Skip to content

ConveyorA distributed, push-based task queue

Persistent tasks with at-least-once execution, retries with backoff, scheduling, and priorities, backed by Postgres or an in-memory broker, with no Redis and no polling.

Overview

How it works

Conveyor has three moving parts: your client enqueues tasks, the server owns them, and your workers process them. A durable broker is the source of truth, and tasks are persisted before they're dispatched, so they survive any crash.

Conveyor architecture: the SDK and CLI enqueue into named, prioritized queues held in a multi-node conveyord cluster backed by a durable Postgres or in-memory broker; the cluster pushes queued tasks to a worker's free concurrency slots, which acknowledge back.
  • ClientYour code that enqueues tasks.
  • Server (conveyord)Owns the queues and decides who runs what, when.
  • WorkerYour code that receives tasks from the server and executes them.
  • BrokerThe durable store (Postgres, or in-memory for dev) that is the single source of truth.

Push, not poll

The server pushes tasks to workers the instant work exists and a worker has free capacity, so there's no poll interval to tune and no Redis. Each worker opens one persistent connection, tells the server which queues it serves and how much it can handle, and receives work over that stream. When a worker is saturated it simply stops accepting more, and the extra work waits safely in the broker.

Released under the Apache-2.0 License.