Architecture

This document explains how Conveyor works on the inside, for contributors and SDK authors. It names the real building blocks (the actor runtime, grains, and broker) that the user-facing README deliberately hides behind plain terms. For the user-level "what is a task queue" picture, see the README's diagram; this document is the engine room.

Mental model

Conveyor is a push-based, durable task queue built on the GoAkt actor framework. Three ideas carry the whole design:

1. The broker is the only durable state. Every task, lease, schedule, and queue flag lives in the broker (Postgres in production, in-memory for dev). Nothing else persists anything.
2. Actors are stateless and rebuildable. The dispatch logic runs in actors and grains that hold only in-memory bookkeeping. On restart, relocation, or failover they rebuild their state from the broker, so losing a node loses no work.
3. Work is pushed, never polled. A worker opens one long-lived stream and declares its capacity; the server leases due tasks and streams them out as capacity frees up. There is no poll interval. (The internal maintenance sweeps described below are recovery backstops, not the dispatch path.)

Component overview

Everything above lives in one conveyord process; clustering replicates the process across nodes (see Clustering & HA). Worker and producer processes are external and speak ConnectRPC; a webhook endpoint is any external HTTP service that receives signed JSON-RPC deliveries and answers with the outcome (see Webhook workers).

The actors

All actor code is in internal/actors. GoAkt gives us three flavors of unit: plain actors (one instance, mailbox-serialized), grains (virtual actors, exactly one live activation cluster-wide, addressed by name, activated on demand), and cluster singletons (one instance across the whole cluster, relocated on node loss).

One rule governs every one of them: a turn never waits on a grain. GoAkt runs actors and grains on a single dispatcher pool sized to the CPU count (floored at two), and a tell to a grain returns only once the grain has run the message. A turn that waited inside such a tell would hold a pool worker while waiting for a grain that needs a worker to run; with as many waiting turns as workers, every grain on the node stalls until the request timeouts, and the maintenance ticks re-create the wedge on their next pass. So every grain call an actor makes goes through tellQueueGrain in grain_tell.go: it resolves the grain by queue name and tells it on its own goroutine, logging a failure rather than returning one, since every such message is a hint the maintenance sweeps recover from. Resolving by name on every call (a registry lookup for an active grain) means an actor caches no grain identity, so a completion can never be dropped for arriving before a cached identity did. Grain-to-actor sends are asynchronous already, and the API's Engine.TellQueue runs on request goroutines, which are not pool workers. TestActorsNeverWaitOnGrainsInsideTurns pins the pool at its floor and fails within seconds if a turn waits.

Engine (engine.go): plain type

The coordination layer and the enqueue entry point the API calls. It builds the GoAkt actor system, installs the Runtime extension, registers the singleton kinds and the QueueGrain grain kind, and starts the system with a detached context (context.WithoutCancel, so a request- or signal-scoped context can never tear down cluster remoting).

Engine.Enqueue assigns a ULID if absent, commits to the broker, then wakes the queue. Waking is coalesced: a per-queue queueWaker collapses an enqueue burst into at most one in-flight TasksAvailable message to the grain (the grain drains the whole broker on each wake, so extra wakes would be wasted). A lost wake is backstopped by the reaper's pending-count sweep.

Runtime (runtime.go): actor-system extension

A shared service object (extension id "broker") that every actor and grain resolves on start. It hands out the broker, the injected clock, server settings, the logger, metric counters, the lifecycle event bus, a monotonic ULID source, and (once the system is up) the node's dependency-resolver pool. It is how stateless actors reach durable state without holding a reference of their own.

QueueGrain (queue_grain.go): grain (one per queue)

The per-queue dispatcher, and the heart of the system. Exactly one activation per queue exists cluster-wide; it is activated on demand and passivated when idle. On activation (OnActivate) it rebuilds all state from the broker: the persisted pause flag, the rate limiter, and the per-key concurrency limit. It holds no durable state, so OnDeactivate is a no-op.

It reacts to wakes (TasksAvailable), gateway registrations and credits, and completion messages. Its core loop, maybeLease, runs whenever the queue is unpaused, no lease cycle is already in flight, and credits are available: it leases up to min(credits, batchMax) due tasks (further capped by available rate-limiter tokens), then distributes them across the registered gateways in proportion to their declared weights (smooth weighted round-robin over the gateways that still have credits), decrementing one credit per dispatched task. A gateway that declared no weight counts as weight one, so an unweighted fleet falls back to plain round-robin. A leased task whose concurrency key is already at the queue's per-key limit is held back (released to redeliver rather than dispatched), so a keyed queue never runs more than its limit per key. Leasing happens off the mailbox turn via PipeToSelf so the grain never blocks on the broker.

Gateway (gateway.go): actor (one per worker session)

The bridge between the actor world and one worker's stream. Spawned per accepted session, long-lived (must not passivate while the stream is open) and relocation-disabled (it is bound to a node-local stream and dies with its node). It is the only component that performs durable execution transitions for its worker's tasks.

• On start and every 30 s (registerTick) it announces its capacity to each queue it serves via RegisterGateway; this re-announcement is what heals a grain that relocated to another node. The capacity announced to a queue is the worker's total concurrency split across the queues it serves in proportion to their weights (splitCapacity), so the worker holds at most concurrency tasks in flight across all its queues, not that many per queue. Each queue keeps a guaranteed weighted share (floored at one slot), which is what gives cross-queue fairness.
• It pushes Dispatch/BatchDispatch frames to the worker and records each as in-flight under a lease id.
• On a Result it maps the outcome to a broker call: Ack (success), Fail with backoff or Archive (retry / exhausted / SkipRetry), or Release (graceful drain), all scoped to the delivery's lease id; a lost lease is logged and dropped.
• A Heartbeat extends every in-flight lease; a lost lease cancels that task on the worker.
• A per-task-type circuit breaker can defer a completion (and its credit refill) briefly when a type is failing, throttling it to probe speed.

Webhook workers (webhook_gateway.go): singleton + per-registration actors

Webhook workers let an external HTTP endpoint process tasks with no SDK: the server leases due tasks to a registered URL and pushes each as a signed JSON-RPC call. Two units implement them, both in webhook_gateway.go.

• WebhookManager: a fourth cluster singleton (registered alongside the maintenance loops). On a reconcile tick it lists the persisted registrations from the broker and converges its children onto them: it spawns a gateway for each active registration, refreshes the snapshot of one that changed, and drains the child of one that was paused, deleted, or lost its secret. Because it is a singleton, a relocated manager rebuilds every gateway on its new host. A registration with no secret is skipped: the gateway signs deliveries and mints lease tokens with the newest secret, so it cannot start without one.
• WebhookGateway: the webhook analog of the per-session Gateway, one plain actor per registration (long-lived, relocation-disabled). A queue grain dispatches to it exactly like any gateway; instead of pushing a stream frame it POSTs a JSON-RPC call to the endpoint off its mailbox turn, maps the response to the durable transition (Ack/Fail/Archive/Release), and reports completion for the credit refill. It re-registers its capacity every tick like the stream gateway, extends the leases of open synchronous deliveries, and carries the same per-task-type circuit breaker. A delivery the endpoint answers accepted parks in asynchronous mode: its slot stays held and its lease is driven by the endpoint's Heartbeat callbacks until a ReportResult callback (both served by WebhookService and routed here by the engine) resolves it, or it stops beating and lease expiry reclaims it. A per-endpoint circuit breaker withholds capacity (announces zero, then a single probe slot) when transport failures pile up, so a dead endpoint's queue stays pending instead of churning.

Cluster singletons: maintenance loops

Three singletons run on one node (the leader) and relocate to a survivor on node loss. Each arms its own recurring tick in PostStart, so after relocation the new host re-arms the cadence; the stale entry on the departed node self-cancels. Each tolerates ErrSingletonAlreadyExists on non-leaders as the desired state.

• Scheduler (scheduler.go), on PromoteTick: promotes due scheduled tasks to pending (PromoteScheduled), materializes due cron entries into real tasks, and wakes affected queues.
• Reaper (reaper.go), on ReapTick: reclaims expired leases (ReapExpiredLeases → retry or archive), purges retention-lapsed terminal rows (PurgeTerminal: completed rows by their per-task retention, archived and canceled rows by the server-wide engine.archive_retention, dropping the stale dependency edges of terminal dependents first), archives tasks past their pre-dispatch TTL (ArchiveExpired), promotes blocked tasks whose dependencies have since reached a terminal state but that inline resolution missed (PromoteReadyDependents), and sweeps for queues with due work whose wake was lost (PendingCount), waking each. It runs under GoAkt's default (stop-on-failure) supervision, so it deliberately logs and skips a failed pass (leaving the next tick to retry) rather than escalating a transient broker error into a crash that would permanently stop all maintenance.
• GroupSweeper (group.go), on GroupSweepTick: reads GroupStats and fires aggregation groups that are past a size, max-delay, or grace-period threshold by telling the owning queue grain FireGroup.

DependencyResolver (resolver.go): per-node router pool

Completion-time dependency resolution runs here, off the gateway and grain turns. A bounded pool of stateless routees sits behind a per-node round-robin router: when a task reaches a terminal state its gateway hands a ResolveDependents to the router with a non-blocking tell, and a routee runs the reconciling broker transaction, promoting each dependent whose dependencies are now satisfied and applying each failed edge's on-failure policy (block, continue, or cascade-cancel). It then wakes the queues that gained work. The pool is node-local and its size is configurable; resolution is best-effort, so a failure is only logged, and a node with no pool (or any missed resolution) falls back to the reaper's PromoteReadyDependents sweep.

The broker

The Broker interface is the sole stateful layer; actors and API handlers never touch storage directly. Two implementations back it, memory and postgres, and both must pass the single brokertest conformance suite, which drives all time-dependent behavior through an injected fake clock (no sleeps) so the two stay semantically identical. Brokers must be concurrency-safe and derive "now" from the injected clock, never the system or DB clock.

The interface methods group as:

Group	Methods
Enqueue	Enqueue, EnqueueBatch (idempotent on id; ErrDuplicateTask on a live unique key; the batch commits all-or-nothing)
Lease / dispatch	Lease, LeaseGroup, ExtendLease, SetProgress
Outcomes (lease-scoped)	Ack, AckBatch, Fail, Release, Archive
Maintenance sweeps	ReapExpiredLeases, PromoteScheduled, PurgeTerminal, ArchiveExpired
Dependencies	ResolveDependents, PromoteReadyDependents
Inspection / admin	PendingCount, QueueStats, GetTask, ListTasks, CancelTask, DeleteTask, RunTaskNow, RescheduleTask, ArchiveTask, SetQueuePaused, QueuePaused, Info
Rate limits (config only)	SetQueueRateLimit, DeleteQueueRateLimit, QueueRateLimit, QueueRateLimits
Concurrency limits (config only)	SetQueueConcurrencyLimit, DeleteQueueConcurrencyLimit, QueueConcurrencyLimit, QueueConcurrencyLimits
Groups	GroupStats, SetGroupConfig, DeleteGroupConfig, GroupConfigs
Cron	UpsertCronEntry, ListCronEntries, ListDueCronEntries, SetCronPaused, UpdateCronNextRun, DeleteCronEntry
Webhook registrations	UpsertWebhookWorker, GetWebhookWorker, ListWebhookWorkers, SetWebhookWorkerPaused, DeleteWebhookWorker
Lifecycle / events	SetEventSink, Close

Persistence model

A task row stores its identity and options plus mutable execution fields (state, retried, last_error, lease_id, lease_expires_at, timestamps). The serialized TaskEnvelope is marshaled into a payload column before dispatch, and that is what makes execution crash-safe. The mutable fields are authoritative in their own columns and are overlaid onto the envelope on read, never written back into the stored blob.

• Leases are a (lease_id, lease_expires_at) pair. Lease-scoped operations match on state = active AND lease_id = ? and return ErrLeaseLost on mismatch. Postgres leasing uses SELECT ... FOR UPDATE SKIP LOCKED in a CTE, so concurrent leasers on different nodes never claim the same row.
• Uniqueness is a partial unique index on unique_key over the incomplete states only; a duplicate maps to ErrDuplicateTask. Lapsed claims are freed before insert and by PurgeTerminal.
• Three distinct TTLs, often confused: expires_at is a pre-dispatch TTL (a still-waiting task past it is archived, and the lease query skips it); deadline cancels an already-running task (cooperative, enforced above the broker); retention is how long a completed row is kept before purge; archived and canceled rows follow the server-wide engine.archive_retention instead.

Encryption decorator

internal/broker/encrypted wraps any Broker so callers see plaintext while storage sees only ciphertext: the server-side, zero-code-change encryption seam. It implements every method by hand (a compile-time var _ broker.Broker assertion) so a future payload-bearing method cannot silently bypass encryption. It seals on write (the Enqueue/EnqueueBatch payloads, the Ack result, and the UpsertCronEntry payload) and opens on read on a clone (Lease, LeaseGroup, GetTask, ListTasks, and the cron list reads). Note there is no result read path in the interface, so Ack seals the result but no symmetric decrypt is needed. This server-side seam is an alternative to SDK end-to-end encryption: a deployment uses one or the other, never both.

Task lifecycle

The task states are defined in protos/conveyor/v1/task.proto. The diagram reads top to bottom: a task lands in one of the pre-dispatch waiting states, is leased into active, and ends in a terminal state. The "single lease path" picture gets two points wrong:

• There are two lease paths. pending and retry tasks are leased individually (Lease); a retry is dispatched straight from retry, not funneled back through pending. A fired aggregation group is leased as a batch straight from aggregating (LeaseGroup) and never passes through pending.
• canceled is a pre-dispatch outcome only. Any waiting task can be canceled (by an admin, or by a dependency that failed under the cascade-cancel policy). An admin cancel of an already-running task cannot undo it: the attempt is aborted and lands in archived, not canceled.

A task with unmet dependencies starts blocked and, once they resolve, is promoted to pending (or to scheduled/aggregating when it is also delayed or grouped). There is no separate "expired" state: a pre-dispatch expiry resolves to archived with last_error = "task expired before dispatch".

The diagram reads top to bottom. The box holds the pre-dispatch waiting states; a task is born into one of them (pending when due, scheduled when delayed, aggregating when grouped, blocked when it has unmet dependencies), and scheduled and blocked promote to pending once they are ready. Dispatch leases the task out to active: Lease for an individual pending or retry task, LeaseGroup for a fired group. A failed attempt returns it to the box as retry, a graceful drain as pending. Any waiting task can leave early: an admin (or a cascade from a failed dependency) cancels it to canceled, and one past its pre-dispatch TTL is archived. Admin actions can also reschedule a waiting task to a later time or revive an archived one. Transitions labeled admin are operator actions; the rest are engine-driven.

Key flows

Enqueue → dispatch (credit-based push)

Credits are the flow-control currency. The server seeds a session's credits from the worker's declared concurrency, split across the queues it serves by weight so the total across queues equals concurrency, and refills exactly one per completion; the optional Credit frame exists for workers that open slots without finishing a task, but the happy path never needs it. A delivery whose lease is lost to another delivery (the worker missed its heartbeats and the reaper reclaimed the task) refunds its credit when the loss is detected, since the worker's eventual result for it is dropped.

Lease recovery & relocation

• Worker disconnect (graceful or crash). When a session ends, the handler asks the gateway to drain: as a serialized mailbox turn it Releases every in-flight task with no retry penalty, so they become due immediately elsewhere.
• Hard death (panic, node loss). Anything that slips past the drain is recovered by lease expiry: the reaper's ReapExpiredLeases turns the stale active tasks back into retry (or archived if exhausted) and wakes the affected queues.
• Grain relocation. A queue grain is a virtual actor; on node loss GoAkt re-activates it elsewhere and OnActivate rebuilds its state from the broker. Lost credits and registrations are re-established by the gateways' 30 s re-registration, and pending work is re-driven by the reaper's sweep.
• Singleton relocation. Scheduler, Reaper, and GroupSweeper relocate to a survivor and re-arm their ticks on PostStart.

Scheduling, cron, and groups

Delayed tasks enter scheduled and are promoted by the Scheduler when due. Cron entries are server-persisted; the Scheduler materializes due ones into real tasks and advances the next-run time with a compare-and-set (UpdateCronNextRun) so a relocating scheduler cannot double-fire a slot. Group members accumulate in aggregating purely in the broker (no per-group actor); the GroupSweeper decides when a group is due and tells the queue grain to lease the whole group with LeaseGroup and deliver it as one ExecuteBatch, consuming a single credit for the entire batch.

API & wire protocol

The wire contract is one proto, protos/conveyor/v1/service.proto, served over a single ConnectRPC port that speaks gRPC, gRPC-Web, and HTTP/JSON. Four services (server/api):

• TaskService, the producer API: Enqueue, EnqueueBatch and EnqueueTx (both capped at 1000; EnqueueBatch reports per-item results, EnqueueTx commits all-or-nothing), GetTask.
• WorkerService: Session, the long-lived bidirectional stream that is the push channel. The handler bridges the stream to a per-session Gateway actor. The first frame must be Hello (queues→weights, concurrency, labels, SDK version, minimum server version, batch types); the server replies Welcome (session id, lease TTL, heartbeat interval = TTL/3, server version, minimum SDK version). Worker→server frames: Hello, Credit, Result, Heartbeat, BatchResult, Progress. Server→worker frames: Welcome, Dispatch, Cancel, Ping, BatchDispatch.
• AdminService: inspection (reads straight from the broker) and mutation (broker write plus a live-grain nudge), including pause/resume, rate and concurrency limits, aggregation-group configs, cancel/run/delete/reschedule/archive and their batch forms, cron management, webhook registration management, a best-effort cluster/worker view, and the WatchEvents live lifecycle-event stream.
• WebhookService: the asynchronous-completion callback surface for webhook workers. An endpoint that answered a delivery accepted calls Heartbeat (extend the lease) and ReportResult (finish the task) here, authenticated by the delivery's lease token alone, with no API bearer token. The handler verifies the token and routes the callback to the owning WebhookGateway.

The normative, language-agnostic version of this contract, for authors building an SDK in a new language, is docs/protocol.md.

Clustering & HA

Clustering is always on; a node with no peers is simply a cluster of one on the identical code path. The Engine builds the actor system with a minimum quorum of 1 and a replica count of 1, registering the singleton kinds and the queue-grain kind. Remoting can be secured with mutual TLS.

Discovery is injected from config through a small SPI in server/discovery.go; DiscoveryProvider mirrors GoAkt's discovery interface without leaking any GoAkt type, so a third-party provider can be registered by name and compiled against this package alone. Built-in selections are static (explicit peers, or self for a cluster of one) and kubernetes (namespace + pod-label + named ports).

The four run modes (standalone, cluster, kubernetes, embedded) are conventional config bundles, not distinct code paths. config.Mode is a label that is validated and logged but does not branch behavior; the real differences come from broker.driver and cluster.discovery. The embedded package is the exception: it reuses server.Server in-process over loopback (default in-memory broker, auth off), handing back real SDK Client/Worker handles. Because it binds only ephemeral loopback ports it is always a cluster of one and cannot join other nodes, so embedded can be durable (pass a Postgres DSN) but is never highly available. HA requires the multi-node deployment: several conveyord nodes clustered over a shared Postgres broker, where singletons relocate and queue grains re-activate on a survivor with no task loss.

Security

• Auth fails closed. A bearer-token interceptor wraps every unary call and incoming stream, comparing tokens in constant time. Outside --dev, config validation refuses to start a server that has no tokens unless allow_unauthenticated is set explicitly; the server logs a loud warning when auth is off.
• Read-only mode. When api.read_only is set, a second interceptor on AdminService rejects the mutating procedures with PermissionDenied while reads, enqueue, and the worker stream pass through.
• Cluster mTLS. When a cert is configured, intra-cluster remoting runs at TLS 1.3; adding a CA turns on mutual verification.

Where things live

Path	Responsibility
internal/actors	Engine, runtime extension, gateway, queue grain, scheduler, reaper, group sweeper, dependency resolver, webhook manager & gateway
internal/broker	Broker interface; memory, postgres, encrypted implementations; brokertest conformance suite
internal/backoff, internal/clock, internal/cron	Retry backoff, injectable clock, cron parsing
internal/wire	ConnectRPC transport plumbing (h2c client, bearer interceptor)
server	Server assembly, config, discovery SPI, telemetry, dashboard wiring
server/api	The three ConnectRPC service handlers and the worker-session state machine
cmd/conveyord, cmd/conveyor	The server binary and the CLI
embedded	In-process server for Go applications
protos	The proto contract (task model + services)
sdks	Go, TypeScript, and Python SDKs

For deployment and configuration, see the operations guide. For the wire contract, see the protocol spec.