Operations guide

How to deploy, configure, scale, secure, observe, and upgrade conveyord.

Administration is driven by the conveyor CLI and the dashboard: rescheduling, running, canceling, deleting, and archiving tasks, pausing and resuming queues, setting rate and concurrency limits, and managing cron are operator actions, kept deliberately out of the SDK. The SDK is the produce and consume surface for application code; operating the system is the CLI and dashboard. For the full command-by-command CLI reference, see the CLI reference.

Deployment modes

One binary, selected by mode (or --mode):

Mode	Discovery	Broker	Use
standalone	self	Postgres (or in-memory with --dev)	a single node, dev, edge
cluster	static peer list	Postgres	VMs / bare metal
kubernetes	pod-label discovery via the API server	Postgres	the flagship mode
embedded	self	memory or Postgres	a Go package run in your process

Clustering is always compiled in: standalone is a cluster of one running the same code path. Artifacts ship under deploy/: a distroless Dockerfile, a Helm chart, a systemd unit, and Compose files. For an end-to-end walkthrough that ties the server, Postgres, and worker tiers together, see high availability.

Configuration

Precedence, lowest to highest: defaults → config file → CONVEYOR_* environment → flags. ${VAR} in the file is expanded from the environment.

conveyord --config=/etc/conveyor/conveyor.yaml
conveyord --mode=kubernetes --config=/etc/conveyor/conveyor.yaml
conveyord --dev   # standalone + in-memory broker + auth off + debug logs

Environment keys mirror the file with CONVEYOR_ and __ between levels. broker.dsn is CONVEYOR_BROKER__DSN, cluster.bind_addr is CONVEYOR_CLUSTER__BIND_ADDR.

Key groups:

• broker.driver (postgres | memory), broker.dsn, and broker.pool.{max_conns,min_conns,connect_timeout,statement_timeout}.
• api.listen (default :8080), api.auth_tokens, api.scoped_tokens, api.tls, and the dashboard settings api.dashboard, api.cors_origins, api.grafana_url, and api.read_only.
• webhooks.allow_private_targets (default false; permits webhook delivery to private and loopback endpoints).
• cluster.discovery, cluster.bind_addr, the remoting/discovery/peers ports, cluster.tls, and cluster.kubernetes (namespace + pod labels).
• engine.lease_ttl, reap_interval, lease_batch_max, promote_interval, passivate_after, default_max_retry, archive_retention, shutdown_timeout.
• engine.rate_limit_enabled (master switch, default true), engine.rate_limit_rate_per_sec and engine.rate_limit_burst (the global default per-queue dispatch limit; per-queue overrides are set at runtime, see rate limiting).
• metrics.listen (default :9464; empty disables the endpoint).
• otel.endpoint (OTLP push for metrics + traces), otel.service_name.
• log.level, log.format.

The Helm chart renders the full configuration into a ConfigMap from deploy/helm/conveyor/files/conveyor.yaml, so that file is also a complete annotated reference.

Scaling

The server is stateless. Durable state lives in the broker. Scale it horizontally:

• More server nodes spread queue ownership and worker sessions across the cluster and survive node loss (a lost node's queues re-activate elsewhere and its in-flight tasks are redelivered). On Kubernetes raise replicaCount. Run at least three nodes in production; see the high-availability guide for why three is the floor.
• More worker capacity comes from running more worker processes or raising a worker's WithConcurrency. Workers are independent of the server cluster.
• The broker is the throughput ceiling. Conveyor commits every task to Postgres before dispatch, so sustained throughput is bounded by the database, not the server. Size the connection pool and the database accordingly, and measure the broker first when tuning.

Priorities and weights shape what runs first: per-task Priority(1..9) orders within a queue, and per-queue weights bias a worker that serves several queues.

Broker sizing (Postgres)

• Size the connection pool with broker.pool.max_conns (and min_conns, connect_timeout, statement_timeout); every replica opens its own pool against the same database, so the database must admit replicas × max_conns. A zero value keeps the driver default. statement_timeout makes a runaway query fail instead of holding a pool slot and stalling dispatch.
• Tasks accumulate rows in the task log. Completed tasks are purged once their per-task Retention lapses (the default is immediate). Archived (dead-lettered) and canceled tasks are kept for engine.archive_retention (default 7 days; 0 keeps them forever) so they can be inspected via the Admin API/CLI before the reaper purges them.
• engine.lease_ttl bounds how long a crashed worker's task waits before redelivery; engine.reap_interval is how often the reaper reclaims expired leases (recovery time after a failure is roughly 2 × reap_interval).
• engine.lease_batch_max caps how many tasks one dispatch cycle claims. Raise it for high-throughput queues, lower it to smooth load.

Security

• Authentication. api.auth_tokens are accepted bearer tokens. Auth is on by default: with no tokens, conveyord refuses to start unless you set api.allow_unauthenticated: true, so a deployment never serves an open API by accident. The --dev preset sets that flag for you; in production set api.auth_tokens instead (the Helm chart's auth.tokensSecret), and only use allow_unauthenticated when a gateway, mTLS, or a private network fronts the API. Clients and workers pass a token with conveyor.WithToken (or CONVEYOR_TOKEN / the CLI --token).

• Token scopes. A token in api.auth_tokens grants everything. To hand out a narrower credential, declare it under api.scoped_tokens instead, where each entry pairs a token with the services it may call: produce for enqueueing, consume for worker sessions, and admin for the administrative API. A recognized token used outside its scopes is refused with PermissionDenied. Scoped tokens are file-only, since each carries its own scope list. Give an application produce, a worker fleet consume, and keep admin for operators. Task inspection (GetTask, which conveyor tasks get uses) is the one call two scopes admit: it lives on the enqueue service, but admin reaches it too.

  api:
    scoped_tokens:
      - token: "${PRODUCER_TOKEN}"
        scopes: [produce]
      - token: "${WORKER_TOKEN}"
        scopes: [consume]

• Webhook targets. Webhook worker endpoints must resolve to public addresses; loopback, link-local, private, and multicast targets are refused so an admin token cannot aim the server at internal services. Set webhooks.allow_private_targets: true when your endpoints are private by design, such as a worker inside the same cluster. See webhook workers.

• TLS. api.tls serves the API over TLS; cluster.tls turns on mutual TLS between cluster peers (set ca_file for peer verification).

• Network. The Helm chart ships an opt-in NetworkPolicy example and keeps the metrics port off the public API listener. Never expose the metrics port (:9464) publicly, since it carries internal topology.

Dashboard

conveyord embeds a read+write operations console, served at the API root and on by default. The dashboard guide covers what each view shows and the actions it offers, signing in, read-only mode, hosting the UI on another origin, and the api.dashboard, api.cors_origins, and api.grafana_url settings.

Observability

• Health. /healthz (liveness) and /readyz (readiness: broker reachable and engine running) on the API port. Wired into the chart's probes.
• Metrics. Prometheus exposition at /metrics on metrics.listen (:9464): conveyor_enqueued_total, …_completed_total, …_failed_total, …_retried_total, …_archived_total, …_released_total, conveyor_active, conveyor_sessions_active, conveyor_pending, plus the health canaries conveyor_lease_expired_total (workers losing leases), conveyor_breaker_open_total (a failing task type), conveyor_events_dropped_total (a slow watcher), and conveyor_maintenance_failures_total{pass} (a reaper, scheduler, or sweeper pass that failed and was skipped until its next tick), and runtime metrics. The chart stamps prometheus.io/scrape annotations and ships an opt-in ServiceMonitor and an opt-in PrometheusRule (prometheusRule.enabled) that alerts on those canaries, pending backlog, and no node exposing metrics; deploy/grafana/ has a dashboard and scrape config.
• Tracing. Set otel.endpoint to push OTLP traces to a collector. Each enqueue opens a span and stamps a W3C traceparent into the task; if your worker process has OpenTelemetry configured, its execution span links back to the enqueue.
• Lifecycle events. A push stream of per-task state transitions for live dashboards, alerting, audit logs, and event-driven chaining; see lifecycle events.
• conveyor cluster info reports cluster membership.

Upgrades & restarts

• Graceful shutdown. On SIGTERM the node drains live worker sessions (releasing in-flight tasks for redelivery) before stopping, bounded by engine.shutdown_timeout. On Kubernetes, terminationGracePeriodSeconds must exceed shutdown_timeout (the chart sets this) so the drain completes before SIGKILL.
• Worker deploys are free. When a worker process shuts down (cancel its Run context, e.g. on SIGTERM), any task it was running is handed back with no retry penalty and no backoff, so it becomes due immediately on another worker rather than counting as a failed attempt. So rolling out a new worker build does not eat into tasks' retry budgets or delay them. A genuine worker crash is different: it is recovered by lease expiry and does count as a retry, which bounds a task that repeatedly kills its worker.
• Rolling restart. Because execution is at-least-once, redelivery during a restart is always safe, so design handlers to be idempotent. The StatefulSet rolls one pod at a time; a PodDisruptionBudget keeps a quorum available. Workers reconnect with jitter to the API Service and keep processing while a node is replaced; tasks held by a restarting node are reclaimed by lease expiry and redelivered. The kind e2e (make e2e) drives load through a full rolling restart and asserts zero task loss.
• Version-skew policy. The wire protocol is additive, so a newer server serves older workers: roll the server tier first, then workers. During a rolling restart the cluster runs mixed server versions briefly; keep upgrades to one minor version at a time. Full mixed-version cluster testing is deferred past v1, so do not run a cluster on more than one server version longer than a rollout takes.
• Schema migrations run automatically on Postgres connect; no manual step.