Problem & Solution
Problem
Background work is easy to lose track of: queues are not durable stores of record, so a crash between the database and the queue loses jobs — and when retry attempts overwrite job state, the history of what actually ran disappears with them.
Solution
Zolt keeps PostgreSQL as the single source of truth with separate jobs and executions tables. The API writes state, a 15-second scheduler claims due jobs into BullMQ, and a worker pool runs only pre-registered handlers, recording every attempt. V0 runs as one scheduler instance; distribution and recovery are V1 work, not claimed here.
Architecture
The API writes job state to PostgreSQL and nothing else. Each tick, the scheduler finds due jobs through a (status, run_at) index, flips them SCHEDULED → QUEUED, and bulk-enqueues into BullMQ. A worker fetches the job row, creates an execution record, runs the registered handler, and writes the outcome back to PostgreSQL.
Engineering Decisions
Jobs ≠ executions
Separate tables keep a stable job identity with per-attempt execution rows (number, timestamps, error). History survives retries instead of being overwritten.
Claim before queue
The scheduler flips SCHEDULED → QUEUED in Postgres — served by a (status, run_at) index — before enqueueing, so the database always reflects reality. Single-instance claiming; SKIP LOCKED is V1.
Closed handlers, validated boundary
Only pre-registered handlers run (SEND_EMAIL); every write endpoint is Zod-validated behind AppError middleware. No arbitrary code, no malformed payloads past the boundary.
Testing & Results
A Vitest suite (run via the repo's vitest script) executes against a dedicated test database, covering execution success, failure handling, and transaction rollback. A recorded checkpoint reported 73 passing tests across 11 files with type-check and build green — historical, not re-verified against the current tree. No load-test or production metrics exist.
Limitations & next steps
- Retries across executions and backoff workflow (V1)
- Concurrent-safe claiming and scheduler reconciliation (V1)
- Execution leases, crash recovery, multi-instance operation (V1)
- Metrics, dashboards, and load tests (V1)