Just Use Postgres: Rethinking Durable Workflow Execution

Durable workflows are a powerful tool for building reliable programs. By regularly checkpointing progress to a database, a program can recover from crashes or failures by reloading from the last completed step—much like saving progress in a video game. Traditionally, this has been achieved through external orchestration, where a central server (like Temporal, Airflow, or AWS Step Functions) coordinates steps and manages state.

However, there is a growing argument that external orchestration is fundamentally overcomplicated. If the core purpose of a durable workflow is to checkpoint state in a database, why introduce a separate orchestrator server? By using the database itself as the orchestrator, developers can simplify their stack, reduce points of failure, and leverage existing database expertise.

The Architecture of Postgres-Backed Execution

In a Postgres-backed system, application servers communicate directly with the database. Instead of a central orchestrator dispatching tasks, the process works as follows:

  1. Submission: A client creates an entry in a Postgres workflows table.
  2. Execution: Application servers poll this table to dequeue and execute workflows.
  3. Checkpointing: As the server executes each step, it checkpoints the output directly to Postgres.
  4. Recovery: If a server crashes, another server can recover the workflow from its last checkpoint.

This design eliminates the need for a central coordinator. Servers cooperatively dequeue workflows using locking mechanisms (such as SKIP LOCKED) to ensure each workflow is handled by exactly one worker. Database integrity constraints prevent duplicate work if multiple workers attempt to execute the same workflow simultaneously.

Key Advantages of the Database-Centric Approach

Scalability and Availability

By shifting orchestration to the database, the system's scalability is tied directly to the database's performance. Postgres can scale vertically to handle tens of thousands of workflows per second, and horizontal scaling can be achieved through sharding or distributed versions like CockroachDB. Because workers are fungible, the system remains available as long as the database is online.

Built-in Observability

One of the most significant advantages is that workflow state is stored in relational tables. This means observability is "free" via SQL. Complex analytical queries—such as identifying all failed workflows in the last month—can be executed declaratively without needing specialized observability tools or key-value store scans.

Reduced Security Surface Area

External orchestrators introduce new points of failure and new security risks, as they often handle sensitive application data. By using Postgres, you reuse existing critical infrastructure. If your application already depends on Postgres, adding durable execution doesn't introduce new infrastructure to harden, audit, or secure.

Community Perspectives and Trade-offs

While the benefits are compelling, the developer community highlights several critical considerations when moving away from dedicated orchestrators.

The "Build vs. Buy" Complexity Trap

Some engineers warn that while the initial setup is simple, the complexity eventually migrates from the orchestrator to the application logic. As a system grows, you may find yourself manually implementing features that dedicated engines provide out-of-the-box:

Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay/debugging semantics, workflow migration, fanout/fanin, long timers, audit trails, and operator tooling, the “just use a database” story becomes “build a poor copy of a workflow engine plus a bunch of workers.”

Correctness and Idempotency

There is a significant debate regarding data correctness. Some argue that simple pseudo-code for durable workflows can lead to data corruption during crashes if not implemented with extreme care. A common alternative suggested is building workflows out of idempotent operations, where the system can simply restart from the top and skip already-completed steps.

Performance at Scale

While Postgres is versatile, some users point out that running a highly durable Postgres instance in the cloud can be expensive. Others suggest that for extreme scale, disk-log based architectures may be more cost-effective than traditional database-backed systems.

Alternative Implementations

Beyond DBOS, several other tools and patterns implement similar philosophies:

  • Absurd: A minimal implementation of durable workflows for Postgres.
  • Oban: A popular job processing library for Elixir that uses Postgres.
  • Custom Rust Implementations: Various teams have built minimal Rust crates to handle durable actors and FSMs (Finite State Machines) within Postgres.
  • SQLite: For smaller workloads, some developers have ported these concepts to SQLite, using simple polling loops instead of LISTEN/NOTIFY.

Conclusion

Moving orchestration into the database is a pragmatic choice for teams already heavily invested in the Postgres ecosystem. It reduces architectural overhead and simplifies observability. However, the trade-off is a shift in ownership: you trade the operational complexity of a separate orchestrator for the development complexity of ensuring correctness and implementing advanced workflow features manually. For many, the ability to keep all data in one place and use SQL for monitoring makes this a winning trade.

Sources