SQLite: The Pragmatic Choice for Durable Workflows and AI Agents

The conversation around durable execution often centers on the need for heavy-duty infrastructure. Recent arguments, such as those from DBOS, suggest that if you already trust your database, you don't need a separate orchestration tier. However, this premise can be pushed even further: for a significant class of durable systems—especially those powering AI agents—SQLite is all you need.

The Architecture of Durable Execution

At its core, durable execution is about the persistence of workflow state. While the compute layer can remain cheap, disposable, and stateless, the progress of the workflow must be preserved. This is typically achieved through an execution log where workflows replay from persisted history and activities can be retried upon failure.

SQLite is an ideal fit for this model because it provides transactional durable state without the overhead of a separate database service. By eliminating the network hop and the operational complexity of a dedicated control plane, developers can maintain workflow progress within a local database file.

Bridging the Gap with Litestream

One of the primary criticisms of SQLite in production is the risk associated with local storage. Litestream solves this by asynchronously streaming SQLite changes to S3-compatible object storage. This creates a hybrid model: state remains close to the runtime for high performance, while a continuous backup exists in the cloud for migration, inspection, and disaster recovery.

It is important to note that because Litestream replication is asynchronous, there is a small window where the most recent writes could be lost if the local volume disappears. For many experimentation and AI workflows, this trade-off is acceptable in exchange for the massive reduction in infrastructure complexity.

Why SQLite Excels for AI Agents

AI-generated workflows are often bursty and experimental. Rather than managing one massive, always-on shared database, a more efficient pattern is to deploy a fleet of tiny servers (in micro-VMs or containers), each with its own self-contained SQLite database and object storage backup.

This approach offers several advantages:

  • Fault Isolation: A failure in one agent's state does not impact others.
  • Cost Efficiency: Reduced reliance on expensive managed database instances.
  • Simplicity: Easier to reason about when each tenant or agent has a discrete unit of state.

The Great Debate: SQLite vs. Postgres

While the "SQLite-first" approach is compelling, it is not a universal solvent. The community discussion highlights a clear divide between those prioritizing simplicity and those prioritizing strict concurrency and scalability.

When to Choose Postgres

Postgres remains the superior choice when your requirements include:

  • High Availability: Synchronous replication and failover capabilities.
  • Shared Scalability: Multiple processes across different machines needing to modify the same data simultaneously.
  • Complex Data Models: Large-scale OLAP data warehousing with intricate permission controls and user roles.
  • Strict Type Systems: Users have noted that SQLite's flexible typing can feel inferior to Postgres's rigorous enforcement.

The Case for SQLite in Production

Conversely, proponents of SQLite argue that it is surprisingly performant for single-node applications. One user reported achieving 7.5k concurrent sessions on a single vCPU with SQLite, whereas Postgres struggled with connection limits. Others have successfully replaced entire SaaS stacks—including billing and issue trackers—with Go and SQLite, citing a 10x reduction in costs and lower tail latency by avoiding "noisy neighbors" in managed cloud environments.

Alternative Perspectives and Tools

Beyond the SQLite vs. Postgres binary, other tools offer specialized advantages for durable state:

  • DuckDB: Recommended for ETL and general scripting where analytical performance is required without the overhead of a server.
  • Temporal: A more robust orchestration engine that uses SQLite for local development but scales to a distributed architecture for high reliability.
  • Cloudflare Durable Objects: A real-world example of this philosophy in action, as they are implemented using a variant of SQLite to provide stateful compute at the edge.

Conclusion

Choosing the right storage engine is often a balance between "YAGNI" (You Ain't Gonna Need It) and strategic over-engineering. While some argue that starting with Postgres avoids future migration pain, the operational simplicity of a local SQLite database backed by Litestream provides a powerful, low-friction starting point. For the modern era of AI agents and disposable compute, the most sensible default may be to start small and only add infrastructure as the state actually demands.

Sources