Waymark: A Durable Workflow Engine for Python
Overview
Waymark is a library designed to orchestrate durable background workflows by compiling code into an intermediate representation rather than replaying it. By leveraging a Rust-based runtime and Postgres as the primary backing store, it allows developers to write standard asynchronous Python code that survives server restarts, crashes, and long-running operations without requiring a separate, complex infrastructure stack.
The Compile-Once Architecture
Unlike traditional durable execution engines such as Temporal or Vercel Workflow, which rely on "replay" mechanisms, Waymark utilizes a compile-once approach. In replay-based systems, workflow code is re-executed from the beginning upon each step, requiring the user to write strictly deterministic code—avoiding non-deterministic functions like random() or datetime.now() within the workflow logic.
Waymark changes this by parsing the Python run() method's Abstract Syntax Tree (AST) into an intermediate representation (IR). This IR is then lowered into bytecode for a custom durable virtual machine. Because the runtime tracks the state of this virtual machine in Postgres, the workflow never needs to re-execute from the start during recovery. This allows developers to use standard Python control flow, including loops, conditionals, and parallel branches, with fewer constraints on determinism.
Core Concepts
Waymark distinguishes between two primary primitives to manage distributed work:
- Actions: These represent the distributed work units, such as network calls, database queries, or any task that may fail and requires independent retries. Actions are executed by nodes and can be configured with specific retry policies and timeouts.
- Workflows: These serve as the durable control flow layer. They orchestrate actions but do not perform heavy lifting themselves. They support standard Python patterns, such as
asyncio.gatherfor parallel fan-out andasyncio.sleepfor durable pauses that persist across process restarts.
Deployment and Usage
Waymark is designed to be "infrastructure-light," requiring only a Postgres database. The system includes a built-in web application, accessible via the waymark-executor, which provides visibility into workflow instances, call timelines, and node load.
While the project is currently in early development and not recommended for production use, it aims to support teams focused on reaching product-market fit by simplifying the background job stack. It is not intended for high-scale environments requiring sub-100ms latency or massive concurrency (e.g., significantly over 10k concurrent actions).
Community Perspectives
The introduction of Waymark has sparked discussion regarding the trade-offs of its design. Some users have noted the potential for tight coupling between the workflow engine and application code, expressing concerns about long-term maintainability as Python versions or third-party dependencies evolve.
Others have highlighted the architectural shift, comparing it to build systems like Bazel or Nix. As one commenter noted:
"Amazing to me that I'm only now seeing the architecture pattern of 'pure coordination language connecting sandboxed imperative components,' which is used by Nix and Bazel, applied to distributed workflows."
Additionally, some developers have pointed out the library's heavy reliance on asyncio, noting that it may be off-putting for teams working primarily with synchronous codebases.