hatchet-dev/hatchet

🪓 An orchestration engine for background tasks, AI agents, and durable workflows

Hatchet – Orchestration engine for background tasks, AI agents and durable workflows

What it is – Hatchet is an open‑source platform that lets you run, schedule and monitor asynchronous work at scale. It is positioned as a durable task queue / workflow engine that can power anything from simple fire‑and‑forget jobs to complex, long‑running AI‑agent pipelines.

Why it matters – Many AI applications need reliable background processing (e.g., model inference workers, data‑pre‑processing pipelines, multi‑step LLM‑agent orchestrations). Traditional queues such as Celery or BullMQ trade durability for speed, while workflow systems like Temporal add durability but can be heavyweight. Hatchet aims to combine the two: a PostgreSQL‑backed durability layer with rich scheduling, routing and observability features, while staying lightweight enough to self‑host.

Core capabilities (as described in the README)

Area Features
Background tasks One‑off functions, fire‑and‑forget or fire‑and‑wait, cron & scheduled runs, retries with exponential back‑off, worker‑label routing, event/webhook triggers
Workflow / orchestration Durable tasks that survive crashes, DAG support for data pipelines, built‑in pause/resume via durable sleep or event waits
Scaling & control Priority queues, dynamic rate‑limiting, concurrency policies (fair scheduling), worker slots to cap per‑worker load
Observability Real‑time web UI, alerts, logs, OpenTelemetry integration, Prometheus metrics, multi‑tenant view with roles
Deployment options Managed SaaS (Hatchet Cloud) with autoscaling, multi‑region, SSO, or self‑hosted via Docker/CLI using PostgreSQL as the persistence layer
Language SDKs Official client libraries for Python, TypeScript, Go, Ruby (the README links to a Go reference badge and an NPM package)

Typical use‑cases

  • AI‑agent pipelines – chain LLM calls, tool invocations and external API requests while guaranteeing that each step can be retried or resumed after a crash.
  • Data‑processing jobs – schedule nightly ETL, run DAGs that transform raw data, and keep a full execution history for debugging.
  • Web‑app background work – offload email sending, image processing, or webhook handling to a reliable queue that can be monitored from a UI.
  • Rate‑limited integrations – enforce per‑user or per‑service rate limits when calling third‑party APIs, using Hatchet’s dynamic rate‑limit feature.

Getting started (quick‑start flow from the README)

  1. Try the hosted service – sign up at cloud.hatchet.run to see a fully‑deployed instance.
  2. Run locally – install the Hatchet CLI (macOS, Linux, WSL) which pulls in Docker, then start the server:
    curl -fsSL https://install.hatchet.run/install.sh | bash
hatchet --version
hatchet server start
  3. Write a task – using one of the supported SDKs, define a function and register it with Hatchet.
  4. Submit work – enqueue the task via the SDK or via a webhook/event.
  5. Monitor – open the real‑time UI (served by the local server) to watch execution, logs and metrics.

Full self‑hosting docs and deeper examples are in the official documentation.

Community & support

  • Discord – primary place for real‑time help and discussion.
  • GitHub Issues & Discussions – for bugs and longer‑form technical conversations.
  • Email – for cloud‑related support (billing, data deletion, etc.).

How it differs from related tools (summary of the README comparison)

  • Versus Temporal/DBOS – Hatchet provides comparable durable workflow semantics but bundles observability, rate‑limiting and multi‑tenant UI out of the box.
  • Versus Celery/BullMQ – It adds durability (full execution history) and richer routing/priority features, at the cost of higher resource usage.
  • Versus Airflow/Prefect/Dagster – Focused on high‑throughput application workloads rather than data‑engineer‑centric ETL; supports custom integrations and lower latency.

TL;DR

Hatchet is an open‑source, PostgreSQL‑backed orchestration platform for reliable background processing and AI‑agent workflows. It offers durable task execution, DAG‑style pipelines, sophisticated routing, built‑in observability and both SaaS and self‑hosted deployment options, with SDKs for Python, TypeScript, Go and Ruby.

Related

  • Project
    datahub-project/datahub

    DataHub is an open‑source, AI‑ready metadata catalog (originating from LinkedIn) that unifies data‑asset information across an organization. It offers real‑time streaming ingestion from 80+ connectors, a searchable UI with lineage and governance views, extensive APIs/SDKs, and native support for LLM agents (Analytics Agent). Deployable via Docker, Helm, or SaaS, it enables discovery, impact analysis, compliance, and natural‑language data queries.

  • Project
    johnson7788/MultiUserClaw

    MultiUserClaw is a Docker‑based SaaS framework that lets you run isolated AI assistants (Hermes agents) for many users. It includes a React front‑end, a FastAPI gateway for auth, container management and LLM proxying, per‑user Hermes containers, PostgreSQL storage, and features like knowledge bases, skill store, cron jobs, multi‑channel bots, and multi‑model support.

  • Project
    slothflowlabs/duckle

    Duckle is an open-source ETL platform that lets you author data pipelines on your laptop with a visual canvas, Python, or SQL, then deploy the same file to your own servers, Docker, or CI. It compiles to SQL on DuckDB, runs headless with scheduling, roles, and an audit trail, and offers a local AI assistant — a self-contained alternative to hosted per-row-priced ETL tools.

  • Project
    Aas-ee/open-webSearch

    Open‑WebSearch is an open‑source, key‑free web‑search service that can be run as an MCP server, CLI, or local HTTP daemon. It aggregates results from multiple public search engines, can fetch full article content (with optional Playwright browser rendering), and is designed for AI agents needing live web lookup. Configuration is done via environment variables, and Docker images are provided for easy deployment.

  • Project
    dograh-hq/dograh

    Dograh AI is an open‑source, self‑hostable platform for building and testing voice conversational agents. It provides a visual workflow editor, telephony integrations (Twilio, Vonage, etc.), and a modular LLM/STT/TTS stack that you can replace with any provider. One‑command Docker setup gets the whole system running locally, and Python/Node SDKs let you create agents programmatically. The project is BSD‑2 licensed and backed by a Slack community.