slothflowlabs/duckle

What it solves

Duckle is an open-source ETL (extract, transform, load) platform that lets teams build data pipelines on their own infrastructure instead of paying per-row fees to hosted vendors like Fivetran or Airbyte. It solves the problem of pipelines being locked into a vendor cloud, hard to audit, and dependent on whoever wrote them. Every pipeline is a plain file in git, so it can be reviewed, rolled back, and outlive its author.

How it works

You author pipelines on your laptop using a visual canvas, Python, or SQL. Duckle compiles the graph to SQL and executes it through the DuckDB columnar engine, which uses every core on the machine. The same pipeline file can be deployed headless to a server with duckle-runner serve, run in Docker, in CI, or as a standalone executable. The server console provides scheduling (cron), roles, an audit log, and alerts. Secrets are resolved from the environment or an encrypted workspace store at run time, not stored in the pipeline file. Scaling is horizontal: more cores, more RAM, more concurrent runs, or more worker machines via duckle-runner work. For very large data, pushdown runs the query verbatim inside Postgres, Oracle, SQL Server, or Snowflake while Duckle handles orchestration. It also ships with an optional local AI assistant (Duckie) that writes pipeline JSON from English descriptions, running locally with no API key.

Who it’s for

Data engineers and teams who want to own their ETL infrastructure — running pipelines on their own servers, Docker, or cloud VMs — and who want a free, auditable, self-contained alternative to hosted per-row-priced ETL tools. It’s also useful for teams migrating from legacy visual ETL tools like Talend, since Duckle can import those jobs.

Highlights

• 366 components ready at install time — sources, transforms, sinks, data-quality checks, and control-flow nodes covering files, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS APIs, FTP, and IMAP.
• Self-contained ~65 MB binary — workspaces are plain files in a folder you choose; engines install on first launch.
• Local AI assistant (Duckie) — describe a pipeline in English and it writes the JSON and drops it onto the canvas, running locally with no API key or telemetry.
• Measured performance — 96 million rows out of live Postgres to Parquet in 39.9s; Oracle extract at 65.0s, faster than python-oracledb with pyarrow on the same machine.
• Deployment flexibility — run headless on a server, in Docker, in CI, or as a standalone executable; console has roles, audit log, and alerts.
• Pushdown — for data bigger than any box, the query runs verbatim inside Postgres, Oracle, SQL Server, or Snowflake while Duckle orchestrates.
• Talend job import — converts legacy visual ETL jobs into Duckle pipelines, with a report of what still needs a person.
• CI-ready — duckle-runner validate compiles pipelines to SQL without opening sources or needing credentials, gating every push.