slothflowlabs/duckle

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

What it solves

Duckle is an open-source ETL (Extract, Transform, Load) platform designed to replace expensive, vendor-locked, or per-row-priced data pipeline tools. It allows teams to build data pipelines visually or via code and run them entirely on their own infrastructure—whether on a local machine, a private server, or a cloud instance—without hidden costs or vendor lock-in.

How it works

Duckle uses a visual canvas where you drag and drop components (sources, transforms, sinks) to build data flows. It compiles these visual graphs into efficient, vectorized SQL that runs on the DuckDB engine. You can author pipelines in the desktop app, via Python, or using SQL, and then deploy the same pipeline file to a headless runner for scheduled execution. It includes an optional, local AI assistant that can generate pipeline configurations from natural language prompts without sending data to external servers.

Who it’s for

It is built for data engineers and teams who need high-performance, local-first data processing and want to maintain full control over their infrastructure, audit trails, and data security. It is particularly useful for those who want to avoid the recurring costs of managed ETL services.

Highlights

  • Native Performance: Leverages the DuckDB columnar engine for high-speed, vectorized data processing.
  • Local AI Assistant: Includes an in-process AI assistant (Duckie) that generates pipelines from natural language prompts without requiring API keys or external data transmission.
  • Git-Friendly: Pipelines, connections, and configurations are saved as plain files, making them easy to version control, branch, and audit.
  • Extensive Connectivity: Supports over 400 components out-of-the-box, including databases, SaaS APIs, object storage, and vector stores.
  • Flexible Deployment: Provides both a visual desktop studio for building and a lightweight headless runner for CI/CD, cron jobs, and containerized environments.

Related

  • Project
    TangleML/tangle

    A visual orchestration system for ML experiments and production pipelines that allows users to build and run containerized batch workflows via a drag-and-drop interface.

  • Project
    ucbepic/docetl

    DocETL is a declarative and agentic framework for processing large-scale structured and unstructured data using LLMs, automating the optimization of prompts and models for cost and accuracy.

  • Project
    pathwaycom/llm-app

    A collection of ready-to-deploy AI pipeline templates for high-accuracy, real-time RAG and enterprise search that syncs directly with live data sources.

  • Project
    hatchet-dev/hatchet

    Hatchet is an open‑source orchestration engine that lets you run reliable background tasks, AI‑agent pipelines and durable workflows. It stores all state in PostgreSQL, provides retries, scheduling, DAGs, rate‑limiting, priority queues and a real‑time UI, and offers SDKs for Python, TypeScript, Go and Ruby. You can use it as a managed cloud service or self‑host it via Docker.

  • Project
    dais-labo/flock

    A DuckDB extension that integrates LLMs and RAG pipelines into SQL queries, enabling multimodal semantic analysis and analytics directly within the database.