mage-ai/mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.

What it solves

Mage OSS provides a self-hosted development environment for building and running production-grade data pipelines. It simplifies the process of automating ETL (Extract, Transform, Load) tasks and orchestrating data flows by replacing traditional script-based workflows with a visual, modular approach.

How it works

The system uses a notebook-style interface where users can build pipelines block-by-block using Python, SQL, or R. It includes prebuilt connectors for databases, APIs, and cloud storage, allowing users to run jobs manually or on a schedule via cron. It also integrates directly with dbt models for data transformation.

Who it’s for

Data engineers and teams who need to build, debug, and orchestrate data pipelines locally before scaling to a production environment.

Highlights

  • Modular Pipeline Design: Build logic in discrete blocks using multiple languages (Python, SQL, R).
  • Notebook UI: An interactive editor for writing and documenting logic with live data previews.
  • Integrated Scheduling: Support for manual triggers and cron-based scheduling.
  • Visual Debugging: Step-by-step logs and error handling to identify issues quickly.
  • Broad Connectivity: Prebuilt connectors for various cloud storage, APIs, and databases.

Related

  • Project
    agent-sh/agentsys

    A modular runtime and orchestration system for AI agents that automates software development lifecycles through structured pipelines, quality gates, and specialized agent roles.

  • Project
    dynamiq-ai/dynamiq

    An orchestration framework for agentic AI and LLM applications that streamlines the creation of RAG pipelines and multi-agent workflows.

  • Project
    deepset-ai/haystack

    An open-source AI orchestration framework for building production-ready RAG systems, autonomous agents, and LLM applications in Python.

  • Project
    kedro-org/kedro

    A Python framework for creating production-ready data engineering and data science pipelines using software engineering best practices.

  • Project
    SamurAIGPT/Vibe-Workflow

    Vibe Workflow is an open‑source, MIT‑licensed node‑based visual editor for building generative image and video pipelines. It consists of a Next.js front‑end, a FastAPI back‑end that calls MuAPI’s generative models, and a reusable workflow‑builder UI library. You can self‑host it (Docker or manual) or use the hosted SaaS version, and you can extend it with custom nodes to call any AI model or external API.