datahub-project/datahub

The Context Platform for your Data and AI Stack

DataHub – Open‑Source AI‑Ready Metadata Catalog

What it is – DataHub is an open‑source “metadata platform” that builds a unified graph of all the data assets in an organization (tables, dashboards, ML models, pipelines, etc.). It was created at LinkedIn, open‑sourced in 2020, and is now maintained by the DataHub project and a community of contributors.

Why it matters – Modern data stacks are fragmented across many tools. Finding the right dataset, understanding where it comes from, and enforcing governance is hard. DataHub solves this by continuously ingesting metadata from dozens of sources, storing it in a real‑time graph, and exposing rich APIs that both humans and AI agents can query.

Core capabilities (as described in the README)

  • Real‑time streaming ingestion – Metadata updates flow through Kafka in seconds, keeping the catalog fresh.
  • 80+ connectors – Pull or push metadata from warehouses (Snowflake, BigQuery, Redshift), lakes, BI tools, dbt, Airflow, ML platforms, etc., capturing schema, lineage, usage stats, profiling, and quality metrics.
  • AI‑ready – Native support for LLM‑powered agents via the Model Context Protocol (MCP) and the new open‑source Analytics Agent that can answer natural‑language data questions with SQL, results, and charts.
  • Rich UI – Search, column‑level lineage, dataset profiles, governance dashboards, tags/terms/domains, and a “universal search” across the whole stack.
  • Developer‑first APIs – GraphQL, OpenAPI, Python & Java SDKs, and a CLI (datahub) for automation.
  • Deployment flexibility – SaaS (DataHub Cloud), Docker‑compose quick‑start, or production‑grade Helm charts for Kubernetes. The platform includes GMS backend, Elasticsearch, MySQL, and Kafka.
  • Enterprise‑grade security – Authentication, authorization, audit trails, and fine‑grained policies.

Typical use cases

Use case How DataHub helps
Data discovery Universal search lets analysts locate any asset instantly; UI shows schema, owners, and documentation.
Data lineage & impact analysis Column‑level lineage graphs reveal upstream sources and downstream consumers, useful for change‑impact assessments.
Governance & compliance Tagging, domain classification, and policy dashboards let teams enforce data‑privacy rules (e.g., PII handling).
AI agent context The catalog supplies up‑to‑date metadata to LLM agents via MCP, enabling reliable natural‑language queries (the open‑source Analytics Agent).
Automation & CI/CD Ingestion recipes (YAML) can be run from CI pipelines; the Python SDK lets programs add documentation or custom properties programmatically.
Self‑service analytics Users can ask plain‑English questions; the Analytics Agent generates SQL, runs it, and returns results/charts.

Getting started (quickest path)

  1. Install the CLI – brew install datahub-project/tap/datahub (macOS/Linux) or pip install acryl-datahub.
  2. Run the Docker quick‑start – datahub docker quickstart. This spins up the full stack (GMS, UI, Elasticsearch, MySQL, Kafka) and loads sample data.
  3. Open a browser at http://localhost:9002 (default credentials datahub / datahub).
  4. Connect your own sources using the ingestion guides (e.g., Snowflake recipe shown in the README) or the Python SDK.

For production you can use the Helm charts under docs/deploy/kubernetes.md or the managed SaaS offering (DataHub Cloud).

Community & resources

TL;DR

DataHub is a production‑grade, open‑source metadata catalog that centralises information about every data asset, keeps it up‑to‑date via streaming ingestion, and exposes powerful APIs for both humans and AI agents. It can be run locally in minutes with Docker, installed via pip/Homebrew, or deployed at scale with Helm, making it a practical foundation for data discovery, governance, and LLM‑driven analytics.

Related

  • Project
    hatchet-dev/hatchet

    Hatchet is an open‑source orchestration engine that lets you run reliable background tasks, AI‑agent pipelines and durable workflows. It stores all state in PostgreSQL, provides retries, scheduling, DAGs, rate‑limiting, priority queues and a real‑time UI, and offers SDKs for Python, TypeScript, Go and Ruby. You can use it as a managed cloud service or self‑host it via Docker.

  • Project
    chenjigang4167/testhub_platform

    TestHub is an open‑source, AI‑enhanced test‑management system built with Django (backend) and Vue 3 (frontend). It covers requirement analysis, test‑case creation & review, API testing, web‑UI automation (Selenium/Playwright), Android automation (Airtest), performance testing (Locust), and a data‑factory toolbox. AI models (OpenAI, Anthropic, Gemini, etc.) can auto‑generate cases and drive browsers via a “smart mode”. The stack uses MySQL, Redis, Celery, Channels, JWT auth, and Allure reporting. The README provides full install steps, Docker support, and contribution guidelines.

  • Project
    NVIDIA-AI-Blueprints/aiq

    A deployable research backend that provides cited answers and in-depth reports through orchestrated shallow and deep research agents.

  • Project
    openhackai/OpenHack

    OpenHack is an open‑source, LLM‑driven security scanner that analyses a codebase through a recon‑→‑hunter‑→‑validator‑→‑verifier pipeline, presenting findings in an interactive terminal UI and optionally verifying exploits in Docker.

  • Project
    qiantongtech/qData

    An open-source data middle platform for enterprise data governance and development, featuring visual ETL, metadata management, and AI-powered Text2SQL data analysis.