magda-io/magda

A federated, open-source data catalog for all your big data and small data

What it solves

Organizations often have data scattered across many places—internal systems, external portals, files, databases, APIs—making it hard to know what exists and where. Magda solves this by providing a single catalog where all data can be listed, described, searched, and tracked, regardless of where it actually lives. It also helps prioritize data and notifies users when changes occur, so people can discover what’s available and trust that it’s up to date.

How it works

Magda is built from small, independent services (microservices) that each handle a specific task, all bundled as Docker containers. The central piece is a Registry that stores metadata as flexible JSON documents called aspects. Connectors go out to external data sources (like open-data portals) and copy their metadata into the Registry. Minions listen for changes and run enrichment tasks—for example, checking if links still work or computing quality scores—and write results back. The metadata is then indexed into an OpenSearch cluster to power fast, scalable searches, and a user interface lets people browse and query everything from one place. The whole system runs on Kubernetes, so it can be installed on any cloud, on-premises, or locally, and it’s designed to be easily extended by adding new Docker-based services.

Who it’s for

Magda is aimed at organizations that want a unified view of all their data—whether they’re managing internal corporate datasets or publishing open data to the public. It has been used in production by data.gov.au for over a year as an open data search engine, and it’s also suitable as a general-purpose data catalogue within organizations (though the README notes that some features are still rough around the edges). Developers who know Kubernetes can install and customize it, and contributors are welcome to help improve it.

Highlights

  • Powerful and scalable search powered by OpenSearch
  • Quickly aggregates metadata from external data sources
  • Unopinionated central metadata store that supports most schemas
  • Federated authentication (via Google, Facebook, WSFed, AAF, CKAN, or custom providers)
  • Cloud-agnostic—deploys to nearly any cloud, on-premises, or local machine via Kubernetes
  • Extensions can be written in any language by adding new Docker images

Related

  • Project
    lightonai/next-plaid

    NextPlaid is a local-first multi-vector search database with a REST API; ColGREP is a semantic code-search tool built on it — a single Rust binary that indexes your codebase locally and answers natural-language queries (optionally via a small embedded agent model), combining regex filtering with semantic ranking. NextPlaid solves the storage cost of multi-vector embeddings (hundreds per document) with quantization, memory-mapped indices, incremental updates, and SQL metadata pre-filtering, all CPU-friendly. Both tools are Apache-2.0 licensed, use ONNX-exported ColBERT models, and keep all data on your machine.

  • Project
    aryn-ai/sycamore

    An AI-powered document processing engine for ETL and RAG that transforms unstructured documents into high-quality data chunks for vector databases.

  • Project
    insitro/redun

    A Python workflow framework that evaluates lazy expressions to build dynamic DAGs, with automatic parallelization, caching, and data lineage logging.

  • Project
    OvidijusParsiunas/myvision

    A free online image annotation tool for generating computer vision ML training data with support for automated bounding box generation and dataset format conversion.