fivetran/great_expectations

Always know what to expect from your data.

What it solves

It addresses the challenge of maintaining data quality and institutional knowledge within data teams. It provides a standardized way to test data, ensuring that datasets remain reliable and consistent over time.

How it works

The project uses "Expectations," which act as expressive and extensible unit tests for data. These tests allow teams to define what their data should look like, validate the results, and and automatically generate documentation for those results to keep the team aligned.

Who it’s for

Data teams who need to ensure the quality and reliability of their data pipelines and datasets.

Highlights

  • Expressive and extensible unit tests for data.
  • Automatic documentation generation for validation results.
  • Common language for data quality tests to foster team collaboration.

Related

  • Project
    evidentlyai/evidently

    An open-source Python framework to evaluate, test, and monitor ML and LLM-powered systems, helping developers detect data drift and ensure output quality.

  • Dispatch
    Optimizing Documentation for the AI Agent Era

    Explore how dari-docs uses simulated developer agents to turn documentation quality from subjective 'understandability' into a measurable, testable metric.

  • Project
    awslabs/deequ

    A library built on Apache Spark for defining unit tests for large datasets to ensure data quality before it is used in applications or ML models.

  • Project
    openml/automlbenchmark

    A framework for evaluating and comparing open-source AutoML systems using standardized environments and curated datasets to ensure reproducible research.

  • Project
    datahub-project/datahub

    An open-source metadata management platform that turns enterprise data into trusted context for humans and AI agents through discovery, governance, and observability.