awslabs/deequ

What it solves

Deequ provides a way to "unit-test" large datasets to find errors early, preventing corrupted or incorrect data from reaching consuming systems or machine learning algorithms. It allows developers to explicitly state assumptions about their data (e.g., no nulls in a specific column) and verify them at scale.

How it works

Built on top of Apache Spark, Deequ works on tabular data (CSV, database tables, logs, etc.) and translates defined constraints into a series of Spark jobs to compute metrics. It can be used via a Scala/Java API or a declarative language called DQDL (Data Quality Definition Language) to define rules like completeness, uniqueness, and range checks. It also supports row-level evaluation to identify exactly which records failed a specific rule.

Who it’s for

Data engineers and ML practitioners who work with very large datasets (billions of rows) in distributed filesystems or data warehouses and need to ensure data quality and consistency.

Highlights

• Scalable Verification: Designed for billions of rows using Apache Spark.
• Declarative Rules: Supports DQDL for readable, simple rule definitions.
• Coded Constraints: Flexible API for defining complex checks like quantiles, URL patterns, and size constraints.
• Advanced Analysis: Includes capabilities for data profiling, anomaly detection on metrics over time, and automatic constraint suggestion.
• Row-Level Tracking: Ability to identify specific rows that fail quality checks.