Desbordante/desbordante-core

What it solves

Desbordante is a high-performance data profiler designed to discover and validate complex patterns within datasets. It helps users identify structural relationships, data quality issues, and hidden insights that can be used for scientific hypothesis generation, business data cleaning, and machine learning feature engineering.

How it works

The tool employs various algorithms to perform three primary types of tasks:

• Discovery: Identifying all instances of a specific pattern type within a dataset.
• Validation: Checking if a specific pattern instance exists and providing explanations (e.g., conflicting rows) if it does not.
• Dynamic Tasks: A specialized variant for certain patterns that updates results based on data changes rather than reprocessing the entire table, significantly increasing speed.

It supports a vast array of patterns, including exact and approximate functional dependencies, inclusion dependencies, association rules, denial constraints, and differential dependencies.

Who it’s for

• Data Scientists and ML Engineers: For feature engineering and ablation studies in machine learning training data.
• Business Analysts: For clearing data errors, removing duplicates, and performing schema matching.
• Researchers: For formulating hypotheses based on patterns found in experimental scientific data.
• Database Administrators: For recovering primary/foreign keys and setting up integrity constraints.

Highlights

• Extensive Pattern Library: Supports dozens of pattern types, including probabilistic, fuzzy, and graph-based dependencies.
• Multiple Interfaces: Available as a command-line tool, a Python library (integrating with pandas DataFrames), and an interactive web application.
• High Performance: Features a C++ core library with Python bindings via pybind11 for efficiency.
• Practical Demo Scenarios: Includes built-in logic for typo detection, data deduplication, and anomaly detection.