capitalone/DataProfiler

What it solves

DataProfiler simplifies the process of analyzing, monitoring, and detecting sensitive information within datasets. It automates the tedious task of identifying data schemas, calculating statistics, and recognizing personally identifiable information (PII) or non-public information (NPI) across various file formats.

How it works

The library uses a Data class to automatically detect and load files (such as CSV, Parquet, JSON, and Avro) into Pandas DataFrames. A Profiler then analyzes this data to generate a profile—a dictionary containing global statistics (e.g., row counts, null ratios) and column-level statistics (e.g., mean, variance, unique counts). For sensitive data detection, it employs a pre-trained deep learning model to label entities like email addresses, credit card numbers, and SSNs. Users can also extend the model with new entities or custom recognition pipelines.

Who it’s for

It is designed for data scientists, data engineers, and security analysts who need to quickly audit datasets for quality or compliance, specifically those needing to identify sensitive data for privacy reasons.

Highlights

• Automated Sensitive Data Detection: Uses a pre-trained deep learning model to identify PII/NPI entities.
• Broad Format Support: Compatible with CSV, TSV, JSON, Avro, Parquet, text files, and URLs.
• Comprehensive Statistics: Provides detailed global and column-level metrics, including Gini impurity and correlation matrices.
• Flexible Profiling: Supports structured, unstructured, and graph-based data profiling.
• Scalable Workflow: Allows updating existing profiles with new data batches or merging multiple profiles using an addition operator.