alan-turing-institute/CleverCSV

What it solves

CSV files are often difficult to work with because they lack a universal standard for formatting, leading to "messy" files with inconsistent delimiters, quotes, or escape characters. CleverCSV solves this by automatically detecting the correct file format (or "dialect") with high accuracy, allowing you to load and process these files without manual inspection.

How it works

CleverCSV uses a research-backed approach that analyzes patterns in row lengths and data types within a CSV file to identify its structure. It functions as a drop-in replacement for Python's built-in csv module, providing improved dialect detection and a suite of helper functions to load data directly into lists or Pandas DataFrames.

Who it’s for

It is designed for data scientists, developers, and researchers who frequently work with real-world CSV files and need a reliable, automated way to handle inconsistent formatting without writing custom parsing logic for every file.

Highlights

• Achieves 97% accuracy in dialect detection, significantly outperforming standard library tools on messy files.
• Provides a command-line interface to quickly inspect, standardize, or generate Python import code for CSV files.
• Offers seamless integration with Pandas for easy data analysis.
• Includes a pre-commit hook to automatically standardize CSV files in your version control repository.