data-privacy-stack/presidio-research

What it solves

It provides a standardized framework for developing, testing, and evaluating Personally Identifiable Information (PII) detection models. It specifically helps developers move beyond simple testing to rigorous evaluation of precision and recall, while also solving the problem of lacking high-quality training data by providing tools to generate synthetic PII datasets.

How it works

The project implements a three-part pipeline:

1. Data Generation: It uses sentence templates (e.g., "My name is {{name}}") and fake PII values to create synthetic datasets. It can then tokenize this data and create tags in formats like IO, BIO, or BILUO.
2. Data Representation: It uses a standardized InputSample object to ensure data can be easily converted between different formats, including CoNLL, spaCy v3, and JSON.
3. Evaluation: It provides tools to measure the performance of the Presidio Analyzer or custom PII recognizers, allowing for detailed error analysis and the mapping of entities across different models.

Who it’s for

• Developers building or refining PII detection models and Named Entity Recognition (NER) systems.
• Data scientists who need to generate synthetic PII data to increase the coverage of entity values in their training sets.
• Users of the Presidio framework who want to evaluate the accuracy of their specific configurations.

Highlights

• Synthetic Data Generator: Creates fake PII sentences based on templates to avoid using real sensitive data for training.
• Multi-format Support: Seamlessly converts datasets between JSON, CoNLL, and spaCy formats.
• Leakage Prevention: Includes tools to split datasets into train/test/validation sets while ensuring the same template doesn't appear in multiple folds.
• Comprehensive Evaluation: Supports precision, recall, and F-score analysis for both system-wide and individual recognizer performance.