moj-analytical-services/splink

What it solves

Splink is designed for probabilistic record linkage and deduplication. It solves the problem of identifying and linking records that refer to the same entity (such as people or companies) across different datasets, or within a single dataset, when those records lack a unique identifier (like a Social Security number or a unique ID).

How it works

Splink uses the Fellegi-Sunter statistical model for record linkage. It employs unsupervised learning, meaning it does not require pre-labeled training data to estimate the parameters of the linkage model. It supports fuzzy matching logic and term frequency adjustments to improve accuracy. To maintain performance at scale, it can execute linkage using DuckDB for local processing or Spark and PostgreSQL for big-data backends.

Who it’s for

Data scientists and analysts in government, academia, and the private sector who need to deduplicate large datasets or link records across multiple sources without unique identifiers.

Highlights

• High Performance: Capable of linking a million records on a laptop in about a minute.
• Scalability: Supports backends like Spark for processing over 100 million records.
• Unsupervised Learning: No training data required for model training.
• Interactive Visualizations: Includes a suite of tools to help users diagnose and understand their linkage models.
• Fuzzy Matching: Supports custom fuzzy matching logic and term frequency adjustments.