moj-analytical-services/splink

What it solves

Splink is designed to deduplicate and link records from datasets that lack unique identifiers. It solves the problem of entity resolution—identifying whether two records refer to the same real-world entity (such as a person or company) when you only have imperfect or inconsistent data across multiple columns.

How it works

Splink uses a probabilistic record linkage algorithm based on the Fellegi-Sunter model. It predicts match probabilities between pairs of records based on multiple non-correlated columns (e.g., name, date of birth, city). The system can estimate model parameters using unsupervised learning (Expectation Maximisation), meaning no pre-labeled training data is required. Once match probabilities are calculated, it clusters these links to assign a unique ID to each group of duplicate records.

Who it’s for

This tool is for data scientists and analysts in government, academia, and the private sector who need to perform record linkage at scale, particularly when dealing with millions of records.

Highlights

• High Performance: Capable of linking a million records on a laptop in about a minute.
• Scalability: Supports multiple backends including DuckDB for local Python execution and Spark or PostgreSQL for datasets exceeding 100 million records.
• Unsupervised Learning: No training data is required to train the model.
• Interactive Diagnostics: Includes a suite of interactive visualizations to help users diagnose and refine their linkage models.
• Fuzzy Matching: Supports term frequency adjustments and user-defined fuzzy matching logic to improve accuracy.