Data Matching software

• Overview
• Software
• Outdated
• Contributing

This is a list of (Fuzzy) Data Matching software. The software in this list is
FOSS (Free and open-source software).

The term data matching is used to indicate the procedure of bringing together
information from two or more records that are believed to belong to the same
entity. Data matching has two applications: (1) to match data across multiple
datasets (linkage) and (2) to match data within a dataset (deduplication).
See the Wikipedia page about
data matching for more information.

Similar terms: record linkage, data matching, deduplication, fuzzy matching,
entity resolution

Overview

The table below gives a dense overview of data matching software properties.
The properties evaluated are Application Programming Interface
(API),
Graphical User Interface
(GUI), Linking,
Deduplication, Supervised
Learning, Unsupervised
Learning and Active
Learning.

Software	API	GUI	Link	Dedup	Supervised <br/> Learning	Unsupervised <br/> Learning	Active <br/> Learning
AtyImo	PySpark	:x:	:white_check_mark:	:white_check_mark:	:x:	:x:	:x:
Dedupe	Python	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:	:white_check_mark:
dirty-cat	Python	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:
fastLink	R	:x:	:white_check_mark:	:grey_question:	:x:	:white_check_mark:	:x:
FEBRL	Python	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:	:x:	:x:
FRIL	Java	:white_check_mark:	:white_check_mark:	:x:	:grey_question:	:white_check_mark:	:x:
FuzzyMatcher	Python	:x:	:white_check_mark:	:x:	:x:	:white_check_mark:	:x:
hlink	PySpark	:x:	:white_check_mark:	:grey_question:	:x:	:x:	:x:
JedAI	Java	:white_check_mark:	:white_check_mark:	:grey_question:	:white_check_mark:	:grey_question:	:grey_question:
PRIL	SQL	:x:	:white_check_mark:	:grey_question:	:grey_question:	:grey_question:	:grey_question:
Python Record Linkage Toolkit	Python	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:
RecordLinkage (R)	R	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:
Reclin2	R	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:	:x:
RELAIS	:x:	:white_check_mark:	:white_check_mark:	:grey_question:	:grey_question:	:white_check_mark:	:x:
ReMaDDer	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:	:white_check_mark:	:x:
RLTK	Python	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:	:x:
Splink	Python	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:
Zingg	Python	:x:	:white_check_mark:	:white_check_mark:	:white_check_mark:	:x:	:x:

:white_check_mark: Yes/Implemented
:x: No/Not implemented
:grey_question: Unknown

Software

This section describes data matching software. The software is
alphabetically ordered.

AtyImo

AtyImo implements a mixture of deterministic and probabilistic routines for data
linkage. Initially developed in 2013 to serve as a linkage tool supporting a joint
Brazil–U.K. project aiming at building a large population-based cohort with data
from more than 100 million participants and producing disease-specific data to facilitate
diverse epidemiological research studies.

License
Language	Python Spark
Latest release	NA
Downloads per month
GitHub stars

Dedupe

Dedupe is a python library for fuzzy matching, deduplication and entity
resolution on structured data. The library makes use of active learning to
match record pairs. Active learning is useful in cases without training data.
Dedupe has a side-product for deduplicating CSV files,
csvdedupe, through the command line.
Dedupeio also offers commercial products for data matching.

License
Language
Latest release
Downloads per month
GitHub stars

dirty-cat

dirty-cat is an open-source Python package that facilitates machine-learning with with dirty data: robust to morphological variants, such as typos. Some of the currently supported features are: fuzzy joining tables on dirty numerical, string or mixed type columns, deduplicating and encoding dirty categorical variables for ML. This example illustrates why to use dirty-cat encoders rather than OneHotEncoder on dirty data and this one shows how to join multiple dirty tables for ML.
The transfomers (TableVectorizer, FeatureAugmenter) are scikit-learn compatible, and easily introduced into ML pipelines.

License
Language	Spark
Latest release
Downloads per month
GitHub stars

fastLink

Implements a Fellegi-Sunter probabilistic record linkage model that allows for
missing data and the inclusion of auxiliary information. This includes
functionalities to conduct a merge of two datasets under the Fellegi-Sunter
model using the Expectation-Maximization algorithm. fastLink is a programming
API written in R. (Enamorado, Fifield & Imai,
2017) [source
code]

License
Language	R
Latest release
Downloads per month
GitHub stars

FEBRL

Febrl (Freely Extensible Biomedical Record Linkage) is a training tool
suitable for users to learn and experiment with record linkage techniques, as
well as for practitioners to conduct linkages with data sets containing up to
several hundred thousand records. Febrl is a data matching tool with a large
number of algorithms implemented and offers a Python programming interface as
well as simple GUI. Febrl doesn't offer unsupervised and active learning
algorithms. The software is no longer actively maintained. (Christen,
2008) [source
code]

License	Custom
Language	Python
Latest release
Downloads per month
GitHub stars

FRIL

FRIL (Fine-grained Records Integration and Linkage tool) is free tool that
enables record linkage through a GUI. The tool implements automatic weights
estimation through the EM-algorithm and offers serveral techniques to make
record pairs. FRIL was developed by the Emory University and is not longer
maintained. [source code]

License	Custom
Language	Java
Latest release
Downloads per month
GitHub stars

FuzzyMatcher

A Python package that allows the user to fuzzy match two pandas dataframes
based on one or more fields in common. The functionality is limited at the
moment. [source code]

License
Language
Latest release
Downloads per month
GitHub stars

hlink

A Python package designed to link two datasets. The primary use case was for linking demographics in the Household -> Person hierarchical structure, however it can be used to link generic datasets as well by skipping household linking tasks. It allows for probabilistic and deterministic record linkage. [source_code]

License
Language
Latest release
Downloads per month
GitHub stars

JedAI

Java gEneric DAta Integration (JedAI) Toolkit is a Entity Resolution Tool
developed by a group of univeristies. JedAI offers a Graphical User Interface.
[source code]

License
Language	Java
Latest release
Downloads per month
GitHub stars

PRIL

PRIL (Point-of-contact Interactive Record Linkage) is a record linkage program
with a GUI. PRIL can be used to link datasets about individuals. (Rentsch CT,
Kabudula CW, Catlett J et al.,
2017) [source
code]

License
Language	SQLPL
Latest release
Downloads per month
GitHub stars

Python Record Linkage Toolkit

The Python Record Linkage Toolkit is a library to link records in or between
data sources. The toolkit provides most of the tools needed for record linkage
and deduplication. The package is developed for research and the linking of
small or medium sized files.

License
Language
Latest release
Downloads per month
GitHub stars

RecordLinkage (R)

Package written in R that provides functions for linking and de-duplicating
data sets. Both supervised and unsupervised classification algorithms are
available. Record pairs can be compared with a limited set of algorithms. The
package is published on CRAN.

License
Language	R
Latest release
Downloads per month
GitHub stars

Reclin2

Package written in R that provides functions for linking data sets. The framework offers
the option to compute the weigths of the Fellegi-Sunter model. It doesn't implement an
undersupervised algorithms to predict the cutoff. The
package is published on CRAN. Formerly https://github.com/djvanderlaan/reclin.

License
Language	R
Latest release
Downloads per month
GitHub stars

RELAIS

RELAIS (REcord Linkage At IStat) is a toolkit providing a set of techniques
for dealing with record linkage projects. IStat is the main producer of
official statistics in Italy.

License	EUPL-1.1
Language	R/Java
Latest release
Downloads per month
GitHub stars

ReMaDDer

ReMaDDer is unsupervised free fuzzy data matching software with a GUI.
ReMaDDer is capable to perform fully automatic fuzzy record matching without
human expert intervention, while attaining accuracy of human clerical review.
NOTE: The software is free, but not open source and requires an internet
connection to work.

License
Language
Latest release
Downloads per month
GitHub stars

RLTK

The Record Linkage ToolKit (RLTK) is a general-purpose open-source record
linkage package. The toolkit provides a full pipeline needed for record linkage
and deduplication.

License
Language	Python
Latest release
Downloads per month
GitHub stars

Splink

Splink is a Python package for probabilistic record linkage at scale.
It supports multiple backends to execute linkage jobs, including DuckDB
Apache Spark and AWS Athena. It is able to perform linking and deduplication of very large datasets
of tens of millions of records with runtimes of less than an hour, including
the clustering of results using connected components. It includes interactive tools
to support the lifecycle of a linking project, from exploratory analysis through to
diagnostics and quality assurance.[source
code]

License
Language
Latest release
Downloads per month
GitHub stars

Zingg

Zingg is an open-source ML based tool for entity resolution with which analytics engineer and the data scientist can quickly integrate data silos and build unified views at scale. Zingg has the ability to connect to disparate data source, local and cloud file systems in any format, enterprise applications and relational, NoSQL and cloud databases and warehouses. It scales to large volume of data and you can define domain specific functions to improve matching.
Not only Zingg support English as well as Chinese, Thai, Japanese, Hindi and other languages, it also has a very active slack community where people around the globe come and help and share their views.

License
Language	Spark
Latest release
Downloads per month
GitHub stars

Outdated/ no longer available

BigMatch (by USA census)

A record linkage tool for use in matching a very large file against a moderate
size file developed by the USA Census Bureau. There are several papers
available about this program (BigMatch,
2007)

The Link King

The Link King’s graphical user interface (GUI) makes record linkage and
unduplication easy for beginning and advanced users. The software requires a
SAS license. SAS

Contributing

Do you know an open source and/or free data matching tool? Please open an
issue or do a Pull Request. The same holds for missing or incomplete
information.

This project is initiated by the author of the Python Record Linkage
Toolkit @J535D165. The aim is to
get a list and comparison of data matching software.

This list is licensed under CC-BY-SA 3.0.