NDLI: A proficient cost reduction framework for de-duplication of records in data integration

Content Provider	Springer Nature : BioMed Central
Author	Sohail, Asif Yousaf, Muhammad Murtaza
Abstract	Background Record de-duplication is a process of identifying the records referring to the same entity. It has a pivotal role in data mining applications, which involves the integration of multiple data sources and data cleansing. It has been a challenging task due to its computational complexity and variations in data representations across different data sources. Blocking and windowing are the commonly used methods for reducing the number of record comparisons during record de-duplication. Both blocking and windowing require tuning of a certain set of parameters, such as the choice of a particular variant of blocking or windowing, the selection of appropriate window size for different datasets etc. Methods In this paper, we have proposed a framework that employs blocking and windowing techniques in succession, such that figuring out the parameters is not required. We have also evaluated the impact of different configurations on dirty and massively dirty datasets. To evaluate the proposed framework, experiments are performed using Febrl (Freely Extensible Biomedical Record Linkage). Results The proposed framework is comprehensively evaluated using a variety of quality and complexity parameters such as reduction ratio, precision, recall etc. It is observed that the proposed framework significantly reduces the number of record comparisons. Conclusions The selection of the linkage key is a critical performance factor for record linkage.
Related Links	https://bmcmedinformdecismak.biomedcentral.com/counter/pdf/10.1186/s12911-016-0280-9.pdf
Ending Page	12
Page Count	12
Starting Page	1
File Format	HTM / HTML
ISSN	14726947
DOI	10.1186/s12911-016-0280-9
Journal	BMC Medical Informatics and Decision Making
Issue Number	1
Volume Number	16
Language	English
Publisher	BioMed Central
Publisher Date	2016-04-12
Access Restriction	Open
Subject Keyword	Health Informatics Information Systems and Communication Service Management of Computing and Information Systems Record linkage/de-duplication Data integration Record comparison reduction Inverted index
Content Type	Text
Resource Type	Article
Subject	Health Informatics Computer Science Applications Health Policy
Journal Impact Factor	3.3/2023
5-Year Journal Impact Factor	3.9/2023

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A proficient cost reduction framework for de-duplication of records in data integration

The effect of data cleaning on record linkage quality

Technical challenges of providing record linkage services for research

Some methods for blindfolded record linkage

De-identified Bayesian personal identity matching for privacy-preserving record linkage despite errors: development and validation

Record linkage under suboptimal conditions for data-intensive evaluation of primary care in Rio de Janeiro, Brazil

Privacy-preserving record linkage using Bloom filters

Medical record linkage in health information systems by approximate string matching and clustering

Optimal strategy for linkage of datasets containing a statistical linkage key and datasets with full personal identifiers

A proficient cost reduction framework for de-duplication of records in data integration

Similar Documents

A proficient cost reduction framework for de-duplication of records in data integration

The effect of data cleaning on record linkage quality

Technical challenges of providing record linkage services for research

Some methods for blindfolded record linkage

De-identified Bayesian personal identity matching for privacy-preserving record linkage despite errors: development and validation

Record linkage under suboptimal conditions for data-intensive evaluation of primary care in Rio de Janeiro, Brazil

Privacy-preserving record linkage using Bloom filters

Medical record linkage in health information systems by approximate string matching and clustering

Optimal strategy for linkage of datasets containing a statistical linkage key and datasets with full personal identifiers

A proficient cost reduction framework for de-duplication of records in data integration