NDLI: Dynamic-ETL: a hybrid approach for health data extraction, transformation and loading

Content Provider	Springer Nature : BioMed Central
Author	Ong, Toan C. Kahn, Michael G. Kwan, Bethany M. Yamashita, Traci Brandt, Elias Hosokawa, Patrick Uhrich, Chris Schilling, Lisa M.
Abstract	Background Electronic health records (EHRs) contain detailed clinical data stored in proprietary formats with non-standard codes and structures. Participating in multi-site clinical research networks requires EHR data to be restructured and transformed into a common format and standard terminologies, and optimally linked to other data sources. The expertise and scalable solutions needed to transform data to conform to network requirements are beyond the scope of many health care organizations and there is a need for practical tools that lower the barriers of data contribution to clinical research networks. Methods We designed and implemented a health data transformation and loading approach, which we refer to as Dynamic ETL (Extraction, Transformation and Loading) (D-ETL), that automates part of the process through use of scalable, reusable and customizable code, while retaining manual aspects of the process that requires knowledge of complex coding syntax. This approach provides the flexibility required for the ETL of heterogeneous data, variations in semantic expertise, and transparency of transformation logic that are essential to implement ETL conventions across clinical research sharing networks. Processing workflows are directed by the ETL specifications guideline, developed by ETL designers with extensive knowledge of the structure and semantics of health data (i.e., “health data domain experts”) and target common data model. Results D-ETL was implemented to perform ETL operations that load data from various sources with different database schema structures into the Observational Medical Outcome Partnership (OMOP) common data model. The results showed that ETL rule composition methods and the D-ETL engine offer a scalable solution for health data transformation via automatic query generation to harmonize source datasets. Conclusions D-ETL supports a flexible and transparent process to transform and load health data into a target data model. This approach offers a solution that lowers technical barriers that prevent data partners from participating in research data networks, and therefore, promotes the advancement of comparative effectiveness research using secondary electronic health data.
Related Links	https://bmcmedinformdecismak.biomedcentral.com/counter/pdf/10.1186/s12911-017-0532-3.pdf
Ending Page	12
Page Count	12
Starting Page	1
File Format	HTM / HTML
ISSN	14726947
DOI	10.1186/s12911-017-0532-3
Journal	BMC Medical Informatics and Decision Making
Issue Number	1
Volume Number	17
Language	English
Publisher	BioMed Central
Publisher Date	2017-09-13
Access Restriction	Open
Subject Keyword	Health Informatics Information Systems and Communication Service Management of Computing and Information Systems Electronic health records Extraction Transformation and loading Distributed research networks Data harmonization Rule-based ETL
Content Type	Text
Resource Type	Article
Subject	Health Informatics Computer Science Applications Health Policy
Journal Impact Factor	3.3/2023
5-Year Journal Impact Factor	3.9/2023

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A clustering approach for detecting implausible observation values in electronic health records data

Text data extraction for a prospective, research-focused data mart: implementation and validation

Enhancing clinical data retrieval with Smart Watchers: a NiFi-based ETL pipeline for Elasticsearch queries

The Generalized Data Model for clinical research

Prediction of Sjögren’s disease diagnosis using matched electronic dental-health record data

An ontology-based approach for harmonization and cross-cohort query of Alzheimer’s disease data resources

Feasibility of extracting data from electronic medical records for research: an international comparative study

Conceptual design of a generic data harmonization process for OMOP common data model

Combining population-based administrative health records and electronic medical records for disease surveillance

Dynamic-ETL: a hybrid approach for health data extraction, transformation and loading

Similar Documents

A clustering approach for detecting implausible observation values in electronic health records data

Text data extraction for a prospective, research-focused data mart: implementation and validation

Enhancing clinical data retrieval with Smart Watchers: a NiFi-based ETL pipeline for Elasticsearch queries

The Generalized Data Model for clinical research

Prediction of Sjögren’s disease diagnosis using matched electronic dental-health record data

An ontology-based approach for harmonization and cross-cohort query of Alzheimer’s disease data resources

Feasibility of extracting data from electronic medical records for research: an international comparative study

Conceptual design of a generic data harmonization process for OMOP common data model

Combining population-based administrative health records and electronic medical records for disease surveillance

Dynamic-ETL: a hybrid approach for health data extraction, transformation and loading