NDLI: Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviews

Content Provider	Springer Nature : BioMed Central
Author	Popoff, E. Besada, M. Jansen, J. P. Cope, S. Kanters, S.
Abstract	Background Despite existing research on text mining and machine learning for title and abstract screening, the role of machine learning within systematic literature reviews (SLRs) for health technology assessment (HTA) remains unclear given lack of extensive testing and of guidance from HTA agencies. We sought to address two knowledge gaps: to extend ML algorithms to provide a reason for exclusion—to align with current practices—and to determine optimal parameter settings for feature-set generation and ML algorithms. Methods We used abstract and full-text selection data from five large SLRs (n = 3089 to 12,769 abstracts) across a variety of disease areas. Each SLR was split into training and test sets. We developed a multi-step algorithm to categorize each citation into the following categories: included; excluded for each PICOS criterion; or unclassified. We used a bag-of-words approach for feature-set generation and compared machine learning algorithms using support vector machines (SVMs), naïve Bayes (NB), and bagged classification and regression trees (CART) for classification. We also compared alternative training set strategies: using full data versus downsampling (i.e., reducing excludes to balance includes/excludes because machine learning algorithms perform better with balanced data), and using inclusion/exclusion decisions from abstract versus full-text screening. Performance comparisons were in terms of specificity, sensitivity, accuracy, and matching the reason for exclusion. Results The best-fitting model (optimized sensitivity and specificity) was based on the SVM algorithm using training data based on full-text decisions, downsampling, and excluding words occurring fewer than five times. The sensitivity and specificity of this model ranged from 94 to 100%, and 54 to 89%, respectively, across the five SLRs. On average, 75% of excluded citations were excluded with a reason and 83% of these citations matched the reviewers’ original reason for exclusion. Sensitivity significantly improved when both downsampling and abstract decisions were used. Conclusions ML algorithms can improve the efficiency of the SLR process and the proposed algorithms could reduce the workload of a second reviewer by identifying exclusions with a relevant PICOS reason, thus aligning with HTA guidance. Downsampling can be used to improve study selection, and improvements using full-text exclusions have implications for a learn-as-you-go approach.
Related Links	https://systematicreviewsjournal.biomedcentral.com/counter/pdf/10.1186/s13643-020-01520-5.pdf
Ending Page	12
Page Count	12
Starting Page	1
File Format	HTM / HTML
ISSN	20464053
DOI	10.1186/s13643-020-01520-5
Journal	Systematic Reviews
Issue Number	1
Volume Number	9
Language	English
Publisher	BioMed Central
Publisher Date	2020-12-13
Access Restriction	Open
Subject Keyword	Medicine Public Health Biomedicine Statistics for Life Sciences Health Sciences Study selection Machine learning Text mining Systematic literature reviews Methods Updates Classification Reasons for exclusion Downsampling Medicine/Public Health
Content Type	Text
Resource Type	Article
Subject	Medicine
Journal Impact Factor	6.3/2023
5-Year Journal Impact Factor	4.5/2023

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Supporting systematic reviews using LDA-based document representations

Iterative guided machine learning-assisted systematic literature reviews: a diabetes case study

Study flow diagrams in Cochrane systematic review updates: an adapted PRISMA flow diagram

Using text mining for study identification in systematic reviews: a systematic review of current approaches

Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study

Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviews

Similar Documents

Supporting systematic reviews using LDA-based document representations

Iterative guided machine learning-assisted systematic literature reviews: a diabetes case study

Study flow diagrams in Cochrane systematic review updates: an adapted PRISMA flow diagram

Using text mining for study identification in systematic reviews: a systematic review of current approaches

Usefulness of machine learning softwares to screen titles of systematic reviews: a methodological study

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records

Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study

Aligning text mining and machine learning algorithms with best practices for study selection in systematic literature reviews