NDLI: Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews

Content Provider	Springer Nature : BioMed Central
Author	Gates, Allison Gates, Michelle DaRosa, Daniel Elliott, Sarah A. Pillay, Jennifer Rahman, Sholeh Vandermeer, Ben Hartling, Lisa
Abstract	Background We evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening and explored whether Abstrackr’s predictions varied by review or study-level characteristics. Methods For a convenience sample of 16 reviews for which adequate data were available to address our objectives (11 systematic reviews and 5 rapid reviews), we screened a 200-record training set in Abstrackr and downloaded the relevance (relevant or irrelevant) of the remaining records, as predicted by the tool. We retrospectively simulated the liberal-accelerated screening approach. We estimated the time savings and proportion missed compared with dual independent screening. For reviews with pairwise meta-analyses, we evaluated changes to the pooled effects after removing the missed studies. We explored whether the tool’s predictions varied by review and study-level characteristics. Results Using the ML-assisted liberal-accelerated approach, we wrongly excluded 0 to 3 (0 to 14%) records that were included in the final reports, but saved a median (IQR) 26 (9, 42) h of screening time. One missed study was included in eight pairwise meta-analyses in one systematic review. The pooled effect for just one of those meta-analyses changed considerably (from MD (95% CI) − 1.53 (− 2.92, − 0.15) to − 1.17 (− 2.70, 0.36)). Of 802 records in the final reports, 87% were correctly predicted as relevant. The correctness of the predictions did not differ by review (systematic or rapid, P = 0.37) or intervention type (simple or complex, P = 0.47). The predictions were more often correct in reviews with multiple (89%) vs. single (83%) research questions (P = 0.01), or that included only trials (95%) vs. multiple designs (86%) (P = 0.003). At the study level, trials (91%), mixed methods (100%), and qualitative (93%) studies were more often correctly predicted as relevant compared with observational studies (79%) or reviews (83%) (P = 0.0006). Studies at high or unclear (88%) vs. low risk of bias (80%) (P = 0.039), and those published more recently (mean (SD) 2008 (7) vs. 2006 (10), P = 0.02) were more often correctly predicted as relevant. Conclusion Our screening approach saved time and may be suitable in conditions where the limited risk of missing relevant records is acceptable. Several of our findings are paradoxical and require further study to fully understand the tasks to which ML-assisted screening is best suited. The findings should be interpreted in light of the fact that the protocol was prepared for the funder, but not published a priori. Because we used a convenience sample, the findings may be prone to selection bias. The results may not be generalizable to other samples of reviews, ML tools, or screening approaches. The small number of missed studies across reviews with pairwise meta-analyses hindered strong conclusions about the effect of missed studies on the results and conclusions of systematic reviews.
Related Links	https://systematicreviewsjournal.biomedcentral.com/counter/pdf/10.1186/s13643-020-01528-x.pdf
Ending Page	12
Page Count	12
Starting Page	1
File Format	HTM / HTML
ISSN	20464053
DOI	10.1186/s13643-020-01528-x
Journal	Systematic Reviews
Issue Number	1
Volume Number	9
Language	English
Publisher	BioMed Central
Publisher Date	2020-11-27
Access Restriction	Open
Subject Keyword	Medicine Public Health Biomedicine Statistics for Life Sciences Health Sciences Machine learning Artificial intelligence Text mining Systematic reviews Methods Efficiency Medicine/Public Health
Content Type	Text
Resource Type	Article
Subject	Medicine
Journal Impact Factor	6.3/2023
5-Year Journal Impact Factor	4.5/2023

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Semi-automated title-abstract screening using natural language processing and machine learning

Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow

Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool

Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study

Statistical stopping criteria for automated screening in systematic reviews

Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer

An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes

Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology

Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews

Similar Documents

Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews

Semi-automated title-abstract screening using natural language processing and machine learning

Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow

Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool

Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study

Statistical stopping criteria for automated screening in systematic reviews

Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer

An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes

Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology

Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews