NDLI: An instrument for quality assurance in work capacity evaluation: development, evaluation, and inter-rater reliability

Content Provider	Springer Nature : BioMed Central
Author	Strahl, André Gerlich, Christian Alpers, Georg W. Gehrke, Jörg Müller-Garnn, Annette Vogel, Heiner
Abstract	Background Employees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To assess whether an individual meets the medical requirements to be considered as disabled, a work capacity evaluation is conducted. However, there are no official guidelines on how to perform an external quality assurance for this evaluation process. Furthermore, the quality of medical reports in the field of insurance medicine can vary substantially, and systematic evaluations are scarce. Reliability studies using peer review have repeatedly shown insufficient ability to distinguish between high, moderate and low quality. Considering literature recommendations, we developed an instrument to examine the quality of medical experts’ reports. Methods The peer review manual developed contains six quality domains (formal structure, clarity, transparency, completeness, medical-scientific principles, and efficiency) comprising 22 items. In addition, a superordinate criterion (survey confirmability) rank the overall quality and usefulness of a report. This criterion evaluates problems of inner logic and reasoning. Development of the manual was assisted by experienced physicians in a pre-test. We examined the observable variance in peer judgements and reliability as the most important outcome criteria. To evaluate inter-rater reliability, 20 anonymous experts’ reports detailing the work capacity evaluation were reviewed by 19 trained raters (peers). Percentage agreement and Kendall’s W, a reliability measure of concordance between two or more peers, were calculated. A total of 325 reviews were conducted. Results Agreement of peer judgements with respect to the superordinate criterion ranged from 29.2 to 87.5%. Kendall’s W for the quality domain items varied greatly, ranging from 0.09 to 0.88. With respect to the superordinate criterion, Kendall’s W was 0.39, which indicates fair agreement. The results of the percentage agreement revealed systemic peer preferences for certain deficit scale categories. Conclusion The superordinate criterion was not sufficiently reliable. However, in comparison to other reliability studies, this criterion showed an equivalent reliability value. This report aims to encourage further efforts to improve evaluation instruments. To reduce disagreement between peer judgments, we propose the revision of the peer review instrument and the development and implementation of a standardized rater training to improve reliability.
Related Links	https://bmchealthservres.biomedcentral.com/counter/pdf/10.1186/s12913-019-4387-4.pdf
Ending Page	11
Page Count	11
Starting Page	1
File Format	HTM / HTML
ISSN	14726963
DOI	10.1186/s12913-019-4387-4
Journal	BMC Health Services Research
Issue Number	1
Volume Number	19
Language	English
Publisher	BioMed Central
Publisher Date	2019-08-09
Access Restriction	Open
Subject Keyword	Public Health Health Administration Health Informatics Nursing Research Work capacity evaluation Insurance medicine Quality assurance Peer review Reliability
Content Type	Text
Resource Type	Article
Subject	Health Policy

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Development and evaluation of a standardized peer-training in the context of peer review for quality assurance in work capacity evaluation

Inter-rater reliability of nursing home quality indicators in the U.S

Intra- and inter-rater reliability of an electronic health record audit used in a chiropractic teaching clinic system: an observational study

A prospective evaluation of inter-rater agreement of routine medical records audits at a large general hospital in São Paulo, Brazil

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Approaches of integrating the development of guidelines and quality indicators: a systematic review

Five-year fidelity assessment of an evidence-based parenting program (GenerationPMTO): inter-rater reliability following international implementation

Assessing time use in long-term institutional care: development, validity and inter-rater reliability of the Groningen Observational instrument for Long-Term Institutional Care (GO-LTIC)

Inter-Rater and Intra-Rater Reliability of Return-to-Work Screening Tests for UK Firefighters Following Injury.

An instrument for quality assurance in work capacity evaluation: development, evaluation, and inter-rater reliability

Similar Documents

Development and evaluation of a standardized peer-training in the context of peer review for quality assurance in work capacity evaluation

Inter-rater reliability of nursing home quality indicators in the U.S

Intra- and inter-rater reliability of an electronic health record audit used in a chiropractic teaching clinic system: an observational study

A prospective evaluation of inter-rater agreement of routine medical records audits at a large general hospital in São Paulo, Brazil

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Approaches of integrating the development of guidelines and quality indicators: a systematic review

Five-year fidelity assessment of an evidence-based parenting program (GenerationPMTO): inter-rater reliability following international implementation

Assessing time use in long-term institutional care: development, validity and inter-rater reliability of the Groningen Observational instrument for Long-Term Institutional Care (GO-LTIC)

Inter-Rater and Intra-Rater Reliability of Return-to-Work Screening Tests for UK Firefighters Following Injury.

An instrument for quality assurance in work capacity evaluation: development, evaluation, and inter-rater reliability