NDLI: The reliability of the College of Intensive Care Medicine of Australia and New Zealand “Hot Case” examination

Content Provider	Springer Nature : BioMed Central
Author	Hoffman, Kenneth R. Swanson, David Lane, Stuart Nickson, Chris Brand, Paul Ryan, Anna T.
Abstract	Background High stakes examinations used to credential trainees for independent specialist practice should be evaluated periodically to ensure defensible decisions are made. This study aims to quantify the College of Intensive Care Medicine of Australia and New Zealand (CICM) Hot Case reliability coefficient and evaluate contributions to variance from candidates, cases and examiners. Methods This retrospective, de-identified analysis of CICM examination data used descriptive statistics and generalisability theory to evaluate the reliability of the Hot Case examination component. Decision studies were used to project generalisability coefficients for alternate examination designs. Results Examination results from 2019 to 2022 included 592 Hot Cases, totalling 1184 individual examiner scores. The mean examiner Hot Case score was 5.17 (standard deviation 1.65). The correlation between candidates’ two Hot Case scores was low (0.30). The overall reliability coefficient for the Hot Case component consisting of two cases observed by two separate pairs of examiners was 0.42. Sources of variance included candidate proficiency (25%), case difficulty and case specificity (63.4%), examiner stringency (3.5%) and other error (8.2%). To achieve a reliability coefficient of > 0.8 a candidate would need to perform 11 Hot Cases observed by two examiners. Conclusion The reliability coefficient for the Hot Case component of the CICM second part examination is below the generally accepted value for a high stakes examination. Modifications to case selection and introduction of a clear scoring rubric to mitigate the effects of variation in case difficulty may be helpful. Increasing the number of cases and overall assessment time appears to be the best way to increase the overall reliability. Further research is required to assess the combined reliability of the Hot Case and viva components.
Related Links	https://bmcmededuc.biomedcentral.com/counter/pdf/10.1186/s12909-024-05516-w.pdf
Ending Page	9
Page Count	9
Starting Page	1
File Format	HTM / HTML
ISSN	14726920
DOI	10.1186/s12909-024-05516-w
Journal	BMC Medical Education
Issue Number	1
Volume Number	24
Language	English
Publisher	BioMed Central
Publisher Date	2024-05-11
Access Restriction	Open
Subject Keyword	Medical Education Theory of Medicine Bioethics Intensive care Examination Credentialling Reliability Generalisability theory Theory of Medicine/Bioethics
Content Type	Text
Resource Type	Article
Subject	Education Medicine
Journal Impact Factor	2.7/2023
5-Year Journal Impact Factor	3.4/2023

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

The validity and reliability of the sixth-year internal medical examination administered at the King Abdulaziz University Medical College

Preparing Australian diabetes educators for practice: workforce perceptions of the credentialling pathway

Pain medicine content, teaching and assessment in medical school curricula in Australia and New Zealand

Leadership curricula and assessment in Australian and New Zealand medical schools

A survey of cariology teaching in Australia and New Zealand

The College of Intensive Care Medicine of Australia and New Zealand supports global collaboration and respect.

Coping strategies and the Salutogenic Model in future oral health professionals

Medical student wellbeing – a consensus statement from Australia and New Zealand

Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission

The reliability of the College of Intensive Care Medicine of Australia and New Zealand “Hot Case” examination

Similar Documents

The validity and reliability of the sixth-year internal medical examination administered at the King Abdulaziz University Medical College

Preparing Australian diabetes educators for practice: workforce perceptions of the credentialling pathway

Pain medicine content, teaching and assessment in medical school curricula in Australia and New Zealand

Leadership curricula and assessment in Australian and New Zealand medical schools

A survey of cariology teaching in Australia and New Zealand

The College of Intensive Care Medicine of Australia and New Zealand supports global collaboration and respect.

Coping strategies and the Salutogenic Model in future oral health professionals

Medical student wellbeing – a consensus statement from Australia and New Zealand

Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission

The reliability of the College of Intensive Care Medicine of Australia and New Zealand “Hot Case” examination