NDLI: How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs

Content Provider	PubMed Central
Author	Margarita, Stolarova Wolf, Corinna Rinker, Tanja Brielmann, Aenne
Copyright Year	2014
Abstract	This report has two main purposes. First, we combine well-known analytical approaches to conduct a comprehensive assessment of agreement and correlation of rating-pairs and to dis-entangle these often confused concepts, providing a best-practice example on concrete data and a tutorial for future reference. Second, we explore whether a screening questionnaire developed for use with parents can be reliably employed with daycare teachers when assessing early expressive vocabulary. A total of 53 vocabulary rating pairs (34 parent–teacher and 19 mother–father pairs) collected for two-year-old children (12 bilingual) are evaluated. First, inter-rater reliability both within and across subgroups is assessed using the intra-class correlation coefficient (ICC). Next, based on this analysis of reliability and on the test-retest reliability of the employed tool, inter-rater agreement is analyzed, magnitude and direction of rating differences are considered. Finally, Pearson correlation coefficients of standardized vocabulary scores are calculated and compared across subgroups. The results underline the necessity to distinguish between reliability measures, agreement and correlation. They also demonstrate the impact of the employed reliability on agreement evaluations. This study provides evidence that parent–teacher ratings of children's early vocabulary can achieve agreement and correlation comparable to those of mother–father ratings on the assessed vocabulary scale. Bilingualism of the evaluated child decreased the likelihood of raters' agreement. We conclude that future reports of agreement, correlation and reliability of ratings will benefit from better definition of terms and stricter methodological approaches. The methodological tutorial provided here holds the potential to increase comparability across empirical reports and can help improve research practices and knowledge transfer to educational and therapeutic settings.
Related Links	http://dx.doi.org/10.3389/fpsyg.2014.00509
Starting Page	509
File Format	PDF
ISSN	16641078
e-ISSN	16641078
Journal	Frontiers in Psychology
Volume Number	5
Language	English
Publisher	Frontiers Media S.A.
Publisher Date	2014-06-01
Access Restriction	Open
Rights Holder	Frontiers Media S.A.
Subject Keyword	Research in Higher Education
Content Type	Text
Resource Type	Article
Subject	Psychology

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs

Reliability and Validity of Parent and Teacher Ratings of Attention-Deficit/Hyperactivity Disorder Symptoms

Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings.

Agreement among adolescents, parents, and teachers on adolescent personality.

Does a child’s language ability affect the correspondence between parent and teacher ratings of ADHD symptoms?

Inter-Parent Agreement on the Syndrome Scales of the Child Behavior Checklist (CBCL): Correspondence and Discrepancies

Reliable Ratings or Reading Tea Leaves: Can Parent, Teacher, and Clinician Behavioral Ratings of Preschoolers Predict ADHD at Age Six?

How to assess the reliability of cerebral microbleed rating?

Parent and teacher mental health ratings of children using primary care services: Inter-rater agreement and implications for mental health screening

How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs

Similar Documents

How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs

Reliability and Validity of Parent and Teacher Ratings of Attention-Deficit/Hyperactivity Disorder Symptoms

Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings.

Agreement among adolescents, parents, and teachers on adolescent personality.

Does a child’s language ability affect the correspondence between parent and teacher ratings of ADHD symptoms?

Inter-Parent Agreement on the Syndrome Scales of the Child Behavior Checklist (CBCL): Correspondence and Discrepancies

Reliable Ratings or Reading Tea Leaves: Can Parent, Teacher, and Clinician Behavioral Ratings of Preschoolers Predict ADHD at Age Six?

How to assess the reliability of cerebral microbleed rating?

Parent and teacher mental health ratings of children using primary care services: Inter-rater agreement and implications for mental health screening

How to assess and compare inter-rater reliability, agreement and correlation of ratings: an exemplary analysis of mother-father and parent-teacher expressive vocabulary rating pairs