NDLI: Inter-rater reliability for emotion annotation in human–computer interaction: comparison and methodological improvements

Content Provider	Springer Nature Link
Author	Siegert, Ingo Böck, Ronald Wendemuth, Andreas
Copyright Year	2013
Abstract	To enable a naturalistic human–computer interaction the recognition of emotions and intentions experiences increased attention and several modalities are comprised to cover all human communication abilities. For this reason, naturalistic material is recorded, where the subjects are guided through an interaction with crucial points, but with the freedom to react individually. This material captures realistic user reactions but lacks of clear labels. So, a good transcription and annotation of the given material is essential. For that, the assignment of human annotators has become widely accepted. A good measurement for the reliability of labelled material is the inter-rater agreement. In this paper we investigate the achieved inter-rater agreement utilizing Krippendorff’s alpha for emotional annotated interaction corpora and present methods to improve the reliability, we show that the reliabilities obtained with different methods does not differ much, so a choice could rely on other aspects. Furthermore, a multimodal presentation of the items in their natural order increases the reliability.
Starting Page	17
Ending Page	28
Page Count	12
File Format	PDF
ISSN	17837677
Journal	Journal on Multimodal User Interfaces
Volume Number	8
Issue Number	1
e-ISSN	17838738
Language	English
Publisher	Springer Berlin Heidelberg
Publisher Date	2013-11-29
Publisher Place	Berlin, Heidelberg
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Affective state Annotation Context influence Inter-rater agreement Labelling User Interfaces and Human Computer Interaction Signal, Image and Speech Processing Image Processing and Computer Vision
Content Type	Text
Resource Type	Article
Subject	Signal Processing Human-Computer Interaction

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Affective transfer computing model based on attenuation emotion mechanism

Emotion-based interaction

A three-component framework for empathic technologies to augment human interaction

Induction, recording and recognition of natural emotions from facial expressions and speech prosody

Emotion recognition in the wild

Facial expression-based affective speech translation

Multimodal user’s affective state analysis in naturalistic interaction

Annotation and interpretation of prosodic data in the HuComTech corpus for multimodal user interfaces

Investigating effects of avatars on primary school children’s affective responses to learning

Inter-rater reliability for emotion annotation in human–computer interaction: comparison and methodological improvements

Similar Documents

Affective transfer computing model based on attenuation emotion mechanism

Emotion-based interaction

A three-component framework for empathic technologies to augment human interaction

Induction, recording and recognition of natural emotions from facial expressions and speech prosody

Emotion recognition in the wild

Facial expression-based affective speech translation

Multimodal user’s affective state analysis in naturalistic interaction

Annotation and interpretation of prosodic data in the HuComTech corpus for multimodal user interfaces

Investigating effects of avatars on primary school children’s affective responses to learning

Inter-rater reliability for emotion annotation in human–computer interaction: comparison and methodological improvements