NDLI: An audio-visual dataset of human–human interactions in stressful situations

Content Provider	Springer Nature Link
Author	Lefter, Iulia Burghouts, Gertjan J. Rothkrantz, Leon J. M.
Copyright Year	2014
Abstract	Stressful situations are likely to occur at human operated service desks, as well as at human–computer interfaces used in public domain. Automatic surveillance can help notifying when extra assistance is needed. Human communication is inherently multimodal e.g. speech, gestures, facial expressions. It is expected that automatic surveillance systems can benefit from exploiting multimodal information. This requires automatic fusion of modalities, which is still an unsolved problem. To support the development of such systems, we present and analyze audio-visual recordings of human–human interactions at a service desk. The corpus has a high degree of realism: all interactions are freely improvised by actors based on short scenarios where only the sources of conflict were provided. The recordings can be considered as a prototype for general stressful human–human interaction. The recordings were annotated on a 5 point scale on degree of stress from the perspective of surveillance operators. The recordings are very rich in hand gestures. We find that the more stressful the situation, the higher the proportion of speech that is accompanied by gestures. Understanding the function of gestures and their relation to speech is essential for good fusion strategies. Taking speech as the basic modality, one of our research questions was, what is the role of gestures in addition to speech. Both speech and gestures can express emotion, so we say that they have an emotional function. They can also express non-emotional information, in which case we say that they have a semantic function. We learn that when speech and gestures have the same function, they are usually congruent, but intensities and clarity can vary. Most gestures in this dataset convey emotion. We identify classes of gestures in our recordings, and argue that some classes are clear indications of stressful situations
Starting Page	29
Ending Page	41
Page Count	13
File Format	PDF
ISSN	17837677
Journal	Journal on Multimodal User Interfaces
Volume Number	8
Issue Number	1
e-ISSN	17838738
Language	English
Publisher	Springer Berlin Heidelberg
Publisher Date	2014-04-04
Publisher Place	Berlin, Heidelberg
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Stressful situations dataset Speech Gestures Multimodal fusion Multimodal communication User Interfaces and Human Computer Interaction Signal, Image and Speech Processing Image Processing and Computer Vision
Content Type	Text
Resource Type	Article
Subject	Signal Processing Human-Computer Interaction

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Multimodal user interface for the communication of the disabled

Emergent leaders through looking and speaking: from audio-visual data to multimodal recognition

Interactive design of multimodal user interfaces : Reducing technical and visual complexity

Cross-disciplinary approaches to multimodal user interfaces

Enriching the user experience with multimodal interfaces

Synchronizing multimodal recordings using audio-to-audio alignment : An application of acoustic fingerprinting to facilitate music interaction research

Communication of musical expression by means of mobile robot gestures

From multimodal analysis to real-time interactions with virtual agents

A french corpus of audio and multimodal interactions in a health smart home

An audio-visual dataset of human–human interactions in stressful situations

Similar Documents

Multimodal user interface for the communication of the disabled

Emergent leaders through looking and speaking: from audio-visual data to multimodal recognition

Interactive design of multimodal user interfaces : Reducing technical and visual complexity

Cross-disciplinary approaches to multimodal user interfaces

Enriching the user experience with multimodal interfaces

Synchronizing multimodal recordings using audio-to-audio alignment : An application of acoustic fingerprinting to facilitate music interaction research

Communication of musical expression by means of mobile robot gestures

From multimodal analysis to real-time interactions with virtual agents

A french corpus of audio and multimodal interactions in a health smart home

An audio-visual dataset of human–human interactions in stressful situations