NDLI: EmoNets: Multimodal deep learning approaches for emotion recognition in video

Content Provider	Springer Nature Link
Author	Kahou, Samira Ebrahimi Bouthillier, Xavier Lamblin, Pascal Gulcehre, Caglar Michalski, Vincent Konda, Kishore Jean, Sébastien Froumenty, Pierre Dauphin, Yann Boulanger Lewandowski, Nicolas Chandias Ferrari, Raul Mirza, Mehdi Warde Farley, David Courville, Aaron Vincent, Pascal Memisevic, Roland Pal, Christopher Bengio, Yoshua
Copyright Year	2015
Abstract	The task of the Emotion Recognition in the Wild (EmotiW) Challenge is to assign one of seven emotions to short video clips extracted from Hollywood style movies. The videos depict acted-out emotions under realistic conditions with a large degree of variation in attributes such as pose and illumination, making it worthwhile to explore approaches which consider combinations of features from multiple modalities for label assignment. In this paper we present our approach to learning several specialist models using deep learning techniques, each focusing on one modality. Among these are a convolutional neural network, focusing on capturing visual information in detected faces, a deep belief net focusing on the representation of the audio stream, a K-Means based “bag-of-mouths” model, which extracts visual features around the mouth region and a relational autoencoder, which addresses spatio-temporal aspects of videos. We explore multiple methods for the combination of cues from these modalities into one common classifier. This achieves a considerably greater accuracy than predictions from our strongest single-modality classifier. Our method was the winning submission in the 2013 EmotiW challenge and achieved a test set accuracy of 47.67 % on the 2014 dataset.
Starting Page	99
Ending Page	111
Page Count	13
File Format	PDF
ISSN	17837677
Journal	Journal on Multimodal User Interfaces
Volume Number	10
Issue Number	2
e-ISSN	17838738
Language	English
Publisher	Springer International Publishing
Publisher Date	2015-08-21
Publisher Place	Cham
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Emotion recognition Deep learning Model combination Multimodal learning User Interfaces and Human Computer Interaction Signal, Image and Speech Processing Image Processing and Computer Vision
Content Type	Text
Resource Type	Article
Subject	Signal Processing Human-Computer Interaction

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A multimodal emotion corpus for Filipino and its uses

Cross-disciplinary approaches to multimodal user interfaces

Video modeling and learning on Riemannian manifold for emotion recognition in the wild

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

Induction, recording and recognition of natural emotions from facial expressions and speech prosody

SVM-based feature selection methods for emotion recognition from multimodal data

Multimodal interaction: a survey from model driven engineering and mobile perspectives

Emotion recognition in the wild

Multimodal user interface for the communication of the disabled

EmoNets: Multimodal deep learning approaches for emotion recognition in video

Similar Documents

A multimodal emotion corpus for Filipino and its uses

Cross-disciplinary approaches to multimodal user interfaces

Video modeling and learning on Riemannian manifold for emotion recognition in the wild

Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis

Induction, recording and recognition of natural emotions from facial expressions and speech prosody

SVM-based feature selection methods for emotion recognition from multimodal data

Multimodal interaction: a survey from model driven engineering and mobile perspectives

Emotion recognition in the wild

Multimodal user interface for the communication of the disabled

EmoNets: Multimodal deep learning approaches for emotion recognition in video