NDLI: Deep Multimodal Learning for Affective Analysis and Retrieval

Content Provider	IEEE Xplore Digital Library
Author	Lei Pang Shiai Zhu Chong-Wah Ngo
Copyright Year	1999
Abstract	Social media has been a convenient platform for voicing opinions through posting messages, ranging from tweeting a short text to uploading a media file, or any combination of messages. Understanding the perceived emotions inherently underlying these user-generated contents (UGC) could bring light to emerging applications such as advertising and media analytics. Existing research efforts on affective computation are mostly dedicated to single media, either text captions or visual content. Few attempts for combined analysis of multiple media are made, despite that emotion can be viewed as an expression of multimodal experience. In this paper, we explore the learning of highly non-linear relationships that exist among low-level features across different modalities for emotion prediction. Using the deep Bolzmann machine (DBM), a joint density model over the space of multimodal inputs, including visual, auditory, and textual modalities, is developed. The model is trained directly using UGC data without any labeling efforts. While the model learns a joint representation over multimodal inputs, training samples in absence of certain modalities can also be leveraged. More importantly, the joint representation enables emotion-oriented cross-modal retrieval, for example, retrieval of videos using the text query “crazy cat”. The model does not restrict the types of input and output, and hence, in principle, emotion prediction and retrieval on any combinations of media are feasible. Extensive experiments on web videos and images show that the learnt joint representation could be very compact and be complementary to hand-crafted features, leading to performance improvement in both emotion classification and cross-modal retrieval.
Sponsorship	IEEE Signal Processing Society IEEE Circuits and Systems Society IEEE Communications Society IEEE Computer Society
Starting Page	2008
Ending Page	2020
Page Count	13
File Size	2553080
File Format	PDF
ISSN	15209210
Volume Number	17
Issue Number	11
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-01-01
Publisher Place	U.S.A.
Access Restriction	One Nation One Subscription (ONOS)
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Visualization Joints Media Videos Semantics Training Feature extraction multimodal learning Cross-modal retrieval deep Boltzmann machine emotion analysis
Content Type	Text
Resource Type	Article
Subject	Signal Processing Electrical and Electronic Engineering Computer Science Applications Media Technology

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Learning Cross Space Mapping via DNN Using Large Scale Click-Through Logs

Rating Image Aesthetics Using Deep Learning

Cross-Domain Feature Learning in Multimedia

Affective Visualization and Retrieval for Music Video

Sparse Ensemble Learning for Concept Detection

Learning Representative Deep Features for Image Set Analysis

Kernel Cross-Modal Factor Analysis for Information Fusion With Application to Bimodal Emotion Recognition

A Continuous Learning Framework for Activity Recognition Using Deep Hybrid Feature Models

Learning Feature Hierarchies: A Layer-Wise Tag-Embedded Approach

Deep Multimodal Learning for Affective Analysis and Retrieval

Similar Documents

Learning Cross Space Mapping via DNN Using Large Scale Click-Through Logs

Rating Image Aesthetics Using Deep Learning

Cross-Domain Feature Learning in Multimedia

Affective Visualization and Retrieval for Music Video

Sparse Ensemble Learning for Concept Detection

Learning Representative Deep Features for Image Set Analysis

Kernel Cross-Modal Factor Analysis for Information Fusion With Application to Bimodal Emotion Recognition

A Continuous Learning Framework for Activity Recognition Using Deep Hybrid Feature Models

Learning Feature Hierarchies: A Layer-Wise Tag-Embedded Approach

Deep Multimodal Learning for Affective Analysis and Retrieval