NDLI: Exploiting Scene Context for Image Captioning

Please wait, while we are loading the content...

Exploiting Scene Context for Image Captioning

News Event Understanding by Mining Latent Factors From Multimodal Tensors

Jointly Representing Images and Text: Dependency Graphs, Word Senses, and Multimodal Embeddings

Multimodal and Crossmodal Representation Learning from Textual and Visual Features with Bidirectional Deep Neural Networks for Video Hyperlinking

Beyond Language and Vision, Towards Truly Multimedia Integration

Cross-modal Classification by Completing Unimodal Representations

User Video Summarization Based on Joint Visual and Semantic Affinity Graph

Semantic Indexing of Wearable Camera Images: Kids'Cam Concepts

Disinformation in Multimedia Annotation: Misleading Metadata Detection on YouTube

Exploiting Scene Context for Image Captioning

Content Provider	ACM Digital Library
Author	Shetty, Rakshith Laaksonen, Jorma R.-Tavakoli, Hamed
Abstract	This paper presents a framework for image captioning by exploiting the scene context. To date, most of the captioning models have been relying on the combination of Convolutional Neural Networks (CNN) and the Long-Short Term Memory (LSTM) model, trained in an end-to-end fashion. Recently, there has been extensive research towards improving the language model and the CNN architecture, utilizing attention mechanisms, and improving the learning techniques in such systems. A less studied area is the contribution of the scene context in the captioning. In this work, we study the role of the scene context, consisting of the scene type and objects. To this end, we augment the CNN features with scene context features, including scene detectors, objects and their localization, and their combinations. We use the scene context features as an initialization feature at the zeroth time step in a LSTM model with deep residual connections. In subsequent time steps, the model, however, uses the original CNN features. The proposed language model, contrary to more conventional ones, thus has access to visual features through the whole process of sentence generation. We demonstrate that the scene context features affect the language formation and improve the captioning results in the proposed framework. We also report results from the Microsoft COCO benchmark, where our model achieves the state-of-the-art performance on the test set.
Starting Page	1
Ending Page	8
Page Count	8
File Format	PDF
ISBN	9781450345194
DOI	10.1145/2983563.2983571
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2016-10-16
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Deep learning Computer vision Image captioning Long-short term memory
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in