NDLI: Joint embeddings of shapes and images via CNN image purification

Content Provider	ACM Digital Library
Author	Qi, Charles Ruizhongtai Fish, Noa Su, Hao Li, Yangyan Cohen-or, Daniel Guibas, Leonidas J.
Copyright Year	2015
Abstract	Both 3D models and 2D images contain a wealth of information about everyday objects in our environment. However, it is difficult to semantically link together these two media forms, even when they feature identical or very similar objects. We propose a $\textit{joint}$ embedding space populated by both 3D shapes and 2D images of objects, where the distances between embedded entities reflect similarity between the underlying objects. This joint embedding space facilitates comparison between entities of either form, and allows for cross-modality retrieval. We construct the embedding space using 3D shape similarity measure, as 3D shapes are more pure and complete than their appearance in images, leading to more robust distance metrics. We then employ a Convolutional Neural Network (CNN) to "purify" images by muting distracting factors. The CNN is trained to map an image to a point in the embedding space, so that it is close to a point attributed to a 3D model of a similar object to the one depicted in the image. This purifying capability of the CNN is accomplished with the help of a large amount of training data consisting of images synthesized from 3D shapes. Our joint embedding allows cross-view image retrieval, image-based shape retrieval, as well as shape-based image retrieval. We evaluate our method on these retrieval tasks and show that it consistently out-performs state-of-the-art methods, and demonstrate the usability of a joint embedding in a number of additional applications.
Starting Page	1
Ending Page	12
Page Count	12
File Format	PDF
ISSN	07300301
e-ISSN	15577368
DOI	10.1145/2816795.2818071
Volume Number	34
Issue Number	6
Journal	ACM Transactions on Graphics (TOG)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-10-26
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	3D shapes Deep learning Embedding
Content Type	Text
Resource Type	Article
Subject	Computer Graphics and Computer-Aided Design

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

GUIBAS L.: Joint embeddings of shapes and images via cnn image purification (2015)

Shape2Vec: semantic-based descriptors for 3D shapes, sketches and images

Identifying style of 3D shapes using deep metric learning

Blind Image Blur Estimation via Deep Learning.

Deep Ranking for Person Re-Identification via Joint Representation Learning.

Joint Learning for Single-Image Super-Resolution via a Coupled Constraint

Learning-Based, Automatic 2D-to-3D Image and Video Conversion

Folding domain-specific languages:deep and shallow embeddings (functional Pearl)

Accurate detection of symmetries in 3D shapes

Joint embeddings of shapes and images via CNN image purification

Similar Documents

GUIBAS L.: Joint embeddings of shapes and images via cnn image purification (2015)

Shape2Vec: semantic-based descriptors for 3D shapes, sketches and images

Identifying style of 3D shapes using deep metric learning

Blind Image Blur Estimation via Deep Learning.

Deep Ranking for Person Re-Identification via Joint Representation Learning.

Joint Learning for Single-Image Super-Resolution via a Coupled Constraint

Learning-Based, Automatic 2D-to-3D Image and Video Conversion

Folding domain-specific languages:deep and shallow embeddings (functional Pearl)

Accurate detection of symmetries in 3D shapes

Joint embeddings of shapes and images via CNN image purification