NDLI: Mutual information guided 3D ResNet for self-supervised video representation learning

Content Provider	IET Digital Library
Author	Xue, Fei Ji, Hongbing Zhang, Wenbo
Abstract	In this work, the authors propose a novel self-supervised learning method based on mutual information to learn representations from the videos without manual annotation. Different video clips sampled from the same video usually have coherence in the temporal domain. To guide the network to learn such temporal coherence, they maximise the mutual information between global features extracted from different clips sampled from the same video (Global-MI). However, maximising the Global-MI leads the network to seek shared content from different video clips and may make the network degenerate to focus on the background of the video. Considering the structure of the video, they further maximise the average mutual information between the global feature and local patches of multiple regions of the video clip (multi-region Local-MI). Their approach, which is called Max-GL, learns the temporal coherence by jointly maximising the Global-MI and multi-region Local-MI. Experiments are conducted to validate the effectiveness of the proposed Max-GL. Experimental results show that the Max-GL can serve as an effective pre-training method for the task of action recognition in videos. Additional experiments for the task of action similarity labelling and dynamic scene recognition also validate the generalisation of the learned representations of the Max-GL.
Starting Page	3066
Ending Page	3075
Page Count	10
ISSN	17519659
Volume Number	14
e-ISSN	17519667
Issue Number	Issue 13, Nov (2020)
Alternate Webpage(s)	https://digital-library.theiet.org/content/journals/iet-ipr/14/13
Alternate Webpage(s)	https://digital-library.theiet.org/content/journals/10.1049/iet-ipr.2020.0019
Journal	IET Image Processing
Publisher Date	2020-06-05
Access Restriction	Open
Rights Holder	© The Institution of Engineering and Technology
Subject Keyword	Average Mutual Information Computer Vision And Image Processing Technique Different Clips Different Video Clips Feature Extraction Global Feature Global-MI Image Recognition Image Representation Knowledge Engineering Technique Learned Representations Learning in AI Local Patches Max-GL Network Degenerate Novel Self-supervised Learning Method Optical, Image And Video Signal Processing Self-supervised Video Representation Learning Statistics Temporal Coherence Temporal Domain Video Clip Video Signal Processing
Content Type	Text
Resource Type	Article
Subject	Signal Processing Electrical and Electronic Engineering Computer Vision and Pattern Recognition Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Spatial–temporal representation for video re-identification via key images

3D Features for human action recognition with semi-supervised learning

Fusing the facial temporal information in videos for face recognition

Capturing the spatio-temporal continuity for video semantic segmentation

Deep facial emotion recognition in video using eigenframes

Self-supervised Temporal Discriminative Learning for Video Representation Learning

Editorial: Special Issue on Unsupervised Feature Learning for Vision

Motion boundary emphasised optical flow method for human action recognition

Fusing HOG and convolutional neural network spatial–temporal features for video-based facial expression recognition

Mutual information guided 3D ResNet for self-supervised video representation learning

Similar Documents

Spatial–temporal representation for video re-identification via key images

3D Features for human action recognition with semi-supervised learning

Fusing the facial temporal information in videos for face recognition

Capturing the spatio-temporal continuity for video semantic segmentation

Deep facial emotion recognition in video using eigenframes

Self-supervised Temporal Discriminative Learning for Video Representation Learning

Editorial: Special Issue on Unsupervised Feature Learning for Vision

Motion boundary emphasised optical flow method for human action recognition

Fusing HOG and convolutional neural network spatial–temporal features for video-based facial expression recognition

Mutual information guided 3D ResNet for self-supervised video representation learning