NDLI: Multi-mode neural network for human action recognition

Content Provider	IET Digital Library
Author	Zhao, Haohua Xue, Weichen Li, Xiaobo Gu, Zhangxuan Niu, Li Zhang, Liqing
Abstract	Video data are of two different intrinsic modes, in-frame and temporal. It is beneficial to incorporate static in-frame features to acquire dynamic features for video applications. However, some existing methods such as recurrent neural networks do not have a good performance, and some other such as 3D convolutional neural networks (CNNs) are both memory consuming and time consuming. This study proposes an effective framework that takes the advantage of deep learning on the static image feature extraction to tackle the video data. After extracting in-frame feature vectors using a pretrained deep network, the authors integrate them and form a multi-mode feature matrix, which preserves the multi-mode structure and high-level representation. They propose two models for follow-up classification. The authors first introduce a temporal CNN, which directly feeds the multi-mode feature matrix into a CNN. However, they show that characteristics of the multi-mode features differ significantly in distinct modes. The authors therefore further propose the multi-mode neural network (MMNN), in which different modes deploy different types of layers. They evaluate their algorithm with the task of human action recognition. The experimental results show that the MMNN achieves a much better performance than the existing long short-term memory-based methods and consumes far fewer resources than the existing 3D end-to-end models.
Starting Page	587
Ending Page	596
Page Count	10
ISSN	17519632
Volume Number	14
e-ISSN	17519640
Issue Number	Issue 8, Dec (2020)
Alternate Webpage(s)	https://digital-library.theiet.org/content/journals/iet-cvi/14/8
Alternate Webpage(s)	https://digital-library.theiet.org/content/journals/10.1049/iet-cvi.2019.0761
Journal	IET Computer Vision
Publisher Date	2020-04-30
Access Restriction	Open
Rights Holder	© The Institution of Engineering and Technology
Subject Keyword	3D Convolutional Neural Networks Computer Vision And Image Processing Technique Convolutional Neural Nets Dynamic Features Feature Extraction Human Action Recognition Image Classification Image Motion Analysis Image Recognition Image Representation In-frame Feature In-frame Feature Vector Intrinsic Mode Knowledge Engineering Technique Learning in AI Multimode Feature Matrix Multimode Neural Network Multimode Structure Neural Computing Technique Pretrained Deep Network Recurrent Neural Nets Recurrent Neural Network Static Image Feature Extraction Video Application Video Data Video Signal Processing
Content Type	Text
Resource Type	Article
Subject	Computer Vision and Pattern Recognition Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Fusing HOG and convolutional neural network spatial–temporal features for video-based facial expression recognition

Learning to recognise 3D human action from a new skeleton-based representation using deep convolutional neural networks

Fully convolutional networks for action recognition

Skeleton-based attention-aware spatial–temporal model for action detection and recognition

DeepFuseOSV: online signature verification using hybrid feature fusion and depthwise separable convolution neural network architecture

Dynamic gesture recognition based on feature fusion network and variant ConvLSTM

Multi-stream 3D CNN structure for human action recognition trained by limited data

Modulation scheme recognition using convolutional neural network

Residual-wider convolutional neural network for image recognition

Multi-mode neural network for human action recognition

Similar Documents

Fusing HOG and convolutional neural network spatial–temporal features for video-based facial expression recognition

Learning to recognise 3D human action from a new skeleton-based representation using deep convolutional neural networks

Fully convolutional networks for action recognition

Skeleton-based attention-aware spatial–temporal model for action detection and recognition

DeepFuseOSV: online signature verification using hybrid feature fusion and depthwise separable convolution neural network architecture

Dynamic gesture recognition based on feature fusion network and variant ConvLSTM

Multi-stream 3D CNN structure for human action recognition trained by limited data

Modulation scheme recognition using convolutional neural network

Residual-wider convolutional neural network for image recognition

Multi-mode neural network for human action recognition