NDLI: Audio-visual speech recognition using deep learning

Content Provider	Springer Nature Link
Author	da, Kuniaki Yamaguchi, Yuki Nakadai, Kazuhiro Oku, Hiroshi G. Ogata, Tetsuya
Copyright Year	2014
Abstract	Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signal-to-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.
Starting Page	722
Ending Page	737
Page Count	16
File Format	PDF
ISSN	0924669X
Journal	Applied Intelligence
Volume Number	42
Issue Number	4
e-ISSN	15737497
Language	English
Publisher	Springer US
Publisher Date	2014-12-20
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Audio-visual speech recognition Feature extraction Deep learning Multi-stream HMM Artificial Intelligence (incl. Robotics) Mechanical Engineering Manufacturing, Machines, Tools
Content Type	Text
Resource Type	Article
Subject	Artificial Intelligence

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Audio-visual speech recognition using deep learning

Incremental Feature Selection

Unsupervised Extraction of Structural Information from High Dimensional Visual Data

MPadal: a joint local-and-global multi-view feature selection method for activity recognition

Activity recognition and intensity estimation in youth from accelerometer data aided by machine learning

Fuzzy Inductive Learning Strategies

Augmenting the Discrimination Power of HMM by NN for On-Line Cursive Script Recognition

Qualitative Geometry for Shape Recognition

Sparse discriminative region selection algorithm for face recognition

Audio-visual speech recognition using deep learning

Similar Documents

Audio-visual speech recognition using deep learning

Incremental Feature Selection

Unsupervised Extraction of Structural Information from High Dimensional Visual Data

MPadal: a joint local-and-global multi-view feature selection method for activity recognition

Activity recognition and intensity estimation in youth from accelerometer data aided by machine learning

Fuzzy Inductive Learning Strategies

Augmenting the Discrimination Power of HMM by NN for On-Line Cursive Script Recognition

Qualitative Geometry for Shape Recognition

Sparse discriminative region selection algorithm for face recognition

Audio-visual speech recognition using deep learning