NDLI: Speech-driven talking face using embedded confusable system for real time mobile multimedia

Content Provider	Springer Nature Link
Author	Shih, Po Yi Paul, Anand Wang, Jhing Fa Chen, Yi Hung
Copyright Year	2013
Abstract	This paper presents a real-time speech-driven talking face system which provides low computational complexity and smoothly visual sense. A novel embedded confusable system is proposed to generate an efficient phoneme-viseme mapping table which is constructed by phoneme grouping using Houtgast similarity approach based on the results of viseme similarity estimation using histogram distance, according to the concept of viseme visually ambiguous. The generated mapping table can simplify the mapping problem and promote viseme classification accuracy. The implemented real time speech-driven talking face system includes: 1) speech signal processing, including SNR-aware speech enhancement for noise reduction and ICA-based feature set extractions for robust acoustic feature vectors; 2) recognition network processing, HMM and MCSVM are combined as a recognition network approach for phoneme recognition and viseme classification, which HMM is good at dealing with sequential inputs, while MCSVM shows superior performance in classifying with good generalization properties, especially for limited samples. The phoneme-viseme mapping table is used for MCSVM to classify the observation sequence of HMM results, which the viseme class is belong to; 3) visual processing, arranges lip shape image of visemes in time sequence, and presents more authenticity using a dynamic alpha blending with different alpha value settings. Presented by the experiments, the used speech signal processing with noise speech comparing with clean speech, could gain 1.1 % (16.7 % to 15.6 %) and 4.8 % (30.4 % to 35.2 %) accuracy rate improvements in PER and WER, respectively. For viseme classification, the error rate is decreased from 19.22 % to 9.37 %. Last, we simulated a GSM communication between mobile phone and PC for visual quality rating and speech driven feeling using mean opinion score. Therefore, our method reduces the number of visemes and lip shape images by confusable sets and enables real-time operation.
Starting Page	417
Ending Page	437
Page Count	21
File Format	PDF
ISSN	13807501
Journal	Multimedia Tools and Applications
Volume Number	73
Issue Number	1
e-ISSN	15737721
Language	English
Publisher	Springer US
Publisher Date	2013-08-17
Publisher Place	Boston
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Real-time speech driven Lip-synch Talking face Hidden markov model (HMM) Multiclass support vector machine (MCSVM) Viseme histogram similarity Confusion matrix Multimedia Information Systems Computer Communication Networks Data Structures, Cryptology and Information Theory Special Purpose and Application-Based Systems
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Hardware and Architecture Software Media Technology

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Speech-driven realtime lip-synch animation with viseme-dependent filters

Real-time lip-synch face animation driven by human voice

Real-time speech-driven lip synchronization

Audiovisual synchrony assessment for replay attack detection in talking face biometrics

HMM trajectory-guided sample selection for photo-realistic talking head

Clustering Persian viseme using phoneme subspace for developing visual speech application

Real-time face detection and phone-to-face distance measuring for speech recognition for multi-modal interface in mobile device

Generating emphatic speech with hidden Markov model for expressive speech synthesis

Comparison of Phoneme and Viseme Based Acoustic Units for Speech Driven Realistic lip Animation

Speech-driven talking face using embedded confusable system for real time mobile multimedia

Similar Documents

Speech-driven realtime lip-synch animation with viseme-dependent filters

Real-time lip-synch face animation driven by human voice

Real-time speech-driven lip synchronization

Audiovisual synchrony assessment for replay attack detection in talking face biometrics

HMM trajectory-guided sample selection for photo-realistic talking head

Clustering Persian viseme using phoneme subspace for developing visual speech application

Real-time face detection and phone-to-face distance measuring for speech recognition for multi-modal interface in mobile device

Generating emphatic speech with hidden Markov model for expressive speech synthesis

Comparison of Phoneme and Viseme Based Acoustic Units for Speech Driven Realistic lip Animation

Speech-driven talking face using embedded confusable system for real time mobile multimedia