NDLI: Speech driven realistic mouth animation based on multi-modal unit selection

Content Provider	Springer Nature Link
Author	Jiang, Dongmei Ravyse, Ilse Sahli, Hichem Verhelst, Werner
Copyright Year	2009
Abstract	This paper presents a novel audio visual diviseme (viseme pair) instance selection and concatenation method for speech driven photo realistic mouth animation. Firstly, an audio visual diviseme database is built consisting of the audio feature sequences, intensity sequences and visual feature sequences of the instances. In the Viterbi based diviseme instance selection, we set the accumulative cost as the weighted sum of three items: 1) logarithm of concatenation smoothness of the synthesized mouth trajectory; 2) logarithm of the pronunciation distance; 3) logarithm of the audio intensity distance between the candidate diviseme instance and the target diviseme segment in the incoming speech. The selected diviseme instances are time warped and blended to construct the mouth animation. Objective and subjective evaluations on the synthesized mouth animations prove that the multimodal diviseme instance selection algorithm proposed in this paper outperforms the triphone unit selection algorithm in Video Rewrite. Clear, accurate, smooth mouth animations can be obtained matching well with the pronunciation and intensity changes in the incoming speech. Moreover, with the logarithm function in the accumulative cost, it is easy to set the weights to obtain optimal mouth animations.
Starting Page	157
Ending Page	169
Page Count	13
File Format	PDF
ISSN	17837677
Journal	Journal on Multimodal User Interfaces
Volume Number	2
Issue Number	3-4
e-ISSN	17838738
Language	English
Publisher	Springer-Verlag
Publisher Date	2009-06-24
Publisher Place	Berlin, Heidelberg
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Mouth animation Audio visual diviseme instance selection Concatenation smoothness Pronunciation distance Intensity distance Image Processing and Computer Vision Signal, Image and Speech Processing User Interfaces and Human Computer Interaction
Content Type	Text
Resource Type	Article
Subject	Signal Processing Human-Computer Interaction

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

An audio-driven dancing avatar

Non-visual identification, localization, and selection of entities of interest in a 3D environment

Comparison between different feature extraction techniques for audio-visual speech recognition

Audio-visual emotion recognition using multi-directional regression and Ridgelet transform

An audio-visual dataset of human–human interactions in stressful situations

RAVEL: an annotated corpus for training robots with audiovisual abilities

A kinect-based interface to animate virtual characters

Interactive design of multimodal user interfaces : Reducing technical and visual complexity

Combining modality-specific extreme learning machines for emotion recognition in the wild

Speech driven realistic mouth animation based on multi-modal unit selection

Similar Documents

An audio-driven dancing avatar

Non-visual identification, localization, and selection of entities of interest in a 3D environment

Comparison between different feature extraction techniques for audio-visual speech recognition

Audio-visual emotion recognition using multi-directional regression and Ridgelet transform

An audio-visual dataset of human–human interactions in stressful situations

RAVEL: an annotated corpus for training robots with audiovisual abilities

A kinect-based interface to animate virtual characters

Interactive design of multimodal user interfaces : Reducing technical and visual complexity

Combining modality-specific extreme learning machines for emotion recognition in the wild

Speech driven realistic mouth animation based on multi-modal unit selection