NDLI: Visual Speech Synthesis Using a Variable-Order Switching Shared Gaussian Process Dynamical Model

Content Provider	IEEE Xplore Digital Library
Author	Deena, S. Shaobo Hou Galata, A.
Copyright Year	1999
Abstract	In this paper, we present a novel approach to speech- driven facial animation using a non-parametric switching state space model based on Gaussian processes. The model is an extension of the shared Gaussian process dynamical model, augmented with switching states. Two talking head corpora are processed by extracting visual and audio data from the sequences followed by a parameterization of both data streams. Phonetic labels are obtained by performing forced phonetic alignment on the audio. The switching states are found using a variable length Markov model trained on the labelled phonetic data. The audio and visual data corresponding to phonemes matching each switching state are extracted and modelled together using a shared Gaussian process dynamical model. We propose a synthesis method that takes into account both previous and future phonetic context, thus accounting for forward and backward coarticulation in speech. Both objective and subjective evaluation results are presented. The quantitative results demonstrate that the proposed method outperforms other state-of-the-art methods in visual speech synthesis and the qualitative results reveal that the synthetic videos are comparable to ground truth in terms of visual perception and intelligibility.
Sponsorship	IEEE Signal Processing Society IEEE Circuits and Systems Society IEEE Communications Society IEEE Computer Society
Starting Page	1755
Ending Page	1768
Page Count	14
File Size	2154689
File Format	PDF
ISSN	15209210
Volume Number	15
Issue Number	8
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-01-01
Publisher Place	U.S.A.
Access Restriction	One Nation One Subscription (ONOS)
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Visualization Hidden Markov models Speech Speech synthesis Switches Data models Animation visual speech synthesis Artificial Talking Head speech-driven facial animation
Content Type	Text
Resource Type	Article
Subject	Signal Processing Electrical and Electronic Engineering Computer Science Applications Media Technology

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Animation of generic 3D head models driven by speech

Joint Audiovisual Hidden Semi-Markov Model-Based Speech Synthesis

Visual Speech Synthesis by Modelling Coarticulation Dynamics using a Non-Parametric Switching State-Space Model

Speech-Driven Facial Animation Using A Shared Gaussian Process Latent Variable Model

Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Animation of generic 3D Head models driven by speech

Dynamic Audio-Visual Mapping using Fused Hidden Markov Model Inversion Method

Speech driven lips animation for the Farsi language

An online speech driven talking head system

Visual Speech Synthesis Using a Variable-Order Switching Shared Gaussian Process Dynamical Model

Similar Documents

Animation of generic 3D head models driven by speech

Joint Audiovisual Hidden Semi-Markov Model-Based Speech Synthesis

Visual Speech Synthesis by Modelling Coarticulation Dynamics using a Non-Parametric Switching State-Space Model

Speech-Driven Facial Animation Using A Shared Gaussian Process Latent Variable Model

Visual speech synthesis by modelling coarticulation dynamics using a non-parametric switching state-space model

Animation of generic 3D Head models driven by speech

Dynamic Audio-Visual Mapping using Fused Hidden Markov Model Inversion Method

Speech driven lips animation for the Farsi language

An online speech driven talking head system

Visual Speech Synthesis Using a Variable-Order Switching Shared Gaussian Process Dynamical Model