NDLI: Real-time expression transfer for facial reenactment

Content Provider	ACM Digital Library
Author	Stamminger, Marc Thies, Justus Theobalt, Christian Nießner, Matthias Valgaerts, Levi Zollhöfer, Michael
Copyright Year	2015
Abstract	We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.
Starting Page	1
Ending Page	14
Page Count	14
File Format	PDF
ISSN	07300301
e-ISSN	15577368
DOI	10.1145/2816795.2818056
Volume Number	34
Issue Number	6
Journal	ACM Transactions on Graphics (TOG)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-10-26
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Depth camera Expression transfer Faces Real-time
Content Type	Text
Resource Type	Article
Subject	Computer Graphics and Computer-Aided Design

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Real-time shading-based refinement for consumer depth cameras

Facial performance sensing head-mounted display

Video-audio driven real-time facial animation

Real-time Expression Transfer for Facial Reenactment

Expression flow for 3D-aware face component transfer

Realtime facial animation with on-the-fly correctives

Precomputed local radiance transfer for real-time lighting design

Real-time non-rigid reconstruction using an RGB-D camera

Displaced dynamic expression regression for real-time facial tracking and animation

Real-time expression transfer for facial reenactment

Similar Documents

Real-time shading-based refinement for consumer depth cameras

Facial performance sensing head-mounted display

Video-audio driven real-time facial animation

Real-time Expression Transfer for Facial Reenactment

Expression flow for 3D-aware face component transfer

Realtime facial animation with on-the-fly correctives

Precomputed local radiance transfer for real-time lighting design

Real-time non-rigid reconstruction using an RGB-D camera

Displaced dynamic expression regression for real-time facial tracking and animation

Real-time expression transfer for facial reenactment