Synthesizing Realistic Facial Expressions from Photographs

Size: px
Start display at page:

Download "Synthesizing Realistic Facial Expressions from Photographs"

Transcription

1 Synthesizing Realistic Facial Expressions from Photographs 1998 F. Pighin, J Hecker, D. Lischinskiy, R. Szeliskiz and D. H. Salesin University of Washington, The Hebrew University Microsoft Research 1

2 Introduction Techniques for creating photorealistic textured 3D facial models from photographs of a human subject and creating smooth transitions between different facial expressions by morphing. Model fitting: 1. User-assisted technique to recover the camera poses corresponding to the views + a sparse set of chosen locations on the subject s face. 2. Scattered data interpolation technique: deforms a generic face mesh to fit the particular geometry of the subject s face. Texture extraction: extracting one or more texture maps from the input images (repeated for several facial expressions). Expression morphing: Generating transitions between the corresponding face models, and blending the textures. 2

3 Motivation Realistic facial synthesis is one of the most fundamental problems in computer graphics, and one of the most difficult. First attempts date back to the 70s. Applications include character animation, computer games, video teleconferencing and facial surgery planning. No perfectly realistic facial animation has ever been generated by a computer. The facial animation Turing test has never been passed. 3

4 Motivation (2) Why is this such a difficult problem? The human face is an extremely complicated geometric form. The face exhibits countless tiny wrinkles and creases, as well as subtle variations in color and texture. Animation is even harder: facial movement is a product of underlying skeletal and muscular forms (+ skin and subcutaneous layers). We look at faces everyday, it is easy for us to detect even the slightest deviation from true faces. We can read facial expressions easily. 4

5 Earlier Work Computer Generated Animation of Faces, Frederic I. Parke (1972): The idea was to produce "realistic" computer generated animated sequences of a human face changing expression. Animation was accomplished using a cosine interpolation scheme to fill in the intermediate frames between expressions. Parke s work introduced simple geometric interpolation between face models. 5

6 Earlier Work (2) Feature-Based Image Metamorphosis (1992): 2D morphing between photographic images. Problems with 2D image morphing: It does not correctly account for changes in viewpoint or object pose. This has been recently addressed by a technique called viewmorphing. 2D morphing still lacks some of the advantages of a 3D model, such as the complete freedom of viewpoint. Michael Jackson s morphing music video 6

7 Similar Work A different approach: performance based animation; measurements from real actors are used to drive synthetic characters. Modeling, Tracking and Interactive Animation of Faces and Heads using Input from Video (1996): Idea: Coupling pixel-by-pixel measurements of surface motion to a physically-based face model and a muscle control model -> detailed spatio-temporal records of both the displacement of each point on the facial surface and the muscle control required to produce the facial motion. 7

8 Advantages of this Technique 1. Gives the user complete freedom in specifying the correspondences, and enables the user to refine the initial fit as needed. 2. Can handle arbitrary camera positions and lenses. 3. Creates realistic face models as well as realistic transitions for face expressions. 4. Is capable of extracting texture maps of high resolution. 5. Can capture subtle changes in illumination and appearance. Price: User intervention in the modeling process. 8

9 General Pipeline Capture multiple views of a human subject (with a given facial expression) using cameras at arbitrary locations. Deform a generic 3D face mesh to fit the face of the particular human subject. Extract texture maps from the photos (single view-independent texture map or view-dependent texture map). Manually mark a small set of initial corresponding points on the face in the different views. Automatically recover the camera parameters (position, focal length, etc.) for each view, and 3D positions. (repeated for different facial expressions) Facial animation: Interpolate between two or more different 3D models, while at the same time blending textures. 9

10 Model Fitting Task: Adapt a generic face model to fit an individual s face and facial expression. Input: 1. Several images of the face from different viewpoints. 2. Generic face model. Output: Face model that has been adapted to fit the face in the input images. User interaction: Selecting feature points and correspondences for refinement. 10

11 Model Fitting (2) 11

12 Model Fitting (3) Consists of 3 stages: 1. Pose Recovery: Apply computer vision techniques to estimate the viewing parameters (position, orientation, focal length) for each input camera -> recover 3D coordinates of feature points. 2. Scattered Data Interpolation: Compute the positions of the remaining face mesh vertices. 3. Shape Refinement: Specify additional correspondences between facial vertices and image coordinates to improve the estimated shape of the face. 12

13 Pose Recovery Technique based on the non-linear least squares structure. The problem is reduced to a series of linear least squares problems that can be solved using very simple and numerically stable techniques. 13

14 Pose Recovery (2) For each camera pose k we have a rotation matrix R k and a translation vector t k. The 3 rows of R k are r k x, rk y and rk z, and the 3 entries in tk are t k x, tk y and tk z. Each feature point p i has 2D screen coordinates in the k th image (x k i, yk i ). Traditional 3D projection equation: 14

15 Pose Recovery (3) Inverse distance World-to-image scale factor Advantage of this formulation: The scale factor can be reliably estimated even if f is too long. In the original formation there is a strong coupling between f and t z. 15

16 Pose Recovery (4) These 2 equations are linear in each of the unknowns p i, t k x, tk y, ηk, s k, and R k. Solve the unknowns in 6 steps in the following order: s k, p i, R k, t k x, tk y, ηk. (maximum numerical stability) Linear least squares is used to solve for each parameter. 16

17 Scattered Data Interpolation Once we have an initial set of 3D feature points p i, we use these values to deform the remaining vertices on the face mesh. Smooth interpolation function: gives 3D displacement between original point positions and the new adapted positions for every vertex in the original generic face mesh. Find u j for every unconstrained vertex j. 17

18 Scattered Data Interpolation (2) Let Radial basis functions: Goal: determine coefficients c i and the affine components M and t. Interpolation constraints: Extra constraints: 18

19 Correspondence-based Shape Refinement After warping the generic face model into its new shape, we can still further improve it by specifying additional correspondences. Re-run the scattered data interpolation step (iteratively) to update the vertices for which no correspondences are given. 19

20 Texture mapping Texture colour for each point on the face model. Methods I. View-independent -> Cylindrical texture map II. View-dependent -> Blending with weights 20

21 21

22 View-independent Feathered visibility map Positional certainty Weight map -> binary visibility map 22

23 Texture Value Weighted map Image function projection coordinates 23

24 24

25 View-dependent texture mapping Render model with different input photo s and blend. Viewing direction d View dependent term 25

26 Results view-(in)dependent blending 26

27 Eyes, teeth, ears, and hair Individual texture map Brightness modulating 27

28 Morphing expressions for animation Linear combination (Vetter and Blanz 1998) -> Simple linear interpolation between coordinates of corresponding vertices of each two face meshes. -> Identical topology: natural correspondence between vertices. ->Blended expression rendered using original texture map with weight at each vertex. 28

29 Local Blending Blend specification: Global Regional Painterly interface 29

30 Facial Action Coding System 30

31 User Interface Keyframe animation Expression gallery Timeline 31

32 Results 32

33 Suggested improvements Color correction. Improved registration. Texture relighting. Automatic modeling. Modeling from video. Complex animations Lip-synching. Performance-driven animation. 33

34 Recent work Facial expressions from video (real-time) Real-Time High-Fidelity Facial Performance Capture 2015 Automatic technique for parameter extraction

35 Dynamic 3D Avatar Creation from Hand-held Video Input 80 images for static reconstruction and > 90 sec. of video for dynamic modeling Dynamic blendshape/morphing Visual feedback still needed Albedo map 35

36 Conclusion Generation of realistic face images. Extract information from face images. Wide range of expressions by linear combination. Specialized for a given person 36

37 Future work More realistic. Behaviour-driven (actor/director and motivation of character). 37

38 Discussion Can this technique be used for anything else? (Example: Animal faces, other body parts, plants, buildings, etc ). How well would this work? We had a good remark that animals have less expressions anyway. Also because of the obstruction of hair it would be probably harder using this technique. But perhaps using this techniques the transformation to a different state (from neural to aggressive) can be generated automatically. Additionally this technique might be used the other way around. Much work has been done for architecture using this technique. 38

39 Discussion (2) How can we improve model fitting, specifically the automation of obtaining valid feature points? We did not discuss this in class but we thought of machine learning as a method to improve the building of the fitted face model. If a computer can learn itself to detect the features based on only the input photographs it can take away the user intervention and this can be used for other applications (architecture, maybe plants?) 39

40 Discussion (3) How far can this technology go? (security/moral issues, fraud ) Based on the reactions during the video presentation, this is not convincing enough, but we can imagine this becoming a problem in the future (especially if speech technology also improves). We already saw in presentation 7 the state of the art techniques that brought single pictures to life and the possible adjustments to that face. 40

41 References F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin Synthesizing realistic facial expressions from photographs. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques(siggraph '98). ACM, New York, NY, USA, A. Ichim, P Kadlecek, L Kavan, M Pauly Physics-based Face Modeling and Animation S. Bouaziz, A. Tagliasacchi, M. Pauly Dynamic 2D/3D Registration. Eurographics Tutorial (Proc. of EG'14) C. Wang, F. Shi, S. Xia Realtime 3D Eye Gaze Animation Using a Single RGB Camera. ACM Transactions on Graphics, 35(4) A.E. Ichim, S. Bouaziz, M Pauly Dynamic 3D Avatar Creation from Hand-held Video Input. ACM Transactions on Graphics (ToG) 34.4 (2015): 45 41

42 References Parke, Frederick I. "Computer generated animation of faces." Proceedings of the ACM annual conference-volume 1. ACM, Beier, Thaddeus, and Shawn Neely. "Feature-based image metamorphosis." ACM SIGGRAPH Computer Graphics. Vol. 26. No. 2. ACM, Essa, Irfan, et al. "Modeling, tracking and interactive animation of faces and heads//using input from video." Computer Animation'96. Proceedings. IEEE,

A Morphable Model for the Synthesis of 3D Faces

A Morphable Model for the Synthesis of 3D Faces A Morphable Model for the Synthesis of 3D Faces Marco Nef Volker Blanz, Thomas Vetter SIGGRAPH 99, Los Angeles Presentation overview Motivation Introduction Database Morphable 3D Face Model Matching a

More information

Faces. Face Modeling. Topics in Image-Based Modeling and Rendering CSE291 J00 Lecture 17

Faces. Face Modeling. Topics in Image-Based Modeling and Rendering CSE291 J00 Lecture 17 Face Modeling Topics in Image-Based Modeling and Rendering CSE291 J00 Lecture 17 Faces CS291-J00, Winter 2003 From David Romdhani Kriegman, slides 2003 1 Approaches 2-D Models morphing, indexing, etc.

More information

Facial Image Synthesis 1 Barry-John Theobald and Jeffrey F. Cohn

Facial Image Synthesis 1 Barry-John Theobald and Jeffrey F. Cohn Facial Image Synthesis Page 1 of 5 Facial Image Synthesis 1 Barry-John Theobald and Jeffrey F. Cohn 1 Introduction Facial expression has been central to the

More information

Image Morphing. Application: Movie Special Effects. Application: Registration /Alignment. Image Cross-Dissolve

Image Morphing. Application: Movie Special Effects. Application: Registration /Alignment. Image Cross-Dissolve Image Morphing Application: Movie Special Effects Morphing is turning one image into another (through a seamless transition) First movies with morphing Willow, 1988 Indiana Jones and the Last Crusade,

More information

SYNTHESIS OF 3D FACES

SYNTHESIS OF 3D FACES SYNTHESIS OF 3D FACES R. Enciso, J. Li, D.A. Fidaleo, T-Y Kim, J-Y Noh and U. Neumann Integrated Media Systems Center University of Southern California Los Angeles, CA 90089, U.S.A. Abstract In this paper,

More information

Faces and Image-Based Lighting

Faces and Image-Based Lighting Announcements Faces and Image-Based Lighting Project #3 artifacts voting Final project: Demo on 6/25 (Wednesday) 13:30pm in this room Reports and videos due on 6/26 (Thursday) 11:59pm Digital Visual Effects,

More information

Image-Based Deformation of Objects in Real Scenes

Image-Based Deformation of Objects in Real Scenes Image-Based Deformation of Objects in Real Scenes Han-Vit Chung and In-Kwon Lee Dept. of Computer Science, Yonsei University sharpguy@cs.yonsei.ac.kr, iklee@yonsei.ac.kr Abstract. We present a new method

More information

Animation of 3D surfaces.

Animation of 3D surfaces. Animation of 3D surfaces Motivations When character animation is controlled by skeleton set of hierarchical joints joints oriented by rotations the character shape still needs to be visible: visible =

More information

Computer Animation Visualization. Lecture 5. Facial animation

Computer Animation Visualization. Lecture 5. Facial animation Computer Animation Visualization Lecture 5 Facial animation Taku Komura Facial Animation The face is deformable Need to decide how all the vertices on the surface shall move Manually create them Muscle-based

More information

FACIAL ANIMATION FROM SEVERAL IMAGES

FACIAL ANIMATION FROM SEVERAL IMAGES International Archives of Photogrammetry and Remote Sensing. Vol. XXXII, Part 5. Hakodate 1998 FACIAL ANIMATION FROM SEVERAL IMAGES Yasuhiro MUKAIGAWAt Yuichi NAKAMURA+ Yuichi OHTA+ t Department of Information

More information

Human body animation. Computer Animation. Human Body Animation. Skeletal Animation

Human body animation. Computer Animation. Human Body Animation. Skeletal Animation Computer Animation Aitor Rovira March 2010 Human body animation Based on slides by Marco Gillies Human Body Animation Skeletal Animation Skeletal Animation (FK, IK) Motion Capture Motion Editing (retargeting,

More information

Animation of 3D surfaces

Animation of 3D surfaces Animation of 3D surfaces 2013-14 Motivations When character animation is controlled by skeleton set of hierarchical joints joints oriented by rotations the character shape still needs to be visible: visible

More information

Performance Driven Facial Animation using Blendshape Interpolation

Performance Driven Facial Animation using Blendshape Interpolation Performance Driven Facial Animation using Blendshape Interpolation Erika Chuang Chris Bregler Computer Science Department Stanford University Abstract This paper describes a method of creating facial animation

More information

Image-based Motion-driven Facial Texture

Image-based Motion-driven Facial Texture Image-based Motion-driven Facial Texture Bing Zhang and Hai Tao and Alex Pang Computer Science and Computer Engineering University of California, Santa Cruz 1 INTRODUCTION Facial animation is a fundamental

More information

CSE452 Computer Graphics

CSE452 Computer Graphics CSE452 Computer Graphics Lecture 19: From Morphing To Animation Capturing and Animating Skin Deformation in Human Motion, Park and Hodgins, SIGGRAPH 2006 CSE452 Lecture 19: From Morphing to Animation 1

More information

Facial Expression Analysis for Model-Based Coding of Video Sequences

Facial Expression Analysis for Model-Based Coding of Video Sequences Picture Coding Symposium, pp. 33-38, Berlin, September 1997. Facial Expression Analysis for Model-Based Coding of Video Sequences Peter Eisert and Bernd Girod Telecommunications Institute, University of

More information

Animation COM3404. Richard Everson. School of Engineering, Computer Science and Mathematics University of Exeter

Animation COM3404. Richard Everson. School of Engineering, Computer Science and Mathematics University of Exeter Animation COM3404 Richard Everson School of Engineering, Computer Science and Mathematics University of Exeter R.M.Everson@exeter.ac.uk http://www.secamlocal.ex.ac.uk/studyres/com304 Richard Everson Animation

More information

INTERNATIONAL JOURNAL OF GRAPHICS AND MULTIMEDIA (IJGM)

INTERNATIONAL JOURNAL OF GRAPHICS AND MULTIMEDIA (IJGM) INTERNATIONAL JOURNAL OF GRAPHICS AND MULTIMEDIA (IJGM) International Journal of Graphics and Multimedia (IJGM), ISSN: 0976 6448 (Print) ISSN: 0976 ISSN : 0976 6448 (Print) ISSN : 0976 6456 (Online) Volume

More information

Data-Driven Face Modeling and Animation

Data-Driven Face Modeling and Animation 1. Research Team Data-Driven Face Modeling and Animation Project Leader: Post Doc(s): Graduate Students: Undergraduate Students: Prof. Ulrich Neumann, IMSC and Computer Science John P. Lewis Zhigang Deng,

More information

Face Morphing using 3D-Aware Appearance Optimization

Face Morphing using 3D-Aware Appearance Optimization Face Morphing using 3D-Aware Appearance Optimization Fei Yang 1 Eli Shechtman 2 Jue Wang 2 Lubomir Bourdev 2 Dimitris Metaxas 1 1 Rutgers University 2 Adobe Systems Figure 1: Our system can generate fully

More information

HIGH-RESOLUTION ANIMATION OF FACIAL DYNAMICS

HIGH-RESOLUTION ANIMATION OF FACIAL DYNAMICS HIGH-RESOLUTION ANIMATION OF FACIAL DYNAMICS N. Nadtoka, J.R. Tena, A. Hilton, J. Edge Centre for Vision, Speech and Signal Processing, University of Surrey {N.Nadtoka, J.Tena, A.Hilton}@surrey.ac.uk Keywords:

More information

Topics in Computer Animation

Topics in Computer Animation Topics in Computer Animation Animation Techniques Artist Driven animation The artist draws some frames (keyframing) Usually in 2D The computer generates intermediate frames using interpolation The old

More information

Advanced Graphics and Animation

Advanced Graphics and Animation Advanced Graphics and Animation Character Marco Gillies and Dan Jones Goldsmiths Aims and objectives By the end of the lecture you will be able to describe How 3D characters are animated Skeletal animation

More information

A New Image Based Ligthing Method: Practical Shadow-Based Light Reconstruction

A New Image Based Ligthing Method: Practical Shadow-Based Light Reconstruction A New Image Based Ligthing Method: Practical Shadow-Based Light Reconstruction Jaemin Lee and Ergun Akleman Visualization Sciences Program Texas A&M University Abstract In this paper we present a practical

More information

Interactive 3D Scene Reconstruction from Images

Interactive 3D Scene Reconstruction from Images Interactive 3D Scene Reconstruction from Images Aaron Hertzmann Media Research Laboratory Department of Computer Science New York University 719 Broadway, 12th Floor New York, NY 10003 hertzman@mrl.nyu.edu

More information

Interactive Deformation with Triangles

Interactive Deformation with Triangles Interactive Deformation with Triangles James Dean Palmer and Ergun Akleman Visualization Sciences Program Texas A&M University Jianer Chen Department of Computer Science Texas A&M University Abstract In

More information

Abstract We present a system which automatically generates a 3D face model from a single frontal image of a face. Our system consists of two component

Abstract We present a system which automatically generates a 3D face model from a single frontal image of a face. Our system consists of two component A Fully Automatic System To Model Faces From a Single Image Zicheng Liu Microsoft Research August 2003 Technical Report MSR-TR-2003-55 Microsoft Research Microsoft Corporation One Microsoft Way Redmond,

More information

Physical based Rigging

Physical based Rigging Physical based Rigging Dinghuang Ji Introduction Computer animation has been a popular research topic since 1970s, when the first parametric facial model is proposed[0]. In the recent few years, a lot

More information

3D Active Appearance Model for Aligning Faces in 2D Images

3D Active Appearance Model for Aligning Faces in 2D Images 3D Active Appearance Model for Aligning Faces in 2D Images Chun-Wei Chen and Chieh-Chih Wang Abstract Perceiving human faces is one of the most important functions for human robot interaction. The active

More information

Lecture 7: Image Morphing. Idea #2: Align, then cross-disolve. Dog Averaging. Averaging vectors. Idea #1: Cross-Dissolving / Cross-fading

Lecture 7: Image Morphing. Idea #2: Align, then cross-disolve. Dog Averaging. Averaging vectors. Idea #1: Cross-Dissolving / Cross-fading Lecture 7: Image Morphing Averaging vectors v = p + α (q p) = (1 - α) p + α q where α = q - v p α v (1-α) q p and q can be anything: points on a plane (2D) or in space (3D) Colors in RGB or HSV (3D) Whole

More information

Feature Tracking and Optical Flow

Feature Tracking and Optical Flow Feature Tracking and Optical Flow Prof. D. Stricker Doz. G. Bleser Many slides adapted from James Hays, Derek Hoeim, Lana Lazebnik, Silvio Saverse, who 1 in turn adapted slides from Steve Seitz, Rick Szeliski,

More information

Speech Driven Synthesis of Talking Head Sequences

Speech Driven Synthesis of Talking Head Sequences 3D Image Analysis and Synthesis, pp. 5-56, Erlangen, November 997. Speech Driven Synthesis of Talking Head Sequences Peter Eisert, Subhasis Chaudhuri,andBerndGirod Telecommunications Laboratory, University

More information

Video based Animation Synthesis with the Essential Graph. Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes

Video based Animation Synthesis with the Essential Graph. Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes Goal Given a set of 4D models, how to generate realistic motion from user specified

More information

2D + 3D FACE MORPHING 1 Yen-Cheng Liu ( ) 2 Pei-Hwai Ciou ( ) Chiou-Shann Fuh ( )

2D + 3D FACE MORPHING 1 Yen-Cheng Liu ( ) 2 Pei-Hwai Ciou ( ) Chiou-Shann Fuh ( ) 2D + 3D FACE MORPHING 1 Yen-Cheng Liu ( ) 2 Pei-Hwai Ciou ( ) Chiou-Shann Fuh ( ) 1, 2 Graduate Institute of Electrical Engineering National Taiwan University Taipei, Taiwan r49213@ntu.edu.tw r49218@ntu.edu.tw

More information

Motion Synthesis and Editing. Yisheng Chen

Motion Synthesis and Editing. Yisheng Chen Motion Synthesis and Editing Yisheng Chen Overview Data driven motion synthesis automatically generate motion from a motion capture database, offline or interactive User inputs Large, high-dimensional

More information

SCAPE: Shape Completion and Animation of People

SCAPE: Shape Completion and Animation of People SCAPE: Shape Completion and Animation of People By Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, James Davis From SIGGRAPH 2005 Presentation for CS468 by Emilio Antúnez

More information

Free-Form Deformation and Other Deformation Techniques

Free-Form Deformation and Other Deformation Techniques Free-Form Deformation and Other Deformation Techniques Deformation Deformation Basic Definition Deformation: A transformation/mapping of the positions of every particle in the original object to those

More information

Pose Space Deformation A unified Approach to Shape Interpolation and Skeleton-Driven Deformation

Pose Space Deformation A unified Approach to Shape Interpolation and Skeleton-Driven Deformation Pose Space Deformation A unified Approach to Shape Interpolation and Skeleton-Driven Deformation J.P. Lewis Matt Cordner Nickson Fong Presented by 1 Talk Outline Character Animation Overview Problem Statement

More information

Interactive Shape Metamorphosis

Interactive Shape Metamorphosis Interactive Shape Metamorphosis David T. Chen Andrei State Department of Computer Science University of North Carolina Chapel Hill, NC 27599 David Banks Institute for Computer Applications in Science and

More information

REALISTIC facial expression synthesis has been one of the

REALISTIC facial expression synthesis has been one of the 48 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 12, NO. 1, JANUARY/FEBRUARY 2006 Geometry-Driven Photorealistic Facial Expression Synthesis Qingshan Zhang, Zicheng Liu, Senior Member,

More information

Graphics, Vision, HCI. K.P. Chan Wenping Wang Li-Yi Wei Kenneth Wong Yizhou Yu

Graphics, Vision, HCI. K.P. Chan Wenping Wang Li-Yi Wei Kenneth Wong Yizhou Yu Graphics, Vision, HCI K.P. Chan Wenping Wang Li-Yi Wei Kenneth Wong Yizhou Yu Li-Yi Wei Background Stanford (95-01), NVIDIA (01-05), MSR (05-11) Research Nominal: Graphics, HCI, parallelism Actual: Computing

More information

K A I S T Department of Computer Science

K A I S T Department of Computer Science An Example-based Approach to Text-driven Speech Animation with Emotional Expressions Hyewon Pyun, Wonseok Chae, Yejin Kim, Hyungwoo Kang, and Sung Yong Shin CS/TR-2004-200 July 19, 2004 K A I S T Department

More information

Real-Time Universal Capture Facial Animation with GPU Skin Rendering

Real-Time Universal Capture Facial Animation with GPU Skin Rendering Real-Time Universal Capture Facial Animation with GPU Skin Rendering Meng Yang mengyang@seas.upenn.edu PROJECT ABSTRACT The project implements the real-time skin rendering algorithm presented in [1], and

More information

Realistic Human Head Modeling with Multi-view Hairstyle Reconstruction

Realistic Human Head Modeling with Multi-view Hairstyle Reconstruction Realistic Human Head Modeling with Multi-view Hairstyle Reconstruction Xiaolan Li and Hongbin Zha ational Laboratory on Machine Perception, Peing University 100871, Beiing, P.R.China Email: {lixlan,zha}@cis.pu.edu.cn

More information

CS 231. Deformation simulation (and faces)

CS 231. Deformation simulation (and faces) CS 231 Deformation simulation (and faces) Deformation BODY Simulation Discretization Spring-mass models difficult to model continuum properties Simple & fast to implement and understand Finite Element

More information

NON-RIGID 3D SHAPE RECOVERY USING STEREO FACTORIZATION

NON-RIGID 3D SHAPE RECOVERY USING STEREO FACTORIZATION NON-IGID D SHAPE ECOVEY USING STEEO FACTOIZATION Alessio Del Bue ourdes Agapito Department of Computer Science Queen Mary, University of ondon ondon, E1 NS, UK falessio,lourdesg@dcsqmulacuk ABSTACT In

More information

Reading. Animation principles. Required:

Reading. Animation principles. Required: Reading Required: Animation principles John Lasseter. Principles of traditional animation applied to 3D computer animation. Proceedings of SIGGRAPH (Computer Graphics) 21(4): 35-44, July 1987. Recommended:

More information

Supplementary Material for Synthesizing Normalized Faces from Facial Identity Features

Supplementary Material for Synthesizing Normalized Faces from Facial Identity Features Supplementary Material for Synthesizing Normalized Faces from Facial Identity Features Forrester Cole 1 David Belanger 1,2 Dilip Krishnan 1 Aaron Sarna 1 Inbar Mosseri 1 William T. Freeman 1,3 1 Google,

More information

Fundamentals of Warping and Morphing

Fundamentals of Warping and Morphing Fundamentals of Warping and Morphing Luiz Velho IMPA - Institututo de Matemática Pura e Aplicada Outline Metamorphosis in Nature Conceptual Framework Overview of Warping and Morphing Applications in Computer

More information

Sample Based Texture extraction for Model based coding

Sample Based Texture extraction for Model based coding DEPARTMENT OF APPLIED PHYSICS AND ELECTRONICS UMEÅ UNIVERISTY, SWEDEN DIGITAL MEDIA LAB Sample Based Texture extraction for Model based coding Zhengrong Yao 1 Dept. Applied Physics and Electronics Umeå

More information

Feature Tracking and Optical Flow

Feature Tracking and Optical Flow Feature Tracking and Optical Flow Prof. D. Stricker Doz. G. Bleser Many slides adapted from James Hays, Derek Hoeim, Lana Lazebnik, Silvio Saverse, who in turn adapted slides from Steve Seitz, Rick Szeliski,

More information

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting R. Maier 1,2, K. Kim 1, D. Cremers 2, J. Kautz 1, M. Nießner 2,3 Fusion Ours 1

More information

Stereo pairs from linear morphing

Stereo pairs from linear morphing Proc. of SPIE Vol. 3295, Stereoscopic Displays and Virtual Reality Systems V, ed. M T Bolas, S S Fisher, J O Merritt (Apr 1998) Copyright SPIE Stereo pairs from linear morphing David F. McAllister Multimedia

More information

Announcements: Quiz. Animation, Motion Capture, & Inverse Kinematics. Last Time? Today: How do we Animate? Keyframing. Procedural Animation

Announcements: Quiz. Animation, Motion Capture, & Inverse Kinematics. Last Time? Today: How do we Animate? Keyframing. Procedural Animation Announcements: Quiz Animation, Motion Capture, & Inverse Kinematics On Friday (3/1), in class One 8.5x11 sheet of notes allowed Sample quiz (from a previous year) on website Focus on reading comprehension

More information

CS 231. Deformation simulation (and faces)

CS 231. Deformation simulation (and faces) CS 231 Deformation simulation (and faces) 1 Cloth Simulation deformable surface model Represent cloth model as a triangular or rectangular grid Points of finite mass as vertices Forces or energies of points

More information

Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac Image stitching Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac Image stitching Stitching = alignment + blending geometrical registration

More information

Deformation Transfer for Triangle Meshes

Deformation Transfer for Triangle Meshes Deformation Transfer for Triangle Meshes a Paper (SIGGRAPH 2004) by Robert W. Sumner & Jovan Popovic presented by Roni Oeschger Deformation Transfer Source deformed Target deformed 1 Outline of my presentation

More information

Midterm Examination CS 534: Computational Photography

Midterm Examination CS 534: Computational Photography Midterm Examination CS 534: Computational Photography November 3, 2016 NAME: Problem Score Max Score 1 6 2 8 3 9 4 12 5 4 6 13 7 7 8 6 9 9 10 6 11 14 12 6 Total 100 1 of 8 1. [6] (a) [3] What camera setting(s)

More information

Real-time Expression Cloning using Appearance Models

Real-time Expression Cloning using Appearance Models Real-time Expression Cloning using Appearance Models Barry-John Theobald School of Computing Sciences University of East Anglia Norwich, UK bjt@cmp.uea.ac.uk Iain A. Matthews Robotics Institute Carnegie

More information

Multimedia Technology CHAPTER 4. Video and Animation

Multimedia Technology CHAPTER 4. Video and Animation CHAPTER 4 Video and Animation - Both video and animation give us a sense of motion. They exploit some properties of human eye s ability of viewing pictures. - Motion video is the element of multimedia

More information

3D FACE RECONSTRUCTION BASED ON EPIPOLAR GEOMETRY

3D FACE RECONSTRUCTION BASED ON EPIPOLAR GEOMETRY IJDW Volume 4 Number January-June 202 pp. 45-50 3D FACE RECONSRUCION BASED ON EPIPOLAR GEOMERY aher Khadhraoui, Faouzi Benzarti 2 and Hamid Amiri 3,2,3 Signal, Image Processing and Patterns Recognition

More information

Online Interactive 4D Character Animation

Online Interactive 4D Character Animation Online Interactive 4D Character Animation Marco Volino, Peng Huang and Adrian Hilton Web3D 2015 Outline 4D Performance Capture - 3D Reconstruction, Alignment, Texture Maps Animation - Parametric Motion

More information

High-Fidelity Facial and Speech Animation for VR HMDs

High-Fidelity Facial and Speech Animation for VR HMDs High-Fidelity Facial and Speech Animation for VR HMDs Institute of Computer Graphics and Algorithms Vienna University of Technology Forecast facial recognition with Head-Mounted Display (HMD) two small

More information

A Review of Image- based Rendering Techniques Nisha 1, Vijaya Goel 2 1 Department of computer science, University of Delhi, Delhi, India

A Review of Image- based Rendering Techniques Nisha 1, Vijaya Goel 2 1 Department of computer science, University of Delhi, Delhi, India A Review of Image- based Rendering Techniques Nisha 1, Vijaya Goel 2 1 Department of computer science, University of Delhi, Delhi, India Keshav Mahavidyalaya, University of Delhi, Delhi, India Abstract

More information

3/1/2010. Acceleration Techniques V1.2. Goals. Overview. Based on slides from Celine Loscos (v1.0)

3/1/2010. Acceleration Techniques V1.2. Goals. Overview. Based on slides from Celine Loscos (v1.0) Acceleration Techniques V1.2 Anthony Steed Based on slides from Celine Loscos (v1.0) Goals Although processor can now deal with many polygons (millions), the size of the models for application keeps on

More information

Peripheral drift illusion

Peripheral drift illusion Peripheral drift illusion Does it work on other animals? Computer Vision Motion and Optical Flow Many slides adapted from J. Hays, S. Seitz, R. Szeliski, M. Pollefeys, K. Grauman and others Video A video

More information

Image warping/morphing

Image warping/morphing Image warping/morphing Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser and Alexei Efros Image warping Image formation B A Sampling and quantization What

More information

Welcome to CS 231. Topics in Computer Animation. Victor Zordan EBU II 337

Welcome to CS 231. Topics in Computer Animation. Victor Zordan EBU II 337 Welcome to CS 231 Topics in Computer Animation Victor Zordan EBU II 337 vbz@cs.ucr.edu Overview Administravia Projects History of animation Research Areas About me Mech Engineering(BS) Computer Science

More information

Animation. CS 465 Lecture 22

Animation. CS 465 Lecture 22 Animation CS 465 Lecture 22 Animation Industry production process leading up to animation What animation is How animation works (very generally) Artistic process of animation Further topics in how it works

More information

3DFaceNet: Real-time Dense Face Reconstruction via Synthesizing Photo-realistic Face Images

3DFaceNet: Real-time Dense Face Reconstruction via Synthesizing Photo-realistic Face Images 1 3DFaceNet: Real-time Dense Face Reconstruction via Synthesizing Photo-realistic Face Images Yudong Guo, Juyong Zhang, Jianfei Cai, Boyi Jiang and Jianmin Zheng arxiv:1708.00980v2 [cs.cv] 11 Sep 2017

More information

Nonrigid Surface Modelling. and Fast Recovery. Department of Computer Science and Engineering. Committee: Prof. Leo J. Jia and Prof. K. H.

Nonrigid Surface Modelling. and Fast Recovery. Department of Computer Science and Engineering. Committee: Prof. Leo J. Jia and Prof. K. H. Nonrigid Surface Modelling and Fast Recovery Zhu Jianke Supervisor: Prof. Michael R. Lyu Committee: Prof. Leo J. Jia and Prof. K. H. Wong Department of Computer Science and Engineering May 11, 2007 1 2

More information

Hierarchical Retargetting of Fine Facial Motions

Hierarchical Retargetting of Fine Facial Motions EUROGRAPHICS 2004 / M.-P. Cani and M. Slater (Guest Editors) Volume 23 (2004), Number 3 Hierarchical Retargetting of Fine Facial Motions Kyunggun Na and Moonryul Jung Department of Media Technology, Graduate

More information

calibrated coordinates Linear transformation pixel coordinates

calibrated coordinates Linear transformation pixel coordinates 1 calibrated coordinates Linear transformation pixel coordinates 2 Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Stratified reconstruction Autocalibration with partial

More information

CHAPTER 3 DISPARITY AND DEPTH MAP COMPUTATION

CHAPTER 3 DISPARITY AND DEPTH MAP COMPUTATION CHAPTER 3 DISPARITY AND DEPTH MAP COMPUTATION In this chapter we will discuss the process of disparity computation. It plays an important role in our caricature system because all 3D coordinates of nodes

More information

Face Tracking. Synonyms. Definition. Main Body Text. Amit K. Roy-Chowdhury and Yilei Xu. Facial Motion Estimation

Face Tracking. Synonyms. Definition. Main Body Text. Amit K. Roy-Chowdhury and Yilei Xu. Facial Motion Estimation Face Tracking Amit K. Roy-Chowdhury and Yilei Xu Department of Electrical Engineering, University of California, Riverside, CA 92521, USA {amitrc,yxu}@ee.ucr.edu Synonyms Facial Motion Estimation Definition

More information

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA

TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA TEXTURE OVERLAY ONTO NON-RIGID SURFACE USING COMMODITY DEPTH CAMERA Tomoki Hayashi 1, Francois de Sorbier 1 and Hideo Saito 1 1 Graduate School of Science and Technology, Keio University, 3-14-1 Hiyoshi,

More information

Homework 2 Questions? Animation, Motion Capture, & Inverse Kinematics. Velocity Interpolation. Handing Free Surface with MAC

Homework 2 Questions? Animation, Motion Capture, & Inverse Kinematics. Velocity Interpolation. Handing Free Surface with MAC Homework 2 Questions? Animation, Motion Capture, & Inverse Kinematics Velocity Interpolation Original image from Foster & Metaxas, 1996 In 2D: For each axis, find the 4 closest face velocity samples: Self-intersecting

More information

RENDERING AND ANALYSIS OF FACES USING MULTIPLE IMAGES WITH 3D GEOMETRY. Peter Eisert and Jürgen Rurainsky

RENDERING AND ANALYSIS OF FACES USING MULTIPLE IMAGES WITH 3D GEOMETRY. Peter Eisert and Jürgen Rurainsky RENDERING AND ANALYSIS OF FACES USING MULTIPLE IMAGES WITH 3D GEOMETRY Peter Eisert and Jürgen Rurainsky Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institute Image Processing Department

More information

Consider a partially transparent object that is illuminated with two lights, one visible from each side of the object. Start with a ray from the eye

Consider a partially transparent object that is illuminated with two lights, one visible from each side of the object. Start with a ray from the eye Ray Tracing What was the rendering equation? Motivate & list the terms. Relate the rendering equation to forward ray tracing. Why is forward ray tracing not good for image formation? What is the difference

More information

Muscle Based facial Modeling. Wei Xu

Muscle Based facial Modeling. Wei Xu Muscle Based facial Modeling Wei Xu Facial Modeling Techniques Facial modeling/animation Geometry manipulations Interpolation Parameterizations finite element methods muscle based modeling visual simulation

More information

What is Computer Vision? Introduction. We all make mistakes. Why is this hard? What was happening. What do you see? Intro Computer Vision

What is Computer Vision? Introduction. We all make mistakes. Why is this hard? What was happening. What do you see? Intro Computer Vision What is Computer Vision? Trucco and Verri (Text): Computing properties of the 3-D world from one or more digital images Introduction Introduction to Computer Vision CSE 152 Lecture 1 Sockman and Shapiro:

More information

CS 523: Computer Graphics, Spring Shape Modeling. Skeletal deformation. Andrew Nealen, Rutgers, /12/2011 1

CS 523: Computer Graphics, Spring Shape Modeling. Skeletal deformation. Andrew Nealen, Rutgers, /12/2011 1 CS 523: Computer Graphics, Spring 2011 Shape Modeling Skeletal deformation 4/12/2011 1 Believable character animation Computers games and movies Skeleton: intuitive, low-dimensional subspace Clip courtesy

More information

Black Desert Online. Taking MMO Development to the Next Level. Dongwook Ha Gwanghyeon Go

Black Desert Online. Taking MMO Development to the Next Level. Dongwook Ha Gwanghyeon Go Black Desert Online Taking MMO Development to the Next Level Dongwook Ha (dongwook@pearlabyss.com) Gwanghyeon Go (xdotdt@pearlabyss.com) 2018-03-23 Black Desert Online Challenges Massive data and contents

More information

Image Warping and Morphing

Image Warping and Morphing Image Warping and Morphing OUTLINE: Image Warping Morphing Beier and Neely s Morphing Method Image Warping Point processing and filtering don t move pixels around. Image warping = rearranging the pixels

More information

Vision-based Control of 3D Facial Animation

Vision-based Control of 3D Facial Animation Eurographics/SIGGRAPH Symposium on Computer Animation (2003) D. Breen, M. Lin (Editors) Vision-based Control of 3D Facial Animation Jin-xiang Chai,1 Jing Xiao1 and Jessica Hodgins1 1 The Robotics Institute,

More information

Real-time Talking Head Driven by Voice and its Application to Communication and Entertainment

Real-time Talking Head Driven by Voice and its Application to Communication and Entertainment ISCA Archive Real-time Talking Head Driven by Voice and its Application to Communication and Entertainment Shigeo MORISHIMA Seikei University ABSTRACT Recently computer can make cyberspace to walk through

More information

Warping and Morphing. Ligang Liu Graphics&Geometric Computing Lab USTC

Warping and Morphing. Ligang Liu Graphics&Geometric Computing Lab USTC Warping and Morphing Ligang Liu Graphics&Geometric Computing Lab USTC http://staff.ustc.edu.cn/~lgliu Metamorphosis "transformation of a shape and its visual attributes" Intrinsic in our environment Deformations

More information

PERFORMANCE CAPTURE FROM SPARSE MULTI-VIEW VIDEO

PERFORMANCE CAPTURE FROM SPARSE MULTI-VIEW VIDEO Stefan Krauß, Juliane Hüttl SE, SoSe 2011, HU-Berlin PERFORMANCE CAPTURE FROM SPARSE MULTI-VIEW VIDEO 1 Uses of Motion/Performance Capture movies games, virtual environments biomechanics, sports science,

More information

2D Image Morphing using Pixels based Color Transition Methods

2D Image Morphing using Pixels based Color Transition Methods 2D Image Morphing using Pixels based Color Transition Methods H.B. Kekre Senior Professor, Computer Engineering,MP STME, SVKM S NMIMS University, Mumbai,India Tanuja K. Sarode Asst.Professor, Thadomal

More information

Art Based Rendering of Fur by Instancing Geometry

Art Based Rendering of Fur by Instancing Geometry Art Based Rendering of Fur by Instancing Geometry Abstract Richie Steigerwald In this paper, I describe a non- photorealistic rendering system that uses strokes to render fur and grass in a stylized manner

More information

arxiv: v1 [cs.cv] 29 May 2018

arxiv: v1 [cs.cv] 29 May 2018 HeadOn: Real-time Reenactment of Human Portrait Videos JUSTUS THIES, Technical University of Munich MICHAEL ZOLLHÖFER, Stanford University CHRISTIAN THEOBALT, Max-Planck-Institute for Informatics MARC

More information

Image-Based Rendering

Image-Based Rendering Image-Based Rendering COS 526, Fall 2016 Thomas Funkhouser Acknowledgments: Dan Aliaga, Marc Levoy, Szymon Rusinkiewicz What is Image-Based Rendering? Definition 1: the use of photographic imagery to overcome

More information

Overview. Video. Overview 4/7/2008. Optical flow. Why estimate motion? Motion estimation: Optical flow. Motion Magnification Colorization.

Overview. Video. Overview 4/7/2008. Optical flow. Why estimate motion? Motion estimation: Optical flow. Motion Magnification Colorization. Overview Video Optical flow Motion Magnification Colorization Lecture 9 Optical flow Motion Magnification Colorization Overview Optical flow Combination of slides from Rick Szeliski, Steve Seitz, Alyosha

More information

Animated Talking Head With Personalized 3D Head Model

Animated Talking Head With Personalized 3D Head Model Animated Talking Head With Personalized 3D Head Model L.S.Chen, T.S.Huang - Beckman Institute & CSL University of Illinois, Urbana, IL 61801, USA; lchen@ifp.uiuc.edu Jörn Ostermann, AT&T Labs-Research,

More information

3D Face Modelling Under Unconstrained Pose & Illumination

3D Face Modelling Under Unconstrained Pose & Illumination David Bryan Ottawa-Carleton Institute for Biomedical Engineering Department of Systems and Computer Engineering Carleton University January 12, 2009 Agenda Problem Overview 3D Morphable Model Fitting Model

More information

VIDEO FACE BEAUTIFICATION

VIDEO FACE BEAUTIFICATION VIDEO FACE BEAUTIFICATION Yajie Zhao 1, Xinyu Huang 2, Jizhou Gao 1, Alade Tokuta 2, Cha Zhang 3, Ruigang Yang 1 University of Kentucky 1 North Carolina Central University 2 Microsoft Research 3 Lexington,

More information

AAM Based Facial Feature Tracking with Kinect

AAM Based Facial Feature Tracking with Kinect BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 15, No 3 Sofia 2015 Print ISSN: 1311-9702; Online ISSN: 1314-4081 DOI: 10.1515/cait-2015-0046 AAM Based Facial Feature Tracking

More information

Neue Verfahren der Bildverarbeitung auch zur Erfassung von Schäden in Abwasserkanälen?

Neue Verfahren der Bildverarbeitung auch zur Erfassung von Schäden in Abwasserkanälen? Neue Verfahren der Bildverarbeitung auch zur Erfassung von Schäden in Abwasserkanälen? Fraunhofer HHI 13.07.2017 1 Fraunhofer-Gesellschaft Fraunhofer is Europe s largest organization for applied research.

More information

Graphics for VEs. Ruth Aylett

Graphics for VEs. Ruth Aylett Graphics for VEs Ruth Aylett Overview VE Software Graphics for VEs The graphics pipeline Projections Lighting Shading Runtime VR systems Two major parts: initialisation and update loop. Initialisation

More information

Image-Based Modeling and Rendering

Image-Based Modeling and Rendering Image-Based Modeling and Rendering Richard Szeliski Microsoft Research IPAM Graduate Summer School: Computer Vision July 26, 2013 How far have we come? Light Fields / Lumigraph - 1996 Richard Szeliski

More information