Data-driven Methods: Faces. Portrait of Piotr Gibas Joaquin Rosales Gomez (2003)

Transcription

1 Data-driven Methods: Faces Portrait of Piotr Gibas Joaquin Rosales Gomez (2003) CS194: Image Manipulation & Computational Photography Alexei Efros, UC Berkeley, Fall 2016

2 The Power of Averaging

3 8-hour exposure Atta Kim

4 Image Composites Multiple Individuals Sir Francis Galton Composite [Galton, Composite Portraits, Nature, 1878]

5 Average Images in Art 60 passagers de 2e classe du metro, entre 9h et 11h (1985) Krzysztof Pruszkowski Spherical type gasholders (2004) Idris Khan

6 100 Special Moments by Jason Salavon Why blurry?

7 Slide by Jun-Yan Zhu Object-Centric Averages by Torralba (2001) Manual Annotation and Alignment Average Image

8 Computing Means Two Requirements: Alignment of objects Objects must span a subspace Useful concepts: Subpopulation means Deviations from the mean

9 Images as Vectors n = m n*m

10 Vector Mean: Importance of Alignment n = = m ½ + ½ = mean image n*m n*m

11 How to align faces?

12 Shape Vector = Provides alignment! 43

13 Appearance Vectors vs. Shape Vectors Appearance Vector 200*150 pixels (RGB) Vector of 200*150*3 Dimensions Requires Annotation Provides alignment! Shape Vector Vector of 43*2 Dimensions 43 coordinates (x,y) Slide by Kevin Karsch

14 Average Face 1. Warp to mean shape 2. Average pixels

15 Objects must span a subspace (0,1) (.5,.5) (1,0)

16 Example mean Does not span a subspace

17 Subpopulation means Examples: Male vs. female Happy vs. said Average Kids Happy Males Etc. Average female Average kid Average happy male Average male

18 Average Women of the world

19 Average Men of the world

20 Deviations from the mean - Image X Mean X = X = X - X

21 Deviations from the mean X X = X - X = + = + 1.7

22 Slide by Kevin Karsch Extrapolating faces We can imagine various meaningful directions. Masculine Sad Current face Feminine Happy

23 Manipulating faces How can we make a face look more female/male, young/old, happy/sad, etc.? Current face Sub-mean 1 Sub-mean 2 Slide by Kevin Karsch

24 Manipulating Facial Appearance through Shape and Color Duncan A. Rowland and David I. Perrett St Andrews University IEEE CG&A, September 1995

25 Face Modeling Compute average faces (color and shape) Compute deviations between male and female (vector and color differences)

26 Changing gender Deform shape and/or color of an input face in the direction of more female original shape color both

27 Enhancing gender more same original androgynous more opposite

28 Changing age Face becomes rounder and more textured and grayer original shape color both

29 Back to the Subspace

30 Linear Subspace: convex combinations Any new image X can be obtained as weighted sum of stored basis images. X = m i= 1 a i X i Our old friend, change of basis! What are the new coordinates of X?

31 The Morphable Face Model The actual structure of a face is captured in the shape vector S = (x 1, y 1, x 2,, y n ) T, containing the (x, y) coordinates of the n vertices of a face, and the appearance (texture) vector T = (R 1, G 1, B 1, R 2,, G n, B n ) T, containing the color values of the mean-warped face image. Shape S Appearance T

32 The Morphable face model Again, assuming that we have m such vector pairs in full correspondence, we can form new shapes S model and new appearances T model as: S = m model a i i= 1 S i T = m model b i i= 1 T i If number of basis faces m is large enough to span the face subspace then: Any new face can be represented as a pair of vectors (α 1, α 2,..., α m ) T and (β 1, β 2,..., β m ) T!

33 Issues: 1. How many basis images is enough? 2. Which ones should they be? 3. What if some variations are more important than others? E.g. corners of mouth carry much more information than haircut Need a way to obtain basis images automatically, in order of importance! But what s important?

34 Principal Component Analysis Given a point set finds a basis such that, in an M-dim space, PCA coefficients of the point set in that basis are uncorrelated first r < M basis vectors provide an approximate basis that minimizes the mean-squared-error (MSE) in the approximation (over all bases with dimension r) x 1 2 nd principal component x 1 1 st principal component x 0 x 0

35 PCA via Singular Value Decomposition [u,s,v] = svd(a);

36 EigenFaces First popular use of PCA on images was for modeling and recognition of faces [Kirby and Sirovich, 1990, Turk and Pentland, 1991] Collect a face ensemble Normalize for contrast, scale, & orientation. Remove backgrounds Apply PCA & choose the first N eigen-images that account for most of the variance of the data. mean face lighting variation

37 First 3 Shape Basis Mean appearance

38 Principal Component Analysis Choosing subspace dimension r: look at decay of the eigenvalues as a function of r eigenvalues Larger r means lower expected error in the subspace data approximation 1 r M

39 Using 3D Geometry: Blinz & Vetter,

40 Walking in the Face-graph! Ira Kemelmacher-Shlizerman, Eli Shechtman, Rahul Garg, Steven M. Seitz. "Exploring Photobios." ACM Transactions on Graphics 30(4) (SIGGRAPH), Aug

41 Photobio

42 Photobio

43 Photobio

44 Challenges Non-rigid (facial expressions, age ) Occlusions (hair, glasses ) Arbitrary lighting, pose Different cameras, exposure, focus But: there are many photos!

45 Image registration Estimate 3D pose Face detection Bourdev and Brandt 05 Fiducial points detection Everingham et al. 06 2D registration Template 3D model Kemelmacher, Shechtman, Garg, Seitz, Exploring Photobios, SIGGRAPH 11

46 Image registration Estimate 3D pose Face detection Bourdev and Brandt 05 Fiducial points detection Everingham et al. 06 3D registration Template 3D model Kemelmacher, Shechtman, Garg, Seitz, Exploring Photobios, SIGGRAPH 11

47 3D transformed photos before after

48 Represent the photo collection as a graph Similarity between 2 photos 3D Head Pose similarity Facial Expression similarity Time similarity

49 Represent the photo collection as a graph Similarity between 2 photos 3D Head Pose similarity Facial Expression similarity Time similarity

50 Represent the photo collection as a graph Similarity between 2 photos 3D Head Pose similarity Facial Expression similarity Time similarity

51 Dreambit

52

53 Image-Based Shaving Nguyen et al., 2008

54 The idea Nguyen et al., 2008 Differences??? Beard Layer Model +

55 Processing steps 68 landmarks Nguyen et al., 2008

56 Some results Nguyen et al., 2008

57 Final Projects Two options: Pre-canned Projects: Do two (2) pre-canned projects for a list, or three (3) projects for a group of two people Triangulation Matting and Compositing ( paper) High Dynamic Range Imaging Vertigo shot Fake Miniatures Video Magnification Gradient Domain Editing Tour into the Picture Image Quilting Propose own project Proposal due end of Oct Final report + presentation RRR or finals week