Facial Processing Projects at the Intelligent Systems Lab

Transcription

1 Facial Processing Projects at the Intelligent Systems Lab Qiang Ji Intelligent Systems Laboratory (ISL) Department of Electrical, Computer, and System Eng. Rensselaer Polytechnic Institute Image Formation and Processing group, Beckman Institute, UIUC, Sept. 7 th,

2 Talk outline Overview of research at ISL Face related projects at ISL Summary and future research 2

3 Research at ISL Computer Vision Probabilistic Graphical Models Object tracking, image segmentation, pose estimation, object recognition, performance evaluation Model learning, active and efficient inference, and mixed graphical models HCI, Transportation, Biometrics, Biology, Medicine, Entertainment, etc.. Applications 3

4 Facial Processing Projects at ISL Multi-view face and eye detection and tracking Facial feature tracking Rigid and non-rigid facial motion separation Facial expression recognition Spontaneous facial action units recognition Eye gaze tracking Face Recognition (IEEE TIP, Zou&Ji, in press) Performance Evaluation of FR system (Wang&Ji, PAMI07) Applications 4

5 Multi-view face and eye detection (Wang&Ji, CVPR05) Perform face and eye detection and tracking under varying pose Propose a recursive Nonparametric Discriminant Analysis (NDA) analysis approach for face and eye detection under different poses 5

6 Features for Multi-View Face Detection Pixel intensity: raw data face vector (400*1) face image(20*20) Haar wavelet features: Haar features essentially are geometric block features. Extracting a vector from an image Linear discriminant feature Fisher discriminant analysis (FDA) Some Haar features y = A Nonparametric discriminant analysis (NDA) T x 6

7 Discriminant Features Extraction Fisher discriminant analysis (FDA): find a linear feature T y = A x that best separates different classes. Disadvantage: It is only optimal for Gaussian distributions assuming equal priors of different classes; only one effective feature is extracted since the rank of between-class scatter matrix is 1 for a two-class problem. Nonparametric discriminant analysis(nda): the full rank intra- and extra-class scatter matrices are calculated from the intraclass nearest neighbors x E I and extra-class nearest neighbors x. α α E E T E I S min( x x, x x ) b Ex[ x ( x x )( x x ) ] γ x = E α I α I I T S = E γ ( x x )( x x ) x x + x x w = γ [ ] x x The mapping matrix A can then be obtained by solving the generalized 1 eigen-value problem: ( Sw Sb) A = λ A Disadvantage: time consuming and needs many training samples to accurately locate the NNs. 7

8 Recursive Nonparametric Discriminant Analysis (RNDA) We propose to apply a recursive strategy in NDA: Search nearest neighbors in transformed feature space. Class 1 Extra-class NNs Class 2 Intra-class NNs of class 1 Intra-class NNs of class 2 Bin i Searching nearest neighbors at the transformed feature space Bin j y Recursively update NNs and discriminant feature until the estimated error rate converges. 8

9 Recursive Nonparametric Discriminant Analysis (RNDA) RNDA Algorithm Begin with the Fisher discriminant analysis result, for i = 0,1, 2, Search nearest neighbors at feature space instead of the original x i space. Compute the nonparametric scatter matrices based on the nearest neighbors. Calculate the new discriminant projection A i +1 based on updated NNs. Continue the above procedure until the error rate converges. y = A 0 A T x Advantage: RNDA relaxes the Gaussian assumption in Fisher discriminant analysis and reduces the computational complexity of traditional nonparametric discriminant analysis. 9

10 10 Feature Selection and Combination with AdaBoost Multiple RNDA features are selected and combined with AdaBoost Extract RNDA feature from training data. Feature histograms are used to represent class distribution: Probabilistic classifier is constructed based on class distributions. AdaBoost iteratively updates the weights of training samples, From the updated weights, more features and classifiers are learned. Finally, we combine all the individual classifiers to form a composite classifier: = t h t x x H ) ( ) ( ( x) h g x x t x w e w ) ( ) ( Ω = Ω x A y P x P T x A y T = t T t T t t T x A y P x A y P x P x P x h + Ω = Ω = = Ω Ω = ) ( ) ( log 2 1 ) ( ) ( log 2 1 ) (

11 Training a Multi-View Face Detector left profile face detector frontal face detector nonface right profile face detector More than 10,000 face images are collected from various sources nonface left full profile face detector nonface left half profile face detector right half profile face detector nonface right full profile face detector nonface multi-view faces Many more non-face images are collected from website. The structure of the multiview face detector 11

12 Multi-View Face Detection Results Some multi-view face detection results 12

13 Eye Localization Results Validate eye localization on above 5,000 2D images in FRGC V1.0. Above 99.0% eyes are automatically detected from the detected face Face and eye detection results horizontal (mean) (std) vertical (mean) (std) Euclidean distance Pixel error Normalized error 2.04% 1.96% 1.31% 1.35% 2.67% Eye localization accuracy on FRGC database 13

14 More Eye Detection Results More eye detection results under different environments 14

15 Face and Eye Detection Demos 15

16 Facial Feature Detection and Tracking (Tong&Ji, PRJ07, Zhi&Ji, ICPR06) Twenty-eight facial features around mouth, nose, eyes and eyebrows are selected. Facial feature detection Facial Feature tracking 16

17 Facial Feature Detection A face-guided facial feature detection algorithm is developed: o Face and eye are first detected in a frontal face o Image is normalized, based on which mean face model is scaled and superimeimposed on the face image, producing the initial feature locations o Gabor wavelet jets are used to refine each feature position via the fast phase-based Gabor Wavelet matching. Approximation Refinement Mean Face Mesh 17

18 Facial Feature Tracking Stage one (online information): o o o Kalman Filtering is used to model the dynamics of each facial feature. Given a the model for each feature point, the fast phase-based displacement estimation is used to locate each facial feature automatically. Each facial feature model is updated in each frame dynamically. Issues: (1) It will drift away due to the accumulated error under the significant appearance changes. (2) No effective measurement to understand the tracking failure situation. 18

19 Facial Feature Correction Stage two (offline information or prior information): o o A feature model that is the most similar to the tracking model in the first stage is selected from a training set for each facial feature collected offline. A new position is estimated via the fast phase-based displacement estimation by using the selected patch as a model for each facial feature. Stage three (Correction using appearance information): Probabilistically combine the results from online and offline information: r x S r r = α x online + ( 1 α) x offline online S offline where --- Similarity measurement in the first stage --- Similarity measurement in the second stage α = S S online online + S offline 19

20 Shape Constraints In order to correct those geometrically violated facial features that deviate far from their actual positions, the geometry constraint among them can be imposed. Using Active Shape Model, local and global shape models are constructed to constrain the global face shape and the shape of each facial component for frontal face Face pose is estimated using a subset of tracked rigid points through RANSAC method For non-frontal face, the ASM models are corrected using the estimated face pose The pose-corrected shape constraints are then imposed on the facial features 20

21 Facial Feature Tracking Demo 21

22 Rigid and Non-rigid Face Motion Separation (zhu and Ji, CVPR 06) The motion of the face is the sum of two independent motions: (1) The rigid motion (face pose) (2) The non-rigid motion (facial expression) (a) Rigid motion (b) Non-rigid motion (c) Coupled motion Issue: Both motions are nonlinearly coupled in the face image, and they need be separated to perform facial expression analysis. The goal of this research is to recover both the 3D rigid and the non-rigid facial motion for facial expression analysis. 22

23 3D Facial Expression Model Given a 3D neutral face model X N, it will vary under facial expressions as follows: X = X N + X X is a vector of 3D points and X is the facial deformation under facial expressions with respect to the neural face X N 23

24 Facial Modeling via PCA Facial expression is revealed from the movements of a small set of facial features. o Issue: still too many parameters ( ) o Method: the facial expression is represented by a linear combination of a set of basis facial deformation vectors: Q j, j =1,..., k α j, j =1,..., k X k α j= 1 3 l j Q j are the 3D basis facial deformation vectors, and are the deformation coefficients. These basis facial deformation vectors are learned from a set of training samples via PCA analysis and represented as follows: Q j = j x 1 j y 1 j z 1 L j x l j y l j z l T, j =1,...,k 24

25 25 Facial Expression Model Integration = + = k i z y x z y x i i i i N N N M v u 1 α By integrating the obtained facial expression model with the image projection model M, a projection model can be derived as follows: The model describes how the effects of face pose (M) and facial expression ( ) are combined together to yield the face image (u,v). α i

26 26 Motion Decomposition The recovery of the pose and expression parameters is formulated as the following minimization process: 2 min 1, 1 = = l j i M k i i j i j i j i N j N j N j j j z y x M z y x M v u α α Once the parameters are recovered, face pose information (M) and the facial deformation ( X) can be derived, based on which we can perform facial expression analysis. Subject to M M = 0 ' = M M and

27 Facial Motion Extraction Demo 27

28 Facial Expression Analysis Given the rigid and the non-rigid facial motions, we want to recognize six basic facial expressions, based on Ekman s Facial Action Coding Systems (FACS) 28

29 Facial Action Coding Systems (FACS) FACS is a method for measuring facial behaviors. It defines expressions as one of 46 "Action Units (AU)", each of which describes a contraction or relaxation of one or more facial muscles. FACS defines the relations between action units and facial expressions 29

30 FACS (cont d) FACS is deterministic FACS is mostly static FACS is qualitative with respect to the AU relations FACS is defined with respect to facial muscles. Measurements are often done through image or through image sensors. 30

31 Probabilistic Facial Expression Modeling (zhang and Ji, ICCV03) The six basic facial expressions can be modeled and recognized using the Dynamic Bayesian Networks: o Reformulate Facial Action Coding System (FACS) in a temporal and probabilistic framework to model the facial expression by accounting for (1) spatial dependency (2) dynamics (temporal behavior) (3) uncertainties with facial feature measurement and facial expression. o Associate facial motion (rigid and non-rigid) measurements with FACS Action Units (AUs). 31

32 AU-based Facial Expression Analysis Grouping AUs as primary AUs and auxiliary AUs for a facial expression Primary AUs are the AUs or AU combinations that can be unambiguously classified as belonging to one of the six expressions An auxiliary AUs provides supplementary support to a facial expression. 32

33 AU Measurements Most AUs are measured by the positions and changes of the facial features, i.e., X, the non-rigid facial motion Other AUs are quantified by head movements (the rigid facial motion). Other AUs are measured by facial wrinkles detected via edge analysis Geometrical relationship of facial feature points Example of wrinkle detection 33

34 Facial Expression Modeling with Dynamic Bayesian Networks 34

35 Probabilistic Facial Expression Modeling Using the model, the six prototypic facial expressions can be recognized under arbitrary face orientations via Dynamic Bayesian Networks (Zhang&Ji, PAMI05). 35

36 Spontaneous Facial Action Unit Recognition (Tong&Ji, CVPR06&07, and PAMI07) Facial actions act in a coordinated way to produce meaningful expressions Facial actions dynamically evolve and relate to each other Facial actions are accompanied with head movements 36

37 Existing Work Most work is for posed expressions for frontal faces, therefore not spontaneous expression Most work ignore the spatial and dynamic relationships among Aus 37

38 Causal Relationships Among Facial Components 2D facial shape could be viewed as a stochastic process generated by three hidden causes: head pose, 3D facial shape, and non-rigid facial muscular movements. 3D facial shape characterizes the intrinsic properties of a subject Non-rigid facial muscular movements represented by facial action units cause the 3D shape deformation of the facial surface 3D head pose characterizes the overall head movement 38

39 Spatial Relationships Among Action Units In a spontaneous facial behavior, there are some relations among AUs: Groups of AUs often appear together to show meaningful expression, e.g. AU6 (cheek raiser) +AU12 (lip corner puller) represents happy Some AUs would appear simultaneously, such as AU1 (inner brow raiser) and AU2 (outer brow raiser) Some AU combinations are nearly impossible, e.g. AU23 (lip tighten) and AU27 (mouth stretch) Muscular anatomy of upper face AUs 39

40 Dynamic Relationships among AUs In a spontaneous facial activity, multiple AUs often proceed in sequence to represent different naturalistic facial expressions. There are two types of temporal relationships among AUs: Intra-AU: AU i at time t-1 to AU i at time t represents the self development of each AU Inter-AU: AU i at time t-1 to AU j (i j) at time t represents the dynamic dependencies among AUs For example, in a spontaneous smile, AU12 (lip corner puller) is first activated to express a slight emotion; then, with the increasing of emotion intensity, AU6 (cheek raiser) is activated; and after both reach their apexes simultaneously, AU6 is relaxed, and next AU12 is released. 40

41 Proposed Solution Propose to use the Dynamic Bayesian Network to systematically represent the uncertainties of AU observations, the spatial and dynamic dependencies among AUs to represent relationships among AUs, head poses, and their measurements to recognize facial actions through probabilistic inference 41

42 A DBN for Facial Activity Modeling First layer: the global constraint Second layer: a set of 2D local component shapes third layer: a set of facial action units The DBN model is employed for modeling: the effect of head motion on 2D global shape; the relationship between 2D global shape and local component shapes the relationship between AUs and 2D local shapes relationships among AUs measurement uncertainty (shaded nodes) dynamic evolution of the temporal variables (self arrows) and dynamic dependencies among AUs (links from t-1 to t) 42

43 Facial Activity Recognition through Probabilistic Inference Given the model, the true joint states of head pose and the AUs can be inferred simultaneously given the measurements of the 3D face, head pose, the 2D global shape, the 2D local shapes, and the AUs by finding the most probable explanation (MPE) of the evidence. Based on the conditional independence encoded in the DBN, the inference could be factorized as below: p( pose, AU O, O, O, O, O ) = c { p( pose) p( S ) 1L N S3 P Sg Sl1LM AU1LN 3D S, S, S C1L K M 3D g l1l M p( O S ) p( S S, pose)[ p( S pa( S ))][ p( C pa( C ))] p( O S ) S3 3D g 3D lj lj k k Sg g j k N M N [ pau ( paau ( ))] po ( pose)[ po ( S)][ po ( AU) ]} i i P Slj lj AUi i i j i K 43

44 AU Recognition on Spontaneous Facial Expressions False positive rate Positive recognition rate 44

45 Experimental results under realworld condition 45

46 Eye Gaze Tracking Objectives : Gaze is important for HCI. It often represents a person s desire or intent. Gaze estimation is, however, often ignored by the computer vision community Develop a real time non-intrusive eye gaze tracking system under natural head movement with minimum personal calibration 46

47 Eye model Gaze is the line of sight or visual axis. The intersection of the visual axis with the object is the gaze point or point of regard. 47

48 Gaze Estimation Techniques Gaze can be estimated with different camera and light configurations A single camera and a single light A single camera with two lights Multiple cameras with multiple lights 48

49 Eye Gaze Tracking with one Camera and Two Lights (Zhu&Ji, CVPR06) Principle of eye gaze estimation 1) Detect pupil and estimate its center 2) Detect the cornea reflection of the lights 3) Estimate the cornea center 4) Determine the optical axis 5) Determine the visual axis through a one-time personal calibration 6) Intersect the visual axis with the screen to produce the gaze point 49

50 Captured image 50

51 Compute virtual pupil using one camera Given the cornea center (c), the virtual pupil (p) can be solved using the following 2 equations p= o+ k p ( o v) p c = K K is a constant for each subject, which can be obtained through a 9 point subject calibration. (The assumption: virtual pupil also on optical axis is validated in Appendix D of the attached paper) (5) (6) 51

52 Compute cornea center using one camera If there are only one light, there are 7 unknowns, 6 equations. If there are N lights, there are 4N+4 unknowns, 5N+3 equations. 52

53 Transfer optical axis to visual axis Add Kappa ( horizontal angle α and vertical angle ) β 53

54 Subject calibration There are 4 subject-depended parameters (R, K, α, β ) in our algorithm. They can be obtained through a subject calibration procedure. During calibration, the subject is asked to fixate at 9 points on the screen sequentially. 54

55 System Overview 55

56 Gaze Tracking Demo 56

57 Eye Gaze Demo 2: Eye Mouse 57

58 Applications: Driver Fatigue monitoring 58

59 Emotion Modeling and Recognition Visual Sensor Pressure Sensor Photo Sensor Emotional Mouse Temperature Sensor GSR Sensor 59

60 Biometric: Facial Recognition 60

61 Facial Motion Capture and Animation Facial motion includes eye movement tracking, facial muscle movement tracking, and head movement tracking 61

62 Summary and Future work Summarize the recent face related projects at ISL. Additional details may be found at We also create an image (mostly face) databse at Future work: Focus on developing real time and non-intrusive system for spontaneous facial activity understanding system Combine computer vision with graphical models for robust and consistent visual understanding and interpretation Apply to different applications human computer interaction (e.g. emotion recognition), transportation, security, medical diagnosis, learning, games, polygraph, entertainment, etc.. 62