IJCAI Dept. of Information Engineering

Transcription

1 IJCAI 2007 Wei Liu,Xiaoou Tang, and JianzhuangLiu Dept. of Information Engineering TheChinese University of Hong Kong

2 Outline What is sketch-based facial photo hallucination Related Works Our Approach Tensor Model TensorPatches Bayesian Tensor Inference Experimental Results

3 Definition Sketch-based facial photo hallucination: hallucinate (imagine) photorealistic faces from sketches, i.e., the backward transform from sketches to photos. Bidirectional transforms on photo-sketch pairs. (a) Forward transform: synthesizing a sketch image from a photo image; (b) backward transform: hallucinating a photorealistic image from a sketch image.

4 Related Works Face Sketching o H. Chen et al. Example-based facial sketch generation with nonparametric sampling, in Proc. of ICCV, o X. Tang and X. Wang. Face sketch recognition. IEEE Trans. on CSVT, 14(1):50-57, o Q. Liu et al. A nonlinear approach for face sketch synthesis and recognition, in Proc. of CVPR, Ideas Pixel-wise non-parametric sampling Global linear model: PCA Local linear model: LLE

5 Motivations Consider the complexity of image spaces and the conspicuous distinction between photos and sketches. Extract the local relations by explicitly establishing the connection between two feature spaces (sketch and photo) formed by a patch-based tensor model. Formulate a Bayesian approach accounting for the statistical inference from sketches to their corresponding photos in terms of the learned tensor model.

6 Tensor Model We make use of a novel tensor model to exclusively account for the representation of images with two styles: photo-style and sketch-style. style As small image patches can account for high-level statistics of images, we take patches as constitutive elements of the tensor model. Based on a patch corpus with photo- o and sketch-styles, syes, we arrange patches into a high-order tensor which will disclose the latent connection between the two styles.

7 Patch-based Tensor Models Generic Images Model a 3 rd order tensor resulting from the confluence of 3 modes: patch examples, patch styles, and patch features. Tensor transfer can learn the hidden relations between photo patch space and sketch patch space. Face Images Model a 4 th order tensor resulting from the confluence of 4 modes: people, patches, styles, and features. Utilize PCA prior of face images. Bayesian Tensor Inference will incorporate the learned relations in which the tensor model entails into a Bayesian framework.

8 Multilinear Analysis Why use Multilinear Analysis? It unifies multiple factors in a framework. It explicitly models the interaction between these factors. Theoretical Foundation Tensor Algebra I Tensor Multidimensional Array 1 I2 In R A Tensor Product ( A U) ( A) ( U) = I k k ii i j i i ii i i i i j i 12 k 1 k k+ 1 n 12 k 1k k+ 1 n k k i k = 1 High Order Singular Value Decomposition (HOSVD) A = C U U U n n

9 How Multilinear Analysis Works Ensemble Representation Ensemble Tensor: Arrange samples based on factors Base Matrix: Span the space of the self variations D = C U U U U n 1 n 1 n n Core Tensor: Control the interaction ti between factors Mode matrices: Capture the variations of each factor

10 How Multilinear Analysis Works Individual Sample Representation C T T T n 1 n 1 n n x= u u u U Core Tensor: Coordinate the interaction between factors The vector representation of each factor Obtain the coefficients of basis

11 TensorPatches Formulation of Multi-Style l Patch Ensembles D = C U U U U 1 people 2 positions 3 styles 4 features Patch ensemble Core tensor coordinating the interaction between ee factors Encoding peoplerelated information Encoding positionrelated information Encoding stylerelated information The basis spanning the variation subspace of patches The tensor decomposition can be done by HOSVD

12 Bidirectional Mapping/Inferring Forward transform: mapping the Photo Patch Space to the Common Variation Space from which inferring the Sketch Patch Space. y A B x y x Backward transform: mapping the Sketch Patch Space to the Common Variation Space from which inferring the Photo Patch Space. x AB y x y x y Illustration of relations among common variation space, photo patch space and sketch patch space.

13 Hidden Relations From common variation space to photo/sketch image spaces: I = A w, I = A w. x x y y From photo/sketch image spaces to common variation space: w= A A A I = B I w= A A A I = B I T 1 1 ( ) T T T x x x x x x, ( y y) y y y y. The people parameter vector w maintains to be solved for new face images.

14 Bayesian Tensor Inference We fulfill the backward transform Iy Ix through w by taking these quantities as a whole into a global optimization formulation. Our inference approach is still deduced from canonical Bayesian statistics, exploiting PCA to represent the photo feature vector Ix (use the latent variable a) to be hallucinated. The advantage of our approach is to take into account the statistics among a, w, and Iy.

15 Bayesian Tensor Inference Perform PCA on the training photo vectors {Ix} I Ua+ μ p a a Λ a x T 1, ( ) exp{ }. Use the learned relations I = A w, w= B I, we have y y x x 2 2 Iy Ayw w Bx ( Ua+ μ) pi ( y w) exp{ }, pwa ( ) exp{ }. λ λ 1 2 We find the MAP solution a* for hallucinating the optimal Ix* as follows * a = wa, pwa Iy = wa, piy wa pwa = arg max wa, pi ( y wpwapa ) ( ) ( ). arg max (, ) arg max (, ) (, )

16 Architecture of our sketch-based facial photo hallucination approach. (1) Learn the TensorPatches model taking image derivatives as features in the training phase. (2) Obtain the initial result applying ppy the local geometry preserving method. (3) Infer image derivatives of the target photo using the Bayesian Tensor Inference method, given input image derivatives extracted from the test sketch face. (4) Conduct gradient correction to hallucinate the final result.

17 Photo hallucination results for Asian and European faces (a) Input sketch images Photo hallucination results for Asian and European faces. (a) Input sketch images, (b) eigentransform method, (c) local geometry preserving method, (d) our method, (e) groundtruth face photos.

18 Thanks! Jan 2007 If any question on this paper, feel free to contact me via Visiting for more information about my other works.