A Novel Technique for Sketch to Photo Synthesis

Transcription

1 A Novel Technique for Sketch to Photo Synthesis Pulak Purkait, Bhabatosh Chanda (a) and Shrikant Kulkarni (b) (a) Indian Statistical Institute, Kolkata (b) National Institute of Technology Karnataka, Surathkal

2 Goal of this work? Texture Transformation No spatial difference Distinct Patches Figure : An Example of Sketch Photo Image Pairs (AR Database)

3 Applications Law Enforcement : Automatic retrieval of photos of suspects from a police mug-shot database, given a sketch artist s rendering from an eye-witness description Film Industry Entertainment Query sketch drawn by artist Image Database

4 Applications Law Enforcement : Automatic retrieval of photos of suspects from a police mug-shot database, given a sketch artist s rendering from an eye-witness description Film Industry Entertainment Multi-touch system Gusture directed system

5 Photo Synthesis Sketch to Photo synthesis techniques Using local texture analysis : Wang, Tang 09 Photo to Sketch synthesis techniques Using a global linear models : Wang 04, 02 Eigen-face methods : Tang 03 Using local texture analysis : Liu & X. Tang 05 Registered Image pairs Global linear models : Input Sketch Projection + +? Synthesized Photo Training Database (Photo-Sketch pair)

6 Photo Synthesis Local Texture Analysis : Patch Matching Algorithm Input Sketch Synthesized Photo Best Match Corresponding Photo patch Comparison MRF Training Database (Photo-Sketch pair) Ref : Q. Liu & X. Tang 05, X. Wang & X. Tang 08 09

7 Drawback of previous models All the models so far are based on Either Learning on Global texture analysis (Tang 2004) Unable to synthesize local information Or Learning on local texture analysis(wang 2009) Can t handle global shape variation Can we learn global shape and Local texture together? Fig : Photo-synthesis using overlapping block matching technique

8 1. We transform all the images (Training + Test) into shape free domain (Image Warping)

9 Input sketch T Shape-free sketch Inv(T) Training Database (Shape-free pair) 2. Learn the shapefree images locally to get shape-free Synthesize photo 3. Then Transform back to it s original Shape Synthesize Photo Shape-free Synthesize Photo

10 Transform images into a fixed shape (Image Warping) Manually Annotated Shape of face Manually Annotated Figure : Annotation points(control points) plotted on photo sketch image pair T s Mean Shape T p

11 Image Warping A piecewise linear transformation is applied separately to each triangular region of the image [1]. Find a Delaunay triangulation of the base control points. Using the three vertices of each triangle, infer an affine mapping from base to input coordinates. Mean shape Figure : Photo and sketch Images after warping

12 Annotation points for test sketch Why manually annotation for Test Sketch is not feasible? Annotation points are in a fixed order Missing a point would blow up Warping algorithm Points should be correctly annotated Time consuming process Alternative solution? Active Shape Model (ASM)

13 Active Shape Model It s a statistical model on shape of objects. Captures the natural variability within a class of shapes of objects in training image. Learn the shape of manually annotated training sketch face Iteratively deform to fit to the face in a new Test sketch on the basis of initial approximation. Initial Approximation 3 rd Iteration 5 th Iteration After convergence

14 Photo Synthesis in Shape-free domain Learning Based Technique : Patch Matching Algorithm (LLE based) Input Sketch Synthesized Photo LLE Nearest k-sketch patch Best Match w 1 w 2 w 3 w k Comparison w 2 w 1 w 3 w k Corresponding sketch patch Corresponding k-photo patches Training Database (Shape-free pair) Ref : Q. Liu & X. Tang 05, X. Wang & X. Tang 09

15 Locally Linear Embedding framework Sketch Patches Test Sketch Patch Photo Patches Target Photo Patch

16 Warping back to it s actual shape ASM Control points of test sketch Train sketch-photo control points MLP Manually Annotated Control points Control points of target photo Shape-free Synthesized Photo Inv(T) After warping back to it s actual shape Synthesized Photo

17 Results : Shape-free Domain After Image Warping Synthesized Photo Original Photos Input Sketches Patch matching in shape-free domain

18 More Results Example 2 : Shape-free photo synthesis from sketch image Results from CUHK Database Results from AR Database

19 Comparison with other models Test Sketch Using patch-based technique Using our Algorithm

20 Conclusion System can synthesize photo correctly for large variation of shape of face. Totally automated system except initial translation of shape of test sketch for ASM. Limitation : Outside of shape may not be correctly reconstructed. Extension of this work : sketch face recognition. Future work : generating 3D face from sketch images

21 Thank You.?