Image Inpainting. Seunghoon Park Microsoft Research Asia Visual Computing 06/30/2011

Transcription

1 Image Inpainting Seunghoon Park Microsoft Research Asia Visual Computing 06/30/2011

2 Contents Background Previous works Two papers Space-Time Completion of Video (PAMI 07)*1+ PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing (SIGGRAPH 09)*2+ 6/30/2011 Seunghoon Park 2

3 What is inpainting? Hole region Result #1 Result #2 Goal: Fill in hole region in a visually plausible way 6/30/2011 Seunghoon Park 3

4 Previous work Partial Differential Equations (PDE) based (Bertalmio et al.[1]) Limited to small gap 6/30/2011 Seunghoon Park 4

5 Previous work Exempler(or Patch) based(criminisi et al.[4]) Discontinuity in case of complex structures (e.g. curves or T(or X) conjunctions) 6/30/2011 Seunghoon Park 5

6 The first paper Space-Time Completion of Video (PAMI 07)*1+ Input Output Mask samples 6/30/2011 Seunghoon Park 6

7 Problem definition : Input image I H I : Hole region S I \ H : Source region * Fill in H such that the resulting image I will have as much global visual coherence with S 6/30/2011 Seunghoon Park 7

8 Completion as a global optimazation Maximizing a global objective function * I S : Resulting : Source Coherence( image region I * S ) p I * max q S sim sim W p, V W q p, V e q d W p, V q 2 2 p, q W p : pixel point x, y, V : fixed - sized windows centered around p and q q d ( W, V ) W ( x, y ) V p q ( x, y ) p q ( x, y ) 2 6/30/2011 Seunghoon Park 8

9 Coherence maximization Two local conditions 1. All windows W 1 W k containing p appear in the source region S: p p V i S, W i p V i 2. 1 k All those V V agree on the color value c at location p: c V i ( p ) V ( p j ) 6/30/2011 Seunghoon Park 9

10 Iterative Scheme Search i 1. Let k i W p i 1 be all windows such that p W p 2. Find V i S maximizing similarity measure 6/30/2011 Seunghoon Park 11

11 Iterative Scheme Vote i i Let c V be the appropriate colors c i M i M i c i p i p M is i, where sim the p i disttransf highest mode orm i from p, : fixed Mean W i p, V i constant - Shift algorithm 6/30/2011 Seunghoon Park 12

12 Iterative Scheme Update Repeated for every hole pixels Multiple iterations until converged 6/30/2011 Seunghoon Park 13

13 Problems Sensitive to filling order Which pixels should be filled first? Priority(C*D) introduced by Criminisi et al.[4] Confidence Data Only confidence Only data 6/30/2011 Seunghoon Park 14

14 Problems Expensive nearest neighbor search NN search rest Total Time[s] Percentage[%] During 1 iteration Use approximate NN search to speed up 6/30/2011 Seunghoon Park 15

15 Results 6/30/2011 Seunghoon Park 16

16 The second paper PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing (SIGGRAPH 09)*2+ 6/30/2011 Seunghoon Park 17

17 Video 6/30/2011 Seunghoon Park 18

18 Problem definition Correspondence Nearest Neighbor(NN) Field 6/30/2011 Seunghoon Park 19

19 PatchMatch algorithm 1. Initialization Assign random values to the NN-field 6/30/2011 Seunghoon Park 20

20 PatchMatch algorithm 2. Propagation A B f ( x, y ) arg min D current, left, above 6/30/2011 Seunghoon Park 22

21 PatchMatch algorithm 3. Random search Box Box width: Box Box width: width: 1 pixel α 2 w w(image αw width) 6/30/2011 Seunghoon Park 23

22 Results Compare against the most popular method: kd-tree with approximate nearest neighbor matching Time [s] Memory [MB] Megapixels PatchMatch kd-tree PatchMatch kd-tree /30/2011 Seunghoon Park 24

23 Results Application to image reshuffling 6/30/2011 Seunghoon Park 25

24 Results Application to image inpainting 6/30/2011 Seunghoon Park 26

25 Summary Global coherence optimization An unified idea used to complete hole regions in both image and video Propagation and random search Accelerate the nearest neighbor search based on patches 6/30/2011 Seunghoon Park 30

26 Q & A 6/30/2011 Seunghoon Park 31

27 References [1] M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, Image Inpainting, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000, [2] Y. Wexler, E. Shechtman and M. Irani, Space-Time Completion of Video, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29, [3] C. Barnes, E. Shechtman, A. Finkelstein and D. B. Goldman, PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, ACM Transactions on Graphics (Proc. SIGGRAPH), 2009, 28, 24:1-24:11 [4] A. Criminisi, P. Perez and K. Toyama Region Filling and Object Removal by Exemplar-Based Image Inpainting IEEE Transactions on Image Processing, 2004, 13, /30/2011 Seunghoon Park 32

28 References [5] J. Sun, L. Yuan, J. Jia & H.-Y. Shum Image Completion with Structure Propagation ACM Transactions on Graphics (SIGGRAPH), 2005, 24, [6] J. Hays and A. A. Efros Scene Completion Using Millions of Photographs ACM Transactions on Graphics (SIGGRAPH), 2007, 26 [7] C. Barnes, E. Shechtman, D. B. Goldman & A. Finkelstein The Generalized PatchMatch Correspondence Algorithm European Conference on Computer Vision, /30/2011 Seunghoon Park 33