Manifold Preserving Edit Propagation

Transcription

1 Manifold Preserving Edit Propagation SIGGRAPH ASIA 2012 Xiaowu Chen, Dongqing Zou, Qinping Zhao, Ping Tan Kim, Wook

2 Abstract Edit propagation algorithm more robust to color blending maintain the manifold structure using LLE LLE : locally linear embedding Kim, Wook # 2

3 Related Work Local Propagation Colorization Using Optimization, SIGGRAPH 2004, Levin et al. Two neighboring pixels s,r should have similar colors if their intensities are similar Kim, Wook # 3

4 Related Work Local Propagation Minimize difference between color r and weighted average of colors at neighboring pixels Large when intensity(r) is similar to intensity(s) r u r s N(r) w rs U(s) 2 Color of pixel r Color of pixel s Have difficulties in dealing with fragmented regions r : pixel index N(r) : neighbors of pixel r Kim, Wook # 4

5 Related Work Global Propagation AppProp: All-Pairs Appearance-Space Edit Propagation, SIGGRAPH 2008, An and Pellacini Consider relationship between all pixel pairs Kim, Wook # 5

6 Related Work Global Propagation Optimization is guided by two principles: pixels with user-specified edits should retain pixels having similar features are more likely to receive similar amount of edits Z measures affinity between i,j Same->1 Different->0 w j z ij (e i g j ) 2 + λ z ij (e i e j ) 2 i j i j Color of stroke j Color of pixel i Allow for a large propagation range, but cannot provide instant feedback i,j : pixel index e : result edit g : user-specified edit Kim, Wook # 6

7 Related Work Accelerating computation Instant Propagation of Sparse Edits on Images and Videos, Pacific Graphics 2010, Li et al. optimization based method -> interpolation based method Kim, Wook # 7

8 Related Work Accelerating computation Interpolate the user specified edits gi Interpolation Function User stroke Gaussian Function Kim, Wook # 8

9 Introduction Object boundaries have blended color of neighboring objects semi-transparency, motion blur.. These pixels are dissimilar from those on main image objects where the user tends to put edit scribbles causes artifacts in results Kim, Wook # 9

10 Example of this problem Pixel A is dissimilar to the user drawn pixels week affinity So other methods cause artifacts on blending region Contribution! Kim, Wook # 10

11 Solution C real solution A stroke A B, B original image & user input C Maintains the manifold structure If A is a linear combination of B and C, our method requires A to be the same combination of B and C Kim, Wook # 11

12 Locally Linear Embedding(LLE) Nonlinear dimensionality reduction by locally linear embedding, Science, Roweis and Saul LLE projects data from a high dimensional space to low dimensional manifold Each sample can be represented by linear combination of its neighbors Kim, Wook # 12

13 Steps of LLE (1) assign neighbors to each data point Xi (2) compute the weights Wij that best linearly reconstruct Xi from its neighbors (3) compute the lowdimensional embedding vectors vi best reconstructed by wij Kim, Wook # 13

14 Algorithm Find K nearest neighbors for each pixel using ANN library [David M. Mount and Sunil Arya ] Compute a set of weights wij that can best reconstruct xi from these K neighbors N i=1 X i K j=1 w ij X ij 2 constrained least square Feature of pixel i Weighted summation of K neighbors Kim, Wook # 14

15 Algorithm Propagate user editing to the whole image result value stroke value weight of neighbors N 2 E = λ i s z i g i 2 i=1 z i z j N i w ij z j reflecting user stroke maintaining manifold structure Kim, Wook # 15

16 Applications Video Objects Recoloring s = user strokes gi = user specified color zi = final RGB color xi = RGB+(x,y,t) Kim, Wook # 16

17 Experiments Video Objects Recoloring Our method generated more nature results Kim, Wook # 17

18 Applications Video Color Theme Editing Source color theme extraction compute original color theme T {t1,t2,,tm} using [Chang et al. 2003] Theme mapping user select a target color theme R {r1,r2,,rm} checking all possible mapping using Euclidean distance in color mood space m i=1 2 (t i r ki ) Color optimization S = T, gi = rki, zi = final RGB color, xi = RGB+(x,y,t) Kim, Wook # 18

19 Experiments Video Color Theme Editing Kim, Wook # 19

20 Experiments Video Color Theme Editing Optical flow based propagation tend to fail in complex videos variable lighting fast motion motion blur Kim, Wook # 20

21 Applications Grayscale Image Colorization Apply an over-segmentation method each superpixel is considered as a point in the feature space S = superpixels that intersect with the user strokes [Achanta et al. 2010] gi = user specified color, zi = final RGB color xi = texture + SIFT features [Chia et al. 2011] + average pixel intensity + coordinate in superpixel Smooth intermediate result guided image filter [He et al. 2010] Kim, Wook # 21

22 Experiments Grayscale Image Colorization Our method Levin et al. [2004] Kim, Wook # 22

23 Applications Image Matting S = the set of definite foreground and definite background pixels marked by the user gi = 1(foreground), 0(background) zi = the alpha value xi = RGB+(x,y) Kim, Wook # 23

24 Experiments Image Matting Kim, Wook # 24

25 Evaluation For 30M pixels and K=30, our methods takes about 200 seconds and 350M RAM Different number of neighbors (varying K) K= 1, 3, 20, 50 Kim, Wook # 25

26 Limitation Not consider semantic information Indistinct features Kim, Wook # 26

27 Conclusion Novel algorithm based on LLE for edit propagation Preserving manifold structure formed by all pixels in a feature space Various application color editing, gray image colorization and matting Kim, Wook # 27

28 One more thing A Sparse Control Model for Image and Video Editing, SIGGRAPHASIA 2013, Li et al. Automatically balance the rival influence of two strokes Kim, Wook # 28