Targil 10 : Why Mosaic? Why is this a challenge? Exposure differences Scene illumination Miss-registration Moving objects

Transcription

1 Why Mosaic? Are you getting the whole picture? Compact Camera FOV = 5 x 35 Targil : Panoramas - Stitching and Blending Some slides from Alexei Efros 2 Slide from Brown & Lowe Why Mosaic? Are you getting the whole picture? Compact Camera FOV = 5 x 35 Human FOV = 2 x 35 Why Mosaic? Are you getting the whole picture? Compact Camera FOV = 5 x 35 Human FOV = 2 x 35 Panoramic Mosaic = 36 x 8 3 Slide from Brown & Lowe 4 Slide from Brown & Lowe Stages in building panoramas Stitching the images together Find alignment between overlapping images Choose motion transformation between images (translation, translation + rotation, affine, homography) Choose compositing surface for warping Assign pixels in the panorama to source images Seamlessly blend images Why is this a challenge? Exposure differences Scene illumination Miss-registration Moving objects Goal invisible seams between images Minimal amount of seams artifacts : edges that did not appear in the original images 5 6

2 Approaches Assuming that the images have already been aligned Stitching The Panorama - Simple seam location Simple seam location + smooth the transition between the images (blend) Feathering (Alpha blending) Pyramid blending Gradient domain blending t t+ t+2 Less suitable when there is miss-alignment or moving objects Search for optimal seam Dynamic programming Min cuts Less suitable for thin strips Less suitable when global differences (intensity) are presence 7 8 Stitching The Panorama - Simple seam location Cut & Paste using Center Strips (Voronoi) Image Blending 2 3 Strip taken from 2 Voronoi diagrams What is the problem of this simple approach? Feathering Affect of Window Size + = Encoding transparency I(x,y) = (αr, αg, αb, α) I blend = I left + I right left right 2 2

3 Affect of Window Size Good Window Size Optimal Window: smooth but not ghosted 3 4 Band-pass filtering Laplacian Pyramid Gaussian Pyramid (low-pass images) Enables to Blend low frequencies Over a large spatial range And high frequencies over a short range Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction 5 6 Pyramid Blending Gradient Domain Blending In Pyramid Blending, we decomposed our image into 2 nd derivatives (Laplacian) and a low-res image Let us now look at st derivatives (gradients): No need for low-res image captures everything (up to a constant) Idea: Differentiate Blend Reintegrate Left pyramid blend Right pyramid 7 8 3

4 Gradient Domain blending (D) Gradient Domain Blending (2D) bright Two signals Trickier in 2D: dark Take partial derivatives dx and dy (the gradient field) Fiddle around with them (smooth, blend, feather, etc) Reintegrate But now integral(dx) might not equal integral(dy) Regular blending Blending derivatives Find the most agreeable solution Equivalent to solving Poisson equation 9 2 Comparisons: Levin et al, 24 Perez et al Perez et al What about moving objects? editing Limitations: Colored backgrounds bleed through Images need to be very well aligned

5 Don t blend, CUT! (Search for optimal seam) Where should the cut pass? Moving objects become ghosts So far we only tried to blend between two images. What about finding an optimal seam? Davis, 998 Another application for this approach: Texture synthesis block Segment the mosaic Single source image per segment Avoid artifacts along boundries Dijkstra s algorithm (Efros & Freeman, 2) B B2 B B2 Input texture B B2 Random placement of blocks Neighboring blocks constrained by overlap Minimal error boundary cut Minimal error boundary Dynamic Programming overlapping blocks vertical boundary For each pixel: Choose best neighbor in term of weight + add my weight 2 _ = In the last row choose the best way and follow the track backwards to find optimal path overlap error min. error boundary

6 3 32 Maximum flow problem Minimum cut problem source S a flow F T Max flow problem: Each edge is a pipe Find the largest flow F of water that can be sent from the source to the along the pipes Edge weights give the pipe s capacity source S a cut C T Min cut problem: Find the cheapest way to cut the edges so that the source is completely separated from the Edge weights now represent cutting costs A graph with two terminals A graph with two terminals Max flow/min cut theorem How does min-cut relates to our problem? Define the problem of finding the seam as a min-cut problem : source S T Max Flow = Min Cut: Maximum flow saturates the edges along the minimum cut. Ford and Fulkerson, 962 Problem reduction! Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution A graph with two terminals 35 the selected path will run between pairs of pixels. 36 6

7 What are the nodes of the graph location of pixels p=(x,y). How does min-cut relates to our problem? What are the edges of the graph each pixel (node) has 4 neighbors pixel The weight of the edge (flow capacity) is the color difference between pairs of pixels that the edge connects W(p, p2,a,b) = A(p) B(p) + A(p2) B(p2) p,p2 are two adjacent pixel A(p) and B(p) be the pixel colors at the location p in image A and B, respectively What are the Source and Target of the graph Pixels we want to define explicitly from which image to take What is the meaning of the resulting cut The cut location will determin where do we want to have the seam between the images 37 Edge Weights If the edge connects pixels from Image B that has same (or very close) colors of image A in both sides of the edges: Weight is very small -> we want the cut this edge (x,y) If the edge connects pixels from Image B that has different colors from image A in any size of the cut Weight is bigger (as the difference is bigger)-> we don t want the cut this edge W(p, p2,a,b) = A(p) B(p) + A(p2) B(p2) Img B Img A (x+,y) Img B Img A s(x,y) t(x+,y) 38 Applying Min cuts on images (simple example à la Boykov&Jolly, ICCV ) the selected path will run between pairs of pixels. Stitch from im hard constraint n-links t a cut Stitch from im2 s hard constraint Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms) 39 7