Stitching and Blending

Transcription

1 Stitching and Blending Kari Pulli VP Computational Imaging Light

2 First project Build your own (basic) programs panorama HDR (really, exposure fusion) The key components register images so their features align determine overlap blend

3 Scalado Rewind

4 We need to match (align) images

5 Detect feature points in both images

6 Find corresponding pairs

7 Use these pairs to align images

8 Matching with Features Problem 1: Detect the same point independently in both images no chance to match! We need a repeatable detector

9 Matching with Features Problem 2: For each point correctly recognize the corresponding one? We need a reliable and distinctive descriptor

10 Harris Detector: Basic Idea flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions

11 Harris Detector: Mathematics Window-averaged change of intensity for the shift [u,v]: xy, [ ] 2 Euv (,) = wxy (, ) I( x+ uy, + v) I(, xy) Window function Shifted intensity Intensity Window function w(x,y) = or 1 in window, 0 outside Gaussian

12 Harris Detector: Mathematics Expanding E(u,v) in a 2 nd order Taylor series expansion, we have, for small shifts [u,v], a bilinear approximation: u Euv (, ) [ uv,] M v where M is a 2 2 matrix computed from image derivatives: M 2 Ix IxI y = w(, x y) 2 xy, II x y Iy I x = I(x, y) I(x 1, y)

13 Eigenvalues λ 1, λ 2 of M at different locations λ 1 and λ 2 are large

14 Eigenvalues λ 1, λ 2 of M at different locations large λ 1, small λ 2

15 Eigenvalues λ 1, λ 2 of M at different locations small λ 1, small λ 2

16 Harris Detector: Mathematics Measure of corner response: M 2 Ix IxI y = w(, x y) 2 xy, II x y Iy det M trace M = λλ 1 2 = λ + λ 1 2 R= det M k M ( trace ) 2 (k empirical constant, k = )

17 Harris Detector: Mathematics R depends only on eigenvalues of M λ 2 Edge R < 0 Corner R is large for a corner R > 0 R is negative with large magnitude for an edge R is small for a flat region Flat R small Edge R < 0 λ 1

18 Harris Detector: Workflow

19 Harris Detector: Workflow Compute corner response R

20 Harris Detector: Workflow Find points with large corner response: R > threshold

21 Harris Detector: Workflow Take only the points of local maxima of R

22 Harris Detector: Workflow

23 Not invariant to image scale! All points will be classified as edges Corner!

24 FAST Corners Look for a contiguous arc of N pixels all much darker (or brighter) than the central pixel p

25 It actually is fast

26 And repeatable! AUR = Area Under Repeatability curve

27 Point Descriptors We know how to detect points Next question: How to match them?? Point descriptor should be: 1. Invariant 2. Distinctive

28 SIFT Scale Invariant Feature Transform

29 SIFT Scale Invariant Feature Transform Descriptor overview: Determine scale (by maximizing DoG in scale and in space), local orientation as the dominant gradient direction Use this scale and orientation to make all further computations invariant to scale and rotation Compute gradient orientation histograms of several small windows (128 values for each point) Normalize the descriptor to make it invariant to intensity change D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints IJCV 2004

30 Match orientations

31 Determine the local orientation Within the image patch estimate dominant gradient direction collect a histogram of gradient orientations find the peak, rotate the patch so it becomes vertical 0 2π D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints IJCV 2004

32 Determine the scale We define the characteristic scale as the scale that produces peak of Laplacian response characteristic scale T. Lindeberg (1998). Feature detection with automatic scale selection. International Journal of Computer Vision 30 (2): pp

33 Blob detection in 2D Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D g = g(x 1, y) 2g(x, y)+ g(x +1, y) 2 x g g 2 g = σ 2 + norm x 2 y 2 2 2

34 Difference of Gaussians (DoG) Laplacian of Gaussian can be approximated by the difference between two different Gaussians

35 Create a pack of DoGs Fast computation, process scale space an octave at a time

36 Determine the scale Find a local maximum in space and scale D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints IJCV 2004

37 ORB (Oriented FAST and Rotated BRIEF) Use FAST-9 use Harris measure to order them Find orientation BRIEF calculate weighted new center reorient image so that gradients vary vertically Binary Robust Independent Elementary Features choose pixels to compare, result creates 0 or 1 combine to a binary vector, compare using Hamming distance (XOR + pop count) Rotated BRIEF train a good set of pixels to compare P xi(x, y) P I(x, y), P yi(x, y) P I(x, y)

38 rbrief vs. SIFT

39 Aligning images: Translation? left on top right on top Translations are not enough to align the images

40 Which transform to use? Translation Affine Perspective 2 unknowns 6 unknowns 8 unknowns

41 Homography Projective mapping between any two PPs with the same center of projection rectangle maps to almost arbitrary quadrilateral parallel lines do not remain parallel but must preserve straight lines is called a Homography " wx' % $ ' $ wy' ' # $ w &' p To apply a homography H = " h 11 h 12 h 13 %" x% $ ' $ ' $ h 21 h 22 h 23 ' $ y' # $ h 31 h 32 h 33 &' # $ 1& ' compute p = Hp (regular matric multiply) convert p from homogeneous to image coordinates [x, y ] (divide by w) H p PP1 PP2

42 Homography from mapping quads Fundamentals of Texture Mapping and Image Warping Paul Heckbert, M.Sc. thesis, U.C. Berkeley, June 1989, 86 pp.

43 Homography from n point pairs (x,y ; x,y ) Multiply out wx = h 11 x + h 12 y + h 13 wy = h 21 x + h 22 y + h 23 w = h 31 x + h 32 y + h 33 Get rid of w (h 31 x + h 32 y + h 33 )x (h 11 x + h 12 y + h 13 ) = 0 (h 31 x + h 32 y + h 33 )y (h 21 x + h 22 y + h 23 ) = 0 Create a new system Ah = 0 Each point constraint gives two rows of A [-x -y xx yx x ] [ x -y -1 xy yy y ] Solve with singular value decomposition of A = USV T solution is in the nullspace of A the last column of V (= last row of V T ) " wx' % $ ' $ wy' ' # $ w &' p = " h 11 h 12 h 13 %" x% $ ' $ ' $ h 21 h 22 h 23 ' $ y' # $ h 31 h 32 h 33 &' # $ 1& ' H h 11 h 12 h 13 h 21 h = h 22 h 23 h 31 h 32 h 33 p

44 Example common picture plane of mosaic image perspective reprojection Pics: Marc Levoy

45 What to do with outliers? Least squares OK when error has Gaussian distribution But it breaks with outliers data points that are not drawn from the same distribution Mis-matched points are outliers to the Gaussian error distribution severely disturbs the Homography Line fitting using regression is biased by outliers

46 RANSAC RANdom SAmple Consensus 1. Randomly choose a subset of data points to fit model (a sample) 2. Points within some distance threshold t of model are a consensus set Size of consensus set is model s support 3. Repeat for N samples; model with the biggest support is the most robust fit Points within distance t of best model are inliers Fit final model to all inliers Two samples and their supports for line-fitting

47 K. Pulli, M. Tico, Y. Xiong, X. Wang, C-K. Liang, Panoramic Imaging System for Camera Phones, ICCE 2010 Hybrid multi-resolution registration Initial guess I.B. I.B. F.B. Image Based Feature Based F.B. F.B. F.B. Registration parameters

48 Feature-based registration Previous estimate Update search range invalid Feature Detection (Harris corners) Feature Matching (spherical coordinates) RANSAC Validity check Apply the previous registration estimate valid New estimate Convert to spherical coordinates Convert from spherical coordinates Best block crosscorrelation match

49 Progression of multi-resolution registration Actual size Applied to hi-res

50 Image blending Directly averaging the overlapped pixels results in ghosting artifacts Moving objects, errors in registration, parallax, etc. Photo by Chia-Kai Liang

51 Alpha Blending / Feathering

52 Alpha Blending / Feathering I blend = αi left + (1-α)I right =

53 Solution for ghosting: Image labeling Assign one input image to each output pixel Optimal assignment can be found by graph cut [Agarwala et al. 2004]

54 Faster solution with dynamic programming Input texture block B1 B2 B1 B2 B1 B2 Random placement of blocks Neighboring blocks constrained by overlap Minimal error boundary cut

55 Minimal error boundary with DP overlapping blocks vertical boundary 2 _ = overlap error min. error boundary

56 New artifacts Inconsistency between pixels from different input images Different exposure/white balance settings Photometric distortions (e.g., vignetting)

57 Solution: Poisson blending Copy the gradient field from the input image Reconstruct the final image by solving a Poisson equation Combined gradient field

58 Problems with direct cloning P. Pérez, M. Gangnet, A. Blake. Poisson image editing. SIGGRAPH

59 Membrane interpolation

60 Solution: clone gradient, integrate colors

61 Copy the details Seamlessly paste onto Just add a linear function so that the boundary condition is respected Gradients didn t change much, and function is continuous

62 SIGGRAPH 2009

63 Smooth interpolation over a triangulation

64 Alpha blending After labeling Poisson blending

65 Pyramid Blending

66 The Laplacian pyramid Gaussian Pyramid G 2 G n expand Laplacian Pyramid L n = G n L - = 2 G 1 - = L 1 G 0 L 0 - =

67 Laplacian Pyramid: Blending 1. Build Laplacian pyramids LA and LB from images A and B 2. Build a Gaussian pyramid GM from selection mask M 3. Form a combined pyramid LS from LA and LB using nodes of GR as weights: LS = GM * LA + (1-GM) * LB 4. Collapse the LS pyramid to get the final blended image

68 Laplacian level 4 Laplacian level 2 Laplacian level 0 left pyramid right pyramid blended pyramid

69 Pyramid Blending

70 Multi-resolution fusion

71 Simplification: Two-band Blending Brown & Lowe, 2003 Only use two bands: high freq. and low freq. Blends low freq. smoothly Blend high freq. with no smoothing: use binary alpha

72 High frequency (λ < 2 pixels) 2-band Blending Low frequency (λ > 2 pixels)

73 Linear Blending

74 2-band Blending

75 Additional reading Image Alignment and Stitching: A tutorial Richard Szeliski Foundations and Trends in Computer Graphics and Vision Computer Vision: Algorithms and Applications Richard Szeliski Chapters 4, 6, 9