Image-based Modeling and Rendering: 8. Image Transformation and Panorama

Transcription

1 Image-based Modeling and Rendering: 8. Image Transformation and Panorama I-Chen Lin, Assistant Professor Dept. of CS, National Chiao Tung Univ, Taiwan

2 Outline Image transformation How to represent the environment with a few images? Cylindrical panorama. Automatic image stitching with SIFT. Several figures are from the following reference lists: Image-based Modeling and Rendering, SIGGRAPH 99 course notes. L. McMillan, G. Bishop, Plenoptic Modeling: An Image-Based Rendering System, Proc. SIGGRAPH 95, pp S.E. Chen, QuickTime VR An Image-Based Approach to Virtual Environment Navigation, Proc. SIGGRAPH 95, pp M. Brown and D.G. Lowe, "Recognising panoramas," Proc., ICCV 23, pp M. Brown and D.G.Lowe, Automatic Panoramic Image Stitching using Invariant Features, Intl. J. Computer Vision, vol.74, no.1, pp R. Szeliski, Image Alignment and Stitching: A Tutorial, Foundations and Trends in Computer Graphics and Vision, vol.2, no.1., pp.1-14.

3 Image Warping and Transformation Rearranging pixels of a picture. It s useful for both image processing and for computer graphics (namely, for texture mapping). Finding corresponding points in the source and destination images. This function is called the mapping or transformation.

4 Image Warping (cont.) v source y destination u x Forward mapping : (x, y) = f (u, v) Inverse mapping : (u, v) = f (x, y)

5 Mapping types How to create the mapping systematically? Simple mappings affine mapping projective mapping bilinear mapping

6 Affine Mappings u = ax+by+c v = dx+ey+f A combination of 2-D scale, rotation, and translation transformations. Allows a square to be distorted into any parallelogram. 6 degrees of freedom. Inverse is of same form (is also affine).

7 Projective Mappings u = (ax+by+c)/(gx+hy+i) v = (dx+ey+f)/(gx+hy+i) Linear numerator and denominator. If g=h= then you get affine as a special case. u = uq/q, v = vq/q Allows a square to be distorted into any quadrilateral. 8 degrees of freedom. Inverse is of same form (is also projective)

8 Every point correspondence can have two equations: x(xa 31 + YA 32 + A 33 ) = (XA 11 + YA 12 + A 13 ) y(xa 31 + YA 32 + A 33 ) = (XA 21 + YA 22 + A 23 ) After rearrangement, a pair of quadrilateral: Projective Mappings 1 * Y X A A A A A A A A A w wy wx * A A A A A A A A A y Y y X y Y X x Y x X x Y X y Y y X y Y X x Y x X x Y X y Y y X y Y X x Y x X x Y X y y Y X y Y X x x Y X x Y X

9 Properties of Image Transformation

10 Performing an Image Warp Source scanning: for v = vmin to vmax for u = umin to umax x = x(u,v) y = y(u,v) copy pixel at source[u,v] to dest[x,y] Destination scanning: for y = ymin to ymax for x = xmin to xmax u = u(x,y) v = v(x,y) copy pixel at source[u,v] to dest[x,y] Is there any problem?

11 Plenoptic Modeling: An Image-based Rendering System Proposed by L. McMillan, G. Bishop, Univ. North Carolina at Chapel Hill Proc. SIGGRAPH 95, pp

12 Why do we use IBR? Very realistic scene descriptions can be acquired with a camera. Geometric models for real-world scenes are complicated and difficult to capture.

13 Images as a Collection of Rays An image is a subset of the rays seen from a given point This space of rays occupies two dimensions

14 Images as a Collection of Rays (cont.) How to represent the whole scene with images? The set of rays seen from all view points.

15 The Plenoptic Function A function that provides a complete description of the scene For every viewpoint and view direction, the plenoptic function describes the incident ray To render a scene, pick the view parameters and get the rays from the function. μ= P(θ, φ, λ, Vx, Vy, Vz, t)

16 The Plenoptic Function (cont.) Taking into account only visible light in a static scene. t is constant λ is not a variable Plenoptic function is 5D μ= P(θ, φ, Vx, Vy, Vz)

17 Panoramic images Choosing a warping equation that can be easily adapted.

18 Cylindrical Reference Images Good representation for real-world scenes Maps to a rectangular grid easily Complete coverage of scene in azimuth Good coverage in elevation Spheres might be better, but they re difficult to map to a plane. Distortion Non-uniform sampling

19 Plenoptic Modeling Store reference images as cylindrical projections Sample PF to get reference images Infer flow field from reference images Resample PF to get the desired image

20 Acquiring Cylindrical Projections Using a regular camera with a panning tripod. Camera model and rotations can be inferred from planar reference images. (without camera calibration) Combine planar images and then compute the projection onto a cylinder.

21 Acquiring Cylindrical Projections S.E. Chen, QuickTime VR, Proc. SIGGRAPH 95

22 Acquiring Cylindrical Projections Any two planar perspective projections of a scene which share a common viewpoint are: Separating intrinsic trans. S and extrinsic trans. R: u H x i S 1 R Sx i R y cos sin 1 sin cos

23 Acquiring Cylindrical Projections Turn off auto-focus and other automatic adjustment function of the camera S is fixed. Estimating the parameters by optimization: 1. Initial guess. 2. Estimating R i and f. 3. Estimating S.

24 Estimating R i and f Using a linear approximation to an infinitesimal rotation. f: focal length, (C x, C y ) is the projection of the optical center. (initial guess: at the image center) Using the estimated translations t i to estimate the rotation angles and focal length f.

25 Estimating S S structure matrix can be decomposed into: Iteratively error minimization S z P x f C C P y x 1 x x x x x cos sin sin cos 1 1 cos sin sin cos z z z z z n i i y i z x y x SI R S I Correlation C C error ), ( 1 ),,,,, ( Initial: C x = width/2, C y = Height/2, =, ρ=1, x =, y =

26 Panoramic Images Perspective projection on view plane

27 Plenoptic Function Reconstruction How to estimate the whole plenoptic function from sparse view positions? Manually assign corresponding points for estimation of cylinder center difference and orientation alignment. How to reconstruct plenoptic function? Any problems? Ambiguity Efficiency of search

28 Visibility After performing the warp, we must take visibility into account. Scene elements may occlude other scene elements with the new center of projection. There is a scheme that uses Painter s algorithm along with rules for deciding which pixels get drawn first.

29 Automatic Panoramic Image Stitching using Invariant Features Proposed by M. Brown and D.G. Lowe, In Proc., ICCV 23 and IJCV 27

30 Introduction Are you getting the whole picture? Compact Camera FOV = 5 x 35

31 Introduction Are you getting the whole picture? Compact Camera FOV = 5 x 35 Human FOV = 2 x 135

32 Introduction Are you getting the whole picture? Compact Camera FOV = 5 x 35 Human FOV = 2 x 135 Panoramic Mosaic = 36 x 18

33 Why Recognising Panoramas?

34 Why Recognising Panoramas? 1D Rotations () Ordering matching images

35 Why Recognising Panoramas? 1D Rotations () Ordering matching images

36 Why Recognising Panoramas? 1D Rotations () Ordering matching images

37 Why Recognising Panoramas? 1D Rotations () Ordering matching images 2D Rotations (, f) Ordering matching images

38 Why Recognising Panoramas? 1D Rotations () Ordering matching images 2D Rotations (, f) Ordering matching images

39 Why Recognising Panoramas? 1D Rotations () Ordering matching images 2D Rotations (, f) Ordering matching images

40 Why Recognising Panoramas?

41 Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

42 Overview Feature Matching SIFT Features Nearest Neighbor Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

43 Invariant Features Schmid & Mohr 1997, Lowe 1999, Baumberg 2, Tuytelaars & Van Gool 2, Mikolajczyk & Schmid 21, Brown & Lowe 22, Matas et. al. 22, Schaffalitzky & Zisserman 22

44 SIFT Features Invariant Features Establish invariant frame Maxima/minima of scale-space DOG x, y, s Maximum of distribution of local gradients Form descriptor vector Histogram of smoothed local gradients 128 dimensions SIFT features are Geometrically invariant to similarity transforms, some robustness to affine change Photometrically invariant to affine changes in intensity

45 Nearest Neighbor Matching Find k-nn for each feature k number of overlapping images (in this work, k = 4) Use k-d tree or other searching tech. for speed-up k-d tree recursively bi-partitions data at mean in the dimension of maximum variance Approximate nearest neighbors found in O(nlogn) Figures from:

46 Overview Feature Matching Image Matching RANSAC for Homography Bundle Adjustment Multi-band Blending Results Conclusions

47 Matching with Features For each keypoint, we have multiple candidates. How to correctly recognize the corresponding one??

48 RANSAC for Point Matching If there re multiple candidates, RANdom SAmple Consensus is an approach to solve the ambiguity. Basic idea is to sample a set of point correspondences. Use the sample to estimate a motion model (mapping between 2 views) E.g. translation, Fundamental matrix, etc. See whether the remaining points can provide support for this solution. Copes with a large proportion of outliers.

49 RANSAC Example E.g. fitting a straight line Figure from Prof. J. Rehg, Computer Vision, Georgia Inst. of Tech.

50 Main Idea Select 2 points at random Fit a line Support = number of inliers Line with most inliers wins

51 Why will this work? The best line has the most support More support -> Better fit Figure from Prof. J. Rehg, Computer Vision, Georgia Inst. of Tech.

52 RANSAC Objective: Robust fit of a model to data S Algorithm Randomly select s points Instantiate a model Get consensus set Si If Si >T, terminate and return model Repeat for N trials, return model with max Si

53 RANSAC for Homography

54 RANSAC for Homography

55 RANSAC for Homography

56 Probabilistic model for verification

57 Finding the panoramas (match 6 images at most )

58 Finding the panoramas

59 Finding the panoramas

60 Finding the panoramas

61 Overview Feature Matching Image Matching Bundle Adjustment Error function Multi-band Blending Results Conclusions

62 Error function Sum of squared projection errors n = #images I(i) = set of image matches to image i F(i, j) = set of feature matches between images i,j r ijk = residual of k th feature match between images i,j Robust error function

63 Homography for Rotation Parameterise each camera by rotation and focal length This gives pairwise homographies

64 Bundle Adjustment New images initialised with rotation, focal length of best matching image

65 Bundle Adjustment New images initialised with rotation, focal length of best matching image

66 Gain Compensation Find the optimize gains of g i according to means of overlapping regions between image pair i and j

67 Overview Feature Matching Image Matching Bundle Adjustment Multi-band Blending Results Conclusions

68 Multi-band Blending Burt & Adelson 1983 Blend frequency bands over range l

69 Linear Blending View-dependent weight: Linear blending by the weights: where w(x) varies linearly from 1 at the centre of the image to at the edge Cause blurring of high frequency detail if there are small registration errors

70 Multi-band Blending Mainly use the most frontal pixels. Propagate the weights to neighbors by Gaussian filter. Divide the image into multiple bands. Apply linear blending on each band, and combine the results.

71 Multi-band Blending

72 Multi-band Blending Band 1 scale to σ Band 2 scale σ to 2σ Band 3 lower than 2σ

73 Linear Blending

74 2-band Blending

75 Results

76 Conclusions Fully automatic panoramas A recognition problem Invariant feature based method SIFT features, RANSAC, Bundle Adjustment, Multi-band Blending O(nlogn) Future Work Advanced camera modelling radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash Full 3D case recognising 3D objects/scenes in unordered datasets