Image stitching. Digital Visual Effects Yung-Yu Chuang. with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

Transcription

1 Image stitching Digital Visual Effects Yung-Yu Chuang with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

2 Image stitching Stitching = alignment + blending geometrical registration photometric registration

3 Applications of image stitching Video stabilization Video summarization Video compression Video matting Panorama creation

4 Video summarization

5 Video compression

6 Object removal input video

7 Object removal remove foreground

8 Object removal estimate background

9 Object removal background estimation

10 Panorama creation

11 Why panorama? Are you getting the whole picture? Compact Camera FOV = 50 35

12 Why panorama? Are you getting the whole picture? Compact Camera FOV = Human FOV =

13 Why panorama? Are you getting the whole picture? Compact Camera FOV = Human FOV = Panoramic Mosaic =

14 Panorama eamples Similar to HDR, it is a topic of computational photography, seeking ways to build a better camera using either hardware or software. Most consumer cameras have a panorama mode Mars: Earth:

15 What can be globally aligned? In image stitching, we seek for a matri to globally warp one image into another. Are any two images of the same scene can be aligned this way? Images captured with the same center of projection A planar scene or far-away scene

16 A pencil of rays contains all views real camera synthetic camera Can generate any synthetic camera view as long as it has the same center of projection!

17 Mosaic as an image reprojection mosaic projection plane The images are reprojected onto a common plane The mosaic is formed on this plane Mosaic is a synthetic wide-angle camera

18 Changing camera center Does it still work? synthetic PP PP1 PP2

19 What cannot The scene with depth variations and the camera has movement

20 Planar scene (or a faraway one) PP1 PP3 PP2 PP3 is a projection plane of both centers of projection, so we are OK! This is how big aerial photographs are made

21 Motion models Parametric models as the assumptions on the relation between two images.

22 2D Motion models

23 Motion models Translation Affine Perspective 3D rotation 2 unknowns 6 unknowns 8 unknowns 3 unknowns

24 A case study: cylindrical panorama What if you want a 360 field of view? mosaic projection cylinder

25 Cylindrical panoramas Steps Reproject each image onto a cylinder Blend Output the resulting mosaic

26 applet plets/projection.html

27 Cylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute pairwise alignments 4. Fi up the end-to-end alignment 5. Blending 6. Crop the result and import into a viewer It is required to do radial distortion correction for better stitching results!

28 Taking pictures Kaidan panoramic tripod head

29 Translation model

30 Where should the synthetic camera be real camera synthetic camera The projection plane of some camera Onto a cylinder

31 Cylindrical projection Adopted from

32 Cylindrical projection

33 Cylindrical projection Adopted from

34 Cylindrical projection unwrapped cylinder y θ f

35 Cylindrical projection y unwrapped cylinder y f θ z

36 Cylindrical projection y unwrapped cylinder y f z s=f gives less distortion

37 Cylindrical reprojection top-down view Focal length the dirty secret Image f = 180 (piels) f = 280 f = 380

38 A simple method for estimating f w p f d Or, you can use other software, such as AutoStich, to help.

39 Input images

40 Cylindrical warping

41 Blending Why blending: paralla, lens distortion, scene motion, eposure difference

42 Blending

43 Blending

44 Blending

45 Gradient-domain stitching

46 Gradient-domain stitching

47 Panorama weaving

48 Assembling the panorama Stitch pairs together, blend, then crop

49 Problem: Drift Error accumulation small errors accumulate over time

50 Problem: Drift ( 1,y 1 ) ( n,y n ) Solution copy of first image add another copy of first image at the end there are a bunch of ways to solve this problem add displacement of (y 1 y n )/(n -1) to each image after the first compute a global warp: y = y + a run a big optimization problem, incorporating this constraint best solution, but more complicated known as bundle adjustment

51 End-to-end alignment and crop

52 Rectangling panoramas video

53 Rectangling panoramas

54 Rectangling panoramas

55 Viewer: panorama eample:

56 Viewer: teture mapped model eample:

57 Cylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute pairwise alignments 4. Fi up the end-to-end alignment 5. Blending 6. Crop the result and import into a viewer

58 Determine pairwise alignment? Feature-based methods: only use feature points to estimate parameters We will study the Recognising panorama paper published in ICCV 2003 Run SIFT (or other feature algorithms) for each image, find feature matches.

59 Determine pairwise alignment p =Mp, where M is a transformation matri, p and p are feature matches It is possible to use more complicated models such as affine or perspective For eample, assume M is a 22 matri ' m11 m12 y' m21 m22 y Find M with the least square error n Mp p' i1 2

60 Determine pairwise alignment Overdetermined system y m m m m y ' ' ' ' y m y m m y m ' ' ' 2 ' 1 ' n n n n n n y y m m m m y y y y y

61 Normal equation Given an overdetermined system A b the normal equation is that which minimizes the sum of the square differences between left and right sides A T A A T b Why?

62 Normal equation 2 ) ( b A E n m nm n m b b a a a a : : : :... : : : : : : nm, n equations, m variables

63 Normal equation n m j j nj i m j j ij m j j j n i m j j nj m j j ij m j j j b a b a b a b b b a a a : : : : : : b A n i i m j j ij b a E ) ( b A

64 Normal equation n i i m j j ij b a E ) ( b A 1 0 E n i i i n i j m j ij i b a a a i n i i m j j ij a b a ) 2( 0 b A A A T T E b A A A T T

65 Normal equation A b 2

66 Normal equation b b b A b A A A b b b A A b A A b A b A b A b A b A b A b A T T T T T T T T T T T T T T T T T T T 2 b A A A T T 2 2 E

67 Determine pairwise alignment p =Mp, where M is a transformation matri, p and p are feature matches For translation model, it is easier. What if the match is false? Avoid impact of outliers. n i i i i i y y m m E 1 2 ' 2 2 ' m E

68 RANSAC RANSAC = Random Sample Consensus An algorithm for robust fitting of models in the presence of many data outliers Compare to robust statistics Given N data points i, assume that majority of them are generated from a model with parameters, try to recover.

69 RANSAC algorithm Run k times: How many times? (1) draw n samples randomly How big? Smaller is better (2) fit parameters with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, c Output with the largest c How to define? Depends on the problem.

70 How to determine k p: probability of real inliers P: probability of success after k trials P 1 (1 p ) n k n samples are all inliers a failure failure after k trials k log(1 P) log(1 p n ) for P=0.99 n p k

71 Eample: line fitting

72 Eample: line fitting n=2

73 Model fitting

74 Measure distances

75 Count inliers c=3

76 Another trial c=3

77 The best model c=15

78 RANSAC for Homography

79 RANSAC for Homography

80 RANSAC for Homography

81 Applications of panorama in VFX Background plates Image-based lighting

82 Troy (image-based lighting)

83 Spiderman 2 (background plate)

84 Reference Richard Szeliski, Image Alignment and Stitching: A Tutorial, Foundations and Trends in Computer Graphics and Computer Vision, 2(1):1-104, December R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and teture-mapped models, SIGGRAPH 1997, pp M. Brown, D. G. Lowe, Recognising Panoramas, ICCV 2003.