Image Warping. Image Manipula-on and Computa-onal Photography CS Fall 2011 Robert Carroll.

Transcription

1 Image Warping Image Manipula-on and Computa-onal Photography CS Fall 2011 Robert Carroll [Some slides from K. Padalkar, S. Avidan, A. Shamir, A. Efros, S. Seitz, and Y. Wang]

2 What is a warp?

3 Original What is a warp?

4 What is a warp? Original Warped Warps are spatial transformations: Points map to points, but colors don t change.

5 Applications Panorama Stitching Resizing/Retargeting Wide-Angle Distortion Shape Manipulation [Igarashi et al. 09] Video Stabilization, Stereoscopic Depth Editing, Object Removal,...

6 Applications Panorama Stitching Resizing/Retargeting Wide-Angle Distortion Shape Manipulation [Igarashi et al. 09] Video Stabilization, Stereoscopic Depth Editing, Object Removal,...

7 Image Transformations image filtering: change range of image g(x) = T(f(x)) f T f x image warping: change domain of image x f g(x) = f(t(x)) T f x x

8 Image Transformations image filtering: change range of image g(x) = T(f(x)) f T g image warping: change domain of image f g(x) = f(t(x)) T g

9 Types of Warps Parametric: - Globally defined by small # of parameters rotation aspect perspective cylindrical Mesh: - Smooth (rubber sheet) transformaaon [Wang et al. 09] Discrete: - Each output pixel can be an arbitrary input pixel [Barnes et al. 09]

10 The Image Resizing/RetargeAng Problem

11 The Image Resizing/RetargeAng Problem

12 Image Retarge-ng Objec-ves 1. Change size 2. Preserve the important content and structures 3. Limit ar*facts created

13 Tradi-onal Methods original squeeze crop hybrid letterbox Letterbox wastes pixels Squeeze/Hybrid introduce distortions Cropping removes important parts

14 Many Exis-ng Resizing Methods

15 Seam Carving for Content-Aware Image Resizing Shai Avidan and Ariel Shamir, Proc. SIGGRAPH, 2007

16 Seam Carving Method

17 Seam Carving Method Seam Carving for Content- Aware Image Resizing, S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007

18 Seam Carving Method Seam Carving for Content- Aware Image Resizing, S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007 In Photoshop called content aware scaling

19 Seam Carving Method Seam Carving for Content- Aware Image Resizing, S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007 In Photoshop called content aware scaling Main idea: Remove the least nohceable pixels How? Define an energy func-on that measures how perceptually no-ceable each pixel is

20 Seam Carving Method Seam Carving for Content- Aware Image Resizing, S. Avidan and A. Shamir, Proc. SIGGRAPH, 2007 In Photoshop called content aware scaling Main idea: Remove the least nohceable pixels How? Define an energy func-on that measures how perceptually no-ceable each pixel is Remove the pixels with low energy and avoid removing pixels with high energy How? Define a criterion for picking which pixels to remove

21 Possible Energy Func-ons Edgeness Gradient magnitude Entropy HOG (Histogram of Gradient) Saliency Caveat: No single energy function performs well across all images

22 Possible Energy Func-ons Edgeness Gradient magnitude Entropy HOG (Histogram of Gradient) Saliency Caveat: No single energy function performs well across all images

23 Pixel Removal Criteria Op#mal: remove the k pixels with lowest energy Output image no longer rectangular Input image Energy image Output image

24 Pixel Removal Criteria Remove k pixels with lowest energy in each row No visual coherence between adjacent rows

25 Pixel Removal Criteria Column: Remove whole column with lowest energy Frequently introduces arhfacts

26 Seam Defini-on

27 VerAcal Seam Seam Defini-on is an 8- connected path of pixels in an n x m image from top to boqom, containing one, and only one, pixel in each row of the image:

28 VerAcal Seam Seam Defini-on is an 8- connected path of pixels in an n x m image from top to boqom, containing one, and only one, pixel in each row of the image:

29 VerAcal Seam Seam Defini-on is an 8- connected path of pixels in an n x m image from top to boqom, containing one, and only one, pixel in each row of the image:

30 Seam Energy Energy of a Seam Minimum Energy Seam

31 Pixel Removal Criteria Seam: Remove the verhcal curve of lowest energy

32 Pixel Removal Effec-veness

33 How to Efficiently Compute Best Seam? Use Dynamic Programming to find lowest energy seam in linear Hme 1. Forward Pass (top row to bowom row for finding verhcal seam) Define M(i,j) = cumula-ve energy at (i,j) M(1,j) = e(1,j) M(i,j) = e(i,j) + min(m(i- 1,j- 1), M(i- 1,j), M(i- 1, j+1)) Find minimum value in last row: min j M(n,j) 2. Backward Pass (bowom row to top row) Trace back path from pixel in bozom row with min value to top row

34 Seams Seams over energy image Seams over input image

35 Shrink Image in 1 Dimension

36 Shrink Image in 1 Dimension Change the image from size n m to n mʹ assume mʹ < m

37 Shrink Image in 1 Dimension Change the image from size n m to n mʹ assume mʹ < m Remove m- mʹ = c seams successively

38 Shrink Image in 1 Dimension Change the image from size n m to n mʹ assume mʹ < m Remove m- mʹ = c seams successively

39 Shrink Image in 1 Dimension Change the image from size n m to n mʹ assume mʹ < m Remove m- mʹ = c seams successively Seam Carving

40 Shrink Image in 1 Dimension Change the image from size n m to n mʹ assume mʹ < m Remove m- mʹ = c seams successively Scaling

41 Shrink Image in Both Dimensions: Op-mal Seam Ordering

42 Shrink Image in Both Dimensions: Op-mal Seam Ordering Change the image from size n m to nʹ mʹ assume mʹ < m and nʹ < n

43 Shrink Image in Both Dimensions: Op-mal Seam Ordering Change the image from size n m to nʹ mʹ assume mʹ < m and nʹ < n What is the best order for seam carving? Remove verhcal seams first? Horizontal seams first? Alternate between the two?

44 Op-mal Seam Ordering

45 Op-mal Seam Ordering Solve ophmizahon problem:

46 Op-mal Seam Ordering Solve ophmizahon problem:

47 Op-mal Seam Ordering Solve ophmizahon problem: where k = r+c, r = (m mʹ ), c = (n nʹ ) and α i is a parameter that determines if at step i we remove a horizontal or veracal seam: α {0,1}

48 Op-mal Seam Ordering

49 Op-mal Seam Ordering Transport map Matrix of size n m Each element T(r,c) holds the minimal cost needed to obtain an image of size n r m c

50 Op-mal Seam Ordering Transport map Matrix of size n m Each element T(r,c) holds the minimal cost needed to obtain an image of size n r m c

51 Enlarging Images

52 Enlarging Images Method 1: Compute the opamal veracal (horizontal) seam s in image and duplicate the pixels in s by averaging them with their le` and right neighbors (top and boqom in the horizontal case) O`en will choose the same seam at each iteraaon, producing noaceable stretching arafact

53 Enlarging Images Method 1: Compute the opamal veracal (horizontal) seam s in image and duplicate the pixels in s by averaging them with their le` and right neighbors (top and boqom in the horizontal case) O`en will choose the same seam at each iteraaon, producing noaceable stretching arafact

54 Enlarging Images Method 1: Compute the opamal veracal (horizontal) seam s in image and duplicate the pixels in s by averaging them with their le` and right neighbors (top and boqom in the horizontal case) O`en will choose the same seam at each iteraaon, producing noaceable stretching arafact

55 Enlarging Images Method 2: To enlarge width by k, compute top k veracal seams (for removal) and duplicate each of them

56 Content Amplifica-on

57 Content Amplifica-on Scale the image; this will scale everything, content as well as non- content Shrink the scaled- image using seam carving, which will (hopefully) carve out the non- content part

58 Content Amplifica-on Scale the image; this will scale everything, content as well as non- content Shrink the scaled- image using seam carving, which will (hopefully) carve out the non- content part

59 Content Amplifica-on Scale the image; this will scale everything, content as well as non- content Shrink the scaled- image using seam carving, which will (hopefully) carve out the non- content part

60 Object Removal

61 Object Removal User marks the target object to be removed Force seams to pass through marked pixels To obtain the original image size, use seam inseraon

62 Object Removal User marks the target object to be removed Force seams to pass through marked pixels To obtain the original image size, use seam inseraon

63 Object Removal User marks the target object to be removed Force seams to pass through marked pixels To obtain the original image size, use seam inseraon

64 Object Removal One shoe removed (and image enlarged to original size) input result

65 Object Removal Object marking to prevent unwanted results: mark regions where seams must not pass

66 Object Removal Object marking to prevent unwanted results: mark regions where seams must not pass

67 Object Removal Object marking to prevent unwanted results: mark regions where seams must not pass

68 Object Removal Object marking to prevent unwanted results: mark regions where seams must not pass

69 Mul-- Size Images Input H V

70 Mul-- Size Images Methods menaoned so far are not real- Ame Input H V

71 Mul-- Size Images Methods menaoned so far are not real- Ame We calculate best seam, remove it, calculate the next seam based on new image, etc. Input H V

72 Mul-- Size Images Methods menaoned so far are not real- Ame We calculate best seam, remove it, calculate the next seam based on new image, etc. Compute Index map, V, of size n m that encodes, for each pixel, the index of the seam that removed it, i.e., V(i, j) = t means pixel (i, j) was removed by the t th seam removal iteraaon Input H V

73 Failures Too much content No space for seam to avoid content

74 Optimized Scale-and-Stretch for Image Resizing 1 Yu-Shuen Wang, 2 Chiew-Lan Tai, 3 Olga Sorkine, 1 Tong-Yee Lee 1 National Cheng Kung University, Taiwan 2 Hong Kong University of Science & Technology 3 New York University

75 Scale-and-stretch warp Allow important regions to uniformly scale Find optimal local scaling factors by global optimization Result: preserve the shape of important regions, distort non-important ones importance map

76 Importance map x = saliency map [Itti et al. 98] importance map gradients only importance map

77 The warping mechanism Grid mesh, preserve the shape of the important quads quads with high importance uniform scaling quads with low importance allow non-uniform scaling Optimize the location of mesh vertices, interpolate image

78 The warping mechanism Grid mesh, preserve the shape of the important quads Optimize the location of mesh vertices, interpolate image

79 Deformation Energy Per quad vʹ i v i f vʹ j v j s f uniform scaling

80 Deformation Energy Total deformation energy importance weights Quadratic energy in vʹ, can minimize by solving a linear system of equations

81 Constraints Corner vertices Horizontal/vertical sliding

82 Deformation energy Problem: grid lines bend a lot; warp not so smooth

83 Deformation energy Problem: grid lines bend a lot; warp not so smooth

84 Grid line bending energy Attempt to keep original edge orientations but allow length scaling note the nonlinear factor f v i vʹ i v j vʹ j

85 Energy minimization Total energy is nonlinear in vʹ Iterative minimization with tricks Keep s f and l ij as additional variables Do alternating minimization steps Fix s f and l ij and optimize vʹ Compute new s f and l ij Sparse direct solver for the linear system and reuse the matrix factorization to gain speed.

86 Grid line bending energy With the bending term added

87 Results

88 Results original SC indirect SC Mesh can move in 2D, with seam carving pixels only move in 1D Scale-and-Stretch

89 Results original SC indirect SC Scale-and-Stretch

90 Results original SC indirect SC Scale-and-Stretch

91 Optimizing Content-Preserving Projections for Wide-Angle Images Robert Carroll Maneesh Agrawala University of California, Berkeley Aseem Agarwala Adobe Systems, Inc.

92 Rectilinear Lens Keith Cooper

93 Fisheye Flickr user kirainet

94 Cylindrical Panorama Flickr user Seb Przd

95

96 Viewing Sphere

97 Viewing Sphere

98 Viewing Sphere

99 Viewing Sphere

100 Courtesy of Flickr user brokendrum70 Cartography

101

102 Linear Perspective Projection

103 Linear Perspective Projection

104 Stereographic Projection

105 Stereographic Projection

106 Cylindrical Projection

107 Mercator Projection

108 Optimized Projection

109 Optimized Projection

110 Goal Given a wide-angle image, produce a projection that preserves straight lines in the scene and the shapes of objects

111

112 Our Approach Mesh the viewing sphere Define mapping constraints Optimize weighted energy function

113 Viewing Sphere Mesh

114 Viewing Sphere Mesh

115 Viewing Sphere Mesh

116 Viewing Sphere Mesh

117 Three Properties 1. Conformality 2. Straight Lines 3. Smoothness of mapping

118 Conformality h k

119 Conformality h k

120 Lines

121 Lines

122 Lines Should be zero

123 Lines Should be zero

124 Lines Should be zero

125 Smoothness

126 Smoothness

127 Smoothness

128 Smoothness

129 What s left? Cannot satisfy all constraints exactly Define weighted least-squares energy terms E total = w 1 E conformality + w 2 E lines + w 3 E smoothness Iterative non-linear optimization

130 Lines Should be zero

131 Lines Should be zero

132 Lines Should be zero

133 Lines fixed n

134 Lines fixed n

135 Lines (1 ) Should be zero

136 Lines fixed

137 Lines fixed

138 Lines fixed n

139 Lines fixed n

140 Lines fixed

141 Lines fixed

142 Implementation Details Converges in 8 iterations Up to 160,000 vertices 15s - 1m, depending on field of view

143 Results

144 Input/Output

145 Input/Output

146 Perspective Mercator Stereographic Our Result

147

148 Perspective Mercator Stereographic Our Result

149

150 Perspective Mercator Stereographic Our Result

151 Image Courtesy Flickr user Aldo

152 Perspective Mercator Stereographic Our Result

153 Image Courtesy Jeff Chien

154 Perspective Mercator Stereographic Our Result

155 Input ( Flickr user Mike Schinkel) Zorin Our Result

156 Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView Mercator Our Result

157 Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView Mercator Our Result

158 Zelnik-Manor et al. MultiPlane Zelnik-Manor et al. MultiView Mercator Our Result

159 Must be cropped Limitations

160 Limitations Some stretching near poles

161 Limitations Some stretching near poles Geodesic Grid

162 Limitations Lines that stretch across the full field of view Flickr user Editor B

163 Thanks!