CSCI 1290: Comp Photo

Transcription

1 CSCI 1290: Comp Photo Fall Brown University James Tompkin Many slides thanks to James Hays old CS 129 course, along with all of its acknowledgements.

2 Smartphone news Qualcomm Snapdragon 675 just announced Triple camera config on mid-tier chipset -> moar cameras! Support for features such as: Telephoto (long focal length camera) Wide angle (short focal length camera) Super-wide angle (?? very short focal length camera??) Enhanced portrait mode (Bokeh) 3D face unlock Epic selfies (???) Limitless slo-mo (limitless in recording time, not in anything else) AI: scene/object detection, image style transfer, portrait relighting

3

4

5

6 Today: pixels, patched, & graphs

7 Project 4 - Minimal error boundaries overlapping blocks vertical boundary 2 _ = overlap error min. error boundary

8 Simple Media Retargeting Operators Letterboxing Scaling?? Michael Rubinstein MIT CSAIL

9 Content-aware Retargeting Operators Important content Contentaware Contentoblivious Michael Rubinstein MIT CSAIL

10 Content-aware Retargeting Scale Crop Content-aware Input less-important content Michael Rubinstein MIT CSAIL

11 Image Retargeting Problem statement: Input Image I of n m pixels, and new image size n m Output Image I of size n m which will be good representative of the original image I. To date, no agreed definition, or measure, as to what a good representative is in this context! Michael Rubinstein MIT CSAIL

12 Image/Video Retargeting In large, we would expect: 1. Adhere to the geometric constraints (display/aspect ratio) 2. Preserve the important content and structures 3. Limit artifacts 4. Perhaps a new representation that will support different sizes? Very Ill-posed! How do we define important? Is there a universal ground truth? Would different viewers think the same about a retargeted image? What about artistic impression in the original content? Michael Rubinstein MIT CSAIL

13 Importance (Saliency) Measures A function S: p [0,1] Wang et al More sophisticated: attention models, eye tracking (gazing studies), face detectors, Michael Rubinstein MIT CSAIL Judd et al. ICCV09 Learning to predict where people look

14 Previous Retargeting Approaches Optimal Cropping Window For videos: Pan and scan Still done manually in the movie industry Michael Rubinstein MIT CSAIL Liu and Gleicher, Video Retargeting: Automating Pan and Scan (2006)

15 Cropping Michael Rubinstein MIT CSAIL

16 Seam Carving Assume m n m n, n < n (summarization) Basic Idea: remove unimportant pixels from the image Unimportant = pixels with less energy Intuition for gradient-based energy: Preserve strong contours Human vision more sensitive to edges so try remove content from smoother areas Simple, enough for producing some nice results See their paper for more measures they have used Michael Rubinstein MIT CSAIL

17 Pixel Removal Michael Rubinstein MIT CSAIL Optimal Least-energy pixels (per row) Least-energy columns

18 A Seam A connected path of pixels from top to bottom (or left to right). Exactly one in each row Michael Rubinstein MIT CSAIL

19 Finding the Seam? Michael Rubinstein MIT CSAIL

20 The Optimal Seam E( I) = I + I x y s * = arg min S E( s) Michael Rubinstein MIT CSAIL

21 The recursion relation The Optimal Seam ( M( i 1, j 1), M( i 1, j), M( i 1, 1) ) M( i, j) = E( i, j) + min j + Can be solved efficiently using dynamic programming in O( s n m) (s=3 in the original algorithm) Michael Rubinstein MIT CSAIL

22 Dynamic Programming Invariant property: M(i,j) = minimal cost of a seam going through (i,j) (satisfying the seam properties) Michael Rubinstein MIT CSAIL

23 Dynamic Programming ( M( i 1, j 1), M( i 1, j), M( i 1, 1) ) M( i, j) = E( i, j) + min j Michael Rubinstein MIT CSAIL

24 Dynamic Programming ( M( i 1, j 1), M( i 1, j), M( i 1, 1) ) M( i, j) = E( i, j) + min j Michael Rubinstein MIT CSAIL

25 Dynamic Programming ( M( i 1, j 1), M( i 1, j), M( i 1, 1) ) M( i, j) = E( i, j) + min j Michael Rubinstein MIT CSAIL

26 Searching for Minimum Backtrack (can store choices along the path, but do not have to) Michael Rubinstein MIT CSAIL

27 Backtracking the Seam Michael Rubinstein MIT CSAIL

28 Backtracking the Seam Michael Rubinstein MIT CSAIL

29 Backtracking the Seam Michael Rubinstein MIT CSAIL

30 H & V Cost Maps High cost Low cost Michael Rubinstein MIT CSAIL Horizontal Cost Vertical Cost

31 Michael Rubinstein MIT CSAIL Seam Carving

32 The Seam-Carving Algorithm SEAM-CARVING(im,n ) // size(im) = mxn 1. Do (n-n ) times 2.1. E Compute energy map on im 2.2. s Find optimal seam in E 2.3. im Remove s from im 2. Return im For vertical resize: transpose the image Running time: 2.1 O(mn) 2.2 O(mn) 2.3 O(mn) O(dmn) d=n-n Michael Rubinstein MIT CSAIL

33 Changing Aspect Ratio Michael Rubinstein MIT CSAIL

34 Changing Aspect Ratio Seam Carving Original Scaling Michael Rubinstein MIT CSAIL

35 Changing Aspect ratio Michael Rubinstein MIT CSAIL Cropping Seams Scaling

36 Changing Aspect Ratio Scaling Michael Rubinstein MIT CSAIL

37 Changing Aspect Ratio Scaling Michael Rubinstein MIT CSAIL

38 Seam Carving in the Gradient Domain Michael Rubinstein MIT CSAIL

39 Image Expansion (Synthesis) Michael Rubinstein MIT CSAIL

40 Image Expansion take 2 Michael Rubinstein MIT CSAIL Scaling

41 Enlarged or Reduced? Michael Rubinstein MIT CSAIL

42 Combined Insert and Remove Insert & remove seams Scaling Michael Rubinstein MIT CSAIL

43 Multi-Size Images We can create a new representation of an image that will allow adapting it to different sizes! 1. Precompute all seams once 2. Realtime resizing / transmit with content First to be removed Last to be removed Michael Rubinstein MIT CSAIL

44 Retargeting Video Michael Rubinstein MIT CSAIL

45 Auxiliary Energy Recall our seam equation ( M( i 1, j 1), M( i 1, j), M( i 1, 1) ) M( i, j) = E( i, j) + min j + Michael Rubinstein MIT CSAIL

46 Michael Rubinstein MIT CSAIL With face detector

47 With User Constraints Michael Rubinstein MIT CSAIL

48 Object Removal Michael Rubinstein MIT CSAIL

49 Object Removal Michael Rubinstein MIT CSAIL

50 Limitations Michael Rubinstein MIT CSAIL

51 Preserved Energy - Revisited Average Pixel Energy Image Reduction Michael Rubinstein MIT CSAIL crop column seam pixel optimal

52 Inserted Energy Michael Rubinstein MIT CSAIL

53 Minimize Inserted Energy Instead of removing the seam of least energy, remove the seam that inserts the least energy to the image! Michael Rubinstein MIT CSAIL

54 Tracking Inserted Energy p i-1,j-1 p i-1,j p i-1,j+1 p i,j-1 p i,j p i,j+1 Three possibilities when removing pixel P i,j Michael Rubinstein MIT CSAIL

55 Pixel P i,j : Left Seam p i-1,j-1 p i-1,j p i-1,j+1 p i,j-1 p i,j p i,j+1 Michael Rubinstein MIT CSAIL

56 Pixel P i,j : Right Seam p i-1,j-1 p i-1,j p i-1,j+1 p i,j-1 p i,j p i,j+1 Michael Rubinstein MIT CSAIL

57 Pixel P i,j : Vertical Seam p i-1,j-1 p i-1,j p i-1,j+1 p i,j-1 p i,j p i,j+1 Michael Rubinstein MIT CSAIL

58 Old Backward Energy E Michael Rubinstein MIT CSAIL

59 New Forward Looking Energy E Michael Rubinstein MIT CSAIL

60 Adding Pixel Energy Michael Rubinstein MIT CSAIL

61 Results Backward Input Backward Forward Input Forward Michael Rubinstein MIT CSAIL

62 Results Michael Rubinstein MIT CSAIL

63 Backward vs. Forward Michael Rubinstein MIT CSAIL Backward Forward

64 Results Michael Rubinstein MIT CSAIL

65 Discrete vs. Continuous [Avidan and Shamir 2007] [Wang et al 2008] Continuous Discrete Michael Rubinstein MIT CSAIL

66 From Images to Videos In general, video processing is a much (much!) harder problem 1. Cardinality Suppose 1min of video x 30 fps = 1800 frames Say your algorithm processes an image in 1 minute 30 hours!! 2. Dimensionality/algorithmic Temporal coherency: human visual system is highly sensitive to motion! Michael Rubinstein MIT CSAIL

67 Seam-Carving Video? Naive frame by frame independently Time Michael Rubinstein MIT CSAIL

68 Frame-by-frame Seam-Carving *Representative seams Michael Rubinstein MIT CSAIL

69 From 2D to 3D 1D paths in images 2D manifolds in video cubes Michael Rubinstein MIT CSAIL

70 Challenges Dynamic Programming no longer applicable Reduction to min-cut graph problem Cut must fulfill seam constraints 1. Monotonic (cut each row exactly once) 2. Connected Cut should be a function! Michael Rubinstein MIT CSAIL

71 Graph Construction Not monotonic Michael Rubinstein MIT CSAIL

72 Graph Construction Monotonic, not connected Michael Rubinstein MIT CSAIL

73 Graph Construction Forward Backward Energy Michael Rubinstein MIT CSAIL

74 Multiresolution Graph Cut Lombaert et al. [2005] Michael Rubinstein MIT CSAIL

75 Video Retargeting Video Michael Rubinstein MIT CSAIL

76 Does retargeting actually work? CR is cropping Michael Rubinstein, Diego Gutierrez, Olga Sorkine, Ariel Shamir A Comparative Study of Image Retargeting ACM Transactions on Graphics, Volume 29, Number 5 (Proc. SIGGRAPH Asia) 2010 Michael Rubinstein MIT CSAIL

77 Texture Synthesis Goal of Texture Synthesis: create new samples of a given texture Many applications: virtual environments, holefilling, texturing surfaces

78 The Challenge Need to model the whole spectrum: from repeated to stochastic texture

79 Efros & Leung Algorithm non-parametric sampling p Synthesizing a pixel Input image Assuming Markov property, compute P(p N(p)) Building explicit probability tables infeasible Instead, we search the input image for all similar neighborhoods that s our pdf for p To sample from this pdf, just pick one match at random

80 input Neighborhood Window

81 Varying Window Size Increasing window size

82 Synthesis Results french canvas rafia weave

83 More Results white bread brick wall

84 Homage to Shannon

85 Summary The Efros & Leung algorithm Very simple Surprisingly good results but very slow

86 Image Quilting [Efros & Freeman] p B Synthesizing a block non-parametric sampling Input image Observation: neighbor pixels are highly correlated Idea: unit of synthesis = block Exactly the same but now we want P(B N(B)) Much faster: synthesize all pixels in a block at once Not the same as multi-scale!

87 block Input texture B1 B2 B1 B2 B1 B2 Random placement of blocks Neighboring blocks constrained by overlap Minimal error boundary cut

88 Minimal error boundary overlapping blocks vertical boundary 2 _ = overlap error min. error boundary

89 Our Philosophy The Corrupt Professor s Algorithm : Plagiarize as much of the source image as you can Then try to cover up the evidence Rationale: Texture blocks are by definition correct samples of texture so problem only connecting them together

90

91

92

93

94

95

96 Failures (Chernobyl Harvest)

97 Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Our algorithm

98 Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Our algorithm

99 Portilla & Simoncelli Xu, Guo & Shum input image Wei & Levoy Our algorithm

100 Political Texture Synthesis!

101 Application: Texture Transfer Try to explain one object with bits and pieces of another object: + =

102 Texture Transfer Constraint Texture sample

103 Texture Transfer Take the texture from one image and paint it onto another object Same as texture synthesis, except an additional constraint: 1. Consistency of texture 2. Similarity to the image being explained

104 + =

105

106 Image Analogies Aaron Hertzmann 1,2 Chuck Jacobs 2 Nuria Oliver 2 Brian Curless 3 David Salesin 2,3 1 New York University 2 Microsoft Research 3 University of Washington

107 Image Analogies A A B B

108

109 Image Analogies Hertzmann et al A is to A as B is to B Learn A = f(a), apply B = f(b) Use patch similarity

110 Blur Filter

111 Edge Filter

112 Artistic Filters A A B B

113 Image Analogies Hertzmann et al. 2000

114

115

116 How does it work?

117 Features Luminance channel (remapped so that statistics of A and B are similar) Steerable pyramid on luminance channel Multi-scale oriented edge kernels That s it.

118 Colorization

119 Texture-by-numbers A A B B

120 Super-resolution Input A A

121 Super-resolution Result B B

122 Graphcut Textures: Image and Video Synthesis Using Graph Cuts Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk and Aaron Bobick To appear in Proc. ACM Transactions on Graphics, SIGGRAPH 2003

123 Graph cut vs Dynamic Programming What s the advantage of a graph cut over dynamic programming here?

124

125

126

127

128 Scene Completion

129 Don t blend, CUT! So far we only tried to blend between two images. What about finding an optimal seam? Moving objects become ghosts

130 Davis, 1998 Segment the mosaic Single source image per segment Avoid artifacts along boundaries Dijkstra s algorithm

131 Graph cuts (simple example à la Boykov&Jolly, ICCV 01) Dynamic programming can t handle loops.. Use graph cuts if we want similar cut-where-things-agree idea, but for closed regions which form a loop. Note: this image example is not a loop; It just shows the idea of a cut as related to graph flow. Image segmentation hard constraint sink n-links source a cut hard constraint Image pixels as nodes Minimum cost cut can be computed in polynomial time (max-flow/min-cut algorithms)

132 Kwatra et al, 2003 Actually, for this example, dynamic programming would work just as well, because the region isn t closed

133 Lazy Snapping This one is a loop! Interactive segmentation using graphcuts

134 Putting it all together Compositing images Have a clever blending function Feathering Center-weighted blend different frequencies differently Gradient based blending Choose the right pixels from each image Dynamic programming optimal seams Graph-cuts Now, let s put it all together: Interactive Digital Photomontage, 2004 (video)

135