CSCI 1290: Comp Photo

Transcription

1 CSCI 1290: Comp Photo Fall Brown University James Tompkin Many slides thanks to James Hays old CS 129 course, along with all its acknowledgements.

2 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Motion and perceptual organization Even impoverished motion data can evoke a strong percept

3 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Motion and perceptual organization Even impoverished motion data can evoke a strong percept

4 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Video A video is a sequence of frames captured over time A function of space (x, y) and time (t)

5 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Motion Applications Background subtraction Shot boundary detection Motion segmentation Segment the video into multiple coherently moving objects

6 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Computer Vision Motion and Optical Flow Many slides adapted from J. Hays, S. Seitz, R. Szeliski, M. Pollefeys, K. Grauman and others

7 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Motion estimation: Optical flow Optic flow is the apparent motion of objects or surfaces Will start by estimating motion of each pixel separately Then will consider motion of entire image

8 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Problem definition: optical flow How to estimate pixel motion from image I(x,y,t) to I(x,y,t+1)? Solve pixel correspondence problem Given a pixel in I(x,y,t), look for nearby pixels of the same color in I(x,y,t+1) Key assumptions I( x, y, t ) I( x, y, t + 1) Small motion: Points do not move very far Color constancy: A point in I(x,y,t) looks the same in I(x,y,t+1) For grayscale images, this is brightness constancy

9 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Optical flow constraints (grayscale images) I( x, y, t ) I( x, y, t + 1) Let s look at these constraints more closely Brightness constancy constraint (equation) I( x, y, t) = I( x + u, y + v, t + 1) Small motion: (u and v are less than 1 pixel, or smoothly varying) Taylor series expansion of I: I I I( x + u, y + v) = I( x, y) + u + v + [higher order terms] x y I I I( x, y) + u + v x y

10 CS 4495 Computer Vision A. Bobick Optical flow equation Combining these two equations 0 = I( x + u, y + v, t + 1) I( x, y, t) I( x, y, t + 1) + I u + I v I( x, y, t) x y Motion and Optic Flow (Short hand: I x = I x for t or t+1)

11 CS 4495 Computer Vision A. Bobick Optical flow equation Combining these two equations 0 = I( x + u, y + v, t + 1) I( x, y, t) I( x, y, t + 1) + I u + I v I ( x, y, t) [ I( x, y, t + 1) I( x, y, t)] + I u + I v I + I u + I v t x y I + I u, v t x y x y Motion and Optic Flow (Short hand: I x = I x for t or t+1)

12 CS 4495 Computer Vision A. Bobick Optical flow equation Combining these two equations 0 = I( x + u, y + v, t + 1) I( x, y, t) I( x, y, t + 1) + I u + I v I ( x, y, t) [ I( x, y, t + 1) I( x, y, t)] + I u + I v I + I u + I v t x y I + I u, v t x y x y Motion and Optic Flow (Short hand: I x = I x for t or t+1) In the limit as u and v go to zero, this becomes exact 0 = I + I u, v t Brightness constancy constraint equation I u + I v + I = x y t 0

13 CS 4495 Computer Vision A. Bobick Motion and Optic Flow How does this make sense? Brightness constancy constraint equation I u + I v + I = 0 x y t What do the static image gradients have to do with motion estimation?

14 The brightness constancy constraint Can we use this equation to recover image motion (u,v) at each pixel? 0 = I + I u, v or t I xu + I yv + It = How many equations and unknowns per pixel? One equation (this is a scalar equation!), two unknowns (u,v) The component of the motion perpendicular to the gradient (i.e., parallel to the edge) cannot be measured 0 If (u, v) satisfies the equation, so does (u+u, v+v ) if I u' v' T = 0 gradient (u,v) (u+u,v+v ) (u,v ) edge

15 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Aperture problem

16 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Aperture problem

17 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Aperture problem

18 CS 4495 Computer Vision A. Bobick Motion and Optic Flow The barber pole illusion

19 CS 4495 Computer Vision A. Bobick Motion and Optic Flow The barber pole illusion

20 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Solving the ambiguity B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence, pp , How to get more equations for a pixel? Spatial coherence constraint Assume the pixel s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25 equations per pixel

21 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Solving the ambiguity Least squares problem:

22 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Matching patches across images Overconstrained linear system Least squares solution for d given by The summations are over all pixels in the K x K window

23 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Conditions for solvability Optimal (u, v) satisfies Lucas-Kanade equation When is this solvable? What are good points to track? A T A should be invertible A T A should not be too small due to noise eigenvalues 1 and 2 of A T A should not be too small A T A should be well-conditioned 1 / 2 should not be too large ( 1 = larger eigenvalue) Does this remind you of anything? Criteria for Harris corner detector

24 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Low texture region gradients have small magnitude small 1, small 2

25 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Edge large gradients, all the same large 1, small 2

26 CS 4495 Computer Vision A. Bobick Motion and Optic Flow High textured region gradients are different, large magnitudes large 1, large 2

27 The aperture problem resolved Actual motion

28 The aperture problem resolved Perceived motion

29 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Errors in assumptions A point does not move like its neighbors Motion segmentation Brightness constancy does not hold Do exhaustive neighborhood search with normalized correlation - tracking features maybe SIFT more later. The motion is large (larger than a pixel) 1. Not linear: Iterative refinement 2. Local minima: coarse-to-fine estimation

30 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Revisiting the small motion assumption Is this motion small enough? Probably not it s much larger than one pixel How might we solve this problem?

31 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Optical Flow: Aliasing Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which correspondence is correct? actual shift estimated shift nearest match is correct (no aliasing) To overcome aliasing: coarse-to-fine estimation. nearest match is incorrect (aliasing)

32 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Reduce the resolution!

33 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Coarse-to-fine optical flow estimation u=1.25 pixels u=2.5 pixels u=5 pixels image 1 u=10 pixels image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2

34 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Coarse-to-fine optical flow estimation run iterative L-K run iterative L-K warp & upsample... image J1 image I2 Gaussian pyramid of image 1 Gaussian pyramid of image 2

35 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Optical Flow Results * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

36 CS 4495 Computer Vision A. Bobick Motion and Optic Flow Optical Flow Results * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

37 Large displacement optical flow, Brox et al., CVPR 2009 CS 4495 Computer Vision A. Bobick Motion and Optic Flow State-of-the-art optical flow, 2009 Start with something similar to Lucas-Kanade + gradient constancy + energy minimization with smoothing term + region matching + keypoint matching (long-range) Region-based +Pixel-based +Keypoint-based

38 CS 4495 Computer Vision A. Bobick Motion and Optic Flow State-of-the-art optical flow, 2015 CNN Pair of input frames Upsample estimated flow back to input resolution Near state-of-the-art in terms of end-point-error Fischer et al

39 CS 4495 Computer Vision A. Bobick Motion and Optic Flow State-of-the-art optical flow, 2015 Synthetic Training data Fischer et al

40 CS 4495 Computer Vision A. Bobick Motion and Optic Flow State-of-the-art optical flow, 2015 Results on Sintel Fischer et al

41 Optical flow Definition: the apparent motion of brightness patterns in the image Ideally, the same as the projected motion field Take care: apparent motion can be caused by lighting changes without any actual motion Imagine a uniform rotating sphere under fixed lighting vs. a stationary sphere under moving illumination.

42 Can we do more? Scene flow Combine spatial stereo & temporal constraints Recover 3D vectors of world motion Stereo view 1 Stereo view 2 t y x t-1 z 3D world motion vector per pixel

43 Scene flow example for human motion Estimating 3D Scene Flow from Multiple 2D Optical Flows, Ruttle et al., 2009

44 Scene Flow [Estimation of Dense Depth Maps and 3D Scene Flow from Stereo Sequences, M. Jaimez et al., TU Munchen]

45 CVMP 2011 Understanding that motion segmentation is hard: AUTOMATIC CINEMAGRAPHS Fabrizio Pece, Kartic Subr, Jan UCL

46 Professional, by hand examples

47 Overview

48 Algorithm

49 Motion segmentation

50 Stabilization

51 Our automatic examples

52 What s interesting: A good example of something that seems simple but rapidly becomes complicated: Registration with parallax (hand-held camera) Motion segmentation in general scenes Fast and HQ loop finding/frame interpolation Try to do this on non-trivial examples. Try to do this on a mobile phone in a few seconds. (Recently released software examples cannot cope).

53 Video Textures Arno Schödl Richard Szeliski David Salesin Irfan Essa Microsoft Research, Georgia Tech [SIGGRAPH 2000]

54 Still photos

55 Video clips

56 Video textures

57 Related work Image-based rendering View Morphing [Seitz 96] Lightfields / Lumigraph [Levoy 96, Gortler 96] Façade [Debevec 96] Virtualized Reality [Kanade 97] many more Novel still views from still images

58 Related work Texture synthesis [Heeger 95] [de Bonet 97] [Efros 99] Infinite extension of 2D pattern

59 Related work Temporal texture synthesis Statistical Learning of Multidimensional Textures [Bar-Joseph 99] [Wei 00] Video as a spatio-temporal texture

60 Related work Video Rewrite [Bregler 97] Assemble video snippets to generate lip motion

61 Related work Periodic motion analysis [Niyogi 94] [Seitz 97] [Polana 97] [Cutler 00] Uses similar analysis tools

62 Problem statement video clip video texture

63 Our approach How do we find good transitions?

64 Finding good transitions Compute L 2 distance D i, j between all frames vs. frame i frame j Similar frames make good transitions

65 Markov chain representation Similar frames make good transitions

66 Transition costs Transition from i to j if successor of i is like j Cost function: C i j = D i+1, j i i+1 ij D i+1, j j-1 j

67 Transition probabilities Probability for transition P i j inversely related to cost: P i j ~ exp ( C i j / s 2 ) high s low s

68 Preserving dynamics

69 Preserving dynamics

70 Preserving dynamics Cost for transition i j C i j = N -1 k = -N w k D i+k+1, j+k i-1 i i+1 i+2 D i-1, j-2 D i, j-1 i j D D i+1, j i+2, j+1 j-2 j-1 j j+1

71 Preserving dynamics effect Cost for transition i j C i j = N -1 k = -N w k D i+k+1, j+k

72 Dead ends No good transition at the end of sequence

73 Future cost Propagate future transition costs backward Iteratively compute new cost F i j = C i j + min k F j k

74 Future cost Propagate future transition costs backward Iteratively compute new cost F i j = C i j + min k F j k

75 Future cost Propagate future transition costs backward Iteratively compute new cost F i j = C i j + min k F j k

76 Future cost Propagate future transition costs backward Iteratively compute new cost F i j = C i j + min k F j k

77 Future cost Propagate future transition costs backward Iteratively compute new cost F i j = C i j + min k F j k Q-learning

78 Future cost effect

79 Finding good loops Alternative to random transitions Precompute set of loops up front

80 Philippe Levieux, James Tompkin, Jan Kautz UCL Interactive Viewpoint Video Textures

81 Interactive Viewpoint Video Textures Philippe Levieux, James Tompkin, Jan Kautz CVMP December 2018

82 Situation Realistic objects take a long time to create and render with computer graphics. For many real-world objects, we can use imagebased rendering. Very appropriate for some applications

83 Motivation [ Product display: Change viewpoint. Convincing depiction. Fast capture. Problem: Only allows static examples.

84 Our goal Introduce motion! Smooth spatial and temporal interpolation. Single input camera. Real-time exploration. Limit to repetitive or stochastic dynamics.

85 Video Textures Schödl, Arno et al., 'Video Textures', in: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, (2000), SIGGRAPH '00.

86 Light Field / Lumigraph Buehler, Chris et al., 'Unstructured lumigraph rendering', in: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, (2001), SIGGRAPH '01.

87 Existing methods Video Textures Our Method Light Fields / Lumigraph Single spatial position. Smooth temporal dynamics. Single camera. Many spatial positions. Smooth temporal dynamics. Single camera. Only repetitive motions. Many spatial positions. Difficult dynamics. Multiple cameras.

88 Our Method

89 Method - Capture

90 Method - Preprocess

91 Method - Preprocess

92 Method - Preprocess

93 Method - Rendering Time Within matrix 1 Within matrix 2 Final frame Across matrix 1->2

94 Method - Rendering Virtual viewpoint is rendered with optical flow. Distance between views is estimated from camera distance measure (slide 23). Flow vectors multiplied by distance to advect frames. Use Brox 2004 approximation by Eisemann 2008 (Floating Textures project).

95 Interactive application

96 Results

97 Scene split in regions

98 Estimating the Camera Distance

99 Varying the number of frames played at each virtual viewpoint

100 Varying the number of virtual viewpoints

101 Timings and Data Costs Capture: 30 minutes Preprocess: 2 to 9 hours n-to-n frame matching for each video and across videos is just expensive. n-to-n is required for good video textures.

102 Timings and Data Costs Rendering real-time interpolating frames: 60Hz 30Hz Additional data costs are small: One n-sided square matrix per video. One n-by-m video between video pairs.

103 Problems

104 How much data can you throw away? Rotations 14 baseline OK. 360 from 24 spatial locations. With short cycle (< 30s), capture would take 30 minutes with no setup or calibration.

105 How much data can you throw away? Dolly shots Focus of expansion (FOE) in dolly shots. Content near FOE is artefact free even at large baselines. Content far from FOE has artefacts even for small baselines. Baseline distances small for this scene as camera close to object. 2 cm 4 cm 7 cm

106 Altering scene content

107 Altering scene content

108 Summary Image-based renderings are often static. Interactive viewpoint video textures! Repetitive / stochastic / psuedo-random motions. Single camera, no calibration. Real-time view + dynamics + content interpolation. Product display / advertising applications.

109 THANK YOU

110 Eurographics 2012, Cagliari, Italy Interactive Multi-perspective Imagery from Photos and Videos Henrik Lieng 1,2 James Tompkin 1 Jan Kautz 1 University College London 1 University of Cambridge 2

111 Eurographics 2012, Cagliari, Italy Multi-perspective Imagery - Picasso 127

112 Eurographics 2012, Cagliari, Italy Multi-perspective Imagery - Hockney 128

113 Eurographics 2012, Cagliari, Italy Multi-perspective Imagery - Escher 129

114 Eurographics 2012, Cagliari, Italy Multi-perspective Imagery - Gonsalves 130

115 Eurographics 2012, Cagliari, Italy Multi-perspective Imagery - Head 131

116 Eurographics 2012, Cagliari, Italy Single-perspective images may not reveal enough information 132

117 Eurographics 2012, Cagliari, Italy Solved by multi-perspective imagery 133

118 Eurographics 2012, Cagliari, Italy Solved by multi-perspective imagery 134

119 Eurographics 2012, Cagliari, Italy Existing work focuses on specific aspects [Brown and Lowe, 2007] [Collomosse and Hall, 2003] [Agarwala et al., 2006] [Nomura et al., 2007] 135

120 Eurographics 2012, Cagliari, Italy 136

121 Eurographics 2012, Cagliari, Italy 137

122 Eurographics 2012, Cagliari, Italy 138

123 Eurographics 2012, Cagliari, Italy 139

124 Eurographics 2012, Cagliari, Italy 140

125 Eurographics 2012, Cagliari, Italy We support any kind of perspective transformation 141

126 Eurographics 2012, Cagliari, Italy Recursive multi-perspective imagery 142

127 Eurographics 2012, Cagliari, Italy OUR SYSTEM 143

128 Eurographics 2012, Cagliari, Italy Overview Code and data available online 144

129 Eurographics 2012, Cagliari, Italy Calibration (Structure from Motion) Bundler for photographs [Snavely et al., 2006]. Voodoo camera tracker for videos [Thormählen, 2006]. Can also use Photosynth (Microsoft) or VisualSFM [Wu, 2011]. Outputs 3D point cloud and camera parameters. 145

130 Eurographics 2012, Cagliari, Italy Prototype Interface 146

131 Eurographics 2012, Cagliari, Italy Portal propagation for image i Project 3D points from SfM to i; for point in user-defined portal p Find n closest projected points to p; Select the closest point in 3D to camera; end end Ensures that we use a visible 3D point. Refine poor propagations. 147

132 Eurographics 2012, Cagliari, Italy Warp and composite 148

133 Eurographics 2012, Cagliari, Italy Warp and composite For each point (x,y) in a propagated portal, move to (x,y ): x = a 0 + a 1 x + a 2 y + a 3 xy y = b 0 + b 1 x + b 2 y + b 3 xy Looking at our quadratic polynomial: Lines map to lines. Parallelism is not preserved. Use α-blending when compositing. 149

134 Eurographics 2012, Cagliari, Italy Warp interpolation (optical flow) With No interpolation 150

135 Eurographics 2012, Cagliari, Italy Warp interpolation (optical flow) SIFT flow [Liu et al., 2011] Robust to illumination changes. Slow (1 minute for 1MP portals). Use real-time GPU optical flow algorithms for more `interactiveness. E.G., [Pauwels and Van Hulle, 2008] 151

136 Eurographics 2012, Cagliari, Italy Colour matching Use [Reinhard et al., 2001]. 152

137 Eurographics 2012, Cagliari, Italy Object Removal 153

138 Eurographics 2012, Cagliari, Italy System recap: Code and data available online 154

139 Eurographics 2012, Cagliari, Italy Evaluation Interviews with professional artists (2D/3D). Positive towards the novelty of the results. Positive towards speed improvement. Popping artefacts not distracting. Nonplussed about automatic colour matching. But essential for video. Compared to artists current systems: Produces as good or better results. Is easier and faster to use. 155

140 Eurographics 2012, Cagliari, Italy Multi-perspective imagery from videos 156

141 Eurographics 2012, Cagliari, Italy 157