Applications of Image Motion Estimation I

Transcription

1 Applications of Image Motion Estimation I Mosaicing Princeton University COS 429 Lecture Oct. 16, 2007 Harpreet S. Sawhney hsawhney@sarnoff.com

2 Visual Motion Estimation : Recapitulation

3 Plan Explain optical flow equations Show inclusion of multiple constraints for solution Another way to solve is to use global parametric models

4 Brightness Constancy Assumption I 1 (p ) p u(p) I 2 (p) p Model image transformation as : I ( p ) = I1( p - u( p )) = I1( p 2 ) Brightness Constancy Reference Coordinate Optical Flow Corresponding Coordinate

5 How do we solve for the flow? I ( p ) = I1( p - u( p )) = I1( p 2 Use Taylor Series Expansion ) T I2 ( p ) = I1( p ) - I 1 u( p ) + O(2 ) Image Gradient Convert constraint into an objective function E T 2 SSD (u ) = ( I1 u( p ) + δi( p )) p R I ( p ) I1( 2 - p )

6 Optical Flow Constraint Equation At a Single Pixel T I2 ( p ) = I1( p ) - I 1 u( p ) + O(2 ) Leads to I T 1 u( p ) + δi( p ) 0 x y Ix u + Iyu + It = 0 - It I Normal Flow

7 Multiple Constraints in a Region E T 2 SSD (u ) = ( I1 u( p ) + I( p )) p R δ X

8 E Solution T 2 SSD (u ) = ( I1 u( p ) + I( p )) p R δ p R E SSD (u ) = 0 u T I ( I u( p ) + δi( p )) = 1 0 p R [ I I - δ T 1 ]u = I I p R Au = b

9 Solution Au = b ] I I I I I I [ A R 2 y R y x R y x R 2 x = δ δ = ] I I I I [ b R y R x - - Observations: A is a sum of outer products of the gradient vector A is positive semi-definite A is non-singular if two or more linearly independent gradients are available Singular value decomposition of A can be used to compute a solution for u

10 Another way to provide unique solution Global Parametric Models E T 2 SSD (u ) = ( I1 u( p ) + δi( p )) p R u(p) is described using an affine transformation valid within the whole region R u (p) = x [ 0 h11 h12 u (p) = Ηp + t H = [ ] h21 h22 y 0 0 x 0 y E SSD (u ) = 0 u ][h 1 E T 11 h12 h21 h22 t1 t2] T 2 SSD (u) = ( I1B(p) β + δi(p)) p R T T B(p) I I B(p)] β = p R p R t1 t = [ ] t2 u(p) = Β(p) β T [ - B(p) IδI 1 A β = b

11 Affine Motion Good approximation for : Small motions Small Camera rotations Narrow field of view camera When depth variation in the scene is small compared to the average depth and small motion Affine camera images a planar scene

12 Affine Motion X X Affine camera: p = s[ ] p s [ ] Y = Y 3D Motion: P = RP + T p = r s [ r T 1 T 2 P P + + T T x y ] = s R T 22 p + r s [ r ]Z + s T xy A 3D Plane: Z 1 = αx + βy + η = [ α β] p s + η p = s R r 1 T 13 22p + s [ ] [ α β]p + η + s Txy r23 s u (p) = p - p = Hp + t

13 Two More Ingredients for Success Iterative solution through image warping Linearization of the BCE is valid only when u(p) is small Warping brings the second image closer to the reference Coarse-to-fine motion estimation for estimating a wider range of image displacements Coarse levels provide a convex function with unique local minima Finer levels track the minima for a globally optimum solution

14 Image Warping I (p) = I1(p 2 - u(p)) (k) Express u(p) as: u(p) = u (p) + δu(p) I (p) (k) = I (p - u (p) - δu(p)) I (k) I1(p - u (p)) I w 1 (p) (p - w δ u(p)) Source Image : Filled Target Image : Empty p - u (k) (p) p

15 Coarse-to-fine Image Alignment : A Primer d(p) - Warper I (p) I (p+ u(p; Θ)) min ( Θ 1 2 ) p { R, T, d(p) } { H, e, k(p) } { dx(p), dy(p) } 2

16 Mosaics In Art combine individual chips to create a big picture... Part of the Byzantine mosaic floor that has been preserved in the Church of Multiplication in Tabkha (near the Sea of Galilee).

17 Image Mosaics Chips are images. May or may not be captured from known locations of the camera.

18 OUTPUT IS A SEAMLESS MOSAIC

19 VIDEOBRUSH IN ACTION

20 WHAT MAKES MOSAICING POSSIBBLE the simplest geometry... Single Center of Projection for all Images Projection Surfaces COP Images

21 Planar Mosaic Construction Align Pairwise: 1:2, 2:3, 3:4,... Select a Reference Frame Align all Images to the Reference Frame Combine into a Single Mosaic Virtual Camera (Pan) Image Surface - Plane Projection - Perspective

22 Key Problem What Is the Mapping From Image Rays to the Mosaic Coordinates? COP P ',P Images Rotations/Homographies Plane Projective Transformations K P ' = RP ' ' pc Rp c p p ' ' p ' RKp K H ' 1 RKp p

23 IMAGE ALIGNMENT IS A BASIC REQUIREMENT

24 PYRAMID BASED COARSE-TO-FINE ALIGNMENT a core technology... Coarse levels reduce search. Models of image motion reduce modeling complexity. Image warping allows model estimation without discrete feature extraction. Model parameters are estimated using iterative nonlinear optimization. Coarse level parameters guide optimization at finer levels.

25 ITERATIVE SOLUTION OF THE ALIGNMENT MODEL Assume that at the kth iteration, I (p ) = I (p ) = I (Ρ w w ( k ) p w ) ( k ) Ρ, is available model the residual transformation between the coordinate systems, p w [I + D]p w T w w w w w p I (p (p;d)) I (p (p;0)) + I D= 0 D w p D D= 0 p w (1+ d11)x + d12y + d d31x + d32y + 1 = d21x + (1+ d22 )y + d d31x + d32y D = I(p) w p D x = 0 y w p and p, as: x xy xy y D= x 0 y Ρ (k+ 1) Ρ (k) [I + D]

26 ITERATIVE REWEIGHTED SUM OF SQUARES Given a solution l i ρ(r & i) r i (m) Θ r θ i k ri θ at the mth iteration, find δθ by solving : l θ l = ρ(r & i r i ) r i r θ i k k w i wi acts as a soft outlier rejecter : ρ& (r) r SS = 1 σ 2 ρ& (r) r GM = 2 2σ 2 (σ + r 2 ) 2

27 PROGRESSIVE MODEL COMPLEXITY combining real-time capture with accurate alignment... Provide user feedback by coarsely aligning incoming frames with a low order model robust matching that covers a wide search range achieve about 6-8 frames a sec. on a Pentium 200 Use the coarse alignment parameters to seed the fine alignment increase model complexity from similarity, to affine, to projective parameters coarse-to-fine alignment for wide range of motions and managing computational complexity

28 COARSE-TO-FINE ALIGNMENT I(t) estimate global shift d r (t) d r (t) delay I (t 1) warp Iˆ ( t 1)

29 VIDEO MOSAIC EXAMPLE Princeton Chapel Video Sequence 54 frames

30 UNBLENDED CHAPEL MOSAIC

31 Image Merging withlaplacian Pyramids Image 1 Image Combined Seamless Image

32 VORONOI TESSELATIONS W/ L1 NORM

33 BLENDED CHAPEL MOSAIC

34 1D vs. 2D SCANNING 1D : The topology of frames is a ribbon or a string. Frames overlap only with their temporal neighbors. (A 300x332 mosaic captured by mosaicing a 1D sequence of 6 frames) 2D : The topology of frames is a 2D graph Frames overlap with neighbors on many sides

35 1D vs. 2D SCANNING The 1D scan scaled by 2 to 600x692 A 2D scanned mosaic of size 600x692

36 CHOICE OF 1D/2D MANIFOLD Plane Cylinder Cone Sphere Arbitrary

37

38

39 THE DESMILEY ALGORITHM Compute 2D rotation and translation between successive frames Compute L by intersecting central lines of each frame Fill each pixel [l y] in the rectified planar mosaic by mapping it to the appropriate video frame

40

41 2D MOSAICING THROUGH TOPOLOGY INFERENCE & LOCAL TO GLOBAL ALIGNMENT automatic solution to two key problems... Inference of 2D neighborhood relations (topology) between frames Input video just provides a temporal 1D ordering of frames Need to infer 2D neighborhood relations so that local constraints may be setup between pairs of frames Globally consistent alignment and mosaic creation Choose appropriate alignment model Local constraints incorporated in a global optimization

42 PROBLEM FORMULATION Given an arbitrary scan of a scene Create a globally aligned mosaic by minimizing min { P i } E = ij G E ij + E i i 2 + σ (Area of the mosaic) Like an MDL measure : Create a compact appearance while being geometrically consistent

43 ERROR MEASURE min { P i } E = ij G E ij + E i i 2 + σ (Area of the mosaic) where Pi : Reference - to - image mapping, ui = PX i E : Any measure of alignment error between neighbors i G E ij i : Graph that represents the neighborhood relations : Frame to reference error term to allow for a priori criterion like least distortion transformation and j

44 ALGORITHMIC APPROACH From a 1D ordered collection of frames to A Globally consistent set of alignment parameters 1. Graph Topology Determination Iterate through Given: pose of all frames Establish neighborhood relations min(area of Mosaic) Graph G 2. Local Pairwise Alignment Given: G Quality measure validates hypothesized arcs Provides pairwise constraints 3. Globally Consistent Alignment Given: pairwise constraints Compute reference-to-frame pose parameters min Eij

45 * Initialize using low order frame-to-frame mosaic algorithm on a plane GRAPH TOPOLOGY DETERMINATION Given: Current estimate of pose* Lay out each frame on the 2D manifold (plane, sphere, etc.) Hypothesize new neighbors based on proximity predictability of relative pose non redundancy w.r.t. current G d ij D ij Specifically, try arc (i,j) if Normalized Euclidean dist d ij << Path distance D ij Validate hypothesis by local registration Add arc to G if good quality registration

46 LOCAL COARSE & FINE ALIGNMENT Given: a frame pair to be registered Coarse alignment Low order parametric model e.g. shift, or 2D R & T Majority consensus among subimage estimates Fine alignment [Bergen, ECCV 92] Coarse to fine over Laplacian pyramid Progressive model complexity, up to projective Incrementally adjust motion parameters to minimize SSD Quality measure Normalized correlation helps reject invalid registrations

47 GLOBALLY CONSISTENT ALIGNMENT Given: arcs ij in graph G of neighbors The local alignment parameters, Qij, help establish feature correspondence between i and j If uil and ujl are corresponding points in frames i,j, then ui uj E ij = 1 1 Pi ( uil) Pj ( ujl ) 2 Eij Incrementally adjust poses P i to minimize min { P i } E = ij G E ij + E i i

48 SPECIFIC EXAMPLES : 1. PLANAR MOSAICS Mosaic to frame transformation model: u P i X Local Registration Coarse 2D translation & fine 2D projective alignment Topology : Neighborhood graph defined over a plane Initial graph topology computed with the 2D T estimates Iterative refinement using arcs based on projective alignment Global Alignment E ij = k 2 ( Aiuik ) ( A jujk ) Pair Wise Alignment Error Ei = 2 k= 1 ( ( A α ) ( A i k β )) ( α j k k 2 β ) k Minimum Distortion Error

49 PLANAR TOPOLOGY EVOLUTION Whiteboard Video Sequence 75 frames

50 PLANAR TOPOLOGY EVOLUTION

51 FINAL MOSAIC

52 SPECIFIC EXAMPLES : 2. SPHERICAL MOSAICS Frame to mosaic transformation model: u FR T i X Local Registration Coarse 2D translation & fine 2D projective alignment Parameter Initialization Compute F and R s from the 2D projective matrices Topology : Initial graph topology computed with the 2D R & T estimates on a plane Subsequently the topology defined on a sphere Iterative refinement using arcs based on alignment with F and R s Global Alignment E ij = k 1 1 R if uik R jf ujk 2

53 SPHERICAL MOSAICS Sarnoff Library Video Captures almost the complete sphere with 380 frames

54 SPHERICAL TOPOLOGY EVOLUTION

55 SPHERICAL MOSAIC Sarnoff Library

56 SPHERICAL MOSAIC Sarnoff Library

57 NEW SYNTHESIZED VIEWS

58 FINAL MOSAIC Princeton University Courtyard

59 Mosaicing from Strips Image 1 Image 2 Image 3 Image 4 Image 5 Mosaic from center strips

60 Problem: Forward Translation

61 General Camera Motion Strip Perpendicular to Optical Flow Cut/Paste Strip (warp to make Optical Flow parallel) Parallel Flow: Straight Strip Radial Flow (FOE): Circular Strip

62 Mosaic Construction Mosaic

63 Simple Cases ax + M = 0 0 ) ( = + + M y x b = 0 a v u = by bx v u Horizontal Translation Zoom (M determines displacement) (M determines radius)

64 Manifold for Forward Motion Stationary (but rotating) Camera Viewing Sphere Translating Camera Sphere carves a Pipe in space

65 Pipe Projection One Image Sequence

66 Concatenation of Pipes

67 Forward Motion Mosaicing

68 Example: Forward Motion

69 Side View of Mosaic

70 Forward Mosaicing II

71 Mosaic Construction

72 OmniStereo: Stereo in Full 360 o Two Panoramas: One for Each Eye Each panorama can be mapped on a cylinder

73 Paradigm: A Rotating Stereo Pair of Slit Cameras Rays are tangent to viewing circle (Gives 360 o stereo) Image planes are radial (Makes mosaicing difficult)

74 Panoramic Projections of Slit Cameras Image Surface Regular panorama (single viewpoint): Projection rays orthogonal to cylinder Directional Viewpoints Left Eye panorama (viewing circle) Projection rays tilted right Right Eye panorama (viewing circle) Projection rays tilted left

75 Slit Camera Model Center Strip: Rays perpendicular to image plane Side Strip: Rays tilted from image plane Pinhole Aperture Vertical Slit (a) (b) Center Strip Image Plane (c) Side Strip

76 Stereo Panorama from Strips Left-eye-Panorama from right strips Image 1 Image 2 Image 3 Image 4 Image 5 Right-eye-Panorama from left strips

77 MultiView Panoramas Right-Eye Panorama Left-Eye Panorama 360 ο 180 ο 0 ο Regular Panorama

78 Stereo Panorama from Video Stereo viewing with Red/Blue Glasses

79 Viewing Panoramic Stereo Printed Cylindrical Surfaces Print panorama on a cylinder No computation needed!!!

80