Global Flow Estimation. Lecture 9

Transcription

1 Motion Models Image Transformations to relate two images 3D Rigid motion Perspective & Orthographic Transformation Planar Scene Assumption Transformations Translation Rotation Rigid Affine Homography Pseudo Perspective

2 Global Flow Estimation Lecture 9

3 Global Motion Estimate motion using all pixels in the image. Parametric flow gives an equation, which describes optical flow for each pixel. Affine Projective Global motion can be used to Remove camera (ego) motion (motion compensation) Object based segmentation generate mosaics

4 Global Motion Compensation Results I Aerial Video - EO Mosaic Alignment Mask

5 Global Motion Compensation Results II Aerial Video - IR Mosaic Alignment Mask

6 Detection Results

7 Video Mosaic

8 Contents Bergen et al method Affine transformation Mann & Piccard Homography (Projective) Pseudo Perspective Bi linear Image Warping Applications Mosaics COCOA system

9 Bergan et al Affine

10 Affine (0,0) Image at t-1 (0,0) Image at t x x (x,y ) (x,y) (1,1) (1,1)

11 Affine

12 Bergan et al Affine

13 Optical flow constraint eq Bergan et al

14 Bergan et al min Homework Linear system

15 Basic Components Pyramid construction Motion estimation Image warping Coarse to fine refinement

16 Coarse to fine global flow estimation u=1.25 pixels u=2.5 pixels u=5 pixels image H u=10 pixels image I Gaussian pyramid of image H Gaussian pyramid of image I

17 Coarse to fine global flow estimation Compute Global Flow Iteratively warp Compute Global Flow Iteratively... image JH Gaussian pyramid of image H image I Gaussian pyramid of image I

18 a 0 Level=2 W M 2 2 W * + a1 * W Level=1 W M W * a 2 * W Level=0 W M f(x,y,t-1) + a 3 f(x,y,t-1) f(x,y,t)

19 Estimation of Global Flow Single Iteration Compute A and B Solve Aa B Image t Image t+1 Warp by a

20 Estimation of Global Flow Iterative Initial Estimate T a a1 a2 b1 a3 a4 b2 Image t Image t+1 Warp by a Compute A and B Solve A a Update a a Warp by B

21 Estimation of Global Flow Iterative Initial Estimate T a a1 a2 b1 a3 a4 b2 Iterate Image t Image t+1 Warp by a Compute A and B Solve A a B Update a

22 Image Warping Warping an image f into image h using some transformation g, involves mapping intensity at each pixel (x,y) in image f to a pixel (g(x),g(y)) in image h such that In case of affine transformation, is transformed to as: Displacement model Instantaneous model

23 Image Warping (Bergan et al) x x warp

24 Image Warping How about values in are not integer. But image is sampled only at integer rows and columns Instead of converting to and copying at we can convert integer values to and copy at

25 Image Warping x x warp

26 Image Warping But how about the values in are not integer. Perform bilinear interpolation to compute at non-integer values.

27 Warping Warped image at t-1 Difference image before Difference image after

28 Global Motion Compensation

29 Football Original frames Aligned frames Optical flow

30 Video Mosaic

31 Video Mosaic

32 mosaic Video Mosaic

33 Sprite

34 Mann & Picard Projective

35

36 Projective The affine model cannot capture camera pan and tilt cannot properly express the keystoning and chirping chirping is the effect of increasing or decreasing spatial frequency with respect to spatial location

37 Motion Models

38 Projective Flow (weighted) Optical Flow const. equation Projective transform

39 Projective Flow (weighted) minimize Homework

40 Projective Flow (weighted)

41 Projective Flow (unweighted)

42 Pseudo Perspective Taylor Series & by removing two square terms and constraining four remaining to 2 degrees of freedom

43 Bilinear Taylor Series & removing Square terms

44 Projective Flow (unweighted) Minimize

45 Bilinear and Pseudo Perspective bilinear Pseudo perspective Homework

46 Algorithm 1 Estimate q (using approximate model, e.g. bilinear model). Relate q to p select four points S1, S2, S3, S4 apply approximate model using q to compute exact p : ( x, y ) k k

47

48 Alignment Features: Harris Corner Feature Descriptor: SIFT Descriptor Matching Projective (homography) fitting

49 Determining Projective transformation using point correspondences If point correspondences (x,y)<-->(x,y ) are known a s can be determined by least squares fit Two rows for each point i

50 Determining Projective transformation using point correspondences

51 Final Algorithm A Gaussian pyramid of three or four levels is constructed for each frame in the sequence. The parameters p are estimated at the top level of the pyramid, between the two lowest resolution images, g and h, using algorithm 1.

52 Final Algorithm The estimated p is applied to the next higher resolution image in the pyramid, to make images at that level nearly congruent. The process continues down the pyramid until the highest resolution image in the pyramid is reached.

53 Video Mosaics Mosaic aligns different pieces of a scene into a larger piece, and seamlessly blend them. High resolution image from low resolution images Increased filed of view

54 Steps in Generating A Mosaic Take pictures Pick reference image Determine transformation between frames Warp all images to the same reference view

55 Applications of Mosaics Virtual Environments Computer Games Movie Special Effects Video Compression

56 Steve Mann

57 Sequence of Images

58 Projective Mosaic

59 Affine Mosaic

60 Building

61 Wal Mart

62 Scientific American Frontiers

63 Scientific American Frontiers

64 Head mounted Camera at Restaurant

65 MIT Media Lab

66 COCOA: A System for Processing of Aerial Videos Saad Ali and Mubarak Shah, COCOA - Tracking in Aerial Imagery, SPIE Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications, Orlando, 2006.

67 COCOA System Flow Aerial Video Telemetry* Ego Motion Compensation Feature based + Gradient Based Motion Detection Accumulative Frame Differencing + Background Modeling + Object Segmentation Object Tracking Kernel Tracking + Blob Tracking + Occlusion Handling COCOA Registered Images Motion Detection Tracks Event Detection & Indexing

68 Registration Result I Aerial Video - EO Mosaic Alignment Mask

69 Registration Result II Aerial Video - IR Mosaic Alignment Mask

70 Detection Results

71 Tracking Results

72 References J. Bergen, P. Anandan, K. Hanna, and R. Hingorani, Hierarchical Model Based Motion Estimation, ECCV 92, pp Video orbits of the projective group a simple approach to featureless estimation of parameters S Mann, RW Picard Image Processing, IEEE Transactions on, 1997 Saad Ali and Mubarak Shah, COCOA Tracking in Aerial Imagery, SPIE Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications, Orlando, R. Szeliski. Video mosaics for virtual environments, IEEE Computer Graphics and Applications, pages,22 30, March 1996.