Global Flow Estimation. Lecture 9

Transcription

1 Global Flow Estimation Lecture 9

2 Global Motion Estimate motion using all pixels in the image. Parametric flow gives an equation, which describes optical flow for each pixel. Affine Projective Global motion can be used to Remove camera (ego) motion (motion compensation) Object based segmentation generate mosaics

3 Global Motion Compensation Results I Aerial Video - EO Mosaic Alignment Mask

4 Global Motion Compensation Results II Aerial Video - IR Mosaic Alignment Mask

5 Detection Results

6 Video Mosaic

7 Contents Bergen et al method Affine transformation Mann & Piccard Homography (Projective) Pseudo Perspective Bi linear Image Warping Applications Mosaics COCOA system

8 Bergan et al Affine

9 Affine (0,0) Image at t-1 (0,0) Image at t x x (x,y ) (x,y) (1,1) (1,1)

10 Affine

11 Bergan et al Affine

12 Optical flow constraint eq Bergan et al

13 Bergan et al min Homework Linear system

14 Basic Components Pyramid construction Motion estimation Image warping Coarse to fine refinement

15 Coarse to fine global flow estimation u=1.25 pixels u=2.5 pixels u=5 pixels image H u=10 pixels image I Gaussian pyramid of image H Gaussian pyramid of image I

16 Coarse to fine global flow estimation Compute Global Flow Iteratively warp & upsample Compute Global Flow Iteratively... image JH Gaussian pyramid of image H image I Gaussian pyramid of image I

17 a 0 Level=2 W M 2 2 W * + a1 * W Level=1 W M W * a 2 * W Level=0 W M f(x,y,t-1) + a 3 f(x,y,t-1) f(x,y,t)

18 Estimation of Global Flow Single Iteration Compute A and B Solve Aa B Image t Image t+1 Warp by a

19 Estimation of Global Flow Iterative Initial Estimate T a a1 a2 b1 a3 a4 b2 Image t Image t+1 Warp by a Compute A and B Solve A a B Warp by a

20 Estimation of Global Flow Iterative Initial Estimate T a a1 a2 b1 a3 a4 b2 Iterate Image t Image t+1 Warp by a Compute A and B Solve A a B Update a

21 Image Warping Warping an image f into image h using some transformation g, involves mapping intensity at each pixel (x,y) in image f to a pixel (g(x),g(y)) in image h such that In case of affine transformation, is transformed to as: Displacement model Instantaneous model

22 Image Warping (Bergan et al) x x warp

23 Image Warping How about values in are not integer. But image is sampled only at integer rows and columns Instead of converting to and copying at we can convert integer values to and copy at

24 Image Warping x x warp

25 Image Warping But how about the values in are not integer. Perform bilinear interpolation to compute at non-integer values.

26 Warping Warped image at t-1 Difference image before Difference image after

27 Video Mosaic

28 Video Mosaic

29 mosaic Video Mosaic

30 Sprite

31 Mann & Picard Projective

32

33 Projective Flow (weighted) Optical Flow const. equation Projective transform

34 Projective Flow (weighted) minimize Homework

35 Projective Flow (weighted)

36 Projective Flow (unweighted)

37 Pseudo Perspective Taylor Series & by removing two square terms and constraining four remaining to 2 degrees of freedom

38 Bilinear Taylor Series & removing Square terms

39 Projective Flow (unweighted) Minimize

40 Bilinear and Pseudo Perspective bilinear Pseudo perspective Homework

41 Algorithm 1 Estimate q (using approximate model, e.g. bilinear model). Relate q to p select four points S1, S2, S3, S4 apply approximate model using q to compute exact p : ( x, y ) k k

42

43 Determining Projective transformation using point correspondences If point correspondences (x,y)<-->(x,y ) are known a s can be determined by least squares fit Two rows for each point i

44 Determining Projective transformation using point correspondences

45 Final Algorithm A Gaussian pyramid of three or four levels is constructed for each frame in the sequence. The parameters p are estimated at the top level of the pyramid, between the two lowest resolution images, g and h, using algorithm 1.

46 Final Algorithm The estimated p is applied to the next higher resolution image in the pyramid, to make images at that level nearly congruent. The process continues down the pyramid until the highest resolution image in the pyramid is reached.

47 Video Mosaics Mosaic aligns different pieces of a scene into a larger piece, and seamlessly blend them. High resolution image from low resolution images Increased filed of view

48 Steps in Generating A Mosaic Take pictures Pick reference image Determine transformation between frames Warp all images to the same reference view

49 Applications of Mosaics Virtual Environments Computer Games Movie Special Effects Video Compression

50 Steve Mann

51 Sequence of Images

52 Projective Mosaic

53 Affine Mosaic

54 Building

55 Wal Mart

56 Scientific American Frontiers

57 Scientific American Frontiers

58 Head mounted Camera at Restaurant

59 MIT Media Lab

60 COCOA: A System for Processing of Aerial Videos Saad Ali and Mubarak Shah, COCOA - Tracking in Aerial Imagery, SPIE Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications, Orlando, 2006.

61 COCOA System Flow Aerial Video Telemetry* Ego Motion Compensation Feature based + Gradient Based Motion Detection Accumulative Frame Differencing + Background Modeling + Object Segmentation Object Tracking Kernel Tracking + Blob Tracking + Occlusion Handling COCOA Registered Images Motion Detection Tracks Event Detection & Indexing

62 Registration Result I Aerial Video - EO Mosaic Alignment Mask

63 Registration Result II Aerial Video - IR Mosaic Alignment Mask

64 Detection Results

65 Tracking Results

66 References J. Bergen, P. Anandan, K. Hanna, and R. Hingorani, Hierarchical Model Based Motion Estimation, ECCV 92, pp Video orbits of the projective group a simple approach to featureless estimation of parameters S Mann, RW Picard Image Processing, IEEE Transactions on, 1997 Saad Ali and Mubarak Shah, COCOA Tracking in Aerial Imagery, SPIE Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and Applications, Orlando, R. Szeliski. Video mosaics for virtual environments, IEEE Computer Graphics and Applications, pages,22 30, March 1996.