CAP5415-Computer Vision Lecture 8-Mo8on Models, Feature Tracking, and Alignment. Ulas Bagci

Transcription

1 CAP545-Computer Vision Lecture 8-Mo8on Models, Feature Tracking, and Alignment Ulas Bagci

2 Readings Szeliski, R. Ch. 7 Bergen et al. ECCV 92, pp Shi, J. and Tomasi, C. CVPR 94, pp Baker, S. and MaPhews, I. IJCV 24, pp Slide Credits: Szeliski, Shah and B. Freeman 2

3 Recap: Es8ma8ng Op8cal Flow Assume the image intensit I is constant Time = t Time = t+dt I, = (, t) I ( + d, + d, t + dt) 3

4 First Assump8on: Brightness Constraint I,! (, t) I ( + d, + d, t + dt) I((t)+u. t, (t)+v. t) I((t),(t),t) Assuming I is differen8able func8on, and epand the first term using Talor d d = Compact representa8on I u + I v + I t = Brightness constanc constraint 4

5 Lecture : Mo8on Models, Feature Tracking, and Alignment Second Assump8on: Gradient Constraint Velocit vector is constant within a small neighborhood (LUCAS AND KANADE) E(u, v) = Z, (I u + I v + I t ) 2 = 2(I u + I v + I t )I = 2(I u + I v + I t )I = 5

6 Lecture : Mo8on Models, Feature Tracking, and Alignment Recap: Lucas-Kanade apple P P apple applep I 2 I I P u P I I I I 2 v = I P t I I t Structural Tensor representa8on apple T T T T apple u v = apple Tt T t u = T tt T T T t T T 2 and v = T tt T T T t T T 2 6

7 Lecture : Mo8on Models, Feature Tracking, and Alignment Piaalls & Alterna8ves Brightness constanc is not sa8sfied Correla8on based method could be used A point ma not move like its neighbors Regulariza8on based methods The mo8on ma not be small (Talor does not hold!) Mul8-scale es8ma8on could be used 7

8 Lecture : Mo8on Models, Feature Tracking, and Alignment Mul8-Scale Flow Es8ma8on u=.25 piels u=2.5 piels u=5 piels image I t- u= piels image I t+ Gaussian pramid of image I t Gaussian pramid of image I t+ 8

9 Lecture : Mo8on Models, Feature Tracking, and Alignment Recap: Horn & Schunck Global method with smoothness constraint to solve aperture problem Minimize a global energ func8on E(u, v) = Z, [(I u + I v + I t ) ( ru 2 + rv 2 )]dd Take par8al deriva8ves w.r.t. u and v: (I u + I v + I t )I 2 ru = (I u + I v + I t )I 2 rv = 9

10 Global Mo8on Models (Parametric) All piels are considered to summarize global mo8on! 2D Models Affine Quadra8c Planar projec8ve (homograph) 3D Models Inst. Camera mo8on models Homograph + epipole Plane + paralla

11 Lecture : Mo8on Models, Feature Tracking, and Alignment Mo8on Models Translation Affine Perspective 3D rotation 2 unknowns 6 unknowns 8 unknowns 3 unknowns

12 Global Mo8on EsOmate mooon using all piels in the image 2

13 Global Mo8on EsOmate mooon using all piels in the image Global MoOon can be used to Remove camera mo8on Object-based segmenta8on generate mosaics 3

14 Global Mo8on EsOmate mooon using all piels in the image Global MoOon can be used to Remove camera mo8on Object-based segmenta8on generate mosaics 4

15 Recap: Object Tracking Track an object over a sequence of images 5

16 Challenges in Object Tracking 6

17 Challenges in Object Tracking Which features to track? 7

18 Challenges in Object Tracking Which features to track? Efficient tracking 8

19 Challenges in Object Tracking Which features to track? Efficient tracking Appearance constraint viola8on 9

20 Challenges in Object Tracking Which features to track? Efficient tracking Appearance constraint viola8on 2

21 Shi-Tomasi Feature Tracker Good Features to Track 2

22 Shi-Tomasi Feature Tracker Good Features to Track Find good features using eigenvalues of Hessian matri (threshold on the smallest eigenvalue when compu8ng Harris corner detec8on) 22

23 Shi-Tomasi Feature Tracker Good Features to Track Find good features using eigenvalues of Hessian matri (threshold on the smallest eigenvalue when compu8ng Harris corner detec8on) Track from frame to frame with LK 23

24 Shi-Tomasi Feature Tracker Good Features to Track Find good features using eigenvalues of Hessian matri (threshold on the smallest eigenvalue when compu8ng Harris corner detec8on) Track from frame to frame with LK Check consistenc of tracks b affine registra8on to the first observed instance of the feature 24

25 Shi-Tomasi Feature Tracker Shi and Tomasi CVPR 994 Good Features To Track. 25

26 KLT Tracking KLT: Kanade-Lucas-Tomasi 26

27 KLT Tracking KLT: Kanade-Lucas-Tomasi Tracking deals with es8ma8ng the trajector of an object in the image plane as it moves around a scene 27

28 KLT Tracking KLT: Kanade-Lucas-Tomasi Tracking deals with es8ma8ng the trajector of an object in the image plane as it moves around a scene Object tracking (car, airplane, person) 28

29 KLT Tracking KLT: Kanade-Lucas-Tomasi Tracking deals with es8ma8ng the trajector of an object in the image plane as it moves around a scene Object tracking (car, airplane, person) Feature tracking (Harris corners) 29

30 KLT Tracking KLT: Kanade-Lucas-Tomasi Tracking deals with es8ma8ng the trajector of an object in the image plane as it moves around a scene Object tracking (car, airplane, person) Feature tracking (Harris corners) Mul8ple object tracking 3

31 KLT Tracking KLT: Kanade-Lucas-Tomasi Tracking deals with es8ma8ng the trajector of an object in the image plane as it moves around a scene Object tracking (car, airplane, person) Feature tracking (Harris corners) Mul8ple object tracking Tracking in single/mul8ple camera(s) 3

32 KLT Tracking KLT: Kanade-Lucas-Tomasi Tracking deals with es8ma8ng the trajector of an object in the image plane as it moves around a scene Object tracking (car, airplane, person) Feature tracking (Harris corners) Mul8ple object tracking Tracking in single/mul8ple camera(s) Tracking in fied/moving camera 32

33 KLT Tracking Algorithm Find GoodFeaturesToTrack Harris Corners (thresholded on smallest eigenvalues) Use LK algorithm to find op8cal flows Use Coarse-to-Fine strateg to deal with large movements When crea8ng long tracks, check appearance of registered patch against appearance of ini8al patch to find points that have drited 33

34 Recent Developments at Op8cal Flow Start with LK or similar methods + Gradient consistenc + Energ minimiza8on with smoothing term + Region matching + KePoint matching Region-based +Piel-based +Kepoint-based Large displacement optical flow, Bro et al., CVPR 29 34

35 Recent Developments at Op8cal Flow Use of Machine Learning Deep Learning (ICCV 25, Fischer et al., FlowNet) 35

36 DeepFlow (Large Displacement Op8cal Flow) Basicall it is a matching algorithm with varia8onal approach [Weinzaepfel et al., ICCV 23]. 9/22/6 Dense correspondence (matching) Self-smooth matching Large displacement op8cal flow hpps:// 36

37 Lecture 8: Mo8on Models, Feature Tracking, and Alignment 9/22/6 Can we use SIFT features for tracking? 37

38 E: SIFT Tracking Frame à Frame 38

39 Lecture 8: Mo8on Models, Feature Tracking, and Alignment 9/22/6 How to evaluate correctness of op8cal flows? 39

40 Op8cal Flow - Quan8ta8ve Evalua8on E ep2 = p (u u ) 2 +(v v ) 2 E ep = u u + v v Where u=(u,v) is computed, u=(u*,v*) ground truth velocit vectors. E ang = arccos ( ut u u u ) 4

41 Interpreta8on of Op8cal Flow Fields 4

42 Interpreta8on of Op8cal Flow Fields Object features all have Zero velocit. 42

43 Interpreta8on of Op8cal Flow Fields 43

44 Interpreta8on of Op8cal Flow Fields Object is moving to the Right. 44

45 Interpreta8on of Op8cal Flow Fields 45

46 Interpreta8on of Op8cal Flow Fields Object is moving Directl toward the camera that is sta8onar 46

47 Interpreta8on of Op8cal Flow Fields 47

48 Interpreta8on of Op8cal Flow Fields Camera is moving into the scene, and an object moving passed the camera 48

49 Interpreta8on of Op8cal Flow Fields 49

50 Interpreta8on of Op8cal Flow Fields Object is rota8ng about the line of sight to the camera 5

51 Interpreta8on of Op8cal Flow Fields 5

52 Interpreta8on of Op8cal Flow Fields Object is rota8ng about an ais perpendicular to the line of sight. 52

53 Applica8on in Image Alignment Mo8on can be used for image alignment 53 = d c b a ' ' Piel loca8ons at 8me t Piel loca8ons at 8me t+

54 Prac8ce: Homogenous Coordinates Q: How can we represent translaoon as a 33 matri? ' = + t ' = + t 54

55 Prac8ce: Homogenous Coordinates 55 t t + = + = ' ' Q: How can we represent translaoon as a 33 matri? = t t Translation

56 Prac8ce: Homogenous Coordinates 56 t t + = + = ' ' Q: How can we represent translaoon as a 33 matri? = t t Translation + + = = ' ' t t t t t = 2 t =

57 Prac8ce: Basic 2D Transforma8ons 57 Θ Θ Θ Θ = cos sin sin cos ' ' = ' ' t t = ' ' sh sh Translate Rotate Shear = ' ' s s Scale

58 Affine Transforma8on 58 Affine transforma8ons are combina8ons of Linear transforma8ons, and Transla8ons Proper8es of affine transforma8ons: Origin does not necessaril map to origin Lines map to lines Parallel lines remain parallel Ra8os are preserved Closed under composi8on Models change of basis = w f e d c b a w ' ' = w i h g f e d c b a w ' ' ' projec8ve

59 Affine Transforma8on 59 = w f e d c b a w ' ' Θ Θ Θ Θ = w s s t t w cos sin sin cos ' ' ' p = T(t,t ) R(Θ) S(s,s ) p Affine matri decomposi8on TranslaOon+rotaOon+scaling

60 Lecture 8: Mo8on Models, Feature Tracking, and Alignment 9/22/6 Ques8ons? PA2 will be including op8cal flow es8ma8ons. 6