Mariya Zhariy. Uttendorf Introduction to Optical Flow. Mariya Zhariy. Introduction. Determining. Optical Flow. Results. Motivation Definition

Transcription

1 to Constraint to Uttendorf 2005

2 Contents to Constraint 1

3 Contents to Constraint 1 2 Constraint

4 Contents to Constraint 1 2 Constraint 3

5 Visual cranial reflex(vcr)(?) to Constraint Rapidly changing scene (video games, virtual reality) Movements are identified by brain Brain sends signals to the eyes Eyes perform opposite movements in order to equilibrate the scene

6 Video sequence to Constraint

7 Video sequence to Constraint

8 Video sequence to Constraint

9 Video sequence to Constraint

10 Video sequence to Constraint

11 Video sequence to Constraint

12 Video sequence to Constraint

13 Video sequence to Constraint

14 Video sequence to Constraint

15 Video sequence to Constraint

16 Video sequence to Constraint

17 Video sequence to Constraint

18 Video sequence to Constraint

19 Video sequence to Constraint

20 Video sequence to Constraint

21 Video sequence to Constraint

22 Video sequence to Constraint

23 Video sequence to Constraint

24 Video sequence to Constraint

25 Video sequence to Constraint

26 Video sequence to Constraint

27 Video sequence to Constraint

28 Video sequence to Constraint

29 Video sequence to Constraint

30 Video sequence to Constraint

31 Video sequence to Constraint

32 Video sequence to Constraint

33 Video sequence to Constraint

34 Video sequence to Constraint

35 Video sequence to Constraint

36 Video sequence to Constraint

37 Video sequence to Constraint

38 Example: Rubik Cube to Constraint Image sequence Velocity field

39 : Assumptions to Constraint Pixel correspondence problem: Given a pixel in the first image, look for a nearby pixel in the second image with the same brightness. Key assumptions: brightness constancy small motion Resulting flow: displacement vector field Problems: great displacements changing illumination

40 Brightness constancy assumption to Constraint If I(x, t) image brightness, then I(x, t) I(x + x, t + t), where x is the displacemente of x after time t. Taylor series I(x + x, t + t) = I(x, t) + I v + I t + H.O.T., where v = x t

41 Constraint to Constraint Brightness constancy and small motion (vanishing H.O.T.) yuild: di(x, t) dt = I v + I t = 0, two velocity components, one equation underdefined problem so called aperture problem another constraint is required

42 Aperture Problem to Constraint Only the normal component of the velocity v in I direction is known: I v n = v I = I t I the tangential component v τ is unknown. Fazit: locally we can not see the tangential motion.

43 Techniques to Constraint Classification: Differential techniques Global methods Horn and Schunck (1st order) Nagel (2nd order) Local methods (Lucas and Kanade) Region-based matching Frequency-based methods Note: All OF techniques can use the hierarchical (coarse-to-fine) refinement.

44 Horn-Schunck Method to Constraint Regularisation: the optical flow constraint I v + I t = 0 combined with a smoothness assumption based on: v x 2 + v y 2 (Another measure of smoothness: v x + v y ) Look for v = (v x, v y ) minimizing functional: ( E(v) = I v + I ) 2 + λ( vx 2 + v y 2 )dx t Ω

45 Horn-Schunck Method to Constraint Minimization of E(v): Variational calculus: I x (I x v x + I y v y + I t ) + λ v x = 0 I y (I x v x + I y v y + I t ) + λ v y = 0 Discretising of derivatives via finite differences: where v = v v, v = v M, are local averages with mask ( 1/12 1/6 ) 1/12 M = 1/6 0 1/6 1/12 1/6 1/12

46 Horn-Schunck Method to Constraint Discretized equations in separated form: (λ + I 2 x + I 2 y )(v x v x ) = I x (I x v x + I y v y + I t ) (λ + I 2 x + I 2 y )(v y v y ) = I y (I x v x + I y v y + I t ) Gauss-Seidel Iteration: v n+1 x = v n x I x I x v n x + I y v n y + I t λ + I 2 x + I 2 y v n+1 y = v n y I y I x v n x + I y v n y + I t λ + I 2 x + I 2 y with v x, v y local averages of v x, v y.

47 Horn-Schunck Method to Constraint Advantages smooth flow global information accurate time derivatives, using more then two frames, possible Disadvantages iterative method: slow unsharp boundaries

48 Lucas-Kanade Method to Constraint Assume the velocity v = (v x, v y ) to be constant over a small neigbourhood Ω x of every x Ω. Minimize for all x Ω: y Ω x W (y)[ I(y, t) v + I t (y, t)], where W (x) is a weight function. Solution via normal equation: where A T W Av = A T W b, A = [ I(x 1 ),..., I(x n )] T W = diag[w (x 1 ),..., W (x n )] b = [I t (x 1 ),..., I t (x n )]

49 Lucas-Kanade Method to Constraint Advantages easy and fast calculation accurate time derivatives Disadvantages errors on boundaries Best combination between accuracy and speed.

50 Region-based Matching to Constraint Define velocity v as shift d = (d x, d y ). Consider sum-of-squared difference between two frames I 1 and I 2 : SSD(x, d) = y Ω x W (y x)[i 1 (y) I 2 (y + d)] 2 = W [I 1 (x) I 2 (x + d)] 2, where W 2-dim window function, d integer. Ω x is a 3 3, 5 5 etc. square with x in the middle. Note: Minimizing SSD is equal to maximizing the cross-correlation, which is the sum over products I 1 (x)i 2 (x + d)

51 Region-based Matching to Constraint Advantages easy to calculate Disadvantages only integer displacements: innacurate only local information used two-frame time derivatives: inaccurate All methods described here work better with presmoothed data.

52 Simple Example: Square to Unsmoothed sequence: Constraint

53 Simple Example: Square to Unsmoothed sequence: Constraint

54 Simple Example: Square to Unsmoothed sequence: Constraint

55 Simple Example: Square to Unsmoothed sequence: Constraint

56 Simple Example: Square to Unsmoothed sequence: Constraint

57 Simple Example: Square to Presmoothed sequence: Constraint

58 Simple Example: Square to Presmoothed sequence: Constraint

59 Simple Example: Square to Presmoothed sequence: Constraint

60 Simple Example: Square to Presmoothed sequence: Constraint

61 Simple Example: Square to Presmoothed sequence: Constraint

62 : to Constraint

63 : to Constraint

64 : to Constraint

65 Horn-Schunck vs Lucas-Kanade to Constraint

66 Horn-Schunck vs Lucas-Kanade to Constraint

67 Horn-Schunk: Rotation to Constraint

68 Horn-Schunk: Rotation to Constraint

69 Lucas-Kanade: Rotation to Constraint

70 Lucas-Kanade: Rotation to Constraint