Tracking Computer Vision Spring 2018, Lecture 24

Transcription

1 Tracking Computer Vision Spring 2018, Lecture 24

2 Course announcements Homework 6 has been posted and is due on April 20 th. - Any questions about the homework? - How many of you have looked at/started/finished homework 6? This week Yannis office hours will be on Wednesday 3-6 pm. - Added an extra hour to make up for change.

3 Overview of today s lecture KLT tracker. Mean-shift algorithm. Kernel density estimation. Mean-shift tracker. Modern trackers.

4 Slide credits Most of these slides were adapted from: Kris Kitani (16-385, Spring 2017).

5

6

7

8 All lectures and talks are free for CMU students! (You only need to pay for registration to attend the free food events.)

9 All lectures and talks are free for CMU students! (You only need to pay for registration to attend the free food events.) Everything in blue is free!

10 15-463/15-663/ Computational Photography Learn about this and other unconventional cameras and build some on your own! cameras that take video at the speed of light cameras that measure depth in real time cameras that capture entire focal stacks cameras that see around corners ICCP covers all of these

11 Kanade-Lucas-Tomasi (KLT) tracker

12

13 Feature-based tracking Up to now, we ve been aligning entire images but we can also track just small image regions too! (sometimes called sparse tracking or sparse alignment) How should we select the small images (features)? How should we track them from frame to frame?

14 History of the Kanade-Lucas-Tomasi Lucas Kanade (KLT) Tracker An Iterative Image Registration Technique with an Application to Stereo Vision Kanade Tomasi Detection and Tracking of Feature Points The original KLT algorithm Tomasi Shi Good Features to Track. 1994

15 Kanade-Lucas-Tomasi How should we track them from frame to frame? Lucas-Kanade Method for aligning (tracking) an image patch How should we select features? Tomasi-Kanade Method for choosing the best feature (image patch) for tracking

16 What are good features for tracking?

17 What are good features for tracking? Intuitively, we want to avoid smooth regions and edges. But is there a more is principled way to define good features?

18 What are good features for tracking? Can be derived from the tracking algorithm

19 What are good features for tracking? Can be derived from the tracking algorithm A feature is good if it can be tracked well

20 Recall the Lucas-Kanade image alignment method: error function (SSD) incremental update

21 Recall the Lucas-Kanade image alignment method: error function (SSD) incremental update linearize

22 Recall the Lucas-Kanade image alignment method: error function (SSD) incremental update linearize Gradient update

23 Recall the Lucas-Kanade image alignment method: error function (SSD) incremental update linearize Gradient update Update

24 Stability of gradient decent iterations depends on

25 Stability of gradient decent iterations depends on Inverting the Hessian When does the inversion fail?

26 Stability of gradient decent iterations depends on Inverting the Hessian When does the inversion fail? H is singular. But what does that mean?

27 Above the noise level both Eigenvalues are large Well-conditioned both Eigenvalues have similar magnitude

28 Concrete example: Consider translation model Hessian when is this singular? How are the eigenvalues related to image content?

29 interpreting eigenvalues l 2 l 2 >> l 1 What kind of image patch does each region represent? l 1 >> l 2 l 1

30 interpreting eigenvalues l 2 horizontal edge l 2 >> l 1 corner l 1 ~ l 2 flat vertical edge l 1 >> l 2 l 1

31 interpreting eigenvalues l 2 horizontal edge l 2 >> l 1 corner l 1 ~ l 2 flat vertical edge l 1 >> l 2 l 1

32 What are good features for tracking?

33 What are good features for tracking? big Eigenvalues means good for tracking

34 KLT algorithm 1. Find corners satisfying 2. For each corner compute displacement to next frame using the Lucas-Kanade method 3. Store displacement of each corner, update corner position 4. (optional) Add more corner points every M frames using 1 5. Repeat 2 to 3 (4) 6. Returns long trajectories for each corner point

35 Mean-shift algorithm

36

37 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975)

38 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Find the region of highest density

39 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Pick a point

40 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Draw a window

41 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the (weighted) mean

42 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

43 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

44 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

45 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

46 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

47 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

48 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

49 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

50 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

51 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

52 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

53 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

54 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window

55 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Compute the mean

56 Mean Shift Algorithm A mode seeking algorithm Fukunaga & Hostetler (1975) Shift the window To understand the theory behind this we need to understand

57 Kernel density estimation

58 To understand the mean shift algorithm Kernel Density Estimation A method to approximate an underlying PDF from samples sum of bumps bumps samples (+) Put bump on every sample to approximate the PDF

59 Say we have some hidden PDF p(x) probability density function cumulative density function

60 randomly sample 1 We can draw samples, using the CDF 0

61 randomly sample 1 0

62 randomly sample 1 0 samples

63 Now to estimate the hidden PDF place Gaussian bumps on the samples samples

64 samples discretized bump

65 samples

66 samples

67 Kernel Density Estimate approximates the original PDF samples

68 Kernel Density Estimation Approximate the underlying PDF from samples from it For example Gaussian bump aka kernel but there are many types of kernels! Put bump on every sample to approximate the PDF

69 Kernel Function returns the distance between two points

70 Epanechnikov kernel Uniform kernel Normal kernel These are all radially symmetric kernels

71 Radially symmetric kernels can be written in terms of its profile profile

72 Connecting KDE and the Mean Shift Algorithm

73 Mean-Shift Tracking Given a set of points: and a kernel: Find the mean sample point:

74 Mean-Shift Algorithm Initialize While place we start shift values becomes really small 1. Compute mean-shift compute the mean compute the shift 2. Update update the point Where does this algorithm come from?

75 Mean-Shift Algorithm Initialize While 1. Compute mean-shift Where does this come from? 2. Update Where does this algorithm come from?

76 How is the KDE related to the mean shift algorithm? Recall: Kernel density estimate (radially symmetric kernels) can compute probability for any point using the KDE! We can show that: Gradient of the PDF is related to the mean shift vector The mean shift vector is a step in the direction of the gradient of the KDE mean-shift algorithm is maximizing the objective function

77 In mean-shift tracking, we are trying to find this which means we are trying to

78 We are trying to optimize this: find the solution that has the highest probability usually non-linear non-parametric How do we optimize this non-linear function?

79 We are trying to optimize this: usually non-linear non-parametric How do we optimize this non-linear function? compute partial derivatives gradient descent!

80 Compute the gradient

81 Gradient Expand the gradient (algebra)

82 Gradient Expand gradient

83 Gradient Expand gradient Call the gradient of the kernel function g

84 Gradient Expand gradient change of notation (kernel-shadow pairs) keep this in memory:

85 multiply it out too long! (use short hand notation)

86 multiply by one! collecting like terms What s happening here?

87 mean shift constant mean shift! The mean shift is a step in the direction of the gradient of the KDE Let Can interpret this to be gradient ascent with data dependent step size

88 Mean-Shift Algorithm Initialize While 1. Compute mean-shift gradient with adaptive step size 2. Update Just 5 lines of code!

89 Everything up to now has been about distributions over samples

90 Mean-shift tracker

91 Dealing with images Pixels for a lattice, spatial density is the same everywhere! What can we do?

92 Consider a set of points: Associated weights: Sample mean: Mean shift:

93 Initialize While Mean-Shift Algorithm (for images) 1. Compute mean-shift 2. Update

94 For images, each pixel is point with a weight

95 For images, each pixel is point with a weight

96 For images, each pixel is point with a weight

97 For images, each pixel is point with a weight

98 For images, each pixel is point with a weight

99 For images, each pixel is point with a weight

100 For images, each pixel is point with a weight

101 For images, each pixel is point with a weight

102 For images, each pixel is point with a weight

103 For images, each pixel is point with a weight

104 For images, each pixel is point with a weight

105 For images, each pixel is point with a weight

106 For images, each pixel is point with a weight

107 For images, each pixel is point with a weight

108 Finally mean shift tracking in video!

109 Goal: find the best candidate location in frame 2 center coordinate of target center coordinate of candidate target candidate there are many candidates but only one target Frame 1 Frame 2 Use the mean shift algorithm to find the best candidate location

110 Non-rigid object tracking hand tracking

111 Compute a descriptor for the target Target

112 Search for similar descriptor in neighborhood in next frame Target Candidate

113 Compute a descriptor for the new target Target

114 Search for similar descriptor in neighborhood in next frame Target Candidate

115 How do we model the target and candidate regions?

116 Modeling the target M-dimensional target descriptor (centered at target center) a fancy (confusing) way to write a weighted histogram Normalization factor function of inverse distance (weight) quantization function bin ID A normalized color histogram (weighted by distance) sum over all pixels Kronecker delta function

117 Modeling the candidate M-dimensional candidate descriptor (centered at location y) a weighted histogram at y bandwidth

118 Similarity between the target and candidate Distance function Bhattacharyya Coefficient Just the Cosine distance between two unit vectors

119 Now we can compute the similarity between a target and multiple candidate regions

120 target image similarity over image

121 target we want to find this peak image similarity over image

122 Objective function same as Assuming a good initial guess Linearize around the initial guess (Taylor series expansion) function at specified value derivative

123 Linearized objective Remember definition of this? Fully expanded

124 Fully expanded linearized objective Moving terms around Does not depend on unknown y Weighted kernel density estimate where Weight is bigger when

125 OK, why are we doing all this math?

126 We want to maximize this

127 We want to maximize this Fully expanded linearized objective where

128 We want to maximize this Fully expanded linearized objective doesn t depend on unknown y where

129 We want to maximize this Fully expanded linearized objective only need to maximize this! doesn t depend on unknown y where

130 We want to maximize this Fully expanded linearized objective doesn t depend on unknown y where Mean Shift Algorithm! what can we use to solve this weighted KDE?

131 the new sample of mean of this KDE is (new candidate location) (this was derived earlier)

132 Mean-Shift Object Tracking For each frame: 1. Initialize location Compute Compute 2. Derive weights 3. Shift to new candidate location (mean shift) 4. Compute 5. If return Otherwise and go back to 2

133 Compute a descriptor for the target Target

134 Search for similar descriptor in neighborhood in next frame Target Candidate

135 Compute a descriptor for the new target Target

136 Search for similar descriptor in neighborhood in next frame Target Candidate

137

138 Modern trackers

139

140

141 Basic reading: Szeliski, Sections 4.1.4, 5.3. References