Video-Based People Tracking. P. Fua EPFL IC-CVLab

Transcription

1 Video-Based People Tracking P. Fua EPFL IC-CVLab

2 Video-Based Tracking Challenges!Thousands of frames!frequent occlusions!poor quality of input images!sudden illumination changes 2

3 Multi-Step Algorithm Given cameras with overlapping fields of view and a discretized ground plane.! Estimate ground occupancy probabilities in individual temporal time frames.! Enforce temporal consistency under very weak assumptions.! A s s i g n i d e n t i t y a n d / o r behavior. 3

4 Ground Occupancy!Input: Binary images!output: Probability of Occupancy Map 4

5 Bayesian Formulation Provide probabilistic estimates of P( X 1,..., X N B 1,...,B C ) with " X i occupancy of location i # $ B j binary images from camera j!for individual multi-view time frames.!given very noisy binary images.!using blob sizes to estimate distance.! Consistent occlusion handling. 5

6 Generative Model A c (..,1,...,1,...,1,...) There are exactly three people at three different locations. 6

7 Interpretation Finds P ( X B) such that the average synthetic image matches the binary images. " Solution of a fixed point problem 7

8 Four Cameras Looking at the Court 8

9 Enforcing Temporal Consistency t=0 t=1 t=2 t=3 9

10 Graph-Based Formulation source sink People can transition from one location at one time to a neighboring one at the next instant. They can only enter and leave through virtual locations that correspond to exit or entrances. They are more likely to pass through high-probability locations. 10

11 Linear Program Maximize t,i log subject to t, i, j, f t i,j 0 t i 1 t fi,j t i j N (i) t, i, f t i,j 1 j N (i) t, i, j fi,j t N (i) k:i N (k) f t 1 k,i 0 j N ( source ) f source,j k: sink N (k) f k, sink 0. > Can be solved in real-time using the K-Shortest Path Algorithm (KSP)! 11

12 Four Cameras Looking at the Court 12

13 Soccer! Color-based appearance models.! 2 goal keepers, 2 sets of 10 players each, 3 referees. 13

14 Handball 14

15 Monocular Result 15

16 Preserving Identity! Read the numbers on the jersey whenever possible.! In practice, not very often. 16

17 Linear Program Appearance information used when available: maximize log t,i,l subject to t, i, j N (i) i(t) l i (t)l 1 i(t) L l=1 j f l i,j(t) 1 N (i) f l i,j(t) Probability of belonging to a specific group. t, l, i, f l i,j(t) f l k,i(t 1) 0 j N (i) k:i N (k) f source,j j N ( source ) t, l, K i=1 j N (i) k: sink N (k) f l i,j(t) t, l, i, j, f l i,j(t) 0. f k, sink 0 N l 17

18 Solving the LP in Real Time Run the KSP algorithm to select grid cells that are occupied. Segment the resulting trajectories into a set of tracklets that form en even more reduced graph. Solve the LP on the reduced graph. 18

19 Basketball! Read the numbers on the jersey whenever possible.! In practice, not very often. 19

20 Facial Identification 20

21 Facial Identification 21

22 Tracking the Ball Ambiguity Motion Blur Occlusion 22

23 Using Additional Knowledge Model both the! physics of the ball s motion,! interactions between ball and players. Interactions only Ballistics only Both 23

24 Volleyball 24

25 Other Sports 25

26 Tech Transfer 2014: PlayfulVision founded. 2015: Acquired by SecondSpectrum. 2016: Player tracking deal announced by the NBA. and it is SHALLOW! 26

27 Deep Background Subtraction and it could go Deep! 27

28 Conclusion Robust approach that can track arbitrary number of people over long periods of time. Does not require appearance information but can use it when available. Can handle the interaction between people and other moving objects. Real-time performance when using tracklets. > In use at the NBA and being extended to return 3D pose. 28

29 References and Code F. Fleuret, J. Berclaz, R. Lengagne and P. Fua, Multi- Camera People Tracking with a Probabilistic Occupancy Map, PAMI J. Berclaz, F. Fleuret, E. Türetken and P. Fua, Multiple Object Tracking using K-Shortest Paths, PAMI H. Ben Shitrit, J. Berclaz, F. Fleuret, and P. Fua, Multi- Commodity Network Flow for Tracking Multiple People, PAMI X. Wang, E. Turetken, F. Fleuret, and P. Fua. Tracking Interacting Objects Using Intertwined Flows, PAMI Code can be downloaded from