Learning to Track: Online Multi- Object Tracking by Decision Making

Transcription

1 Learning to Track: Online Multi- Object Tracking by Decision Making Yu Xiang 1,2, Alexandre Alahi 1, and Silvio Savarese 1 1 Stanford University, 2 University of Michigan ICCV

2 Multi-Object Tracking Visual surveillance Robot navigation Sport Analysis Autonomous driving 2

3 Batch Mode vs. Online Mode Batch Mode t-2 t-1 t t+1 t+2 time axis Online Mode t-2 t-1 t t+1 t+2 time axis 3

4 Tracking by Detection 4

5 Data Association? Tracks at time t-1 Detections at time t time axis 5

6 Challenges Noisy detection: false alarms and missing detections 6

7 Challenges Occlusion 7

8 Similarity Function for Data Association Zhang et al., CVPR 08 Berclaz et al., TPAMI 11 Breitenstein et al., TPAMI 11 Pirsiavash et al., CVPR 11 Butt & Collins, CVPR 13 Milan et al., TPAMI 14 Etc. Tracks at time t-1 Detections at time t time axis Ours Simple similarity measure + Powerful optimization 8

9 Learning to Track Different features/cues between targets and detections Appearance Location Motion Etc. φ 1 (, ) Similarity = w w n φ n (, ) Weights to combine different cues (to be learned) 9

10 Offline-learning vs. Online-learning 10

11 Offline-learning vs. Online-learning Training time With supervision Before Tracking Offlinelearning Onlinelearning During Tracking Use history of the target Li et al., CVPR 09 Kim et al., ACCV 12 Etc. 11

12 Offline-learning vs. Online-learning Training time With supervision Before Tracking Offlinelearning Onlinelearning During Tracking Use history of the target Song et al., ECCV 08 Kuo et al., CVPR 10 Bae et al., CVPR 14 Etc. 12

13 Our Solution: Tracking by Decision Making The target is tracked The target is occluded The target is tracked again 13

14 Inverse Reinforcement Learning tracked lost tracked Ground truth trajectory Markov Decision Process (MDP) Supervision Tracked Lost Tracked 14

15 Comparison between Different Learning Strategies Offlinelearning Onlinelearning Ours Training time Before Tracking During Tracking Before Tracking With supervision Use history of the target 15

16 Comparison between Different Learning Strategies Offlinelearning Onlinelearning Ours Training time Before Tracking During Tracking Before Tracking With supervision Use history of the target 16

17 Outline Markov Decision Process (MDP) for a Single Target Online Multi-Object Tracking with MDPs Experiments Conclusion 17

18 Outline Markov Decision Process (MDP) for a Single Target Online Multi-Object Tracking with MDPs Experiments Conclusion 18

19 Markov Decision Process for a Single Target object detection Tracked Active Lost Inactive 19

20 Markov Decision Process for a Single Target object detection Tracked Active Lost Inactive 20

21 Markov Decision Process for a Single Target object detection Tracked Active Lost Inactive 21

22 Markov Decision Process for a Single Target object detection Tracked Active Inactive 22

23 Markov Decision Process for a Single Target object detection Tracked Single object tracking Active Lost Inactive TLD Tracker. Z. Kalal, K. Mikolajczyk, and J. Matas. Tracking-learning-detection. TPAMI, 34(7): ,

24 Template Tracking in Tracked States Frame 50 Frame 51 24

25 Template Tracking in Tracked States Frame 50 Frame 51 25

26 Template Tracking in Tracked States Frame 50 Frame 51 26

27 Template Tracking in Tracked States Frame 50 Frame 51 Tracked 27

28 Template Tracking in Tracked States Frame 50 Frame 57 28

29 Template Tracking in Tracked States Frame 50 Frame 57 29

30 Template Tracking in Tracked States Frame 50 Frame 57 30

31 Template Tracking in Tracked States Frame 50 Frame 57 Tracked Lost 31

32 Markov Decision Process for a Single Target object detection Tracked Active Lost Inactive If lost for more than T frames 32

33 Data Association in Lost States tracked lost? t-2 t-1 t time axis 33

34 Learning the Similarity Function φ 1 (, ) Similarity = w w n φ n (, ) + b Inverse reinforcement learning: tracking objects in training videos! (, 1 ) (, 2 ) (, M ) Hard positive examples (, 1 ) (, 2 ) (, N ) Hard negative examples 34

35 Inverse Reinforcement Learning Ground truth trajectory Supervision 1 tracked lost t-2 t-1 t time axis 35

36 Inverse Reinforcement Learning Ground truth trajectory Supervision 1 tracked lost t-2 t-1 t time axis 36

37 Inverse Reinforcement Learning Ground truth trajectory Supervision 1 Wrong decision! Update your weights! tracked lost 2 (, 1 ) 3 Negative example 4 t-2 t-1 t time axis 37

38 Inverse Reinforcement Learning Ground truth trajectory Try it again Supervision 1 tracked lost No association 2 Wrong decision! Association to this one! Update your weights! 3 (, 2 ) 4 Positive example time axis t-2 t-1 t 38

39 Inverse Reinforcement Learning Try it again Ground truth trajectory Supervision 1 tracked lost 2 Good job! Keep going! No update of the weights 3 4 t-2 t-1 t time axis 39

40 Markov Decision Process for a Single Target object detection Tracked Active Lost Inactive 40

41 Outline Markov Decision Process (MDP) for a Single Target Online Multi-Object Tracking with MDPs Experiments Conclusion 41

42 Ensemble MDPs for Online Multi-Object Tracking MDP1 MDP2 MDP3 t-2 t-1 t time axis 42

43 Step 1: Process tracked targets MDP1 MDP2 MDP3 t-2 t-1 t time axis 43

44 Step 2: Process lost targets MDP1 MDP2 MDP3 Hungarian algorithm for lost targets t-2 t-1 t time axis 44

45 Step 3: Initialize new targets MDP1 MDP2 Initialize new targets MDP3 Terminate detection t-2 t-1 t time axis 45

46 Online Multi-Object Tracking with MDPs MDP1 Tracked Lost Tracked MDP2 Tracked Lost Tracked MDP3 Tracked Tracked Tracked 46

47 Outline Markov Decision Process (MDP) for a Single Target Online Multi-Object Tracking with MDPs Experiments Conclusion 47

48 Experiments: Dataset Multiple Object Tracking Benchmark [1] 11 training sequences 11 test sequences Object detections from the ACF detector [2] [1] L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler. MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking. arxiv: [cs], [2] P. Dollár, R. Appel, S. Belongie, and P. Perona. Fast feature pyramids for object detection. TPAMI, 36(8): ,

49 Experiments: Analysis on Validation Set Contribution of different components 49

50 Experiments: Analysis on Validation Set Contribution of different components object detection Tracked Active Lost Inactive MOTA: multiple object tracking accuracy 50

51 Experiments: Analysis on Validation Set Contribution of different components object detection Tracked Active Lost Inactive MOTA: multiple object tracking accuracy 51

52 Experiments: Analysis on Validation Set Contribution of different components object detection Tracked Active Lost Inactive MOTA: multiple object tracking accuracy 52

53 Experiments: Analysis on Validation Set Contribution of different components Similarity = w 1 φ 1 (, ) + + w n φ n (, ) + b MOTA: multiple object tracking accuracy 53

54 Experiments: Analysis on Validation Set Contribution of different components Similarity = w 1 φ 1 (, ) + + w n φ n (, ) + b MOTA: multiple object tracking accuracy 54

55 Training Sequences Experiments: Analysis on Validation Set Cross-domain tracking MOTA: multiple object tracking accuracy TUD-Stadtmitte ETH-Bahnhof ADL-Rundle-6 KITTI-13 PETS09-S2L1 Testing sequences 55

56 Training Sequences Experiments: Analysis on Validation Set Cross-domain tracking MOTA: multiple object tracking accuracy TUD-Stadtmitte ETH-Bahnhof ADL-Rundle-6 KITTI-13 PETS09-S2L1 Testing sequences 56

57 Training Sequences Experiments: Analysis on Validation Set Cross-domain tracking MOTA: multiple object tracking accuracy TUD-Stadtmitte ETH-Bahnhof ADL-Rundle-6 KITTI-13 PETS09-S2L1 Testing sequences 57

58 Experiments: Evaluation on Test Set Tracker Tracking Learning MOTA DP_NMS [1] Batch N/A 14.5 TC_ODAL [2] Online Online 15.1 TBD [3] Batch Offline 15.9 SMOT [4] Batch N/A 18.2 RMOT [5] Online N/A 18.6 CEM [6] Online N/A 19.3 SegTrack [7] Batch Offline 22.5 MotiCon [8] Batch Offline 23.1 MDP (Ours) Online Online 30.3 MOTA: multiple object tracking accuracy [1] Pirsiavash et al., CVPR 11 [2] Bae et al., CVPR 14 [3] Geiger et al., TPAMI 14 [4] Dicle et al., ICCV 13 [5] Yoon et al., WACV 15 [6] Milan et al., TPAMI 14 [7] Milan et al., CVPR 15 [8] Leal-Taixé et al., CVPR 14 58

59 Tracking Results 59

60 MDP [Ours] MotiCon [Leal-Taixé et al., CVPR 14] 60

61 MDP [Ours] MotiCon [Leal-Taixé et al., CVPR 14] 61

62 MDP [Ours] MotiCon [Leal-Taixé et al., CVPR 14] 62

63 Outline Markov Decision Process (MDP) for a Single Target Online Multi-Object Tracking with MDPs Experiments Conclusion 63

64 Conclusion Single Object Tracking Tracked Active Lost Object Detection Inactive Data Association Target Re-identification 64

65 Code 65

66 Single Object Tracking Tracked Active Lost Object Detection Inactive Data Association Target Re-identification Thank you! 66