Performance Measures and the DukeMTMC Benchmark for Multi-Target Multi-Camera Tracking. Ergys Ristani

Performance Measures and the DukeMTMC Benchmark for Multi-Target Multi-Camera Tracking Ergys Ristani

In collaboration with: Francesco Solera Roger Zou Rita Cucchiara Carlo Tomasi

Outline Problem Definition Performance Measures DukeMTMCT Benchmark Summary

Problem Definition DukeMTMCT Benchmark Multi-Target Multi-Camera Tracking Given n camera streams, determine who/what is where at all times

Problem Definition Multi-Target Multi-Camera Tracking Given n camera streams, determine who/what is where at all times MTMCT Evaluation DukeMTMCT Benchmark How well does a tracker determine who/what is where at all times?

Evaluation Paradigms Suspect

Evaluation Paradigms Computed IDs ID1 Tracker (a) Suspect

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification Tracker (c) Suspect 7 switches 83% identification

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification Tracker (c) Suspect Researcher: How often does the tracker switch identities? 7 switches 83% identification

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification Tracker (c) Suspect 7 switches 83% identification

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification Tracker (c) Suspect End-user: How well does the tracker explain identity? 7 switches 83% identification

Evaluation Paradigms Computed IDs ID1 ID2 Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification Tracker (c) Suspect 7 switches 83% identification

Outline Problem Definition Performance Measures Existing Measures Issues Identity Measures DukeMTMCT Benchmark Summary

Performance Measures MTMC Tracker Performance

Performance Measures Interaction model Object detection Optimization Appearance model Parameter tuning Training data Motion model

Performance Measures MTMC Tracker Performance

Performance Measures MOTA IDS MTMC Tracker Performance MT ML FRG [1] Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. Image Video Proc. 2008 [2] Learning to Associate: HybridBoosted Multi-Target Tracker for Crowded Scene. CVPR 2009

Performance Measures MOTA IDS MT ML FRG MTMC Tracker Performance MCTA Handover (X, FRG) [3] An Equalized Global Graph Model-Based Approach for Multi-Camera Object Tracking. IEEE TCAS 2016 [4] Inter-camera Association of Multi-target Tracks by On-Line Learned Appearance Affinity Models. ECCV 2010

Performance Measures MOTA IDS MT ML FRG MTMC Tracker Performance MCTA Handover (X, FRG)

[5] Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol. TPAMI 2008 [6] A New Benchmark and Protocol for Multi-Object Detection and Tracking. arxiv CoRR 2015 [7] Evaluating Multi-Object Tracking. CVPRWS 2005 Performance Measures PR-MOTA MOTA IDS MCTA FIO FIT STDA-D ATA MT ML FRG MTMC Tracker Performance Handover (X, FRG) TP OP

Errors in Tracking Mismatches Fragmentations T 1 True Computed C 1 C 2 Merges T 1 T 2 C 1 False Positives/Negatives T 1 C 2

Key Evaluation Steps T 1 C 1 C 3 True Computed C 2 T 2

Key Evaluation Steps Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 T 2

Key Evaluation Steps Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2

Key Evaluation Steps Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2 PR-MOTA STDA-D ATA MOTA IDS MT ML MTMC Tracker Performance FRG MCTA Handover (X, FRG) FIO FIT OP TP

CLEAR MOT Mapping At each frame t Preserve mapping of detections from frame t-1 if still valid Solve bipartite matching at frame t for unmapped detections T 1 C 1 C 3 True Computed C 2 T 2 time

CLEAR MOT Mapping At each frame t Preserve mapping of detections from frame t-1 if still valid Solve bipartite matching at frame t for unmapped detections T 1 C 1 C 3 True Computed C 2 T 2 1 0 0 0 time

Key Evaluation Steps Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2 PR-MOTA STDA-D ATA MOTA IDS MT ML MTMC Tracker Performance FRG MCTA Handover (X, FRG) FIO FIT OP TP

Common Scoring Functions Multiple Object Tracking Accuracy (MOTA) Multiple Camera Tracking Accuracy (MCTA) Trajectory Scores MT, PT, ML Handover Errors

Outline Problem Definition Performance Measures Existing Measures Issues Identity Measures DukeMTMCT Benchmark Summary

Issues Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2 PR-MOTA STDA-D ATA MOTA IDS MT ML MTMC Tracker Performance FRG MCTA Handover (X, FRG) FIO FIT OP TP

CLEAR MOT Mapping Identity mapping bijective at each frame Not bijective overall T 1 C 1 T 2 C 2 T 1 C 1 C 3 True Computed C 2 T 2 time

Issues Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2 PR-MOTA STDA-D ATA MOTA IDS MT ML MTMC Tracker Performance FRG MCTA Handover (X, FRG) FIO FIT OP TP

Scoring Functions Multiple Object Tracking Accuracy (MOTA) Multiple Camera Tracking Accuracy (MCTA) Trajectory Scores MT, PT, ML Handover Errors

Scoring Functions Multiple Object Tracking Accuracy (MOTA) Multiple Camera Tracking Accuracy (MCTA) Trajectory Scores MT, PT, ML Handover Errors Brittle due to mapping

Handover Errors Camera 1 Camera 2 Truth A A A A A A A A A A A A A A A A

Handover Errors Camera 1 Camera 2 Truth A A A A A A A A A A A A A A A A Tracker 1 b b b b b b b c b b b b b b b b time

Handover Errors Camera 1 Camera 2 Truth A A A A A A A A A A A A A A A A Tracker 1 b b b b b b b c b b b b b b b b Tracker 2 b b b b b b b b b b b b b b b time

Scoring Functions Multiple Object Tracking Accuracy (MOTA) Multiple Camera Tracking Accuracy (MCTA) Trajectory Scores MT, PT, ML Ignore identity Handover Errors

Trajectory Scores Truth A A A A A A A A A A time

Trajectory Scores Truth A A A A A A A A A A time Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores Truth A A A A A A A A A A Tracker 1 b b c c e e g g f f time Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores Truth A A A A A A A A A A Tracker 1 b b c c e e g g f f MT = 1 time Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores Truth A A A A A A A A A A Tracker b b c c b g g g b b MT = 1 FRG = 4 time Mostly Tracked: Percentage of GT trajectories which are covered by tracker output for more than 80% in length Fragments: The number of times that a GT trajectory is interrupted in tracking result

Trajectory Scores Truth A A A A A A A A A A Tracker b b c c e e g g f f MT = 1 FRG = 4 time

Trajectory Scores Truth A A A A A A A A A A Tracker b b c c e e g g f f MT = 1 FRG = 4 Tracker++ b b c b b g b g b b time

Trajectory Scores Truth A A A A A A A A A A Tracker b b c c e e g g f f MT = 1 FRG = 4 Tracker++ b b c b b g b g b b MT = 1 FRG = 4 time

Scoring Functions Multiple Object Tracking Accuracy (MOTA) Multiple Camera Tracking Accuracy (MCTA) Bizarre combination Trajectory Scores MT, PT, ML Handover Errors

MCTA Camera 1 Camera 2 Truth A A A A A A A A A A A A A A A A Tracker 1 b b b b b b b c b b b b b b b b Tracker 2 b b b b b b c b b b b b b b b b time

MCTA Camera 1 Camera 2 Truth A A A A A A A A A A A A A A A A Tracker 1 b b b b b b b c b b b b b b b b Tracker 2 b b b b b b c b b b b b b b b b time Tracker 1 Fragmentations Handover Within camera Tracker 2

Scoring Functions Multiple Object Tracking Accuracy (MOTA) Multiple Camera Tracking Accuracy (MCTA) Mapping not one-to-one Trajectory Scores MT, PT, ML Handover Errors

MOTA True Computed A A A A A A A A b b b b c c c c time True A A A A A A A A Computed b b b b b b b c time

MOTA True Computed A A A A A A A A b b b b c c c c Half explanation time True A A A A A A A A Computed b b b b b b b c Nearly full explanation time

Outline Problem Definition Performance Measures Existing Measures Issues Identity Measures DukeMTMCT Benchmark Summary

Identity Measures Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2 PR-MOTA STDA-D ATA MOTA IDS MT ML MTMC Tracker Performance FRG MCTA Handover (X, FRG) FIO FIT OP TP

Identity Measures Proposed Truth-to-Result Matching Camera 1 Camera 2 True A A A A A A A A A A A A A A A A Computed b b b b b b c c b b b b b b time

Identity Measures Proposed Truth-to-Result Matching Camera 1 Camera 2 True A A A A A A A A A A A A A A A A Computed b b b b b b c c b b b b b b time A b c

Identity Measures Proposed Truth-to-Result Matching Camera 1 Camera 2 True A A A A A A A A A A A A A A A A Computed b b b b b b c c b b b b b b time A 4 14 16 12 2 0 0 b c Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures Proposed Truth-to-Result Matching Camera 1 Camera 2 True A A A A A A A A A A A A A A A A Computed b b b b b b b b b b b b time A 4 b Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures Proposed Truth-to-Result Matching Camera 1 Camera 2 True Computed A A A A A A A A A A A A A A A A c c time A 14 c Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures Proposed Truth-to-Result Matching Camera 1 Camera 2 True Computed A A A A A A A A A A A A A A A A time A 16 Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures Proposed Truth-to-Result Matching True Camera 1 Camera 2 Computed b b b b b b b b b b b b time b 12 Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures Truth-to-Result Matching T 1 C 1 C 3 True Computed C 2 Scoring Function T 2 PR-MOTA STDA-D ATA MOTA IDS MT ML MTMC Tracker Performance FRG MCTA Handover (X, FRG) FIO FIT OP TP

Identity Measures Proposed Scores ID Precision ID Recall F 1 -score False Negatives True Positives False Positives True Detections Computed Detections

Identity Measures Proposed Scores IDP: Precision Fraction of computed detections that are correctly identified. ID Recall F 1 -score False Negatives True Positives False Positives True Detections Computed Detections

Identity Measures Proposed Scores IDP: Precision Fraction of computed detections that are correctly identified. IDR: Recall Fraction of ground-truth detections that are correctly identified. F 1 -score False Negatives True Positives False Positives True Detections Computed Detections

Identity Measures Proposed Scores IDP: Precision Fraction of computed detections that are correctly identified. IDR: Recall Fraction of ground-truth detections that are correctly identified. FIDF1: 1 -score Ratio of correctly identified detections over the average number of ground-truth and computed detections False Negatives True Positives False Positives True Detections Computed Detections

Identity Measures Properties True and computed identities are mapped 1-1 Notion of identity switch is not present in ID measures The truth-to-result matching is the most favorable to the algorithm Applicable to single- and multi-camera tracking

Scope and Limitations ID measures are useful in: Surveillance Sport MTT for autonomous driving Not necessary when: Tracking indistinguishable targets (ants, sheep) Not applicable to: Tracking targets that merge/split (cells)

Practical Implications MOT 16 Challenge Tracker MOTA HCC 0.492 LMP 0.487 FWT 0.477 NLLMPa 0.475 MDPNN16 0.471 NOMT_16 0.464 JMC 0.462 QuadMOT16 0.441 oicf_16 0.432 MHT_DAM_16 0.429 LINF1_16 0.410 EAMTT_pub 0.388 OVBT 0.384 LTTSC-CRF 0.375 LP2D_16 0.357 TBD_16 0.337 CEM_16 0.331 DP_NMS_16 0.321 GMPHD_HDA 0.305 SMOT_16 0.297 JPDA_m_16 0.261 Tracker IDF1 NOMT_16 0.533 LMP 0.512 HCC 0.506 oicf_16 0.493 NLLMPa 0.473 JMC 0.463 MDPNN16 0.462 MHT_DAM_16 0.457 LINF1_16 0.456 FWT 0.442 EAMTT_pub 0.424 LTTSC-CRF 0.420 QuadMOT16 0.382 OVBT 0.378 CEM_16 0.357 LP2D_16 0.341 GMPHD_HDA 0.333 JPDA_m_16 0.311 DP_NMS_16 0.287 TBD_16 0.251 SMOT_16 0.185

IDP/IDR Curves Public detections Private detections KDNT DeepSORT POI NOMTwSDP FWT LMP HCC NLLMPa JMC NOMT oicf CEM JPDA

Performance Measures IDF1 IDP IDR PR-MOTA MOTA IDS MCTA FIO FIT STDA-D ATA MT ML FRG MTMC Tracker Performance Handover (X, FRG) TP OP [8] Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking. ECCV 2016 BMTT Workshop

Outline Problem Definition Performance Measures Existing Measures Issues Identity Measures DukeMTMCT Benchmark Summary

Previous MTMC Data Sets NLPR 1, 2 EPFL NLPR 3 APIDIS ISSIA PETS2009

DukeMTMC 8 static cameras x 85 minutes of 1080p 60 fps video More than 2,000,000 manually annotated frames More than 2,000 identities Annotation by 5 people over 1 year More identities than all existing datasets combined Unconstrained paths, diverse appearance

DukeMTMC@MOTChallenge

DukeMTMC@MOTChallenge Evaluation Single Camera Multi-Camera Ranking IDF1 score

DukeMTMC@MOTChallenge Trainval 0-50 min Test-hard 50-60 min Test-easy 60-85 min

Baseline

Baseline: BIPCC - +1 +0.4 +1 +1 Camera i Person detection +1-0.7-0.8 +1 +1 +1 +1 Multi-Camera Identities MTMC Tracking as Correlation Clustering Tracking Multiple People Online and in Real Time, E. Ristani and C. Tomasi, ACCV 2014

Baseline

Baseline Incorrect Matches

Baseline Correct Matches

DukeMTMCT Fragmentations Merges IDR IDP

DukeMTMCT Fragmentations Merges IDR IDP ID switches in CLEAR MOT don t correlate with ID measures

Outline Problem Definition Performance Measures DukeMTMCT Benchmark Summary

Summary MTMC trackers are complex - multiple uncorrelated measures are required to understand tracker performance PR-MOTA STDA-D MOTA IDS IDF1 IDP IDR Interaction model Optimization Object detection Appearance model MTMC Tracker Performance MCTA FIO FIT TP ATA MT ML Parameter tuning FRG Motion model Training data Handover (X, FRG) OP

Summary Large-scale realistic benchmarks make comparisons meaningful

Summary Different applications require different measures ID measures show bottom line performance: They are useful for end users of MTMC tracking systems Tracker (a) Suspect 1 switch 67% identification Tracker (b) Suspect 7 switches 67% identification Tracker (c) Suspect 83% identification

Thank you! Code/Data: vision.cs.duke.edu/dukemtmc/ [1] Performance Measures and Dataset for Multi-Target, Multi-Camera Tracking Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, Carlo Tomasi ECCV 2016 Workshop on Benchmarking Multi-Target Tracking [2] Tracking Social Groups Within and Across Cameras Francesco Solera, Simone Calderara, Ergys Ristani, Carlo Tomasi, Rita Cucchiara IEEE Transactions on Circuits and Systems 2016 [3] Tracking Multiple People Online and in Real Time Ergys Ristani and Carlo Tomasi ACCV 2014