Modeling and Propagating CNNs in a Tree Structure for Visual Tracking

Transcription

1 The Visual Object Tracking Challenge Workshop 2016 Modeling and Propagating CNNs in a Tree Structure for Visual Tracking Hyeonseob Nam* Mooyeol Baek* Bohyung Han Dept. of Computer Science and Engineering *Equal contributors

2 Key concept 2 Multiple CNNs trained on tracking results from different time slices Modeling and propagating CNNs in a tree structure

3 Deep Learning for Visual Tracking Response map from CNN features HCF [Ma2016] FCNT [Wang2015] DeepSRDCF [Danelljan2015] 3 Pre-train CNN with tracking datasets MDNet [Nam2015]

4 Challenges Less training data Catastrophic forgetting 4 Online update Only a single labeled frame CNN forgets previous information Overfitting Difficult to train Unstable model Easily drifts

5 Our method CNNs trained on frames from different time slices State estimation Model update 5

6 Overall Algorithm 1 Initialize model with ground-truth 6 CNN1 1 st frame Model ground-truth positive patches negative patches

7 Overall Algorithm 2 Estimate target location for T consecutive frames 7 CNN1 (T+1) 23 nd rd st frame Model tracking results target candidates

8 Overall Algorithm 3 Update model 8 CNN1 21 nd st frame 31 rd st frame (T+1) 1 st st frame CNN2 Model tracking results positive patches negative patches Choose CNN with the best target score as a parent Maintain N latest CNNs

9 Overall Algorithm 4 Iterate until the end of sequence 9 CNN1 CNN2 (2T+1) (T+2) (T+3) nd rdst frame Model tracking results target candidates

10 Overall Algorithm 4 Iterate until the end of sequence 10 CNN1 (T+2) 1 st nd frame (T+3) 1 st rd frame (2T+1) 1 st frame st frame CNN2 Model CNN3 tracking results positive patches negative patches Choose CNN with the best target score as a parent Maintain N latest CNNs

11 Structure of CNN 11 Convolution layers are from VGG-M [Chatfield2014] network. Output: normalized vector [φ(x), 1 φ(x)] T Fully-connected layers are randomly initialized. Update fully-connected layers only (fc4, fc5, output)

12 State Estimation Weighted-average scores from multiple CNNs w N : min(max(φ N (x)), β N ) affinity of CNN N with t th frame reliability of CNN N 12 CNN w 1 CNN w 2 t th frame patch x* CNN 3 CNN N models w 3 w N scores φ N (x*) weights 0.79 score

13 Model Update 1 Sample training data for model update Store target scores of each CNN for parent selection 13 CNN w 1 CNN w 2 CNN w score t th frame patch CNN N models 0.66 w N scores weights positive patches negative patches score log

14 Model Update 2 14 Model CNN 1 Score log Target scores t+1 t+2 t+t Avg Disable old model (maintain N latest models) CNN 1 β 1 CNN 2 CNN max 0.91 CNN 2 β 2 CNN 3 β 3 CNN N 0.66 positive patches negative patches 0.91 CNN N+1 CNN N Finetune β N min. edge score along the path β N+1 =min(β 3,0.91)

15 Model Update an Example Soccer sequence 15 node: CNN image: representative training sample Multiple CNN captures multi-modal appearances Tree structure isolates erroneous models in a branch

16 Experiment Dataset VOT2016, VOT2015 [Kristan2015], OTB50 [Wu2013] 16 Experiment settings Parameters fixed throughout the whole experiment Speed: 1.5 fps Intel Core i7 3.3Ghz with NVIDIA TITAN X MATLAB / MatConvNet

17 Results VOT2016 Overlap: 0.56, Failures: 0.83, Unsupervised overlap: 0.49 Red box: tracking result, Black box: ground-truth <fish4> Overlap=0.45, Failures=0.00 <iceskater2> Overlap=0.62, Failures= <motocross1> Overlap=0.51, Failures=0.00 <nature> Overlap=0.58, Failures=2.00

18 Results VOT2015 [Kristan2015] 18 Trackers *MDNet excluded Accuracy Robustness Expected overlap ratio Rank Score Rank Score DSST [Danelljan2014] MUSTer [Hong2015CVPR] MEEM [Zhang2014] Struck [Hare2011] RAJSSC [Zhang2015] NSAMF [Li2014] SC-EBT [Wang2014] spst [Hua2015] LDP [Lukežič2016] SRDCF [Danelljan2015ICCV] EBT [Zhu2016] DeepSRDCF [Danelljan2015ICCVW] TCNN (ours) Red: best, Yellow: second best

19 Results OTB50 [Wu2013] 19 TCNN (ours) compared to state-of-the-art trackers HCF [Ma2016], CNN-SVM [Hong2015ICML], MEEM [Zhang2014], SRDCF [Danelljan2015ICCV], DeepSRDCF [Danelljan2015ICCVW], MUSTer [Hong2015CVPR], FCNT [Wang2015], DSST [Danelljan2014], Struck [Hare2011]

20 Ablation study Single Single CNN Linear Models propagating in a linear chain Tree-mean Models propagating in a tree structure Equal weights for every model Tree-weighted Models and weights propagating in a tree structure Results on OTB50 Variations Precision Success AUC Single Linear Tree-mean Tree-weighted

21 Comparison with MDNet Results on VOT2016 Results on OTB50 21 Trackers Accuracy Robustness MDNet-N Trackers Precision Success AUC MDNet-N TCNN (ours) TCNN (ours) MDNet MDNet MDNet-N: MDNet without pretraining using tracking videos

22 Conclusion Multiple CNNs are helpful to achieve multi-modality and reliability of target appearances. 22 The CNNs and their weights propagating in a tree structure efficiently isolates error in a temporal manner. Please refer to our arxiv paper for further details.

23 23 Thank you.