Haishan Wu School of Computer Science, Fudan University Shanghai, China

Transcription

1 1 Haishan Wu School of Computer Science, Fudan University Shanghai, China

2 When simulation meets empirical data Different sensors Computer vision techniques: pattern recognition, feature extraction, image processing, stereo reconstruction, machine learning Measurement data 2!

3 Related works on motion measurement methods On small groups Human, fish, ants, bees, cells, bacteria On large groups Bacterial swarms, Drosophila groups, bird flock, human crowd, bat flock There is no robust implementation publicly available for tracking large groups of targets yet. 3!

4 General Procedure Detecting each individual Tracking each individual in 2D or 3D throughout the whole data sequences Target Detection is a task-specific and challenging problem. Maintaining the identity individually is very difficult. (false positive detections, occlusions etc.). Computation speed should also be considered. Computer vision techniques that may be used: image segmentation, object detection, image denoising feature extraction Computer vision techniques that may be used: multi-view geometry, multi-target tracking, combinatorial optimization, feature extraction and matching 4!

5 Very useful references Four excellent books Multiple view geometry in computer vision Computer Vision: Algorithms and Applications the latest draft can be downloaded in Pattern Recognition and Machine Learning Multiple-target tracking with radar applications 5!

6 Multi-view geometry and stereo reconstruction Epipolar geometry Image rectification 6!

7 Framework for 3D tracking :1 Image sequences of multi-camera setup, often acquired by two or more CCD-cameras in a convergent arrangement Tracking of particles in 2D image space with different approaches (filter + data association) Stereoscopic correspondences problem, matching of the 2D tracks to generate particle trajectories in object space, requires system calibration and orientation 7!

8 Framework for 3D tracking: 2 Image sequences of multi-camera (more than two) setup Establishment of particle correspondences with consequent use of epipolar constraints and determination of 3D particle positions, treating each single time step separately, requires system calibration. Tracking is performed in 3D object space, one 3D point cloud as input for each time step 8!

9 Framework for 3D tracking :3 Establishment of particle correspondences with epipolar constraints and determination of 3D particle positions, in addition storage of all relevant data from particle detection process 9!

10 My work Drosophila group tracking in 3D Fish school tracking in 2D (I bring my poster here ) +,-./012345,60$(% %"" &"" '""!"" ("" #"" )"" $"" *"" 10!!"" #"" $"" %""" %&"" %!"" %#""

11 Tracking fruit fly group in 3D Why it is a challenging problem: Frequent occlusions make many tracking methods failed.! Tracking or matching failures will lead to the broken or incomplete trajectories in 3D space! Appearance feature will be also ineffectual for tracking or matching across various views! 11!

12 Tracking fruit fly group in 3D (cont.) By using two high-speed cameras, we solved the abovementioned problems by using one compact framework: linear assignment problem (LAP) 12

13 Tracking fruit fly group in 3D (cont.) The first LAP 2 steps: single particle state updating and data association State updating: alpha-beta or Kalman filter! 13!

14 Project 1: Tracking fruit fly group in 3D (cont.) The first LAP The first LAP is:! To handle these difficulties:! 14!

15 Tracking fruit fly group in 3D (cont.) The second LAP Now our problem is how to assign (match) a 2D trajectory in one view to that of another view The appearance feature of targets, however, is useless in our experiments, so how to define the matching cost? Observations: l Two true matching trajectories will submit to epipolar constrains l The longer the length of matched trajectories, the larger the possibility of correct matching is. 15!

16 Project 1: Tracking fruit fly group in 3D (cont.) The second LAP Our matching cost: MECL(maximum epipolar co-motion length)! 16!

17 Tracking fruit fly group in 3D (cont.) The third LAP 3D trajectories can be recovered by using multiview geometry technique. However, tracking errors make these trajectories broken into segments We solve this problem by formulating it as pairwise tracklet matching problem! 17!

18 Tracking fruit fly group in 3D (cont.) The third LAP We incorporate temporal and kinematic information into linking cost. l temporal information l kinematic information! 18!

19 Tracking fruit fly group in 3D (cont.) 19

20 Large scale fish school tracking (I bring the poster here ) 20!

21 Acknowledgements Thank Prof. Tamas Vicsek for offering me the opportunity to give a talk here. Many thanks also go to Zsuzsa Ákos. Thank Qiyuan Tian, Qi Xu, Yihao Zhou, Miaohui Wang, Ye Liu, Jinlong Shi, Iain Couzin for their valuable comments in Drosophila group tracking. Thank Linguo Li and Wei Li at School of Life Sciences of Fudan University for providing fruit flies for the experiments. Thank Simon Leblanc in Princeton University for providing the fish school videos 21!