Observing people with multiple cameras

Transcription

1 First Short Spring School on Surveillance (S 4 ) May 17-19, 2011 Modena,Italy Course Material Observing people with multiple cameras Andrea Cavallaro Queen Mary University, London (UK)

2 Observing people with multiple cameras Andrea Cavallaro youtube.com/smartcameras videotracking.org Topics covered Target detection and tracking with multiple cameras Fusion and behaviour recognition Calibration Content-based view selection Keywords Information filtering; metadata; Calibration; Tracking; Fusion; Behaviour understanding. 1

3 Resources Acknowledgements Emilio Maggio Murtaza Taj Gabin Kayumbi Nadeem Anjum Matteo Bregonzio Stefan Karlsson Fahad Daniyal Huiyu Zhou Fabio Poiesi Tahir Nawaz EU FP7 project APIDIS UK EPSRC project MOTINAS 2

4 Motivation Dramatic increase in the scale of camera networks Large-scale applications that can observe detect track events of interest across large areas Why now? technological advances in sensor design communication computing reduction of cost of image sensors supporting network infrastructure Deployment example: surveillance 3

5 Sport: crowds, occlusions, fast changes, clutter! Excellent testbed for surveillance algorithms! Synchronisation External trigger based synchronisation Trigger generated through NI PCI-6601 trigger board Each camera captures at exactly the same time Ethernet Data cables Signal replicator Trigger board Synchronization signal 4

6 Synchronisation Visio-Box for synchronised acquisition Each audio-visual data stream is captured by a separate processor Audio amplifier Video Audio Scenario 5

7 What for? multi-camera processing 6

8 Context Task Cameras signal processing attributes pattern recognition Content production camera selection content summaries perceptual coding Scene understanding behaviours interactions identities a priori information and info from other cameras a priori information and info from other cameras a priori information and info from other cameras input video pre-filtering object detection postprocessing symbols event detection 3D analysis tracking and classification a priori information and info from other cameras a priori information and info from other cameras a priori information and info from other cameras 7

9 (Visual) Information filtering Objective To extract relevant information from a scene (through images) with an internal/external processing unit with the help of other cameras (or sensors) The format of the output format may be different from the signal that was captured by the sensor (transmoding) From visual information to knowledge Objective to extract the main content message e.g., automatic video object segmentation (detection & tracking) classification of the pixels in two classes: foreground + background pixels or: patches, feature points, edges,... Use a priori information Application dependent Alternative probabilistic description of the observations, supported by time and/or multi-camera integration 8

10 From detection to description Detection (i.e., how to find relevant information) motion classification change detection object classification Tracking (i.e., how to propagate relevant information) over time (in the same camera) across cameras despite occlusions despite multiple simultaneous objects despite local and global changes in illumination Object and scene description (compact & unambiguous) communication storage Multi-camera processing: 2 approaches First fuse, then track Data fusion object tracking First track, then fuse object tracking object tracking Data fusion object tracking 9

11 Video tracking: approximations Annotation of a video in terms of its component objects to localize objects of interest to link instances of the same object over time (tracking) Typical object approximations for tracking Polygonal approximation Bounding ellipse Bounding box Position only Challenges Occlusions Illumination (shadows) Over-segmentation 10

12 Tracking: challenges Track management issues Track initiation Track termination Occlusions handling Partial / total occlusions Spawning and splitting / merging Target model update (drift problem) What makes tracking difficult? Noisy observations Changes in object pose / scale Changes in scene illumination Other objects and background with similar appearance (clutter) Tracking: approaches (1/2) Feature-based tracking Track parts of objects (features) corner / feature point detection and pattern matching Problems low-resolution targets feature grouping Contour-based tracking Track object boundaries Full boundaries (e.g., snakes) Partial boundaries (e.g., omega-shaped edges for heads) Problems partial occlusions (handled e.g. with Kalman-snakes) 11

13 Tracking: approaches (2/2) Region-based tracking Track blobs which roughly correspond to objects Problem partial occlusions Model-based tracking A priori knowledge of the shape Problems computationally intensive need a large set of (3D) models Object state Estimate the state x k from the observations z 1:k x k is a vector of object parameters (position, velocity, shape, etc.) z k is k the frame at time k (appearance-based) the output of an object detector (detection-based) State affine transformation parameters of a patch w.r.t. the first image parameters of a shape parameters of the contour speed and acceleration parameters describing target appearance 12

14 Target model: template initialization later in the sequence... model I T Score function: Α x, w I w d( x) IT 2 w Target model: template Target template stores luminance (color) values + their location translational model (or more complex motion models) Advantages: simple and computationally fast Disadvantages: over time becomes non-representative of the object appearance (noise, partial occlusions, pose or scale changes) Improved templates temporal update model each pixel in the template as a mixture of Gaussians that is updated over time include initial template to reduce drifting evolve based on a modified Kalman filter robustness to illumination variations phase of wavelet coefficients (instead of color) 13

15 Improved model: histogram Observation The complete pixel information may be (is) redundant / misleading for the tracking task Target representation: desired properties descriptive enough to disambiguate object vs. background flexible enough to cope with changes of target scale pose scene illumination partial occlusions histograms Color histograms for tracking Color histograms invariance to scaling (normalized) rotation robustness to partial occlusions data reduction and efficient computation 14

16 Likelihood of a candidate (score) k th frame model ˆq candidates ˆp y, 1 u 1 d pˆ y qˆ pˆ y qˆ m u u Target representation: template vs. color histogram Pixel-based representation (template) data reduction No Yes Statistical representation (normalized color distribution) rotation & size invariance No Yes flexibility No Yes spatial information Yes No (*) If correctly updated is more descriptive 15

17 Integrating detection and tracking Tracking propagates the initialization information model: template, statistical representation, parts, should update the model should self-initialize Approaches multiple single-target trackers initialized with a detector/classifier Detect, model, search Birth and death governed by heuristics Usually poor clutter handling Associate detections over time (data association filters) Nearest neighbour filter, JPDAF, MHT, Data association filters Objective to associate detections Approaches Nearest Neighbour (NN) filter one hypothesis per trajectory Joint Probabilistic Data Association Filter (JPDAF) evaluate all associations between two frames handle clutter complexity: exponential with number of targets Multiple Hypotheses Tracker (MHT) evaluate hypotheses over multiple frames complexity: exponential with time and with number of targets gating to reduce computational cost extending the Bayes recursion to multiple targets (using Random Finite Sets) 16

18 Extending the Bayes recursion to multiple targets PHD filter for multi-target tracking Filter the detections particularly effective in removing clutter Linear complexity with the number of targets Incorporates scene contextual information Learning scene context for multiple object tracking E. Maggio and A. Cavallaro IEEE Trans. on Image Processing, August 2009, pp Efficient multi-target visual tracking using Random Finite Sets E. Maggio, M. Taj, A. Cavallaro IEEE Trans. on Circuits and Systems for Video Technology, August 2008, pp How can I evaluate and compare (easily!) my results? 17

19 PFT: a Protocol for Evaluating video Trackers Set of trials Robustness of trackers to initialization perturbations, noise, frame dropping, illumination changes Evaluation measure (AUC λ ) Overlap measure Lost track ratio Dataset PETS00 PETS10 T. Nawaz, A. Cavallaro, PFT: a Protocol for Evaluating video Trackers, ICIP 2011 Software, data and results available at Clemson SPEVI Surveillance Performance EValuation Initiative SPEVI Surveillance Performance EValuation Initiative One-stop web site collecting existing datasets Pointers to other evaluation programmes / datasets Server hosting received datasets and ground-truth Free distribution (for research) Requested citation acknowledgment If you want to contribute to this collection of datasets please contact info@spevi.org 18

20 Approach 1: track and then fuse Approach 2: fuse and then track Trajectory association and fusion on ground plane Object detection and tracking Information fusion Object detection and tracking Object detection and tracking 19

21 A simple example: 2 cameras Global trajectory reconstruction 1:N trajectory matching Normalized cross correlation C 1 Detection and tracking Ground plane projection Feature extraction Feature matching Complete trajectory C 2 Ground plane projection Directional distance Trajectory shape Average velocity Dominant turn angles PCA components 20

22 Global trajectory reconstruction Detection & tracking Ground-plane projection Feature extraction Trajectory of interest Ground-plane matching Candidates Detection & tracking Ground-plane projection Feature extraction Image-plane verification Association Trajectory association and fusion across multiple partially overlapping cameras N. Anjum, A. Cavallaro IEEE AVSS 2009 [best paper award] Fusion Linkage Approach 1: track and then fuse Approach 2: fuse and then track 21

23 Track before detect Information fusion Object detection and tracking Multi-camera track-before-detect M. Taj, A. Cavallaro Proc. of ACM / IEEE ICDSC 2009 Multi-camera player tracking P1 P2 R1 R2 22

24 How about other sports? Trajectory association across multiple cameras Trajectory clustering feature feature extraction 1 extraction Mean-shift Mean-shift clustering clustering Y1 Y 2 cluster merging Y1 Y 2 cluster merging feature extraction M M Mean-shift clustering Y M cluster merging _ Y M cluster fusion common patterns cluster analysis outliers Multi-feature object trajectory clustering for video analysis N. Anjum, A. Cavallaro IEEE Trans. on Circuits and Systems for Video Technology, 18(11), Nov

25 Ball detection - challenges Occlusions partial total Appearance similarity player player player court ball - court ball - players Where is the ball? Estimation of the approximate location of the ball based on players behaviour Use of motion flow to estimate regions of convergence of players during attacks 24

26 Position estimation Detector-less ball position estimation where is the ball? Detector-less ball localisation using context and motion flow analysis, F. Poiesi, F. Daniyal, A. Cavallaro ICIP

27 View selection c 1 c N 1 I t N I t motion detection motion detection 1 d t N d t object detection and tracking 1 X t N X t object score object score 1 t N t content ranking and camera selection c ~ t N X 1 t feature extraction feature extraction 1 t N t event detection event detection 1 t N t View selection / video ranking View selection given n views of an object m objects in the scene n views of m objects in the scene Applications Video summarization for multiple-video streams Camera scheduling based on automatic rating of content [demo video] Quality of view: components Location Orientation Visibility Motion path Event Bag 26

28 View selection Tracking - example 1 Production example A Production - example with audio Surveillance: generic pipeline camera calibration global trajectory reconstruction trajectory clustering common patterns outliers 27

29 Surveillance: generic pipeline camera calibration global trajectory reconstruction trajectory clustering common patterns outliers Localisation of cameras Generic pipeline camera calibration global trajectory reconstruction trajectory clustering common patterns outliers Localisation of cameras Complete trajectory 28

30 Surveillance: generic pipeline camera calibration global trajectory reconstruction trajectory clustering common patterns outliers Localisation of cameras Complete trajectory Clustering of trajectories Open challenges 29

31 Calibration and topology Camera network layout non-overlapping fields of view interpreting the dynamics of moving objects in wide areas partially overlapping or overlapping fields of view observing objects from different view-points (3D interpretation) to disambiguate occlusions Calibration image to 3D space correspondence (points) relative location D i and orientation Φ i of the cameras Intrinsic calibration relationship between camera-centric and image coordinates camera matrix Calibration of a camera network Camera network layout Partially overlapping Non-overlapping 30

32 Calibration methods Small scale (overlapping fields of view) multi-view geometry epipolar geometry projective transformations feature detection and matching Large scale (partially/non overlapping fields of view) observations of an object simultaneous calibration and tracking simultaneous calibration and synchronization Non-overlapping cameras How to localize randomly placed cameras? non-overlapping field of view no intrinsic calibration top-down view optical axis is perpendicular to the ground plane, or removed perspective deformation Approach Learn regression parameters from available trajectories Extrapolate trajectory data to unobserved regions Kalman filtering on available and estimated trajectory estimated trajectory observed moving target FOV 31

33 Example: estimation result Automated localization of a camera network N. Anjum, A. Cavallaro IEEE Intelligent Systems, (to appear) 2011 Multi-camera processing: architecture Centralized Decentralized Distributed Distributed and decentralized multi-camera tracking M. Taj, A. Cavallaro IEEE Signal Processing Magazine, Vol. 28, Issue 3, May

34 Multi-camera processing: crowded scenes Ref: TREC Video Retrieval Evaluation 2008 Dataset description - scenario 33

35 Research outlook / interesting topics What s next? Algorithms learning a model (on-line) / model update probabilistic description of the observations, supported by multi-camera integration Sensors heterogeneous sensors (video is good, but not the only modality!) Applications embedded systems large-scale behavior recognition, detection of unusual events Datasets tracking on real sequences (crowds, UAVs, ) + evaluation! To play with OpenCV library Image Processing & Computer Vision library Camellia Hybrid tracker 3D, pattern recognition, tracking, Other links Andrea Cavallaro Queen Mary University of London 34