Event Sensing on Distributed Video Sensor Networks Edward Chang Associate Professor Department of Electrical and Computer Engineering University of California
Collaborator Prof. Yuan-Fang Wang Industry Partners Proximex ArgusSenses Intel Students Acknowledgement Ankur Jain (lead), Diana Han, Dan Koppel, Kyle Kakligian, Arun Qamra, David Trainor, Tomy Tsai
Berkeley smart dust project Stanford multi-camera array
Desired Features [refs #1,7,8,12] Independent Cameras Multiple Cameras Static, Wired Cameras Mobile, Wireless, PTZ cameras Manual Event Recognition Automatic or Semi-automatic Event Recognition Tape-based Storage RAID-based Storage No Protocol Issues Smart Semantic-based Protocols
Potential Applications and Needs Applications Emergency search and rescue in natural disaster Deterrence of cross-border illegal activities Reconnaissance and intelligence gathering in digital battlefields Needs Rapid deployment, dynamic configuration, and continuous operations Robust and real-time data fusion and analysis Intelligent event modeling and recognition
advertisement of objectvideo.com Wide-area Surveillance
Surveillance Scenarios (1) (1) Intrusion Detection (2) (2) Passenger Screening Monitor and alert on tailgating, loitering, exit/closed entry, other unauthorized access (3) (3) Perimeter Monitoring Use biometric facial recognition to identify individuals of interest through existing closed circuit TV surveillance (4) (4) Unattended Baggage Z Z Object tracking and biometric facial recognition to determine vehicles and humans exhibiting suspicious behavior Copyright 2004 Proximex Corp. Identify unattended baggage (or other objects) left for long periods of time
Intrusion Detection Loitering, Tailgating 2 Raise low severity alert!! 0 Secure entrance 1 Suspect loiters X seconds, Loitering pattern recognized 8 Correlate alerts and raise critical severity alert if if it s it s the same person! 3 Capture high resolution facial image & soft biometric data 6 Suspect tailgates authorized person in in to to secure area. System captures suspect s high resolution facial image & soft biometric data 7 Raise medium severity alert!! 4 Match watch-list 5 Raise critical severity alert if if there s a match!! Copyright 2004 Proximex Corp.
Problem Statement Video Surveillance with Multiple cameras Mobile, wireless networks Intelligent content analysis Real-time query processing Focus of Our Current Work Event sensing including Motion detection Data representation, and Event recognition Sensor network management for Bandwidth and power resource conservation Dynamic routing
Research and Development Framework Event detection Far-field coordination and update Near-field sensor data fusion Event representation Hierarchical multiple levels of detail Invariant insensitive to incidental changes Event recognition Temporally correlated event signature Imbalanced training set
System Architecture s 1 s 2 Multi-camera Data Fusion s Multi-level level Sequence Descriptor Event Recognition..... s n! Security Alert
Validation Scenario p 1 t ) = ( x ( t ), y ( t )) ( 1 1 T z 1 x 1 y 1 Slave station y m Internet x m z m P ( t ) = ( X ( t ), Y ( t ), Z ( t )) T Z Master station z 2 y 2 Y X x 2
Event Detection: Coordination and Update [Refs #5, 6] Dual Kalman filters Minimizing Bandwidth and power consumption under pre-specified accuracy constraints Update necessary only when predications diverge Cache dynamic algorithms instead of static data
Event Detection: Sensor Data Fusion [Refs #2, 9] Sensing coordination and intelligent data fusion Two-level hierarchy of Kalman filter Top level (feed backward) Fill in missing or occluded trajectory pieces Camera pose & frame rate control z x (0) (0) () t = T (0) x (0) real world p = p & p&& (0) (0) (0) X Internet Bottom level (feed forward) Summarize trajectories in local state vectors X = T Merge state vectors from multiple cameras through registration parameters Slave station world real z ( i) () t Master fusion station ( 0) x P X= P& P&& (0) X= T Slave station x ( i) p = p & p&& ( i) ( i) ( i) z ( m 1) x () t ( m 1) world real x ( m 1) = T ( m 1 ) real ( m 1) x ( p ( = p & ( p&& world ( m 1) X Slave station m 1) m 1) m 1)
Sequence Data Learning [Ref #3] Sequence Sequence Representation Numeric-valued Representation Symbolic-valued Representation Discrete Fourier Transform Wavelet Piece-wise Linear Singular Value Decomposition Natural Language Strings
Requirements Different Event Queries of Different Granularities Coarse: temporal pattern query Fine: exact match Same Events under Different Representations Circling behavior in daytime (normal) Circling behavior with abrupt turn (suspicious) Circling behavior during midnight (suspicious) Multi-level Representation Select proper resolutions in a concept and context dependent way
Kernel Design Steps 1. Design primary kernel based on symbolic description 2. Design secondary kernels based on multi-attribute vector description 3. Fuse kernels
Kernel Fusion Definition x: input vector; y: event label {K 1,,K D } be a set of D kernels for D representations K d is a (dis)similarity matrix of the training instances K d should be positive semi-definite! Kernel Fusion K fusion (x,y)= F(K 1,,K D ) F: Tensor product F: Weighted sum F: Non-linear super kernel
Event Recognition: Imbalanced Data Set [Refs #4, 10] Negative samples significantly outnumber positive samples Bayesian risk associated with false negative significantly outweighs false positive Adaptive conformal mapping at decision boundary
Sensor Network Management [Ref #8] Bit-per-joule metric Multi-hop routing Enclosure algorithm for dynamic neighbor discovery
Experimental Results: Semantic Indexing [Refs #9, 11]
Experimental Results: Statistical Learning
Concluding Remarks Multi-sensor registration and fusion Hierarchical and invariant event representation Sequence data learning using imbalanced data set Mobile camera network bandwidth and power consumption management
References 1. A collaborative Camera System for Surveillance, A. Jain, Yuan-Fang Wang, E. Y. Chang, et al., UCSB Technical Report, November 2004. 2. The Anatomy of a Multi-camera Security Surveillance System, L. Jiao, Y. Wu, G. Wu, E. Y. Chang, Y.-F. Wang, ACM Multimedia System Journal Special Issue, Volume 10, Number 2, October 2004. 3. Distance Function Design and Fusion for Sequence Data, Y. Wu and E. Y. Chang, ACM International Conference on Information and Knowledge Management (CIKM), DC, November 2004 (19% accepted). 4. Aligning Boundary in Kernel Space for Learning Imbalanced Dataset, G. Wu and E. Y. Chang, IEEE International Conference on Data Mining (ICDM), United Kingdom, November 2004 (39/452 or 9% accepted). 5. Adaptive Sampling for Sensor Networks, A. Jain and E. Y. Chang, International Workshop on Data Management for Sensor Networks in conjunction with VLDB, Toronto, August 2004. 6. Adaptive Stream Resource Management Using Kalman Filters, A. Jain, E. Y. Chang, and Y.-F. Wang, ACM International Conference on Management of Data (SIGMOD), pp.11-22, Paris, June 2004 (16% accepted). 7. Toward Building a Robust and Intelligent Video Surveillance System: A Case Study (invited paper), E. Y. Chang, Y.-F. Wang, and I-J. Wang, IEEE International Conference on Multimedia, Taipei, June 2004. 8. Distributed Video Data Fusion and Mining, E.Y. Chang, Y.-F. Wang, and Volkan Rodoplu, SPIE Conf. on Defense and Security --- Sensors, and Command, Control, Communications, and Intelligence Technologies for Homeland Security and Homeland Defense (co-chaired by DARPA, Dod/Doj), Orlando, April 2004. 9. Multi-camera Spatio-temporal Fusion and Biased Sequence-data Learning for Security Surveillance, G. Wu, Y. Wu, L. Jiao, Y.-F. Wang, and E. Y. Chang, ACM International Conference on Multimedia, pp. 528-538, Berkeley, November 2003 (17% accepted). 10. Adaptive Feature-Space Conformal Transformation for Imbalanced-Data Learning, G. Wu and E.Y. Chang, The Twentieth International Conference on Machine Learning (ICML), pp.816-823, Washington DC, August 2003. 11. Invariant Feature Extraction and Biased Statistical Inference for Video Surveillance, Y. Wu, L. Jiao, G. Wu, E. Y. Chang, and Y.-F. Wang, IEEE International Conference on Advanced Video and Signal Based Surveillance, pp.284-289, Miami, July 2003. 12. A Framework for Detecting Hazardous Events, Y. Wu, G. Wu, and E.Y. Chang, IS&T/SPIE International Conference on Storage and Retrieval for Media Databases, San Jose, January 2003. 13. More on http://www.db.stanford.edu/~echang/