Functionalities & Applications. D. M. Gavrila (UvA) and E. Jansen (TNO)

Transcription

1 Functionalities & Applications D. M. Gavrila (UvA) and E. Jansen (TNO)

2 Algorithms, Functionalities and Applications Algorithms (Methods) Functionalities (Application building blocks) Applications (Systems)

3 The ADABTS scenario Dynamic and noisy environments involving Varying visual/audio backgrounds Illumination changes, specularities Reverberating surfaces Multiple persons in close proximity (group / crowd), occlusions, potential for wrong assignments Challenging sensor analysis conditions Multiple (overlapping) cameras and microphone array technology

4 Overview Functionalities Video: Multi-Person Detection and Tracking Video: Human Body Orientation Estimation Video: Anomaly Detection (with or without Tracks) Audio: Sound Classification Audio/Video: Visualizations ( Basic, Advanced ) Audio/Video: Action Recognition (Aggression)

5 Video: Multi-Person Detection and Tracking Videoclip: FOI Camera overlays and top view (Green estimated track. Red annotated ground truth)

6 Video: Multi-Person Detection and Tracking Videoclip: BAE Systems Top view and image overlay (four overlapping cameras)

7 Video: Multi-Person Detection and Tracking By combining person detections in the individual cameras in 3D world coordinates, it is possible to keep track of targets as they move in and out of view of the individual cameras, providing more reliable long-term tracks. A system using overlapping cameras could select viewpoint automatically based on visibility considerations (body, head area) Zoom capabilities can be added (PTZ camera or image-based) Functionality stands at the basis for more advanced analysis (body orientation estimation / trajectory analysis / anomaly detection) BAE Systems ptrack System Novel multi-person detection and tracking from 4 overlapping cameras Threat Alerts generated from track processing : 1) Loitering near Sensitive Area; 2) Entering Sterile Zone System can perform reliably with people in a scene with high track accuracy. High frame rates achieved using a GPU Algorithms and software largely developed in ADABTS Bespoke hardware developed under BAE Systems PV funding

8 Video: Human Body Orientation Estimation Estimation of overall body orientation using overlapping cameras

9 Video: Human Body Orientation Estimation Estimation of overall body orientation using overlapping cameras Videoclip: UvA

10 Video: Human Body Orientation Estimation Establish person identity: Automatically establish in which camera frames and viewpoints faces are best viewed (display vs. automatic face recognition) Automatically detect focus of interest (e.g. bystanders observe a fight) Automatically detect person interaction (e.g. mark me all the persons that this person interacted with in the last 5 minutes ). Persons of interest could be chosen by operator or be system-triggered.

11 Video: Trajectory Analysis / Anomaly Detection All tracks Typical tracks Anomalous tracks Based on track data of persons in the scene, the system automatically determines typical and anomalous movement patterns, without need for additional user input.

12 Video: Trajectory Analysis / Anomaly Detection The system can determine and visualize typical movement patterns of persons in a scene (this could be extended with interaction patterns) It can furthermore detect and visualize the movement patterns that are anomalous. The future system could feature manual or automatic camera viewpoint selection to inspect the results, and zoom-in capabilities. This functionality could be useful for detecting a wide range of anomalous activities: pickpockets, terrorism reconnaissance, panhandling, fights, calamities etc.

13 Video: Track-less Anomaly Detection Automatically detect fighting and other anomalous events in video Not dependent on person detection and tracking Based on a single camera view Low requirements on camera calibration accuracy

14 Video: Track-less Anomaly Detection Videoclip: FOI Color: instantaneouos anomaly/fight score Graph: anomaly/fight variations over time

15 Audio/Video: Visualization Basic Localization, Visualizes operator where sound sources are on a plane. This helps to select best camera (angle) to monitor the scene. High noise level Low noise level Enhancement of sound, increased signal-to-noise ratios allows for eavesdropping: listening to the acoustic beam in a user-defined specific direction. Allows the operator to potentially better understand what is being said/shouted in the scene.

16 Audio/Video: Visualization Basic Videoclip: TNO

17 Audio: Sound Classification Real-time detecting and labeling anomalies (abnormal events) in the audio signals of a microphone array. Determination which type of sound is emitted from which location. System highlights areas in which events are detected. Selection of sound events: screaming, glass breaking and gun shots.

18 Audio: Sound Classification Videoclip: TNO

19 Audio: Sound Classification Immediately draws the attention of the operator to the right area in the right screen with right camera. PTZ camera could automatically and directly zoom in on the action.

20 Audio/Video: Visualization Basic Videoclip: TNO Advanced Videoclip: UvA Overlay microphone array on single camera image Based on multi-person tracking: Assign sound sources to persons given overlapping cameras. Temporal audio volume analysis for person tracks.

21 Audio/Video: Visualization Advanced Another example Videoclip: UvA

22 Audio/Video: Visualization Advanced System can detect and highlight persons based on their audio volume characteristics over a certain time period (e.g. give me all the loud persons within the last 5 minutes). A future system could add manual/automatic camera viewpoint selection and zoom-in capabilities, to inspect the results.

23 Audio/Video: Action Recognition (Aggression) Videoclip: UvA A system that automatically detects aggressive behavior based on audio and video analysis in overlapping cameras.

24 Audio/Video: Action Recognition (Aggression) A system that automatically detects aggressive behavior based on audio and video analysis. System output could be either a re-ranking of camera feeds presented to operator, or a visual or acoustical characterization of alert level.

25 Summary of Functionalities Video: Multi-Person Detection and Tracking Video: Human Body Orientation Estimation Wewantyourfeedback: How useful are these functionalities for you? Video: Anomaly Detection (with or without Tracks) Please participate in the ADABTS end-user survey! Audio: Sound Classification Audio/Video: Visualizations ( Basic, Advanced ) Audio/Video: Action Recognition (Aggression)

26 Further Reading Video: Multi-Person Detection and Tracking Video: Anomaly Detection Video: Body Orientation Estimation Audio: Sound Classification Audio/Video: Visualizations Audio/Video: Action Recognition (Aggression) ADABTS Deliverable 5.3 M. Liem and D. M. Gavrila. A comparative study on multi-person tracking using overlapping cameras. Proc. of the Int. Conf. on Computer Vision Systems, St.Petersburg, Russia, ADABTS Deliverable 5.3 J. Kooij, G. Englebienne and D.M. Gavrila. A Non-parametric Hierarchical Model to Discover Behavior Dynamics from Tracks. Proc. of the European Conference on Computer Vision, vol. 6, pp , Florence, Italy, ADABTS Deliverable 5.3 M. Liem and D. M. Gavrila. Person Appearance Modeling and Orientation Estimation using Spherical Harmonics. Proc. of the IEEE International Conference on Automatic Face & Gesture, Shanghai, China, 2013 ADABTS Deliverable 5.2 ADABTS Deliverable 5.3 ADABTS future publication

27 Thank you! October, 2010 June, 2012