DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats

Transcription

1 DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats Sponsor: Air Force Office of Scientific Research FA Program Manager: Dr. Erik Blasch PIs: Young-Jun Son 1, Jian Liu 1, Jyh-Ming Lien 2 Students: S. Lee 1, Y. Yuan 1, H. Na 1, H. Yang 1 1 Systems and Industrial Engineering, University of Arizona 2 Computer Science, George Mason University PI Contact: son@sie.arizona.edu; AFOSR DDDAS PI Meeting Sep. 7 th,

2 Agenda Problem Definition and Modeling Framework Model Description and Implementation via Sensors Systems Software 2

3 Border Surveillance and 3-Level Framework High altitude level (HAL) Low altitude level (LAL) Surface level (SL) 3

4 Major Challenges in Border Surveillance 3-D Surveillance System for aerial and ground targets Effective detection, recognition and identification of targets Heterogeneous data from complex targets by 3 levels of sensors Multi-level information aggregation Active or pro-active surveillance strategies Realistic scenarios and model validation based on data collection from our research partners (AFRL, Raytheon, University Partners ) 4

5 3-Level Measurement System in Border Surveillance 3-Levels Sensors Measurement Data EO/IR Image High altitude level (HAL) Low altitude level (LAL) SAR Image Spectral Image Lidar Image Thermal Images Surface level (SL) Magnetic Data 5

6 LiDAR Data Processing A Velodyne HDL64 LiDAR integrated with a golf cart for outdoor scanning A Velodyne HDL32 LiDAR integrated with a powered wheelchair for indoor scanning Registered LiDAR data captured on George Mason Campus LiDAR+Photos = Better Localization [Arsalan, Kosecka, Lien, ICRA 2015] 6

7 DDDAMS Framework Group Prediction Group Prediction 7

8 Agenda Problem Definition and Modeling Framework Model Description and Implementation via Sensors Systems Software 8

9 DDDAMS Framework : Target DRI (Detection, Recognition, Identification) Group Prediction Group Prediction 9

10 Target DRI (1) : Detection Goal: Discover the presence of a person, object, or phenomenon Sensing technologies: Cameras (UAV, UGV), Seismic Sensors Algorithms: Motion detection, Control Charts * Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Group Prediction 10

11 Target DRI (2) : Recognition Goal: Determination of the nature of a detected person, object or phenomenon, and its class or type * Algorithms: Wavelet decomposition, Classification * Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Group Prediction 11

12 Target DRI (3) : Identification Goal: Discrimination between recognizable objects as being friendly or enemy * Algorithms: Information-aggregation method, Extended BDI (Belief-Desire-Intention) framework * Military, U. S. (2005). Dictionary of military and associated terms. US Department of Defense. Group Prediction 12

13 Crowd Detection Module UAV Goal: Moving Target Detection via Sliding Window I t I t+ t I t+2 t S1. Feature extraction: Good features to track K t K t+ t K t+2 t S2. Feature tracking: Optical flow. S3. Image registration: RANSAC, Homography (T) I t+ t (T) I t+2 t S4. Background elimination: Absolute differences (T) I t+2 t (T) I t+ t S5. Targets segmentation: Motion history, Dilation-erosion (a) (b) (c) Minaeian, S., Liu, J., & Son, Y. J. (2016). Vision-Based Target Detection and Localization via a Team of Cooperative UAV and UGVs. Systems, Man, and Cybernetics: Systems, IEEE Transactions on, 46(7), Minaeian, S., Liu, J., & Son, Y.-J. (2017). Effective and Efficient Detection of Moving Targets from a UAV s Camera. Submitted to Intelligent Transportation Systems, IEEE Transaction on, (Under Review). 13

14 Individual Detection/Recognition Module UGV Goal: HOG (Histogram Oriented Gradient) based Target D/R Gradient computation : 3x3 derivative mask [-1, 0, 1] Orientation binning : Weighted voting over cells 6x6 pixel cells HOG over des. blocks : Grouping cells into blocks 3x3 cell blocks Blocks Block normalization : L2-norm Cells Classify the target : OpenCV classifier Minaeian, S., Liu, J., & Son, Y.-J. (2015, January). Crowd Detection and Localization Using a Team of Cooperative UAV/UGVs. In IISE Annual Conference. Proceedings: Institute of Industrial Engineers-Publisher. 14

15 Detection Ground Seismic Sensor Goal: Target Detection based on Control Charts Detect presence of an object (mv) Normal (mv) Target Appearance Detected (sec) (sec) Geophone Amplifier Gateway Software System Components: 15

16 Feature Extraction Methods: Fisher s Discriminant Analysis Recognition Ground Seismic Sensor Goal: Target s Characteristics Recognition (e.g., walking/running) Challenge: Raw data are noisy and mathematically inseparable Training Data Knowledge Wavelet Coefficients C = C run C walk X t = Φ run t C run + Φ walk t C walk Model: Wavelet decomposition Regression New Observation SVM Classifier 16

17 Identification Behavior Models Goal: Target identification via behavior models of drug traffickers and ground patrol agents under varying environmental conditions Behavioral Models of Drug Traffickers Decision 1 Decision 2 Decision 3 Selection at Departure Time Location Route Choice Fastest Way Rugged Road En-Route Planning Hiding/Avoiding Sudden Direction Change Hiding Direction Change 17

18 Extended Belief-Desire-Intention (EBDI) Framework Bratman, 1987 Rao and Georgeff, 1998 Zhao and Son, 2008 Lee, S., Y.-J. Son, and J. Jin (2010), Integrated human decision making and planning model under extended belief-desire-intention framework, ACM Transactions on Modeling and Computer Simulation, 20(4), 23(1)~23(24). 18

19 Identification Fusion of Sensor Data Goal: Target s Identification (e.g., Friend/Foe) Challenge: Fusion of different sensor data (UAV vision and seismic) 19

20 Superhuman Vision via Information Fusion Wide range visibility Border patrol UAV Motion control Aerostat occluded subjects DDDAS in border patrol s computing unit Task: Augment patroller agent s vision system with real-time imagery data collected by UAVs and the patrol agent Inputs: (1) Low-resolution but less-occluded image/3d data from UAV (2) high-resolution but much-occluded captured by the border patrol Output: Fused and registered data captured by the UAV to images captured by the border patrol agents Method: The key in enhancing the border patrol s vision is in finding the correspondences between the features extracted from the images captured by the UAV and the patrol agents and identifying occluded objects in patrol agents view. 20

21 Agenda Problem Definition and Modeling Framework Model Description and Implementation via Sensors Systems Software 21

22 Agent-based Hardware-in-the-loop Simulation Agent-based Simulation Repast Simphony with 3D GIS Sensory Data Assembled UAV (e.g. GPS) (APM:Copter / Arducopter) Hardware Interface: MAVproxy Control Commands (MAVLink Messages) Assembled UGV (APM:Rover / Ardurover) Wi-Fi / XBee PRO 900HP; APM one 22 Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. -J., & Liu, J. (2013). Agent- based hardware-in-the-loop simulation for modeling UAV/UGV surveillance and crowd control system. In Proceedings of the winter simulation conference 2013, Washington, DC, USA.

23 Physics Based Simulation for Data Generation Real Pictures from UAV Generated Pictures from Simulation 23

24 Physics Based Simulation for Data Generation Goal: Visionary data generation using various simulation objects. Ground Patrol UGV UAV (fly low) UAV (fly high) 24

25 Detection module (Simulated) UAV Goals: Applying detection algorithm using data from simulated UAV. 25

26 Detection module (Simulated) UGV Goal: Applying HOG detection algorithm to simulated data from UGV. 26

27 DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats Son and Liu, Arizona; Lien, George Mason Summary of Effort AF relevance to autonomous systems; collaborative/cooperative control; sensor-based processing; multi-scale simulation technologies; cognitive modeling Key Focus of Scientific Research Active or pro-active border surveillance strategies Multi-level information aggregation involving heterogeneous data from 3 levels of sensors Handling latency in detection, recognition, & identification of targets Other performers on project Y. Son (UA), J. Liu (UA), J. Lien (George Mason) Students: S. Lee, Y. Yuan, H. Na, H. Yang 27

28 DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats Son and Liu, Arizona; Lien, George Mason Accomplishments Developed/refined a DDDAMS-framework for a 3 level border surveillance system Developed preliminary models and algorithms for target DRI, group prediction, and mission control Hardware in the loop simulation (agent; physics simulation) Awards (PhD graduate at U of Arizona) Dr. Sara Minaeian, August 2017 (Joined Siemens) (Best paper award in Service and Work Systems track) S. Lee (PhD student) and Y. Son, Extending Decision Field Theory to a Multi-agent Decision-making with Forgetting, Proceedings of 2016 IISE Annual Meeting, Anaheim 28

29 DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats Son and Liu, Arizona; Lien, George Mason Reporting Minaeian, S., Liu, J., & Son, Y.-J. (2016). Vision-Based Target Detection and Localization via a Team of Cooperative UAV and UGVs. Systems, Man, and Cybernetics: Systems, IEEE Transactions on, 46(7), Minaeian, S., Liu, J., & Son, Y.-J. Effective and Efficient Detection of Moving Targets from a UAV s Camera. Submitted to Intelligent Transportation Systems, IEEE Transaction on, Special Issue on Robust & Efficient Vision Techniques for Intelligent Vehicles, (Under Review) Minaeian, S., Liu, J., & Son, Y.-J. Effective and Efficient Multi-target Data Association via Dynamically Adjusted Affinity Scores, (to be submitted to a journal in September, 2017) Minaeian, S., Liu, J., & Son, Y.-J. (2016). Analysis of Network Communications between Cooperative Unmanned Vehicles for Autonomous Surveillance. In IIE Annual Conference, 2016 Proceedings. Institute of Industrial Engineers-Publisher. 29

30 DDDAMS-based Border Surveillance and Crowd Control via UVs and Aerostats Son and Liu, Arizona; Lien, George Mason Coordination/Synergy Exploring collaboration and access to aerostat data with Raytheon (no access yet) Discussed collaboration opportunities with Tathagata Mukherjee (Intelligent Robotics, Inc) at the 2017 PI meeting in Dayton, and will schedule follow-up meetings together with Eduardo Pasiliao at AFRL Exposure/Use by other groups Hardware-in-the-loop demos to visitors (STEM students, summer students, visitors from industry and other universities) 30

31 Acknowledgements Air Force Office of Scientific Research FA Program Manager: Dr. Erik Blasch PIs: Young-Jun Son 1, Jian Liu 1, Jyh-Ming Lien 2 Students: S. Lee 1, Y. Yuan 1, H. Na 1, and H. Yang 1 1 Systems and Industrial Engineering, University of Arizona 2 Computer Science, George Mason University PI Contacts: son@sie.arizona.edu; jianliu@ .arizona.edu; jmlien@cs.gmu.edu;