07ATC 22 Image Exploitation System for Airborne Surveillance

Transcription

1 07ATC 22 Image Exploitation System for Airborne Surveillance Karunakaran. P, Frederick Mathews, S.H.Padmanabhan, T.K.Sateesh HCL Technologies, India Copyright 2007 SAE International ABSTRACT The operational design of Image exploitation system to process real time video and flight data acquired by an Unmanned Aerial Vehicle is presented. The system comprises of image processing tasks along with image processing applications embedded on vision processor board. Image exploitation makes use of image processing algorithms for information extraction using selective and adaptive techniques such as image enhancements, filtering, automatic target tracking, and computation of geo-reference location of target. The system operates in real time mode to display mission parameters on a multi display system. The results indicated the system is qualified to be used for surveillance applications. A sample set of the results are presented. INTRODUCTION Airborne surveillance has been widely used in different range of applications in civilian and military applications, such as search and rescue missions, border security, resource exploration, wildfire and oil spill detection, target tracking, surveillance, etc [1]. The unmanned airborne vehicle is equipped with special sensors (day/ night) to image objects in ground and assigns the actual recognition task (surveillance) to the crew or record image data and analyze them off-line on the ground. Pilot less airborne vehicle with sensors carrying platforms transmitting data to a ground control station for analysis and data interpretation. Description of Image exploitation system developed to acquire and process real time video with flight data of a pilot less vehicle while flying over a specified area is presented. The system is generic in nature for image processing algorithms and techniques essentially for any airborne vehicles. IMAGE EXPLOITATION Image exploitation is a process carried out by acquisition and processing of sensory information of a scene or targets for surveillance applications.this process involves processing large volume of image data acquired by multi sensor such as optical, infrared and radar data. For Information extraction and quick analysis the image data has to be processed using fast computational image processing algorithms and efficient embedded processor [2, 3]. This involves development of automated image exploitation tools such as selective and adaptive image enhancements,filtering,boundary detection of targets, feature extraction,target acquisition and tracking, identifying geo location of the target, Zooming,target area measurements, etc. The results of processing are stored in a mission data base for further analysis and interpretation by decision makers. Image exploitation system developed consists of hardware and exploitation software for processing aerial vehicle data. The hardware system consists of the following components: an Industrial PC, Frame grabber, embedded vision processor and a graphic card for multiple displays. The image exploitation system has built in tools with functional features to acquire, store, retrieve, process, analyze, interpret and display information from imagery during a vehicle mission.the data captured by the camera is transferred to work station and analyzed to provide imagery information. Imagery information is in the form of video clips, video frames and the corresponding flight data, the calculated location of the targets and related information.the extraction and exploitation of imagery intelligence from aerial surveillance enhances understanding and interpretation of scene contents, allows vehicle to see distant targets, and enhances surveillance capabilities. The block diagram of the image exploitation system is given in figure 1.

2 Figure 1 Block diagram of imaging system SYSTEM DEVELOPMENT The hardware architecture of Image exploitation system core consists of a Computing System (Industrial PC) with multiple imaging boards, frame grabber card, graphics card to support multiple displays, and network interface card along with supporting peripherals. The Host PC is connected to 3 Embedded Vision Processor (EVP) cards (256 MB RAM) with add-on modules and a Frame Grabber card (FG).The EVPs and the FG boards are interconnected through the on-board Auxiliary Bus to ensure data transfers are carried out independently of the PCI bus as given below in Figure 2. PC s Graphics card (Dual display 256 MB DDR memory) and the output of each embedded processor is viewed on separate monitors connected and controlled by graphics card on embedded processor board. Real-time imaging applications require high data bandwidth from the video source to the display output [4]. These systems also demand very low latencies from the processing system because a human operator is badly affected by any apparent lag between the input image and the presented image when in a moving platform (such as an aircraft). This means that the processing delays in the system need to be minimized and often kept below 12.5 fps (which is a typical frame time). Add to this further complex operations (Image processing algorithms) need to be applied to each and every pixel, perhaps several times. Real time Video image processing is realized by using the EVP, optimized image processing library and distributing the image processing functions to multiple EVP. Image acquisition and processing is done in parallel and the data goes to host as well as EVP. One such framework is to use the processing ability of multiple embedded processors where 80 ms is the required time to process 1 frame and 3 embedded boards are used for fast processing of images. Further an auxiliary bus with multicast mode allows images to be sent for concurrent processing on two or more embedded processors.the Software architecture of the system with application layer, hardware layer and interface layer is given in Figure 3 as below. Figure 2 Host Interface The Frame Grabber digitizes the incoming PAL analog video (color/grayscale) of 25 frames/sec (1 Frame 768x576) and transfers it (120 MB/sec) to Host PC via PCI bus or to embedded processor via dedicated data bus called AB (Auxiliary Bus). AB is a dedicated bus that makes transfer of video from Frame grabber to embedded processor at a very high speed (200 MB/sec) and bandwidth with out depending or affecting the Host PC. The output of the video capture and processing are viewed on the HOST PC s monitor through the HOST Figure 3 Software Architecture Diagram APPLICATION SOFTWARE Image exploitation system contains application software which extracts and exploits the information of aerial imagery obtained from onboard sensors mounted on UAV or other reconnaissance platforms. The output is obtained from the exploitation of aerial video imagery captured by camera mounted on UAV with Flight telemetry data.imagery intelligence is obtained by applying image processing techniques and algorithms

3 where target feature information is extracted automatically from real time video data for image analysis. The basic input to the application software is the flight video image and data captured by the unmanned mission. The system processes the images using Image processing application software which displays input and processed images for analysis. Some of the processing functions required by users are as follows: 1. Mission settings, acquisition of video flight data and image display. 2. Video Image analysis using Image Processing functions such as Enhancements, edge detection, filtering, zooming, etc. 3. User interaction and processing such as single frame and multiple frame target calculation, terrain measurements, area based retrieval, etc. 4. Real time target tracking from video images. 5. Real time Plotting of flight parameters (Health). 6. Creating,updating and retrieval of Imagery intelligence database 7. Printing and saving the image file data in a CD. METHODOLGY The methodology is based on the development of fully automated decision criteria tools [3, 5 and 6] for feature information extraction from video images which involves various image processing algorithms such as: 1. Video Image enhancement techniques for selective (linear and non-linear enhancements) and automatic enhancements to search for a target region in the video imagery. In selective mode of enhancement user has an option to select standard enhancement techniques such as Histogram equalization, contrast stretching, contrast stretching with clipping population on a single tail or both tails of the image histogram. Enhancement techniques are also applied on a moving window of user specified size in the video image to assist user in selection of target. Once the target is selected by user, various parameters for input image are calculated such as average intensity, standard deviation and Intensity histogram. Adaptive video Image enhancement techniques are applied for real time enhancement using lookup table [LUTs]. Lookup table is created based on derived Image statistics [Range, entropy, coefficient of variation, mean and contrast] obtained from real time video Image data as given in table 1. Table 1 Image Information Statistics Analyzing various data sets it is found that contrast information derived is more stable than other image statistics as it depends more on grey level range in the scene data. Contrast information is derived from real time video image data as : Contrast (%) = Max GL Min GL / Max GL + Min GL where Max GL and Min GL are the maximum and minimum gray level from input image. LUTs are created based on the input image contrast computed and corresponding LUTs are assigned in real time for Image enhancement. 2. Target region processing {TRP] tools to process user selected region of the video frame.option to select various Image filtering techniques (Low pass and high pass, median),image sharpening tools, selected area zooming, Edge boundary information of the target area by various edge detection (Sobel, Canny,Laplace,Robert, etc ) and to display all the processed output. 3 Target tracking tool to track a particular target in the video frame using template pattern matching algorithm.the algorithm is based on normalized cross correlation method. The Position of target in the first image is identified as Window area and the target in the subsequent frame is identified as Search area.the problem is then to estimate the position of the template in subsequent images. Normalized correlation template matching algorithm is used to search target in the subsequent frame and to estimate the position of the target accurately. 4 Algorithm for computation of geo referenced position of the target selected by the user using the flight data. By click of mouse on target, the look angle, slant range, easting/northing or Latitude and longitude of the selected area is displayed on Map. 5 Algorithm for retrieval of area based target within a user specified area. Retrieves the targets that lie within the user defined area or the maximum Range of search. Area to be specified in terms of Circular range around the current UAV position. 6 Tool for Real time Flight parameters display such as Roll, Pitch, azimuth, altitude and heading angle of the Vehicle. 7 Algorithm for Terrain measurement and to measure specific terrain features. Measures includes linear distance on ground, tracing of curvilinear path, area coverage of interesting region and perimeter. RESULTS The basic input to the Image exploitation system consists of mission path, probable target locations and map data. During a mission Image exploitation system

4 acquires image data from the camera and flight parameters from onboard systems. Frame grabber digitizes image data and transfers it to Host PC and multiple embedded processor boards. Application software in host uses multiple embedded processor boards to achieve real time image processing, analysis and display of various parameters to user during mission. Application software is tested using the playback flight video image and data captured by the unmanned mission in a form of DVD connected to the embedded vision system. The system processes the images using Image processing application software in the form GUI menu which displays input and processed images for analysis. Video data with flight parameters is processed in real time where Roll, Pitch, heading, height of the the vehicle, azimuth, elevation are displayed in meter display as given in Figure 4.. The geo-referenced location of the target position on the ground is computed from a single and multiple frames associated with the target using flight data parameters. Once the user clicks a Target, the Look Angle, Slant Range, Easting/ Northing or Latitude and Longitude of the selected target is displayed with the target image as given in figure 7. Figure 7 Target Position computation A Target Tracking function is developed to track the particular target(s) in the video frame using template matching techniques. Window frame area of target frame is matched with searched area window of the next frame using normalized cross correlation as a similarity measure. A sample result of tracking is given in Figure 8 as below. Figure 4 Real Time Video Image display. A sample result of a moving window area enhancement is given in figure 5. Figure 8 Sample Target Tracking Figure 5. Moving window Area Enhancement Target region processing of a target region in the video frames with variable window sizes and image Processing algorithms such as edge detection /enhancement/ Zooming/sharpening tools, etc is applied for selected area.a sample results of processing is given in figure 6. Figure 6 Selected area Image Processing Target area measurements on ground are computed using the geo-referenced values. Target area is selected by the user on a freezed frame. Distance, area and perimeter of the target are computed using Euclidian distance and it is displayed on map/ zoomed map. User has an option to select and view previous video frames of the incoming video with specified frame interval for analysis. Whenever user acquires a target in the host, this module displays the list of last few target images. Further user can retrieve targets that lay within the user defined or the maximum range of search.when user access this module, dialog is displayed with UAV current position and all target locations on the map. Retrieve target list is maintained where user can see the target details: Target name, easting, Northing, distance from the current location, look angle,azimuth and elevation with the target images in the form of context diagram as given in figure 9

5 . REFERENCES [1] M.Kontitsis, Kimon P. Valavanis et al. A UAV Vision System for Airborne Surveillance, Proceedings of the 2004 IEEE, International Conference on Robotics & Automation, New Orleans, LA, April 2004 P [2] P. Doherty et. al, The WITAS unmanned aerial vehicle Project.In. Proc. of the 14 th European conf. of. Artificial Intelligence, [3] Brian Hoerl, ARIES migrates Image Exploitation to UAVs, VME bus Systems, Mercury /Aug 2005 Figure 9 Target Image Retrieval An Image intelligence database is generated for Targets such as geo-location of the target, normalized view of the target, history information, Video clipping, and auxiliary data like survey maps, satellite image and digital elevation model, etc. CONCLUSION This study demonstrate the development of a image exploitation system with potential of using image processing tools for processing real time video and flight data for an unmanned aerial vehicle. The image exploitation system is developed using multiple embedded processing boards connected to the host and results of processing is displayed in multiple displays using graphic card. Real time Video Image acquisition and image analysis using multiple embedded processor with optimized image algorithms library is realized.the work involves the design and development of the system meeting project specific requirements and integrating the hardware, software features. The system is tested using simulated flight data collected for a surveillance mission. The advantages of this architecture being scalable and flexible to increase embedded processor boards as per requirement. To maximize productivity and minimize the decision making cycle of the request, algorithm parallelism effort will seek to achieve better latency. Further exploration of multiple sensor and study of fusion techniques may improve target detection performance. [4] Jamie Heather & Moira Smith, Analysis of Registration requirements & Techniques for Imaging Sensor Suites on UVs, 1st SEAS DTC Technical Conference Edinburgh 2006, [5] R. Gonzalez and R. Woods, Digital image Processing, Addison Wesley, [6] Paul Robertson et al. Adaptive image analysis for Aerial Surveillance, IEEE Intelligent systems, IEEE 1999, P-1094 ACKNOWLEDGMENTS Authors gratefully acknowledge Dr. Jharna Majumdar for her constant encouragement and technical guidance throughout the project.