Active Tracking of Surface Targets in Fused Video 1

Size: px

Start display at page:

Download "Active Tracking of Surface Targets in Fused Video 1"

Edwin Dickerson
5 years ago
Views:

1 Active Tracking of Surface Targets in Fused Video 1 Allen Waxman, David Fay, Paul Ilardi, Pablo Arambel and Jeffrey Silver Fusion Technology & Systems Division BAE SYSTEMS Advanced Information Technologies Burlington, MA, U.S.A. allen.waman@baesystems.com Abstract - We seek to establish a quality metric for sensor fused video based on the performance of a multisensor system that actively tracks multiple surface targets over an extended field of regard. Such a system must, in real-time, fuse multisensor/spectral imagery, detect targets reliably, track the targets for extended time periods as they move, stop, approach, cross, hide/emerge, over an extended field of regard. The system must automatically control the pointing, zoom and modes of cameras so as to maintain and re-establish track as targets move outside the current field of view. All this must be done for an increasing number and density of targets in clutter. Individual performance metrics relating to these various stages of processing include percent detection vs. false alarm rate, track identity lifetime, track purity, track accuracy, field of regard, target revisit rate, and number of targets tracked. All of these metrics reflect the quality of the multisensor imagery and the algorithms (image fusion, detection, tracking, sensor resource manage-ment) for the multitarget tracking task. This same task applies to human observers tracking multiple surface targets in displays of multisensor/spectral video. One intuitively expects that the "better" the image quality, the more "effectively" one can track multiple targets. We are exploring this relationship between tracking and fused video as a quality metric. This paper introduces the computational fused video tracking system that forms the basis of this new effort. Future papers will provide results and performance metrics noted above. Keywords: Image fusion, image quality, target detection, multi-target tracking, sensor resource management 1 Introduction With growing interest in the use of fused imagery for enhanced vision and target detection, and the emergence of real-time systems that can fuse multisensor video using a variety of alternative approaches, the question of fused image & video quality has arisen. A variety of traditional metrics exist for visible and infrared still imagery, and were summarized in [1] in the context of visible, thermal and fused image quality metrics. For dynamic multisensor imagery, we propose to develop fused video quality metrics based on performance of an explicitly dynamic task, that is, multi-target tracking over an extended field of regard. This is a task that can be quantified for human performance in a psychophysical context, and for a completely automatic target tracking system that utilizes fused multisensor video (e.g., VNIR & or SWIR & or RGB & or VNIR, SWIR & ), detects targets, fingerprints targets, tracks multiple targets, and controls its sensors to point among targets while maintaining wide-area search over the field of regard. In this paper we introduce our system for active tracking of surface (water & ground) targets in sensor fused video. Each stage of processing has its own set of relevant metrics, however, it is the system as a whole that must perform the task of detecting and tracking multiple targets while maintaining wide-area situational awareness. Overall performance will depend on many real-world parameters that ultimately impact the quality of the fused video derived from input imagery streams. Comparison of automated multi-target tracking to human performance using the same fused video will enable us to establish a relationship between quantifiable metrics derived from the automated system and the expected performance of human observers endowed with sensor fused displays (e.g., goggles, cockpit displays, or UAV turret displays). Establishing this relationship to human performance is the subject of future collaborative research. Here we focus on the fused video tracking system that supports this research. 2 Image Fusion and Quality Factors We have reviewed our opponent-color neural architecture for color image fusion in many previous publications [1 4], and noted alternative approaches to fusing imagery into panchromatic and color products [5 8]. These alternative methods for fusing imagery can start with identical input data and produce dramatically different results. Various measures of image quality would, no doubt, yield different results for these alternative fusion products. Thus, the method for image fusion itself can affect fused image quality (as clearly shown in Fig. 12 of ref. [1]). Our multi-resolution opponent-color image fusion architecture is shown in Figure 1. It combines gray-scale 1 This research sponsored, in part, by the U.S. Air Force Office of Scientific Research.

brightness fusion on multiple scales [5] with opponentsensor contrast mapped to human opponent-colors [2 4], and can be implemented in real-time on inexpensive commercial DSP boards [4, 9].

& color = center-surround shunting dynamics Figure 1: Multi-resolution opponent-color fusion network uses center-surround shunting dynamics to support local adaptive contrast enhancement and gain

Typically, one uses this approach to fuse images containing complementary information, e.g., reflected light in day or night (VNIR or SWIR) with emitted light (MWIR or ) from the scene.

Clearly, as the spectral diversity of the input imagery increases, the resulting fused image quality is improved, inasmuch as it supports spontaneous visual segmentation into scenic objects.

7µm) Fused SWIR/ Figure 3: Dual-sensor fusion of VNIR/ using an intensified-ccd for the VNIR channel, and SWIR/ using an InGaAs imager for the SWIR channel, both using an uncooled microbolometer for

3 Target Detection in Fused Video In order to track multiple targets, it is first necessary to detect the targets reliably from the fused imagery.

However, in a tracking context, the temporal stability of the real targets should sustain them, whereas the likely temporal instability of the false alarms should suppress them at the tracker output.

2 brightness fusion on multiple scales [5] with opponentsensor contrast mapped to human opponent-colors [2 4], and can be implemented in real-time on inexpensive commercial DSP boards [4, 9]. of the SWIR sensor (InGaAs focal plane) is much less than the VNIR sensor (intensified-ccd), the resulting apparent resolution and quality of the fused SWIR/ appears higher than the fused VNIR/ imagery. VNIR Rev. + - Y Q I Rotate Hue Hue Map Map to to RGB VNIR - + Fused Result Register and Noise Clean LACEG Local adaptive contrast & gain Opponent Contrast: Multi-sensor/scale contrasts drive display brightness & color = center-surround shunting dynamics Figure 1: Multi-resolution opponent-color fusion network uses center-surround shunting dynamics to support local adaptive contrast enhancement and gain (LACEG), and two stages of opponent-color that are interpreted as the elements of the Y, I, Q human opponent-color space. Typically, one uses this approach to fuse images containing complementary information, e.g., reflected light in day or night (VNIR or SWIR) with emitted light (MWIR or ) from the scene. This same architecture can be used to fuse two, three or four imaging sensors, as shown in Figure 2. Clearly, as the spectral diversity of the input imagery increases, the resulting fused image quality is improved, inasmuch as it supports spontaneous visual segmentation into scenic objects. InGaAs SWIR ( µm) Fused SWIR/ Figure 3: Dual-sensor fusion of VNIR/ using an intensified-ccd for the VNIR channel, and SWIR/ using an InGaAs imager for the SWIR channel, both using an uncooled microbolometer for the channel. Taken in overcast full-moon (40 mlux), the SWIR/ image has better quality than the VNIR/ image, though the I 2 CCD has higher resolution than the InGaAs focal plane. 3 Target Detection in Fused Video In order to track multiple targets, it is first necessary to detect the targets reliably from the fused imagery. It is often the case that false alarms may also be detected. However, in a tracking context, the temporal stability of the real targets should sustain them, whereas the likely temporal instability of the false alarms should suppress them at the tracker output. Thus, the temporal component of fused video serves to extend image quality beyond a static metric such as percent detection of targets vs false alarms (P d vs P fa ). In order to detect targets in fused imagery, we utilize the opponent-color contrasts created in the fusion process itself together with the multisensor inputs, to create a feature vector as shown in Figure 4 [9]. Pixels Fused VNIR/SWIR/ Fused VNIR/SWIR/MWIR/ Figure 2: The architecture of Figure 1 is used to fuse 2, 3, or 4 sensors, including VNIR/SWIR/MWIR/ shown here, as collected under quarter-moon (27 mlux). Image quality improves, with increased color contrast, as more complementary spectral bands are fused. Even the choice of spectral bands and sensing technology can affect image quality, due to a combination of the natural illumination in that band (e.g., VNIR vs SWIR at night) and the noise factor of the sensor which impacts the noise-limiting resolution of the imagery. Figure 3 illustrates this with a comparison of fused VNIR/ and SWIR/. Though the sampling resolution Features Color fused Visible SWIR MWIR Opponent-Band contrasts Contours Textures 3D Context Figure 4: Multi-modality and cross-band contrast image layers contribute to a feature vector at each pixel....

This feature vector is then input to an adaptive classifier (shown in Figure 5) based on Adaptive Resonance Theory (ART) [10, 11], which we ve modified for salient feature subspace discovery and

3 This feature vector is then input to an adaptive classifier (shown in Figure 5) based on Adaptive Resonance Theory (ART) [10, 11], which we ve modified for salient feature subspace discovery and real-time implementation [9]. Recognized Class F3 F3 Target Non-Target Match Tracking I 2 CCD Category-to-Class Learned Mapping Unsupervised Clustering F2 F2 Reset Bottom-Up and Top-Down Adaptive Filters ρ T Complement Coding F1 F1 f 1 f 3 f i ρ NT Input Feature Vector Figure 5: Modified Fuzzy ARTMAP neural classifier is used to learn category & salient feature representations of targets and normalcy models of backgrounds. In this form, the classifier learns fused representations of targets by example and counter-example as designated by an operator. Or, it can learn backgrounds through random sampling or operator designation, and then detect fused spectral anomalies as potential targets which can be learned as categories. Typically, with only two sensors (e.g., VNIR/ or SWIR/), the classifier learns a more generic target detector, such as a man-detector or a boat-detector, as opposed to a specific target detector that serves to fingerprint a particular target among many similar targets. Clearly, this kind of target detector is not based on target shape or resolution. It is, essentially, a spectral-based target detector that exploits cross-sensor spectral contrast. This is illustrated in Figures 6 & 7. VNIR MWIR Fused Detections Kayaks Figure 6: The same neural architecture shown in Figure 1 is used to fuse three sensors VNIR/MWIR/ here. The various outputs of the opponent-colors are used to train a target detector for the spatially unresolved kayaks. Fused Figure 7: Dual-sensor video is fused in real-time. One man-target is designated by an operator as a target of interest, and the classifier rapidly learns a detector that then detects all men moving around in the scene, and runs in real-time on compact commercial hardware. Key advantages of fused multisensor/spectral target detection are independence from both target shape and target motion. Independence from target shape enables detection of small unresolved targets, hence, supporting a wider field-of-view. When tracking targets from a moving platform, there is no need to stabilize the background in order to detect movers while coping with false alarms due to imperfect background stabilization. Stationary targets can be detected easily as well. However, purely spectralbased target detection also has its shortcomings, when other scenic elements have similar spectral content in the limited sensing bands available. As with the human visual system, the most robust target detectors will likely utilize a combination of the color, shape and motion pathways.

4 Nonetheless, we have been quite successful in detecting a large variety of target types in different backgrounds. The nature of our target detector, being based on continuous learning in an ART network (Fig. 5), supports adaptation of the representation while tracking the target. This helps tune the detector on-the-fly as the target moves through extended operating conditions and among confusers. 4 Multi-Hypothesis Tracking and Sensor Resource Management Modern target trackers utilize a combination of Kalman filtering, interacting motion models, and multi-hypothesis association strategies [12]. This enables the tracker to deal with multiple targets undergoing maneuvers and coasting through temporary occlusions. Commonly associated with radar tracking, similar methods have been applied to airto-ground video tracking [13]. This same association strategy can be extended to track-to-track fusion as well as detection-to-track fusion [14], providing a means to fuse tracks generated by multiple sensor platforms. Typical performance metrics for trackers include track identity lifetime, track purity, and track accuracy. Figure 8 illustrates fusion, detection and tracking of multiple targets on the sea surface. Some of these targets are stationary and some are moving. The sensors, consisting of a daylight CCD and MWIR imager, were in a turret aboard an aircraft in flight. This example was computed in real-time from recorded flight imagery. We have conducted related experiments in real-time while flying over water-borne and ground-based targets, both moving and stationary. The fused color imagery has sufficient quality to enable detection of all targets in the field-of-view without any false alarms. In order to track targets over an extended field-ofregard that exceeds the field-of-view of the sensors, it is necessary to have a dynamic sensor control strategy that supports both wide-area search as well as multi-target tracking. The combined control of sensor pointing (e.g., a turret or pan-tilt unit), field-of-view or zoom, and operating mode (e.g., frame-rate, resolution, or possibly integration time), is known as sensor resource management (SRM). Figure 8: Real-time VNIR/MWIR image fusion, detection and tracking of multiple water-borne targets from an airborne platform. Originally hazy daylight TV imagery is enhanced and fused with the MWIR imagery to produce high quality color fused imagery that supports detection of targets with no false alarms [9]. Tracking is performed using MHT multi-target tracking based on ATIF [14]. Figure 9: (Top) Sensor resource manager (SRM) uses the current set of tracker outputs (positions & uncertainties) to plan the next view (pointing, zoom, mode) in order to optimize a cost function (Bottom). The arrow (in the upper panel) shows the planned shift in view that minimizes the cost function (the bright red spot in the bottom panel). In support of the DARPA VIVID program, we have developed SRM methods based on approximate stochastic dynamic programming [13] that have been adapted for the multisensor fused imaging approach. The method utilizes a cost function that aims to optimize the number of targets

imaged in a field-of-view, the size of the field-of-view, the time between revisiting targets, and accounts for a planning horizon that includes not only the next view but following views as well.

5 Tower-Based Field Experiment In order to study the real-time fusion, detection and tracking of targets in a controlled environment, we have constructed a multisensor pan-tilt-zoom unit and conduct

5 imaged in a field-of-view, the size of the field-of-view, the time between revisiting targets, and accounts for a planning horizon that includes not only the next view but following views as well. This is illustrated in Figure 9, where the SRM uses a prediction of multiple target locations from the tracker to plan its next optimal view. 5 Tower-Based Field Experiment In order to study the real-time fusion, detection and tracking of targets in a controlled environment, we have constructed a multisensor pan-tilt-zoom unit and conduct experiments on a tower overlooking an open range. The sensors and tower are shown in Figures 10 & 11. A human observer can also be tasked with detecting and tracking the ground targets being imaged by these sensors, either with direct viewing, enhanced viewing using intensifier tubes or thermal imagers at night, or by observing the monitor displaying the fused imagery from the sensor array. observers, humans with fused goggles or controlling fused turrets, and automated fused sensing & tracking systems. Correlating results between human observers and automated systems remains a challenge for the future. Figure10. Multisensor pan-tilt unit carries a daylight RGB zoom camera, an intensified-ccd (VNIR) sensor, a SWIR camera, a thermal imager, and a laser range finder (LRF). The pointing of the pan-tilt and zoom, as well as the various sensors are under control of the SRM. Figure 12 illustrates the view from the tower, showing the RGB visible, thermal, and fused imagery with target detections overlaid, as well as the target tracks bounded by the instantaneous field-of-view of the sensors. The scene shown is quite simple, with the targets prominently visible. But as the targets become smaller and denser, embedded in background clutter, and of lower contrast in the individual modalities, the task of detecting and tracking multiple targets gets progressively harder. The system will be faced with missed detections, false alarms, occluded targets, and proximity to confusers. This can easily be the case with man-targets moving among the trees at night, similar to those shown in Figure 7. We can expect that as overall image quality degrades in one or more sensor modalities, the task of multi-target detection and tracking becomes harder for both human Figure 11: Test tower 75ft high, used for field testing of the fused imaging and active multi-target tracking system. 6 Conclusions We have previously considered various aspects of fused image quality [1], and in this paper turned to fused video quality in its ability to support detection and tracking of multiple targets. We have developed the capability to do this in real-time, combining methods of opponent-color image fusion, target learning & detection, multihypothesis multi-target tracking, and sensor resource management. Examples are shown for controlled tower tests, field imaging at night, and airborne sensing of vessels on the sea surface. It will be necessary to develop metrics of human performance in this task domain, providing observers with joystick control over the pan-tilt unit (or possibly fused goggles), and requiring them to perform wide-area search, detection and tracking of multiple targets. Then, by correlating human performance with automated target detection & tracking performance using the same imaging modalities and image fusion methods, we expect to be able to predict human performance in such tasks under extended operating conditions. This is the subject of our future research on performance metrics using fused video.

6 Figure 12: Fused RGB/ imaging, target detection & tracking of ground vehicles from the test tower shown in Figure 11. Active tracking involves dynamic control of the multisensor pan-tilt unit to support both wide-area search and tracking of multiple targets in the field-of-regard. References [1] A. Waxman, D. Fay, P. Ilardi, E. Savoye, R. Biehl and D. Grau, Sensor fused night vision: Assessing image quality in the lab and in the field, Proceeds. of the 9 th International Conference on Information Fusion, Fusion 2007, Florence, [2] A. Waxman, A. Gove, D. Fay, J. Racamato, J. Carrick, M. Seibert, and E. Savoye, Color night vision: Opponent processing in the fusion of visible and IR imagery, Neural Networks, 10, 1-6, [3] A. Waxman, D. Fay, E. Savoye, et al., Solid-state color night vision: Fusion of low-light visible and thermal infrared imagery, Lincoln Laboratory Journal, 11(1): 41-60, [4] M. Aguilar, D. Fay, D. Ireland, J. Racamato, W. Ross, and A. Waxman, Field evaluations of dual-band fusion for color night vision, SPIE-3691, Enhanced and Synthetic Vision, [5] A. Toet, L. van Ruyven, and J. Valeton, Merging thermal and visual images by a contrast pyramid, Optical Engineering, 28, , [6] A. Toet, and J. Walraven, New false color mapping for image fusion, Optical Engineering, 35, , [7] A. Toet, J. IJspeert, A. Waxman, and M. Aguilar, Fusion of visible and thermal imagery improves situational awareness, SPIE-3088, Enhanced and Synthetic Vision, , [8] P. Burt and R. Kolczynski, Enhanced image capture through fusion, Fourth International Conference on Computer Vision, , Los Alamitos: IEEE Computer Society Press, [9] D. Fay, P. Ilardi, N. Sheldon, D. Grau, R. Biehl, and A. Waxman, Real-time image fusion and target learning & detection on a laptop attached processor, Proceeds. of the 7 th International Conference on Information Fusion, Fusion 2005, Stockholm, [10] G.A. Carpenter, S. Grossberg, N. Markuzon, J.H. Reynolds, and D.B. Rosen, Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps, IEEE Transactions on Neural Networks, 3, pp , [11] T. Kasuba, Simplified Fuzzy ARTMAP, AI Expert, pp , Nov 1993.

7 [12] Y. Bar-Shalom and X.-R. Li, Multi-target multisensor tracking: Principles and techniques, YBS Inc., [13] P. Arambel, M. Antone, M. Bosse, J. Silver, J.Krant, and T. Strat, Performance assessment of a video-based air-to-ground multiple target tracker with dynamic sensor control, SPIE Defense and Security Symposium, Signal Processing, Sensor Fusion, and Target Recognition XIV, Orlando, [14] S. Coraluppi, C. Carthel, M. Luettgen and S. Lynch, All-source track and identity fusion (ATIF), National Symposium on Sensor and Data Fusion, 2000.

Daniel A. Lavigne Defence Research and Development Canada Valcartier. Mélanie Breton Aerex Avionics Inc. July 27, 2010

A new fusion algorithm for shadow penetration using visible and midwave infrared polarimetric images Daniel A. Lavigne Defence Research and Development Canada Valcartier Mélanie Breton Aerex Avionics Inc.