Cooperative Fusion Architecture in a Network Centric Environment

Transcription

1 Cooperative Fusion Architecture in a Network Centric Environment Dr. Omar Aboutalib System Analysis and Sensor Fusion Corporation, 1142 South Diamond Bar Boulevard, M/S 247 Diamond Bar, California 91765, USA omaraboutalib@hotmail.com Abstract Modern network centric technology within the global information grid (GIG) permits the rapid and effective sharing of information to such a degree that nodes should be able to "pull" information, rather than having centralized locations predict their information needs and "push" it to them. In this paper, we introduce innovative cooperative fusion architecture of nodes with similar on-board sensors and algorithms followed by fusion of relevant sources that may be available in the vertical layered sensing environment. This leads to flattening of the traditional network hierarchy and enhanced target location and reliable combat target identification. Keywords-component; network centric fusion, cooperative fusion, decision level fusion, relevant and common operating picture, GPS-denied navigation. I. PROBLEM STATEMENT The role of information fusion and the need to share fused information quickly has become critical in light of modern asymmetric warfare. There is a need of networking sensors, commanders, and shooters to flatten the network hierarchy, and enhance information fusion precision and reliability. Modern network centric technology within the GIG permits the rapid and effective sharing of information to such a degree that nodes conducting military missions themselves, should be able to "pull" information, rather than having centralized locations predict their information needs and "push" it to them. This would imply a major flattening of traditional network centric architecture. However, the extensive amount of data and information being collected in a network centric enterprise can potentially overwhelm the war fighters to the point that critical event or threat information may not be addressed on time. Therefore, the war fighters need effective processing of the vast amount of data and associated contextual information to generate timely information to mediate multiple hypotheses and make the proper decisions. The advantages of network centric fusion are the ability to adapt to unanticipated and unforeseen situations, eliminate single points of failure, and remain continuously operational while being dynamically updated. The advent of network-centric systems has served to accelerate the move toward systems of systems. Some of the challenges in building effective network centric fusion architecture are the ability of the architecture to network all relevant nodes into one self-forming, self-healing network; network coordination where every piece of mobile node which may be a piece of equipment or a human participant becomes a potential source or relay of information; maintaining reliable common and/or relevant operating picture throughout; the accurate location awareness where global positioning system (GPS) coverage is weak or denied; and the considerable interest in providing data fusion as a service in a serviceoriented architecture (SOA) environment. The SOA concept was first initiated by the computer science community. In a SOA environment, functional capabilities are made available to users as services that interact through the use of the core enterprise services that include security control, messaging, and discovery services that allow users to become aware of available services. This need will impact the design of the fusion architecture to operate in the SOA environment. Multitier fusion in vertically integrated ISR enterprise assumes that each node within the network knows its position fairly accurate. This assumption is not valid if attacks on key nodes of the GPS occurred. Therefore, a fallback method of navigation is required in time of crisis. The all source fusion of vision and navigation sensors is required to provide continuous and precision navigation in GPS as well as non-gps environment. II. NETWORK CENTRIC FUSION ARCHITECTURE In this section, we propose innovative two-tier fusion architecture. Tier 1 is cooperative fusion of similar nodes with similar on-board sensors and similar algorithms followed by tier 2, which fuses each node with relevant sources that may have access to in the vertical layered sensing environment. Tier 1 fusion yields enhanced fusion performance by enabling the distribution of all participating nodes in preferred configurations that enhance the fusion performance and generate reliable common or relevant operating picture for nodes participating in cooperative fusion. In tier 1 fusion, all participating nodes have the same sensors and processing algorithms and communicate using fast local network. In Tier 2, each node fuses its operating picture with any available information from sensors or sources in the vertical layer sensing environment. A new update to the common or relevant operating picture is then performed among the participating nodes in tier 1 as shown in Figure 1. The Tier 1 fusion represents the fusion of sensors on-board a single platform and across platforms in a given formation of same platforms using similar on-board sensors and algorithms, 1379

2 and tier 2 fusion represents the additional vertical fusion with other sources in the layered sensing environment. It should be noted as discussed in sections A and B that care needs to be exercised in the multi-tiered fusion to avoid fusing dependent or duplicate information which will lead to building false belief. Figure1: Two-tier fusion in network centric environment maintains common or relevant operating picture Tier 1 fusion encompasses the fusion of the detections or tracks of the on-board sensors of each aircraft to produce Single Platform Multi-Sensor (SPMS) database as discussed in more details in section A. Subsequently, cooperative fusion among platforms within the formation yielding a common or relevant operating picture by producing a Multi-Platform Multi-Sensor (MPMS) database as shown in Figure 3. Tier 2 fusion is the fusion of each platform operating picture with the relevant off-board assets in the vertically integrated sensing environment, and a subsequent cooperative fusion among the platforms within the formation yielding an updated common/relevant operating picture. A relevant operating picture includes the fused results in the common footprint area and any tracks outside the common footprint that deemed to be of interest to a given platform (e.g. tracks that are expected to show up in the platform field of regard). A common operating picture includes the fused results in the common footprint area as well as the combination of all other tracks. A. Tier 1 Fusion for Target Tracking Figure 2 depicts tier 1 single platform multi-sensor fusion. As shown in Figure 1, raw detections from each single sensor on-board each platform in the formation are correlated and fused into single sensor tracks. The single sensor tracks are also used to build a sensor detections database of detections correlated to sensor tracks (marked as database 1 in Figure 2), which may be used for centralized fusion with other on-board sensor detections as shown in Figure 2. There are two methods of fusion to enhance tracking performance, centralized fusion (marked as block A in Figure 2) and track-level-fusion (marked as block B in Figure 2). Track fusion suffers from the presence of cross correlation of the tracking error from two or more sensors tracking the same target. This cross-correlation of these errors should be taken into account in fusing the track data [1]-[3]. In general, centralized fusion tends to be more accurate than track fusion. A hybrid of the two approaches may be more appropriate, where track fusion is used for situation awareness for targets at long ranges while centralized fusion is used for fire control and targeting at closer ranges. A SPMS track database is first initiated using single tracks. The correlation of single sensor tracks and SPMS tracks are then used to develop centralized or track fusion databases as show in Figure 2. Cooperative fusion are performed as part of tier 1 fusion, where a portion of the on-board SPMS database is communicated to the other platforms within the formation via the in-flight data link as shown in Figure 1, and Figure 3. The transmit filter shown in Figure 3 selects the transmitted database based on geographical and temporal constraints. For example, the transmitted database may represent targets in the common footprint of two platforms for cooperative fusion with the other platform s tracks, or may represent relevant tracks to the other platform. The platform receive filter will receive data from the other platforms which may include sensor detections and/or tracks for multi-platform multi-sensor (MPMS) centralized or track fusion. It should be noted that the received data needs to be inspected using a maintained pedigree table to determine if the received data has been contaminated by ownship database as shown in Figure 3. Different track fusion methods apply to remove estimation error cross-correlation with ownship tracks [1]-[3]. A common and/or relevant operating picture is then formed and shared as shown in Figure 1. Figure 2: Tier 1 Single platform multi-sensor fusion for target continuous and precision tracking. B. Tier 1 Fusion for Combat Identification The Existing combat identification (CID) implementations lack consistency and the required high confidence due to: 1) stovepipe architecture leading to disparate CID contributors with limited fusion; 2) lack of CID algorithms characterization for proper fusion; 3) available ISR data is not fully utilized due to latency and ambiguity; 4) lack of target models that are representative of all potential target articulations and variations; 5) large variability in the extended operating conditions including deployment of partially obscured targets and targets employing denial and deception (D&D) techniques; and 6) 1380

3 black box implementations of CID systems which lead to expensive software validation and verification (V&V) as the CID technology matures and new and advanced CID algorithms become available. There are various levels of CID fusion including, pixel-level fusion, feature or attribute level fusion, and decision-level fusion [4]-[7]. Let us assume without the loss of any generality that we have a formation of two platforms within the formation. Each aircraft has a wide area search Electronic Warfare (EW) or Electronic Support Measure (ESM) sensor for detection and identification of emitting air defense radars, a high resolution Synthetic Aperture Radar (SAR) for long range target detection and identification, and an Electro-Optical/Imaging Infrared (EO/IIR) as a terminal sensor. Figure 4 illustrates some of the potential CID fusion levels for the sited example. requirements for CID algorithms characterization. Level 1 demands the least amount of requirement from each CID algorithm, where a single confusion matrix represents the minimum amount of information for the characterization of a single CID algorithm. Level 2, requires multiple confusion matrices representative of various levels of probability of correct classifications (e.g. low, medium, and high) as well as extended operating conditions (e.g. different look down angles, and various weather states). Level 3 occur if the multi-sensor CID algorithms are characterized jointly. It should be noted that in level 3, the CID algorithms are tightly coupled, but yields the most reliable fusion results. In level 1, the CID algorithms are loosely coupled, but may yield unreliable fusion performance. The various competing CID algorithms will start as loosely coupled but overtime they become tightly coupled through an adaptive learning process. For track fusion methods, we need to remove estimation error cross- correlation among tracks as described in [1]-[3]. Figure 3: Cooperative fusion among the nodes of a given formation C. Tier 2 CID Decision-Level Fusion and Track Fusion In a vertically integrated sensing environment, CID decision-level fusion and track level fusion are most applied. CID decision-level fusion methods used in industry are not quite effective. In some situations, there is no performance gains experienced due to fusion, and in other situations the fused decision performance is less than an individual sensor algorithm decision performance. This is due to making assumptions to simplify the mathematics of decision level fusion. Specifically, making the assumption that the joint decision probability of multiple identification algorithms is the product of individual decision probabilities. An optimal CID decision level fusion for either tier 1 or tier 2 fusions should avoid making such assumption, and should utilize all historical available information to perform global optimization of the joint decision probability. All CID algorithms are characterized offline (this process known as automatic target recognition algorithms training) and their respective decision performance metrics are collected. These historical CID performance metrics should be used for the decision optimization. Figure 5 illustrates various levels of Figure 4: Potential CID fusion levels in cooperative fusion Figure 5: CID algorithms characterization and decision level fusion performance D. CHALLENGES OF GPS DENIAL IN LAYERED SENSING While the GPS/INS navigation performance is fairly accurate, navigation performance will degrade significantly if 1381

4 the GPS is denied. Multi-tier fusion in vertically integrated ISR enterprise assumes that each node within the network knows its position fairly accurate. This assumption is not valid if attacks on key nodes of the GPS occurred. Terrain-relative navigation can be used as an alternative navigation to rectify the GPSdenied problem. Optical flow processing can be used to infer the velocity vector of the flying platform. When the platform with the imaging sensor or camera moves with respect to a stationary three-dimensional (3-D) point, the 3-D velocity of the platform as seen by that stationary point can be mapped onto the sensor s image plane as a two-dimensional (2-D) image motion. As the platform moves, projected points on the image plane seem to be traveling away from the optical flow center, known as the focus of expansion (FOE). The FOE defines the point on the image plane at which the velocity vector of the platform pierces the earth surface. The field of computer vision has witnessed many approaches for computing ego-motion which estimates an observer s movement (sensor or biological organism) from optical flow measurements. Optical flow is defined as the apparent motion of the brightness patterns. The optical flow is characterized by a field of 2-D velocity vectors which are the projections of the 3- D velocity vectors of the surface points onto the image plane [8]-[11]. The 2-D velocity vectors derived from sequences of images are shown in Figure 6. Adaptive fusion in real-time of all available navigation data such as IMU/GPS, altimeters, star tracker, horizon sensor, passive imaging sensor, and digital elevation database is a promising approach for navigation in non-gps environment. For example, the integration of passive imaging sensors has some important advantages. Foremost, the sensors are completely passive, and can operate in an environment where the GPS signal may be difficult to receive. Secondly, the sensors are immune to disruptions in the radio spectrum. The adaptive fusion of all navigation and sensory will provide optimal estimate of the vehicle position, velocity, and attitude, as well as estimates of the accelerometer, rate gyro, and clock biases. The architecture of the fusion filter should be independent of specific sensor types and data rates, and hence is easily adaptable to the addition of new sensors, changing data rates, and loss or interruption of data from any given sensor. The fusion filter can be hosted on multiple platforms (airborne, ground), small robots, or dismount man-wearable equipment to provide adaptive fusion in real-time of all sources or signals of opportunities that may be available to any node in the network leading to distributed and collaborative navigation solutions. At the tier 1 level, the on-board single platform sensors such as EO/IR, SAR, altimeter, horizon sensor, inertial measurement unit (IMU), and star tracker should be used for navigation as shown in Figure 7. Collaboration among multiple platforms within a formation of platforms and other platforms in the vertically integrated sensing environment will be used to enhance navigation performance and transfer navigation accuracy to platforms flying over featureless terrain such as desert or water. Fusion is the main focus for alternate navigation in GPS denied environment as described in section E. Figure 6: Optical flow field The measurement of the 2-D image velocity at a given point can be used to infer the 3-D velocity of the imaging platform. Since the optical flow is a projection of a 3-D velocity vector onto a 2-D image plane, there is an inherent ambiguity in inferring the platform 3-D velocity vector. The observed optical flow therefore needs to be fused with other measurements such as the scene depth or height to remove the ambiguity and to provide an estimate of the 3-D velocity vector of the imaging platform. Knowing the observer velocity vector and the last GPS position estimate will enable, in theory, navigation in non-gps environment. Using only optical flow measurements for navigation in non-gps environment encounters difficulties due to the variation of depth across the image field and the presence of noise in the sequence of images. Legacy approaches combine precision radar sensors with digital terrain elevation databases to serve as a back-up to the GPS system. Such approaches require high-level of accuracies of digital terrain data and radar sensor and will fail if the platform travels in featureless terrain. Figure 7: Network centric autonomous and collaborative navigation in GPS denied environment E. Fusion for Navigation in non-gps Environment Fusion of all available vision and navigation sensor data will provide an alternative navigation in a GPS-denied environment. Various sensors have various contributions to continuous and precision navigation in GPS-denied environment as described below: 1382

5 Inertial Measurement Unit (IMU) - IMUs usually contain three orthogonal rate gyros which measure the angular rotation rates of the vehicle s roll, pitch, and yaw, and three orthogonal accelerometers measuring the vehicle linear accelerations in the vehicle coordinate system. By processing signals from these six sensors at high rate (usually 100Hz or more), it is possible to use initial estimates of vehicle heading and position developed at lower rates to perform dead reckoning and generate the vehicle s heading and position at 100 Hz as shown in Figure 8. Figure 9: Surface features are detected and tracked Figure 8: Dead reckoning of vehicle heading and position The fusion of IMU measurements for navigation in non- GPS environment needs to account for the various types of IMU error such as misalignment, moving bias, random walk noise, and scale factors. IMU sensing can be unreliable at very high altitudes. Star Trackers The star tracker provides measurements of vehicle attitude by detecting stars of high magnitudes, and performing star pattern recognition and vehicle s attitude determination based on line of sight of vectors to the known stars. Star tracker measurements can be used when the IMU sensing is unreliable at very high altitudes and can also be fused with IMU measurements to reduce the dead reckoning error. With the rapid advancement of imaging sensor hardware and high-speed computer processors, current star trackers are small and can achieve 1-arc-second attitude accuracy at a sampling rates range from 1 to 10 Hz. Horizon Sensor - horizon sensor is a scanning infrared sensor (using a small focal plan array, µmeter wave length). It senses the comparative warmth (radiated intensity) of atmosphere compared with much colder cosmic background to determine the horizon. Attitude measurements using star tracker can be combined with the horizon plane position developed using a horizon sensor to provide the vehicle s geolocation. Electro-Optical / Imaging Infrared (EO/IIR) Sensor EO/IIR are useful vision sensors. Optical flow measurements and surface features as shown in Figure 9 can be detected from imagery stream and can then be correlated in real time with the database of landmarks [12]-[19]. Optical flow measurements can contribute to the estimation of vehicle velocity vector. Landmark measurements are important navigation aids because they provide positions as well as orientations fixes as shown in Figure 10. EO/IIR sensors have limitations for alternate navigation in high altitude missions and low cloud ceiling where visibility is limited. Figure 10: Landmarks can be used for position and orientation fixes Altimeter altitude measurements can be combined with EO/IIR optical flow measurements to determine vehicle velocity and position. Altitude can be measured using a barometer based on atmospheric pressure. Radar altimeter measures altitude more directly, using the time taken for a radio signal to reflect from the surface back to the aircraft. The radar altimeter is used to measure height above ground level during landing in commercial and military aircraft. Synthetic Aperture Radar (SAR) SAR sensor can provide all-weather, autonomous navigation. By forming SAR imagery of the terrain and then correlating persistent SAR features with a stored reference obtained from optical photography or a presorted SAR imagery database, a navigation position fix can be obtained. Position accuracies of less than a SAR resolution cell can be obtained. Magnetometer A magnetometer is an instrument used to measure the strength or direction of magnetic fields. As we mentioned IMUs are naturally subject to drift since they are essentially integrators, and any error in the sensor reading is compounded by the integration. They are often augmented by some other sensor to correct for drift. One method is to use a magnetometer to get an orientation measurement based on the earth magnetic field. Magnetometers will help correct orientation error but have no direct contribution in correcting vehicle s position or velocity. Digital Terrain Elevation Data (DTED) There is a great utility of DTED databases in navigation in non-gps environment. However, the limitations and accuracies of these databases need to be considered. The utility of DETED database and the use of a radar altimeter has been demonstrated to provide position fix for low-altitude missions in many legacy systems. 1383

6 III. CONCLUSIONS There is a need of networking sensors, commanders, and shooters to flatten the network hierarchy, and enhance information fusion precision and reliability. Modern network centric technology within the GIG permits nodes within the network to "pull" information, rather than having centralized locations predict their information needs and "push" it to them. We introduced innovative cooperative fusion architecture of nodes with similar on-board sensors and algorithms followed by fusion of relevant sources that may be available in the vertical layered sensing environment. Tier 1 fusion yields enhanced fusion performance by enabling the distribution of all participating nodes in preferred configurations that enhance the fusion performance and generate reliable common or relevant operating picture for nodes participating in cooperative fusion. Shortcomings of existing combat identification (CID) implementations and optimized decision-level fusion are discussed. Continuous and high precision navigation is also required for network centric fusion. Multi-tier fusion in vertically integrated ISR enterprise assumes that each node within the network knows its position fairly accurate. This assumption is not valid if attacks on key nodes of the GPS occurred. Therefore, a fallback method of navigation is required in time of crisis. At the tier 1 level, the on-board single platform sensors such as EO/IR, SAR, altimeter, horizon sensor, inertial measurement unit (IMU), and star tracker can be used for terrain relative navigation. The concept of the all source fusion of vision and navigation is introduced for navigation in non-gps environment. The contribution of various types of vision and navigation sensors is also presented. For higher tier fusion collaboration among multiple platforms will be used to enhance navigation performance and transfer navigation accuracy to platforms flying over featureless terrain. REFERENCES [1] Y. Bar-Shalom and L. Campo, The effect of common process noise on the two-sensor fused-track covariance, IEEE Transactions on Aerospace and Electronic Systems, vol. 22 no 6, pp , 1986 [2] J. Oliver E. Drummond and David Dana-Bashian, "Comparison of tracklet filter methods for non-maneuvering targets", Proc. SPIE 5913, 59131B (2005). [3] X. R. Li and Z. L. Zhao, Measuring estimator s credibility: Noncredibility index, in Proceedings of the 9th International Information Fusion Conference, pp. 1 8, July [4] G.S. Robinson, and A. Omar Aboutalib," Trade-off Analysis of Multi-sensor Fusion Levels", 2nd National Symposium on sensor fusion, Orlando, Florida, Vol. II, March 1989, pp [5] A. Omar Aboutalib, L. Tran, and C. Hu, Fusion of Passive Imaging Sensor Data for Target Acquisition and Identification", 5th National Symposium on Sensor Fusion, Orlando, Florida, Vol. I, April 1992, pp [6] A. Omar Aboutalib, Luong Tran, Jeff Cameron, " Enhancement of 4-D Laser Radar Imagery for Target Identification", 1993 Infrared Information Symposia on Active Systems, Monterey, California, Proc. IRIS, vol. II, November 1993, pp [7] O. Aboutalib and Timothy J. Klausutis, All source adaptive fusion for aided navigation in non-gps environment, SPIE, March [8] Berthold K.P. Horn and Brian G. Schunck, Determining Optical Flow, Artificial Intelligence 17, , [9] B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision, Proc. DARPA Image Understanding Workshop, pp , [10] Joachim Wickert and Christoph Schnorr, Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint, J. of Mathematical Imaging and Vision 14: , [11] S.S. Beauchemin and J.L. Barron, The Computation of Optical Flow, ACM Computing Surveys, Vol 27 No. 3, , [12] Jianbo Shi and Carlo Tomasi, Good Features to Track, IEEE Conference on Computer Vision and Pattern Recognition (CVPR94), [13] Berthold K.P. Horn and Brian G. Schunck, Determining Optical Flow, Artificial Intelligence 17, , [14] B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision, Proc. DARPA Image Understanding Workshop, pp , [15] Joachim Wickert and Christoph Schnorr, Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint, J. of Mathematical Imaging and Vision 14: , [16] S.S. Beauchemin and J.L. Barron, The Computation of Optical Flow, ACM Computing Surveys, Vol 27 No. 3, , [17] Jianbo Shi and Carlo Tomasi, Good Features to Track, IEEE Conference on Computer Vision and Pattern Recognition (CVPR94), [18] Artur M. Arsenio, "An Embodied Approach to Perceptual Grouping," cvprw, p. 51, Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04), [19] David G. Lowe, Distinctive Image Features from Scale- Invariant Keypoints, International Journal of Computer Vision,