Boeing Fusion Performance Analysis (FPA) Tool

Transcription

1 12th International Conference on Information Fusion Seattle, WA, USA, July 6-9, 2009 Boeing Fusion Performance Analysis (FPA) Tool Paul Jacson Electronics Prototyping and Integration Center (EPIC) The Boeing Company St. Louis, MO Jeffery D. Musia, PhD Advanced Intelligence & Security Systems The Boeing Company Seattle, WA Abstract - The Fusion Performance Analysis Tool (FPAtool) is a software application that computes Technical Performance Measures for any fusion application. The FPA Tool reads fusion application input and output files, performs computations, saves data to a file. The tool supports both interactive and batch processing modes. In interactive mode it displays an operator control panel, configurable plots and a trac display of the data. In batch processing mode, the tool runs using settings in a parameter file, and outputs image files in addition to the text output files to a specified output directory. The FPA Tool is written in Java and will run on any platform with a Java Virtual Machine. Trac CML Truth Trajectory FPA Tool Entity Keywords: Fusion Metrics 1 Introduction The FPAtool (Fusion Performance Analysis tool) is a GUI-driven software application that computes Technical Performance Measures (TPMs) for any tracer, as long as it conforms to the format of the fused trac file. FPAtool reads TIDS input and output files, performs computations, saves data to a file, and displays plot and text results upon demand. FPAtool is written in Java and has been run on Windows 2000, XP, Linux and Apple OS X operating systems. 2 Fusion Performance Analysis (FPA) Tools 2.1 Overview The FPAtool is typically used as a post-run analysis tool. However, it also supports a real-time mode when connected to a tracer output using messaging service called CML, which listens to socet data from the tracer containing trac data.. TPM Summary.txt Plots & Graphs Figure 1 FPAtool Data Flow Diagram The trac file and/or CML trac message contains the following information used by the FPAtool: time, trac number, sensor number, confirmation counter, target number (truth), latitude, longitude, altitude, position and velocity in ECEF, covariance matrix, and a list of id categories and their estimated probabilities. The truth trajectory input file contains the time, target number, position, velocity in ECEF coordinates The entity input file contains the entity numbers, their true category name, and the entity name for each object. The entity name file is only needed if discrimination TPMs are selected ISIF 1444

2 2.2 Metrics There are three categories of TPMs, which can be selected. The TPMs below are divided according to these categories: State Estimation TPMs TPM 1: Estimated Position Accuracy Difference between a trac s inematic state position estimate at time and the true target position at the same time. A cubic spline method is used to convert the true position to account for time mismatches between the trac and the truth. velocity) at some future time and the true target state at the same time. This metric produces a text with these differences in position and velocity. This data can easily be plotted with excel for analysis purposes. TPM 201: Angular Error The trac input file supports angle-only inputs. This measure determines the bearing error of angle-only tracs (if provided) TPM 202: Normalized Error - For each trac, plot normalized, in terms of sigma, distance between a trac and the truth object. The expected range is 1-3 sigma, more than this range is a an indication of possible divergence. R = RE RTC (1) Where the trac estimated position vector is R E, and the converted true position vector is R TC. TPM 2: Estimated Velocity Accuracy Difference between a trac s inematic state velocity estimate at time t and the true target velocity at the same time. V = V E V TC (2) Where the trac estimated velocity vector is V E, and the converted true velocity vector is V TC. TPM 3: Composite Trac Accuracy - Mean and Median of TPM1 and TPM2 over all tracs. TPM 5: Velocity Coherence - Numerical derivative with respect to time of the trac position errors, providing an estimate of the miscorrelation occurring in the system. Large spies in this measurement show either very large position errors or the use of a miscorrelated report for the update. TPM 5V = [d(r E R TC )/dt](t) (3) TPM 6: State Consistency - For this TPM, we compare the integral of the velocity state estimate of a trac with the position state estimate. Differences (minus a constant offset) indicate inconsistency within the tracer. TPM 6 = R E ( V E dt + R E0 ) (t) (4) TPM 112: Future Kinematic Accuracy - Difference between a trac s inematic state estimate (position and NE = T ( xp xp ) Pp 1 ( xp xp ) / 3 j Note the 3 in the denominator for 3 degrees of freedom, where Pp is the 3x3 position covariance matrix in ECEF coordinates, xp is the positional state in ECEF coordinates for the trac, and xp j is the positional state in ECEF coordinates for the truth object This process is repeated for the velocity covariance also but using the velocity terms of the trac state and covariance matrix. Trac Quality TPMs TPM 10: Trac Continuity - The number of trac segments associated with a particular target gives a measure of continuity. TPM 11: Trac Purity - This TPM measures how well the tracer associates sensor reports with tracs. It also provides a metric on trac swapping". For each object n, contributing to trac m, we can compute the trac purity: R TPM = ftp ( m, n) = R j mn m (5) 11 (6) where R m is the number of reports (or tracs) comprising trac m, and R mn is the number of reports (or tracs) incorporated into trac m that come from object n. By definition, for each trac, we have Σ f TP ( m, n) = 1 N (7) where N signifies summation over all N objects. For any one trac, the highest contributor from any one object is used to determine the purity for that trac, and falls in the range of 0 to 1. TPM 17: Number of Swaps- For each trac, a count will be ept of the number of transitions ( swaps ) between 1445

3 intervals when sensor reports are received from one object and when sensor reports received from some other object. For all M confirmed tracs: Mtracs transitions TPM17 mean = (8) M TPM 18: Mean duration of Intervals Between Swaps - For the same transition statistics developed in TPM17, we can also define the time intervals (or durations) between object report transitions (or swaps). = Mtracs TPM18 mean (9) (8) Mtracs duration swaps TPM 19: Probability of Swap - The probability that a trac will swap one object for another by TPM mean = M swap 19 (10) (8) M The total assignment ratio is the number of trac reports assigned from a primary trac to each object, summed over objects divided by the total reports. AR( t) J c( j) j = 1 = (12) N( t) Where c(j) = max number of assignments from this truth object (column) to any one trac and NT (t) = total number of trac reports. The assignment ratio is computed by taing the max from each column (object), and summing these, and dividing by the total number of trac updates. Ideally the correct assignment ratio will equal 1.0. Potentially a tracer will have good trac purity if it produces many short tracs with good purity for each trac. However, the correct assignment ratio would liely be poor (less than 1.0). Liewise if a trac represents many objects, this will result in a good assignment ratio (near 1.0), but a poor trac purity (much less than 1.0). Below is a example of computing both metrics. where M swap is the number of tracs that had at least one object swap. TPM 21: Redundant Trac Ratio This is simply the number of confirmed tracs divided by the number of objects. TPM 41: Number of Confirmed Tracs This is the number of tracs which are confirmed. The threshold for the confirmation counter can be adjusted on the FPAtool GUI. In this way, short-lived tracs can be filtered from consideration if desired. TPM 211: Confusion Matrix For this metric, a table is constructed showing which objects have been associated to which tracs. From this table the trac purity and correct assignment ratio is computed. The trac purity is computed by taing the max from each row (trac), and summing these tracs over the total number of tracs. TP ( t) NM NT ( t) j ( t) = (11) Where NM j (t) = maximum number of tracs from any one object j for trac (row) at time t and NT (t) = total number of trac reports. NSRs Generated For Each Truth Object Truth Object Number (TON) Trac # DiscriminationTPMs Figure 2 Trac Purity NSRs Generated For Each Truth Object Truth Object Number (TON) Trac # Column Max Column Total 443 Corr Assign Ratio Figure 3 Assignment Ratio TPM 26: Probability of Correct ID - This TPM requires a entity file input. A confusion table(matrix) is constructed from the true ID categories and the estimated ID of the tracs. Rows of the matrix are labeled by true Row Max Row Total Purity 1446

4 ID, and columns are labeled by the ID estimates produced in the tracer. Each PID vector is placed in a particular row based on the actual ID of the object that went into the feature observation. All such row vectors for a run are summed up and then each row is normalized by the number of observations added to it. The resulting matrix shows the probability that a particular true ID would be estimated correctly (diagonal elements of the matrix) or incorrectly (off diagonal elements). TPM 27: Confusion Matrix/ID Settling Time This TPM requires an entity file. It computes and displays a confusion matrix similar to TPM26. It also displays the cumulative ID probabilities for each trac over time. TPM 29: Accurate ID Duration Computes the percentage of time when trac ID is "correct" as determined by a threshold (which can be adjusted). Results are computed for each ID category and for the aggregate over all ID categories Figure 5 - TPM 1: Estimated Position Accuracy 2.3 Tool Usage The FPAtool, when started loos similar to the following. The trac, truth and entity files can be adjusted since the last time it was run. The selected TPMs can also be modified. Figure 6 - TPM 2: Estimated Velocity Accuracy Figure 4 - FPAtool Control Panel After adjusting the inputs and parameters as desired, the user hits the run button. Not that the FPA tool can also run in a batch mode if desired with non GUI for montecarlo testing. Below are screenshots for some of the various TPMs. Figure 7 - TPM3 Composite Accuracy 1447

5 Figure 8 - TPM 5: Velocity Coherence Figure 11 - TPM 11 Trac Purity Figure 12 - TPM 17,18,19 Trac Swapping Figure 9 - TPM6 State Consistency Figure 13 - TPM 21 Redundant Trac Ratio Figure 10 - TPM 10 Trac Continuity 1448

6 Figure 14 - TPM 27: Confusion Matrix/ID Settling Time (1of 2) Figure 17 - TPM 41 Number of Confirmed Tracs Figure 15 - TPM 27: Confusion Matrix/ID Settling Time (2 of 2) Figure 18 - TPM 202 Normalized Error Figure 16 - TPM 29 Percent Time ID Correct Figure 19 - TPM211 Confusion Matrix A text summary output file is saved and displayed as shown below : 1449

7 accuracy and coherence of individual tracs. The trac quality TPMs measure the quality of the overall trac picture, namely the trac purity, trac continuity, etc. The discrimination TPMs measure the ID correctness of individual tracs. The FPA tool measures the outputs of a single tracer. In the case of a distributed fusion metrics, a good reference of this is reference [3]. The typical approach in this document for multiple sensor platforms is to average the metrics across all the platforms using different weighting techniques, time-averaged or object count averaged for example. 2.4 Summary The FPAtool has been proven to be a very maintainable and useful tool which has been primarily used in for testing the trac outputs of the Boeing Fusion Architecture. However it has been applied to other tracing algorithms in the past as well. While the example TPM s illustrated in this paper are all for ballistic targets FPAtool will calculate the same TPM s for any type of target motion (e.g. air, surface or subsurface). There are three major categories of metrics implemented by the FPA tool: state estimation, trac quality, and discrimination. The state estimation TPMs measure the References [1] FPAtool User's Manual, Build 7.2 Release, Document Number D , Rev 45, Dec 31, 2007 [2] Fusion Simulation and Analysis Tool Algorithm Description Document, Document, D , original, July [3] Single Integrated Air Picture (SIAP) Attributes (distribution statement A: approved for public release, distribution unlimited), Version 2.0, August 2003, written by the SIAP System Engineering Tas Force. 1450