Fuson Performance Model for Dstrbuted rackng and Classfcaton K.C. Chang and Yng Song Dept. of SEOR, School of I&E George Mason Unversty FAIRFAX, VA kchang@gmu.edu Martn Lggns Verdan Systems Dvson, Inc. 4 Key Boulevard, Sute Arlngton, VA 9 martn.lggns@verdan.com Abstract he research goal of dstrbuted fuson systems n the past has prmarly focused on establshng a computatonal approach for fuson processng algorthms. here are very few research works focused on fuson system performance evaluaton models. hs paper presents a Fuson Performance Modelng (FPM) to characterze the relatonshp between sensed nformaton nputs avalable to the dstrbuted fuson system and the ualty of fused nformaton output. he model ncludes both knematc and classfcaton components and focuses on the two performance measures: postonal error and classfcaton error. he results provde hghlevel performance bounds gven the sensor mx and sensor ualty for system control such as sensor resource allocaton and evaluate sensed nformaton reurements. Understandng these ssues has broad mplcatons for fuson system archtecture desgn, the evaluaton of system alternatves, and understandng the relatve trade-offs between varous sensng technologes. Keywords: Fuson Performance Model, Mutsensor data fuson, Bayesan networks. Introducton Multsensor Data Fuson has been appled n modern estmaton or trackng systems, both n mltary and cvlan applcatons. By combnng nformaton from multple sensors and sources, the estmate of the state (poston, velocty, dentfcaton, etc.) of enttes wll be enhanced. In fuson systems, each of these sensors detects multple targets and may create ther own tracks. he resultng detectons or tracks from these sensors are fused together to provde more accurate knematc and classfcaton estmates. o enhance estmaton performance, dfferent types of fuson archtectures such as centralzed, herarchcal, and dstrbuted systems have been developed. Accordngly, many fuson algorthms have also been developed. In the past, much research has been conducted wth a focus on establshng a computatonal approach for fuson processng and algorthm development. In realty, before a data fuson system s deployed, developers and users need to be able to assess ther fuson system performance under varous condtons. Recently, a study was done on fuson system performance estmaton modelng []. In the study, the performance models were desgned to model bnary detecton processes based on extended Dempster-Shafer theory, and the goal was to predct fuson gan n terms of system operatng characterstcs, such as probablty of detecton (PD), probablty of false alarm (Pfa), probablty of correct decson (PCD), and probablty of false track update (Pftu). However, one of the prmary goals of the fuson system s to localze and dentfy mportant targets. Instead of focusng on statc performance predcton as done n [], ths paper proposes a Fuson Performance Model (FPM) to focus dynamcally on the two most mportant performance measures: postonal error and classfcaton error. he performance model s based on Bayesan theory and a combnaton of smulaton and analytcal approaches. One of the purposes of developng the FPM s to characterze the relatonshp between sensed nformaton nputs avalable to the fuson system and the ualty of fused nformaton output n order to understand fuson system performance. he other purpose s to provde hgh level performance bounds gven sensor mx and sensor ualty for system control, such as sensor resource allocaton and to estmate sensed nformaton reurements. Understandng these ssues has broad mplcatons for fuson system nformaton archtecture
desgn, the evaluaton of system alternatves, and the understandng of relatve trade-offs between varous sensng technologes. Note that the purpose of FPM s not to evaluate performance but to predct performance gven sensor sute and operatng condtons. For a sensor fuson system, a typcal ueston that could be asked would be what s the best achevable performance, and s t good enough? FPM wll be able to answer the frst ueston and f the answer s not good enough, a seuence of what f scenaros can be added for FPM to conduct new assessments. hose scenaros may nclude changng operatng condtons, such as SNR, geometry, revst rate, etc. of the exstng sensors or addng new sensors. he assessment results can then be used to better control sensors and allocate system resources. Internal to a fuson system, the FPM performance predcton can be used to check aganst current trackng/fuson performance. he results wll ndcate any potental flaws n the system and help to adjust system parameters, such as gatng logc, assgnment polcy, track prunng strategy, etc. he paper s organzed as follows: Secton descrbes the hgh level techncal approach; Sectons and 4 focus on dfferent aspects of the performance models; they provde both an overvew of the FPM system and the models behnd the establshment of the fuson performance model. Both knematcs and classfcaton models are descrbed. Secton 5 shows some experment results, and Secton 6 presents potental future work. echncal Approach he goal s to predct both knematc and classfcaton fuson performance. o do so, we need to understand the ndvdual sensor observaton model as well as how the fuson process takes place. o predct optmal performance bounds, a centralzed fuson archtecture s assumed. Our approach s based on the followng steps: () Wth each sngle sensor, develop ts observaton model, such as measurement error covarance, probablty of detecton (PD), false alarm probablty (Pfa), confuson matrx for classfcaton at feature or target ID levels, revst pattern, and regstraton error. () Consder the operatng concept of the fuson process, ncludng target dynamcs, error propagaton, asynchronous sensor revsts, fuson rules, etc. () For the knematc performance model, develop performance bounds based on a covarance analyss by propagatng the error wth smulaton or analytcs. We wll frst generate performance bounds by consderng a hgh level model wth deal condtons where perfect data assocaton s assumed. We wll then enhance the model fdelty to predct performance under other operatng condtons. (4) In the smulaton approach, we randomly smulate the sensor revsts based on a gven pattern and the detecton probablty (when the target s not detected, the nformaton matrx wll be replaced by a zero matrx,.e., predcton only, no update). We propagate the error covarance based on the target dynamcs (process nose covarance, stop and go, etc.) and sensor ualty (measurement nose covarance). Note that n the smulaton approach, the process of mperfect data assocaton can also be modeled by replacng the measurement covarance by the effectve measurement covarance. In the analytcal approach, we conduct a steady state analyss of the error covarance based on target dynamcs and sensor operatng condtons (ualty, revst rate, etc). When detecton probablty and data assocaton are mperfect, the nformaton matrx needs to be modfed and dscounted n the covarance propagaton. Knematcs Performance Model In the knematcs performance model we model sngle sensor operatng characterstcs such as measurement error, Pd (probablty of detecton), Pfa (probablty of false alarm), confuson matrx, revst rate, regstraton error, etc. We also consder the operatng concept of the fuson process ncludng target dynamcs, error propagaton, asynchronous sensor revsts, fuson rules, etc. Based on covarance analyss by propagatng the error usng smulaton or analytcs, performance bounds are developed for knematcs performance. We wll frst generate performance bounds by consderng a hgh level model wth deal condtons where perfect data assocaton s assumed. We wll then enhance the model fdelty and nclude a more realstc scenaro n order to predct performance under any operatng condtons.. Sensor uncertanty model We buld a sensor uncertanty model through the followng sensor operatng characterstcs to smulate the sensors nherent uncertanty:. Observaton error covarance: he error covarance could be statonary and gven a pror or dynamcally generated and provded by the smulator.. Revst pattern: he pattern could be determnstc where a constant revst rate s assumed or stochastc where an exponentally dstrbuted tme between arrvals s assumed.
. Detecton model: he detecton probablty could be statonary and gven a pror or dynamcally defned based on sensor target geometry, sensor operatng characterstcs such as SNR, etc.. arget dynamc model he target dynamc s assumed to be a two-dmensonal knematc model as gven n the followng euaton, x ( t ) ( t ) = F [ x( t ) x ( t ) y( t ) y ( t )] t t ( θ ) ( t ) F t t ( θ )' + Q ' = F t t t t ( θ ) ( θ ) x( t ) + w( θ ) () where Ft ( ) t θ s the x transton matrx between t and t, θ s a two-state (stop/go) Markov process represented by the transton matrx. It PSG PSG can be shown that the steady state of the process s PSG, [ Pr( θ = Go ) Pr( θ = Stop) ] = + PSG + PSG namely, one could specfy the process by a sngle P number GS, whch ndcates the average target + PSG stoppng probablty. Both the transton and the process nose covarance matrx are a functon of the tme dfference as well as the target state (stop/go),.e., F t t ( θ ) = Qt ( ) t θ = where = t t ( θ ). Fuson Process, yy yy he fuson rules are based on the nformaton matrx fuson concept. In Fgure, we assume there are two sensors, UGS (unattended ground sensor) and GMI and ther reports arrve n a gven random fashon. he gven covarance matrces n the fgure are assumed to be the poston error uncertanty at the tme of the report. he fuson process can be descrbed by the followng euatons, yy yy () FP FP FP ( t ) = ( t ) = ( t ) = UGS ( t ) [( Ft ( ) ( ) ( )' ( )) ( ) ] t θ FP t Ft t θ + Qt t θ +GMI t [( F ( θ ) ( t ) F ( θ )' + Q ( θ )) + ( t ) ] t t FP t t t t GMI Note here that the error covarances from dfferent reports are assumed to be condtonally ndependent. When they are not ndependent due to ether common pror nformaton or common process nose, a more precse fuson rule, such as MAP fuson can be used []. Fgure Fuson Process wth Asynchronous Sensors.4 Poston Error Predcton here are two ways to predct poston error performance, one s to use a smulaton approach, and the other s to use an analytcal approach. he smulaton approach combnes the sensor, target, detecton and fuson process models and reles on the smulaton of random asynchronous arrval of sensor reports to estmate the error covarance at any nstance of tme. On the other hand, the analytcal approach reles on the steady state flter concept (e.g., alpha-beta flter) to predct the steady state error covarance based on the gven target and sensor models. he latter approach assumes statonary target and sensor model parameters and does not allow tme-dependent sensor error covarance such as the ones could be generated from the smulator..5 Examples UGS ( t ) ( t ) t ) t ) GMI Here we apply four sensors (GMI, IMIN, SIGIN, and UGS) along wth ther sensor operatng characterstcs to two approaches, the smulaton approach and the analytcal approach. he followng fgures (Fgures -) show the average poston error (meter) for the two cases respectvely. For a smulaton tme of,, wth target process nose = 5., target stoppng rate., = yy GMI( GMI ( 4 four types of sensors are fused to predct poston error for a ground target. Fgure shows the results from the smulaton approach wth an average poston error of 4.68 (m). ()
Mantanng the same sensor types and operatng characterstcs parameters, we obtaned an analytcal poston error curve based on dfferent process nose as shown n Fgure. From ths curve, wth process nose = 5., we can see that the poston error s = yy approxmately.8 (m). When comparng the results of these two approaches, as expected, they are comparable. It s recommended that the analytcal approach should be used whenever feasble. However, n more complcated cases where the scenaro nvolves the stuatons such as dynamc sensor geometry, terran obscuraton, road network constrants, etc., t s more approprate to use the smulaton approach. assocaton, P CA ) of the optmal data assocaton algorthm n a m-dmensonal space can be approxmated ( C by m β + D m β FA) m P CA e, m+ C = π Γ, where ( ) m D m = Γ( m) Γ( m ), β s the normalzed target densty defned as the average number of objects n a onesgma regon n the measurement space, and β s the FA normalzed false alarm densty defned as the average number of extraneous objects n the same regon [6]. When the target detecton probablty ( P D ) s less than unty wth false alarms present, n a -dmensonal space, the performance of the optmal data assocaton can be ( β + β FA ) approxmated by PCA PD e. In order to assess the trackng accuracy under mperfect data assocaton, an effectve measurement error model was developed. It can be shown that n a -dmensonal space, the effectve ~ ~ measurement error s = R + (Q R) β exp( β ) R E where R s the measurement error covarance, Q s the track predcton error covarance, and the effectve object densty [6]. m ~ β = β + β s FA Wth the effectve measurement error covarance, we can replace the fuson euaton n euaton () by, Fgure Results of Poston Error Based on Smulaton Approach FP( t ) = ( t ) [( F ( θ ) ( t ) F ( θ )' + Q ( θ )) P ( t ) R ( t ] FP( t ) = t ) t FP t t t t + D E (4) 4 Classfcaton Performance Model Fgure Results of Poston Error Based on Analytcal Approach We also develop the model to enhance knematcs performance model fdelty by takng nto account Probablty of Detecton (PD), Probablty of False Alarm (Pfa), target densty, and mperfect data assocaton. It s known that the performance (probablty of correct As n the knematc performance assessment, there are two ways to predct the classfcaton performance. One s to use the smulaton approach where we randomly smulate the sensor revsts based on a gven pattern and detecton probablty. We recursvely update the classfcaton performance based on the Bayesnet model and the observatons. he classfcaton performance could ether be the confuson matrx or the probablty of correct classfcaton as defned later. It s ntally assumed that there s no error n data assocaton. We wll relax the assumpton later. In the analytcal approach, we derve the steady state confuson matrx and average the dagonal elements (probablty of correct classfcaton) to obtan the average probablty of correct classfcaton (Pcc). here are varous approaches to model and update classfcaton probablty. Our approach s to use the Bayesan network to model the relatonshp between target ID and varous levels of observables. Frst we defne some termnology.
() Confuson Matrx: Local confuson matrces are the ones based on sngle sensor feature level observatons; and they wll be used to compute the global confuson matrx whch s defned as the probablty of nferred target class gven true target class. () Percentage of Correct Declaraton: Percentage of track nstances where the hghest probablty class s the true class of the assocated target (when t s known) or the class declared by the track. () Average Probablty of Correct Classfcaton: Average probablty of the class correspondng to the true class of the assocated target or the class declared by the track. he Bayesan network has a fxed topology where the arcs ndcate ether the functonal relatonshps or probablstc relatonshps between nodes, and each node s a random varable n the model. he sensor models are specfed by the confuson tables whch wll be the condtonal probablty tables of the observable/evdence nodes gven ther parent nodes. Note that these tables could be sensor dependent as well as scenaro dependent (tme, geometry, etc.). he purpose of buldng the Bayesan network s to compute the updated track and the assocaton lkelhood gven an extrapolated track and a measurement n the classfcaton space. Computng the updated track state dstrbuton s rather straghtforward. Frst, substtute the gven extrapolated track state dstrbuton nto the pror probablty dstrbuton of the root node arget Classfcaton/ID,.e., the pror pdf of the target classfcaton s the current extrapolated target classfcaton pdf. For a gven measurement, attach t to the leaf node of the correspondng model, then execute a probablstc nference algorthm [] to compute the a posteror dstrbuton of the node arget Classfcaton/ID gven that measurement. he result s the pdf of the updated track and s the frst element to be computed. Computng the assocaton lkelhood between a gven track and a measurement s, however, more complcated. One dea s to use a smple two-step nference algorthm [4]. he dea s to predct the observaton dstrbuton usng forward nference gven a target ID, and then to compute the lkelhood based on the dfference between the predcted observaton and the actual observaton. For a partcular set of observatons, the frst step s to do forward nference from the root nodes for each possble combnaton of root node values to the observaton nodes. hs step can be done ether wth the smulaton method or an exact (eg., Juncton tree []) algorthm dependng on the network confguraton. In a network where all nodes are dscrete except for the leaf nodes, the exact algorthm can be used to compute the pror dstrbuton of observaton nodes gven each root node confguraton. However, n general, when a network contans non- Gaussan contnuous nodes, the smulaton method s the only way to do forward nference. he second step s to compute the lkelhood based on the observed evdence. If the observaton node s a dscrete varable, ths step can be done by selectng approprate entres from the pror pdf of the observaton node whch corresponds to the specfc observaton. If the observaton node s a contnuous varable, then frst of all, the pror pdf of the observaton needs to be represented approxmately by a un-modal or mult-modal gaussan dstrbuton. he lkelhood s then computed based on the degree of closeness between the actual observaton and the predcted observaton. In order to evaluate the classfcaton performance, wth the Bayesnet completely specfed, we can compute p ( y x) and p ( x y) where x s the target ID, and y s the sensor report. he confuson table for the specfc sensor can then be computed as, p x = x x = x ) = p( x = x y) p( y x = x ) (5) ( j y j Note that ths wll gve us a suare matrx where each row ndcates that f the target ID x = x s true, what s the probablty of nferrng t as x = x j gven the report from sensor S. he performance measure can then be defned y as the average correct classfcaton probablty, whch can be obtaned as, P = p( x = x x = x ) CC N where N s the total number of target types. When there s more than one sensor, the dea s smlar, namely, p x = x x = x ) = p( x = x y, y ) p( y, y x = x ) ( j y, y j (6) where y, y are the reports from two dfferent sensors and S. S y y As mentoned above, the classfcaton performance assessment was frst developed based on the assumpton that there was no error n data assocaton. In realty, the mpact of mperfect data assocaton due to hgh target densty and low sensor resoluton on classfcaton performance needs to be taken nto account. However, ths s a non-trval problem. One dea s to approxmate the performance by degradng the global confuson matrx wth the probablty of correct assocaton accordngly, namely, P E ( Obs. rue ID)
PCA Pr( Obs. rue ID) + ( PCA ) Pr( Unform Obs rue ID) (7) One nterestng research drecton s to develop steady state classfcaton performance. Smlar to the steady state knematc covarance matrx obtaned by the Kalman flter, one should be able to develop the steady state global confuson matrx gven sensor characterstcs. 5. Exprment Results he Bayesan network model was bult based on the target and feature taxonomy as well as varous sensor characterstcs (local feature level confuson tables). he model was constructed and the nference algorthm was mplemented usng MALAB BN toolbox [6]. A Bayesan network example for classfcaton s shown n Fgure 4. he example shows that wth the observatons from the three sensors (UGS, GMI, and Vdeo), we are able to dentfy the target (GAZ-66) wth very hgh confdence. In the current system, we are able to create both poston error covarance hstory and classfcaton probablty hstory for smulated scenaros under any operatng condtons. For example, Fgure 5 shows the fuson performance assessment of a smulated scenaro. he left wndow shows that there are nne dfferent sensors where fve of them detect more than one measurement on the target. he wndows on the rght show the postonal errors and classfcaton errors over tme of the target track. he fuson performance model provdes upper bounds of knematc and classfcaton performance assumng perfect data assocaton. In addton to the smulaton approach, we also bult and tested the analytcal steady-state performance model. In ths case, we mplemented the analytcal FPM model by ncludng the uncertantes such as detecton probablty, false alarm densty, and mperfect data assocaton. Fgure 6 shows a screen dump of the system. In ths demo system, we are able to vary up to two operatng condtons for target characterstcs and any combnaton of the four sensors. For example, n Fgure 6, we chose GMI and IMIN sensors whle we vared target process nose from to (m/s)^ and GMI revst rate from to seconds. he D chart on the rght shows the steady state performance (postonal error SD) under each combnaton of the operatng condtons. 6. Summary For a multsensor fuson system, t would be very useful to be able to characterze the relatonshp between sensed nformaton nputs avalable to the fuson system and the ualty of fused nformaton output. hs paper presents a fuson performance model (FPM) for a general multsensor fuson system. One of the purposes of developng FPM s to provde hgh level performance bounds gven sensor mx and sensor ualty. he assessment results can be used to better control sensors and allocate system resources. he FPM developed n ths paper ncludes both knematc and classfcaton components and focuses on the two performance measures dynamcally: postonal error and classfcaton error. he performance model s based on the Bayesan theory and a combnaton of smulaton and analytcal approaches. References [] J. Hoffman and J. Petty Fuson System Performance Estmaton Modelng, DARPA DBB fnal report, Lockheed Martn Astronautcs Operatons,. [] K. C. Chang, Zh an, Shozo Mor, and Chee-Yee Chong, MAP rack Fuson Performance Evaluaton, n Proc. Fuson, Washngton DC, July,. [] S. L. Laurtzen and D. J. Spegelhalter, "Local Computatons wth Probabltes on Graphcal Structures and ther Applcaton n Expert Systems", Journal Royal Statstcal Socety B, 5, 988. [4] K. C. Chang, Jun Lu, and Jng Zhou Bayesan Probablstc Inference for arget Recognton, Proceedngs of SPIE, Orlando, Aprl, 996. [5] Kevn Murphy; he Bayes Net oolbox for Matlab, Computng Scence and Statstcs,. [6] S. Mor, K. C. Chang, and C. Y. Chong, Performance Analyss of Optmal Data Assocaton wth Applcaton to Multple arget rackng, Multtarget-Multsensor rackng: Applcatons and Advances, Vol. II, Chapter 7, Edted by Y. Bar- Shalom, Artech House, 99.
Fgure 4. Bayesan Network Example for Classfcaton Fgure 5. FPM Smulaton Envronment Interface
Fgure 6. Steady State Knematc Fuson Performance Predcton