Data fusion in multi sensor platforms for widearea

Transcription

1 1 Data fuson n mult sensor platforms for wdearea percepton Ars Polychronopoulos, Member IEEE, Nkos Floudas, Angelos Amdts, Member IEEE, Drk Bank, Bas van den Broek Abstract there s a strong belef that the mprovement of preventve safety applcatons and the extenson of ther operatve range wll be acheved by the deployment of multple sensors wth wde felds of vew (FOV). The paper contrbutes to the soluton of the problem and ntroduces dstrbuted sensor data fuson archtectures- and algorthms for an effcent deployment of multple sensors that gve redundant or complementary nformaton for the movng objects. The proposed fuson archtecture s based on a modular approach allowng exchangeablty and benchmarkng usng the output of ndvdual trackers, whereas the fuson algorthm gves a soluton to the track management problem and the coverage of wde percepton areas. The test case s LATERAL SAFE sensor confguraton, whch montors the rear and lateral areas of the vehcle. Results show that wth the gven approach the system s able to mantan the ID of all objects n transton (an object enters a sensor s FOV) and blnd areas (no sensor coverage). Index Terms Data Fuson, Data assocaton, ADAS, lane change, lateral collson warnng I. INTRODUCTION L ATERAL SAFE s a subproject of the EU co-funded Integrated Project PReVENT [1] that ams at developng a cluster of safety applcatons that prevent lateral/rear area related accdents and assst the drver n adverse or low vsblty condtons and blnd spot areas. The man objectves of LATERAL SAFE as reflected to ts three applcatons are: - To develop s Lateral and Rear area Montorng (LRM) applcaton enhancng the drver s percepton and decreasng the rsk of collson n the lateral and rear area of the vehcle. - To develop a Lateral Collson Warnng (LCW_ applcaton that detects and tracks obstacles n the lateral and rear feld and warns the drver about an mmnent rsk of an accdent and - To develop a stand-alone Lane Change Assstance (LCA) Ths work s supported by the European Commsson under the project PReVENT sub-project LATERAL SAFE. A. Polychronopoulos, N. Floudas and A. Amdts are wth nsttute of Communcaton and Computer Systems, Athens, 9, Iroon Polytechnou St,, 15773, (phone: ; fax: ; e-mal: arsp@ccs.gr ). D. Bank s wth DamlerChrysler AG Research and Technology, Wlhelm-Runge-Str. 11, Ulm, Germany. Bas van den Borek wth the TNO Defense en Velghed, Oude Waalsdorperweg 63, P.O. Box 96864, 2509 JG Den Haag, Netherlands. system wth ntegrated blnd spot detecton assstng the drver n lane change maneuvers whle drvng on roads wth more than one lane per drecton. These applcatons and n general enhanced drver assstance systems have specfc requrements on envronmental percepton: wder feld of vew.e. more complementary sensors (lateral and rear area n ths case 270 deg); enhanced accuracy for longtudnal (rear n ths case) and lateral knematcs.e. more redundant sensors. Ths requrement s addressed n ths paper and a global soluton s gven n terms of archtecture and algorthm development. Sensor-level dstrbuted fuson was selected snce t s flexble n the number and type of sensors and therefore t allows addton, removal or substtuton of sensors or sensor systems wthout havng to alter the fundamental structure of the fuson algorthm. In addton, the processng at the sensor level before data entry nto the fuson processor reduces the computatonal load n the central fuson processor. Moreover, snce many commercal sensors provde only tracks and not raw data, track level fuson appears to be the rght soluton. Related work on mult sensor data fuson for preventve safety has been carred out n a seres of research actvtes such as the ARCOS project [2] wth a forward collson mtgaton combnng stereo vson and laser scanner, the EUCLIDE research project [3] whch s a forward collson warnng and vson enhancement applcaton usng far nfrared and mmw radar sensor. The same sensor combnaton was used n the PAROTO project [4], whle n the CARSENSE [5] project nformaton from a radar, vdeo sensors and a laser was fused. All above projects focus on redundant sensor nformaton for lmted coverage areas, thus not coverng the requrements set n the prevous paragraph. On the other hand, LATERAL, offers a useful test case, where three ndependent sets of sensoral systems are used ncludng, for the man applcatons, ten (10) dfferent sensors; a backward lookng long range radar (LRR), two trplets of short range radars n both sdes settng a homogeneous radar network (SRR)- and three cameras at the mrrors and at the rear wndsheld. In the proposed soluton, there s an ntal level of processng at each sensor set, wth trackng takng place n most cases, whle fuson s also conducted n the short range radar network case. The challenge of ths archtecture s to fuse effcently, n a successve level, the avalable

2 2 nformaton n order to provde the safety applcatons wth a substantal all around object descrpton. The structure of the paper s the followng: n Secton II, the LATERAL SAFE system archtecture s descrbed, and n Secton III the sensor level processng modules are brefly presented. A descrpton of the fuson algorthms that handle the four track arrays and produce the syntheszed percepton of the movng objects follow n Secton IV. At the same Secton, smulaton results of the fuson algorthm are ncluded. The Conclusons are provded n Secton V. II. MULTI-SENSOR ARCHITECTURE The fuson module n LATERAL SAFE has the man task of coordnatng, ntegratng data and provdng the output of the so-called percepton layer. The percepton layer s specfed as the layer n the LATERAL SAFE archtecture whch ntermedates between the sensor system and the applcatons. Ths layer gves a realstc representaton of the envronment and ams at enhancng the performance of sngle sensor systems provdng more robust output to the applcaton. Summarzng, the role of the percepton s to: - Carry out percepton enhancement" tasks ndependent of the applcaton - Descrbe n a formal way the envronment and the traffc scenaro - Support LATERAL SAFE functons under request - Act as a gateway between sensor systems and applcatons wth well defned nterfaces and I/O protocols. LRR + LRR Processng SRR1... SRRn Camera1 Camera2 SRR Processng SVIP Processng LRR Output SRR Output SVIP Output FUSION FUS Output LCA LCW LRM LCA Output LCW Output Output LRM LRM Output Output HMI are depcted n the system archtecture n Fg. 1. The fuson algorthm generates global fused objects and fulfls the generc percepton layer objectves that are summarsed above. Regardng the sensors avalable n the LATERAL SAFE confguraton, a test car s equpped wth: (a) Long Range Radar (LRR) - 2 nd generaton long range radar by Bosch whch transmts a lst of through CAN-nterface to the percepton layer (b) Short Range Radar Network, whch s a data fuson trackng system that uses nformaton from short range radar samples from M/A-COM Tyco Electroncs and processes the raw measurement data to generate a unfed lst of lateral objects at both sdes of the vehcle and (c) Vson systems whch are usng three CMOS INKA-NSC640PG cameras by Aglaa GmbH (two n the sde mrrors and one n the mddle of the rear wndow). III. SENSOR LEVEL PROCESSING A. Long Range Radar trackng The measurements avalable on CAN bus by the LRR sensor are: Dstance [m], Transversal Dstance [m] and Radal Velocty (relatve) [m/s]. The output nformaton of LRR trackng sent to the fuson module s a tracked object lst ncludng: number of objects, X poston [m], Y poston [m], relatve X velocty [m/s], relatve Y velocty [m/s], relatve X acceleraton [m/s 2 ], relatve Y acceleraton [m/s 2 ], object ID, estmaton error covarance matrx. Smlar nformaton s provded by all the other sensor processng modules. The LRR trackng algorthm s llustrated n Fg. 2; t s a trackng system talored for automotve applcatons separated n three (3) man sub-modules: data assocaton, track management and flterng & predcton. Vehcle VCAN Output Percepton Layer Applcaton/HMI Layer Fgure 1: System archtecture The Percepton Layer s always actve ( ON ) and t always montors and models the envronment. The percepton layer modules that produce track arrays to the fuson module are: Long Range Radar trackng (LRR), Short Range Radar network sgnal processng (SRR) wth separate track arrays for each sde of the vehcle and the Syntheszed Vson Image Processng (SVIP) processed data. The Percepton layer modules and ther role n the LATERAL SAFE applcatons Fgure 2: LRR trackng algorthm

3 3 The basc flterng approach n the LRR trackng s the Extended Kalman Flter (EKF), as defned n [6]. A sngle Constant Acceleraton (CA) model s used for moton modelng as t s proved to be adequate; regardng data assocaton two workng modes are selected; Global Nearest Neghbor (GNN) and Jont Probablstc Data Assocaton (JPDA), usng 1-to-1 and N-to-1 measurements to track assgnment. The standard 2D assgnment problem s solved va the aucton algorthm [7]. Accordng to the results of the assgnment, f a track has been assocated wth a measurement t has a ht n current scan; otherwse t has a mss. The values of hts and msses are stored for each track and are attached to t. The track conssts of the state vector, the covarance matrx and the ID vector. For confrmaton and deleton of tracks, an ad hoc rule for track deleton and confrmaton s followed; when a new track s confrmed or an exstng track s deleted, the tracked object lst s updated accordngly. B. Short Range Radar trackng SRR network sgnal processng s based on sensor arrays nstalled on both sdes of the vehcle. The sensors are orentated perpendcular to the sde of the car. A sensor model has been developed, whch consders dstance detecton range, dstance accuracy, bearng detecton range, bearng accuracy (for dfferent bearng ranges) of the employed sensors as well as reflecton propertes of radar waves. The raw sensor data (measurements) are fltered and sngle-sensor-mult-targettrackng s realzed for each sensor ndependently by applyng an EKF wth mxed coordnates. A GNN approach s used to assocate new measurements wth exstng tracks. Track management s carred out smlarly to the LRR trackng algorthm. The sngle-sensor-tracked-targets obtaned by all sensors belongng to the same array are ntegrated by multsensor data ntegraton. Ths process nvolves gatng accordng to the gven uncertantes of the sensors. Whenever the gates of n 2 sngle-sensor-tracked-targets ntersect, the n sngle-sensor-tracked-targets wll be assocated wth each other and wll be merged to an ntegrated target. The weghted average of poston and accuracy s calculated, usng the accuracy as weghtng factor. Based on the ntegrated targets, mult-sensor-mult-target-trackng s acheved by applyng another Kalman flter. For ths purpose, the approach of sngle-sensor-mult-target-trackng can be adapted. A Converted Measurement Kalman Flter (CMKF) as descrbed n [8] s used n order to get small errors due to the transformaton from polar coordnates nto rectangular coordnates. For the mplementaton of the flter a Constant Velocty) CV model s chosen. The GNN approach s also used to assocate new ntegrated targets wth the exstng tracks. C. Syntheszed Vson Image Processng The vson system conssts of several cameras mounted n the sde mrrors and behnd the rear wndow. Three operatonal modes can be chosen: - Blnd spot mode, wth mono vson processng of the mrror cameras - Full mono vson, wth processng of all cameras - Mono and stereo vson The stereo vson provdes lateral locaton and dstance for objects n the regon, where the felds-of-vew of cameras overlap. Mono vson provdes an ndcaton of presence of vehcles n the blnd-spot regon, wth less accuracy n locaton. The combned coverage of the vson sensors covers a large area around the car, complementng the LRR and SRR sensors n some areas, and overlappng wth them n others. The mono vson algorthm for detecton of vehcles from a sngle camera s based on detecton of objects not belongng to a background. When a vehcle s correctly detected, the dstance may be roughly estmated. The mono vson processng also ncludes lne detecton, provdng nformaton about lanes. Dstance can be computed from the dsparty between two correspondng ponts n the stereo mages. In order to fnd the dspartes, ponts from the left mage must be matched to those of the rght mage. Ths can be smplfed by determnng the eppolar geometry [9] of the stereo rg, and convertng mages so a pont n a scene s mapped to the same lnes n both cameras. The dfference n locaton on these lnes, the dsparty, ndcates the dstance to the pont. A calbraton s needed to determne the eppolar geometry, ntrnsc parameters (such lens dstorton) and extrnsc parameters (relatve poston and orentaton of the cameras). For the stereo vson system n the car, a sngle calbraton would be done after nstallaton. Durng use, the relatve camera postons and orentaton change, due to for example flexblty n door and mrror. Therefore, an onlne recalbraton s done, estmatng these changes. In sparse stereo algorthms, detectors are used to fnd features such as edges or corners, and only these are matched. Ths approach s computatonally attractve because only a lmted set of features has to be matched. In dense stereo vson approaches all pxels are matched, allowng detecton of objects from the dsparty. The man drawback of ths approach s the ncreased computatonal cost. However, an algorthm was developed that allows a real-tme dense dsparty algorthm to run wthout specal hardware [10]. For detecton use s made of the dsparty estmaton to detect the road surface, and detected pxels that do not belong to ths surface. Ths can be done usng V-dsparty [11], whch s obtaned by computng a hstogram of dsparty values for each mage lne. The road surface can then be detected usng the assumpton that n a gven mage lne, the most common dstance s that to the road surface. All dspartes above ths lne correspond to objects above the road. By detectng clusters n ths data, vehcles at dfferent dstances can be obtaned. Detectons are combned n a tracker. Ths allows the combnaton of the outputs of mono- and stereo vson processng, provdes a more stable estmaton of the locaton of vehcles, and provdes velocty and accuracy nformaton.

4 4 The tracker uses a lnear velocty model and Kalman flter. After trackng a velocty estmate of detected objects s avalable, as well as an accuracy estmate n the form of a covarance matrx. The track nformaton s used to flter tracks, reducng false alarms. system. IV. DATA FUSION A. Algorthm The fuson algorthm handles four track arrays, namely a LRR track array; a SRR left track array; a SRR rght track array and a SVIP track array and ncludes: - The synchronzaton of track arrays to a common tme reference and the propagaton of track nfo to ths tme - The transformaton n a common spatal base accordng to the vehcle centered chosen coordnate system. Then the tracks are separated to the tracks that are lkely to be fused and those that are not. Ths s acheved due to the known and predefned FOVs of the sensor systems and possble complementartes and redundances. Track-to-track assocaton follows for the case of fuseable tracks, together wth the fused object management (ntalzaton, confrmaton, and deleton) and the fuson update appled to the assocated track arrays. Fgure 4: Synchronzaton procedure of LATERAL SAFE track arrays The fuson algorthm dvdes the fuson problem to subproblems accordng to the regon of each object follows. In the case of the LATERAL SAFE confguraton, seven areas are defned. These areas are depcted n Fg. 5, wth dark and lght green are the SRR areas, wth lght blue are the SVIP areas and wth red the LRR rear area. The areas where fuson can be applcable are 2 and 6 between SVIP and SRR left and rght respectvely and area 4 between LRR and SVIP at the rear. Each segment s a certan sensor s FOV based on the sensor wth the narrowest FOV - that s ncluded nsde the FOV of a sensor wth wder coverage. Ths s done for the SRR objects n the mrror cameras FOV, and the SVIP tracks nsde each radar s FOV. Ths process reduces the tme delay of track to track assocaton. Fgure 5: Areas of sensor survellance Fgure 3: Fuson algorthm data flow The tme cycle for the LRR tracked objects s 100ms, ms (a value about 40ms s achevable) for the SVIP tracked objects: and 40ms for the SRR tracked objects. The fuson algorthm could follow the step of LRR tracked objects and update the other track arrays to acqure a common tme. The process s llustrated n Fg. 4. State vectors and covarance matrces are extrapolated to the fuson tme after calculaton of all tme delays. The track arrays are easly transformed to the vehcle centered common coordnate In the case of fuson between two tracks data assocaton and fuson update s carred out usng the cross-covarance matrx method [12]. Let two (2) tracks, j have state vectors xˆ, ˆ and covarance matrces P, P respectvely, the state x j dfference and the cross-covarance matrx between the estmaton errors of tracks are defned: ~ x = xˆ xˆ j j j j [ P ( l, m) P ( l, )] 1/ 2 P ( l, m) = ρ m (1) where l, m are the elements of covarance matrces one-toone, and ρ s the correlaton coeffcent. A statstcal dstance between tracks, j usng the cross-covarance s defned as: d 2 j = T [ P + P j Pj Pj ] ~ xj ~ T 1 xj After the calculaton of the dstance, the assgnment j (2)

5 5 problem can be solved n a smlar manner as the 2D data assocaton n a trackng problem; accordng to the performance of sensor level trackng t can be solved qute explctly n most cases or, f necessary, usng the aucton algorthm for 1-1 assgnment. The fused object update s done usng the Covarance Intersecton (CI) method [13]. CI method deals wth the problem of nvald ncorporaton of redundant nformaton. The fused state and covarance are calculated: P f [ wp ( ) ] w 2 P x + ( w) P = P [ w x 2 ] x f = P f (3) where w n the nterval [0,1]. The fnal step of fuson n the LATERAL SAFE s that of the mantenance of the global fuson ID; for that reason the tracked object ID s used. The tracked objects of the four track arrays are assgned to one of the seven areas as defned n Fg. 6. Each tracked object can be fused wth another tracked object of another track array f t belongs to areas 2, 4 and 6; otherwse tracked objects that are not fused wth others are smply added n the fused objects array. Every fused object observed for the frst tme obtans a fuson ID whch s accompanes wth the ntal track(s) IDs that produced t. If the track(s) ID concde wth one of the exstng tracked IDs the fuson ID does not change. Ths method requres from each track array to have fxed and dfferent IDs for each tracked object. 500 scans of the scenaro s llustrated n Fg. 6. The smulaton scenaro ncludes three target vehcles and the egovehcle; one target vehcle appears at the rear, accelerates and overtakes the ego vehcle; a second target vehcle moves at the rght sde n parallel to the ego-vehcle and a thrd appears from the rght sde and s overtaken. For these three vehcles the performance of the LATERAL SAFE fuson n smulaton condtons s tested. The true poston and velocty, as well as the system scans of appearance and dsappearance n sensor FOVs are a pror know; the desred performance of fuson algorthm s the elmnaton of estmaton errors on one hand and f the mantenance of the fuson IDs for these objects for the whole duraton of ther presence In the fgure 7, the 100 Monte Carlo runs estmaton performance for the x-y poston of target vehcle 1 when n areas of LRR and SVIP at the rear s depcted. In both cases fuson s found to be superor to both radar and camera trackers. The Root Mean Square Error (RMSE) of measurements s shown n these fgures. Smlar results have been establshed for the other target vehcles and for the velocty estmates. RMS [m] Poston X Meas-R Meas-C Est-R Est-C Fused scan Poston Y Meas-R Meas-C Est-R Est-C Fused RMS [m] Fgure 6 : The fuson performance scenaro B. Results The challenges that the descrbed fuson algorthm has to confront are (a) the mproved estmaton compared to sngle sensor processng and (b) the mantenance of a global ID for the fused objects. For an ntal testng of the overall algorthm a representatve smulaton scenaro was mplemented. The scenaro conssts of the host vehcle and three vehcles movng around t. The smulaton ncludes the specfc sensors that the actual LATERAL SAFE applcaton wll ntegrate and uses confgurable parameters for the sensor characterstcs, measurement parameters, uncertantes, sensor locatons, ranges, drectons and FOVs. The cumulatve plot of a total of scan Fgure 7: The fuson performance In table 1, we focus on the performance of ID mantenance task. In order to evaluate such performance, the followng measures are defned. The probablty of successful performance of the fuson algorthm under the assumptons of the specfc scenaro s defned: P =1 P P P (4) succ mss false dupl mss = probablty of falure to dentfy an exstng object,

6 6 false = probablty of dentfyng a new false object dupl = probablty of duplcatng an object The other ndex s the ID mantenance defned as: P ma = 1 R. I. total (5) nt The ID mantenance evaluates the false re-ntalzaton (RI = the number of false re-ntalzatons) of a track per total number of objects; the optmal case s R.I. = 0 and the worst s R.I. = 962 (no ID s preserved). 962 s the maxmum number of expected fused objects to be detected n the scenaro. Alternatvely, consderng that three objects IDs are always expected, on average 2.33 excessve IDs for the case of no fuson and 1.67 and 1.37 more IDs for the two fuson cases are obtaned. In Table 1, results are gven the no fuson (NO) case - data of four trackers - and the fuson method presented (LS-1); LS-2 s usng the same algorthm wth an addtonal check to elmnate the duplcaton of fuson IDs. Table 1: Fuson performance FUS MISS FA DUPL F- REI SUCC MAIN NO LS LS All four trackng systems of the smulatons are GNN-EKF trackers as were descrbed n prevous sectons. The sgnfcant problem of false unnecessary objects when usng just the output of the trackers s elmnated usng the fuson approach descrbed here. V. CONCLUSIONS The paper proposed a generc soluton for data fuson problems, where applcatons mpose requrements for wder and more accurate percepton. The soluton s based on dstrbuted fuson archtectures and s appled on the system confguraton of LATERAL SAFE applcatons, whch montors the lateral and rear area of the ego-vehcle. The ndvdual trackers are evaluated usng recorded real-data, whle the fuson algorthm s evaluated through Monte Carlo smulatons. The next step s the recordng of synchronzed data, whch s planned for the begnnng of January, and wll be used for the fnal evaluaton of the algorthms. All processng modules and the fuson algorthm wll be ntegrated n a mult-sensor platform n a test car (provded by Fat Research Centre) and wll run n real-tme provdng a map target vehcles to the LATERAL SAFE applcatons. The algorthm can be appled to other confguratons and applcatons. In the near future, t wll be extended n order to cover N-D track assocaton tasks usng optmzaton theory. The goal s that the algorthm s able to be reconfgurable and react automatcally wth dfferent sensor confguratons. In the fnal paper, more results wll be ncluded usng synchronzed data recordngs. REFERENCES [1] LATERAL SAFE webste: [2] R. Labayrade, C. Royere, D. Aubert, A Collson Mtgaton System usng Laser Scanner and Stereovson Fuson and ts Assessment, Proc. of 2005 IEEE Intellgent Vehcle Symposum, Las Vegas, Nevada, USA, June 6-8, 2005, pp [3] A. Polychronopoulos, U. Scheunert, F. Tango, Centralzed data fuson for obstacle and road borders trackng n a collson warnng system, Proc. of the ISIF 7th Internatonal Conference on Informaton Fuson, Stockholm, Sweden, 28/06-01/ , pp [4] C. Blanc, L. Trassoudane, Y. Le Gulloux, R. Morera, Track to track fuson method appled to road obstacle detecton, Proc. of the ISIF 7th Internatonal Conference on Informaton Fuson, Stockholm, Sweden, 28/06-01/ , pp [5] C. Coué, Th. Frachard, P. Bessère and E. Mazer, Usng Bayesan Programmng for Mult-Sensor Data Fuson n Automotve Applcatons, Proc. of 2002 IEEE Intellgent Vehcle Symposum, Versalles, France, June 18-20, [6] Y. Bar-Shalom, X. Rong L, T. Krubarajan, Estmaton wth Applcatons to Trackng and Navgaton, Theory Algorthms and Software, John Wley & Sons Inc., New York, [7] D. Bertsekas, Aucton Algorthms for Network Flow Problems: A Tutoral Introducton, Journal of Computatonal Optmzaton and ts Applcatons, May [8] D. Lerro, Y. Bar-Shalom, Trackng wth debased consstent converted measurements versus EKF, IEEE Trans. Aerospace Electronc Systems 29 (3) (July 1993) [9] M. Pollefeys, Self-Calbraton and Metrc Reconstructon from Uncalbrated Image Sequences, PhD Thess, Faculty of Appled Scences, Unversty of Leuven, Belgum, May [10] Wannes van der Mark, Daru M. Gavrla, "Real-Tme Dense Stereo for Intellgent Vehcles", to appear n IEEE Transactons on Intellgent Transportaton Systems, [11] R. Labayrade, D. Aubert, J-P Tarel, Real Tme Obstacle Detecton n Stereovson on Non Flat Road Geometry Through V-dsparty Representaton, Proc. of IEEE Intellgent Vehcle Symposum, June 18-2, Versalles, France [12] S. Blackman and R. Popol, Desgn and Analyss of Modern Trackng Systems, Artech House, Boston, [13] S. Juler and J. Uhlmann, General Decentralzed Data Fuson wth Covarance Intersecton (CI), Handbook of Multsensor Data Fuson, 2001, edted by D. Hall and J. Llnas, Chapter 12.