A Framework for Applying Point Clouds Grabbed by Multi-Beam LIDAR in Perceiving the Driving Environment

Transcription

1 Sensors 215, 15, ; doi:1.339/s Aricle OPEN ACCESS sensors ISSN A Framewor for Applying Poin Clouds Grabbed by Muli-Beam LIDAR in Perceiving he Driving Environmen Jian Liu 1,2, *, Huawei Liang 2, Zhiling Wang 2 and Xiangcheng Chen 1 1 Deparmen of Auomaion, Universiy of Science and Technology of China, Hefei 2326, China; chenxgcg@usc.edu 2 Insiue of Applied Technology, Hefei Insiues of Physical Science, Chinese Academy of Sciences, Hefei 2326, China; s: hwliang@iim.ac.cn (H.L.); zlwang@hfcas.ac.cn (Z.W.) * Auhor o whom correspondence should be addressed; fdlj@mail.usc.edu.cn; Tel./Fax: Academic Edior: Felipe Jimenez Received: 23 June 215 / Acceped: 24 Augus 215 / Published: 31 Augus 215 Absrac: The quic and accurae undersanding of he ambien environmen, which is composed of road curbs, vehicles, pedesrians, ec., is criical for developing inelligen vehicles. The road elemens included in his wor are road curbs and dynamic road obsacles ha direcly affec he drivable area. A framewor for he online modeling of he driving environmen using a muli-beam LIDAR, i.e., a Velodyne HDL-64E LIDAR, which describes he 3D environmen in he form of a poin cloud, is repored in his aricle. Firs, ground segmenaion is performed via muli-feaure exracion of he raw daa grabbed by he Velodyne LIDAR o saisfy he requiremen of online environmen modeling. Curbs and dynamic road obsacles are deeced and raced in differen manners. Curves are fied for curb poins, and poins are clusered ino bundles whose form and inemaics parameers are calculaed. The Kalman filer is used o rac dynamic obsacles, whereas he snae model is employed for curbs. Resuls indicae ha he proposed framewor is robus under various environmens and saisfies he requiremens for online processing. Keywords: dynamic obsacle modeling; muli-beam LIDAR; muli-feaure ground segmenaion; road curb modeling

2 Sensors 215, Inroducion Environmen percepion is a ey research area as a source of informaion flow in developing unmanned ground vehicles (UGVs). Wihou an accurae and quic undersanding of he driving environmen, a vehicle is unable o mae he righ decisions when i moves a a high speed. Road curbs and dynamic road obsacles are he mos imporan road elemens in he driving environmen. Road curbs indicae he road area for a vehicle, whereas road dynamic obsacles specify he areas o avoid. To obain quic and accurae undersanding of he driving environmen, various sensors such as cameras, sereo vision, infrared cameras, and 2D sequenial lasers are employed o perceive he environmen. A common problem in using hese sensors is heir limied descripion of he environmen. Image processing has been sudied inensively and applied exensively in modeling he driving environmen o deec lanes, road boundaries, raffic lighs, ec. [1 8]. However, unlie ha of oher sensors, he informaion provided by a camera is resriced o a cerain view direcion ha covers a narrow field of view (FOV); moreover, disance informaion is lacing. Cameras are also suscepible o changing ligh condiions. To address he lac of disance informaion provided by ordinary cameras, sereo cameras have been developed and used in auonomous driving o address he lac of disance informaion in ordinary cameras [9 11]. However, sereo cameras have a narrower FOV han ordinary cameras. To overcome his ligh-limied disadvanage, sudies on he applicaion of infrared cameras have gained significan aenion because of he capabiliy of hese cameras o deec obsacles and heir insensiiviy o illuminaion [12 17]. However, he high price and low resoluion of infrared cameras limi heir applicaions. Meanwhile, vehicles are ypically equipped wih a 2D sequenial laser o deec obsacles in a cerain direcion under any weaher condiion [18 22]. However, he sparse informaion offered by a 2D sequenial laser is insufficien for a vehicle o mae is own driving decision. To exend 2D and 2.5D o 3D, a muli-beam LIDAR is employed o replace he 2D sequenial laser ha can only provide poins in a fixed pich angle. Alhough considerable research has been conduced on 2D and 2.5D percepion, only a few researchers have addressed problems in 3D percepion hus far. Pieces of 2D and 2.5D informaion are ypically represened in he form of an image, whereas a poin cloud is adoped in 3D percepion. The poin cloud was firs used in remoe sensing [23 26] o model errain. However, he mehod developed for remoe sensing is unsuiable for UGVs because of wo reasons. Firs, he densiy of he poin cloud used in remoe sensing is differen from ha of he poin cloud used in UGVs. A daa se is considered dense if he conneciviy of scanned surfaces can be capured wih he conneciviy of non-empy cells (i.e., cells wih a leas one daa poin), whereas empy cells exiss in he sparse poin cloud. The poin cloud used in remoe sensing is he dense poin cloud, whereas wha employed in UGVs is he sparse poin cloud; hus, daa are sored and processed differenly [27]. Second, real-ime requiremens for remoe sensing are differen and significanly lower from hose for UGVs. One of he mos ime-consuming aspecs in remoe sensing is deermining he closes poin in a poin cloud because he sored poins are unorganized. To overcome his problem, poin srucure should be considered for differen ypes of sensors. The sensor adoped in his sudy is he Velodyne HDL-64E LIDAR, which is a new ype of 64-beam LIDAR ha is exensively used in UGVs. Wih is 36 horizonal FOV by 26.8 verical FOV, 5 15 Hz user-selecable frame rae, and over 1.3 million poins per second oupu rae, he Velodyne HDL-64E LIDAR can model he 3D environmen in a poin cloud. Various sudies on he Velodyne HDL-64E LIDAR have been presened in lieraure. Some

3 Sensors 215, conceps in image processing, e.g., model maching, have been inegraed ino LIDAR daa processing. Pascosl e al. [28] used he superquadrics fiing mehod o segmen and model an obsacle simulaneously, whereas a plane model was applied in [29] o fi he drivable area. Range scan lielihood models were sudied in [3] by direcly considering he range parameer. However, he resuls of our experimen deermined ha he poins obained from he LIDAR daa of a cerain objec varied wih he relaive posiion of he vehicle o he deeced obsacle, paricularly when he obsacle was far from he vehicle. In his case, only a few poins were projeced ono he obsacle, which caused he mach o fail. Thus, he presen sudy focuses on exracing local feaures from raw daa using muli-line LIDAR, in which poins belonging o he same obsacle are grouped ogeher afer he poins ha are projeced ono he ground are removed. In conras o sensors, e.g., Ibeo, which allow easy ground filering by collecing four parallel horizonal scan lines and maring he readings ha liely come from he ground [15], he daa from he Velodyne LIDAR conains poins projeced ono he road surface. Ground segmenaion mehods are caegorized based on he organizaion of a poin cloud or he uilizaion of he informaion conained in a pace. In he firs caegory, he viciniy informaion in raw daa is no considered and only he posiions of he poins are obained. Azim e al. [31] deermined ha a praical means o idenify a dynamic obsacle was o deec he change in occupancy in an ocree. Meanwhile, a 1D Gaussian process (GP) regression wih a non-saionary covariance funcion was used o disinguish he ground poins or obsacles in each segmen of a polar coordinae sysem in [32]. Excellen resuls based on he viciniy informaion in he pace obained from he Velodyne LIDAR were observed afer he 24 and 25 Grand Challenges and he 27 Urban Challenge held by he Defense Advanced Research Projecs Agency o boos he developmen of UGVs. Von Hundelshausen e al. [33] proposed an obsacle deecion mehod based on he differen values of poins locaed wihin he same grid cell produced by a single beam. This mehod was also applied in [34]. The number of poins projeced ono he same grid and he heigh difference in he same grid were considered in [35]. Heigh difference was also employed in [36 38], wih he addiion of he range comparison reurned by wo adjacen beams presened in [21]. Moosmann e al. [36] projeced a poin cloud ono a cylinder whose axis was he roaional axis of he scanner; he local convexiy crierion was applied o segmen he ground. In our experimen, various feaures were esed and a conclusion was presened, ha is, muli-feaures wih a loose hreshold should be considered o address challenges in various environmens. Afer an obsacle is deeced, he dynamic obsacles and road curbs are raced. Tracing muliple dynamic objecs is a complex problem ha is generally divided ino wo pars: daa filering and daa associaion. Filering is he sequenial esimaion of he sae of a dynamic objec. This process is ypically performed using Bayesian filers and requires a specific moion model o predic he posiions of raced models in an environmen. Afer predicing he posiions of exising racs, daa associaion is performed o assign observaions o exising racs. Alhough he framewor is he same as he one previously menioned, deails vary. A bounding box was employed o classify he characerisics of he obsacle, and he global neares neighbor was applied in daa associaion o predic obsacles in [31]. The Junior [37] raced an obsacle by idenifying he area where changes occurred; a se of paricles were hen iniialized as possible objec hypoheses o implemen recangular objecs wih various dimensions a slighly differen velociies and locaions o rac he dynamic obsacle. Anoher proposed approach was grouping he classified obsacle range reurns ino local line feaures ha are raced across consecuive scans using a muliple-hypohesis Kalman filer [39]. In road curb racing, wo ypical Bayesian mehods

4 Sensors 215, were combined in [21], namely, he ineracing muliple model-probabilisic daa associaion filer approach. Anoher wor [4] focused on he paricle filer. A framewor for deecing and racing road curbs and dynamic obsacles is presened in his sudy. Firs, ground segmenaion is performed wih dynamic obsacles clusered simulaneously by combining muli-feaures from boh he poin cloud and he obsacle grid map generaed from he poin cloud. This process was esed robusly in various urban and rural environmens. A general approach o deec obsacles is ground segmenaion followed by obsacle clusering. However, a novel mehod is presened in he curren wor wherein obsacles are clusered during ground segmenaion, which reduces ime consumpion. In addiion, he feaures exraced from he Velodyne LIDAR raw daa are sudied comprehensively in his secion. To our nowledge, his wor is he firs aemp o conduc such a sudy. Second, geomeric parameers are obained separaely for dynamic obsacles and road curbs. Local informaion is applied o dynamic obsacle deecion using he Karhunen Loeve ransformaion, whereas global informaion is applied o road curb deecion using disance ransformaion. Third, he processes involved in he racing procedures for dynamic obsacles and road curbs are compleely differen. A Kalman filer is employed for dynamic obsacles, whereas road curbs are fied using he snae model. The Kalman filer has been proven o be a minimum-variance sae esimaor for linear dynamic sysems wih Gaussian noise and he bes linear esimaor for non-gaussian noise [41], which is suiable for his wor. Moreover, he improved real-ime performance of he proposed approach compared wih oher Bayesian filers, e.g., paricle filer, maes i applicable o high-speed unmanned vehicles. The snae model is seleced for is capabiliy o combine local curvaure informaion wih overall coninuous informaion. To our nowledge, racing and deecing dynamic obsacles and road curbs are performed separaely in previous wors. By conras, hese wo processes are combined in he curren wor. In he proposed framewor, previous informaion can be applied o deec obsacles. The succeeding porions of his paper are organized as follows: The dynamic obsacle and road curb deecion process are described in Secion 2. The process of racing road curbs and dynamic obsacles is discussed in Secion 3. The experimenal procedures are presened in Secion Deecing Road Curbs and Dynamic Obsacles 2.1. Ground Segmenaion As described earlier, he Velodyne LIDAR provides a comprehensive descripion of he ambien environmen. I was employed in our experimens as follows. The Velodyne LIDAR was mouned on he es car, called Inelligen Pioneer, and he direcion of he vehicle was mared as he saring direcion from which 64 poins would be sampled from he 64 lasers every.2. Thus, a poin cloud frame ha consised of D poins ha described he all-around car environmen would be obained afer one spin of he Velodyne LIDAR. The coordinaes of he poins were ranslaed from he polar coordinae sysem ino he Euclidean coordinaes, wherein he up direcion was se as he z axis and he forward direcion was se as he y axis. To uilize he informaion in he raw daa sen by he Velodyne LIDAR, he poins were sored in an D marix called Clouds. In his marix, he column represens a circle of poins generaed by one laser in one spin, whereas he row represens 64 poins generaed by 64 lasers in one roaing posiion. The 18 poins generaed by one laser in a

5 Sensors 215, single spin ha was projeced ono he fla horizonal ground would form a circle. By conras, he 64 poins generaed by he 64 lasers in a fixed roaing posiion would form a sraigh line. The esed car and he poin cloud frame obained using he Velodyne LIDAR are shown in Figure 1. (a) (b) Figure 1. (a) The es car; (b) A poin cloud frame. Based on he preceding analysis, road curbs or dynamic obsacles can be regarded as consising of poins ha are no in he posiion hey should have been if projeced ono a fla horizonal plane. If he poins are projeced ono such a plane, he poins will form concenric circles, and changing he posiions of he poins of he obsacles will affec local geomeric characerisics and preven concenric circles from forming. Tha is, deecing obsacle poins is modeled as removing poins projeced ono a road surface, he local geomery of which is similar o hose projeced ono a fla horizonal plane. However, a road surface is no an enirely horizonal plane and he effecs of various lasers are differen because of varying scan ranges; hence, segmening ground poins by using only a single characerisic is difficul. When only a single characerisic is applied, a sric hreshold causes lea deecion of impercepible obsacles, whereas a loose hreshold resuls in false deecion, paricularly in field environmens where road condiions are complex. The principle behind our wor involves applying various feaures and inensively esing each feaure using a relaively loose hreshold. Given ha he scan frequency is 1 Hz and he local feaures are only affeced by neighboring poins, which are generaed nearly simulaneously, he ego-moion of he vehicle has minimal influence on he local feaure. The feaures applied in his wor are described in he following secions Change in Radius beween Neighboring Poins in One Spin The mos significan feaure for a circle formed by one laser in one spin is he nearly similar disances beween he LIDAR and he poins. However, he disance will change significanly when he laser beam is bloced during roaion. Thus, he raio beween he radii of neighboring poins in one circle can be designaed as a feaure. If he raio is wihin [1 δ,1 + δ ], hen he poin is designaed as a road surface poin and is removed. 1 1

6 Sensors 215, Deecing Broen Lines This mehod exracs line segmens from he raw daa obained from he sensor in he polar coordinaes. The line segmens are classified ino road and obsacle segmens. A deailed descripion of his mehod is provided in [2]. This feaure is applied o deec only sraigh lines alone as a supplemen of he poins filered by oher feaures. The poins beween wo broen lines are fied wih a line model. The poins will be designaed as non-ground if he line model fis well. This feaure is paricularly applicable o boundary poins wih a sraigh model, e.g., sraigh road curbs, vehicle edges, ec. However, i will fail when applied o curved boundaries Tangenial Angle As menioned earlier, a circle will form if he poins generaed by he same laser in one spin are projeced ono a fla horizonal plane. However, his process will no occur if an obsacle exiss. This siuaion is illusraed in Figure 2. (a) (b) Figure 2. (a) A sample poin cloud; (b) he poins formed by one laser in one spin. The figure depics a driving environmen on an urban road whose boundaries are formed by a parerre. Figure 2b shows a porion of Figure 2a. Hence, Figure 2b is formed by one laser in one spin, whereas Figure 2a is formed by a oal of 64 lasers in one spin. As shown in Figure 2b, siuaion varies according o projecion posiion. Based on his figure, if a poin, e.g., poin P, is projeced ono a road surface, hen he angenial angle formed by is radial direcion, i.e., OP, and is angenial direcion, which is expressed by wo of is symmerical neighboring column poins in Clouds, i.e., MN, is nearly perpendicular. Oherwise, if he poin, e.g., poin Q, is projeced ono an obsacle, hen is radial direcion, i.e., OQ, and is angenial direcion, i.e., GH, will form an acue angle. Thus, ground segmenaion can be described as follows: if he absolue value of he cosine of he angenial angle is less han he hreshold δ2, hen i is regarded as a road surface poin and is removed.

7 Sensors 215, This feaure is paricularly applicable o obsacles ha are far from he vehicle because he farher a poin is, he larger he value of he angenial angle formed by he obsacle Local Heigh Difference Assuming ha he road is fla; he heigh difference in a local area will hen be small. Neverheless; an obsacle causes a sudden change in heigh. The poin cloud is projeced ono a grid map; wherein each pixel covers a range of 2 cm 2 cm. Hence; a range of approximaely 1 m 1 m of he environmen will be covered. The maximum heigh difference will be calculaed in each pixel. The pixel will be mared as an obsacle area if he heigh difference exceeds he hreshold δ Gradien in he Radial Direcion wih a Dynamic Threshold As menioned earlier, he 64 poins generaed by 64 laser beams in one direcion will form a sraigh line when hey are projeced ono a fla plane. However, he line will be broen if i is bloced by an obsacle. Thus he gradien in he radial direcion will be changed. The dynamic hreshold of he change in gradien is obained from he inner poins in his direcion by ripling he variance. A deailed descripion of his wor is provided in [42] Deermining he Threshold of he Feaures A oal of 17 feaures are esed in our experimen. The feaures menioned from Secions are proven o be effecive and robus, whereas oher feaures cause lea deecion of he obsacles. For example, he heigh difference beween neighboring poins in one spin from a laser, which is a feaure adoped in many wors, will remove disan road curb poins because heigh difference changes gradually when poins are projeced. To provide an analogy, if he heigh of a road curb is 1 cm and up o 1 poins are projeced ono i from one laser in one spin, which is possible in an acual siuaion, hen he average heigh difference is 1 cm. Thus, he poin will be removed. To deermine he hreshold for he firs and hird feaures, an experimen was conduced o obain he saisics for he disribuion of he feaure values of he poins projeced ono he ground. The resul is shown in Figure 3. As shown in he figure, he range for he change in radius beween neighboring poins in one spin is [.997, 1.3]. Thus a looser hreshold of.5 is adoped in his wor. Alhough he range for he hird feaure is [.915,.965], our experimen demonsraes ha a hreshold wih a value near 1 removes he poins belonging o non-ground poins near he vehicle. Therefore, a considerably looser hreshold of.6 is adoped. The hreshold for he fourh feaure was se o 15 cm in [38]. However, a looser hreshold of 1 cm is adoped in his wor o deec unobvious obsacles. Deails are shown in Table 1. Table 1. Thresholds adoped in he experimen. δ 1 for feaure in Secion δ 2 for feaure in Secion δ 3 for feaure in Secion

8 Sensors 215, Figure 3. Disribuions for feaures 1 (a) and 3 (b) of poins projeced ono he road surface Obsacle Clusering To our nowledge, previous wors have processed road surface segmenaion and obsacle clusering separaely. By conras, clusering and ground segmenaion are performed simulaneously in our wor. Hence, a poin is assigned o an obsacle poin cluser once i is deeced. The proposed mehod offers wo advanages over previous approaches. Firs, processing ime is reduced because an exra obsacle clusering sep is eliminaed. Clusering is conduced during obsacle deecion and he posiion of he es poin can be obained; hus, clusering can be performed locally insead of globally. Second, previous clusering informaion can be uilized in subsequen deecion. This advanage is demonsraed in our experimen. Neighboring informaion can also be regarded as a feaure o deec obsacles. For example, he poins on op of a bus will be removed if all crieria are saisfied. However, if he boundary of he bus is deeced afer all he crieria are saisfied, hen he poins on op of he bus can be deeced using a looser hreshold because deecing he boundary of he bus sugges ha neighboring poins are liely o be obsacle poins. If he neighbor of a poin is designaed as an obsacle, hen he poin can be idenified as an obsacle wih less consrain. In our experimen, if no obsacle poin is deeced among he neighbors of he esed poin, hen he combinaion of all he crieria menioned in Secion 2.1 excep for he second crierion, called crierion se 1, should be saisfied o designae he poin as an obsacle poin. Oherwise, only he combinaion of he local heigh difference and angenial angle, called crierion se 2, should be saisfied. An ID is aached o each obsacle cluser during he deecion process and an obsacle map is used o record he clusers ha have been buil based on he grid map. However, unlie ha in he grid map, he informaion regisered in an obsacle map is he ID of he cluser o which he pixel belongs o. Moreover, a lis of deeced clusers is mainained and updaed during he process. The process flow is shown in Figure 4.

9 Sensors 215, Figure 4. Process flow for clusering Calculaing Obsacle Cluser Shape Characerisics A number of clusers are obained afer he process described in Figure 4 has been performed. For each cluser ζ ha consiss of poins {( xi, yi, z i)}, wih i = 1,2,3,,N, he shape characerisic for he projecion on he x y plane is calculaed as follows: (1) The cener posiion O of he cluser is calculaed as follows: cenerx cenery 1 N = xi (1) N i = 1 1 N = yi (2) N i = 1 (2) The covariance marix in he x y plane is calculaed as follows: 1 C x cenerx y cenery x cenerx y cenery (3) N T = [ i, i ][ i, i ] N i = 1 (3) The eigenvalue and eigenvecor of C are calculaed and saved in marices P and X: 1, λ P = 2, λ 1, 2, 1, 2, x x X = n n = 1, 2, y y CX = XP (4)

10 Sensors 215, As proven in he Karhunen-Loeve ransformaion, he principal componen denoes he direcion in which he cluser poins are irrelevan. The principal componen can be obained by ransforming he recangular axis hrough roaion ransformaion defined by he eigenvecor of he covariance marix, i.e., X. The dynamic road obsacle are mosly vehicles, which can be represened by a cuboid in 3D or a recangle in 2D. Thus, he forward direcion of he dynamic obsacle can be represened by is irrelevan 1, direcion, i.e., n, in 2D. 1, 2, The x y poins in ζ are projeced ono wo direcions defined by n, n, which are perpendicular o each oher. Half of he lengh and widh can be deermined by calculaing he maximum disance of he projecion ono he cener. Hence, he minimum bounding box can be obained in he obsacle map. Moreover, he heigh can be deermined by calculaing he maximum difference in he z direcion. The heigh, lengh, and widh of an obsacle are saved in he sae variables H, L, and W, respecively Deecing Road Curbs The mos significan feaure of a road curb is is coninuiy. However, local disconinuiy may affec his feaure. Disconinuiy may resul from naural road curbs or he lea deecion of a road curb because i is unobvious. To enhance he coninuiy informaion of a road curb and reduce local disconinuiy, a smoohing filering echnique is employed on he grid map. Filering is performed hrough he following seps: (1) As a ey sep, binarizaion is conduced on he obsacle map wherein obsacle pixels are mared as, whereas oher pixels are designaed as 255. This sep is followed by smoohing filering, in which he possibiliy of a pixel mared as a road surface is designaed as is inensiy. The inrinsic logic of filering is as follows: when a pixel is far from he pixel mared as an obsacle, hen he possibiliy ha i is a road surface increases. In he map, he difference beween he pixel and is neighboring eigh pixels should no exceed 2. To saisfy his resricion, inensiy difference hreshold filering is performed from he op lef corner o he boom righ corner and hen from he boom righ corner o he op lef corner. To illusrae his sep, a pixel and is eigh neighboring pixels are shown in Figure 5. Figure 5. P and is eigh neighboring pixels. In he firs ieraion from he op lef corner o he boom righ corner, he effec of pixel P on is neighboring pixels is described as follows: Pi = min{ P + 2, Pi} i = 1~4 (5) In he second ieraion from he boom righ corner o he op lef corner, he effec of pixel P on is neighboring pixels is described as follows:

11 Sensors 215, P min{ P 2, P} i = + i = 5~8 i (6) (2) A hreshold filer is used on he map o idenify he road area. The hreshold is se o 2 in his wor. A pixel value larger han 2 is se o ; oherwise, i is se o 255. (3) A neighboring road pixel search algorihm is employed o deec he road area. The algorihm sars from (256, 1), which is he posiion of a vehicle, o search he eigh neighboring pixels. The pixel is designaed as a road surface pixel only if all of is 24 neighboring pixels in a 5 5 grid are. Once a pixel is designaed as a road surface pixel, i is hen added o he road surface area and a search is performed on is eigh neighboring pixels ieraively. A drivable area will be idenified a he end of he ieraion. To accelerae he process, he search is resriced o a recangular area in he vehicle forward direcion. (4) A road curb is idenified by searching he boundary of he road area, which is described in deail i in our previous wor [42], and sored in C = { P, i= 1~ N}. Finally, a leas square fi is applied o C o form a quadraic curve. An example of his process is shown in Figure 6. (a) (b) (c) (d) (e) (f) Figure 6. Example of road curb deecion. (a) Original image obained by he camera ha describes he local environmen; (b) Original grid map obained by he Velodyne LIDAR; (c f) Resuls for Seps 1 4, respecively.

12 Sensors 215, Tracing Road Curbs and Dynamic Obsacles Tracing is a complex problem ha can be divided ino wo seps: predicion and updaing he predicion. Predicion is he sequenial esimaion of he sae. I is ypically performed using Bayesian filers ha require a specific moion model o predic he posiions of he raced objecs in he environmen. Afer predicing he posiions of exising racs, daa associaion is performed o assign he observaions o exising racs. The racing of road curbs and dynamic obsacles is addressed separaely in our wor. A road curb is a saic obsacle in he environmen; hence, he variaion of is posiion beween differen frames merely depends on he moion of a vehicle. For dynamic obsacles, however, he variaion in sae depends on boh he moion of a vehicle and he sae of he dynamic obsacles. The moion informaion of a vehicle can be obained from he Global Posiioning Sysem (GPS) and inerial navigaion sysems (INSs). The processes of racing road curbs and dynamic obsacles are described in deail in he following secions Tracing Road Curbs Predicing Road Curbs As menioned earlier, a road curb can be considered a saic obsacle and predicing a road curb merely depends on he moion saes of a vehicle. The moion of a vehicle is simulaneously modeled as a ransfer moion and a roaing moion. The moion of he road curb can be modeled as he relaive moion in he negaive direcion of he vehicle, which can be obained direcly from Synchronous Posiion, Aiude, and Navigaion (SPAN)-CPT. The SPAN echnology combines wo differen bu complemenary echnologies: a global navigaion saellie sysem (GNSS) and an Inerial Navigaion Sysem (INS). The absolue accuracy of GNSS posiioning and he sabiliy of Inerial Measuremen Uni (IMU) gyro and acceleromeer measuremens are ighly coupled o provide an excepional 3D navigaion soluion ha is sable and coninuously available, even hrough periods when saellie signals are bloced. The sae space equaion for his saic model is presened o model he road curb moion C= { P i, i= 1~ N} as follows: where: ^ i i P 1 AP B C: pixel se ha consiss of a road curb, P i = ( x i, y i ) T : a curb pixel, cos( ϑv, ) sin( ϑv, ) A = sin( ϑv, ) cos( ϑv, ), Δ ( xv, + xv, )cos( ϑv, ) ( Δ yv, + yv, )sin( ϑ v, ) + x v, B =, ( Δ xv, + xv, )sin( ϑv, ) ( Δ yv, + yv, )cos( ϑ v, ) + yv, : roaion of he vehicle, ϑ v, x, y : x and y coordinaion of he vehicle in he map, v, v, Δx, Δ y : ransformaion in he x and y direcions. v, v, + = + (7)

13 Sensors 215, Updaing he Predicion of a Road Curb To increase he accuracy of he prediced road curb poins, he prediced road curb should converge o he road curb in his frame by adoping he resul of ground segmenaion. As menioned earlier, he mos significan feaure of a road curb is is coninuiy. However, local disconinuiy occurs because of wo reasons. Firs, disconinuiy is caused by he scene, e.g., he adjacen parerre. Second, lea deecion of a road curb is ineviable because road curbs are no obvious obsacles ha may be concealed by dynamic obsacles on he road. Considering global coninuiy and local disconinuiy, he snae algorihm is applied o updae predicion. The snae model was proposed by Kass [43]. A snae is an energy-minimizing spline guided by exernal consrain forces and influenced by image forces ha pull i oward feaures such as lines and edges. The energy funcion is defined by he inegraion of hree iems, as follows: E ( ) ( ( ))) snae = Ein ν(s) + Eimage ν s ds (8) 2 2 where Ein ( ν(s) ) = ( α ( s) υs( s) +β( s) νss( s) )/2 represens he inernal energy of he spline because 2 of bending, and Eimage ( ν( s)) = γ( s) I( ν ) denoes he exernal energy ha corresponds o he edge feaures of he image. Parameers α(), s β(), s γ () s denoe he weighing coefficiens of elasic energy, curvaure energy, and exernal energy, respecively; whereas ν(s) = (x(s),y(s)) represens he pixel coordinaes on he spline. To decrease he ime consumed in updaing he road curb, he elevaion map is applied. Some noise poins are deeced in he grid map because only he heigh difference feaure is applied o updae predicion. This decision is based on wo consideraions. Firs, he elevaion map can be obained immediaely once he LIDAR poin is parsed, which can improve real-ime performance. Second, he heigh difference in our experimen is relaively robus and only a few discree noise poins remain, which will no affec he final resul because he snae model considers global coninuiy informaion. The ieraion sars from he prediced road curb poin obained using Equaion (7) and coninues unil he local minimum value of Esnae is obained. Our experimen proves ha he curve converges o he road boundary when he minimum value of he Esnae is obained. For a rapid convergence, he greedy snae [44] is adoped in our wor. The vehicle dynamic informaion obained by GPS and INS is accurae and a road curb is a saic obsacle; hence, he local opimum value is sufficien o converge o he acual road curb in our experimen Tracing Dynamic Obsacles The moion sae of a dynamic obsacle in a frame reflecs is relaive posiion wih a vehicle. Thus, is moion sae depends on boh he dynamic obsacle and on iself. Unlie a road curb whose moion sae beween frames can be obained from he sensors in a vehicle, a moion sae vecor is mainained for each raced dynamic obsacle. The rac of dynamic obsacles is modeled as a linear ime invarian sysem in our experimen, and he Kalman filer is used. The moion sae for each dynamic obsacle is composed by and given as: S = ϑ α,,,, [ x x y x y ] T y v v a a L W

14 Sensors 215, where (, x, y, O = x y ) is he cener posiion of a dynamic obsacle; v = ( v, v ) is he velociy of a x, y, dynamic obsacle; a = ( a, a ) is he acceleraion of a dynamic obsacle; ϑ is he forward direcion of a dynamic obsacle; α is he angle acceleraion of a dynamic obsacle; and L, W are he lengh and widh of a dynamic obsacle, respecively. To iniialize S, once a new cluser O (, = x y ) is deeced, ϑ and L, W can be obained hrough he process presened in Secion 2.3 and he ohers are iniialized as. The dynamic obsacle moion sysem is modeled as follows: S = 1 AS + + w (9) Z = S + v (1) T T T T 2 1 T 1 T where A = 1 is he sysem ransfer marix; and 1 1 T he independen zero-mean Gaussian noise variables o process and measure wih covariance R v,, respecively Predicing Dynamic Obsacles w and v are w, R and The moion sae and is covariance marix of each dynamic obsacle in he presen frame from he las frame is prediced as follows: ^ 1 The covariance P ^ + 1 for S ^ + 1 is esimaed as follows: S + = AS (11) ^ T w, P 1 AP A R + = + (12) Updaing Dynamic Obsacles on a Road In his secion, daa associaion is performed beween he newly deeced obsacle poin and he hisorical dynamic obsacle lis. Once a poin is designaed as an obsacle by applying he rules in Secion 2.1, he poin is esed o deermine wheher i is locaed on he road. A road is saic and he measuremen of he sensors in he vehicle and is accurae; hence, he prediced road area beween he prediced road curb

15 Sensors 215, obained in Secion is employed. If he projecion of he poin ono he x y plane is no on he road surface, hen he poin is no clusered because i does no affec driving decision. Oherwise, i is designaed as a road obsacle. The poin will be mached firs wih he dynamic obsacles in he hisorical dynamic obsacle lis. If he mach succeeds, hen he moion sae of he mached dynamic obsacle will be updaed. Oherwise, a new dynamic obsacle will be creaed. The neighboring pixels will be searched o deermine wheher hey are aached o a cluser. If a neighboring pixel is aached, hen i is included in he same cluser. Oherwise, he pixel is evaluaed o idenify he cluser o which i belongs o. For poins P = ( x, y) and cluser ς, he process is as follows: n = (cos( ϑ ),sin( ϑ ) (13) l, d = O P* n (14) w, d = O P n (15) pos = 1 2πLW e l,2 w, 2 1 d ( d ) 2 L W (16) The larges pos among pos is deermined. If a pos is less han he hreshold, hen he poin is deal wih, as indicaed in Secion 2.2. Oherwise, he h cluser is updaed wih he new poin. The process is shown as follows: δp updae he ς pos = max( pos ) <δ p deal as shown in Secion 2.2 Afer processing all he poins, a lis of clusers is esablished for his frame. If a cluser is newly deeced, hen i is iniialized as described earlier. Oherwise, he observed sae vecor,,,, z [ x y x y ] T = x y v v a a ϑ α L W can be obained as follows. O = ( x, y ), ϑ, and L, W can be idenified hrough he process described in Secion 2.3. The oher componens of he sae vecor can be obained as follows: v v a a x, 1 (17) x x = (18) T y y = (19) T y, 1 v v = (2) T x, x, x, 1 v v = (21) T y, y, y, 1 The sae vecor and is covariance are updaed as follows: ϑ ϑ 1 α = (22) T

16 Sensors 215, Kg = ^ P ^ P + R v, (23) ^ ^ S = S + Kg ( Z S ) (24) ^ P = ( I Kg ) P (25) 4. Resuls and Discussion The experimen was performed on our UGV, called he Inelligen Pioneer, which was equipped wih various sensors, e.g., a camera, an infrared camera, an Ibeo LIDAR, a Sic LIDAR, a Velodyne LIDAR, and SPAN-CPT. The sensors used in his wor were he Velodyne LIDAR and SPAN-CPT. The processor used for he LIDAR daa was a Core i7-361qe wih a process frequency ha could be increased o 3.3 GHz and a cache of 6 M. Firs, ground segmenaion experimens were conduced under various road condiions. The proposed muli-feaure ground segmenaion algorihm was proven o be effecive under various road condiions. The experimen was performed in boh urban and rural environmens. Four represenaive scenes are shown in Figure 7. The original image obained using he camera and he original grid map developed by he Velodyne LIDAR are shown in he firs and second columns, respecively, whereas he resuls of ground segmenaion and curb fiing are shown in he hird column. The yellow area denoes he drivable area, whereas he red and green lines represen he road curb and driving guide line, respecively. Two urban scenes are shown in he firs wo rows. The firs row shows a sraigh road ha is no fla. On he one hand, he road surface is craced. On he oher hand, i is no clean because of he presence of mud. Given such disurbances, he poins projeced ono he road surface do no form a smooh circle bu an arc wih a high-frequency noise. However, based on he hird column, he road surface was deeced as clean and wihou noise. This resul was achieved by adoping feaures in wo aspecs: he local poin characerisics described in Secion 2.1 and he local area coninuiy feaure processed similarly in Secion 2.4. Muli-feaure road surface segmenaion removed nearly all he noise poins on he road surface because a muli-feaure scheme wih a relaively looser hreshold was applied. The remaining noise poins, if hey exised, were eroded by he surrounding coninuous road surface area. The curve shown in he second row demonsraes he capabiliy of he algorihm o deec road curbs. In general, he curve fis he general road curb well. However, some gap exiss in our experimen; hese gaps are mainly caused by wo reasons. Firs, he curve is a quadraic, and hus, i may be unsuiable for a curb in an acual scene. Second, he dynamic obsacle near a road curb will be deeced as a road curb. However, driving decision will be unaffeced because he coninuiy of he drivable area is no influenced by he dynamic area, as shown in Figure 8. As shown in his figure, he paced cars and pedesrians along a road curb are deeced as he road edge, which causes he irregulariy of he road curb. However, his phenomenon will no affec driving decision because he given drivable area is reasonable.

17 Sensors 215, Figure 7. (Lef column) Original image aen by a camera; (Middle column) Original grid map obained by he Velodyne LIDAR; (Righ column) Ground segmenaion and curb fiing resuls. The hird and fourh rows in Figure 7 depic he rural environmen, wherein he boundary is represened by bushes, i.e., posiive obsacles, in he hird row, whereas he boundary is represened by a dich, i.e., a negaive obsacle, in he fourh row. The challenge in ground segmenaion in a rural environmen is considerably greaer han ha in an urban environmen because poins projeced ono he road surface are irregular. Moreover, diffuse reflecion occurs because road surface is coarse. Hence, he

18 Sensors 215, local poin feaure becomes increasingly irregular. In he experimen conduced in he rural environmen, heigh measuremens were inaccurae because of he posiion of he horizonal projecion, which migh be caused by he jol of he vehicle in he rural environmen. Based on he analysis, he local coordinae sysem eeps changing wih he flucuaion in pich of he vehicle during a jol, which resuls in he incorrec calculaion of he z value of he poin. Each single feaure was esed in our experimen, and he resuls were unsaisfacory. However, combining muliple feaures yielded an excellen resul. Figure 8. Scene where he vehicle pared along he road side was designaed as he road curb. (Lef column) Original image aen by camera; (Middle column) Original grid map obained by Velodyne LIDAR; (Righ column) Ground segmenaion and curb fiing resuls. (a) (b) (c) (d) Figure 9. Comparison beween muli-feaure road surface segmenaion and segmenaion based on heigh difference. Lef o righ columns: (a) Original image aen by a camera; (b) Image resuling from he heigh difference feaure wih a sric hreshold; (c) Image resuling from he heigh difference feaure wih a loose hreshold; (d) Image resuling from he muli-feaure scheme wih a loose hreshold.

19 Sensors 215, The vehicle drives under various road condiions; hence, he LIDAR finds performing effecive ground segmenaion o be difficul because of he irregulariy of he source poin cloud, which resuls from vehicle flucuaion and he diffused reflecion of he ground. To address his problem, he philosophy behind ground segmenaion is a muli-feaure scheme wih a relaively looser hreshold. Experimens were conduced by applying differen algorihms under various road surface condiions in ground segmenaion o exhibi he advanages of he muli-feaure scheme. Heigh difference is a widely applied characerisic in ground segmenaion; hence, a comparison beween segmenaion ha applies heigh difference and segmenaion ha applies he fusion of feaures is presened in Figure 9. The wo scenes shown in he figure represen urban and rural environmens. This figure indicaes ha when road surface is fla and road curb is relaively low, a loose hreshold is more suiable when applying he single heigh difference feaure because he sric hreshold classifies a rue road curb as road surface. The rees on he roadside are classified as road curbs insead, which is shown in he second column of he firs row. However, he environmen is more complex in rural areas. Based on he image, nearby poins are classified as non-ground poins and road surface is resriced o a small area. Figure 1. Tracing experimen on he disance and velociy of a vehicle under various velociies. In racing dynamic obsacles; an experimen was performed o rac vehicles under various velociies. The posiions and velociies of a arge vehicle ha was moving a various velociies were recorded by CPT. Meanwhile; he es vehicle was sopped o esimae he posiion and velociy of he arge vehicle. Three groups of experimens are shown in Figure 1; wih speeds of approximaely 2; 3; and 6 m/s. The daa obained by CPT are shown in blue; whereas he esimaions are indicaed in saffron yellow. The op image illusraes he rac of he disance of he arge vehicle from he es vehicle; whereas he boom image shows he rac of he velociy of he arge vehicle. This figure also indicaes ha racing under a low speed is accurae; whereas racing under a high speed is relaively inaccurae. Based on he analysis; his finding is caused by wo reasons. Firs; he resul occurred because of he propery of he Kalman filer; which considers previous informaion. Thus when relaive speed is high; he frames ha can be raced for he arge vehicle are few. Second; his phenomenon occurred because flucuaion increases wih speed. However; he relaive speed beween he es vehicle and he arge vehicle is low.

20 Sensors 215, Thus; when he vehicle is on an acual driving environmen; our algorihm exhibis good capabiliy in racing he vehicle. The wo scenes in Figure 11 show our experimen conduced in an acual driving environmen. The deeced dynamic obsacle is mared in blue and surrounded by a minimum bounding box. The number on he righ column denoes he velociy of he arge vehicles. As shown in Figure 11, he firs row indicaes he rac of a single vehicle on a sraigh road, where he es vehicle is sopped by he road curb. As he es vehicle sops, he velociy of he arge vehicle is refleced. However, his velociy is he relaive velociy beween he es vehicle and he arge vehicle. Hence, he vehicle moving oward he opposie direcion has high speed, whereas he velociy of he vehicle moving in he same direcion is relaively low. This phenomenon is illusraed in he second scene, which represens a crossroad environmen. The lef arge vehicle is moving oward he opposie direcion wih higher velociy, whereas he righ vehicle is moving oward he same direcion wih lower velociy. Figure 11. Dynamic obsacle rac resul. (Lef column) Original image aen by a camera; (Middle column) Original grid map obained by he Velodyne LIDAR; (Righ column) Ground segmenaion and dynamic obsacle rac resul. To evaluae he real performance of he proposed algorihm, experimens on urban and rural environmens were conduced. The saellie aerial phoographs of wo scenes are shown in Figure 12. The urban scene Figure 12a is composed of various road condiions: sraigh roads, crossroads, and curved roads. In he rural environmen Figure 12b, he road condiion is more complex. The bad road condiion causes he vehicle o jol, which maes he generaed poin cloud more irregular. The lengh of he roue is abou approximaely 5 m in Figure 12a and approximaely 8 m in Figure 12b.

21 Sensors 215, Firs, he real-ime performance for each frame is obained and he resul is shown in Figure 13. As shown in he figure, he real-ime performances of he urban scene and he rural scene are nearly he same. Moreover, he figure shows ha he real-ime performance is invarian o ime. The disribuion of process ime for each frame is illusraed in Figure 14. As shown in he figure, mos of he samples fall wihin [3,4] in boh cases. (a) (b) Figure 12. Saellie aerial phoographs of he urban scene (a) and he rural scene (b). Time/ms Scene 1 Time/ms Scene Frame (a) Frame (b) Figure 13. Real-ime performances of he urban scene (a) and he rural scene (b). An experimen on he robusness of road curb deecion was also conduced. Ground segmenaion is he ey procedure for an effecive of he road curb deecion. As shown in our experimen, accurae ground segmenaion can guaranee he effeciveness of road curb deecion and a minor segmenaion error was olerable because of he error eliminaion process in he following sep. Only significan errors will lead o incorrec deecion of he road curb. Thus, a saisic is proposed. This saisic consiss of four conens: he percenage of successful ground segmenaion, he percenage of ground segmenaion wih minor errors whereas wherein he road curb deecion is no affeced, he percenage of incorrec ground segmenaion which caused lea deecion of he road, and he percenage of incorrec ground segmenaion which caused false deecion of he road. The resuls are shown in Table 2. As indicaed in he able, performance is beer in urban scenes, wherein road condiions are beer han in rural scenes.

22 Sensors 215, Lea deecion of he road curb occurred in cases wherein he road curb is no disincive from he road surface. However, false deecion occurred in crossroad siuaions which are no acled in his wor in urban scenes and in hose cases wherein he road siuaion is severe in rural scenes. 5 Scene Scene Frequency Frequency Time/ms Time/ms Figure 14. Disribuion of process ime for each frame in he urban scene (a) and he rural scene (b). Table 2. Saisics on ground segmenaion and road curb deecion. Scene Accurae Minor Error Lea Deecion False Deecion Scene % 1.85%.55%.16% Scene % 6.54% 1.56% 2.61% 5. Conclusions/Ouloo A framewor for he applicaion of a muli-line LIDAR o model ey elemens ha comprise he driving environmen, i.e., road curbs and dynamic obsacles, is proposed in his wor. The framewor combine he modeling of he road informaion and he road dynamic obsacle as an organic enirey. This approach is proven o be robus and saisfied he requiremen for he online process in he experimen presened in earlier secions. A quic and robus modeling of dynamic obsacles and road curbs consiss of several procedures ha all pose a significan influence on he final resul. To segmen he ground robusly, in conras o he applicaion of single or very few feaures in ground segmenaion in previous wor, he philosophy of muli-feaures wih loose hresholds is employed in his wor, which proves o be adapive o various environmens. As for he dynamic obsacles, in conras o previous wors where dynamic obsacles are clusered afer he grid map is generaed, dynamic obsacles are clusered in his wor once hey are classified as non-ground poins. The laer approach has wo benefis. Firs, he differen sraegies are applied depend on he neighborhood informaion of he esed poin. When he neighbor poins of he es poin are designaed as non-ground poins, he possibiliy ha he es poin is a non-ground poin increases, and hus, a looser hreshold can be applied for he poin o be designaed as non-ground poin. Second, clusering ime is reduced because clusering is performed locally insead of globally. Moreover, shape feaures are calculaed by adoping he Karhunen-Loeve ransformaion, which is he firs ime such a procedure is employed. In he racing process, he shape feaures menioned before are used o fuse curren informaion wih hisorical informaion. In road curb

23 Sensors 215, deecion, he disance image is applied o deec he road curb, whereas he snae model is used for racing because of is capabiliy o consider boh local and global informaion. To our nowledge, he snae model has never been applied in road curb racing for LIDAR daa processing. By modeling road area, a vehicle can deermine he pah of is movemen. By modeling dynamic obsacles on he road, a vehicle can deermine places o avoid. The presened framewor aimed o provide a novel framewor for he applicaion of muli-beam LIDAR under various road environmen. Unlie he framewors presened before, he purpose of he presened framewor is o dig deep ino he process of he applicaion of muli-beam LIDAR o find a beer way o organize he informaion flow. The robusness and he real-ime performance are conradicory o some exen. The way o solve his problem is by finding a way o combine processes as much as possible. By combining differen process ogeher, informaion loss is eliminaed beween differen procedures. Wha s more, he process ime is reduces because he applicaion of local viciniy informaion. The framewor is esed under various road condiions and proved o be able o model he road curb and he dynamic road obsacle robusly on-line. Furher research will focus on simulaneous locaion and mapping using muli-beam LIDAR by inegraing local and global informaion. Acnowledgmens We would lie o acnowledge all of our eam members, whose conribuions were essenial for he success of our inelligen vehicle. We would also acnowledge he suppor of hree Naional Naure Science Foundaions of China: Sudy on he robus racing of dynamic obsacle based on he significance learning of he vision and consciousness (61591), Key echnologies and plaform for unmanned vehicle in urban inegraed environmen (911237) and Key echnologies and plaform for unmanned vehicle based on visual and audiory cogniive mechanism (913231). Auhor Conribuions All four auhors conribued o his wor during he enire phase. Jian Liu was responsible for he lieraure search algorihm design and daa analysis. Huawei Liang and Zhiling Wang made subsanial conribuions in he plan and design of he experimens. Jian Liu was responsible for he wriing of he paper and Xiancheng Chen helped modify i. Finally, all he lised auhors approved he final manuscrip. Conflics of Ineres The auhors declare no conflic of ineres. References 1. Dollar, P.; Woje, C.; Schiele, B.; Perona, P. Pedesrian Deecion: An Evaluaion of he Sae of he Ar. IEEE Trans. Paern Anal. Mach. Inell. 212, 34, Andrilua, M.; Roh, S.; Schiele, B. People-racing-by-deecion and people-deecion-by-racing. In Proceedings of he 28 IEEE Conference on Compuer Vision and Paern Recogniion (CVPR), Anchorage, AK, USA, June 28; pp.1 8.