Real-Time Object Detection Using a Sparse 4-Layer LIDAR

Transcription

1 Real-Time Object Detection Using a Spase 4-Laye LIDAR Micea Paul Muesan, Segiu Nedevschi, Ion Giosan Compute Science Depatment Technical Univesity of Cluj-Napoca Cluj-Napoca, Romania {micea.muesan, segiu.nedevschi, ion.giosan}@cs.utcluj.o Abstact The obust detection of obstacles, on a given oad path by vehicles equipped with ange measuement devices epesents a equiement fo many eseach fields including autonomous diving and advanced diving assistance systems. One paticula senso system used fo measuement tasks, due to its known accuacy, is the LIDAR (Light Detection and Ranging). The commecial pice and computational intensiveness of such systems geneally incease with the numbe of scanning layes. Fo this eason, in this pape, a novel six step based obstacle detection appoach using a 4-laye LIDAR is pesented. In the poposed pipeline we tackle the poblem of data coection and tempoal point cloud fusion and we pesent an oiginal method fo detecting obstacles using a combination between a pola histogam and an elevation gid. The esults have been validated by using objects povided fom othe ange measuement sensos. Keywods Spase LIDAR, Object detection, 3D Point-Cloud, Digital Elevation Maps I. INTRODUCTION The obust and eliable epesentation of the envionment is an impotant task fo any application woking in outdoo suoundings. Efficiently detecting and identifying obstacles fom continuous steamed 3D point clouds in vaious scenaios ae key poblems fo all intelligent vehicle applications. As descibed in [1] any intelligent vehicle can be descibed by thee commonly accepted modules: peception, planning and contol modules. In the scope of the cuent wok, we ae inteested in the peception module, which builds an intenal epesentation of the envionment based on the data collected fom the sensos. Fo gatheing 3D infomation, in the case of autonomous vehicles, the peception system intepets, and commonly eceives its input fom steeo cameas [2, 3], 3D-LIDARs [4, 5] and RADAR systems [6]. Even though steeo solutions ae moe affodable, a majo limitation of a steeo system is the difficulty in dealing with lack of textue, bad illumination, pespective effect and dakness among othe. RADAR sensos ae geat at detecting moving objects made out of metal mateial, howeve they fail to detect items made of poous plastic o wood. Even moe, RADARs usually have a naow field of view, and to compensate fo this issue they ae usually used in aays that slightly ovelap in ode to obtain lage fields of view. One of the biggest disadvantages of RADARs is that they omit objects in ode, not to ove-epot. While this chaacteistic is vey efficient when omitting the oad suface, it has vey high isks since it can also omit static objects (fo example vehicles paked on the oad) [7]. LIDAR sensos can detect static objects and ae less sensitive to weathe and illumination conditions, howeve they have a vey high puchasing cost. The pice of the LIDAR sensos inceases with the numbe of scanning layes, the ones having 32 and 64 layes ae among the most expensive. LIDARs having 4-layes ae much cheape, they have a much lage woking ange than the pevious mentioned LIDAR systems and they ae geneally faste. The dawback of the 4-laye LIDARs is that they have a smalle scanning angle and ultimately povide fewe scanning points. The task of detecting obstacles fom spase point clouds is theefoe vey challenging fo numeous easons. Fist of all the aw point clouds, obtained afte measuement, can be noisy and uneven in consecutive fames. Secondly, in the case of teestial LIDAR measuements, objects can eceive stongly coupted geometic popeties such as missing pats o defomed shapes due to inceased point cloud densities fom the diection of the measuement [8]. In the context of autonomous vehicles, data fom multiple sensos is fused in ode to povide a moe obust envionment epesentation. In this pape we will tackle the poblem of obstacle detection using 4Laye LIDARs and we will elaboate on the necessity of each stage fom the given pipeline in the context of spase LIDARs. The est of the pape is stuctued as follows: section II pesents the elated wok in the field of object detection using LIDAR sensos, in section III we pesent the poposed pipeline fo object detection, section IV povides expeimental esults and finally in section V we pesent the conclusions and futue wok. II. RELATED WORK Thee ae numeous appoaches available in the liteatue fo detecting obstacles fom 3D point clouds. Depending on the data-stuctue used, they can be split into two categoies: tee based and gid based appoaches. In the tee based object detection, pe-computed tee-like data stuctues such as octee o ange tee can be used [9, 10]. These methods ae vey good at ange seach, howeve they ae computationally intensive at initialization. Othe ecent appoaches use diffeent egion gowing in ode to obustly detect objects. The authos in [11] pesent an octee based occupancy gid that models the envionment nea the vehicle and detects moving obstacles fom inconsistencies between scans.

2 The gid based methods ae the second categoy of appoaches that focus on fast 3D pocessing fo object detection. In [12] a segmentation of the 3D point cloud and objects by using a standad connected component algoithm on a 2D occupancy gid is pesented. In [13] the concept of elevation (o 2.5 gids) maps that stoe in each cell the height of objects above gound level is poposed. Roth et al. [14] and Moavec [15] popose a 3D gid made up fom voxels. This method equies lage amounts of computational esouces since the voxels defined cove the whole space, even if in eality only a few measued points ae pesent in a specific egion. Fo the task of detecting obstacles, usually gid based methods wok in conjunction with some appoaches of detecting the gound suface. In [16] the RANSAC method is used to estimate the gound plane. This method is efficient when the gound is plana, howeve if the numbe of points is not lage enough o the gound is cuvy the method fails to detect the oad suface. A method that solves the issue of oads with unequal elevation is pesented in [17]. A quadatic suface model is initially fitted to the egion in font of the vehicle to estimate the oad plane. In the cuent spase LIDAR scenaio such a technique would not wok since the numbe of points on the oad is vey small. The V-dispaity [18] is anothe method in which the oad suface can be estimated in case of steeo econstuction, howeve the dispaity epesentation is not a natual way to epesent 3D points. III. PROPOSED SOLUTION In this pape we will tackle the poblem of object detection when using spase 4-laye (4L) LIDARs. The easons why classical object detection methods might fail is because of the point cloud noise and spasity. We will pesent a method fo coecting the point cloud data in case of a mobile obot. LIDAR Data Acquisition Point Cloud Motion Coection Point Cloud Tempoal Fusion 3D points Motion Coected Points TF 3D points Anothe difficult poblem which appeas when detecting objects using spase LIDAR sensos is the estimation of points belonging to the gound suface. Fo solving this issue, in this section we will descibe a method that uses a pola histogam to detemine oad points. We will also use an elevation gid fo efficiently labelling and gouping object points afte the oad points have been emoved. The six step pipeline fo efficiently detecting objects is pesented in Fig. 1. A. Point Cloud Motion Coection and Tempoal Fusion 1) Motion Coection The 4L LIDAR is measuing the envionment by means of lase beams. The complete pofile of the envionment can be built, by the pemanent otation of the mio which is in connection with the lase beam. A difficult scenaio aises when we ae scanning the envionment fom a mobile platfom. Due to the fact that the ca is moving, the points fom evey individual scan will be affected by displacement eos. An intuitive depiction of this phenomenon is displayed in figue 2 bellow. Fig. 2. Measuement eos caused by motion 4L LIDARs can geneally povide a timestamp in one of the thee untime moments: when the data acquisition stats, when it ends o at the middle of the acquisition. Consideing that the timestamp of each 4L LIDAR fame is the timestamp of the beginning of the acquisition, we compute the timestamp of each lase point, knowing the esolution of the scan, duation of a scan, and the channel ID fo each point all these infomation ae available in the datasheet of the senso o in diffeent papes that evaluate the senso capabilities [18]. An intuitive image of the data povided by the senso used is illustated in Fig. 3. Road Points Detection and Point Cloud Filteing Filteed 3D points Ceation of 2.5 Elevation Gid Gid with Labels Channel DEG 145 DEG t Channel 580 Gouping Obstacle Cells to Define Obstacle Limits Fig. 3. Depiction of spase LIDAR capabilities Object filteing List of Objects Knowing all this infomation we can compute the timestamp fo each individual point using the equation (1) below. Final Objects Fig. 1. Object detection pipeline t TSi Channeli * NumbeOfChannels (1)

3 In the equation above TS i epesents the timestamp fo point i, Channel i epesents the i th channel fom the acquied data, NumbeOfChannels is the total numbe of channels available fo a LIDAR senso and Δt efes to the amount of time needed to pefom a scan. The cuent motion coection appoach computes tansfomations on each individual point as pesented in [19], but also elies on ego motion infomation povided by an inetial measuement unit (IMU) available on the vehicle. When computing the cloud tansfomation matix we ae taking into account the displacements on x, y, z and the pitch, yaw and oll infomation. In equations 2, 3 and 4 bellow we show how to compute the coection tansfomation of each point. TT defines the taget timestamp, TP means the timestamp of the fist acquied point, TansfomationMatix efes to the matix obtained using the EGO infomation and tanslation displacement values, Tp i efes to the timestamp of point i and is the coection tansfomation fo point i. Cloud TT TP (2) 1,1 1,2 1,3 t x TansfomationMati 2,1 x 2,2 2, 3 t y 3,1 3,2 3,3 t z Ci (3) Tp i log( TansfomationMatix) Cloud e (4) Consideing t p the i th point taken at timestamp t, with i infomation on axis x, y and z, the coection fo the entie cloud becomes (5). In this equation, N epesents the numbe of points and denotes the iteation pocess though the entie point cloud. N t CoectedCloud (tagettime) C i p i (5) i 0 In Fig. 4, the effect of the point cloud coection can be obseved. With white we obseve the uncoected points and with ed we can see the coected data. The points fom the bottom of the image eceive the highest coection wheeas the one fom the top of the image is less coected. Also note that the diffeences between the two timestamps ae small this is why the ed coected points ae not vey much modified compaed to the white uncoected points. 2) Tempoal Fusion In ode to achieve a highe data density seveal point clouds ae fused togethe. The easoning behind this is that, if we have the useful infomation in one fame, fo example some oad points, they will be popagated up futue fames making the late jobs of futue tasks in the pipeline easie. In ou paticula scenaio we have fused 6 consecutive LIDAR fames. The tansfomation function between consecutive point clouds is computed using the motion coection module pesented befoe. By we ae efeing to the motion coection tansfomation applied to consecutive point clouds. The equation of fusing multiple point clouds is pesented in (6). Pcl 1 5 FinCl T * Pcl Pcl i1 Pcl 2 ii, 1 i 6 Fig. 5. Gaphical epesentation of the TF pocess The esult of the fusion can be seen in Fig. 6. In the left hand side we have the motion coected point cloud, while in the ight hand side we have the coected and tempoal fused point cloud. Fig. 6. Coected point cloud and tempoal fused point cloud B. Road Point Detection and Ceation of Elevation Gid Road estimation fom a 4L LIDAR is a challenging task due to the fact that thee ae not so many points falling on the gound suface. The fist step fo finding the gound plane is to filte out points which have a high height (z coodinate) above a theshold of 0.5 m. We ae futhe filteing the esulted 3D points by consideing only the points that have an amplitude geate than 0.8. The high amplitude can usually come fom lane makings o pedestian cossings. We conside that the oigin of the cente of coodinates is in font of the vehicle, at gound level, and we can assume that the oad fo metes will be a line passing though this oigin. We can define this line by the pitch angle that it makes to the hoizontal line passing though the defined oigin. So, we begin to count the points falling on each line fo a numbe of angles in the side view pojection. An intuitive image of the pocess is pesented in Fig. 7. Pcl 6 (6) Fig. 4. Uncoected (white) vs. coected (ed) LIDAR points

4 In Fig. 9 we pesent the elevation gid and the gouped objects. The lines in the gid ae illustated with geen, with ed we depict the clusteed objects, with white we denote the oad points o object points which have been filteed, and with blue we illustate the bounding boxes fo object boundaies. Fig. 7.Gaphical depiction of a line sweeping though the points to find the most suitable paametes fo the oad plane A histogam stoes the numbe of points falling on each line. Finally we identify the line coesponding to the oad as the line with the maximum numbe of points. We then emove the gound points fom the initial 3D points available. To identify the points that fall on the lines we convet the pola coodinates to Catesian coodinates (7), and compute the equation of the line, that passes though the oigin, in Catesian coodinates (the value of being equal to 20). x cos( ); y sin( ) (7) In Fig. 8 the points belonging to the oad ae encicled with a ed cicle. In the left hand side we have the top view and in the ight hand side we have the oad points seen in the side view image (distance-height plane). Fig. 9. Labeled object cells, object boundaies and filteed points The detected objects can also epesent buildings o othe lage stuctues, so in ode to allow only items of inteest we have to filte the obtained esults. This task is achieved by imposing size constaints. This means that we will conside an object to be viable only if the length and width ae smalle than pedefined values (in ou case width 3m and length 12m). We also do not conside objects that ae smalle than 2 gid cells. The final esult can be seen in Fig. 10. Fig. 8. Points belonging to the gound ae highleted with ed C. Obstacle Detection and Filteing Afte emoving the oad points, we ceate a 2.5 elevation gid. We split the 3D space into cells having a dimension of 40x60 cm, and we ceate the elevation gid fo the points that have a distance smalle than 50m and a width smalle than 20m (10 mete in left and ight of the ego vehicle). The 3D points ae pojected in the gid in a top view manne. In each cell we stoe the maximum height fom all the points that fall in that cell and the numbe of points that belong to the cell (the point density in the cell). We label a cell with an object label if the density in that cell is lage than a pedefined theshold (in ou case the density theshold value is 4, but this geneally depends on the numbe of fused lase fames; if the numbe of fused fames is lage, the density theshold should be lage as well), and the height is smalle than 4m. Afte labelling all cells we pefom a clusteing appoach in ode to fuse the object cells togethe. The object cells ae fused if they ae neighboing each othe in 1 of 8 diections (up, down, left ight and fou diagonal ways). Fig. 10. Oientative intensity image of the scene (top); Unfilteed labeled objects (middle); Filteed objects in yellow bounding boxes (bottom)

5 Afte the filteing pocess thee might still be othe objects detected, like poles, longitudinal baies on the side of the oad o smalle pats of buildings which, due to the points spaseness, get labeled as diffeent objects. As humans we ealize what such stuctues mean, because we also have visual infomation about the envionment, howeve just by looking at the points we cannot say whethe they epesent useful infomation o not. Fo this eason we have decided to epot them as objects and not filte them out. In Fig. 11 with yellow we mak the identified LIDAR objects, with blue the RADAR object, with white the filteed 3D points and with geen any found association. In Fig. 12 we illustate the RADAR-LIDAR association in a cowded intesection. The significance of the colos used emains the same. IV. EXPERIMENTAL RESULTS In this section we pesent an evaluation of the poposed algoithm in tems of quality and unning time. We will compae the object list given by ou solution to the object list given by a 77 GHz long ange ada in a taffic situation. Fo this eason we popose two scenaios: the fist scenaio is following just one vehicle fo a numbe of fames and see how many times ou algoithm is not able to detect the vehicle and the ada is able to detect it; the second scenaio is in an intesection whee thee ae multiple cas and, fo a numbe of fames, we count how many vehicles does ou solution miss compaed to the ada appoach. Even though the two sensos ae diffeent we have selected scenes whee the weathe conditions ae good fo both LIDAR and RADAR, and the objects can be detected by both sensos. The system on which we have tested ou method contains an Intel i CPU with 3 GHz fequency, no hadwae acceleation methods have been used in the algoithm. Both LIDAR and RADAR objects ae in the same efeence fame and the RADAR objects have also been motion coected to a common timestamp with the LIDARs. Fo simplicity easons we have pojected, in a top view manne, the 3D RADAR objects into the same vitual image as the LIDAR objects, and to each 3D wold object we have associated a 2D vitual object. Fo each vitual LIDAR object we ty to identify the RADAR object that is closest to it, in a cicle of adius 60 pixels in the vitual image. In case thee ae seveal objects in this cicle we also look to the object that has the dimensions most simila to the LIDAR vitual object. In Fig. 11 we illustate an association between a LIDAR object and a RADAR object. Please note that the RADAR object is in the association cicle of the LIDAR. Fig. 12. Senso object associtation in an intesection In Table I we pesent the numbe of unassociated ada objects, as pecentage, fo a single oad object, fo 300 fames. In the second ow of the table, we pesent the pecentage of unassociated oad objects fo ove 1000 images. We would like to mention the fact that in some scenaios the LIDAR might not be able to captue the object in font of it due to occlusion, while the RADAR might be able to identify it. We came acoss this scenaio when we evaluated the algoithm in a cowded intesection whee multiple consecutive vehicles ae placed one in font of each othe. This phenomenon happens because of the sensos positioning on the vehicle. Fig. 13. Patialy occluded object scenaio; the top image epesents the data fom the camea; bottom image epesents the top view image containing LIDAR and RADAR objects. Fig. 11. LIDAR-RADAR object association

6 In figue 13 we can see that the vehicle in font is coectly detected and associated to a RADAR object. The second vehicle fom ou position is not detected by the LIDAR algoithm due to the fact that the fist vehicle in font of the ego vehicle is obstucting the field of view. The achieved pocessing fame ate of the poposed algoithm on the specified hadwae is of 15 fames pe second. TABLE I. OBJECT DETECTION ACCURACY Scenaio N. of fames Accuacy Single object detection % Multiple object detection % V. CONCLUSIONS AND FUTURE WORK In this pape we have poposed an oiginal eal time solution fo detecting objects using a spase 4L LIDAR. One of the main easons fo which classical detection schemes might fail to detect objects when using 4L LIDARs is the 3D points spaseness. Anothe eason which has detemined us to investigate this poblem is the high pice of dense laye (32L, 64L) LIDARs. In the development of ou algoithm we have povided solutions fo poblems like motion coection, tempoal fusion, oad detection, elevation gid ceation and so on. The oiginal object list is filteed using size constaints such that buildings o othe vey lage stuctues ae eliminated. The six step algoithm was able to successfully detect objects in eal time in vaious taffic scenaios. Fo evaluating ou solution we used objects povided by a RADAR senso and we have pefomed a 2D object association to view how many RADAR objects do not get coupled to the LIDAR objects. In futue wok we will ty to coect the positions of the points belonging to moving objects using thei elative velocity; in the cuent appoach we ae using only using the ego speed fo the motion coection. Anothe impovement that would incease the obustness of ou solution would be the implementation of object tacking fo the identified objects. ACKNOWLEDGMENT This wok was suppoted by the EU H2020 poject, Up- Dive unde gant n This wok was also suppoted by the MULTISPECT gant (Multispectal envionment peception by fusion of 2D and 3D sensoial data fom the visible and infaed spectum) of the Romanian National Authoity fo Scientific Reseach and Innovation / UEFISCDI, poject code PN-III-P4-ID-PCE , contact numbe 60/2017. REFERENCES [1] R. Muphy, Intoduction to AI obotics, MIT pess, [2] M. P. Muesan, S. Nedevschi and R. Danescu, "Patch waping and local constaints fo impoved block matching steeo coespondence," 2016 IEEE 12th Intenational Confeence on Intelligent Compute Communication and Pocessing (ICCP), Cluj-Napoca, 2016, pp [3] J. Žbonta and Y. LeCun, "Computing the steeo matching cost with a convolutional neual netwok," 2015 IEEE Confeence on Compute Vision and Patten Recognition (CVPR), Boston, MA, 2015, pp [4] S. Hwang, N. Kim, Y. Choi, S. Lee and I. S. Kweon, "Fast multiple objects detection and tacking fusing colo camea and 3D LIDAR fo intelligent vehicles," th Intenational Confeence on Ubiquitous Robots and Ambient Intelligence (URAI), Xi'an, 2016, pp R. Nicole, Title of pape with only fist wod capitalized, J. Name Stand. Abbev., in pess. [5] M. Montemelo, J. Becke, S. Bhat, H. Dahlkamp, D. Dolgov, S. Ettinge, D. Haehnel, T. Hilden, G. Hoffmann, B. Huhnke, D. Johnston, S. Klumpp, D. Lange, A. Levandowski, J. Levinson, J. Macil, D. Oenstein, J. Paefgen, I. Penny, A. Petovskaya, M. Pfluege, G. Stanek, D. Stavens, A. Vogt, S. Thun, Junio: The stanfod enty in the uban challenge, Jounal of Field Robotics 25 (9) (2008) [6] F. J. Botha, C. E. van Daalen and J. Teunicht, "Data fusion of ada and steeo vision fo detection and tacking of moving objects," 2016 Patten Recognition Association of South Afica and Robotics and Mechatonics Intenational Confeence (PRASA-RobMech), Stellenbosch, 2016, pp [7] Diveless: Intelligent Cas and the Road Ahead By Hod Lipson, Melba Kuman [8] Behley, J., Steinhage, V., Cemes, A.B.: Pefomance of histogam desciptos fo the classification of 3D lase ange data in uban envionments. In: ICRA. pp IEEE [9] Benedek, C., Moln á, D., Szi ányi, T.: A Dynamic MRF Model fo Foegound Detection on Range Data Sequences of Rotating Multi- Beam Lida. In: Intenational Wokshop on Depth Image Analysis, LNCS. Tsukuba City, Japan (2012) [10] Rusu, R.B., Cousins, S.: 3D is hee: Point cloud libay (pcl). In: Intenational Confeence on Robotics and Automation. Shanghai, China (2011) [11] Azim, A., Aycad, O.: Detection, classification and tacking of moving objects in a 3D envionment. In: Intelligent Vehicles Symposium. pp (2012) [12] Himmelsbach, M., Mulle, A., Luttel, T., Wunsche, H.J.: LIDAR-based 3D Object Peception. In: Poceedings of 1st Intenational Wokshop on Cognition fo Technical Systems. M ünchen (Oct 2008) [13] M. Hebet, C. Caillas, E. Kotkov, I. S. Kweon, T. Kanade, Teain mapping fo a oving planetay exploe, in: Robotics and Automation, Poceedings IEEE Intenational Confeence on, IEEE, 1989, pp [14] Roth-Tabak, R. Jain, Building an envionment model using depth infomation, Compute 22 (6) (1989) [15] H. Moavec, Robot spatial peception by steeoscopic vision and 3d evidence gids, Peception,(Septembe). [16] M. Oliveia, V. Santos, A. Sappa, P.Dias, Scene epesentations fo autonomous diving: an appoach based on polygonal pimitives, in: 2 nd Ibeian Robotics Confeence, [17] Floin Oniga, Segiu Nedevschi, Pocessing Dense Steeo Data Using Elevation Maps:Road Suface, Taffic Isle and Obstacle detection, IEEE Tansactions on Intelligent Tanspotation Systems, vol 12 No 4, Decembe 2011, pp [18] Zeisle, J., and H. G. Maas. "ANALYSIS OF THE PERFORMANCE OF A LASER SCANNER FOR PREDICTIVE AUTOMOTIVE APPLICATIONS." ISPRS Annals of Photogammety, Remote Sensing and Spatial Infomation Sciences (2015):