Height restriction barriers detection from traffic scenarios using stereo-vision

Transcription

1 Heght restrcton barrers detecton from traffc scenaros usng stereo-vson Mara-Irna Barbu, Ion Gosan, Tberu Marța Computer Scence Department Techncal Unversty of Cluj-Napoca, Romana {Ion.Gosan, Abstract Heght restrcton barrers detecton s mportant usually for trucks drvng assstance systems. In ths paper we propose a novel approach that uses stereo-vson and combnes ntensty and depth nformaton for heght barrers detecton and trackng. Hgh qualty stereo-reconstructon s carred out by the SORT-SGM algorthm. Canny edges are extracted from 2D ntensty mage and fltered out by the 3D nformaton. Horzontal lnes are determned usng the Hough transformaton on the fltered edges and then valdated by an ntensty correlaton approach takng nto consderaton that usually heght restrcton barrers have a repettve textural pattern. Neghborng lnes are clustered together n order to form the barrer regon of nterest. The heght of the barrer s also approxmated from the 3D ponts that belong to the barrer s regon of nterest. SURF features are extracted for the detected barrers from the ntensty mage and used for trackng them across frames. The whole heght restrcton barrers detecton system performs real tme wth hgh accuracy results. Keywords heght restrcton barrers; ntensty nformaton; depth nformaton; detecton; heght estmaton; trackng I. INTRODUCTION Nowadays, drvng assstance systems are frequently used n many ntellgent vehcles. The most common functons ntegrated n a drvng assstance system are: obstacle collson warnng, speed keepng assstance, lane departure warnng, lane keepng assstance, traffc sgn recognton, pedestran collson warnng and avodance etc. These are used n on-board drvng assstance systems on passenger cars whch usually don t have very hgh heght. Fgure 1. Coach collson wth a traffc heght restrcton barrer n Tanjn, Chna [1] In case of busses and trucks, ther hgher heghts may cause unexpected collsons wth traffc heght restrcton barrers due to the drver nattenton or even due to the fact that ther heght s not sgnalzed and the drver doesn t correctly apprecate t (see Fgure 1). Ths motvates the researchers for developng robust solutons for preventng and avodng any collsons of busses or trucks wth heght restrcton barrers. Many dfferent sensors lke RADARs, LIDARs, ultrasound sensors, nfrared sensors and vdeo cameras may be used for acqurng the traffc scene nformaton. Among these sensors, we prefer a par of two gray levels cameras n stereo setup for scene mages acquston. Stereo-vson [2] offers both pxel ntensty nformaton and also the possblty of computng depth. Usually the obstacles texture [3] may be detected from the ntensty mages whle the depth nformaton [4] may be very useful for search space reducton and for valdatng them [5]. In our case, we are focusng on detectng the heght restrcton barrers regons of nterests n the traffc scenes by usng both ntensty and depth nformaton. A barrers detecton method usng stereovson was proposed n [6], but the subject s not so common n the lterature. A hgh qualty of stereo-reconstructon s absolutely necessary for obtanng hgh accurate 3D reconstructed scene ponts. We use the SORT-SGM stereo-reconstructon algorthm [7] mplemented on a GPU that offers 3D ponts accuracy n a very low processng tme. Even wth a hgh qualty stereoreconstructon avalable, the 3D ponts from the neghborhood of the heght restrcton barrers are not so accurate or the reconstructed ponts mght be mssng. Ths s due of the fact that stereo cameras are dsplaced left-rght n the stereo setup confguraton and the barrers are also horzontal structures. The methods that perform traffc obstacle detecton by means of 3D ponts groupng [5, 8] cannot be appled for detectng a heght barrer. Usually the ponts that are hgher than a specfed threshold or they are belongng to hangng obstacles are usually rejected by these approaches because they affect the on-road obstacle detecton. Although there are some methods for determnng the elevaton map [9], road surface [10] or the drvng area [11], n our approach we use only the prmary 2D ntensty nformaton and the 3D reconstructed ponts and make the assumpton of a planar road n front of the ego-vehcle. We combne the ntensty Canny edge detecton [12] result wth the power of the Hough transform [13] for detectng the barrer s canddate horzontal lnes. The assumpton of repettve

2 texture pattern of the heght barrers s made and a correlaton score computaton approach s proposed for selectng the lnes that are on the barrer surface. A clusterng procedure s also proposed for groupng the barrer s lnes wth the closest canddate lnes n order to determne the entre regon of nterest of the barrer. A trackng procedure based on SURF features [14] s used for keepng the detected barrer across frames, n the stuatons when the barrer t s not detected n the current frame but t was prevously detected n other frames. All these steps are descrbed n detals n the next sectons. II. SYSTEM ARCHITECTURE In Fgure 2, the components of our proposed heght restrcton barrers detecton and trackng system, the relatonshps between them and also the nput and output of each module are depcted. Gray levels mages of the traffc scenes are acqured wth a par of stereo-vson cameras. A hgh qualty stereoreconstructon s acheved by usng the SORT-SGM algorthm [7] whch offers accurate and dense depth map, whch are very mportant for the subsequent processng steps. Heght restrcton barrers are clearly dstngushable from the neghborng background. Canny algorthm s employed for detectng all the scene edges ncludng those that separates the barrers from the background. In order to select only the edges near the barrer s structure, we flter out all the other edges based on some heght constrants. The heghts values are known for all the 3D reconstructed ponts. Hough lnes detecton follows the edge detecton. It helps fndng the canddate lnes of the barrer. Usually the heght restrcton barrers have a repettve textural pattern, so a texture correlaton valdaton procedure s proposed for selectng the lnes that le onto the barrer. Near the true lnes of the barrer there may appear some canddate lnes that also belong to the barrer, so we cluster all the lne canddates wth the barrer s lnes based on ther vcnty. A cluster of lnes s consdered to defne a detected barrer. Across sequences of multple frames, there can be few ntercalated frames n whch the barrer s not detected. Ths fact can appear due to multple factors. A trackng procedure based on SURF features of the detected barrer n the prevous frames s proposed for solvng ths problem n the current frame. The output of the entre system conssts n sgnalzng the presence of the heght barrers and ther correspondng heghts n traffc mages. III. METHOD DESCRIPTION In ths secton, we descrbe n detals all the modules of the heght restrcton barrers detecton and trackng system. Stereo-vson grayscale mages wth 512x383 pxels are acqured. Preprocessng algorthms for correctng the dstortons and mage rectfcaton are also appled. Stereo-reconstructon s performed usng a hgh qualty SORT-SGM algorthm. In Fgure 3 we present an example of stereo-reconstructed ponts. At each row and column j n the left mage, the correspondng stereoreconstructed pont t s drawn wth one of the three dfferent colors cyan, magenta and brown accordng to ts heght Y n the scene: Fgure 2. Heght restrcton barrers detecton and trackng system archtecture Color( Pont ) brown, f Y ( Pont ) 3000 mm cyan, f 200 mm Y ( Pont ) 3000 mm magenta, f Y ( Pont ) 200 mm The orgn (0, 0, 0) of the scene system coordnates (X, Y, Z) s stuated exactly n front of the ego-vehcle, n ts mddle and on the road surface. (1)

3 a) a) b) Fgure 3. Stereo-reconstructon: a) left ntensty mage; b) 3D reconstructed ponts projected onto the left mage b) Fgure 4. Edges on the left mage: a) orgnal edges obtaned from the Canny algorthm; b) fltered edges usng some constrants Canny edges are extracted n the left mage consderng the two hysteress thresholds T low =50 and T hgh =200. For the next steps we keep nto analyss only the edge ponts at postons (, j) that have a stereo-reconstructed pont assocated whch also satsfy the followng constrants: Y( Pont ) 2500 mm Y( Pont ) 5000 mm Z( Pont ) mm The constrants (2) take the assumpton that a traffc heght restrcton barrer s usually placed between 2,5m and 5m heght and also restrct the scene searchng space not far away than 30m. The extracted edges on the scene depcted n Fgure 3 and also the fltered edges that wll be used for the next processng steps are presented n Fgure 4. (2) We use a probablstc Hough method [15] appled on the fltered edges n order to determne all the mage lnes ncludng the real ones that belong to the heght restrcton barrers. The parameters determned emprcally and set for ths method are: d 1pxel d 1 MnVotes 50 MnLneLength 20 pxels MaxLneGap 40 pxels The lnes that are consdered to be canddates for the heght restrcton barrer have a maxmum drft of ±5º from the horzontal drecton. We make the assumptons of traffc scene scenaro wth planar roads and the ego-vehcle that s (3)

4 approachng a heght barrer havng a very small roll angle. Only these lnes are consdered for the next steps. Not all these canddate lnes are stuated on a heght restrcton barrer. In order to valdate them, we analyze the pxel ntenstes along them wth the purpose of selectng only the lnes that have the ntenstes formng a repettve pattern. We terate all the pxels located on each canddate lne and store them n an array I. Several correlaton coeffcents Corr k for each ntensty lag k are computed: ti tki I () t I( t k) Corr k I k mncorr mn Corr k maxcorr max Corr lowlag, hghlag k The drft range for k was selected emprcally: lowlag=10, hghlag=70. The mnmum and maxmum values for all the correlaton coeffcents are computed and the lne s consdered to be part of a heght restrcton barrer only f these two boundary values are satsfyng the followng condtons: mncorr lowthresholdcorr (5) maxcorr hghthresholdcorr We use the two thresholds values: lowthresholdcorr=50 and hghthresholdcorr=85. It s obvous that there can exst other canddate lnes that are close to these heght restrcton barrer detected lnes and whch should be taken nto consderaton. A clusterng procedure s proposed n order not to mss detect some canddate lnes. Every barrer detected lne d form a cluster wth other canddate lne c f the extremtes of the canddate lne P 1c and P 2c are closer than a threshold tbetween=10 to the detected lne and there s at least one pont P 1c or P 2c closer than a threshold taway=40 from one of the extremtes P 1d or P 2d of the detected lne: k, (4) In Fgure 5, the boundng rectangle of the heght restrcton barrer s drawn wth blue color. It s the largest cluster from all the clusters contanng the detected barrer lnes and the canddate lnes that were valdated wth the constrants (6). Fgure 5. Heght restrcton barrer boundng rectangle (wth blue color); detected barrer lnes (red color); vald canddate lnes (green color) For approxmatng the barrer s heght we take nto consderaton all the reconstructed 3D ponts closer than 30m whch have the 2D left mage projectons nsde the prevously determned boundng barrer rectangle. The heght of the barrer s selected to be the medan value of the heghts: Heghts { Y ( Pont ) Z( Pont ) mm Projecton( Pont ) P P barrerrectangle} barrerheght medan( Heghts) (8) d( P, P ), c( P, P ) 1d 2d 1c 2c dstance( d, P ) tbetween 1c dstance( d, P ) tbetween 2c (, j) P { P, P } P { P, P } 1c 2c j 1d 2d dstance( P, P ) taway After ths step, a number nc of barrer lnes clusters C, each contanng one pure detected barrer lne and other canddates are formed. The boundng rectangle boundngrectangle of every cluster s determned and the real heght restrcton barrer boundng rectangle barrerrectangle s chosen to be the wdest one: j (6) barrerrectangle wdest( boundngrectangle ) (7) C 1, nc Fgure 6. Detected heght restrcton barrer (boundng rectangle wth red color) wth the specfed heght above (wth blue color)

5 The computed heght (wth approxmaton of one decmal) for the detected heght restrcton barrer n the current frame s dsplayed above ts boundng rectangle (see Fgure 6). Accordng to our methods steps proposed above for heght restrcton barrer detecton, there can appear few ntercalated frames n a sequences of frames n whch the proposed methodology doesn t detect any barrer even f t s present. In order to fll n these gaps we propose a trackng procedure of the detected barrers n consecutve frames. Ths wll ncrease the robustness of our method. The SURF key ponts K and the assocated feature vector FV to every key pont are extracted from the detected barrer boundng rectangle n the frame P. The trackng procedure s appled n the case when n the current frame C there sn t detected any barrer and n the prevous frame P exsted one. The same key ponts and feature vectors are extracted for the scene n the current frame. A fast approxmated nearest neghbor search algorthm [16] s used for matchng the barrer s features from the prevous frame aganst the scene features. The barrer s consdered to be tracked only f the mnmum matchng score dstance s lower than a threshold tfeatdst=0.02: K { k }, FV { FV }, 1,..., nrkeyponts mn dstance( FV, FV ) tfeatdst 1, K P We fnd then a perspectve transformaton between the prevous frame P and the current frame C by applyng a RANSAC algorthm [17] on the matchng pars of key ponts between the detected barrer n frame P and the correspondng ponts n the current frame C. The RANSAC algorthm fnds the perspectve transformaton matrx parameters M. The boundng rectangle of the barrer n the prevous frame P s projected nto the current frame C by usng the transformaton matrx M. A fnal refnement s employed for flterng out spurous tracked barrers. The number of edge fltered ponts that are nsde the barrer boundng rectangle n the current frame C are counted. It s used for computng the edges densty n the barrer boundng rectangle. A barrer s consdered to be a good track only f t has the edge ponts densty above a threshold tedges=0.1: C (9) Fgure 7. Tracked heght restrcton barrer (boundng rectangle wth magenta color) of the detected barrer from Fgure 6 IV. EXPERIMENTAL RESULTS In ths secton, we present the proposed heght restrcton barrer detecton and trackng system s results. The results were obtaned from two stereo-vson mage sequences contanng heght restrcton barrers havng thousands of frames. In terms of qualtatve evaluaton shows that the heght restrcton barrers n front of the ego vehcle are well detected and tracked. Mss detectons can occur due to some poor stereoreconstructed ponts around the barrer or due to the drt on the barrer s surface that sometmes may affect the correlaton score of checkng the repettve textural pattern. False alarms may occur n scenaros wth non-planar road surface where the estmated heghts aren t correct at all or rarely on frontal buldngs wth many horzontal edges and repettve texture. Some sample results ncludng three correctly detected barrers and a false postve are presented n Fgure 8. EP { Pont (, j) boundngrectangle} EP edgedensty Area( boundngrectangle) edgedensty tedges (10) In Fgure 7, an example of a tracked heght restrcton barrer s presented. The mage represents the next frame of the mage from Fgure 6, where a detected heght restrcton barrer was detected. The boundng rectangle n Fgure 7 s obtaned by the perspectve projecton of the prevous frame barrer boundng rectangle and t s drawn wth the magenta color. The heght of the tracked barrer s dsplayed above and t s coped from the prevously detected barrer. a)

6 In terms of quanttatve evaluaton, the system has the followng performance: approx. 95% true postve rate n heght barrers detecton less than one false postve heght barrer detected at 200 frames barrers heght estmaton error n the range of [ 0.2m, +0.2m] b) All the experments were done n good weather condtons wth hgh vsblty traffc envronment. Adverse condtons lke ran, nght tme, fog are obvously reducng the system s performance. However these scenaros were not nvestgated and they are beyond the scope of ths paper. The entre processng necessary for the heght restrcton barrers detecton and trackng was done on a sngle core of an Intel Core GHz, the system achevng an executon tme less than 40ms/frame, resultng a processng power of more than 25 fps. c) V. CONCLUSIONS We proposed a novel approach for heght restrcton barrers detecton and trackng by processng stereo-vson mages. Our method can be successfully ntegrated wthn an advanced drvng assstance system for avodng mpacts wth these knd of structures. The system combnes the 3D nformaton used for reducng the search space wth the 2D ntensty nformaton that offers the possblty of horzontal structures detecton and ther valdaton by analyzng the repettve textural patterns specfc to heght restrcton barrers. These structures are clustered together n order to determne the boundng rectangle of the heght restrcton barrers. A trackng procedure s also proposed for mprovng the prevously mentoned detecton method n order to acheve contnuous detectons n consecutve frames. The system performs real tme and offers hgh accuracy results. As a future work, n order to mprove the system s performance, we nvestgate the possblty of usng a road surface estmaton module whch elmnates the flat road assumpton and a better module for valdatng the barrers lnes by consderng multple features not only on ther repettve textural pattern. It would be also great to nvestgate the possblty of detectng the heght restrcton barrers n lowvsblty traffc scenaros n order to help the drver avodng such an obstacle n dffcult vsblty condtons. d) Fgure 8. Sample results of heght restrcton barrers detecton: a)-c) true postves; d) false postve ACKNOWLEDGMENT Ths work has been supported by UEFISCDI (Romanan Natonal Research Agency) n the natonal research project Mult-scale Mult-modal Percepton of Dynamc 3D Envronments Based on the Fuson of Dense Stereo, Dense Optcal Flow and Vsual Odometry Informaton (MultSens), project code PN-II-ID-PCE

7 REFERENCES [1] ChnaDaly.com.cn. (2015). At least 2 dead, dozens hurt after bus hts road barrer. Avalable: [2] S. Nedevsch, R. Danescu, D. Frentu, T. Marta, F. Onga, C. Pocol, et al., "Hgh accuracy stereo vson system for far dstance obstacle detecton," n IEEE Intellgent Vehcles Symposum, 2004, pp [3] J. Malk, S. Belonge, T. Leung, and J. Sh, "Contour and Texture Analyss for Image Segmentaton," Internatonal Journal of Computer Vson, vol. 43, pp. 7-27, [4] Z. Zhen, W. Yfe, J. Brand, and N. Dahnoun, "Real-tme obstacle detecton based on stereo vson for automotve applcatons," n European DSP Educaton and Research Conference, 2012, pp [5] C. Pocol, S. Nedevsch, and M. M. Menecke, "Obstacle Detecton Based on Dense Stereovson for Urban ACC Systems," n Internatonal Workshop on Intellgent Transportaton (WIT), Hamburg, Germany, 2008, pp [6] T. Marta, "Barrers detecton method for stereovson-based ACC systems," n Proceedngs of Internatonal Conference on Intellgent Computer Communcaton and Processng 2009, pp [7] C. D. Pantle and S. Nedevsch, "SORT-SGM: Subpxel Optmzed Real- Tme Semglobal Matchng for Intellgent Vehcles," IEEE Transactons on Vehcular Technology, vol. 61, pp , [8] C. Pocol, S. Nedevsch, and M. A. Obojsk, "Obstacle Detecton for Moble Robots, Usng Dense Stereo Reconstructon," n IEEE Internatonal Conference on Intellgent Computer Communcaton and Processng, 2007, pp [9] F. Onga, S. Nedevsch, M. M. Menecke, and T. Thanh-Bnh, "Road Surface and Obstacle Detecton Based on Elevaton Maps from Dense Stereo," n IEEE Intellgent Transportaton Systems Conference, 2007, pp [10] F. Onga and S. Nedevsch, "Processng Dense Stereo Data Usng Elevaton Maps: Road Surface, Traffc Isle, and Obstacle Detecton," IEEE Transactons on Vehcular Technology, vol. 59, pp , [11] R. Danescu, F. Onga, and S. Nedevsch, "Modelng and Trackng the Drvng Envronment Wth a Partcle-Based Occupancy Grd," IEEE Transactons on Intellgent Transportatn Systems, vol. 12, pp , [12] J. Canny, "A Computatonal Approach to Edge Detecton," IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 8, pp , [13] R. O. Duda and P. E. Hart, "Use of the Hough transformaton to detect lnes and curves n pctures," Communcatons of the ACM, vol. 15, pp , [14] H. Bay, A. Ess, T. Tuytelaars, and L. V. Gool, "Speeded-Up Robust Features (SURF)," Journal of Computer Vson and Image Understandng, vol. 110, pp , [15] J. Matas, C. Galambos, and J. Kttler, "Robust Detecton of Lnes Usng the Progressve Probablstc Hough Transform," Computer Vson and Image Understandng, vol. 78, pp , 4// [16] M. Muja and D. G. Lowe, "Fast approxmate nearest neghbors wth automatc algorthm confguraton," n VISAPP Internatonal Conference on Computer Vson Theory and Applcatons, 2009, pp [17] M. A. Fschler and R. C. Bolles, "Random sample consensus: a paradgm for model fttng wth applcatons to mage analyss and automated cartography," Communcatons of the ACM, vol. 24, pp , 1981.