FLane: An Adaptive Fuzzy Logic Lane Tracking System for Driver Assistance

Transcription

1 Gulo Rena 1 Department of Engneerng for Innovaton, Unversty of Salento, Va per Arnesano, Lecce, Italy e-mal: gulo.rena@unsalento.t Annalsa Mlella Insttute of Intellgent Systems for Automaton, Natonal Research Councl, Va G. Amendola 122/D, Bar, Italy e-mal: mlella@ba.ssa.cnr.t FLane: An Adaptve Fuzzy Logc Lane Trackng System for Drver Assstance In the last few years, drver assstance systems are ncreasngly beng nvestgated n the automotve feld to provde a hgher degree of safety and comfort. Lane poston determnaton plays a crtcal role toward the development of autonomous and computer-aded drvng. Ths paper presents an accurate and robust method for detectng road markngs wth applcatons to autonomous vehcles and drver support. Much lke other lane detecton systems, ours s based on computer vson and Hough transform. The proposed approach, however, s unque n that t uses fuzzy reasonng to combne adaptvely geometrcal and ntensty nformaton of the scene n order to handle varyng drvng and envronmental condtons. Snce our system uses fuzzy logc operatons for lane detecton and trackng, we call t FLane. Ths paper also presents a method for buldng the ntal lane model n real tme, durng vehcle moton, and wthout any a pror nformaton. Detals of the man components of the FLane system are presented along wth expermental results obtaned n the feld under dfferent lghtng and road condtons. DOI: / Introducton Wthn the last few years, research nto ntellgent vehcles has greatly expanded. Systems that montor drver ntent, warn drvers of lane departure, or provde vehcle assstance are all emergng. Specfcally, lane detecton and trackng s a well-researched problem n computer vson wth a wde range of applcatons n autonomous vehcles and drver support systems. Lane detecton can be employed n the followng drver assstance applcatons 1 : Lane-departure-warnng system. The system predcts the trajectory of the vehcle wth respect to the lane boundary 2 Fg. 1 a. Drver-attenton montorng system. The system montors the drver s attentveness to the lane-keepng task usng parameters such as the smoothness of the lane followng 3 Fg. 1 b. Automated vehcle-control system. The system automatcally gudes safely the vehcle wthn the lane by controllng the lateral poston error 4 Fg. 1 c. Fndng whte markngs on a dark road can turn nto a very complex problem when shadows, physcal barrers, occlusons by other vehcles, changes n road surfaces, and dfferent types of lane markngs come nto play. A robust and effcent lane detecton system must be able to flter out all dsturbances and extract the markngs of nterest from cluttered roadways n order to produce an accurate and relable estmate of the vehcle poston relatve to the road. In Fg. 2, a sample mage set demonstrates the varety of road and envronmental condtons that can be encountered. Fgure 2 a shows a scene where lane detecton can be consdered relatvely easy thanks to a clearly defned, sold markng and a unform road texture. In Fg. 2 b, extracton of road markng s more dffcult due to the presence of a curb and a manhole cover. Fgure 2 c shows a more complex road markng wth transversal sold lnes due to sde road enters, whle n Fg. 1 Correspondng author. Contrbuted by the Dynamc Systems Dvson of ASME for publcaton n the JOURNAL OF DYNAMIC SYSTEMS, MEASUREMENT, AND CONTROL. Manuscrpt receved July 9, 2008; fnal manuscrpt receved: July 23, 2010; publshed onlne February 11, Assoc. Edtor: Marco de Queroz. 2 d a nonunform road texture s shown. Fnally, Fgs. 2 e and 2 f refer to low lghtng scenes due to overpasses and nght tme. Many researchers have developed lane detectors based on varous technques. A commonly used approach s the Hough transform, whch fts lnes to detected edges 5,6. Ths approach typcally suffers from heavy computatonal requrements that make t a dffcult real-tme mplementaton and can easly fal n stuatons where many extraneous lnes exst. Neural networks have been used to attempt to detect lanes and control vehcles 7 but have dffcultes on roads not ncluded n ther tranng set. Technques usng tangent vectors have also been demonstrated to be qute robust on well-marked roads but can fal when lane markngs are not well-defned 8. Other researchers have attempted to overcome problems of dfferng lane markngs by usng multple detectors. For example, Gehrg et al. 9 detected bots dots on Calforna hghways usng specfc matched flters and detected sold lane markngs usng more classcal methods. Others, such as the authors of Refs , proposed the use of partcle flterng to mprove robustness to lghtng and road changes, whle Bertozz and Brogg 13 developed the generc obstacle and lane detecton GOLD system for robust obstacle and lane detecton. McCall and Trved 14 used steerable flters for accurate and robust lane markng detecton. Frequency-based technques, such as the lane-fndng n ANother doman LANA system 15, have been shown to be effectve n dealng wth extraneous edges. Other technques, such as the rapdly adaptng lateral poston handler RALPH system 16, based the lane poston on an adaptve road template. Such technques generally assume a constant road surface texture and can fal n stuatons lke the one n Fg. 2 d. Whle these methods are all very effectve at performng lane detecton n several contexts, they tend to be hghly nfluenced by the road type or condtons. Robust lane detecton remans, therefore, an open research area, snce, n order to have a robust lane detector, the system must be nvarant to dfferent road markngs, road condtons, lghtng changes, shadowng, and occlusons. In ths paper, we nvestgate an alternatve method based on a Hough transform enhanced by fuzzy reasonng to provde a realtme, robust, and accurate lane detecton and trackng system n hghly dynamc envronments. Fuzzy logc allows one to cope wth complex dynamcal contexts that are dffcult to model wth mathematcal approaches 17,18. A few authors have proposed Journal of Dynamc Systems, Measurement, and Control MARCH 2011, Vol. 133 / Copyrght 2011 by ASME

2 fuzzy systems n the context of lane detecton n order to mprove specfc functons, such as edge detecton 19,20. Here, nstead, fuzzy logc serves as a general framework to deal wth the whole process of lane detecton and trackng. Although lane detecton systems have been extensvely studed, one commonly-recognzed ssue s the lack of unform performance characterzaton methodologes 1. Several metrcs have been proposed but they all tend to be very specfc. In addton, most proposed algorthms have shown lmted numercal results or sample mages to demonstrate the performance of the algorthms, thus makng a quanttatve comparson across dfferent classes of algorthms very dffcult. In ths paper, we evaluate the entre vson-based algorthm for lane trackng by measurng the occurrence rate of false postves, false negatves, and msdentfcatons. A key ssue of lane detecton systems s that of defnng an adequate model for the lane markng to be tracked over subsequent mages. Ths s partcularly challengng at the start of the vehcle moton, when no pror nformaton s avalable, and whenever the system fals and starts over. In ths respect, the FLane system features a specal module, referred to as dynamc model buldng DMB. The DMB module provdes onlne lane model constructon by processng a short sequence of mages usng what we call the cumulatve Hough matrx CHM n conjuncton wth fuzzy logc operatons. Extensve testng of the proposed approach s performed wth a commercal automoble equpped wth a low cost webcam and operatng under dfferent drvng and envronmental condtons. The results demonstrate that fuzzy reasonng s a proper framework to operate under uncertanty n vsual data for lane detecton and drve montorng applcatons. Theoretcal detals of the FLane method and ts modules are provded n Sec. 2. Expermental results to valdate ths approach and assess the system performance are presented n Sec. 3. Fnally, Sec. 4 concludes ths paper. Fg. 1 Drver assstance systems that requre lane poston: a lane-departure warnng, b drver-attenton montorng, and c vehcle-control 2 The FLane System The FLane module performs ts task usng a robust Hough transform, enhanced by fuzzy logc operatons, whch provde the system wth the ablty to adapt rapdly to varyng operatonal condtons. In ths secton, a theoretcal analyss of the method s presented, also provdng expermental evdences of ts effectveness n the feld. Fg. 2 Sample mages of road markngs and condtons: a smple road wth sold lane markng, b dsturbances due to curb and manhole cover, c transversal sold lnes due to sde road enters, d nonunform pavement texture, e freeway overpass causng lghtng change and reducng road-markng contrast, and f low lghtng and shadowng at nght tme / Vol. 133, MARCH 2011 Transactons of the ASME

3 Fg. 3 Model of the lane markng n the mage plane. Note that the parameters 1, 2, and are expressed n pxels. 2.1 Lane Model. The presence of a sdeward-facng camera mounted on the vehcle body wth a feld of vew on the ground plane correspondng to a 90 cm long 120 cm wde area s assumed. It s also consdered that the locaton of the camera relatve to the ground s known and fxed durng travel. Although ths assumpton s of lmted valdty snce a car s suspenson system allows body tltng, n a prevous work 21, ths approach was proved to be rather robust to relatvely small changes n the poston and orentaton of the camera wth respect to the ground. Under the further assumpton that the porton of the lane markng LM n the mage s relatvely small, the lane markng curvature can be neglected, and t s possble to refer to a lane model composed of a par of parallel lnes wth constant offset. In the mage reference frame S, denoted wth P 1 = 1, 1 and P 2 = 2, 2, the polar parameters of these two lnes, namely L 1 and L 2 as explaned n Fg. 3, the pose P=, of the LM model can be defned as = = In addton, the lane marker s characterzed by an ntensty level I m, whch s defned as the average gray level of the pxels comprsed between the lane marker borders. Real world nformaton can be obtaned wth good accuracy from mage data by nverse perspectve projecton technques, and the LM model can be defned n the real world Fg. 4 by the followng parameters the absolute value of the relatve angle between the borders L 1w and L 2w = and 2 beng the orentaton of the vehcle wth respect to L 1w and L 2w, respectvely; the absolute value of the wdth W of the marker, whch s equal to W = d 1 d 2 d 1 and d 2 beng the mnmum dstance of the vehcle relatve to L 1w and L 2w, respectvely. The varaton range for both and W can be consdered known by road legslaton. Ths turns nto two constrants that can be exploted for lane detecton,.e., = ˆ Fg. 4 Model of the lane markng n the real world. Note that the dstances d 1 and d 2 are expressed n mllmeters. W = Ŵ Note that ˆ s null, beng the lnes formng the marker parallel, and Ŵ ranges approxmately between 150 mm and 200 mm, dependng on road type. 2.2 Lane Trackng. The FLane system s composed of two man submodules: 1 The fuzzy edge detecton (FED) module. Ths module provdes an ntellgent fuzzy bnarzaton of the mage that allows the classcal Hough transform to be appled wth a substantally reduced computatonal requrement and a more robust and accurate mplementaton. 2 The fuzzy lane recognton (FLR) module. Ths module recognzes, between the lnes extracted from the mage, those lnes that best ft to the lane markng model. The selecton s performed by combnng geometrcal and ntensty data of the mage through fuzzy reasonng. It should be noted that the FLane system updates the reference lane at each new acquston. One crtcal aspect connected wth ths approach les n buldng the ntal model and updatng t after the system fals to detect the lane marker e.g., when false negatves arse or when no marker s present n the scene. In order to solve ths specfc problem, the FLane system employs the DMB module, as explaned n Sec It s also worth mentonng that the knowledge of the pose of the lane marker n one mage s used to determne the regon of nterest ROI to be processed for lane detecton n the next frame. Ths makes the lane search more accurate and reduces computatonal requrement by elmnatng much of the scene. Theoretcal detals of the FED, FLR, and DMB modules are presented n the remander of ths secton Fuzzy Edge Detecton. Hough transform s a commonly used technque to ft lnes to detected edges. However, t typcally suffers from heavy computatonal tme and ts performance largely depends on the result of the edge detecton process. In order to apply effectvely the Hough transform n real tme and n a hghly dynamc envronment, the FLane system employs a fuzzy logc-based edge detecton algorthm. The proposed approach s 6 Journal of Dynamc Systems, Measurement, and Control MARCH 2011, Vol. 133 /

4 Fg. 5 Membershp functon of the ntensty ndcator. If the degree of membershp s greater than a threshold T T=0.7 n our case, then the pxel s accepted, and t s set to 1 whte, otherwse t s dsregarded, and t s set to 0 black. ntended to extract whte lane markngs regardless of varatons n lghtng condtons and road texture characterstcs. It s suted to detect both sold and segmented lnes. The basc dea of the FED module s that of extractng mage ponts that satsfy two condtons: to have a specfed gray level and to belong to a border regon. To ths end, a gven pxel n the ROI s processed by what we defne an ntensty ndcator I that estmates the lkelhood that ths pxel belongs to the lane model. The I compares the ntensty level of the pxel wth the average ntensty value of the lane marker detected n the prevous frame. Our hypothess s that a large dfference n ntensty value suggests that the pxel does not belong to the model. We express ths hypothess usng fuzzy logc. The trangular membershp functon used for the I,.e., the curve that maps each pont n the nput space to a membershp value or grade between zero and one, s shown n Fg. 5. The ntensty ndcator uses one nput and one output. The nput s the relatve change n the ntensty level of the pxel j of the ROI of mage wth respect to the prevous frame 1 defned as I j = I 1 j I m I m where I 1 j s the ntensty level of the pxel j of the frame and I m s the average ntensty level of the lane markng detected n the frame 1. The output s a dmensonless factor rangng from zero to one that expresses the degree of confdence we have that the pxel j belongs to the model. It s mportant to notce that the performance of the FED module greatly depends on the value of the lower and upper lmts of the trangular membershp functon, I mn and I max, respectvely, n Fg. 5. In the proposed mplementaton, I max s well-expermentally determned as 90%, and I mn vares adaptvely, dependng on the average lghtng change wth respect to the prevous frame. The detals of the fuzzy-based regulaton of the lower bound are ncluded n Appendx for completeness. A typcal result of the thresholdng usng the I s shown n Fg. 6 for a sample mage. Specfcally, Fg. 6 a shows the orgnal mage ROI. Fgure 6 b depcts the bnary mage obtaned by the fuzzy thresholdng, whereas Fg. 6 c demonstrates the result of an ndependent Canny s edge detecton 22. Two bnary mages are avalable: one contanng ponts whose gray level s smlar to the gray level expected for the lane marker Fg. 6 b and one contanng strong edge ponts Fg. 6 c. A fnal bnary mage, sutable for Hough manpulaton, can be obtaned wth a Boolean AND-operaton, as shown n Fg. 6 d. Fg. 6 Results of a sample mage bnarzaton usng the fuzzy edge detecton module: a selected ROI, b fuzzy thresholdng usng the Intensty Indcator, c Canny edge detecton, and d Boolean AND of the two prevous operatons. Note that the negatves of the bnary mages are shown for vsualzaton s sake. One should note that relyng only on the edge detecton operator would have brought a more complex and uncertan thresholdng of the scene, as apparent by comparng Fg. 6 d wth Fg. 6 c Fuzzy Lane Recognton. A set of lane canddates s made avalable by applyng Hough transform to the output of the FED step. The FLR module allows the par of lnes that best agrees wth the lane model to be selected. The general approach s based on comparng the geometrcal propertes of each canddate wth those of the LM model n both the mage plane and the real world and defnng determnstc condtons for model matchng. The output of the FLR module s a fuzzy quantty that expresses our certanty that the lne par matches the lane model. If n lnes are detected n the mage, then, there wll be c lane marker canddates LM j wth j=1,2,...,c, and n! c = 8 2! n 2! In the mage plane, we can compute the pose P j = j, j for each one of the lane marker canddates LM j relatve to frame and compare ths value wth the pose of the lane marker obtaned n the prevous frame P 1 = 1, 1. Under the assumpton of a relatvely small dsplacement of the vehcle wth respect to the road markng between two consecutve frames, we can regard P 1 as a good reference value. If the lne par pose P j agrees wth P 1, then one can expect good correspondence between that par and the lane model. Poor correspondence suggests low lkelhood of matchng. Smlarly, we can compare the geometrcal propertes of the lane marker LM j n the real world,.e., W j and the orentaton j, wth the analogous parameters of the model obtaned from the prevous frame. A small dfference n the values of wdth and orentaton suggest hgh lkelhood of matchng of the canddate wth the model. We agan adopt fuzzy logc to express these hypotheses. The trangular membershp functons of the nference system for the FLR module are shown n Fg. 7. The fuzzy data fuson uses four nputs and one output. The nputs are the geometrcal data,.e., the absolute dfference n dstance and orentaton estmated n the mage plane, denoted wth j and j, between the canddate pose and the model pose n the prevous frame, and the absolute dfference n wdth and orentaton, denoted wth W j and j, respectvely, between the canddate and the model n the real world. The output s a dmensonless factor rangng from zero to one that expresses the degree of confdence we have that the lne par matches the lane model / Vol. 133, MARCH 2011 Transactons of the ASME

5 Fg. 7 Membershp functons of the FLR module The set of f-then rules used by the fuzzy nference system to fuse the geometrcal nformaton s shown n Table 1. Those rules express our physcal understandng of the problem, and they were chosen to gve the best performance over other alternatves usng a tral and error process. The rule set s not unque; new rules may be thought of and mplemented to mprove the output of the system. The output of the FLR module s shown n Fg. 8 overlad over the orgnal scene of the runnng example of Fg. 6. Fve lnes are obtaned by applyng Hough transform; thus, ten lane markng canddates exst. Table 2 collects the match confdence estmated Rule number Table 1 Fuzzy logc rules used by the FLR module Input Output j j W j j confdence Match 1 Small Small Small Small Hgh 2 Small Large Small Large Medum 3 Large Small Large Small Low 4 Large Large Large Large Low 5 Large Large Small Small Low 6 Small Small Large Large Medum Fg. 8 Fuzzy lane selecton appled to a sample mage. Fve lnes were selected formng ten lane marker canddates. Journal of Dynamc Systems, Measurement, and Control MARCH 2011, Vol. 133 /

6 Table 2 Degree of confdence for the lane markng canddates of Fg. 8, as derved by the FLR module Canddate number Lnes nvolved Match confdence % 1 L 1,L L 1,L L 1,L L 1,L L 2,L L 2,L L 2,L L 3,L L 3,L L 4,L Table 3 Degree of confdence for the lane markng canddates of Fg. 9 as derved by the FLR module Canddate number Lnes nvolved Match confdence % 1 L 1,L L 1,L L 1,L L 1,L L 2,L L 2,L L 2,L L 3,L L 3,L L 4,L by the FLR system for each canddate. As expected, the lane marker bounded by lnes L 1 and L 2 Fg. 8 yelds the greatest confdence level 86.0%, and t s, therefore, selected as the best match. It s worth notcng that the effcent flterng provded by the prevous FED module see Fg. 6 d, allowed all the detected lnes to be concentrated at the borders of the actual lane marker. The presence of multple lnes can be explaned when consderng the resoluton of the Hough space. Ths makes the task for the FLR system relatvely easy. However, ths s not always the case. As an example, Fg. 9 shows a dfferent scene where spurous road sgns appear, resultng n a fcttous peak n the Hough space due to lne L 2 n Fg. 9 b. Match confdences of the lane marker canddates for ths case are reported n Table 3. All the strpes comprsng the spurous lne are labeled wth low confdence by the FLR module and the lane model s correctly determned as the one formed by lnes L 3 and L 4, wth 82.1% of confdence, attestng to the feasblty of ths approach. Once the lane marker has been detected, the vehcle poston and orentaton relatve to the lane can be estmated usng nverse perspectve projecton 23, thus provdng a vald nput to vehcle-control and drver warnng systems Dynamc Model Buldng. The accuracy of a lane detector greatly depends on the accuracy of the model adopted for the road markng. The best choce of road model s tghtly connected wth the envronmental condtons n whch the system s used. For example, a statc model, bult upon the ntal geometrcal and ntensty propertes of the road lane, could soon fal or gve poor results because of changes n lghtng condtons and lane markng shape or wdth durng vehcle travel. The proposed DMB module allows the lane model to be bult onlne followng a multframe approach by processng a short sequence of mages typcally, from 10 to 20 frames, wth less than 1 s perod of tme, are suffcent. It kcks n at the start of the lane detecton operaton or when the system needs to update the model after falure. The only underlyng assumpton s that the vehcle s properly postoned on the road and that the lane markng s wthn the camera feld of vew. In each frame of the sequence, the DMB module looks for the best model followng a two-step approach. Frst, a Canny s edge detecton s performed. Second, a bounded Hough transform s appled. Thus, a set of marker canddates j can be defned n terms of ther parameters W j and j and compared wth the nomnal model to fnd the lne par that best satsfes constrants 5 and 6 for the gven mage. Fuzzy logc stll represents a feasble and effectve soluton to ths problem. Two nputs and one output are used n the fuzzy nference system, wth the relatve membershp functons shown n Fg. 10. The nputs Fg. 10 are the absolute dfference W j 0 between W j and the nomnal wdth Ŵ and the absolute dfference j 0 between j and ˆ. The output expresses the match confdence of the lane marker j. The rules for the fuzzy nference engne are collected n Table 4, expressng the dea that the more the strpe s smlar to the nomnal model, the greater s the confdence that t actually represents the lane marker. In order to combne robustly the results obtaned from the sngle scenes of the sequence, a so-defned cumulatve Hough matrx s proposed, whose conceptual scheme s shown n Fg. 11. For each mage, the computed model s added to the CHM, expressed n terms of coordnates of ts border lnes n the Hough space, namely, ponts P 1, P 2. After processng m frames, m lane marker canddates LM wth =1,2,...,m, wll be ncluded n the CHM, dstrbuted n cells, representng a small span of the Hough parameters. The larger the number of ponts fallng nto a gven cell, the hgher the lkelhood that one of the two boundares of the model belongs to ths cell. Note that the vehcle s poston and orentaton wth respect to the lane markng s assumed not to Fg. 9 Lane detecton for a sample mage where extraneous transversal whte road markngs are present: a output of the FED module and b ndcaton of the lane marker canddates / Vol. 133, MARCH 2011 Transactons of the ASME

7 Fg. 10 DMB: nput and output membershp functons change sgnfcantly durng the model buldng stage. Eventually, the two cells wth the largest number of ponts are selected cells denoted by a dashed rectangle n Fg. 11, and the last ponts added to these cells denoted wth P 1 K and P 2 K n Fg. 11, are chosen as the best-updated estmate of the model. Once the lane marker has been dentfed, ts geometrcal propertes are known n both the mage plane and the real world. In addton, the appearance propertes of the marker,.e., the average ntensty of the pxels bounded by the two lnes, can also be determned. All the estmated propertes of the lane marker are passed on to the FLane trackng system that can start ts trackng task. Representatve results obtaned from the DMB module are shown n Fg. 12 for a sample set of ten frames, acqured n a 0.5 Rule number Table 4 Fuzzy logc rules used by the DMB W j 0 Input j 0 Output Match confdence 1 Small Small Hgh 2 Small Large Medum 3 Large Small Medum 4 Large Large Low Fg. 11 Conceptual scheme of the CHM s wndow, wth the vehcle movng at about 40 km/h Specfcally, Fgs. 12 a and 12 b show frames #1 and #5 of the sequence, wth the overlad selected lane marker and the assgned confdence level. Fgure 12 c llustrates the CHM for the test sequence. The two cells wth the largest number of ponts orgnate two peaks n the CHM, correspondng to the lane marker borders depcted n Fg. 12 d. Note that, n ths example, the lane marker was properly dentfed n every frame of the sequence, and the border lnes have ther correspondng Hough ponts concentrated n only two cells of the CHM. However, t may occur that ether the marker s not detected or msdentfcatons arse n one or more frames due to bad llumnaton condtons, wde occlusons, presence of multple whte strpes, or even absence of the marker. In such a case, the use of multple frames helps to keep a robust estmaton of the model, as demonstrated by the sequence shown n Fg. 13. Fnally, one should note that the relablty of the DMB output can be assessed by evaluatng the score assgned to the lane model, and the process can be possbly repeated untl a suffcently hgh-confdence model s acheved. 3 Expermental Results In ths secton, we present a comprehensve set of experments to valdate our approach. The FLane system was tested n the feld on a commercal automoble, as shown n Fg. 14. A C compled mplementaton of the algorthm processed mages n real tme at 20 Hz on a 1.86 GHz Pentum III-M laptop. The executable verson of the code requred 120 KB of memory for the program, wth an addtonal 225 KB of memory for executon. These low requrements suggest that the algorthm s sutable for on-board mplementaton wth lmted computatonal resources. Addtonally, a cost-effectve webcam, mounted sdeways, was used for mage acquston to demonstrate the effectveness of the algorthm wth poor hardware resource. The webcam was calbrated usng the MATLAB camera calbraton toolbox 24. Data were collected from portons of four knds of roads at dfferent tmes of the day. Detals are gven n Table 5. Sequence A refers to urban road wth sold or dotted lane markngs and heavy dsturbances due to curbs, manhole covers, etc. Typcal freeway condtons wth sold or dotted road markngs and complex shadowng Journal of Dynamc Systems, Measurement, and Control MARCH 2011, Vol. 133 /

8 Fg. 12 Result of the DMB module appled to a sample sequence: a frst frame of the sequence, b ffth frame of the sequence, c representaton of the CHM, and d selected lane marker model. Images a and b report ndcaton of the confdence assgned to the selected lane marker. Fg. 13 Example of robust marker model buldng: a - l consecutve frames used for model buldng; m marker model obtaned as output of the DMB module. Although the absence of the man lane marker n d or the presence of multple lnes n e, f and results n msdentfcatons, the DMB module retans a correct model / Vol. 133, MARCH 2011 Transactons of the ASME

9 Fg. 14 The test bed used for expermental valdaton of the FLane system Table 5 Set of sequences showng the envronmental varablty caused by road markngs and surfaces and lghtng Set Road Road markng Day tme A Urban and rural Sold or dotted lnes Occlusons and dsturbances Low contrast between road texture and lne Noon B Hghway Sold or dotted lnes Noon C Hghway Sold or dotted lnes Dusk D Hghway Sold lnes Nght Table 6 Performance of the FLane system under varous lghtng and road condtons Set Frames False postves % False negatves % Msd. % A 10, B 15, C 11, D 7, Fg. 16 Example of false negatve due to poor mage segmentaton: a black lnes: lane canddates; and b output of the FED module. due to overpasses and other cars and changes n road surface materal are shown n sequences B, C, and D at dfferent tmes of the day: noon, dusk, and nght, respectvely. The FLane system was tested over a total of 44,850 frames approxmately 35 km of total travel dstance, showng the results summarzed n Table 6. The percentage of false postves, false negatves, and msdentfcatons s shown for each mage set. False postves occur when a lane marker s recognzed when actually there s no lane markng. Ths s due to spurous objects n the scene, whch somehow match the lane model. As an example, Fg. 15 shows a scene where the FLane system erroneously detected a lane marker, msled by the grds of a manhole cover. In our tests, the percentage of false postves was always less than 3%. Conversely, false negatves arse when the lane marker s present n the mage but the system s not able to detect t at all, and t does not return any nformaton. The percentage of false negatves was less than 6% and due manly to partally-deleted road markng, poor mage segmentaton, and camera calbraton errors, as shown by the example of Fg. 16, where the FLane system faled. However, we should emphasze that ths type of error may be greatly mtgated by adoptng a more sophstcated hardware set. Fnally, msdentfcatons refer to cases n whch a lane marker s present n the mage but the system fals n recognzng t properly and returns wrong nformaton. In all tests, msdentfcatons were less than 3%. As expected, set A presented error rates greater than sets B, C, and D. Table 7 shows some typcal results for sample mages extracted from each one of the sequences nvestgated. The ntal model of the lane marker was constructed onlne wth our DMB approach employng a set of ten frames. The ntal models are also shown n the frst column of Table 7 wth overlad the confdence level. Fnally, Table 8 collects the output of the FLane system for partcularly challengng stuatons. Specfcally, the mages extracted from sequence A refer to two scenaros characterzed by hgh level of nose, low contrast between road texture and lane marker, and sudden lghtng varatons. Although the applcaton of the FED module resulted n the detecton of multple spurous lnes denoted by black lnes due to poor mage segmentaton, the FLR module correctly dentfed the lane marker grey lnes or red lnes n the onlne verson of the paper. The mages from set D also confrm the effectveness of the proposed approach n very poor lghtng condtons and n presence of multple reflectons and shadows. In concluson, the FLane system proved effectve n feld testng provdng a fast measurement update every few meters of Fg. 15 Example of false postve due the presence of a manhole cover: a grey lnes red lnes n the onlne verson of the paper : erroneous lane marker estmated by the FLane system and black lnes: lane canddates; and b output of the FED module Journal of Dynamc Systems, Measurement, and Control MARCH 2011, Vol. 133 /

10 Table 7 Typcal results obtaned from the FLane system for dfferent envronmental condtons, as descrbed n Table 5. Please refer to Secs and for more detals on the mage processng. SET DMB FLane Fuzzy Edge Detecton A B C D travel dstance e.g., 1.4 m at a speed of 100 km/h that enforced the small varaton assumpton adopted for the lane model. 4 Conclusons In ths paper, we presented a method for detectng and trackng lateral lane markngs n real tme and n a hghly dynamc envronment, referred to as fuzzy logc lane trackng system. The FLane system uses Hough transform n conjuncton wth fuzzy reasonng to provde hgh flexblty and ablty to adapt to dfferent roads and envronmental condtons. Expermental results, obtaned wth our system ntegrated wth a commercal automoble, and usng a cost-effectve webcam showed the feasblty of our approach and ts robustness to varatons n lghtng and road condtons, wth a worst-case of less than 6% of faled observatons n urban roads. It was shown that the FLane module could be effectvely employed n the development of autonomous vehcles and drver assstance systems. Appendx If we ntroduce an average relatve ntensty varaton between the ROI of two consecutve frames and 1 as I ROI = I 1 ROI I ROI 1 I ROI 100 A1 where I ROI s the average ntensty value wthn the ROI of frame and I 1 ROI s the average ntensty value wthn the ROI of frame 1. Then, we can defne I mn as I mn = I 1 mn + k I ROI A2 wth k beng a weghtng factor, whch depends on the value of I ROI. For a fner gradaton, we also express ths relatonshp wth fuzzy logc. The nput to the fuzzy nference system s the value of and the output s the gan k rangng from zero to one. I ROI / Vol. 133, MARCH 2011 Transactons of the ASME

11 Table 8 Results obtaned from the FLane system n challengng stuatons encountered n set A urban and rural road and D nght-tme acquston. Canddate lnes are shown n black; grey red lnes ndcate the selected lane marker. References 1 McCall, J. C., and Trved, M. M., 2006, Vdeo-Based Lane Estmaton and Trackng for Drver Assstance: Survey, System, and Evaluaton, IEEE Trans. Intell. Transp. Syst., 7 1, pp Godthelp, H., Mlgram, P., and Blaauw, G. J., 1984, The Development of a Tme Related Measure to Descrbe Drvng Strategy, Hum. Factors, 26, pp McCall, J., and Trved, M. M., 2004, Vsual Context Capture and Analyss for Drver Attenton Montorng, Proceedngs of the IEEE Conference on Intellgent Transportaton Systems, Washngton, DC, pp Taylor, C., Košecká, J., Blas, R., and Malk, J., 1999, A Comparatve Study of Vson-Based Lateral Control Strateges for Autonomous Hghway Drvng, Int. J. Robot. Res., 18 5, pp Hough, P., 1962, Methods and Means for Recognzng Complex Patterns, U.S. Patent No McDonald, J., 2001, Detectng and Trackng Road Markngs Usng the Hough Transform, Proceedngs of the Conference of the Irsh Machne Vson and Image Processng, Maynooth, Ireland, pp Pomerleau, D., 1995, RALPH: Rapdly Adaptng Lateral Poston Handler, Proceedngs of the IEEE Symposum on Intellgent Vehcles, Detrot, MI, pp Furusho, H., Shrato, R., and Shmakage, M. A., 2002, Lane Recognton Method Usng the Tangent Vectors of Whte Lane Markers, Proceedngs of the Internatonal Symposum on Advanced Vehcle Control, pp Gehrg, S., Gern, A., Henrch, S., and Woltermann, B., 2002, Lane Recognton on poorly Structured Roads The Bot Dot Problem n Calforna, Proceedngs of the Conference on Intellgent Transportaton Systems, pp Southall, J. B., and Taylor, C. J., 2001, Stochastc Road Shape Estmaton, Proceedngs of the Conference on Computer Vson, Vancouver, Canada, pp Km, Z. W., 2008, Robust Lane Detecton and Trackng n Challengng Scenaros, IEEE Trans. Intell. Transp. Syst., 9 1, pp Danescu, R., and Nedevsch, S., 2009, Probablstc Lane Trackng n Dffcult Road Scenaros Usng Stereovson, IEEE Trans. Intell. Transp. Syst., 10 2, pp Bertozz, M., and Brogg, A., 1998, GOLD: A Parallel Real-Tme Stereo Vson System for Generc Obstacle and Lane Detecton, IEEE Trans. Image Process., 7 1, pp McCall, J., and Trved, M., 2005, Performance Evaluaton of a Vson Based Lane Tracker Desgned for Drver Assstance Systems, Proceedngs of the IEEE Symposum on Internatonal Vehcles, pp Kreucher, C., and Lakshmanan, S., 1999, LANA: A Lane Extracton Algorthm That Uses Frequency Doman Features, IEEE Trans. Rob. Autom., 15 2, pp Pomerleau, D.; Jochem, T., 1996, Rapdly Adaptng Machne Vson for Automated Vehcle Steerng, IEEE Expert, 11 2, pp Mendel, J. M., 1995, Fuzzy Logc Systems for Engneerng: A Tutoral, Proc. IEEE, 83 3, pp Sler, W., and Buckley, J. J., 2005, Fuzzy Expert and Fuzzy Reasonng, Wley, New Jersey. 19 Russo, F., and Rampon, G., 1994, Edge Extracton by FIRE Operators, Proceedngs of the IEEE Internatonal Conference on Fuzzy Systems, pp We L, G. T., and Wang, Y., 1997, Recognzng Whte Lne Markngs for Vson-Guded Vehcle Navgaton by Fuzzy Reasonng, Pattern Recogn. Lett., 18 8, pp Rena, G., and Yoshda, K., 2008, Estmaton for Lunar and Planetary Robots, Int. J. Intell. Contr. Syst., 13 1, pp Canny, J., 1986, A Computatonal Approach to Edge Detecton, IEEE Trans. Pattern Anal. Mach. Intell., PAMI-8 6, pp Mlella, A., Venturno, L., Gramegna, T., and Ccrell, G., 2004, Robot Localzaton and Path Followng Based on Detecton of Vsual Structures, Proceedngs of the IEEE Internatonal Conference on Mechatroncs and Machne Vson n Practce, pp Bouguet, J., Camera Calbraton Toolbox for MATLAB. 2004, avalable onlne at Journal of Dynamc Systems, Measurement, and Control MARCH 2011, Vol. 133 /