Sequential Monte-Carlo Based Road Region Segmentation Algorithm with Uniform Spatial Sampling

Transcription

1 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) [DOI: /psjtcva.8.1] Regular Paper Sequental Monte-Carlo Based Road Regon Segmentaton Algorthm wth Unform Spatal Samplng Zdeněk Procházka 1,a) Receved: March 27, 2015, Accepted: November 2, 2015 Abstract: Vson based road recognton and trackng are crucal tasks n a feld of autonomous drvng. Road recognton methods based on shape analyss of road regon have the potental to overcome the lmtatons of tradtonal boundary based approaches, but a robust method for road regon segmentaton s the challengng ssue. In our work, we treat the problem of road regon segmentaton as a classfcaton task, where road pxels are classfed by statstcal decson rule based on the probablty densty functon (pdf) of road features. Ths paper presents a new algorthm for the estmaton of the pdf, based on sequental Monte-Carlo (SMC) method. The proposed algorthm s evaluated on data sets of three dfferent types of mages, and the results of evaluaton show the effectveness of the proposed method. Keywords: road segmentaton, sequental Monte Carlo, vehcle mounted camera, road trackng 1. Introducton In recent years, many researchers and car manufacturers are strongly nterested n the development of autonomously drvng cars. Autonomous drvng systems rely on varous sources of sensory nformaton, ncludng radar, ldar, camera etc., to handle stuatons around the vehcle. Snce a human drver reles manly on vsual nformaton, the transport nfrastructure s adapted to maxmally ft human vson. For ths reason the vson sensor, lke vehcle mounted camera, s a very mportant source of sensory nformaton. To explot such nformaton, varous computer vson technques are beng ntensvely studed n the feld of autonomous drvng. Road recognton and trackng are crucal tasks for autonomous vehcles. The most common approach to road recognton s based on the detecton of lane markng or road boundares, and fttng some geometrc model to the lane markng or boundary ponts. Very good surveys can be found n Refs. [1], [2], [3]. The geometrc model plays a very mportant role n boundary based technques, because t mposes constrants to acheve robust results. However on the other hand, t also poses very strong lmtatons on the method, because the geometrc model s usually based on the assumpton that roads or lane boundares can be modeled by a par of parallel curves. Although such assumptons hold for roads wthout sgnfcant changes (e.g., hghways), t s volated when mergng or splttng of traffc lanes appears, and for crossngs or changes n road wdth caused by parkng cars. Hllel et al. mentons such lmtatons n ther survey report [4] as a challenge for further development. Furthermore, there are also ssues surroundng how to handle stuatons where lane markng s occluded by obstacles, or how to handle roads wthout markng 1 Natonal Insttute of Technology, Ota College, Ota , Japan a) zdenek p@ota-ct.ac.jp and unstructured roads respectvely. An approach opposte to the boundary based one s to perform segmentaton of road regons, and analyze ther shapes and topology to understand the road. Ths approach has a potental to overcome lmtatons of boundary based methods, however although study on a regon based approach has started at the almost same tme lke the boundary one (see e.g., Crsman et al. [5]), the majorty of research was devoted to development of boundary based technques, and regon based methods are not well developed yet. The man challenge of the regon based approach s a lack of robust method for road regon segmentaton, and development of such method s stll an open queston. The road can take varous shapes wth dfferent topology and, the goal of the road regon segmentaton s to dentfy multple, possbly dsconnected, parts of road surface wth smlar features. Ths means that rather than to perform general partton of nput mage frame followed by dentfyng of road parts among parttoned segments, we can formulate the segmentaton of road regon as a two class decson problem, where we want to decde whether a pxel n an nput mage frame s part of road or not. The queston s now how to buld such a classfer. Some approaches from feld of machne learnng based on neural networks [6] or SVM [7], [8] have been reported. However as s also mentoned n Ref. [9], the dffculty wth such methods les n the fact that the classfer has to be very adaptve to handle current vsual appearances of the road surface, and the learnng of the classfer cannot be smple performed n off-lne manner. Another approach to such a classfcaton task s based on an dea of statstcal decson. Ths means that statstcal dstrbuton of road features s modeled by parametrc or non-parametrc probablty dstrbuton functon (pdf), and the pdf s used to decde whether a gven mage pxel s a part of road or not. Methods based on smple Gaussan color model [5], Gaussan mxture model [10], hstogram of color components [11], hstogram of lc 2016 Informaton Processng Socety of Japan 1

2 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) lumnaton nvarant features [12] have been proposed. Agan, snce the classfcaton method has to be adaptve to the current road appearance, we need to estmate pdf n an adaptve way. Furthermore, there s another ssue. To estmate pdf for a gven frame, we need to acqure samples of road regon features, however to acqure such samples we need to know where the road regon s. Ths clearly poses a chcken-and-egg problem whch has to be solved. As an approach to ths ssue, a road regon segmentaton method based on sequental Monte-Carlo (SMC) has been developed n the prevous work [13]. Wth ths method, the estmaton of the pdf of road features can be performed n an adaptve way, and t also ntroduces a systematc soluton to the above mentoned chcken-and-egg problem. However, ths orgnal method stll doesn t produce optmal results, especally for wde roads or roads wth multple traffc lnes. Ths paper ntroduces a new algorthm, whch sgnfcantly mproves the shortcomngs of the orgnal method. Snce ths paper s focused on mprovement of the estmaton method tself, t doesn t deeply dscuss whch knd of mage features s optmal for road regon segmentaton, although some drectons are remarked. However, t should be emphaszed that the proposed method can be very easly modfed to other mage features too. The paper s organzed n the followng way. In Secton 2, we frst brefly summarze the prevously developed SMC segmentaton algorthm to explan ts lmtatons, and next, we gve a detaled explanaton of the proposed algorthm. In Secton 3, we evaluate the proposed algorthm, and compare t to the orgnal one, as well as to the conventonal road regon segmentaton methods. Secton 4 contans several addtonal remarks and dscussons. Fnally, n Secton 5, we gve a concluson and show some drectons for future work. 2. SMC Based Road Regon Segmentaton Algorthm When a vehcle moves, the appearance of road n the mage sequence, captured by camera mounted on the vehcle, changes wth respect to ts shape, topology, placement, or vsual features lke color or texture. To buld-up a classfer for decson between road and non-road pxels n the captured mage, we have to handle such changes n an adaptve way. A method based on recursve estmaton of a statstcal dstrbuton of the road features has been already developed n the earler work [13]. Although ths method can potentally avod some ssues dscussed n the Secton 1, and t s qute robust to parameter tunng, t stll doesn t produce satsfactory results under certan crcumstances. The goal of ths paper s to develop a new algorthm wth hgher segmentaton ablty, whch also preserves the strong ponts of the orgnal SMC based algorthm. In ths secton we frst recaptulate the orgnal SMC based algorthm to explan ts lmtatons. After that we propose a new revsed algorthm. 2.1 Orgnal Segmentaton Algorthm Let f be a vector of road features. In ths work, a fve dmensonal feature vector f = (x,y,r,g,b) T s used, where x and y stand for pxel coordnates, and r, g, b stand for color components. By ncludng spatal coordnates nto the feature vector, we can estmate pdf wth respect to x, y, as well as to the mage features smultaneously, whch enable us to avod the chckenand-egg problem dscussed n Secton 1. Although the mage features used n the segmentaton algorthm are smply r, g, b components, the algorthm s generally not lmted to them, and t can be easly modfed to other type of mage features too. To smplfy further notaton, we defne also vectors x = (x,y) T and c = (r,g,b) T as shorthand for vector of coordnates and color components respectvely. Wth ths notaton, the feature vector can be also wrtten as f = (x T, c T ) T. Our goal s to estmate pdf p( f) of the features f n an adaptve way, and use the estmated pdf to classfy road and non-road pxels. Snce the road can take varous shapes wth dfferent topology, p( f) s expected to be a non-gaussan dstrbuton at least wth respect to the x. Ths suggests that we need an estmaton method whch can handle general types of dstrbutons. Snce the road usually looks very smlar between two successve frames, and overall vsual appearance changes relatvely slowly compared to frame rate, temporal contnuty can be nvolved nto the pdf estmaton, and t should be possble to model a vsual changes of road as a stochastc process. Takng ths nto account, we have focused to SMC (Sequental Monte-Carlo) method as a basc tool for estmaton of the p( f). The orgnal segmentaton algorthm was nspred by CONDEN- SATION algorthm [14]. The pdf s expressed n dscrete form as a set of weghted samples S, S = { ( f 1,w 1 ), ( f 2,w 2 ), ( f N,w N ) } (1) where f s a sample of the feature vector f and w s an mportance weght assgned to the sample. Estmaton s performed by repettve update of the S accordng to the algorthm shown n Fg. 1. The superscrpt k stands for dscrete tme here. The ν n Eq. (2) means a vector of Gaussan random numbers wth zero mean and varances σ 1, σ 5.Thep(z c ) n Eq. (3) s an observaton pdf defned by Eq. (5). Fg. 1 Orgnal algorthm. c 2016 Informaton Processng Socety of Japan 2

3 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) Fg. 2 Intalzaton of the algorthm. ( p(z c ) = exp 1 ) 2 (c z ) T Σ 1 m (c z ) (5) The symbol Σ m here means a dagonal matrx of varances, controllng wdth of exponental functon for each color component, and z means a color components observed at postons predcted by Eq. (2). Smply sad, the p(z c ) s a measure of smlarty between color components c predcted by Eq. (2), and color components observed at predcted postons. The calculated smlarty s then assgned as an mportance weght. The above descrbed algorthm estmates the pdf n a form of weghted samples. To dentfy road pxels, however, we have to evaluate pdf for an arbtrary pxel n the nput frame, and to acheve ths, we perform a smoothng of the estmated pdf n a way smlar to non-parametrc densty estmaton, usng kernel functon φ. Hence, the pdf p( f) for an arbtrary value of f s calculated as N p( f) = w φ( f, f ) (6) =1 φ( f, f ) = β exp ( 1 ) 2 ( f f ) T Σ 1 s ( f f ) where the symbol Σ s s a dagonal matrx of parameters, controllng the wndow wdth for each component of f, and the β s a normalzng constant. The road regon s then obtaned as the followng set of ponts R x { R x = x θ< p( f) max f p( f) } where θ stands for decson threshold. Let us show behavor of the descrbed algorthm. The algorthm s ntalzed wth help of smple heurstc that the bottom part of the mage should be road. Rectangular regon shown n Fg. 2 (a) s set n the frst frame of the mage sequence, and ntal samples are drawn unformly from the rectangular regon. By the sequental update, accordng to the algorthm from Fg. 1, samples spread over the mage, as s shown n Fg. 2 (b) (d). Segmentaton results for three types of mages are shown n Fg. 3. Fgure 3 (a) shows the result for a relatvely narrow road, and segmented road regon shown n rght column seems to be plausble for ths case. However, f we look at Fg. 3 (b), we can see that the segmented road regon lacks bottom left part of the road. The reason s that ths part of road s not covered by samples, and pxels are classfed as non-road ones when p( f)sevaluated. Fnally, Fg. 3 (c) shows a result for synthetc mage. Snce (7) Fg. 3 Example of segmentaton results by the orgnal algorthm. color of the road s almost homogeneous, the color components are optmal features for segmentaton, and road regon should be segmented very easly. However the result s not optmal due to the mproper dstrbuton of the samples. The results shown above mply that the dstrbuton of the samples n the orgnal algorthm tends to be nonunform n spatal doman, what s undesrable for the purpose of road segmentaton. Furthermore, snce the estmaton of the pdf of road features s treated as an stochastc process, certan nose has to be ntroduced nto the system model. By ths, the postons of samples change from frame to frame, and segmented parts of road covered only by a few of samples could change drastcally, although there are no sgnfcant changes n road appearance between successve frames. Ths leads to unsteady segmentaton results. The behavor descrbed above poses a very serous drawback to the orgnal segmentaton algorthm, and the algorthm has to be revsed to get a better performance. 2.2 Proposal of New Algorthm The drawback of the orgnal algorthm s caused by mproper dstrbuton of samples durng the estmaton process. To obtan proper segmentaton result, the samples should be spread unformly over the road regon. The algorthm from Fg. 1 generates samples accordng to dstrbuton of weghts, therefore the samples wth hgh weghts are chosen more lkely then the samples wth low weghts. However such samplng cannot guarantee that the samples wll be spread unformly over the road regon. In the orgnal algorthm, the only way to spread samples more wdely over the road regon s to ncrease the amount of the transton nose added to spatal components, however, on the other hand, ths causes more sgnfcant fluctuatons of the segmented regon. The concluson s that the drawback of the orgnal algorthm cannot be solved by smple parameter tunng, and a fundamentally new soluton s requred. The dea to solve the ssue s based on the prncple of mportance samplng [15]. Ths framework supposes that we can easly draw samples from some probablty densty functon q( f (k), z (k) ), whch s called mportance densty. Update of weghts for samples generated from the mportance densty s c 2016 Informaton Processng Socety of Japan 3

4 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) done accordng to the followng formula. w (k) w (k 1) p(z (k) f (k) )p( f (k) q( f (k), z (k) ) ) Although the prncple of mportance samplng provdes sound theoretcal framework, the specfc forms of p(z (k) f (k) ), p( f (k) )andq( f (k), z (k) ) are not obvous, and strongly depend on the type of partcular applcaton. In our case, q( f (k), z (k) ) was chosen to be a unform dstrbuton wth respect to spatal components x, wth x R (k 1) x, where R (k 1) x means a road regon at the prevous tme step. To evaluate p( f (k) ), we need to now correspondences between f (k) and f (k 1). However, snce the samples at k (new samples) are pcked up unformly from R (k 1) x, there are no drect correspondences wth samples at k 1 (old samples), and we have to establsh them. The dea here s to assgn old samples n a way descrbed n Fg. 4 for sngle varate case. Let the x 1,, x 4 stand for old samples, and let the ntervals hghlghted by lght gray color are regons of x axs segmented at prevous step. To generate a new sample set, we frst draw unformly samples x, = a, b, c, d (shown by ) from the lght gray regons. Usng weghts of the old samples, we can evaluate weghted kernel functon w j φ(x), j = 1, 2, 3, 4, and dvde lght gray regons to the sub-regons X j, where { } X j = x w jφ(x) = max (w mφ(x)) ; x R (k 1) x. (9) m=1,...,4 A label l() of the old sample assgned to the -th new sample s then decded as follows. (8) crcles) drawn unformly from the segmented road regon. Color parts of samples are not defned yet. Fgure 5 (c) shows dvson of the road regon to sub-regons, and fnally Fg. 5 (d) shows the resultng assgnment between old and new samples. By the above descrbed procedure, we can generate spatal part of samples for the new sample set, however to buld-up set of complete samples we have to generate color part for each new sample too. To do t, we apply transton equaton to the color part of the assgned old sample. c (k) = c (k 1) l() + c ν (11) Here the c ν means a color part of ν from Eq. (2), and the subscrpt l() means a label of the assgned old sample. After the new sample set f (k) has been completed, we can evaluate transton pdf n weght update equaton as p ( ) ( ) f (k) l() = φ f (k), f (k 1) l() (12) where φ has the same form as n Eq. (6), and l() s a label of sample assgned to the -th one. The last component requred for the weght update calculaton of Eq. (8) s the observaton pdf p(z (k) f (k) ), and t has the same l() = j f x X j (10) The resultng assgnment for the case shown n Fg. 4 s (x a, x 1 ), (x b, x 3 ), (x c, x 2 ), (x d, x 4 ). Extenson to our case of f s straghtforward. The man dfference s that the evaluaton of the kernel functon φ and dvson to sub-regons s performed only over x,.e. over two-dmensonal spatal part of the feature vector f. The kernel functon φ was chosen to have the same functonal form as s shown by Eq. (6), snce t allows us to reuse values from pdf evaluaton already calculated at the prevous tme step. The process descrbed above s schematcally shown n Fg. 5. Pcture n Fg. 5 (a) shows a current sample set, and a road regon segmented at the prevous step. Weghts of the samples are depcted by sze of crcle. Fgure 5 (b) shows new samples (whte Fg. 5 Assgnment n real segmentaton algorthm. Fg. 4 Assgnment between subsequent samples. Fg. 6 Proposed algorthm. c 2016 Informaton Processng Socety of Japan 4

5 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) Fg. 7 Segmentaton results for proposed algorthm. form here as Eq. (5). The whole proposed algorthm s summarzed n Fg. 6. Snce we draw samples accordng to the unform dstrbuton, the value of mportance densty q s same for all samples, and t vanshes at the normalzng step 9. Therefore, the value of q s omtted n the weght update Eq. (14) of the step 8. At the end of ths secton, we show n Fg. 7 segmentaton results obtaned by the proposed algorthm for the mages from Fg. 3. The ntal regon s the same as has been shown n Fg. 2. We can see that the dstrbuton of samples over the road regon s sgnfcantly mproved, and the results of segmentaton are closer to the true road regon. More detaled evaluaton of the proposed algorthm s descrbed n the next secton. 3. Evaluaton of the Proposed Algorthm 3.1 Image Data Sets The evaluaton of the proposed algorthm has been performed on the followng sets of mage data. A: Sequences of Stll Images Approxmately 200 mages, capturng frontal roads vewed from a car, were randomly collected. Half of them are structured roads, second half are unstructured and rural roads. For every mage, a sequental mage was created repeatng the same stll mage. The resultng sequental mages show real roads, however snce there are no temporal changes between successve frames, these data can be used to assess temporal stablty of the resultng segmentaton. B: Synthetc Image Sequence Sequence of rendered CG mages, already shown n Fg. 3, was created. The sequence smulates a vehcle movng by 60 km/h on road wth 4 curves (R = 150 m, 100 m, 80 m, 50 m). Steerng of the vehcle s smulated by changes n lateral poston and ptch angle. Snce the sequence s composed of road mages wth almost homogeneous colour, the perfect segmentaton of road regon can be done very easly, and ths sequence can be used to assess qualty of segmentaton. Fg. 8 Real mage sequences. C: Real Image Sequences Real mage sequences, already mentoned n report [16], wth representatve frames shown n Fg. 8, were prepared. Snce the sequences are real data, they can be used to assess overall ablty of the segmentaton algorthm. 3.2 Reference Segmentaton Methods Although ths paper s manly nterested n solvng drawbacks of the orgnal SMC based algorthm, we have decded to mplement two conventonal segmentaton methods as a reference for an assessment of the SMC based approach. In the frst reference method, pdf of mage features (color components) s modeled by Gaussan mxture model, and EM algorthm s used to estmate mxture parameters. More specfcally, after the ntalzaton, color components of pxels segmented as a road regon at the prevous tme step are used to estmate mxture parameters, and estmated pdf s used to classfy road and non-road pxels n the current frame. Segmentaton s done n the smlar way as Eq. (7). Ths method s referred as REF-GM n the followng. Hyper-parameters to be optmzed n the REF-GM are number of mxture components and threshold value θ. The second method s nspred by reports of Alvarez et al. [12] and Oh et al. [11], and t uses a normalzed hstogram of mage features (color components) to model pdf. Smlarly as the REF-GM, after the ntalzaton, color components of pxels segmented as a road regon at the prevous tme step are used to form a normalzed hstogram and the hstogram s used to classfy road and non-road pxels n the current frame accordng to Eq. (7). Ths method s referred as REF-HIST n the followng, and hyper-parameters to be optmzed are sze of hstogram bn and threshold value θ. 3.3 Evaluaton Method To evaluate the segmentaton ablty of proposed SMC based, orgnal SMC based, REF-GM, REF-HIST algorthms respecc 2016 Informaton Processng Socety of Japan 5

6 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) tvely, we have compared the segmentaton results by each algorthm wth ground truth. The ground truths, means deally segmented road regons, for the data sets from Secton 3.1 were created manually for the data A and C. For the data B, the ground truth was generated automatcally durng CG renderng. As a quanttatve measure of how well a segmentaton result match the ground truth, we have adopted Jaccard ndex J, defned by Eq. (13). J = R s R g R s R g (13) Here, R s s a set of road pxels obtaned by segmentaton algorthm, and R g s a set of road pxels of correspondng ground truth. Symbol means a set sze (number of pxels) here. Jaccard ndex J takes value 1 f both sets R s and R g are dentcal, and 0 f one of sets s a complement of the other one. Hyper-parameters Σ m, Σ s, ν, θ for SMC based algorthms, as well as the hyper-parameters for REF-GM and REF-HIST were optmzed, usng ground truth and Jaccard ndex J. To assess robustness of each algorthm to parameter settng, once optmzed hyper-parameter values were used for all data sets A, B, C, and no addtonal tunng was done for specfc data set. It should be remarked here that the optmzaton for REF-GM was extremely dffcult, and best results were obtaned for the model wth only sngle mxture component (same as smple Gaussan model). But REF-GM stll faled for some cases, as wll be shown later. Rectangular regon wth wdth 1/8 of frame wdth and heght 1/8 of frame heght, centered at pont located n 50% of frame wdth and 10% of frame heght, shown n Fg. 2 (a), was used as an ntal regon for all tested algorthms. 3.4 Evaluaton Results for Data Set A Both SMC based algorthms as well as REF-GM and REF- HIST were appled to data set A, and Jaccard ndex J was calculated frame by frame for each sequence (transent frames at the begnnng of the each sequence were skpped). After that, temporal averages μ J and standard devatons σ J of Jaccard ndex were calculated separately for each sequence, and resultng values of μ J and σ J were used to form hstograms. Resultng hstograms for the μ J are shown n Fg. 9. Ideal value of μ J s μ J = 1, where values closer to 1 ndcate segmentaton results closer to the ground truth. As can be seen from Fg. 9, peak of the hstogram for the proposed SMC algorthm s closer to 1, whch means that for majorty of the data A, the proposed algorthm produces segmentaton results more closer to the ground truths then other tested algorthms. The results for REF-GM and REF-HIST algorthms are surprsngly wrong. The values of average of Jaccard ndex are dstrbuted n a wde range and t s dffcult to fnd some tendency n the resultng hstograms. Ths result ndcates that REF-GM and REF-HIST are not robust to the settng of hyper-parameters. The hstograms for σ J are shown n Fg. 10. Snce algorthms REF-GM and REF-HIST are completely determnstc, and sequences of data set A are created by repeatng of the same frame, after skppng of transent frames at the begnnng each sequence, both algorthms REF-GM and REF-HIST produces segmentaton Fg. 9 Hstogram of means of Jaccard Index for data set A. Fg. 10 Hstogram of means of Jaccard Index for data set A. results wth no temporal changes. Therefore, the hstograms of σ J for these two algorthms have no meanng, and only the hstograms for SMC based algorthm are shown here. The deal value of σ J s σ J = 0, whch means completely steady result wth no fluctuatons. As we can see from Fg. 10, peak of the hstogram for proposed SMC based algorthm s closer to the deal value 0, whch means that the fluctuatons of segmentaton results are effectvely suppressed n the proposed algorthm. 3.5 Evaluaton for Data Set B Both SMC based algorthms as well as REF-GM and REF- HIST were appled to data set B, and Jaccard ndex J was calculated frame by frame, whle transent frames at begnnng of the sequence were skpped. The temporal sequences of the calculated values J for each tested algorthm were formed nto graphs shown n Fg. 11. As can be seen, the result for the proposed SMC based algorthm Fg. 11 (a) has less amount of fluctuatons and hgher values of J than the result for the orgnal SMC based algorthm shown n Fg. 11 (b). The result for REF-GM n Fg. 11 (c) shows that the trackng fals for several tmes around frames 500 and Snce a poston of horzon moves n up and down drectons due to the smc 2016 Informaton Processng Socety of Japan 6

7 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) Fg. 11 Sequences of Jaccard Index for for data set B. ulated changes of ptch angle n ths parts of the mage sequence, outlers appear n the data fed nto EM algorthm, whch causes EM algorthm fal to estmate proper pdf. Fnally, a very poor result for REF-HIST n Fg. 11 (d), because REF-HIST seems to be very senstve to settng of hyperparameters, whch are not optmal for ths partcular sequence. 3.6 Evaluaton for Data Set C Both SMC based algorthms as well as REF-GM and REF- HIST were appled to data set C, and Jaccard ndex J was calculated frame by frame for each sequence, whle transent frames at the begnnng of the sequence were skpped. The temporal sequence of the calculated values J were formed nto graphs shown n Fg. 12. Smlar tendency as for data sets A and B can be observed. The proposed SMC based algorthm gves hgher values of J wth less fluctuatons then other tested algorthms. The results for orgnal SMC based algorthm shows relatvely hgh values of J for sequences No.1-3, whle fluctuatons of J can be observed. The results of orgnal SMC based algorthm for sequences No.4-6 shows lower values of J, whch s due to multple traffc lnes and the ssue dscussed n Secton 2.1. Both reference algorthms REF-GM and REF-HIST gve sgnfcantly worse results then SMC based algorthms, only results for sequences No.4 and No.6 are comparable to orgnal SMC based algorthm. For sequence No.3, the REF-GM fals n the second half of the sequence, because t s dstracted by outlers, and other regon than the road s segmented as result. REF-HIST shows qute unstable behavor between frames 400 and 500 for ths sequence. Although the results of the proposed SMC based algorthm are sgnfcantly better than for other tested algorthms, decrease of J for several tmes between frames 50 and 200 can be observed for sequence No.5. Let us now look more closely on segmentaton results for such frames. Fgure 13 shows result for frame 125 of the sequence No.5. Ths s a stuaton mmedately after another car has passed an opposte traffc lne. Snce the samples dsappear from road parts occluded by a passng car, a few teratons of the algorthm are needed to spread samples over the area exposed after the car has been passed. Ths cause temporary decrease of J Fg. 12 Sequences of Jaccard Index for data set C. due to the dfference between result and ground truth. 4. Addtonal Remarks Ths secton contans some addtonal remarks and dscussons related to the proposed algorthm. Frst of all, let us dscuss the computatonal complexty of the proposed algorthm. Both of the predcton and weght update steps can be performed wth computatonal complexty O(DN), where D s a dmenson of feature space and N s number of samples. We suppose here that the results computed at the prevous tme step are properly reused. The hghest computatonal effort s spent for classfcaton of road pxels, descrbed by Eq. (6) and c 2016 Informaton Processng Socety of Japan 7

8 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) Fg. 15 Segmentaton result for mage wth shadows. Fg. 16 Segmentaton result for ntrnsc mage. Fg. 13 Frame 125 of sequence No.5. Fg. 17 Example of challengng scenaro. Fg. 14 Road wth precedng vehcle. Eq. (7). If we lmt spatal support of the kernel functon φ to sze M M pxels, the road pxels are obtaned wth a complexty O(DNM 2 ). Here, only pxels wthn spatal support of φ for each sample f are evaluated. In the evaluaton experments descrbed above, the sample set S from Eq. (1) comprsed N = 1500 samples, and sze of spatal support was set to pxels after the parameter tunng. Wth such settng, our mplementaton of the algorthm n C++ was able to run at frame rate about 3 fps (Intel dual core 3.2 GHz processor, 2 GB of memory), on mages wth resoluton pxels. However there s stll room for mprovement, e.g. usng wndow functons that can be evaluated wth lower costs than exponental ones, or replacng the exact computaton of wndow functons by LUT strategy. Knowledge on effcent calculaton of non-parametrc densty estmators may be also helpful. The algorthm s also well suted for parallel processng. Next, let us show an example of behavor of the algorthm under presence of obstacles. Fgure 14 shows a frame from mage sequence where the vehcle approaches a precedng vehcle n the same traffc lne. The dstrbuton of samples changes automatcally, and only vsble parts of road surface appear n the segmentaton result, what s an expected behavor from the pont of vew of segmentaton algorthm. It should be however mentoned that the segmentaton result s only a knd of sensory nformaton, and complete understandng of obstacles requres a hgher semantc level of processng. We would also lke to dscuss some drectons for mprovements of the proposed algorthm as well as ts lmtatons. Snce the proposed algorthm uses color components as mage features, the algorthm n presented form s not able to handle shadows. Ths stuaton s llustrated n Fg. 15. The part of road under sunshne s segmented properly, although ts color s not completely homogeneous. However the color of the shadowed part dffers too drastcally, and the algorthm s not able to handle ths part properly. To handle shadows, the color compo- nents have to be replaced by features more robust to llumnaton, or combned wth some knd of pror knowledge. Fnlayson et al. [17] shown that under assumptons of black body radator, colors captured by camera wth narrow band sensors under dfferent llumnatons form straght lnes n log-chromatcty space. Illumnaton nvarant features than can be obtaned by projecton of data ponts n log-chromatcty space to drecton perpendcular to these lnes. The mage formed by such features s called ntrnsc mage. In Fg. 16 (a), the mage from Fg. 15 (a) has been replaced by the correspondng ntrnsc mage, and color components n the proposed algorthm has been replaced by values of ntrnsc mage. Correspondng dstrbuton of samples and segmentaton result are shown n Fg. 16 (b) and Fg. 16 (c) respectvely. As can be seen from the fgure, segmentaton result s sgnfcantly mproved, and the methodology of ntrnsc mage shows an mportant drecton to deal wth ssue of shadows. However more detaled study on ntrnsc mages have to be carred out. Snce some nformaton s lost due to formng of ntrnsc mage, nonshadow regons from the orgnal mage may appear smlar n the ntrnsc mage too, whch can deterorate separablty of such regons. Several scenaros, however, are more challengng then the ssue of shadows. One of them s a patched surface wth dfferent types of pavement, lke the one shown n Fg. 17 (a). Correspondng dstrbuton of samples and segmentaton result are shown n Fg. 17 (b) and Fg. 17 (c) respectvely. As can be seen from the result, only the rght part of the road s segmented out properly. Segmentaton result of the left dark part s naccurate due to texture of the surface, and whte part s mssng from segmentaton result, snce ts color dffers too drastcally from other parts of road. For such mages, the assumpton that the pxels formng the same road surface have also smlar mage features s strongly volated, and such stuatons are very challengng for any knd of mage segmentaton method. Other challengng scenaros are also nght condtons, roads captured aganst the sun, rany condtons, tunnels etc. c 2016 Informaton Processng Socety of Japan 8

9 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) Fg. 18 Results for graph-cuts based general segmentaton method. Let us dscuss some dfferences of the proposed approach wth general mage segmentaton methods. The goal of general segmentaton methods s to partton nput mages nto multple nonoverlappng segments n a such way, that each segment consst of pxels wth smlar mage features. A popular segmentaton technque s based on energy mnmzaton va graph cuts, and ts effcent mplementaton s reported e.g., by Felzenszwalb et al. [18]. Some results by ths segmentaton method for our case are shown n Fg. 18. The result n Fg. 18 (a) seems good for road segments, but we can see that the wall and trees at rght sde of the road form the same segment. In the Fg. 18 (b) the road and surroundng grass forms the same segment, because color features of gray road and green grass seem to be close n the feature space. Although the segmentaton technques based on energy mnmzaton are a powerful tool n general, t s not trval to control the energy functon to get desrable results for a specfc task. Le et al. [9] have used the above mentoned segmentaton method [18] for detecton of the pedestran lne n an unstructured envronment. After parttonng of the nput mage, the connected mage segments are merged nto a sngle lne by a greedy search algorthm, where lane templates are used as pror knowledge. Although Le s method shows promsng results for a set of statc mages wth sngle pedestran lne, t doesn t solve another ssue wth general segmentaton methods that s how to dentfy multple traffc lnes n the set of mage segments. If we apply the general segmentaton method to an mage of road wth multple traffc lnes, separated by panted center lne (lke the one n Fg. 18 (a)), the road surface wll be dvded nto multple segments, whch are not mutually connected. Therefore, even f we have a perfect segmentaton result, we need another knd of processng to dentfy all parts of road surface n the total set of mage segments, whch s not a trval task. Besdes ths, almost no general segmentaton method takes care about temporal stablty of segmentaton results for mage sequences, although t s requred for stable road trackng. In contrast, the proposed approach, attempts to classfy road pxels n the nput mage, nstead of complete parttonng of the mage, whch enables us to dentfy non-connected road segments n a straghtforward way. Furthermore, the temporal factor s treated naturally as an essental part of the proposed algorthm. Fnally, let us also show some segmentaton results for KITTI vson benchmark. The benchmark for road detecton algorthms has been ntroduced by Frtsch et al. [19], and some results for ths data set are shown n Fg. 19. As can be seen from the fgure, the results are consstent wth our prevous dscusson. The proposed algorthm s able to perform proper segmentaton for roads wth contnuously varyng colors, however t s not yet capable to Fg. 19 handle hard cast shadows. Results for KITTI vson benchmarks. 5. Concluson and Further Work The paper has been focused on the ssue of road regon segmentaton from sequental mages captured by a vehcle mounted camera. The road regon segmentaton has been treated as a classfcaton task, where road pxels are classfed by statstcal decson, usng a pdf of road features. The paper has presented a new algorthm for the sequental estmaton of the pdf. The proposed algorthm s based on the sequental Monte-Carlo (SMC) method, and the goal was to develop an algorthm wth a hgher segmentaton ablty, whle preservng strong ponts of the algorthm developed n earler work on ths topc. The key dea of our proposal s based on the prncple of mportance samplng, and spatal parts of samples are generated by unform samplng of spatal coordnates from road regon segmented at prevous tme step. Assgnment between successve sample sets s performed, and wth ths help, color parts of samples are generated. Transton pdf and observaton pdf are then evaluated to update sample weghts. Wth the proposed algorthm, we have acheved a better spatal dstrbuton of the samples and got better segmentaton results then wth the earler one. The proposed algorthm has been evaluated on three types of data sets, and compared to the orgnal SMC based algorthm, as well as to two types of conventonal algorthms. The evaluaton results shows that the proposed algorthm produces sgnfcantly better segmentaton results than the orgnal SMC algorthm, and comparson to the conventonal algorthms shows the valdty of the SMC based approach. It should be especally remarked that the proposed algorthm produced good results on dfferent mage sets, although the same hyper-parameters were used. Ths shows robustness of the proposed approach, and demonstrates clear dfference wth reference algorthms, whch seem to be very senstve to the hyper-parameter tunng. Snce RGB features are not guaranteed to be optmal for road regon segmentaton, more detaled study on the type of features must be carred out. Especally a dfferent color model or the c 2016 Informaton Processng Socety of Japan 9

10 IPSJ Transactons on Computer Vson and Applcatons Vol (Feb. 2016) above mentoned ntrnsc mage needs to be taken nto account to fnd features more robust to crtcal llumnaton condtons lke hard-cast shadows. The proposed approach can serve as a complementary method to a stereo vson based approach. Snce t s dffcult to fnd matchng ponts on homogeneous low textured road surfaces to compute dense dsparty map n a stereo based approach, many road detecton algorthms perform 3D road detecton by fndng road markngs or road boundares n par of stereo mages [20], [21]. However, the regon based approach proposed n ths paper can produce very good segmentaton results especally for homogeneous road surfaces wth almost no texture, therefore t has potental to produce more powerful results f combned wth the stereo based approach. Study on ths topc remans as a feature work. Another approach to road segmentaton s based on other knds of sensory nformaton than cameras. A method based on LIDAR s reported e.g. by Fernandez et al. [22]. Although LIDAR has generally lower resoluton than a camera sensor, t can produce more accurate 3D measurements of road surfaces then stereo based technques. Combnng LIDAR data wth mage features has also potental to buld up more powerful method, however the man drawback of the LIDAR s the relatvely hgh costs of such sensors. Fnally, more effectve methods for evaluaton of the segmentaton ablty of the algorthm must be developed, because the evaluaton method relyng on manually created ground truth s mplausble for evaluaton on large data sets. Acknowledgments The author would lke to thank Mr. Paul Kostamo for hs knd help n proofreadng. References [1] Bertozz, M., Brogg, A., Cellaro, M., Fascol, A., Lombard, P. and Porta, M.: Artfcal Vson n Road Vehcles, Proc. IEEE, Vol.90, No.7, pp (2002). [2] McCall, J.C. and Trved, M.M.: Vdeo-based lane estmaton and trackng for drver assstance: Survey, System, and Evaluaton, IEEE Trans. Intellgent Transportaton Systems, Vol.7, No.1, pp (2006). [3] V. Kastrnak, M.Z. and Kalatzaks, K.: A Survey of Vdeo Processng Technques for Traffc Applcatons, Image and Vson Computng, Vol.21, No.4, pp (2003). [4] Hllel, A.B., Lerner, R., Lev, D. and Raz, G.: Recent progress n road and lane detecton: A survey, Machne Vson and Applcatons, Vol.25, No.3, pp (2014). [5] Crsman, J. and Thorpe, C.: SCARF: A Color Vson System that Tracks Roads and Intersectons, IEEE Trans. Robotcs and Automaton, Vol.9, No.1, pp (1993). [6] Foedsch, M. and Takeuch, A.: Adaptve Real-Tme Road Detecton Usng Neural Networks, Proc. 7th Int. Conf. Intellgent Transportaton Systems, Washngton D.C (2004). [7] Zhou, S., Gong, J., Xong, G., Chen, H. and Iagnemma, K.: Road detecton usng support vector machne based on onlne learnng and evaluaton, Intellgent Vehcles Symposum 2010, pp (2010). [8] Shang, E., An, X., Ye, L., Sh, M. and Xue, H.: Unstructured road detecton based on hybrd features, Proc nd Internatonal Conference on Computer and Informaton Applcatons (ICCIA 2012), pp (2012). [9] Le, M.C., Phung, S.L. and Bouzerdoum, A.: Lane Detecton n Unstructured Envronments for Autonomous Navgaton Systems., Proc. 12th Asan Conference on Computer Vson (ACCV 14), pp (2014). [10] Ramström, O. and Chrstensen, H.: A method for followng of unmarked roads, Proc. Intellgent Vehcles Symposum 2005, pp , IEEE (2005). [11] Oh, C., Son, J. and Sohn, K.: Illumnaton robust road detecton usng geometrc nformaton, 15th Internatonal IEEE Conference on Intellgent Transportaton Systems (ITSC2012), pp (2012). [12] Alvarez-Mozos, J., Lopez, A. and Baldrch, R.: Illumnant-nvarant model-based road segmentaton, Intellgent Vehcles Symposum, 2008 IEEE, Endhoven, Netherlands, pp (2008). [13] Prochazka, Z.: Road regon segmentaton based on sequental Monte- Carlo estmaton, Proc. ICARCV 2008, pp (2008). [14] Isard, M.A.: Vsual Moton Analyss by Probablstc Propagaton of Condtonal Densty, PhD Thess, Unversty of Oxford (1998). [15] Doucet, A., Godsll, S.J. and Andreu, C.: On Sequental Monte Carlo Samplng Methods for Bayesan Flterng, Statstcs and Computng, Vol.10, No.3, pp (2000). [16] Prochazka, Z.: Pathway estmaton for vson based road followng sutable for unstructured roads, Proc. ICARCV 2012, pp (2012). [17] Fnlayson, G.D., Drew, M.S. and Cheng, L.: Entropy Mnmzaton for Shadow Removal, Internatonal Journal of Computer Vson, Vol.85, No.1, pp (2009). [18] Felzenszwalb, P.F. and Huttenlocher, D.P.: Effcent Graph-Based Image Segmentaton, Internatonal Journal of Computer Vson, Vol.59, No.2, pp (2004). [19] Frtsch, J., Kuehnl, T. and Geger, A.: A New Performance Measure and Evaluaton Benchmark for Road Detecton Algorthms, Internatonal Conference on Intellgent Transportaton Systems (ITSC) (2013). [20] Danescu, R. and Nedevsch, S.: Probablstc Lane Trackng n Dffcult Road Scenaros Usng Stereovson., IEEE Trans. Intellgent Transportaton Systems, Vol.10, No.2, pp (2009). [21] Bóds-Szomorú, A., Dabócz, T. and Fazekas, Z.: A Lane Detecton Algorthm based on Wde-Baselne Stereovson for Advanced Drver Assstance, 7th Conference of the Hungaran Assocaton for Image Processng and Pattern Recognton (KPAF 2009), Budapest, Hungary, John von Neumann Computer Socety, pp.1 10, Paper (2009). [22] Fernandez, R., Premebda, C., Pexoto, P., Wolf, D. and Nunes, U.: Road Detecton Usng Hgh Resoluton LIDAR, Proc. Vehcle Power and Propulson Conference (VPPC), pp.1 6 (2014). Zdeněk Procházka was born n 1970 n former Czechoslovaka. He receved hs M.Sc degree n rado-electroncs n 1994 from the Faculty of Electrcal Engneerng, Czech Techncal Unversty n Prague, Czech Republc. In 2000 he receved hs Ph.D. degree from Unversty of Electro-Communcatons n Tokyo, Japan. In 2007 he became an assocated professor of Natonal Insttute of Technology, Ota College n Ota, Japan. (Communcated by Yu-Wng Ta) c 2016 Informaton Processng Socety of Japan 10