II. THE ALGORITHM. A. Depth Map Processing

Transcription

1 Lerning Plnr Geometric Scene Context Using Stereo Vision Pul G. Bumstrck, Bryn D. Brudevold, nd Pul D. Reynolds CS229 Finl Project Report December 15, 2006 Abstrct A relible method for detecting plnr regions in video/stereo scene would be of gret use to the field of computer vision. Solutions to this problem re pplicble to object recognition, scene identifiction, nd robot-relted pplictions. In this pper we present plne-finding lgorithm tht uses dt from binoculr stereo cmer system to produce lbeled output imges showing the mjor plnes in video scene. The lgorithm is bsed on the three-dimensionl Hough Trnsform but lso presents mny useful pproches nd heuristics pplicble to generl plne-finding. T I. INTRODUCTION he gol of our project ws to use imge nd depth dt from stereo cmer system to locte the mjor plnes in n imge. A successful lgorithm would find immedite ppliction in res such s scene identifiction, object recognition, nd robot-environment interction. After trying severl pproches built round unsupervised lerning lgorithms, we converged on n lgorithm bsed on the three-dimensionl Hough Trnsform. Our dt ws gthered using binoculr stereo cmer with 4-cm bseline. Our input dt set consisted of imges of indoor office nd hllwy scenes with vrying levels of clutter. Ech entry in the dt set consisted of n imge pir: one, monochrome imge from the cmer's left eye, nd two, depth mp contining the estimted distnce vlue for ech pixel (provided directly by the cmer softwre [1]). Figure 1 shows two exmple pirs of imges. Drk blue regions in the depth mp indicte points where no distnce redings were returned due to lck of distinct fetures in tht re. One of our primry ssumptions ws tht there were two types of evidence for plnes: one, loclized nd contiguous evidence found on textured surfces (such s desks), nd two, sprse nd scttered evidence when most of the plne is fetureless nd returns no depth dt (such s wlls nd ceilings). Our lgorithm focuses primrily on utilizing the first type of evidence since it ws typiclly the most relible. In regions where the second type of evidence predomintes, our lgorithm uses the mono imge dt to ssist plne clssifiction. A. Depth Mp Processing II. THE ALGORITHM Before proceeding with computtion, we observed tht the stereo depth mps suffered from two min sources of noise: Gussin noise on the distnce redings nd nother source kin to slt-nd-pepper noise which ws cused by poor feture mtching from the stereo cmer (returning distnce redings mny meters off from the true vlues). To combt both of these sources of noise the depth mp ws subjected to Fig. 1 Two exmple mono cmer imges (left side) nd their corresponding depth mps (right side). Drk blue in the depth mps indictes plces where no dt ws returned. Fig. 2 Effects of modified low-pss filtering on the point cloud obtined by bck-projecting the depth mp shown on the lower right in Fig. 1. Here () is the originl, unfiltered depth mp nd (b) is the filtered one. Note tht the three verticl lines from the door re lost in () but pper strongly in (b). b

2 dρ dρ more hevily thn points frther wy, thus roughly preserving the density of dt points per surfce re in the point cloud. In imges with mny dt points, the decimtion rtio ws usully 10:1, but in sprse imges no decimtion ws performed so s to retin ll of the given dt from the stereo cmer. b c Fig. 3 Exmple of the sensitivity of the 3D Hough Trnsform to dρ. The mximum response of the Hough Trnsform in ech cse is shown in yellow. In () dρ is chosen well nd the mximum response selects strong plne; in (b) dρ is slightly too lrge nd the plne fit is off; nd in (c) dρ is much too lrge nd ll plnr informtion is lost. dρ Fig. 4 Exmple of the mximum response of the Hough Trnsform being skewed by seprte groups of plnr points even when dρ is chosen well. c Fig. 5 The effects of segmenting the point cloud in () re shown s color-coded groups in (b). (c) is obtined by projecting the color-coded groups bck to 2D imge. modified low-pss filtering before it ws used. After bckprojecting the smoothed depth dt into 3D point cloud, the beneficil effects of this step cn be seen in Figure 2. The depth mp ws lso decimted in order to reduce computtion time. We used modified rndom decimtion lgorithm where points closer to the cmer were decimted B B. 3D Hough Trnsform Next the 3D Hough Trnsform is run on the 3D point cloud. In the Hough Trnsform, every point votes for every plne tht psses within some distnce of it. Thus the mximum response over the trnsform indictes the best guess for plne in the region. Plnes in the 3D Hough Trnsform re described by their norml vectors which re specified by two ngles (zimuth, θ, nd elevtion, φ) nd the vector s Eucliden norm (ρ). Since the Hough Trnsform is discrete, it is prmeterized by the step size in ll three of these vribles. Of the three of these, the Hough Trnsform s mximum response ws most sensitive to dρ (s illustrted in Figure 3). We ddressed this sensitivity by setting dρ to be 10 cm since this ws smll enough to detect most plnes throughout the dt set while still lrge enough to ccommodte the noise. Step sizes of 5º proved sufficient for both dθ nd dφ. Another problem ws tht the Hough Trnsform s mximum response over severl, unconnected groups of points ws often non-plnr even when the individul groups themselves were very plnr (this is illustrted in Figure 4). We ddressed this problem by first performing 3D segmenttion on the point cloud nd then running the Hough Trnsform on ech of those segmented groups in isoltion. Our 3D segmenttion lgorithm works by quntizing the entire point cloud into series of quntum boxes of size 30x30x30 cm. Next it selects the quntum box contining the highest number of points nd connects to it ll of the other contiguous quntum boxes lso contining high number of points. This group is then lbeled nd removed from the set of points. This procedure is repeted until 80% of the point cloud hs been grouped (the remining 20% were typiclly outliers). A smple result of this 3D segmenttion is shown in Figure 5. After this, the Hough Trnsform is run on ech segmented group of points in the following mnner. First, it is run over ll of the points nd the mximum response is determined (this is shown in Figure 6. with the red points). Then ll of the points corresponding to the mximum response re extrcted nd the Hough Trnsform is re-run on the remining points (Figures 6.b nd 6.c show these successive pplictions of the Hough Trnsform). This is repeted until 70% of the originl points hve been exhusted (processing the finl 30% often generted mny wek plne guesses). C. Plne Guess Processing After the Hough Trnsform step, the lgorithm possesses series of plne guesses specified by their norml vectors, centroids, nd ll of their ssigned points. These rw results often contin mny flse plnes so they re first decimted: those guesses tht re drwn from too few depth points or

3 which occupy too little surfce re re thrown wy. Also, becuse of the fineness of dρ, these guesses often contin mny repetitious guesses for single plne; thus the guesses re clustered. In this step, two or more plnes re combined if their norml vectors re highly ligned nd if their centroids lso stisfy some similrity conditions. Also fctored in re elements from ech point cloud s singulr vlue decomposition which contined vitl informtion on ech plne s geometry. We lso mde use of the requirement tht vlid plnes do not occlude too mny of the given stereo dt points (we ssume ll plnes re solid nd opque, so line-of-sight constrints must be stisfied). Thus, before ny two plne guesses re combined, the supposed new plne is checked to see if it occludes too much depth dt (this is illustrted in Figure 7). Single plne guesses re lso dropped if they violte this occlusion rule themselves. Figures 9.e nd 10.e show the results fter this step: colored regions indicte points in evidence for the finl plne guesses, nd the ttched point cloud plots show the norml vectors of ll plne fits. D. Plne Region Lbeling Finlly we desired to mke the plne lbels mtch the monochrome imges better. The output lbels of the lst step did not mtch very well becuse they were drwn entirely from the stereo dt nd so hd little direct reltion to the mono imge boundries. We corrected this by introducing imge segmenttion on the mono imges. We used superpixel-, grph-bsed pproch published by Felzenszwlb nd Huttenlocher [2]. An exmple output of their segmenttion lgorithm is shown in Figure 8. For the segmented imges, we djusted the prmeters so tht the segmenttion ws fine enough tht ech segment overlpped with only one finl output plne with high probbility. Thus, in the finl step of the lgorithm, ech imge segment is ssigned to the plne lbel with which it hs the most overlp (exmples of these processed imges pper in Figures 9.c nd 10.c). This hs the effect of spreding out the finl plne lbels into loctions better-defined by the imge boundries, nd it lso llows smll mount of depth dt to provide plne lbel for lrge, texture-less regions tht returned no dt. c Fig. 6 Itertive procedure for running the Hough Trnsform. () shows the first mximum response; (b) shows the next mximum response fter the points from the first hve been removed; nd (c) shows the lst mximum response. (d) shows these three plne lbels on the corresponding mono imge. Fig. 7 Plne occlusion exmple. Here the lgorithm considers combining the two seprte light blue groups into one plne. It clcultes the convex hull (blue outline) of the new, proposed plne nd checks whether it occludes ny points. In this exmple, the purple points lie in front of the plne nd re not occluded, but the yellow nd green points lie behind the plne nd re occluded. Thus the two blue plnes re not combined into single plne. b d III. RESULTS Figures 9 nd 10 depict some representtive results. For reference, imges showing the results on ll of our input dt re ttched to this report. Figure 9 shows typicl result on textured, close-up imge. Our lgorithm performs best on these types of imges since the plnes re highly-textured nd re close to the cmer, producing dense point clouds nd enbling very relible results. Figure 10 shows result on cluttered indoor scene with most objects lying frther wy from the cmer thn in Figure 9. Here the lgorithm still succeeds t finding most of the Fig. 8 Smple output of the imge segmenttion lgorithm by Felzenszwlb nd Huttenlocher [2]. Prmeters set to k=500, σ=0.3.

4 plnes but it hs trouble ssigning them to the proper imge regions. It lso exhibits mny more spurious results thn in the previous exmple. b IV. CONCLUSIONS AND FUTURE WORK Overll our lgorithm succeeded in finding mny plnes in certin settings. It works best in scenes with high-texture nd low clutter where it is ble to identify both the correct plnes nd their orienttions with high ccurcy. It is still wek in finding occluded plnes in cluttered environments s well s in low-texture imges with very sprse depth dt. And even when it finds the correct plnes it is still prone to mislbeling them in the output step. The results chieved so fr re promising, but there re mny possible directions for improvement. For instnce, our lgorithm produces hrd clssifictions, but nother pproch would be to build probbilistic model tht estimtes the number of plnes through n dptive method nd then ssigns weights to ech point indicting how likely they re to pper in ech plne. One could lso employ prior ssumptions bout the geometry of the room to reduce spurious results. Some possible ssumptions include tht there re lwys wlls bounding the scene, tht the mjor plnes should be orthogonlly oriented (the wlls, floor, nd ceiling), nd tht the best plnes tend to be strictly horizontl or verticl (tble surfces, desk tops, nd doors). b c e Fig. 10 Another set of exmple results with the sme imge rrngement s given in Figure 9. Here the scene contins more clutter nd is more open shot. d c e d Lstly, we only exmined the tsk of finding plnes given single imge, but in mobile robot ppliction it would be possible to mke use of informtion from multiple, djcent video frmes when ttempting plne clssifiction. ACKNOWLEDGMENTS We would very much like to thnk the mny people who provided dvice nd direction on this project, including Prof. Andrew Ng, Prof. Jn Koseck, Stephen Gould, Ashutosh Sxen, nd the members of the Fll 2006 STAIR Vision tem. REFERENCES [1] Tyzx stereo cmer informtion. [2] P. F. Felzenszwlb nd D. P. Huttenlocher. "Efficient Grph-Bsed Imge Segmenttion" Interntionl Journl of Computer Vision, Volume 59, Number 2, September Fig. 9 Exmple results. () shows the originl mono imge, (b) the originl depth mp, (c) the process nd lbeled output, (d) the hndlbeled ground truth, nd (e) composite imge showing the points for ech finl plne s well s their norml vectors.

5 Appendix Left Cmer Imge Depth Mp Segmented Imge Lbeled Plnes Ground Truth Plnes

6 Appendix Left Cmer Imge Depth Mp Segmented Imge Lbeled Plnes Ground Truth Plnes

7 Appendix Left Cmer Imge Depth Mp Segmented Imge Lbeled Plnes Ground Truth Plnes

8 Appendix Left Cmer Imge Depth Mp Segmented Imge Lbeled Plnes Ground Truth Plnes