wrobot k wwrobot hrobot (a) Observation area Horopter h(θ) (Virtual) horopters h(θ+ θ lim) U r U l h(θ+ θ) Base line Left camera Optical axis

Transcription

1 Selective Acquisition of 3-D Information Enoug for Finding Passable Free Spaces Using an Active Stereo Vision System Atsusi Nisikawa, Atsusi Okubo, and Fumio Miyazaki Department of Systems and Human Science Graduate Scool of Engineering Science, Osaka University, Osaka , Japan Abstract We address te acquisition of 3-D structure around free spaces needed for veicle navigation using an active stereo vision system tat can xate on any 3-D point of te scene. Our metod is based on te ypotesis tat te larger and deeper free spaces exist in te direction to wic te larger and farter visible objects exist. We rst estimate te direction of promising free spaces by xating on te innity point. Ten we acquire te partial 3-D structure around te free spaces by successively xating on several 3-D points in te estimated direction. We nally present experimental results of te proposed metod applied to actual scenes. 1. Introduction Wit a robot moving in an unknown 3-D environment, it is absolutely necessary to work out a metod for nding passable free spaces. It is also an important tecnology even in case tat te robot is given a map of te environment in advance, because te robot must cope wit unknown obstacles or errors of te map. Metods ave been so far proposed[3][4][5][7] tat suggest representation of te environment in terms of free spaces using stereo vision. Most of tese studies, owever, generate te scene description by uniformly analyzing te wole image data including unnecessary information for nding free spaces wic means a vast amount of computation is required. Wen te objective of a robot is to reac te goal, te robot does not require a full/detailed description of te wole scene but only needs te information about passable free spaces to te goal and te partial 3-D structure around te free spaces. Tere ave been several works on selectively acquiring tese data. Takizawa et al. [11] proposed a scene observation metod tat suggests generation of coarse 3-D description from low-resolution stereo images, and ten selective zooming-in on te most promising pat (image region) estimated troug detailed analysis of te 3-D description. Tis metod, owever, needs a lot of computation to make te initial scene description because it is based on te constrained Delaunay triangulation. Jasiobedzki[6] proposed a metod of selective detection of oor regions using a planar oor model and a vision guided range detection sensor. It, owever, requires detailed calibration of te sensor-world model in advance. We propose a new metod to selectively acquire 3- D information enoug for nding passable free spaces wit an active stereo vision system tat xates on a 3-D point of te scene. We start by describing an algoritm to acquire 3-D information around a xation point. 2. Basic algoritm Let us consider two pin-ole cameras wit teir optical axes in te same plane. Assume tat te cameras can pan independently to make vergence movements and tey are mounted on te common platform tat tilts up/down as well as pans full circle so tat te system can xate on any 3-D point of te scene. Now suppose tat te active stereo rig xates on a 3-D point P ; let be te xation angle. We consider a circle passing troug te two nodal points of te cameras and te xation point as sown in Fig. 1(a). Tis circle is te set of points wose binocular disparity is zero and in general referred to as oropter[2]. We denote te oropter wit te xation angle by (). Also, let U l and U r be te set of points wose projection exists on respectively te left and te rigt image window(see Fig. 1(a)). Now we dene an observation area A as A = U l \ U r \ f( + 1) j 0 1 < 1 lim g were 1 lim is te upper limit of 1. A metod for acquiring te 3-D structure of te ob-

2 servation area as been already proposed by ourselves. Tis metod is based on te use of virtual oropters 1 as was proposed by Rougeaux et al.[9]. Te principle of our metod is simple: rst te input left image is sifted orizontally by s pixels. Ten, te input rigt image witin te window and te sifted left image are respectively divided into a set of small patc regions (see Fig. 1(b)) and compared in te corresponding patc location by a simple correlation metod between local image regions so tat te paired patc region wose correlation value is larger tan a tresold is extracted. Repeating te above process for s = 0; 1;... ; s max, only te projection of object surfaces lying in te observation area A remains on te matcing output. Tese 3-D positions can be easily computed by using te triangulation principle wit te rotation angle of eac camera and te sift value. Denoting te focal lengt of te camera and te widt of a pixel by f and w pixel respectively, we ave te following equation: s max = round-up [f 1 tan(1 lim )=w pixel ] (1) were round-up(x) indicates te function for rounding up x to nearest ones position. Te details of tis metod can be found in [8]. In tis algoritm, te xation angle and 1 lim, te pose of cameras to realize, and te size of image window are te indenite parameters tat depend on te given task. We ave worked out a metod for acquiring 3-D information enoug to searc for passable free spaces troug determination of tese parameters. Te proposed metod consists of te following two steps: (1)initial observation process in wic te robot nds te direction to wic promising free spaces exist, and (2)selective observation process in wic te robot turns to te direction of te detected free space and acquires te partial 3-D structure around te direction wit te successive vergence movements of te stereo cameras. In te following sections, te aforementioned two steps are considered in detail. 3. Initial observation process Te initial observation process is based on te ypotesis tat te deeper and larger free spaces(te more 1 s generated by sifting te left(or rigt) image orizontally by some pixels. In Rougeaux et al.[9], it was pointed out tat, near te image center, small sifts(s pixel) of te left image are almost equivalent to small virtual rotations (1 = tan 01 (s 1 wpixel=f )) of te left camera. Here, f is te focal lengt and wpixel is te widt of a pixel. Observation Fixation area point A P U l Optical axis U r (θ) Rigt camera W w W w W P s (sifting) P W Left image θ Image window Optical axis (θ+ θ lim) Rigt image (a) Observation area (Virtual) oropters (θ+ θ) j comparison between te intensity values at te corresponding position (m,n) m m n n i Y Sifted left image X j i Y Rigt image (b)image segmentation robot k wwrobot W f Image plane d min (c)image eigt Image plane d min Ww (d)image widt Figure 1. Active stereo vision based scene observation ( 0+ θ lim) = ( θ lim) θ lim b (a) (1 lim ) dfar Rigt camera (θ imax ) θ imax b (b) ( i max ) f dmin Rigt camera Figure 2. geometry for deriving Eqs.(4) and (5) promising pats) exist in te direction to wic te farter and larger visible surfaces exist. Now we dene a far object as a 3-D object wose dept is greater tan d far. In tis process, in order to nd far objects at rst, te xation angle is set to = 0(deg) by rotating te stereo cameras suc tat teir optical axes are mutually parallel and perpendicular to te baseline. Besides, te widt of image window (below denoted by W w (0) ) is set as widely as possible for te purpose of covering a wide range of te scene, wile te eigt of te image window(below denoted by ) is decided based on te minimal dept of te scene, say d min, and te size of te robot as sown in Fig. 1(c). Tat is, we ave X k robot w = I w ; (2) = round-up [f(k 1 robot )=(d min 1 pixel )] (3) wrobot

3 were I w, robot and pixel mean te image widt, te eigt of te robot and te eigt of a pixel respectively. k is a coecient for navigating te robot safely (see Fig. 1(c)). Furtermore, denoting te baseline lengt by b, 1 lim is geometrically(see Fig. 2(a)) given by 1 lim = 2 tan 01 (b=2d far ): (4) Under tese settings, te basic algoritm is executed. As a result, only te projection of far objects remains in te matcing output. In te present paper, te blob-coloring algoritm[1] is applied to te matcing result for region growing, so tat te direction from te viewpoint troug te centroid of te far object region wose area(= te number of pixels) is greater tan a tresold is selected as te direction to be explored in wic promising free space exists. 4. Selective observation process Now we explain te selective observation process tat follows te initial observation process. In tis process, at rst te camera platform is rotated to te direction of te detected large and deep free space and ten te partial 3-D structure around te direction is acquired wit te successive vergence movements of te stereo cameras suc tat te xation angle is i = 1 lim 2 i(i = 1; 2;... ; i max ). In so doing, te left and rigt camera is rotated wit 1 lim =2(deg) in opposite directions respectively. As sown in Fig. 2(b), i max = imax =1 lim = round-up 2 (2 tan 01 (b=2d min ))=1 lim 3 (5) so tat te 3-D object wose dept is greater tan d min can be extracted along te selected direction. On te oter and, bot te widt and te eigt of image window(below denoted by W w and W respectively) are decided based on te minimal dept of te scene and te size of te robot as sown in Fig. 1(c)(d). Tat is, we ave W w = round-up [f(k w 1 w robot )=(d min 1 w pixel )] ;(6) W = (7) were w robot and w pixel indicate te widt of te robot and te widt of a pixel respectively. k w is anoter coecient for navigating te robot safely (see Fig. 1(d)). 5. Experiments Experiments were conducted wit selective 3-D reconstruction using te scene sown in Fig. 3(a), to validate te proposed metod. Doll3 Doll2 (a) Experimental scene Can Tilt Vergence Camera Pan Weel (b) Our mobile robot equipped wit an active stereo vision system Figure 3. Experimental setup 5.1. Scene and parameters Te experimental scene includes tree dolls(doll 13), a box() and a can(can). Te distance from our robot(see Fig. 3(b)) to tem is about 1:3 1:5(m). Beind tese objects, tere are a lot of objects suc as desks, cairs, and bookselves; te distance from te robot is about 3(m). Tus, we set te two tresolds about dept to d min = 1100(mm), d far = 2000(mm) respectively. Note tat tere is a large free space between and. In tis paper we only sow two experimental results: te proposed metod was rst applied to te scene of Fig. 3(a) (EXPERIMENT I). After removing Doll 2 from te scene in order to make anoter large free-space, te same experiment was ten conducted(experiment II). Main experimental parameters are as follows : te image size is (pixel), te baseline lengt b is 120(mm) and focal lengt f is 16(mm), and te pixel size (w pixel ; pixel ) = (0:0277; 0:0218)(mm), so tat 1 lim 3:44(deg)(from Eq.(4)), te maximal sift value s max is 35(pixel)(from Eq.(1)), and i max = 2(from Eq.(5)). On te oter and, (w robot ; robot ) = (292; 272)(mm) and k w = 1:3; k = 1:1, ten, from Eqs.(2) and (3), te size of image window (W w (0) ; ) = (512; 200)(pixel) and, from Eqs.(6) and (7), (W w ; W ) = (200; 200)(pixel) Results and discussion EXPERIMENT I (case tat tere is a large free space in te scene): First of all, our robot nds te direction to wic larger and deeper free spaces exist by te initial observation process. Fig. 4(a) sows one of te stereo images(left one) 2 witin te (W w (0) 2 ) sized image window and Fig. 4(b) sows te extraction 2 In order to eliminate bot te ig-frequency components suc as noise and te low-frequency components suc as sading, we used te tree intensity level image[10] generated by Laplacian-Gaussian ltering followed by dual-tresolding instead of te original gray level image.

4 Doll 3 Doll 2 Can (a) Left image witin te image window Doll3 Doll2 Can z(mm) Selected Viewing direction (te direction of free space) (but not considered in te selective observation process) 1000 (b) Detected objects Figure 4. Detection result of far objects(initial observation process, EXPERIMENT I) Mobile Robot (a) Actual 3-D structure 500 Mobile Robot Detected object x(mm) (b) Acquired 3-D structure Figure 6. Te partial 3-D structure acquired by te proposed metod (EXPERIMENT I) (a) Left image (b)rigt image (c) Objects Figure 5. Detection result of objects (Selective observation, = 1, EXPERIMENT I) result of far objects; te wite area indicates te objects in te observation area (far objects), tat is, te direction to wic deep free spaces exist. We can see te large wite area between te projection of and tat of. As a result, te robot understands tat tere is a large and deep free space in te front direction. Secondly, te robot turns to te direction of te detected large and deep free space and acquires te partial 3-D structure around te direction wit te successive vergence movements of te stereo cameras suc tat te xation angle is i (i = 1; 2;... ; i max ). Fig. 5 sows te extraction result of objects around te selected free space; (a) and (b) are te stereo images witin te (W w 2 W ) sized window and (c) indicates te extracted objects (wite area). Tese were all obtained at te xation angle = 1. It is seen tat te rigt part of and te left part of are detected. By using te triangulation principle wit te rotation angle of te camera ead, te xation angle, and te sift value, te robot can easily compute teir 3-D positions. Fig. 6 illustrates te acquired 3-D structure of te scene. Comparing (a)(actual 3-D structure) wit (b)(acquired one), we can easily see tat te partial 3-D structure around te selected viewing direction(a part of and ) is acquired correctly. EXPERIMENT II (case tat tere are two large free spaces in te scene): First, te robot detects free spaces in te same way as EXPERIMENT I was conducted. Fig. 7 sows te detection result of deep free spaces. As a result, te robot knows tat tere are two large and deep free-spaces in te front and left direction. Ten, te robot turns to te direction of te detected free spaces and, wit te vergence movements, makes a selective reconstruction of 3-D objects around te direction. Fig. 8 sows te extraction result of objects in te front direction((a)to(c)) and tose in te left direction((d)to(f)). We can see te wite area of te rigt part of and te left part of in Fig. 8(c) and tat of te rigt part of Doll 3 and te left part of in Fig. 8(f). Te actual and acquired 3-D structure are sown in Fig. 9(a) and (b) respectively. We nally evaluate te cost of computation of te proposed metod. In eac step(initial/selective observation process), te basic algoritm is repeated (s max + 1) times for eac image patc, wile te number of te patc regions is proportional to te processing image size. Let K denote te proportional constant. Ten, te total computational complexity of our metod(denoted by C partial ) in terms of te number of computation of correlation is: C partial = (K 2 W w (0) (s max + 1) 2 i max 2 n max were n max indicates te number of selected viewing directions. Next let us consider te case of acquiring 3-D information by uniformly analyzing te wole image data. As te dept range is [d min ;1], te range of stereo disparities is [0; D] were D = (f=w pixel ) 1 b=d min. Te ) 2 (s max + 1) + (K 2 W w 2 W ) 2

5 2nd Viewing direction 1st Viewing direction z(mm) V2 V Doll 3 Can (a) Left image witin te image window Doll (but not considered in te selective observation process) Can 1500 (b) Detected objects Mobile Robot Doll3 Figure 7. Detection result of far objects (Initial observation process, EXPERIMENT II) (a) Actual 3-D structure Detected object x(mm) (b) Acquired 3-D structure Figure 9. Te partial 3-D structure acquired by te proposed metod (EXPERIMENT II) (a) Left image (b)rigt image (c) Objects 1st View (Front direction) experiments wit actual scenes. In future, we are going to explore a metod/strategy to nd proper viewpoints(i.e.,sub-goals to reac te goal) were bot unknown/occluded region and positional errors can be reduced. Doll 3 Doll 3 References (d) Left image (e)rigt image (f) Objects 2nd View (Left direction) Figure 8. Detection result of objects (Selective observation, = 1, EXPERIMENT II) maximal disparity D corresponds to te maximal sift value s max. Terefore, te complexity of tis case, say C all, is: C all = K 2 I w 2 I 2 (D + 1) were I w and I mean te image widt and te image eigt respectively. In te experiments, C partial =C all 2 100(%) = 41.7(%) (for EXPERIMENT I) and 60.1(%) (for EX- PERIMENT II), wic demonstrates te eectiveness of our metod in terms of computational cost. 6. Conclusion In tis paper, a metod was proposed for selective acquisition of 3-D information enoug for a robot to nd passable free spaces using an active stereo vision system. Te ig eciency of te proposed metod in terms of computational cost was veried troug [1] D. H. Ballard and C. M. Brown. Computer Vision. Prentice- Hall, Inc., Englewood Clis, NJ, [2] D. J. Coombs and C. M. Brown. Cooperative gaze olding in binocular vision. IEEE Control Systems, pages 24{33, [3] T. Ecigo. Segmentation of a 3-D scene into free areas and object surfaces by using occluded edges of trinocular stereo. In 1991 IEEE/RSJ Int. Worksop on Intelligent Robots and Systems, pages 863{868, [4] O. D. Faugeras, E. L. Bras-Melman, and J. D. Boissonnat. Representing stereo data wit te delaunay triangulation. Artif. Intell., 44:41{87, [5] E. Grosso and M. Tistarelli. Active/dynamic stereo vision. IEEE Trans. on Pattern Analysis and Macine Intelligence, 17(9):868{879, [6] P. Jasiobedzki. Detecting driveable oor regions. In 1995 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 264{270, [7] J. Miura and Y. Sirai. Modeling obstacles and free spaces for a mobile robot using stereo vision wit uncertainty. In 1994 IEEE Int. Conf. on Robotics and Automation, pages 3368{ 3373, [8] A. Okubo, A. Nisikawa, and F. Miyazaki. Selective reconstruction of a 3-D scene wit an active stereo vision system. In 1997 IEEE Int. Conf. on Robotics and Automation, pages 751{758, [9] S. Rougeaux, N. Kita, Y. Kuniyosi, S. Sakane, and F. Cavand. Binocular tracking based on virtual oropters. In 1994 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 2052{2057, [10] K. Sumi, M. Hasimoto, and H. Okuda. Tree-level broadedge matcing based real-time robot vision. In 1995 IEEE Int. Conf. on Robotics and Automation, pages 1416{1422, [11] H. Takizawa, Y. Sirai, and J. Miura. Selective renement of 3-D scene description by attentive observation for mobile robot. In 1994 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 1118{1125, 1994.