University of Alberta, range data with the aid of an o-the-shelf video-camera.

Transcription

1 Single Camea Steeo fo Mobile Robot Wold Exploation Dmity O. Goodnichy and William W. Amstong Depatment of Computing Science, Univesity of Albeta, Edmonton, Albeta, Canada T6G 2H1 Abstact This pape intoduces a single-camea-based steeo vision system used fo ceating local 3D occupancy models. The design of the system is descibed, the ange data eo analysis is pesented and the senso model which assigns the values of evidence to the egisteed data is built. The application of the poposed system fo mobile obot wold exploation is shown. Data obtained by unning a single camea mobile obot ae pesented. Keywods: Occupancy gids, visual senso model, ange data fusion, evidence theoy. 1 Intoduction In wold exploation, the occupancy model of the wold is one of the most commonly used [3, 20, 7, 25]. In this model, the evidence that a point in space is occupied is calculated, based on the data egisteed by a ange senso. Oiginally developed fo building 2D maps [9], the occupancy-based appoach has ecently been extended to build 3D models of the wold [15, 17], which povide much moe infomation about the envionment. Thee ae two poblems howeve with building 3D occupancy wold models. The st poblem is the epesentation of the occupancy infomation. The conventional way of stoing the occupancy data using gids equies a consideable amount of memoy and is time consuming. It is also inecient fo map extaction. The second poblem concens ange sensos. Sona sensos ae not expensive and thei models ae known [3, 16]. Howeve, they do not yield accuacy sucient fo 3D modeling [15]. At the same time, lase ange sensos and highly calibated steeo systems, which also have well dened senso models, ae vey expensive and cannot be used in many situations. These wee the poblems we addessed in the Boticelli poject. As a solution, st, we poposed a egession-based technique fo fusing ange data, which allowed us to build 3D occupancy models epesented in a paametic way, and second, we designed a single camea visual senso, which allowed us to egiste 3D ange data with the aid of an o-the-shelf video-camea. Boticelli is the name of the obot we used fo the poofof-concept demonstations. It exploes an unknown envionment by building local occupancy wold models based on the visual data captued by a single camea. While the issues of ange data fusion, occupancy model epesentation and vision-based navigation ae coveed in [4], [5] and [1], espectively, this pape is dedicated to the vision pat of the poject. We show how a single camea visual senso can be designed so as to povide ange data needed fo building 3D occupancy models of the equied quality. This includes designing the steeo ig, pesenting the eo analysis of the steeo algoithm, building the visual senso model and showing the advantages of using the poposed visual senso fo mobile obot wold exploation. The pape is oganized as follows. The design of the steeo setup is pesented in the next section. The senso model which assigns the values of evidence to egisteed depth data is built in Section 3. Expeimental esults and discussions conclude the pape. 2 Visual senso design The design of a visual senso depends on the objective of the poject. In ou poject the senso is used fo building local occupancy models, whee we want the models to be fast in calculation and compact in epesentation. Thee ae many applications whee such models can be used and in this pape we concentate on thei application fo the mobile obot exploation task. Let us outline this task.

2 Camea Mobile Base PTU Boticelli LOOK: - depth acquisition - looking fo goal THINK: - fusing ange data - wold modeling - map extaction DRIVE: - path planning - moving the obot MIL,Pentium 200,Windows NT a) b) c) Figue 1: Boticelli (a), its achitectue (b) and an envionment to exploe (c). 2.1 The exploation task We conside the task of exploing the envionment fo the pupose of locating a hidden taget. Figue 1.c shows the oom which is used as a testbed envionment in the poject. Stating fom an abitay position, the obot has to nd a taget, which ischosen to be a cone of a geen tiangle glued to white pape seen on the back wall in the gue. The decision how to exploe is detemined by 1)the knowledge of aleady obseved obstacle points, 2) the knowledge of exploation points, i.e. points whee no infomation is obtained yet, and 3) the knowledge of the taget location, if available. This detemines a theemodule achitectue of the obot, which we efe to as the \Look-Think-Dive" achitectue (see Figue 1.b). The st module is the vision module, duing opeation of which the obot ties to locate the taget and collects ange data aound itself. The amount of these data should not be vey lage so as not to impede the mobility of the obot. On the othe hand, it should suce to build a pecise enough 3D occupancy model of the wold, whee the pecision is measued by the ability tonavigate using the maps extacted fom the model. 2.2 Single camea steeo Many poblems in wold exploation by mobile obots ae attibuted to the odometic eos of the obot. Theefoe, it is desiable to get as much infomation aound the obot without having the obot move. This is achieved with a camea which has enough degees of feedom to captue the entie envionment. We use a Sony XC-999 camea mounted on an L- shaped suppot on a Diect Peception pan-tilt unit (PTU) on the top of a mobile base as shown in Figue 1. The angle and the length of the suppot ae chosen in such a way that the camea can obseve completely the pat of the wold fom the oo to the height of the obot, within a ange fom one decimete to thee metes. A gabbe gabs 640x480 colou (RGB) images, which ae then pepocessed with an aveaging lte to poduce 160x120 pixel images 1. Depth acquisition is done on these lowe esolution images. The Matox Imaging Libay (MIL) is used as an image gabbing and pocessing tool. The eadings fom a single camea steeo, which ae 3D depth data, ae obtained by the following thee step pocedue (see Figue 2). A set of featues is selected in the st fame (step 1). Each featue is then tacked along the epipola line in the second fame, which is gabbed afte the camea has moved, and the best match is obtained (step 2). The depth to those featues which ae selected and successfully tacked is then calculated on the basis of the dispaity of the featues in the two fames (stage 3). In the next subsection, we povide moe specics on the pocedue and below we addess anothe impotant issue the issue of uncetainty of visual senso data. Uncetainty of ange data As a esult of camea distotion, changing light conditions and incoect egistation of featues, the obtained 3D infomation is not cetain. Figue 4 shows the image of monochome geen ectangles as obseved by a camea. The waping of the pictue and the dieent intensities of the unifomly geen suface can be clealy seen. This esults in impefect tacking and matching of featues, which, in tun, esults in unde- o oveestimating the depth value coesponding to a featue 1 This size of image has been found optimal not only by us, but othe eseaches [7].

3 1 Evidence of occupancy Z θ Y h Ω Φ L z y 0 θ x Y Z θ L Φ θ m=(x,y,z) Figue 3: Depth calculation pocedue. Focus δθ pixel Figue 2: Depth data obtained using a single camea. as illustated in Figue 2. The gue also shows anothe majo eason fo uncetainty in depth estimation limited esolution of the camea. All these has to be taken into account when building a visual senso model. At the same time, in ode to decease the depth estimation eo, we esot to the following two techniques. Fist, we disegad the maginal featues (as in [25, 7]), since they intoduce high eo not only because of the image quantization but also because of the waping of the image. Second, when the obot views the suounding wold, we make sue that each selected featue is obseved at least twice, so that it appeas at least once in the middle of the image, whee its eo is low. This is achieved by adjusting the angle of pan otation D data egistation Accoding to the objective to build a wold model just good enough fo exploation and in ode to make fusion and wold modeling faste, we select only about 500 featues pe image. In paticula, the pixels with a high intensity deivative in the vetical diection ae selected as featues. The second fame is gabbed afte the camea moves vetically down, which explains ou choice of selecting featues. The angle of the camea tilt otation is =7:7 o and the leve length L = 21 cm, which esults in the baseline h 3 cm. This poduces the dispaityof 10 pixels on aveage, which is the same as in [7]. In the second fame, each featue is tacked along the epipo- la line, which ischosen 3 pixels wide to account fo waping of the image, using a 5 by 5 scanning window centeed on a pixel. A featue is consideed successfully tacked if the eo E between the best match and the oiginal featue is lowe than a cetain theshold E thesh. By loweing the theshold E thesh, we can educe the amount of uncetain data. This ltes away appoximately 60% of featues. The match eo E is calculated as the Euclidean distance between the nomalized 2 N-dimensional vectos (N = 25) obtained by using the scanning window: E = k ~ V, ~ V 0 k 2 = NX n=1 (V [n], V 0 [n]) 2 ; (1) which is a standad appoach in featue tacking [6]. Finally, the depth to those featues which ae selected and tacked is calculated, using the tiangulation based on the pojective camea model [8]: ~m, ~ h = R~m 0 0 ; (2) whee ~m =[i; : j; F ] unit and ~m 0 : =[i 0 ;j 0 ;F] unit designate unit vectos detemined by the positions of a featue in the st and the second fame espectively. F is the focal length of the camea, which is known fom the camea specications o calculated in advance using the vanishing point technique descibed in [8]. The camea we use has F = 150, i 2 [,53; 53] and j 2 [,40; 40]. R is the otation matix and ~ h is the tanslation vecto of the camea. Both ae known, since only the pan-tilt unit moves and not the obot duing depth acquisition. Since a featue moves vetically only, Eq. 2 can be ewitten, using the coodinate method (see Figue 3), as 0 sin( +, 0 )=Ztan(, ), h cos 2 0 cos( +, 0 )=Z + h sin ; (3) 2 2 Moe exactly, the intensity of a cente pixel is subtacted fom the intensities of all pixels in the window.

4 whee tan( 0 )= j 0, tan() = j and is the angle of F F the camea suppot. Dividing the st equation by the second one yields the following fomula fo (X; Y; Z) coodinates of a featue in the coodinate system centeed on the st location of the camea as shown in Figue 2: 8>< >: Z = h cos 2 +h sin 2 tan(+,0 ) tan(,)+tan(+, 0 ) X = Z tan Y = Z tan x ; whee tan x = i F (4) To obtain the coodinates (x; y; z) of a featue in the PTU-centeed coodinate system, vecto ~m = (X; Y; Z) is multiplied by a homogeneous matix descibing the cuent position of the camea, which is a function of camea pan and tilt angles. Afte depth is calculated fo the cuent position of the camea, the camea is panned on the PTU clockwise and the pocedue is epeated fo the new angle of view, until nally all pats of the wold aound the obot ae obseved. A thing to be mentioned about the single camea vision system is the paallelism of its opeations the depth is calculated, while the camea is moving. Because of that, the time needed to acquie depth infomation about the suounding envionment is just equal to the time needed to complete the full otation of the camea. It takes 15 dieent pan positions of camea to obseve the whole envionment and the whole pocess of a building a spase depth map of the entie envionment takes about one minute. Figues 6.a and 6.b show the depth infomation acquied by the obot by looking aound fom two dieent locations. Registeed 3D featues ae shown pojected on the oo (Oxy plane), the obot is located in the cente. The gues also show gabbed images (in left top cones) and pais of 2D featues used in depth calculation (in left bottom cones): in white ae the featues selected in the st fame, while in black ae the featues which ae tacked in the second fame. 2.4 Seaching fo the taget As opposed to a steeo setup with a xed camea conguation, a single camea steeo allows abitay motion of the camea. This gives moe exibility not only in tacking the featues but also in seaching fo the taget. In this poject, we ae not concened with the issue of taget ecognition. Instead we choose the taget to be invaiant to the distance, which explains ou choice of a cone of an object as the taget. The taget is sought by checking each image fame fo the existence of a patten peviously stoed in memoy. MIL has a function Figue 4: The couption of an image by a camea. which can do this opeation eciently. If the taget is found, the same depth calculation outine which is used fo featues is used again to poduce the location of the taget with espect to the cuent position of the obot. 3 The visual senso model 3.1 Evidential appoach In senso fusion, the concept of the senso model is of pime impotance. Pobabilistic appoaches [22, 9] de- ne senso model as the conditional pobability P (~ = occj~ S ) that a point ~ in space is occupied, given a ange senso measuement ~ S. It has been agued howeve that pobabilistic appoaches ae not valid in building a senso model when a senso is not eliable [23, 16]. Fo example, if a senso woks popely only 3 times out 4 (because of powe failues o othe poblems), then a measuement ~ S, which, we may say, is 75% eliable, povides some infomation about the occupancy of a point, but it does not give any data about the negation, i.e. about the emptiness of this point. The evidential appoach has been suggested to cicumvent this poblem. Rathe than dealing with pobabilities, this appoach consides two values of evidence: the evidence m occ (~) that a point is occupied as well as the evidence m emp (~) that a point is empty. These values of evidence ae the functions of the paametes descibing the eliability of the measuement and ae obtained using the intinsic chaacteistics of the senso. The evidential appoach is also cedited with esolving the \unknown vs contadictoy" ambiguity, which aises in st moment pobabilistic appoaches. Pobabilistic appoaches, which use second moments of the unknown vaiables, like Kalman lte appoaches [13],

5 ae too computationally expensive and theefoe ae not vey suitable in mobile obotics. The main citicism of the evidential appoach concens the Dempste-Shafe ule, which is used to combine the evidence data. This ule assumes that souces of evidence ae distinct and independent, so that no evidence is counted epeatedly [24]. In senso fusion howeve, the same piece of evidence is often obseved moe than once. This is why in [4]we poposed a new, egession-based technique fo combining ange evidence data. This technique is used fo fusing ange data egisteed by the single camea steeo and is fomulated as follows. Given a set of sample points ~ along with thei evidence values m occ and m emp povided by the senso model, nd a smooth piece-wise linea appoximation of functions m occ = m occ (~) and m emp = m emp (~) on the entie input domain. This detemines the design of the single camea steeo senso model. 3.2 Uncetainty of egisteed data Industially manufactued sona sensos [3] and lase ange ndes [17] have well dened senso models povided by a manufactue. Howeve, thee is no geneal senso model of visual ange sensos, which is due to the divesity of the visual system setups. Thus, we have to design ou own model of the single camea ange senso. Taking into account the quantization eo As mentioned in Section 2, the depth data obtained by a vision system is not cetain fo many easons. Due to the nite esolution of the image, the angle 0 in the Eq. 4 is known only with the pecision 0 = 1 F (see Figue 2.b). This esults in the ange eo, which can be estimated by taking a deivative of = (X; Y; Z) with espect to 0 in Eq. 4. Anothe way of estimating the ange eo is to use the esults obtained fo non-convegent dual camea steeo systems. The analysis of the uncetainty due to image quantization has been done in [2, 10,18] and using the esult obtained in [18], we get the following estimate of the ange eo: = 22 hf + : (5) Taking into account the match eo Calculation of the evidence values assigned to the egisteed ange data is based on the following idea. If we ae 100% condent in the ange data, then the ange data should get the evidence value one. On the othe a) 1 m occ 1 m emp b) 1 m occ 1 δ=0.1 m emp Figue 5: Visual senso model fo ideal (a) and eal (b) senso. hand, if the senso is completely uneliable, then the ange data should get the evidence value zeo. In the case of the single camea steeo, the measue of condence of egisteed depth data ~ is povided by the match eo E obtained duing the depth calculation pocedue (Eq. 1). In paticula, we obtain the evidence of a 3D point m occ (~), by applying the Tuckey by-weight to the eo E: (1 E, ( m occ (~) = E max ) 2 ) if E<E max ; (6) 0; othewise which is a common appoach in obust estimation [11, 19]. E max is a constant which ischosen in ageement with the theshold value E thesh used in lteing the outlies in Section 2.3. This appoach is dieent fom that of [14] and esembles that of [15]. It poduces the value of evidence in ange [0,1], which is used in fusing the ange data. 3.3 Linea epesentation In the case of the ideal visual senso, all points between the camea and the obseved point will be given the evidence values m emp (~) and m occ (~), as illustated in Figue 5.a. Figue 5.b shows the visual senso model fo the eal visual senso which is built accoding to the ideas descibed above. The maximum value of evidence is detemined by Eq. 6 The width of the ange eo is appoximated using the Eq. 5 as =0:1. We also make the evidence gow gadually fom zeo to its maximum value, using the ange eo as a guide in detemining the steepness of the slope, so that not to have innite deivatives of the occupancy function. The evidence behind the obseved point is zeo fo both occupancy and emptiness evidence values. The piece-wise linea epesentation of the senso model is chosen because of two easons. Fist, it facilitates the appoximation of the occupancy function with linea sufaces. Second, it signicantly educed the amount of sample data used in fusion. In paticula, the senso model can be epesented with only δ 2 δ

6 a) b) c) d) Figue 6: Depth data obtained by a single camea steeo (a,b) and the aea available fo navigation acquied fom the depth data (c,d) obtained at two dieent locations of the obot. few sample points on the ay of view, poviding that thee ae cetain constaints imposed on the function, which is descibed in moe detail in [4, 5]. This povides the solution to the poblem of edundancy of pocessed data, which, as was mentioned in the intoduction, is the majo poblem of the occupancy-based appoach. 4 Discussions The single camea steeo vision system descibed in the pape is tested using a mobile autonomous obot Boticelli. The obot is placed in an appoximately 5 by 6by 1.5 m oom suounded by walls which it has to exploe in ode nd a taget which is hidden behind one of the walls. Figue 1.c shows the oom and the taget. Stating fom an abitay location, the obot exploes the envionmentuntil it nds a taget. The exploation policy of the obot is detemined by the knowledge of obstacle and navigation points, which ae extacted fom multiple 3D local occupancy models built on the basis of the ange data egisteed by the single camea visual senso, and also by the knowledge of the taget location, which is acquied by the same senso. In ode to ensue that thee ae enough visual featues in the envionment, we put camouage clothes on the walls. These can be seen in Figues 1.c and Figue 6. Othe objects inside the exploation aea include a tee (seen in Figue 6.b), a couple of boxes (seen in Figue 6.a) and extension cods lying on the oo. Figues 6.c and 6.d show 3D occupancy models obtained fom depth data shown in Figues 6.a and 6.b. The points with occupancy values m occ highe then 0.6 ae shown pojected on the oo. The obot is located in the cente of the gues and is suounded by anunoccupied aea. This aea is consideed to be available fo the navigation. The occupancy models constucted fom the egisteed visual data ae found to be sucient fo making the navigation decisions. In ou expeiments, the obot successfully locates the taget while avoiding obstacles and the aeas aleady exploed. Thus we conclude that the single camea steeo vision system poposed in the pape, which is able to egiste eciently visual featues aound the obot, is vey suitable fo mobile obot exploation. Fo moe details on how occupancy models ae built fom the visual ange data see [4, 5]. The technique we use fo featue selection and tacking (Section 2), while simple and not time consuming,

7 suces fo applications like the one descibed above. Yet, if thee is a need fo a moe pecise depth data egistation, then the following steps can be undetaken to impove the pefomance of a single camea steeo: { using a digital camea instead of an analog one [21]; { ectifying the images [15], if an analog camea is used; { using an inteest opeato to select featues [15]; { using obust tacking appoaches, e.g. like those descibed in [12, 11]. As fo the visual senso model (Section 3), a bette appoximation of the ange eo should be used fo lage scale envionments. In addition, othe appoaches in assigning the evidence values to egisteed ange data can also be tied. Howeve, since the nal map of an aea available fo navigation is detemined by a theshold on an occupancy function, this assignment seems not to aect much the navigation planning pocess. Acknowledgments We gatefully acknowledge the suppot by the Defense Reseach Establishment Sueld (contact W R594/001) and by the Natual Sciences and Engineeing Reseach Council of Canada (OGP ). We also thank Xiaobo Li fo valuable feedback on the poject. Help of Monoe Thomas and Ron Kube in designing the PTU softwae is gatefully acknowledged as well. Refeences [1] W.W. Amstong, B. Coghlan, and D.O. Goodnichy. Reinfocement leaning fo obot navigation. In Intenational Joint Confeence on Neual Netwoks (IJCNN'99) poceedings, Washington DC, July 21-23, [2] S. Blostein and T. Huang. Eo analysis in steeo detemination of 3-d point postition. IEEE Tansactions on Patten Analysis and Machine Intelligence, 9(6):752{772, [3] L. Feng, J. Boenstein, and H.R. Eveett. Whee am i? sensos and methods fo autonomous mobile obot. Technical Repot UM-MEAM-94-21, The Univesity of Michigan, [4] Dmity O. Goodnichy. On using egession in ange data fusion. In Canadian Confeence on Electical and Compute Engineeing (CCECE'99) poceedings, May 9-12, Edmonton, [5] D.O. Goodnichy and W.W. Amstong. A paametic altenative to gids fo occupancy-based wold modeling. In Quality Contol by Aticial Vision (QCAV'99) confeence poceedings, May 18-21, [6] D.O. Goodnichy, W.W. Amstong, and X. Li. Adaptive logic netwoks fo facial featue detection. In Lectue Notes in Compute Science, Vol 1311 (ICIAP'97 Poceedings, Vol. II), pp , Spinge, [7] C. Jennings and D.Muay. Steeo vision based mapping and navigation fo mobile obots. In Poc. IEEE Intenational Confeence on Robotics and Automation, pp , [8] K. Kanatani. Geometic Computation fo Machine Vision. Oxfod Univesity Pess, [9] M.C. Matin and H.P. Moavec. Robot evidence gids. Technical Repot CMU-RI-TR-96-06, CMU RI, [10] L. Mathies and S. Shafe. Eo modeling in steeo navigation. IEEE Jounal of Robotics Automation, 3/3:239{247, [11] P. Mee, D. Mintz, A. Rosenfeld, and D. Kim. Robust egession methods fo compute vision: A eview. Intenational jounal of compute vision, 6(1):59{70, [12] C. Menad and A. Leonadis. Steeo matching using m- estimatos. In LNCS{1296, (Poc. of CAIP'97), pages 305{312, [13] A. Mitiche. Computational Analysis of Visual Motion. Plenum Pess, New Yok and London, [14] J. Miua and Y. Shiai. Vision-motion planning fo a mobile obot unde uncetainty. Int. J. of Robotics Reseach, 16:806{825, [15] Hans P. Moavec. Robot spatial peception by steeoscopic vision and 3d evidence gids. Technical Repot CMU-RI-TR-96-34, CMU RI, [16] D. Pagas, E. Nebot, and H. Duant-Whyte. An evidential appoach to pobabilistic map-building. In Reasoning with Uncetainty in Robotics (RUR'95) Inten. Wokshop poceedings, pages 165{169, [17] P. Payeu, P. Hebet, D. Lauendeau, and C.M. Gosselin. Pobabilistic octee modeling of a 3d dynamic envionment. In Poc. IEEE Int. Conf. on Robotics and Automation, pp , [18] J. Rodiguez and J. Aggawal. Stochastic analysis of steeo quantazation eo. IEEE Tansactions on Patten Analysis and Machine Intelligence, 12/5:467{470, [19] P.J. Rousseeuw and A.M. Leoy. Robust egession and outlie detection. New Yok : Wiley, [20] S. Thun. The mobile obot hino. AI Magazine, 15:31{ 38, [21] A. Ude and R. Dillmann. Vision-based obot path planning. In Advances in Robot Kinematics and Computational Geomety, Kluwe, Dodecht, pp 505{512,, [22] J. van Dam, B. Kose, and F. Goen. Neual netwok application in senso fusion fo an autonomous mobile obot. In Reasoning with Uncetainty in Robotics (RUR'95) Inten. Wokshop poceedings, pages 263{ 277, 1995.

8 [23] Fans Voobaak. Reasoning with uncetainty in ai. In Reasoning with Uncetainty in Robotics (RUR'95) Inten. Wokshop poceedings, pages 52{90, [24] Pei Wang. A defect in dempste-shafe theoy. In Uncetainty in Aticial Intelligence Confeence Poceedings ( [25] B. Yamauchi, A. Schultz, and W. Adams. Mobile obot exploation and map-building with continuous localization. In Poceedings of the 1998 IEEE Intenational Confeence onrobotics and Automation, Leuven, Belgium, 1998.