Efficient stereo vision for obstacle detection and AGV Navigation

Size: px

Start display at page:

Download "Efficient stereo vision for obstacle detection and AGV Navigation"

Elfreda Tate
5 years ago
Views:

1 Efficient stereo vision for obstacle detection and AGV Navigation Rita Cucchiara, Emanuele Perini, Giuliano Pistoni Diartimento di Ingegneria dell informazione, University of Modena and Reggio Emilia, ITALY, Vision-e srl, ITALY {erini, Abstract The aer resents the design and imlementation of a system of autonomous navigation of a mobile robot with stereo vision 1. Introduction One of the main ools of automation engines in modern industrial warehouses is the automatic transort of the goodies in the environment. AGVs (autonomous guided vehicles) can often reresent the most suitable and flexible solution. In the lasts years, several tye of vehicles and mobile robots have been realized and many aroaches have been roosed for semi- and fully autonomous navigation. AGVs generally exloit sensor-based information: as well as lasers, infrared and PIR sensors, cameras are very often exloited with comuter vision techniques to rovide self-localization, obstacle detection and other navigation tasks. De Souza and Kak in 2002 resented a comlete survey for comuter vision based autonomous navigation [5]. Here, many roosals are reviewed, which rocess single or coule of images for ma-based, ma-building-based and maless navigation. In this aer, we address ma-based autonomous navigation of mobile robots for industrial alications, based on visual sensors only. The goal is to create an integrated framework, sufficiently efficient to run on standard PCs in real-time, caable of roviding selflocalization, ma correction, navigation and obstacle detection for the autonomous guide of simle AGVs in indoor or outdoor environments. The roosal fully exloits stereo vision, integrates and imroves the state of the art of comuter-vision algorithms and exloits the multimedia exansion of Intel instruction set Architecture in order to otimize the software seed. Using the images catured from a stereo camera, the system calculates the distance between the vehicle and artificial atterns laced either on the floor either on the wall and uses this information to correct the osition readed from the weel encoders. Moreover, self-localization is imroved with attern recognition of ainted lines on the floor. Then, the disarity ma of the scene in front of the camera is builded and after an oeration of floor removal, it s able to localize ossible 3D obstacles. The main novelty of the work is the interleaved integration of different techniques for both localization and obstacle detection, with an efficient otimization, that allows a real-time autonomous navigation of a mobile robot, controlled by a standard PC. Due to the low comutational costs, and the limited hardware requirement, the rototye can be suitably adoted in many industrial alications. The aer is structured as follows. Section 2 reorts a brief list of related works in the toic of Autometed Guided Vehicle. Section 3 describes the system roosed and the imlemented comuter-vision techniques. Section 4 describes the imrovements of the algorithms and the rovided system tests; conclusions follow in Section Related works The general challenges of mobile vehicle navigation can be summarized with three questions: where am I, where am I going and how should I get there [1]; in order to answer these questions, the system should know the outut of a lot of sensors and mix them to obtain the most exactly osition in the environment [5]. Some sensors, such as lasers, cannot be used outdoor or can be dangerous for humans. Others have a low recision. Visual sensors, such as cameras in visible sectrum, have a wide usability and flexibility. Therefore, comuter vision techniques alied to images acquired in real time by cameras mounted on mobile robots could give an efficient solution to the fore mentioned questions. Vision based navigation is sreading in industrial alications; nevertheless, u to now, the high comlexity of the algorithms and the severe comutational costs limited de facto a wider diffusion. In this toic two main grous of techniques can be devised: the relative osition measurements, and the absolute osition measurements; in the first grou, the

2 odometry-based techniques [2,3] are mostly adoted, where two or more encoders measure the wheel rotation and/or steering orientation. Some system use also inertial navigation, like gyroscoes and accelerometers to measure the rate of rotation and acceleration. These methods can be affected by errors and imrecision, which increase during the time and often can derive. In the second grou, sensor-based techniques that rovide absolute localization are accounted: for instance, active beacons [4] comute the osition of the vehicle from measuring the direction of incidence of three or more actively transmitted beacons; model matching aroaches [6][17], comare the information acquired from the onboard sensors to the information stored in the ma of the environment to estimate the vehicle s absolute osition. In addition, there are techniques based on natural or artificial landmark recognition where distinctive features or artificial atterns are maed in a know location in the environment; laser range finders [7], or cameras [8] are exloited. The natural landmarks are distinctive features resent in the environment, detectable without the need of initial setting. They must be known in advance. Sometimes, very simle features are used and they can be easily confused each other; instead the main advantage of the artificial landmarks is that they can be designed for otimal detecting even under adverse environmental condition. An examle of artificial landmark detection is given in [11]. We use an imroved aroach, based on [11], to rovide the so-called incremental localization [5]: the integration of attern and landmark detection corrects an absolute osition comuted with odometry. Another roblem of mobile vehicle navigation is the obstacle detection; the aroaches roosed in literature can be divided in two gous indeendently from the used sensors: the former is based on the 3D recostruction of the environment [9] and the latter is based on homograhy tecniques [10]. In [10] to detect ossible obstacle on the road, the homograhy is followed by a line detector (straight or curved) to recognize the road lines and a simle and efficient obstacle detection with an algorithm called Inverse Persective Maing (IPM) using a coule of images. In this work, we use homograhy and line detection as in [10]. Conversely, IPM is substituted with a more comlex obstacle detection method, since IPM is not suitable in indoor or small environment where i) many obstacles could be resent at the same time or ii) some obstacles cannot be comletely visibile by both cameras or iii) aarent obstacles such as the walls of oen doors can be detected. In the matter of fact IPM does not detect obstacle but only the boundaries of disarity between floor and obstacles. An obstacle is detected by couling the two eaks of disarity. In revious conditions, the eaks couling is not always strigthforward. For this reason, IPM can be very useful only in some secific cases with single large obstacle, such as a car in front of the autonomous moving vheicle. In [15], a roosal for an autonomus robot using disarity ma is roosed. One of the urose of this work is to remove the floor form the calculated disarirty ma. This allows the detection of the obstacle ma in front of the veichle after floor removal. We adot a roosal similar to [15] integrated with the line detection [10] and the landmark detection [11] to make self localization and obstacle detection efficient and robust. 3. The roosal As introduced, in this roject we use basically the odometry to define the relative vehicle s osition on the environment, as in most industrial AGV. Then, stereo vision rovides incremental localization and error correction in the osition. Self- localization joints two techniques: artificial line detection and self landmarks recognition. The results of line detector correct the absolute angle of vehicle. The landmarks ositions correct the absolute coordinates. Line and landmark detection are interleaved to imrove the seed of the system. Conversely, at each frame, a fast stereo disarity ma algorithm [14] [12] is used: after a floor removal oeration, the relative osition of obstacles over the vehicle s ath can be comuted. Fig. 1 gives an overview of the comuter vision techniques adoted. Figure 2: Eiolar geometry 3.1 Eiolar calibration

Stereo visioning requires the air of stereo cameras to be registered, such that corresonding eiolar lines are identified. Classical eiolar geometry is recalled the icture of Fig. 2.

3 Stereo visioning requires the air of stereo cameras to be registered, such that corresonding eiolar lines are identified. Classical eiolar geometry is recalled the icture of Fig. 2. Registration needs a calibration rocedure: an efficient method is described in [13]. In our alication, the cameras are re-calibrated by manufacturer and the eiolar calibration arameter are stored in the flash memory of the cameras. The software driver rovide calibration during the acquiring rocedure. It s ossible to transform into a similar, antisimilar function s by substituting log x for x. S 2( log x log x ) s ( x) = S(log x) = In Fig.3 is indicated an examle of Self-similar attern generated using the function (3). (3 ) Figure 3: Self-similar Landmark Figure 2: Eiolar geometry 3.2 Self similar landmarks recognition A function f : R + R is -similar for a scale factor if If at least three landmarks are visible in both images catured from the stereo camera as in Fig. 4, the deicted distances D1, D2 and D3 can be calculated, using the information available with eiolar geometry. The algorithm detects all ossible landmarks that have a sufficient size not to be confused with other natural or artificial atterns. Using the information about the osition of the artificial landmarks in the sace, the angles α, β of Fig.4 can be derived and the absolute osition of the vehicle on the ma can be comuted., 0 < < 1, if ƒ(x) = ƒ (x) x > 0 (1) Self-similar intensity atterns, generated by (1) with a fixed, are attractive for landmark recognition since the roerty of -similarity is invariant to scale: this makes the recognition robust and insensitive to variable distance between cameras and atterns [11]. The eriodic square wave function S 1 0, x x < S( x) = 2 = 2( x x ) (2) 1 1, x x 2 has the roriety that S ( x + 1) = S( x) and 1 S( x + ) = 1 S( x) i.e, it is a eriodic function that is 2 similar under a translation of 1 and anti-similar under a translation of. Figure 4: Landmarks triangulation This comuted osition is affected by an intrinsic stereo camera error (see Fig 5) This error deends on camera-landmark distance and many other factors like resolution, focal length and base-length. Accordingly, the correction of the osition of the vehicle with this absolute localization, w.r.t. the relative osition

4 comuted with odometry, is accetable only if the delta correction is less than the error ossible at that distance. In many cases, just only one landmark could be visible. Therefore, we integrate this information with another measure comuted by searching linear atterns on the floor. assuming that some artificial straight lines of bright colour with directions arallel or orthogonal at the coordinate system are drown. The line are segmented in the homograhic z=0 lane of the W sace using a Sobel edge detector filter. After the edge detection, a simle Hough line detector is imlemented to extract the direction of each line. In a standard vehicle alication it's ossible to assume that the disarity decreases from bottom to the to image. In most of common situation objects lie to the floor and this assumtion ermits to restrict the minimum and maximum disarity allowed, deending to the y value of the image. The couled adotion of landmarks and lines detectors make the system articularly robust and flexible. During autonomous navigation, either lines, landmarks or both can be detected to rovide localization. 3.5 Disarity Ma Figure 5: Landmark osition error If only one landmark is visible, the value of the distance between the vehicle and the landmark is couled with the absolute angle observed by the line detector (described in sub-section 3.4) to estimate the osition of the vehicle. The chosen landmark have also a sort of bar code in the vertical art that allows the identification of the landmark. This is necessary when many landmarks are visible to have the correct matching. It could be used also in ma-building based navigation in order to identify the ath. 3.3 Homograhy The rocedure to remove the ersective effect resamle the original image, by remaing each ixel toward a different osition and roducing a new twodimensional array of ixel [16]. W = { (x,y,z) E 3 } I = { (u,v) E 2 } (4) The camera acquires the image defined in the I sace while the remaed image is defined as the Z = 0 lane of the 3D world sace W. The I W maing function roduces the homograhy as a 2D image that reresents a to view of the region in front of the vehicle, as it was observed from a significant height. 3.4 Line Detector In structured environment, such as for industrial alication, robot localization can be imroved with floor s line detection. Line detector is erformed by The Census algorithm is a stereo corresondence algorithm imlemented in our system: it is described in detail in [12]. The Census algorithm is chosen because it rovides a robust, fast a non-arametric dense disarity ma. Rather that arametrically comaring intensity value across images, the Census algorithm erforms a transform on the inut images based on intra-image comarison and then use the transformed images to determine the corresondence. The first ste of the algorithm is to aly the Census transform to the left and right image. Given these images, the best matching ixel P' for a given ixel P is determined to be one of the minimal accumulated Hamming distance between P and P'. In order to restrict the range of disarity value, the hyothesis that the maximum disarity decreases roortionally from bottom to to image is stated. This ermitted to reduce the noise of the disarity ma, or to have the same noise level using an half-resolution disarity ma. 3.6 Obstacle Detection In our aroach of obstacle detection, obstacles are defined as all those blobs sufficiently large that don't lie to the floor. To estimate the floor and remove it from the disarity ma, the information from the y direction is exloited: in the matter of fact, the disarity of the floor increases linearly. This consideration allows a labelling of increasing disarity and the floor segmentation. Segmented floor can be subtracted to the disarity ma to have only obstacles.. After a blob segmentation, it is ossible to assign a disarity and a deth to each object. For the nearest

After 3 cycles the angular difference is 17.1 degree and the osition difference is about 26 cm. Figure 7 : Comarison between corrected (left) and uncorrected (right) ath 4.

5 object on the trajectory of vehicle, the time to collision is comuted. Fig. 6 describes two examles of obstacles detected by the mobile robot. In Fig 7 is ossible to see the difference between the ath covered with the roosed system correction (left) and the ath covered using the common odometry osition system only (right) is indicated. After 3 cycles the angular difference is 17.1 degree and the osition difference is about 26 cm. Figure 7 : Comarison between corrected (left) and uncorrected (right) ath 4. Test Figure 6: Obstacle detection The roosed system was tested whit a Bumblebee Stereo camera mounted on a Pioneer 3DX robot, using a PC with a 3.2 GHz P4 Intel Processor. The Self-similar attern recognition algorithm of [11] has been modified, to allow the detection in wide range of scale. This is obtained by considering a search window of variable dimension. In this manner landmarks can be laced on floor as well as on the walls. This is very imortant in industrial environment where walls are not always visible. The code has been imroved using the SSE3 P4 Intel instruction set. The SSE3 otimization ermits a valuable comutational time reduction, or having the same time available as in [11], allows an imrovement in the global accuracy with a higher stereo resolution and a higher number of scanlines. With this imrovement, using a 640x480 image, the system finds correctly all atterns in a range of distance from 0.8 to 6 meters. This range deends on the camera lacement and the camera orientation: as shown in Fig.8 camera is mounted at less than a fifty centimetres to the floor. The obstacle detection system is been tested using some artificial obstacle laced on the vehicle test ath. The system finds correctly obstacle height more than 10 centimetres in a range of distance from 1 to 4 meters. Table 1 gives a measure of the comutational time required, in average for each ste of the navigation task. Here, only the comuter- vision stes are accounted. The majority of comutational costs are caused by the landmark detection: as reviously introduced, this art has been otimized for Intel architecture in order to limit the comutational time and to have a high recision. Comutational times are not strictly roortional to the ixel number: images with one quarter of resolution need about a half of comutational time only. Oeration (msec) Resolution 640x x240 1 Grab+Calibration Landmark recognition 3 Obstacle Detection* Line Detection Table 1: Time measures (in msec) * Obstacle detection is made at half resolution. To test the system, we have rogrammed the robot guide in a simle ath that the vehicle reeats three times: the ath is about 2x2 meters Correction is carried out with an association of the angle comuted with line detection and the detection of a single selfsimilar landmark.

Figure 8: The develoed mobile robot In the system the obstacle detection are interleaved with landmarks recognition (frame A) and line detector (frame B); the results frames rate are summarized in

6 Figure 8: The develoed mobile robot In the system the obstacle detection are interleaved with landmarks recognition (frame A) and line detector (frame B); the results frames rate are summarized in Table 2. The system rovides a fs of about 3fs and 5.5 fs for images of 640x480 and 320x240 resolution, resectively. Frame (fs) Resolution 640x x240 Frame A (1,2,3) Frame B (1,3,4) Conclusion Table 2: Frame rates In this work we have imlemented a working rototye of AGV guided by a standard PC for localization and obstacle detection. It is mainly used in ma-based autonomous navigation even if the same techniques can be used for absolute maless navigation. The rototye is tested for industrial alications, like the autonomous roduct transort. The roject has been funded by Regione Emilia Romagna, Italy PRIITT roject in collaboration with ItalVision srl. 6. References 1. L.Feng, J. Borenstein, H.R. Everett WHERE AM I? Sensor and Methods for Mobile Robot Positioning A.K. Peters Ltd, Wellesley 2. F. Chenavier e J. Crowley, 1992 Position Estimation for a Mobile Robot Using Vision and Odometry.Proceedings of the 1992 IEEE International Conference on Robotics and Automation 3. I. Cox., 1991 Blanche- An Exeriment in Guidance and Navigation of an Autonomous Mobile Robot IEEE Transaction on robotics and automation, Vol. 7 N.2 Aril B. U. Shoval, S. e E. Lenz, 1995 Control and Positioning of an AGV for Material Handling in an Industrial Environment. Manufacturing System Vol.25 N DeSouza G. and Kak A. C, 2002 Vision for mobile robot navigation: A survey. IEEE Trans. Pattern Anal. Mach. Intell. Vol. 24 No.2, February J. Crowley, World Modeling and Position Estimation for a Mobile Robot Using Ultrasonic Ranging. Robotics and Automation, Proceedings 7. Arsenio, A.; Ribeiro, M.I., 1998 Absolute localization of mobile robots using natural landmarks. Electronics, Circuits and Systems, 1998 IEEE International Conference 8. Yuen, D.C.K.; MacDonald, B.A.,2002 Natural landmark based localisation system using anoramic images Robotics and Automation. Proceedings. ICRA '02. IEEE International Conference 9. R Labayrade, D Aubert, JP Tarel 2002 Real Time Obstacle detection in stereovision on non flat road geometry through v-disarity reresentation.. Procs. IEEE Intelligent Vehicles Symosium A. Broggi. M. Bertozzi 1998 GOLD: A Parallel Real-Time Stereo Vision System for Generic Obstacle and Lane Detection. IEEE Transaction in Image Processing, Vol. 7, No.1 January Briggs, A.J.; Scharstein, D.; Braziunas, D.; Dima, C.; Wall, P.; 2000 Mobile robot navigation using self-similar landmarks Robotics and Automation, Proceedings. ICRA ' R. Zabih, J. Woodfill, May 1994 Non-arametric Local transforms for Comuting Visual Corrisondence, Proceedings of 3rd Euroena Conference on Comuter Vision 13. R. Tsai An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision. In Proc. of CVPR' M.Z. Brown, D. Burschka, and G.D. Hager, 2003 Advances in comutational stereo, IEEE Trans. on Pattern Analysis and Machine Intelligence 15. E. L. Hall D. W. Rosselot, 2004 Processing real-time stereo video for an autonomous robot using disarity mas and sensor fusion. Proceedings of SPIE Volume 5608 Intelligent Robots and Comuter Vision XXII: Algorithms, Techniques, and Active Vision 16. Newman, W.M.; Sroull, R.F Princiles of Interactive Comuter Grahics, 2d ed. McGraw-Hill, New York. 17. R. Talluri e J. Aggarwal, Position Estimation Techniques for an Autonomous Mobile Robot - a Review Handbook of attern recognition & comuter vision ag

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2 Mingliang Chen 1, Weiyao Lin 1*, Xiaozhen Zheng 2 1 Deartment of Electronic Engineering, Shanghai Jiao Tong University, China