DEPTH ESTIMATION USING STEREO FISH-EYE LENSES

Transcription

1 DEPTH ESTMATON USNG STEREO FSH-EYE LENSES Shishir Shah and J. K. Aggamal Computer and Vision Research Center Department of Electrical and Computer Engineering, ENS 520 The University of Texas At Austin Austin, TX ABSTRACT This paper presents the estimation of depth in an indoor, structured environment based on a stereo setup consisting of two fish-eye lenses, with parallel optical axes, mounted on a robot platform. The use of fish-eye lenses provides for a large field of view to estimate better the depth of features very close to the lens. To extract significant information from the fish-eye lens images, we first correct for the distortion before using a special line detector, based on vanishing points, to extract significant features. We use a relaxation procedure to achieve correspondence between features in the left and right images. The process of prediction and recursive verification of the hypotheses is utilized to find a one-to-one correspondence. Experimental results obtained on several stereo images are presented, and an accuracy analysis is performed. Further, the algorithm is tested using a pair of wide-angle lenses, and the accuracy and difference in the spatial information obtained are compared. 1. NTRODUCTON Stereopsis is a common technique used for passive computation of range information about a scene. The objective of scene analysis by stereopsis is to recover the threedimensional (3D) location of features from their projection in two-dimensional (2D) images. The central problem in realizing this objective is the ability to accurately determine the correspondence between the left and right images. Researchers have proposed various solutions to this problem. A common stereo system includes a nonconvergent stereo imaging system with two fixed cameras, separated by some baseline distance, and having parallel optical axes whose parameters are calculated in the calibration stage. With this setup, a left and a right image are obtained. By computing the displacement, or disparity, between two corresponding features in the left and right image, the 3D coordinates of an image point in the scene are found. The computational stereo paradigm then consists of three major steps: stereo camera modeling, feature detection, and matching. The major problem lies in establishing the correspondence between the two stereo images. Generally, two broad classes of techniques have been used: feature-based and area-based. This research wa8 supported in part by the DoD Joint Services Electronic Program through the Air Force Office of Scientific Research (AFSC) Contract F C-0027, and in part by the Army Research Ofice under contract DAAL03-91-G-0050 Featurebased solutions employ simple, geometric primitives such as line segments and planar patches [l]. Such models are appropriate for simple, well-structured environments consisting of man-made objects. These techniques have generally been more successful overall as the matching process is much faster than the area-based techniques and there are fewer feature points to be considered [2]. The area-based algorithms represent depth at each pixel in the image. These techniques promise more generality; however, much remains to be done on both the mathematical and system aspects of this approach [3, 41. So far, many techniques have proved to be unsatisfactory as they produce poorly defined matches, and thus it becomes difficult to de termine when a match has been established. They are also highly sensitive to distortion in gray level and geometry, and are computationally very expensive. A survey of stereo algorithms is found in [5]. Various techniques that have been used include dynamic programming [6] and relaxation methods [2, 7, 8, 91. n this paper we present a stereo vision procedure for the guidance and navigation of autonomous mobile robots. We use a parallel stereo geometry consisting of two fish-eye lenses mounted on a robot platform. Fish-eye lenses provide a large field of view and are important in applications where an accurate estimate of the distance of features very close to the lens is crucial. This would not be possible using a normal lens. Figure 1 contrasts a scene as imaged by the fish-eye lens, on the left, and as imaged by a wide-angle lens, on the right, from the same position. Distortion seen in the fish-eye lens image has to be corrected before linear edge segments can be detected. n this paper, we discuss methods for distortion correction of fish-eye lens images, segmentation, and stereo correspondence to estimate the spatial information of a scene. This paper has been organized in the following five sections: Section 2 briefly describes the stereo vision setup and geometry used. Section 3 discusses the algorithms for distortion correction, segmentation, and stereo correspondence. Section 4 presents the experimental results obtained on several pairs of images, and Section 5 discusses the accuracy recorded in estimating the depth of detected features in the scene. Also, we present a comparison between the spatial information obtained by using the fish-eye lenses and the wideangle lenses. Finally, Section 6 summarizes the key points of this paper and offers important conclusions /94 $ EEE 740

2 3. MAGE ANALYSS n the stereo setup described above, a pair of fish-eye lens cameras with a calculated focal length of 3.8- are used. The baseline distance is set to be 398 and the parallel optical axes geometry is calibrated. The procedure followed for calculating the spatial information from the given setup is presented in the following sections. Figure 1: (a)fish-eye and (b)wideangle lens images Figure 2: Stereo setup 2. STEREO SYSTEM n an image, a detected 2D feature is the perspective projection of a 3D feature in the scene. A number of 3D points can project onto the same 2D point, which results in the loss of depth information. To recover this lost information, two images taken from different perspectives are required. The first step involved in recovering 3D spatial information is to establish the perspective transformation relationship between the scene and its projection onto the left and right images. As shown in Figure 2, a point P defined by its coordinates (z, y, z) in the real world, will project as the corresponding 2D image coordinates (zr, yr) and (zr, yr) onto the left and right images respectively. The two cameras are separated by a fixed baseline distance D and have a known focal length f. Considering 0 to be the origin, which coincides with the image center in the left camera, then the perspective projections can be defined through simple algebra. x = xr.d/d (1) where d is defined as the disparity between the two corresponding features in the left and right images: d = (151 - xr (4) These equations provide the basis for deriving the 3D depth from stereo images Distortion Correction n order to take advantage of the large field of view of the fish-eye lens, the inherent distortion has to be corrected so that we can use the data with minimal loss of information. The images obtained exhibit a combination of radial and tangential distortion, and to remove this distortion, we have used a polynomial transform that is a non-linear mapping of points in the image plane. We use an inverse mapping technique to recover a complete gray-scale image. The result is an undistorted image with minimal loss of information. The corrected image results in blank areas in the vertical direction due to the larger field of view in the diagonal direction. Detailed procedure for calibration and distortion correction is given in [lo]. The distortion correction is achieved by determining a mapping of points in the world coordinate system and their corresponding point locations in the image plane. A fifth order polynomial is employed. The coefficients of the polynomial are determined using a Lagrangian minimization technique. Figure 3 shows an image (distorted) acquired with the fish-eye lens and the corrected (distortion removed) image Segmentation mportant line segments are extracted from the images by using a line detector based on vanishing points. The vanishing points are determined from the heading information obtained from the calibration parameters, and are updated based on the horizontal lines in the image. As the goal is to navigate through a structured environment, we look for lines only in three directions: horizontal lines, vertical lines, and lines going into the image plane. The lines going into the image plane are not very significant for the process of correspondence, as the exact beginning and ending of such lines is not known, making 3D recovery difficult. n our calculations, we have ignored these lines, but they can prove to be important in reconstructing a model of the scene or object. For further details, refer to [ll] and [12]. The detected lines are grouped according to their orientations and can be processed separately. Figure 4 shows the segmented images Stereo Correspondence Having grouped the detected lines according to their orientations, the correspondence problem is greatly simplified. As the parallel stereo geometry is used, the depth extraction process is based on triangulation, which is the determination of the 3D point from the intersection of two rays. This procedure makes use of the parallax, which is the displacement of the perspective projection of a point due to 741

3 Figure 5: Disparity gradient Figure 3: (a)distorted and (b)undistorted image pair i J. _-- Figure 4: Detected segments in an image translational change in the position of the lens (131. To get depth information for navigation purposes, it is important to know the depths of the vertical and the horizontal lines in the scene. We consider only two orientations of lines for the matching algorithm, as they provide us with the most significant information. The horizontal lines provide important information when a change in motion direction has to be made by the robot. We are not looking for accurate depth information from horizontal matches, but only a guideline towards making the transition in direction. The correspondence within the two images is achieved by associating weights to the probable matches, which are calculated on the basis of disparity estimate, length estimate, edge intensity value, and the disparity gradient value. The feature with the largest weight value is chosen as the corresponding match. Estimating the disparity value prior to the matching process can lead to a higher speed in feature matching by limiting the search area to only a certain section of the image. n order to achieve this, the baseline length of the lenses is calculated from the hardware setup and the lens calibration. The maximum search area in the stereo algorithm, which is the maximum disparity that can be encountered, is calculated as dispmaz = D.f/Zm,n, (5) where D is the calculated baseline, f is the focal length, and Zmin is the closest possible depth that can be detected. Further, the area is also restricted based on the epipolar line constraint. As the orientation of each line is known, we can isolate the search to lines of similar orientation for the purpose of stereo matching. We use the method of iterative search in our stereo algorithm. Based on the value of maximum disparity, the search is iterated until a match is established. Knowing the end points, we can easily calculate the length of each line. t is very likely that lines in both the left and right images will have almost the same length. Thus the search is also weighted according to lengths recorded for each match. The intensity value around the edge point of the lines is also used to associate weights to each correspondng match. An eight neighborhood of pixels around the edge point of the line is chosen and a summed intensity value is calculated. This is done as: n n i j where a,j is the edge pixel. This process is repeated for the edge of lines in the other image. A value is associated to the absolute difference of the intensity values between the two lines as: ; = llli - Till (7) where ; is the intensity value, 1; and T; are the edge intensities of the left and right line respectively. Weight is associated to the matched pair according to the absolute intensity difference value and the minimum valued pair is chosen as a probable match. This process is repeated for each line in the database and weights are associated with each match. The final criteria for associating weights is based on the disparity gradient between two lines. Of the probable matches, two lines near to each other are selected, as shown in Figure 5, and one edge of each line is taken into account for each iteration. The disparity gradient value is then calculated based on the following equations. At, Z31, A,, and E, are the edge points of the selected lines and the coordinates of each point is given as: A = (az~, ~yl); Ar = (arr, agr) (8) B = (bz1, bgl); Br = (bzr, by?) (9) The average coordinates of the two edge points between the two lines is given by: A = (a,[ + azr)/21 (ayl + ayr)/2 (10) B = (6,1+ 6,,)/2, (b,~ + byr)/2 (11) Now a separation value is calculated according to: S(A,B) = d G (12) 742

4 ~~ where; The disparity gradient is calculated as the ratio of the disparity between the two edge points and the calculated separation value. This is given by: The spatial information for the segments was calculated and then compared to information previously obtained using the fish-eye lens stereo setup. Further, we compared the estimates of segments in two regions. We considered lines which were closer than 4.3 meters and then those further away. The break up in distance is based on the calculated maximun distance that a lens would see based on our setup, and is given by the equation: Z = f.d/dmin The minimum valued match is associated with the greatest weight. The overall criteria for a match is set by examining the sum of the weights associated with each probable match. The match with the highest weight is then chosen as the correct match. The overall aim of the algorithm is to minimize the difference in the endpoints of the line in the left and right images and then reinforce the match by minimizing the difference in the length of each line segment, edge gradient, and disparity gradient. Thus, the approach of prediction and recursive verification of the hypothesis is used [14]. We have found that this procedure results in 98% true matches. Based on the matches we can then successfully calculate the depth of each segment and construct a map. where f is the focal length, D is the baseline distance, and dmin is the minimum disparity observed in the image pair. The results of the comparision are shown in Table 2. We have found that lines closer to the lens can be estimated with higher accuracy by using the fish-eye lens while lines further away are better estimated with the wide-angle lens. 4. RESULTS The algorithm was tested on several sets of images. A left and right image pair as imaged through the fish-eye lens camera is shown in Figure 6. Using the distortion correction algorithm, the images were corrected, and based on the calculated vanishing points, the significant line segments were detected as shown in Figure 7. Stereo correspondence was achieved and matched line segments from the left and right pair were extracted as shown in Figure 8. Based on the triangulation process, the three-dimensional information is recovered by projecting the two-dimensional line segments to their respective spatial coordinates. This can be seen in Figure 9(a). Lines have been recorded according to their spatial coordinates and are joined by lines going into the image plane. Lines at the same depth have been joined by horizontal lines. n order to determine the accuracy of the estimated depth for each line segment, we physically measured the depths of over 100 detected line segments in several image pairs. The average of these measurements is shown in Table 1 and the average error is also presented. 5. ACCURACY Results obtained by using fish-eye lens and those obtained by using wide-angle lens were compared to establish the advantage of using fish-eye lenses for autonomous navigation. A similar stereo setup was achieved using a pair of wide-angle lenses. The difference in the information perceived has already been shown. A pair of images was acquired from the same position as the fish-eye lens. Once again the significant lines were detected and the stereo correspondence established. The reconstructed model is shown in Figure 9(b). For further details refer to [15]. Figure 6: Left and right image pair Figure 7: Segmented left and right image pair Figure 8: Matched left and right image pair 743

5 Segments: Vertical Horizontal Total (a) Figure 9: Spatial maps; (a)fish-eye & (b)wide-angle lens Dist.:(m) Less than 4.3 Detected line classification 1 Average Distance Estimation( ) # Estimated Measured Error % % % Table 1: Estimated distance distribution 8. CONCLUSON n this paper we have presented a simple and effective procedure for estimating depth in a structured, indoor scene using a parallel stereo setup with two fish-eye lenses. The procedure includes the distortion correction of fish-eye lens images, segmentation procedure based on vanishing points which enables the system to extract only the most significant line segments, and, finally, a correspondence algorithm which is based on a recursive hypotheses formulation and verification procedure. Finally, the triangulation process is used to estimate the depth of each line segment in the scene. The system is successfully implemented on real scene images and the accuracy determined. A comparison is made between the use of fish-eye lenses and a similar setup of wide-angle lenses. The viewing range of the lenses is calculated based on the hardware setup. Accuracy in estimating the depth of segments near and far from the lens is evaluated. t is thus established that the fish-eye lens provides more information and higher accuracy when the application requires spatial perception of objects close to the lens. More than 4.3 Average Distance Estimated s Company, Segments Estimated( ) Measured( ) ~.,, Error 8.7 % % Segments Estimated( ) Measured( ) , Error 12.9 % 9.1 % Table 2: Lens error distribution 7. REFERENCES W. E. L. Grimson. Computational Experiments with a Feature Based Stereo Algorithm. EEE Transactions on Pattern Analysis and Machine ntelligence, vol. 7, 17-34, 1985 G. Medioni and R. Nevatia. Segment-based Stereo Matching. Computer Vision, Graphics, and mage Processing, 2-18, 1985 S. T. Barnard. Stochastic Stereo Matching over Scale. nternational Journal of Computer Vision, vol. 3, 17-31, L. H. Matthies, R. Szeliski and T. Kanade. Kalman Filter-based Algorithms for Estimating Depth from mage Sequences. nternational Journal of Computer Vision, vol. 3, , Umesh R. Dhond and J. K. Aggarwal. Structure from Stereo-A Review. EEE Transactions on Systems, Man, and Cybernetics, vol. 19, , H. H. Baker and T. 0. Binford. Depth from Edge and ntensity Based Stereo. Proc. 7th nt. Joint Conf. on Artificial ntelligence, , S. B. Marapane and M. M. Trivedi. Edge Segment Based Stereo Analysis. SPE, Applications of Artificial ntelligence V, vol. 1293, , S. T. Barnard and W. B. Thompson. Disparity Analysis of mages. EEE Transactions on Pattern Analysis and Machine ntelligence, vol. 2, , Y. C. Kim and J. K. Aggarwal. Positioning Three- Dimensional Objects Using Stereo mages. EEE Journal of Robotics and Automation, vol. 3, , Shishir Shah and J. K. Aggarwal. A Simple Calibration Procedure for Fish-Eye (High Distortion) Lens Camera. Proc. EEE nt. Conf. Robotics and Automation, , X. Lebbgue and J. K. Aggarwal. Extraction and nterpretation of Semantically Significant Line Segments for a Mobile Robot. Proc. EEE nt. Conf. Robotics and Automation, , X. Lebbgue and J. K. Aggarwal. Detecting 3-D Parallel Lines for Perceptual Organization. Proc. Second European Conf. on Computer Vision, , R. M. Haralick and L. G. Shapiro. Computer and Robot Vision. Volume 11. Addison-Wesley Publishing N. Ayache and B. Faverjon. A Fast Stereo Vision Matcher based on Prediction and Recursive Verification of Hypotheses. Proc. of the Third Workshop on Computer Vision: Representation and Control, 27-37, Shishir Shah and J. K. Aggarwal. Segment-based Stereo Correspondence using Fish-Eye Lens. Technical Report: Computer and Vision Research Center, TR ,