An Embedded Calibration Stereovision System

Transcription

1 2012 Intelligent Vehicles Symposium Alcalá de Henares, Spain, June 3-7, 2012 An Embedded Stereovision System JIA Yunde and QIN Xiameng Abstract This paper describes an embedded calibration stereovision system that is able to be strongly online and autocalibrated without placing a calibration device in front of the stereo camera, but with the device hidden inside the cavity of the system via a half-mirror. The stereo camera simultaneously observes a scene passing through the half-mirror and the calibration device reflected from the half-mirror, and makes the formation of an embedded calibration stereo pair containing both scene and the calibration device. The features of the calibration patterns are extracted from the embedded calibration stereo pair to estimate the stereo camera s parameters. We use a polyhedral-mirror to generate multiple virtual images of the calibration device to occupy large part of a scene image for accurate estimation. We also use several mirrors to extend the optical path from the calibration device to the stereo camera for depth recovery of distance objects. The system can be easily used in a wide range of applications without considering variation of camera parameters. I. INTRODUCTION Stereovision system is an effective means of 3D scene understanding and obstacle avoidance of autonomous robotic vehicles,and has been traditional one of the most attractive topics in computer vision and robotics. Camera calibration is a necessary step of stereo vision to extract metric information from 2D images. Conventional camera calibration [1], [2], [3] has generally been done by a procedure whereby a known calibration device (whose geometry in 3D space is known with very good precision) is placed in front of a camera, and the imaged positions of features on this device have been used to determine the camera parameters. The advantages of conventional methods are the robustness to estimate camera parameters and the possibility to recover the absolute scale of a scene owing to having some explicit 3D information. Such a calibration must be done at the beginning of working session, and often requires human interaction to place and position the calibration device, and then remove it after calibration. In many applications, the vision system is changing due to mechanical and thermal variation, so such a one-off calibration method is just about useless. Currently in industrial robot vision, re-calibration was integrated into working with data available during normal use [4]. And in unmanned vehicle navigation, the calibration tool was taken with the vehicle for on-site re-calibrating its vision system at intervals. For example, in the planetary exploration, the calibration device was mounted on the vehicle for onsite stereo calibration [5]. In order to develop practical and efficient vision systems, many researchers continued to JIA Yunde and QIN Xiameng are with Beijing Lab of Intelligent Information Technology, School of Computer Science, Beijing Institute of Technology, Beijing , PR CHINA. jiayunde@bit.edu.cn, qxm0405@bit.edu.cn contribute much to design high efficient procedures of the conventional calibration. Zhang [6] and Bouguet [7] have developed popular toolboxes allowing others to implement calibration in their own research with well-user interface. But all these operations are labor intensive, cost expensive or halt in working. Moreover, the conventional method cannot be used in some applications where focusing and zooming of cameras are needed. Recently, Katayama [8] proposed a calibration method that uses a transparent calibration tool with patterns of dots of color filter. This tool is installed in front of the camera and not required resetting or removal to greatly simplify the calibration process. This is an online automatic robust calibration technique, but the accuracy of this method is not comparable with the conventional method because the calibration tool is located far from the plane of focus. Another problem is the size and location of the calibration tool which occupies the space of the field of view and makes the system not compact. In this paper, we propose an embedded calibration stereovision method that is able to strongly online auto-calibrate a stereovision system with a calibration device hidden inside the cavity of the system via a half-mirror, instead of placing the device in front of the system. The half-mirror can completely reflect the calibration device hidden inside the system to the stereo camera while allowing scene rays passing through it to the same camera. This means the stereo camera is able to simultaneously observe a scene and the calibration device, i.e. to form an embedded calibration stereo pair containing both scene and the calibration device. We can extract the features of the calibration patterns from the embedded calibration images to estimate camera intrinsic and extrinsic parameters, we can also use the same images to compute dense depth maps. In our system, a polyhedralmirror is used to generate multiple virtual images of the calibration device to occupy large part of images for estimating accurate camera parameters. And we use several mirrors to extend the distance from the calibration device to the camera for depth recovery of a distance object. The system can be easily used in a wide range of applications without considering variation of camera parameters. Another online auto-calibration method is the selfcalibration which has been explored intensively in last two decades [9], [10]. This method is only using a number of image correspondences to estimate camera parameters. But even under ideal conditions, when self-calibration methods are most reliable, they are routinely outperformed by the conventional calibration method [11] /$ IEEE 1072

2 Light Source Dense depth maps Stereo Computing Half- Scene Polyhedral- Fig. 1. The configuration of the embedded calibration stereo System. Half- Half- Fig. 2. A photograph of our system. II. THE SYSTEM OVERVIEW Fig. 1 shows the configuration of our embedded calibration stereo vision system which consists of a stereo head, a mirror group, a calibration device, a light source, and a stereo computing device. Fig. 2 is the system photograph. The stereo head comprises two cameras with fixed focal length lenses or zoom lenses. The calibration device is made of multiple planes, and each plane is covered with a chess-board pattern. The mirror group is composed of a half-mirror, a polyhedral-mirror and several mirrors to set up required the optical paths of the calibration device to the camera. The half-mirror is the fundamental element in the system which can be used to completely reflect the calibration device, hidden inside the system, to the stereo camera while allowing scene rays passing through it to the same camera. The stereo camera is able to simultaneously observe a scene and the calibration device to form an embedded calibration stereo pair, as shown in Fig. 3. In order to estimate more accurate camera parameters, the conventional method requires fabricating the calibration device as big as possible to occupy large part of images. Actually, the size of calibration device is ultimately limited, so most of researchers have to move the device manually in a wide area of interest or by using a robot arm to the different position of working space, and all data then are used to estimate camera parameters [6]. In our system, we use a polyhedral-mirror to generate multiple virtual images of the calibration device which can be projected to different regions in the field of view. Fig. 4 illustrates the formation of virtual calibration devices with a polyhedral-mirror. In practice, the calibration device has to be placed to the common visible area as well as the depth of field, and this distance is usually to be meters in front of the camera, e.g. 2 meters. We can use several mirrors in this system to fold this distance for reducing the system volume. The stereo computing device is an FPGA-based high performance computer developed in our lab. The device can simultaneously acquire images from the two color cameras, pre-process embedded calibration stereo pairs, estimate the camera parameters, and compute dense depth maps with a resolution of pixels and 32 disparity level in video rate (30pfs). The stereo computing device is developed based on our previous work [12], [13]. Different from the con- 1073

3 Fig. 3. Virtual An embedded calibration stereo pair. Virtual Virtual from an embedded calibration image is a challengeable task because the calibration process is online and full automatic, and the calibration device is embedded into a scene which might be complicated. Fortunately, in our system the virtual positions of the calibration device in stereo pairs are not changed, and we can set up a prior set of features with accurate coordinates as the initial values of online autocalibration. Before online auto-calibration, we use PCA (Principle Component Analysis) to train the positions of calibration planes and the four initialized corners of the chess-board in each plane. During calibration, the homography matrix is first computed from these four corners, and then the positions of the rest of corners are calculated based on projective transformation as the initialized value of iteration. The new positions of all calibration corners are detected by using Harris detector in the small windows around the initialized corners to update the homography matrix. Thus, we can iteratively detect all the corners with accurate coordinates. Fig.5 shows the positions of corners in calibration planes. Polyhedral- Fig. 4. The formation of virtual calibration device with a polyhedral-mirror. ventional stereovision system, our system can automatically work well when camera parameters are changed, and also run well without any human interaction. Moreover, our system is able to recover stereo pairs with zooming lenses. Indeed, it is a real zoom stereo camera. III. CAMERA CALIBRATION The system is able to perform camera calibration and depth recovery by using the same embedded calibration stereo pair. We follow the traditional stereo vision method to realize camera calibration first and then computing the corresponding depth map. A. pattern extraction The features of the calibration patterns must be extracted robustly and accurately for estimating accurate camera parameters. Conventional methods use simple background for ease of feature detection or often require human interaction for feature detection initialization, such as best-know calibration techniques [6], [7] available on the Internet need to manually point the four chess-board corners on all stereo pairs. In our system, the feature detection of the calibration patterns B. model The pixel location of a feature is denoted by m = (u,v) T, and the corresponding 3D point is denoted by M = (X,Y,Z) T. We use m = (u,v,1) T and M = (X,Y,Z,1) T to represent their homogeneous coordinates. A pinhole camera model is used and the projective relationship between m and M is given by s m = α γ u 0 0 β v 0 [ R t ] M = A [ R t ] M, (1) where α,β,γ,u 0,v 0 are the camera intrinsic parameters and A is intrinsic matrix. R and t are the extrinsic parameters, which are the rotation and translation between the world coordinates and the the camera coordinate system. Suppose an m-polyhedral-mirror is used to generate m virtual images of the calibration device in an embedded calibration stereo pair. Given n features on each calibration pattern image and corresponding 3D coordinates. The maximum likelihood estimate can be obtained by minimizing the following functional [6]: m n i=1 j=1 m i j ˆm(A 0,k i1,k i2,r i,t i,m j ) 2, (2) where ˆm(A 0,k i1,k i2,r i,t i,m j ) is a corresponding image point of 3D point M j, and k i1,k i2 are distortion factors. C. Image rectification The perspective projection matrix for image rectification [14], [15] is used in our system. A new coordinate system, a 3D affine coordinate system, is established for making the epipolar lines parallel to the scan line of images. The new X axis and Y axis are parallel to the baseline of the stereo camera and the intersecting line of the two imaging planes respectively, and the new Z axis is perpendicular to the XY plane. 1074

4 lighting the calibration device (e.g. conventional stereo pair), and Fig. 6(c) is the dense depth mapping from Fig. 6(b) only using a simple SAD matching method. In most applications, such as robot navigation and auto machinery, the focal length of lenses is usually fixed and the calibrated system should keep working well for a long time. Thus, our system can work in calibration mode with lighting the calibration device for system calibration, and depth recovery mode without lighting the calibration device for dense depth mapping of scenes, as conventional stereo system. Fig. 5. The close-up of features extracted from embedded calibration images in Fig. 3, in which red crosses are corners detected by Harris detector and blue crosses are refining results after iteration. It is easy to infer the original camera model based on the original intrinsic parameter matrix A and rotation matrix R: λ [ c r 1 ] T = AR(X T), (3) where c and r are column and row pixel coordinates in the original camera coordinate system. By using the new camera parameters, the rectified camera model is given by λ [ c r 1 ] T = A R (X T), (4) where A and R are the rectified intrinsic parameter matrix and rotation matrix, respectively, and c and r are column and row pixel coordinate in the rectified camera coordinate system. Combining (3) and (4), we have the rectification transform equation: λ [ c r 1 ] T = A R R 1 A 1 [ c r 1 ] T. (5) IV. EXPERIMENTS Our system uses two FFMV-03MTC Color Cameras (Point Gray Inc.) with U-Tron 4mm lenses to simultaneously acquire stereo pairs, as shown in Fig. 2. The calibration device is composed of three planes with different orientations and each plane is covered with an chess-board with a grid size of 1cm 1cm. We used a polyhedral-mirror with only two planar mirrors to form two mirror images (two virtual calibration devices) of the calibration device. We also used another mirror together with the polyhedral-mirror to extend the optical path of the calibration device to the stereo camera up to 1.2 meters. We tested our system in real scenes and get good results. Fig. 6(a) demonstrates an example of embedded calibration stereo pair with lighting the calibration device for system calibration. Fig. 6(b) is an example of stereo pair without A. Dense depth recovery from embedded calibration stereo pairs In object tracking and gazing areas, Stereo cameras with zooming lenses are often used, such as humanoid robots and surveillance systems. In this case, the system has to work in calibration mode high frequently with lighting the calibration device. Thus, the dense depth maps recovers directly from the embedded calibration stereo pairs is an interesting task. But a challenging problem is to separate the calibration device from embedded calibration stereo pairs. Some techniques [16], [17] have been proposed to separate the reflection components from images. We only employ a simply physical method to solve this problem and to demonstrate the feasibility. This method takes account of the cumulating of charges in imaging sensor which is related to the illumination intensity and illumination time (exposure time). Let I E denote an embedded calibration image and I C an image only containing the calibration object. The model of image separation is I S = I E αi C, where I S is the separated scene image, α is a coefficient and defined as α = F (mean(i E I C )), where mean( ) represents the mean gray. F( ) is a cubic function that is fitted by Least Square method. Fig. 7(a) shows the stereo pair after separation of the calibration device from the embedded calibration stereo pairs in Fig. 6(a). We can see that the separated image is very much resembling to the real scene image(fig. 6(b)). Fig. 7(b) is the dense depth map that recovers from the stereo pair of Fig. 7(a). As can be seen, the result is acceptable and comparable with the result from the conventional stereo pair(fig. 6(c)). V. CONCLUSIONS This paper has presented an embedded calibration stereo vision method that is able to strongly calibrate a camera 1075

5 (a) (a) (b) (b) Fig. 7. Depth map recovery from an embedded calibration stereo pair. (a) Gray stereo pair after separation of the embedded calibration device from the images of Fig. 6(a). (b) Dense depth map from the stereo pair. used in a wide range of applications without considering changeable parameters of cameras and zooming lenses. It is a real a complete standalone machine and can be easily used by any person out of the field. (c) VI. ACKNOWLEDGMENTS Fig. 6. An example of dense depth maps from our stereovision system with an embedded calibration device. (a) Stereo pair with lighting the embedded calibration device for camera calibration. (b) Stereo pair without lighting the embedded calibration device for 3D scene recovery. (c) Dense depth map computed from (b). This work was supported in part by the 973 Program of China under Grant No. 2012CB and Natural Science Foundation of China (NSFC) under Grant No R EFERENCES without placing a calibration device in front of the camera. We used a half-mirror to completely reflect the calibration device hidden inside the system to a camera while allowing scene rays passing through it. The stereo camera is able to simultaneously observe a scene and the calibration device to form an embedded calibration stereo pair. A polyhedralmirror is used to generate multiple virtual images of the calibration device to occupy large part of images for estimating accurate camera parameters, and several mirrors are used to extend the optical path from the calibration device to the camera for depth recovery of distance objects. The embedded calibration stereo vision system can be easily 1076 [1] Olivier Faugeras and Giorgio Toscani, The calibration problem for stereo, In Proceedings of the International Conference on ComputerVision and Pattern Recognition, Miami Beach, FL, 1986, pp [2] R.Y. Tsai, An Efficient and Accurate Camera Technique for 3D Machine Vision, Proceedings of the International Conference on ComputerVision and Pattern Recognition, Miami Beach, FL, 1986, pp [3] J. Weng and P. Cohen and M. Herniou, Camera calibration with distortion models and accuracy evaluation, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, 1992, pp [4] N.A. Thacker and J.E.W.Mayhew, Optimal Combination of Stereo Camera from Arbitrary Stereo Images, Image and vision computing, vol. 9, 1991, pp [5] Zhengyou Zhang, A stereovision system for a planetary rover: calibration, correlation, registration, and fusion, Machine Vision and Applications, vol. 10, 1997, pp

6 [6] Zhang Zhengyou, A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, 1961, pp [7] J.Y. Bouguet, Camera Toolbox for Matlab, [8] Y. Katayama, Camera with a Transparent Tool Using Color Filters: Application to for a Distant Object, in Eighteenth International Conference on Pattern Recognition, 2006, pp [9] O. Faugeras, 3D Computer Vision, MIT Press, [10] R. Hartley and A. Zisserman, Multiple View Geometry in computer vision, Cambridge University Press, [11] S. Bougnoux, From projective to Euclidean space under any practical situation, a criticism of self-calibration, in Sixth International Conference on Computer Vision, 1998, pp [12] Chen Lei and Jia Yunde and Li Mingxiang, An FPGA-Based RGBD Imager, Machine Vision and Applications, Machine Vision and Applications,Vol. 23, no. 3, 2012, Page [13] Yunde Jia and Xiaoxun Zhang and Mingxiang Li and Luping An, A miniature stereo vision machine (MSVM-III) for dense disparity mapping, in Seventeenth International Conference on Pattern Recognition, 2004, pp [14] N. Ayache, and C. Hansen, Rectification of images for binocular and trinocular stereovision, in Ninth International Conference on Pattern Recognition, vol. 1, 1988, pp [15] A. Fusiello and E. Trucco and A. Verri, A compact algorithm for rectification of stereo pairs, Machine Vision and Applications, vol. 12, 2000, pp [16] R. Szeliksi and S. Avidan and P. Anandan, Layer extraction from multiple images containing reflections and transparency, Computer Vision and Pattern Recognition, 2000, pp [17] A. Levin and Y. Weiss, User assisted separation of reflections from a single image using a sparsity prior, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 9, 2007, pp