Feature-Based Image Mosaicing

Transcription

1 Systems and Computers in Japan, Vol. 31, No. 7, 2000 Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J82-D-II, No. 10, October 1999, pp Feature-Based Image Mosaicing Naoki Chiba and Hiroshi Kano Mechatronics Research Center, Sanyo Electric Co., Ltd., Hirakata, Japan Michihiko Minoh Center for Information and Multimedia Studies, Kyoto University, Kyoto, Japan Masashi Yasuda Mechatronics Research Center, Sanyo Electric Co., Ltd., Hirakata, Japan SUMMARY We propose an automatic image mosaicing method which can construct a panoramic image from a collection of digital still images. Our method is fast and robust enough to process a nonplanar scene with unrestricted camera motion. The method includes the following two techniques. First, we use a multiresolution patch-based optical flow estimation for making feature correspondences to obtain a homography automatically. This enabled us to transform images fast and accurately. Second, we have developed a technique to obtain a homography from only three points, instead of four in general, in order to divide a scene into triangles. Experiments using real images confirm the effectiveness of our method Scripta Technica, Syst Comp Jpn, 31(7): 1 9, Introduction Image mosaicing has become an active area of research in the fields of photogrammetry, computer vision, image processing, and computer graphics. Traditional applications include construction of aerial and satellite photographs. Recently, creation of virtual environments has been collecting attention [2, 9]. This is an interactive system which can provide a novel view from panoramic image mosaics. The other applications include video surveillance and image compression. However, previous methods have limitations regarding camera motion or the geometric properties of shooting scenes. On the other hand, in the research area of motion analysis, optical flow estimation techniques have been intensively investigated. Recently, a reliable optical flow technique has been introduced for 3D shape recovery [3]. In the area of stereo camera calibration, a good method has been introduced to obtain the geometric relationship, the epipolar constraints, between two cameras [12]. We propose a feature-based method which is fast and can deal with nonplanar scenes with unrestricted camera motion. The camera motion includes not only rotations such as panning or tilting, but also translations. As for the geometric property of shooting scenes, we consider not only planar scenes but also arbitrary depth scenes which are common indoors. Our method consists of the following three steps. First, it makes feature correspondences between consecutive images by using optical flow estimation. Second, it Scripta Technica

2 computes a geometric transformation when the shooting scene can be regarded as planar. Third, when we cannot regard the scene as a planar surface, it divides the scene into several triangles, by using the epipolar constraints between images to obtain each transformation for each triangle. Then we transform images with the transformations. The remainder of this paper is structured as follows. After reviewing related work in Section 2, we show how to apply the Lucas Kanade optical flow estimation to image mosaicing in Section 3. Section 4 describes a technique of image mosaicing when the scene is not planar. Section 5 presents our experimental results using real scene images. We close with a discussion and future work. 2. Related Work One of the previous methods uses a cylindrical projection for image mosaicing [2]. However, this method has the following two problems. First, it needs a tripod to restrict the camera motion to be horizontal. Second, the focal length of the camera must be known in advance. Recently, several methods have been introduced to solve these problems by using planar projective transformation. Given two images taken from the same viewpoint, or images of a planar scene taken from different viewpoints, the relationship between the images can be described by a linear projective transformation called a homography. Once we obtain a homography between two images, we can construct a panoramic image by transferring one image to the other with the homography matrix. Szeliski has introduced a method which can obtain the optical flow vectors and the homography between images by using a nonlinear minimization technique [8]. Zoghlami and colleagues have introduced a method which is based on random sampling framework to obtain both feature correspondence and a homography at the same time [13]. However, these methods have the following two problems. First, their computations are expensive. Szeliski s method has a tendency to fall into a local minimum, since it is based on a nonlinear minimization framework. To deal with nonplanar scenes, several methods have been introduced. However, these methods have the following problems. Their computational cost is expensive. There is instability in obtaining solutions since the methods are based on nonlinear minimization framework with many parameters. One of the reasons for these problems is that the methods do not use feature correspondence to obtain transformation, but they try to obtain the feature correspondence and transformation at the same time. We solve this problem by using feature correspondence. 3. Feature-Based Image Registration 3.1. Optical flow estimation The Lucas Kanade method is one of the best regarding optical flow estimation, because it is fast to compute, easy to implement, and controllable because of the availability of tracking confidence [1]. The major drawback of the gradient methods, including the Lucas Kanade method, is that they cannot deal with large motion. Coarse-to-fine multiresolution strategy has been applied to overcome this problem. A major problem, however, still remains. Low-textured regions have unreliable results. We solved this problem in Ref. 3, with a dilation-based filling technique after thresholding unreliable estimates, at each pyramid level Overlap extraction We need to extract the overlapping region before applying the optical flow estimation, when the difference between images is large. This is because optical flow estimation may fail, even though our flow estimation incorporates a hierarchical multiresolution technique. We find a rough displacement by maximizing the score of the normalized cross correlation, in the overlapping region, defined by the following equation: where C is the cross correlation coefficient of an overlapping region w (size: B M u N between images I 1 and I 2 having displacement d. Ii and V i are the mean and the variance of the overlapping part, respectively, in each image. We make low-resolution images for faster computation, because the following optical flow estimation will obtain finer displacement. Figure 1 shows the original images and Fig. 2 the extracted overlapping region. Fig. 1. Original images. (1) 2

3 Fig. 2. Overlap extraction. When the scene is a planar surface or when the images are taken from the same point of view, images are related by a linear projective transformation called a homography. Figure 4 illustrates the principal of the planar projective transformation. When we see a point M on a planar surface from two different viewpoints C 1 and C 2, we can transform the image coordinates m 1 = x 1, y 1, 1 t to m 2 = x 2, y 2, 1 t using the following 3 u 3 planar projective transformation matrix H [5]: km 2 = Hm 1 (2) where k is an arbitrary scale factor and image coordinates m 1 and m 2 are represented by homogeneous coordinates. This relationship can be rewritten using the following equations: 3.3. Feature correspondence After obtaining the overlapping region of the two images, we first select prominent features in the first image from their image derivatives and the Hessians [11]. This method finds small rectangular regions such as corners automatically. Next, we estimate sparse optical flow vectors for small rectangular patches such as 13 u 13 pixels by using the improved Lucas Kanade method described in Section 3.1. To obtain the point correspondence, we interpolate the results of nearest four patch bilinearly in subpixel order. We discard false correspondences by measuring the cross correlation coefficients. Figure 3 shows an example of the selected and corresponding features Planar projective transformation We solve a homography from feature correspondences. We can solve this homography from four pairs of feature correspondences, because one pair of feature correspondence provides two equations for eight unknown parameters. If we have more than four pairs of feature correspondences, we can use a least-squares method for obtaining a homography Homography-based image mosaicing By using planar projective transformation, we can extend the image plane virtually. The dotted line in Fig. 4 shows an extended region where it is invisible from viewpoint C 1, but visible from viewpoint C 2. Pixels in this area at viewpoint C 1 can be obtained by transferring pixels from C 2 with the homography between the two images. Extend- (3) Fig. 3. Feature correspondence. 3

4 4.1. Piecewise homography computation We need at least four points to obtain a homography since it has eight independent parameters. (It is a 3 u 3 matrix and is invariant to scaling.) We describe a novel technique to compute a homography from three points using the epipolar constraint. The epipolar constraint between two images is described by the fundamental matrix F, with a point m on an image I and the corresponding point mc on the other image Ic as follows: Fig. 4. Planar projective transformation. ing image plane corresponds to placing a wide-view lens whose focal length is short. We blend pixel intensities in the overlapping region. Averaging the intensities in that region causes a seam due to the brightness difference between the two images. Therefore, we blend intensities with the weights depending on the distance to each image. When we have three or more images, we cascade transformation iteratively. We select the middle image as a reference image by judging from the rough alignment by using normalized correlation described in Section 3.2. We obtain the transformation for each image to the reference image, after obtaining those between consecutive images. We can obtain a transformation to the reference image with the following equation, even though the image does not have any overlapping portion with the reference image: 4. Nonplanar Scene We propose a method that can construct a panorama from nonplanar, arbitrary depth scenes with unrestricted camera motion. If the assumption for homographies is not satisfied, that is, taking images of a nonplanar, arbitrary scene from different viewpoints, there will be misregistration caused by motion parallax. Figure 9 (left) shows an example. We propose a feature based method which triangulates a scene with corresponding feature points between images for obtaining the homography for each triangle. We use the Delaunay triangulation method. This is a wellknown triangulation method which is based on Voronoi diagrams [5]. (4) Recently a good method has been developed to obtain a fundamental matrix between uncalibrated cameras [12]. Using it, we first obtain the fundamental matrix between images from their corresponding points. By substituting Eq. (2) into Eq, (5), we have Since H T F means the outer product of vector m, it should be skew symmetric: We obtain 6 equations from Eq. (7), because diagonal elements and the additions of skew symmetric elements should be 0. We already have 6 equations from three sets of corresponding points with Eq. (2). We can compute a homography by least squares with three points and the fundamental matrix, because a total of 12 equations are available for eight unknown parameters Homography test We can determine whether we should use a single homography or divide the scene into several triangles by the following test. We measure the reprojection error D by using a homography with the following equation: where p i is a coordinate of a feature point, p i g is a coordinate of the reprojected feature point by the homography, and N is the number of feature points. If D is larger than a predefined threshold, we divide the scene into triangles. (5) (6) (7) (8) 4

5 4.3. Image mosaicing using triangular patches Here we show how to make image mosaics with triangular patches. First, we obtain an average homography H computed from all of the feature correspondences. Then we obtain each homography H t for each patch after triangulation. In the overlapping region, we use homographies H t for triangles. Outside the overlapping region, we use the generic homography H. Fig. 5. Landscape. 5. Experiments This section introduces the results of applying our feature-based techniques to image mosaicing. These images were taken with a hand-held digital still camera with no tripod Single planar scene Figure 5 shows an example. It took 11 seconds on a Pentium 300 MHz PC, 2 seconds for extracting the overlapping region, 4 seconds for matching features and obtaining the homography, 3 seconds for warping the images, and 2 seconds for blending intensities weighted by the distance from the boundaries, for an image size of 640 u 480 pixels. The reduced image size for obtaining the overlapping region was 40 u 30. We used four levels of pyramid for optical flow estimation. Figure 6 shows a comparison with a cylindrical projection. Our result was better even though the camera motion contained twisting around the optical axis and the scene had perspective distortion. On the other hand, the result with a cylindrical projection was poor. This is because the camera motion contained rotation besides panning. We did not blend intensities for evaluation purposes. Figure 7 shows another example. It demonstrates that our method does not restrict camera motion to horizontal rotation. Our method can make a panoramic image, even though the camera motion includes not only panning but also tilting or twisting. We used only G channel of RGB color for fast processing to obtain transformation, although all of the images are color. This is because the human eye is regarded to be sensitive to green Multiple planar scene Figure 9 shows the result of a multiple planar scene. Since the scene has a certain amount of depth range, we cannot regard it as a planar surface. The overlapping part of the image is divided into 40 triangles using feature points. The average of absolute intensity differences in the overlapping region has been reduced to 16.0%, compared with a single planar model. Here some triangles do not correspond to actual planes. This is because we cannot guarantee that the inside of the triangles always contain flat objects, although the vertices are on planes. There is no distortion when the triangles are on actual planes. However, if they are not on planes, there is significant misregistration. To solve this problem, we postprocessed for those triangles. First we determined whether the triangles contain nonplanar objects or not by measuring the intensity difference with normalized correlation. When the coefficients are Fig. 6. Clock tower (left: planar projective transform; right: cylindrical projection). 5

6 Fig. 7. Canal. beyond the predefined threshold, we took the whole region from either of the images rather than blending. This operation can reduce geometric distortions Comparison with previous work Fig. 8. Triangulation. We compared the processing time with one of the previous methods, namely, Szeliski s method. This method uses the following sum of squared intensity differences E between two images I and Ic to obtain the homography. It finds the optical flow vectors and the homography by minimizing the cost E with the Levenberg Marquardt nonlinear method. Fig. 9. Arbitrary depth scene (left: single; right: multiple). 6

7 (9) Table 2. Accuracy of overlap extraction However, Szeliski s nonlinear minimization method has the following two problems, although it has been widely used. First, it is computationally expensive, requiring iterative warpings. Second, it easily falls into a local minimum. This problem is common to the nonlinear minimization framework method. Then it is very important to provide a good initial estimation. Table 1 shows a comparison with Szeliski s nonlinear minimization method. Our method is 4 to 7 times faster than Szeliski s. We have measured the processing time of computing homographies on an SGI Indigo 2 (195 MHz). We had to prepare four levels of pyramid to avoid local minima. We have set the number of maximum iterations to be Accuracy of overlap extraction We measured the accuracy of our overlap extraction described in Section 3.2. We used 200 pairs of images which were taken with a hand-held camera. The image size ranges from 240 u 320 to 1024 u 768. A size of 40 u 30 was used for overlap extraction. Table 2 shows the result. We counted the number of successful image pairs versus the correct answers by hand. We conducted an experiment not only with normalized correlation but also with the sum of squared difference (SSD). The image pairs that SSD failed but correlation succeeded, had large brightness differences. Using normalized correlation took 2 seconds to process, which was 0.5 second longer than that of SSD, on a Pentium II 300 MHz with an image size of 640 u Discussion The chief advantage of our method is that it is faster and more accurate than previous methods at computing homographies. We can categorize mosaicing techniques that use planar projective transformation into two types. One uses nonlinear minimization framework and the other uses features. Our method is 4 to 7 times faster than Szeliski s method, which is based on nonlinear minimization framework. Feature-based techniques can be categorized into two types. One uses manual feature correspondence and the other uses automatic feature correspondence. There is a method of automatic feature correspondence which makes feature correspondence randomly [13]. This method was reported to take 15 minutes depending on images. Our method is much faster than this method. Traditional feature matching techniques such as area-based matching can be used. However, these techniques take longer time to process and tend to have false matching when the shooting scene contains repetitive pattern. On the other hand, our method is faster and more accurate than these methods, because it is based on a gradient-descent method. All types of image-based techniques have difficulties in handling low-textured images. Our method is not an exception, although it does not depend on a type of geometric properties. With low-textured images, having no feature correspondence or false matches cannot be avoided. Moving objects such as people or automobiles might be a problem since our method assumes static scenes. However, we can avoid this problem by using robust estimation when the scene can be regarded as a single plane or be taken from the same viewpoint. When the scene contains arbitrary depth and is taken from different viewpoints, we have to correct false matching by hand. Our method can afford some range of rotation around the optical center, although our overlapping extraction is based on translation. We have confirmed that it tolerates around 10 degrees of rotation, although it is not as good as Zoghlami s method [13]. Table 1. Processing time comparison (seconds) 7. Conclusions In this paper, we have illustrated a feature-based image mosaicing method. The greatest advantage of our method is that we can make a panoramic image of a nonplanar scene with unrestricted camera motion. First, we showed that our feature-based method is faster and more robust than previous featureless methods, because it is based on linear techniques. 7

8 Second, for a scene that we cannot assume to be a single plane. taken from different viewpoints, we described a method of dividing images into triangles with corresponding features. Our method provides the homography for each triangle from three points using the epipolar constraint, although existing methods require four points to compute. The technique is fast and robust, because it is linear. In future work, we plan to develop a technique using line features which we can expect to have more accuracy. We are also interested in creating a virtual environment that can provide a novel view from any viewpoints in 3D space. REFERENCES 1. Baron J, Fleet D, Beauchemin S. Performance of optical flow techniques. Int J Comput Vision 1994;12: Chen SE. QuickTime VR an image-based approach to virtual environment navigation. SIG- GRAPH 95, p Chiba N, Kanade T. A tracker for broken and closely spaced lines. Syst Comput Japan 1999;30: Kumar R, Anandan P, Hanna K. Direct recovery of shape from multiple views: A parallax based approach. Proc ICPR, p , Faugeras O. Three-dimensional computer vision: A geometric viewpoint. MIT Press; Lucas B, Kanade T. An iterative image registration technique with an application to stereo vision. 7th Int Joint Conference on Artificial Intelligence (IJCAI- 81), p Szeliski R. Video mosaics for virtual environment. IEEE Computer Graphics and Applications, p 22 30, Szeliski R. Image mosaicing for tele-reality applications. CRL Technical Report 94/2, Digital Equipment Corporation, Szeliski R, Shum HY. Creating full view panoramic image mosaics and environment maps. SIG- GRAPH 97, p Sawhney HS, Ayer S. Compact representations of videos through dominant and multiple motion estimation. IEEE PAMI, p , Tomasi C, Kanade T. Shape and motion from image streams: A factorization method Part 3 detection and tracking of point features. CMU-CS , Carnegie Mellon University, Zhang Z. Determining the epipolar geometry and its uncertainty: A review. Int J Comput Vision 1998;27: Zoghlami I, Faugeras O, Deriche R. Using geometric corners to build a 2D mosaic from a set of images. CVPR 97, p AUTHORS (from left to right) Naoki Chiba (member) received his B.S. degree in mechanical engineering from Kobe University in He then joined the Research and Development Headquarters of Sanyo Electric Company, and is currently a chief researcher there. He was a visiting scientist at the Robotics Institute of Carnegie Mellon University from 1995 to His current research interests include feature tracking, image mosaicing, and 3D reconstruction. Hiroshi Kano (member) received his M.S. degree from Kyoto University in He then joined the Research and Development Headquarters of Sanyo Electric Company in 1984, and is currently a lab leader there. He was a visiting scientist at the Robotics Institute of Carnegie Mellon University from 1993 to He received his Ph.D. degree from Kyoto University. 8

9 AUTHORS (continued) (from left to right) Michihiko Minoh (member) received his Ph.D. degree from Kyoto University in He was a visiting research scientist at the University of Massachusetts from 1987 to He became an associate professor at Kyoto University in 1989 and a professor in Masashi Yasuda (non-member) received his M.S. degree from Kyoto University in He then joined the Research and Development Headquarters of Sanyo Electric Company in 1980, and is currently a section leader there. He received his Ph.D. degree from Kyoto University. 9