Underwater Mosaic Creation using Video sequences from Different Altitudes

Transcription

1 Underwater Mosaic Creation using Video sequences from Different Altitudes Nuno Gracias and Shahriar Negahdaripour Underwater Vision and Imaging Lab Electrical and Computer Engineering Department University of Miami Coral Gables, FL Abstract This paper presents a method for the automatic creation of 2D mosaics of the sea floor, using video sequences acquired at different altitudes above the sea floor. The benefit of using different altitude sequences comes from the fact that higher altitude sequences can be used to guide the motion estimation of the lower ones, thus increasing the robustness and efficiency of the mosaicing process. When compared to the case of single sequence mosaic creation, we show that by combining geometric information from different sequences, we are able to successfully estimate the registration topology of much lower altitude sequences. This results in higher resolution mapping of the sea floor. Illustrative results are presented using sequences of the same coral reef patch, captured with a single video camera. The sequences present some of the common difficulties of underwater 2D mosaicing, namely non-flat, moving environment and changing lighting conditions. The importance of this work is emphasized by fact that the presented methods require inexpensive image acquisition and processing equipment, thus potentially benefiting a very large group of marine scientists. I. Introduction The creation of wide area views of the sea floor is an important optical sensing capability for benthic exploration and mapping. Such views can easily be interpreted by a human operator on a survey mission or be used as a spatial representation for navigation. When compared with land or aerial environments, the light underwater is subject to intense attenuation and scattering. These factors severely limit the definition and range of underwater imagery. Under such conditions, video mosaicing methods are suited to creating large visual representations of the sea floor, through the registration of many close range images. This paper presents a method for the automatic creation of 2D mosaics of the sea floor, using video sequences acquired at different altitudes above the sea floor. We implement and test two different methods to illustrate the advantages of this approach. One of the methods implies the rendering of the higher altitude mosaics to serve as a map upon which the lower altitude images are matched against. The other method does not require a rendered mosaic, but only the topology of the sequence, i.e. the arrangement of the images on a common spatial reference frame. The images of the lower altitude se- ngracias@isr.ist.utl.pt. The work described in this paper has been partially supported by SERDP Proj. CS 1333 and the Portuguese Foundation for the Science and Technology BPD/1462/23. quences are matched directly against the higher altitude ones. For both methods the resulting set of geometric constraints is used in a bundle adjustment step, thus promoting the final geometric accuracy. The two methods are compared using a challenging video sequence and tested for accuracy and computational efficiency using ground truth. II. Background and Related work Underwater video mapping requires the registration of large sets of images. Most commonly this is performed by pair wise image registration in chronological order. The resulting motion estimates are then concatenated to infer the relation between any pair of images. However, even small amounts of noise in the estimation process will result in large accumulated error. To create spatially coherent mosaics, a number of authors tackled the problem of registration for multiple image superpositions from camera trajectories loops [1], [2], [3], [4]. Related to this, bundle adjustment techniques from the photogrammetry literature have been successfully adapted to image registering applications []. Recent years have witnessed significant progress in the optical mosaicing and 3D reconstruction of large underwater areas [6], [7]. The use of specialized platforms, with navigation sensors providing pose information, can considerably assist the problem of image motion and structure estimation [8]. However such equipment is still too expensive to be used by a large group of marine scientists. Conversely, vision only systems are more sensitive to image quality and departures to the assume motion and structure models. However they allow for inexpensive acquisition equipment which can be, at the limit, a standard underwater handheld video camera. Our paper addresses the issue of mosaicing sequences of low image overlap among strips. This relates to the work of Singh et al. [9] where an higher altitude mosaic is used to merge separate mosaic strips that were previously constructed from lower altitude. The registration uses a high number of manually matched points and is followed by a finite-element warping. This contrasts with our approach, which is based on automatic image matching. The only user intervention is in defining a contact area among the sequences, in the form of 2 coarse matches over 2 images. 1

2 III. Background mosaic creation In this paper, we refer to the high altitude image set as the background sequence and the low altitude as the foreground. The background mosaic creation method, which is summarized in this section, mirrors previous work on underwater video mosaicing[1] and pose estimation[11]. For a detailed description please refer to [3]. The method comprises three major stages. The first stage consists of the sequential estimation of the image motion, using a reduced image motion model. The set of resulting consecutive homographies is cascaded, in order to infer the approximate topology of the camera movement. The topology information is then used to predict the areas where there is image overlap resulting from nonconsecutive images. Secondly, the overall topology is refined by iteratively executing the following two steps. (1) Point correspondences are established between non-adjacent pairs of images that present enough overlap. (2) The topology is updated by searching for the set of homographies that minimizes the overall sum of distances in the point matches. The final stage of the algorithm estimates the set of homographies and a world plane description that best fit the observation data. As the main concern is attaining high registration accuracy, a general parameterization of the homographies with 6 DOF for the pose is used, which is capable of modelling the effects of wave induced general rotation and translation. An essential building block consists of image registration of pairs of images. This is done as follows. 1. A set of point features corresponding to textured areas, are extracted from one of the images, using the Harris corner detector [12]. 2. For each feature (defined as a small square image patch centered at the detected corner location), a prospective match is found in the other image, using normalized cross-correlation. We assume that prior information exists on the expected image motion (typically in the form of a homography). This information is used for two main purposes. It establishes the location of the correlation window center. It defines the required warping of the image feature so that the search over the other image becomes essentially a translation (2D) search. This allows for the use of area correlation for heavily rotated or slanted images. 3. A robust estimation technique is used to remove outliers using a Least Median of Squares criterion, based on a planar motion model. The background mosaic was created using a 28 minute video sequence of a survey of a coral reef patch. The coral patch is approximately flat at site scale, but contains innumerous benthic structures with vertical relief up to 7 cm from the average plane. The sequence was acquired from a single camera, deployed by a custom modified commercial ROV [13]. The camera was internally calibrated to reduce image distortion from the lens and housing [14]. Although the vehicle is equipped with additional sensors, such as a depth gauge, compass and inclinometers, only the video information was used to create the mosaics. The vehicle covered an area of approximately 4 square meters, at an altitude of 2. to 3 meters. It followed a lawn mower s pattern of side by side strips followed by the same pattern rotated 9 degrees. The purpose of this was to ensure full coverage of the area and provide a high number of superposition areas among the strips. On the first part of the algorithm, 496 key frames were selected out of the complete set of 61 images, using a criterion of 72% overlap between time consecutive images. On the second part, a total of 1292 image pairs were matched, over 12 cycles of iterative superposition prediction and matching. The resulting mosaic is shown in Figure 1. It was rendered using the whole set of 61 images. The registration parameters for the non key frames were obtained by linear adjusting of the sequential matching, constrained by the registration parameters of the two closest key frames. The ground resolution ranges from 2. to 3 millimeters per pixel. The original frames are 124x768 pixels. Fig. 1. Complete mosaic from the higher altitude sequence, covering approximatelly 4 square meters. IV. Foreground sequence registration To illustrate the benefit of combining sequences of different altitudes, we implemented and tested two different approaches. 2

3 Image to Mosaic Relies on a existing mosaic image, created from the background sequence. The registration of the foreground sequence is carried out by concurrently performing image to mosaic registration and image to image registration over the foreground sequence. Image to Image Uses the original images of the background sequence, along with their global registration parameters. No mosaic image is required. The foreground images are matched directly against the background counterpart. The background registration constraints are considered static, so no background to background matching is attempted. Furthermore the background registration parameters are not changed. To avoid extensive search over the complete set of background and foreground images, both methods require prior knowledge on a superposition area between the two sequences. For the testing in this paper, the algorithm assumes an estimate of the registration between one frame of the foreground sequences to either the background mosaic or one of the background images. This is provided as two or more manually matched points. These points are used to assist a fully automatic matching, by constraining the search area and providing a prior to the feature warping. As such, the accuracy requirements for the manual matches are not high. Both approaches were implemented with minimal changes to the standard global mosaicing algorithm described in Section III. The algorithm for the image to mosaic approach adds the mosaic image to the set of foreground images and treats it as any other image. The image to image approach joins both image sets, but restricts the matching and the optimization to the foreground data set. A. Accuracy Assessment To quantify the accuracy of the methods we perform a simple geometric distortion analysis, using ground truth data. This data consists of a set of scene points of known metric 2D locations, that are easily located on the mosaic images. The ground truth points were extracted from a high resolution image acquired from a still camera. This image was captured from the a distance of meters from the bottom, near the water surface. The coordinate mapping from the still image to world coordinates was obtained using a set of 13 fiducial points. Givenitisdifficulttodirectly measure the XY locations of points underwater, the estimation of fiducial data was done indirectly, using a network of distance measurements between ground points. This process followed standard surveying guidelines [1]. The fiducial point data was created by placing 1 markers rigidly attached to the sea bed, and distributed over the test area. Additionally, 4 stakes (with no markers) were also used. The required field data was collected by divers who measured the distances of each marker to the control stakes, together with the distances among the 4 control stakes. This resulted in 46 distance measurements. According to [1], the uncertainty in the measurements can be suitably modelled as additive Gaussian noise of zero mean and constant standard deviation. Typically the standard deviation is cm, mainly due to the elasticity of the measuring tape. We are ignoring the effects of terrain elevation which may lead to the measurements not being done over a straight line. However, the markers were positioned in locations that represent the main underlying plane (avoiding crevices and peaks), in order to reduce terrain induced distortions. Given a set of distance measurements, we want to estimate the 2D locations of all points with respect to a common metric reference frame. Let d ij be measured distance between points i and j. The observed noisy measurement relates with the ideal noise-free distance d ij as d ij = d ij + ε, where ε is a additive noise term. Each point is represented by its 2D coordinates P i =(x i,y i ). The observations relate to the sought parameters as d ij = (x i x j ) 2 +(y i y j ) 2 + ε. Using a Least-Squares criteria, the problem can be formulated as finding the set of ( x i, ŷ i ) such that ) 2 ( x i, ŷ i ) = arg min ( dij d ij. (x i,y i) To fix the gauge freedoms, additional constraints need to be imposed. These can be conveniently defined as x 1 = y 1 = y 2 =, which sets the origin at point 1 and the world X axis along the line between point 1 and 2. The coordinates of the fiducial point set were estimated using a standard non linear least squares algorithm [16]. The resulting residues follow a normal distribution with standard deviation of 2. cm. The image coordinates of the fiducial points were extracted manually from the still. The mapping from the still image to world coordinates was computed as a planar homography, parameterized by the camera pose, as described in [17]. A set of 22 prominent points were selected from the still image, covering the whole image (Figure 2). Care was taken in selecting points that where representative of the underlying average plane, to minimize projection distortions. The metric coordinates of the these points were obtained using the homography. The advantage of using a calibrated still image, as opposed to using the fiducial data set directly, is twofold: it provides more geometrically consistent data, and allows for the use of larger ground truth set. V. Results This section presents illustrative results using a lower altitude sequence. It comprises 36 seconds of video, acquired at approximately 1.4 meters above the sea floor, covering an area of 3 square meters. The sequence was i,j 3

4 gt1 gt(m9) gt19 gt4 gt6() gt7(m) gt8(m2) gt12(m6) gt1(m7) gt22 gt16(m3) gt1 gt9(m2) gt13(m13) gt17 gt11 gt14(m18) gt2 gt18 gt21 gt3 gt2 Fig. 2. High resolution still image, used for defining the ground truth point set (marked). The circular markers are 12cm diameter disks (CDs) positioned around permanently fixed nails. This still image was captured at a different data collecting trip than the mosaic video sets. The PVC quadrats indicate 4 permanent stakes, but are not exactly on the same positions as in the video data sets. selected as an example of a challenging data set. The camera undergoes a lawn mower s pattern trajectory, with 6 strips. Most strips have very low superposition among them, making it difficult to create a geometrically accurate mosaic. Fig. 3. Resulting mosaic from using the global mosaicing approach on the lower altitude sequence alone. The low overlap between non sequential images leads to large registration errors which are clearly visible. The first two strips on the left hand side should be overlapping. This is apparent from the broken image of the quadrat, made of white PVC tubing. A. Single sequence mosaic creation As a baseline for comparison with the approach of this paper, we created a mosaic using the method outlined in section III over the lower altitude sequence. The first part of the algorithm selected 24 images from the original set of 17. This subset was also used for the image to mosaic and image to image tests. Figure 3 presents the resulting mosaic. The algorithm executed 11 cycles of topology refinement, establishing point correspondences among 417 pairs of overlapping images. The small overlap among strips hindered adequate loop closure, leading to noticeable geometric distortions, visible on the left hand side. B. Image to Mosaic The image to mosaic approach used, as the background, the image in Figure 4, taken from the central area of the mosaic in Figure 1. Two point correspondences were manually established among one image from the foreground sequence and the background mosaic. These correspondences helped the automated images matching procedure which established 8 inlier point matches. After 12 cycles of topology refinement, 1141 matched pairs of images were found, from which 197 were image to mosaic matches. The resulting mosaic if shown in Figure. The improvement with respect to the single sequence mosaicing is visible on the left and right hand side strips, which are closer to the central part of the mosaic. Fig. 4. Section of the mosaic used as a background for the image to mosaic registration. C. Image to Image For the image to-image approach, we selected one frame from each sequence that overlap. Similarly to above, two point correspondences were manually established, from which 131 inlier matches were automatically found. The topology refinement executed 1 cycles, establishing 188 matched pairs of images from which 844 were between the two sequences and 141 among the foreground sequence. The resulting mosaic, shown in Figure 6, is close in appearance to the one produced by the image to mosaic approach. For comparison purposes, we have also created a mosaic 4

5 Fig.. Resulting mosaic from the image to mosaic registration. This image was rendered with only the foreground sequence images. Fig. 6. Resulting mosaic from the image to image registration, rendered with only the foreground sequence images. using joint optimization, where the registration parameters for both the background and foreground sequence were estimated. The resulting mosaic exhibits minor improvements, with no visible difference to the one in Figure D. Accuracy Results To quantify the geometric distortions and compare the methods, the location of the 22 ground truth points were manually identified on the mosaics. We define a residue vector for each ground truth point of metric coordinates (x i,y i ) and mosaic image coordinates (u i,v i )as [ ] ] [ ] rxi xi = r yi y i [ h1u i+h 2v i+h 3 h 7u i+h 8v i+1 h 4u i+h v i+h 6 h 7u i+h 8v i+1 where h = [ h 1 h 8 ] T are the parameters of a 8 degrees of freedom projective mapping. This mapping was computed using standard least squares as h = arg min h ( r 2 xi + ry 2 ) i. As geometric distortion criteria we consider the standard deviation of all residues (r x1,r y2,,r y22 ) and the maximum distance error d max = max rx 2 i i + ry 2 i. The results for these criteria are shown in Table I, along with the cost function execution time. The latter refers to the time required to compute the residue vector and the associated Jacobian, used in the Levenberg Marquadt iterations of the non linear least squares optimization. The histograms for the residues are shown in Figure 7. i (Global Simple) (Image to mosaic) (Image to image) (image to image joint estim.) Fig. 7. Histograms of the residues for geometric accuracy for the methods of table I. The stardard deviations of the fitted normal distributions are given in the table. These results testify the advantages of using a higher altitude sequence, either as a mosaic or as a set of images, to guide the registration of a lower altitude image set. Both image to mosaic and image to image approaches allowed for establishing image matches among the foreground sequence that were not possible on the case of the simple global registration of the foreground set. The image to image method attains lower distortion than the image to mosaic for both criteria. This can explained by the fact that the image to mosaic matching is affected by rendering artifacts and imperfections of the background mosaic. For the experiments of this paper,

6 Method St. Dev. Max. dist. error Cost eval. time Global Simple 17.2 cm cm.9 sec Image to mosaic 6.3 cm 18.3 cm 1.47 sec Image to image 4.7 cm 1.97 cm 6.82 sec Image to image with Joint Estim cm cm 16.1 sec TABLE I Statistics for the geometric distortion and computation effort Standard deviation of residues, maximum distance error and the cost function evaluation time. the background mosaic was rendered using the contribution of the closest image (to any given mosaic point) without histogram equalization among the contributing images. This originates seams on the boundaries of the contributions, due to the unaccounted 3D structure and illumination variations, which degrade the matching performance. The image to image matching is not affected by such artifacts, but requires the matching of much larger number of image pairs. This is reflected on the higher execution time. As expected, the joint estimation of both background and foreground sequence registration improves the geometric distortion, due to the higher number of matching constraints available. VI. Conclusions This paper presented an approach for the creation of 2D mosaics of the sea floor, using video sequences acquired at different altitudes above the sea floor. The benefit of using different altitude sequences comes from the fact that higher altitude sequences can be used to guide the motion estimation of the lower ones, thus increasing the robustness and accuracy of the mosaicing process. The methods were devised with emphasis on automated processing. The results presented only require user input for the selection of a small number of correspondences, to relate the background and foreground sequences. Applications such as the mapping of shallow water natural habitats present difficulties due to the departure from some of the assumptions typically used image registration. In this work we used a motion model that assumes flat and static environment, with constant illumination. Clearly, for the low altitude sequence such as the one reported here, these assumptions may not hold. The net effect of this is that the trajectory estimation from the concatenation of motion estimates suffers from rapid error growth. This hinders image registration on closed trajectory loops, which is essential for accurate mosaic construction. The approach and methods presented in this paper addressed this problem without resorting to other sensor modalities. The presented methods require inexpensive image acquisition and processing equipment, thus potentially benefiting a very large group of marine scientists. In Proc. European Conf. on Computer Vision, Freiburg, Germany, June [2] H. Sawhney and R. Kumar. True multi-image alignment and its application to mosaicing and lens distortion correction. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, June IEEE Computer Society Press. [3] N. Gracias and J. Santos-Victor. Underwater mosaicing and trajectory reconstruction using global alignment. In Proc. of the Oceans 21 Conference, pages , Honolulu, Hawaii, U.S.A., November 21. [4] R. Garcia, J. Puig, P. Ridao, and X. Cufi. Augmented state Kalman filtering for AUV navigation. In Proc. Int. Conf. on Robotics and Automation, pages 41 41, Washington DC, USA, May 22. [] P. McLauchlan and A. Jaenicke. Image mosaicing using sequential bundle adjustment. In Proc. of the British Machine Vision Conference BMVC2, Bristol, U.K., September 2. [6] O. Pizarro,, and H. Singh. Towards large area underwater mosaicing for scientific applications. IEEE Journal of Oceanic Engineering, 28(4):61 672, 23. [7] S. Negahdaripour and H. Madjidi. Stereovision imaging on submersible platforms for 3 d mapping of benthic habitats and sea floor structures. IEEE Journal of Oceanic Engineering, 28(4):62 6, October 23. [8] O. Pizarro, R. Eustice, and H. Singh. Large area 3D reconstructions from underwater surveys. In Proc. of the IEEE Oceans 24 Conference, volume 2, pages , Kobe, Japan, November 24. [9] H. Singh, J. Howland, and O Pizarro. Advances in large area photomosaicking underwater. IEEE Journal of Oceanic Engineering, 29(3): , July 24. [1] N. Gracias and J. Santos-Victor. Underwater video mosaics as visual navigation maps. Computer Vision and Image Understanding, 79(1):66 91, July 2. [11] N. Gracias and J. Santos-Victor. Trajectory reconstruction with uncertainty estimation using mosaic registration. Robotics and Autonomous Systems, 3: , July 21. [12] C. Harris and M. Stephens. A combined corner and edge detector. In Proceedings Alvey Conference, pages , Manchester, UK, August [13] X. Xu. Vision based ROV System. PhD thesis, University of Miami, Coral Gables, Miami, May 2. [14] J.Y. Bouguet. Matlab camera calibration toolbox. 22. [1] P. Holt. The site surveyor guide to surveying underwater. Technical report, 3H Consulting Ltd, 2. [16] W. Press, S. Teukolsky, W. Vetterling, and B. Flannery. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, [17] N. Gracias. Mosaic based Visual Navigation for Autonomous Underwater Vehicles. PhD thesis, Instituto Superior Técnico, Lisbon, Portugal, June 23. References [1] H. Sawhney, S. Hsu, and R. Kumar. Robust video mosaicing through topology inference and local to global alignment. 6