AUTOMATIC TERRAIN EXTRACTION USING MULTIPLE IMAGE PAIR AND BACK MATCHING INTRODUCTION

Transcription

1 AUTOMATIC TERRAIN EXTRACTION USING MULTIPLE IMAGE PAIR AND BACK MATCHING Bingcai Zhang, Chief Photogrammetrist Scott Miller, Chief Engineer Kurt DeVenecia, Product Manager Stewart Walker, Director of Marketing BAE Systems National Security Solutions Technology Place San Diego, CA ABSTRACT Automatic Terrain Extraction (ATE) is a key component of digital photogrammetric software. Image correlation has been widely used in ATE and has proved to be a reliable and accurate approach. This paper discusses two algorithms for ATE: (1) multiple image pairs; and (2) back matching. With standard methods of image acquisition, a point on the ground is often covered by multiple stereo image pairs. By correlating on these multiple pairs, ATE computes multiple elevations, which can then be used to detect image correlation blunders and improve elevation accuracy. In difficult terrain such as airports and streets, image correlation algorithms may generate false correlations due to lack of texture and repeated patterns. The back matching algorithm effectively detects those false correlations by checking the consistency between forward matching and backward matching. Initial test results indicate that the multiple image pair algorithm can improve elevation accuracy up to 30%. The back matching algorithm has successfully detected and removed false correlations in a difficult test stereo pair over an airport. This paper presents the theoretical foundation of these two algorithms, supported by test results. INTRODUCTION The driving force for digital photogrammetric systems is the high cost of extracting geospatial data from raw imagery into GIS databases. As analyzed in Stevenson s paper (1995) more than 50% of GIS cost goes to data extraction. Digital photogrammetric systems can dramatically reduce the cost of data extraction by automating the process. For example, the SOCET SET software from BAE Systems can capture over 1,000,000 elevation points in one hour. This is equivalent to many days of manually digitizing from stereo plotting. The unarguable advantage of digital photogrammetric systems over traditional stereo plotters is the ability to automatically generate a 3-D digital terrain model (DTM). The key issue of digital photogrammetry is to establish correspondence between individual points in two images such that each matched points are truly of the same object point. The elevation of this point can then be computed using projective geometry. There are two major approaches for establishment of correspondence: intensity-based methods, and edge-based methods. Both methods may produce elevation blunders due to false matching results. Blunder detection in DTM generation becomes a critical component of DTM production. Intensity-based Methods Based on image intensities, these methods match the conjugates of a point along conjugate epipolar lines. Such a strategy assumes that scene points have the same or similar intensity in each image and there is significant image intensity variation (uniqueness) over both images. There are a large number for algorithms for image matching based on image intensities (Ackermann, 1984; Calitz and Ruther, 1996; Helava, 1988; Li, 1991; Okutomi and Kanade, 1992; Rosenholm, 1987; Zhang and Miller, 1997). The most widely used method is the normalized cross correlation (Vosselman, et al, 2004).

2 Intensity-based methods use a window (template and search window) of image to compute X parallax. When the window size is too small, there may not enough signal power to identify a reliable match. When the window size is too large, the X parallax within the window may not be the same. Okutomi and Kanade (1992) use a signal matching algorithm that can select an appropriate window size adaptively according to image intensity variations so as to obtain precise and stable estimates of correspondences. They state: Matching two signals by calculating the sum of squared differences (SSD) over a certain window is a basic technique in computer vision. Given the signals and a window, there are two factors that determine the difficulty of obtaining precise matching. The first is the variation of the signal within the window, which must be large enough, relative to the noise, that the SSD values exhibit a clear and sharp minimum at the correct disparity. The second factor is the variation of the disparity within the window, which must be small enough that signals of corresponding positions are duly compared. These two factors present conflicting requirement to the size of the matching window, since a larger window tends to increase the signal variation, but at the same time tends to include points of different disparity. A window size must be adaptively selected depending on local variations of signal and disparity in order to compute a mostcertain estimate of disparity at each point. The least squares image matching method is widely used in digital photogrammetry (Calitz and Ruther, 1996; Helava, 1988; Hermanson et al, 1993). There are three approaches: multi-grid multi-point matching; multi-point matching with multi-resolution; and combination of multi-grid and multi-resolution. Li (1991) uses the combination of multi-grid and multi-resolution for multi-point matching. The multi-point matching algorithm does not have the problem of window size and signal variations as discussed by Okutomi and Kanade (1992). There are some other approaches such as the multiple-baseline stereo method (Kanade and Nakahara, 1992) multicolor image matching, and multi-camera image matching (Dahman, 1994). The basic idea of these approaches is to acquire redundant data of the same scene and perform extra image matching. These approaches may improve image matching accuracy to some extent. Edge-based Methods Edge-based methods establish correspondence between image points by matching image-intensity patterns along conjugate epipolar lines. They detect edges and then seek matches between these edges' intersections with conjugate epipolar lines. This approach will not work well if the image regions have no edges or if it is difficult to detect edges precisely. There are numerous edge-based algorithms (Medioni and Nevatia, 1985; Ohta and Kanade, 1985). This approach assumes that there are enough edges in the images and they can be accurately detected. If this assumption holds, the edge-based image matching has certain advantages over the intensity-based image matching because image noise has less effect on edges than on intensities. Image noise is an unavoidable problem in digital photogrammetry and is very complex. As Greenfeld (1991) analyzed, there are so many problems associated with image matching in digital photogrammetry. He states: 1. Photometric (radiometric) problems: resolution, reflectance, illumination, foreshortened effect, exposure parameters differences, lab processing noise, digital camera radiometric calibration differences, digital camera noise. 2. Geometric problems (perspective projection): relief displacement and occluded areas, projective deformation, scale variation, base/height ratio. 3. Textural problems: featureless surface, repetitive texture, hanging surfaces, ambiguous levels such as tree top and ground below them, thin objects. There exists one matching system that has the capabilities to overcome the matching difficulties presented above. That is the human vision system. Considering all the above problems, intensity-based image matching seems to be hopeless. Marr, in his book Vision (1982), states: The job of stereo fusion is to match items that have definite physical correlates, because the law of physics can guarantee only that items will be matchable if they correspond to things in space that have a well-defined physical location. Gray-level pixel values do not. Hence, gray-level correlation fails. The major difficulties with edge-based image matching in digital photogrammetry are: (1) there are usually not enough edges in real terrain to generate a DTM, (2) there is no perfect edge detector available, and (3) edge-based matching is still based on image intensities and hence is not free of noise and discontinuities. Thus, on balance,

3 despite the problems with intensity based approaches, ATE has had good success with this method in producing high quality DTMs by adding heuristic algorithms that compensate for the inherent problems described above. Blunder Detection Blunder detection in ATE is critical to reduce DTM editing time. Unfortunately, there is not much research done due to lack of redundancies in DTM generation. Based on the characteristics that adjacent elevation points have certain properties, such as slope, in common, ATE can perform some blunder detection for spike and well points. For example, when a point elevation is higher than its 4 adjacent neighbors by a certain threshold, this point can be considered as a spike point. This type of blunder detection is dependent on a number of factors such as the point spacing of a DTM, terrain type (smooth or rugged), with or without trees and buildings, etc. As a result, this type of blunder detection is in general not very reliable and successful. The key to reliable and successful blunder detection in ATE is to increase redundancies. When ATE computes the elevation of the same point using more than one method or different stereo image pairs, there are redundancies. Based on the redundancies, ATE can perform more reliable and successful blunder detection. Back matching as described in section 3 is one method to double the redundancies in ATE. The multiple image pair algorithm as described in section 2 is another. This paper presents the theoretical foundation of multiple image pairs and back matching algorithms and considers issues of practical implementation. It also includes some comparison results with and without using the multiple image pairs algorithm, as well as with and without using the back matching algorithm. The rest of the paper is organized as follows: (2) multiple image pair algorithm; (3) back matching algorithm; (4) test results; (5) future research direction; and (6) conclusions. MULTIPLE IMAGE PAIR ALGORITHM Standard aerial image capture techniques provide stereo imagery with significant overlap between frames. As illustrated in Figure 1, a typical frame image: 30% side overlap and 60% end overlap may have up to 76% of the ground area covered by more than two stereo image pairs. With new digital sensors such as Leica Geosystems ADS40 and multiple heterogeneous sensors, the number of stereo image pairs for a given point increases dramatically. It is quite common to have 9 to 12 stereo image pairs for a given point. Today, LIDAR systems are also making a big impact on the production of reliable elevation data. LIDAR systems work better with narrow scan widths so as to avoid occlusions. Thus it is common to fly many LIDAR flight lines with narrow scan widths. Photogrammetry can also avoid these occlusions by a) using more than one stereo pair and b) flying strips with 60% side and forward overlap. This has the added benefit of higher quality orthophoto production as relief displacement is minimized. By correlating on multiple stereo image pairs, ATE generates multiple elevations of the same point and avoids more occlusions. With multiple elevations, ATE can perform blunder detection and select the most reliable and accurate elevation.

4 4 or more 30% 3 or more 2 images 3 or more 30% 20% 20% 4 or more Figure 1. Typical frame image: 30% side overlap and 60% end overlap may have up to 76% of the ground area covered by more than two images. Image correlation is computationally intensive. When correlating on multiple stereo image pairs, ATE may run significantly slower than a single best stereo image pair in order to produce a higher quality result. ATE allows users to specify the maximum number of stereo image pairs per point. For example, a point is covered by 12 stereo image pairs and the user specifies 4 as the maximum number of stereo image pairs per point. ATE will rank the 12 stereo image pairs from the best to the worst for this point and use the first 4 stereo image pairs for image correlation. Since image relief distortions and obscurities due to relief are two of the major problems for image correlation, ATE gives a higher score to the image pair which displays less of these problems. The total amount of overlap in the area of interest is also a factor. Figure 2 illustrates the ranking algorithm described above. In this example, three images taken from different perspectives cover the same area on the ground. The better image pair in this example would be the middle and right images because of the high relief distortions that would appear in the left image. For different image matching areas, the better image pair or pairs will change accordingly. The better image pair is a function of terrain slope and orientation, and sensor geometry as shown in Figure 2. images poor better good terrain image matching area Figure 2. Better image pair selection based on terrain slope and orientation, combined with sensor geometry (the better image pair consists of the middle and right images for the image matching area indicated).

5 BACK MATCHING ALGORITHM The image correlation algorithm uses two images: (1) left image, and (2) right image as illustrated in Figure 3 to perform image matching. For ground point, the corresponding image locations on the left image and the right image are not conjugate points as shown with the two cursors. The X parallax is caused by the initial Z being inaccurate. The purpose of image matching is to determine the X parallax. Once the X parallax is computed, ATE can compute the change in elevation. In areas without sufficient texture such as water bodies, streets, airports, repeated patterns, image correlation is prone to false matching. In the forward matching, a patch of pixels centered around the cursor in the left image are used to match a patch of pixels centered around the cursor in the right image as illustrated in Figure 3. The center of the patch in the left image is fixed while the center of the patch in the right image moves along the epipolar line. In Figure 3, when the center of the patch in the right image moves to the left by a few pixels, the image correlation has the greatest value. The few pixels to the left (1) are the X parallax. In the backward matching, the center of the patch in the right image is fixed while the center of the patch in the left image moves along the epipolar line. In Figure 3, when the center of the patch in the left image moves to the right by a few pixels, the image correlation has the greatest value. The few pixels to the right (2) are the X parallax. When the delta elevations computed from 1 and 2 are far apart, one or both of the image correlations have blunders and the results are not reliable. When the delta elevations are very close to each other, the image correlations on this point are reliable. When either the forward matching or backward matching fails, this indicates that the point is difficult to match and the one successful matching result may not be very reliable. left image right image Figure 3. Forward matching and backward matching. In the forward matching, a 15x15 pixel patch window moves along the epipolar line in the right image while the same 15x15 pixel patch window in the left image centered around the left cursor does not move. In the backward matching, the 15x15 pixel patch window centered around the right cursor does not move, but the same patch window in the left image moves. Because the 15x15 pixel patch window in the left image and the right image are different, this process generates two matching results. Since there is only one elevation at a given XY location, the two matching results have to be consistent. TEST RESULTS The first test site consists of two-strips with 4 frame images in each strip. The DTM lies in the overlap area of the two strips and covers a large building. A manually edited grid DTM of 960 check points was compared with two automatically derived DTMs, one using one as the maximum number of image pairs per point (MNIPPP) and the other using four as the MNIPPP. The elevations of the same 960 check points were computed from both DTMs and compared with the manually measured corresponding elevations. The elevation differences used to compute the root mean square error (RMS) and standard deviation (STD). The accuracy improvements of the DTMs were compared by the differences between MNIPPP 4 RMS and MNIPPP 1 RMS, and MNIPPP 4 STD and MNIPPP 1 STD, divided by MNIPPP 1 RMS and STD, respectively. The accuracy improvement and image correlation success rate was computed by the difference between MNIPPP 4 and MNIPPP 1 success rates divided by (100% - MNIPPP 1 success rate). Test data for the first site is tabulated in Table 1. It should be noted that the large RMS and STD errors and low success rate are due to elevation discontinuities and a lack of texture caused by a large building as illustrated in Figure 4. MNIPPP 4 is 3 times slower than MNIPPP 1 as indicated in Table 1.

6 Table 1. Multiple image pairs matching vs. single image pair matching (number of check points = 960) MNIPPP 4 MNIPPP 1 Performance Improvement RMS error (meter) % STD (meter) % Success rate (%) % Points per second % Figure 4. A manually edited grid DTM of 960 points with a contour interval of 1 meter and a post spacing of 3 meters. Figure 5. DTM generated by ATE using a single best image pair vs. using 4 image pairs per post of 960 points with contour interval of 1 meter and post spacing of 3 meters. DTM on the left is from a single best image pair. DTM on the right is from 4 image pairs per post.

7 The second test site was Memphis airport. Two images were used to test the back matching algorithm. As illustrated in Figure 6 and 7, the back matching algorithm correctly detected and removed blunders on the runways. Figure 6 is the result without using back matching. Figure 7 is the result when back matching is used. Figure 6. Memphis airport DTM generated by ATE without using back matching. Huge blunders on the runways are caused by lack of texture particularly along the epipolar lines. The contour interval is 5 feet. Figure 7. Memphis airport DTM generated by ATE using back matching. Notice that blunders on the runways are correctly detected and removed by the back matching algorithm. Contour interval is 5 feet.

8 FUTURE RESEARCH DIRECTIONS Although the multiple image pair algorithm and the back matching algorithm improve ATE DTM generation, they do not solve the elevation discontinuity problem caused by buildings and trees in urban areas of large scale imagery. All intensity-based image matching algorithms are based on the assumption that the elevation within an image window such as 15 pixels by 15 pixels is somewhat flat without elevation discontinuity. Buildings and trees are not elevation. However, to intensity-based image patch algorithms, they appear as such. From building tops to building bases, the elevations change abruptly. As a result, intensity-based image matching algorithms fail on buildings and trees. As illustrated in Figure 8, a 15x15 pixel window centered at point x2 on the left image is significantly different from the same window on the right image due to the elevation discontinuity or the height of the building. The hybrid algorithm offers a promising solution for the elevation discontinuity problem. Whenever there is an elevation discontinuity, there are likely very good edge pixels. The hybrid algorithm uses both intensity-based matching and edge-based matching. The intensity-based matching can generate an approximate DTM such that the conjugate edge pixels are within 5-10 pixels. Reducing the search distance significantly reduces the difficulty of edge matching. At the same time, the edge matching results can help intensity-based image matching by segmenting image window based on edges, by seeding and constraining intensity-based image matching. In Figure 8, edge pixels x2, x6, and x9 are at the locations of elevation discontinuity. Intensity-based image matching algorithms fail on them. Figure 8. Hybrid matching algorithm. Intensity-based image matching algorithms generate an approximate DTM such that conjugate edge pixels are constrained to within only 5-10 pixels. This constraint significantly reduces the difficulty of edge matching. At the same time, reliably matched edge pixels can be used to seed and constrain intensity-based image matching. Edge pixels x1, x4, x7, and x11 can be uniquely matched using the 5-10 pixel search range from intensity-based matching. The bare earth technology transforms a digital surface model (DSM) into a digital elevation model (DEM or bare earth) as illustrated in Figure 9. It was developed in SOCET SET v5.1. It needs two parameters: (1) minimum height of objects such as trees and buildings; and (2) maximum terrain slope. It works with both GRID DTM format and TIN DTM format. For DSM, apriori building shapes and locations can help ATE to switch from image correlation algorithm to edge pixel detection and matching algorithm. As a result, DSM has sharp shapes at building edges. For DEM, a DSM with sharp building edges produces more accurate DEM when running the bare earth algorithm.

9 Figure 9. Bare earth algorithm. DTM on the left is a DSM from ATE. DTM on the right is a DEM after applying the bare earth algorithm. The difference between DSM and DEM reveals the approximate locations and shapes of buildings and trees. Our future research direction is to develop a next generation photogrammetry automation system, which takes a number of oriented stereo images as input, and generates much more accurate DSM, DEM, true ortho images, and feature database of buildings as illustrated in Figure 10. The oriented stereo images are: (1) a next generation Automatic Triangulation Manager (ATM) that automatically triangulates input images that have associated georeferencing metadata as input; and (2) oriented digital images directly from digital sensors. The accuracy of DSM and DEM should be compatible with the accuracy of LIDAR. The accuracy of the true ortho images should be much higher than the current technology can provide. The feature database of buildings can be used for applications such as telecommunications for line of sight analysis, airport surveys including obstruction models, and visualization for simulators and image based virtual tours, to name a few. Both DSM and DEM have elevations for every pixel. Using DSM and DEM as input, the really true ortho image generation engine can produce a true ortho image with much higher accuracy.

10 a number of oriented stereo images multiple image pair algorithm back matching algorithm hybrid image matching algorithm bare earth algorithm adaptive image matching algorithm TIN technology accurate DSM and DEM (compatible with LIDAR accuracy) building dimension (height and width) really true ortho image generation engine DSM DEM edge matching Hough transform true ortho images feature database of buildings Figure 10. Next generation photogrammetry automation system. CONCLUSIONS Automatic Terrain Extraction (ATE) is a very important component of digital photogrammetric software. Image correlation has been well studied as a means to producing DTMs. As used in ATE, this approach has been shown to provide good results quickly. However, when greater reliability is required, significant improvements are possible by using the methods described in this paper. Blunder detection for image correlation is a critical component of DTM production. By using multiple stereo image pairs and back matching, ATE can significantly increase the redundancies of image correlation. Using these redundancies, ATE can reliably perform elevation blunder detection and removal and reduce the human time required to complete terrain extraction. Initial test results indicate that the multiple image pair algorithm can improve elevation accuracy up to 30%. The back matching algorithm can effectively detect false correlations by checking the consistency between forward and backward matching. The back matching algorithm has successfully detected and removed false image

11 correlations in an airport. Computing time may increase substantially using multiple image pairs and back matching algorithms. However, ATE can run in batch and uses multiple CPUs, and for most production situations, this can be alleviated by running overnight or on weekends. Users also have the option to select the maximum number of multiple image pairs per point for image correlation and the on and off option for back matching to control the amount of time to complete ATE. The proposed next generation photogrammetry automation system offers great promise for future research and development. This system should be able to generate DSM and DEM with accuracy similar to LIDAR. Based on accurate DSM and DEM, this system should be able to produce very accurate true ortho images with accuracy similar to ortho images from ISTAR system. With accurate DSM, DEM, this future system offers the promise to extract buildings automatically. The accurate extraction of DSM and DEM is a key component to a truly automatic end-to-end photogrammetry product. ACKNOWLEDGMENTS This research was supported by BAE Systems under an internal research and development project. Special thanks are due to Gail Nagle for her numerous comments and editing. REFERENCES Ackermann, F. (1984). Digital image correlation: performance and potential application in photogrammetry. Paper presented at the 1984 Thompson Symposium, Birmingham, England. Calitz, M.F., and H. Ruther (1996). Least absolute deviation (LAD) image matching. ISPRS Journal of Photogrammetry & Remote Sensing, Vol. 51, pp Dahman, N.A. (1994). Application of multiocular stereo to digital photogrammetry. Seminar Paper, Department of Civil and Environmental Engineering, University of Wisconsin-Madison, Madison, Wisconsin. Greenfeld, J.S. (1991). An operator-based matching system. Photogrammetric Engineering & Remote Sensing, Vol. 57, No. 8, pp Helava, U.V. (1988). Object-space least-squares correlation. Photogrammetric Engineering & Remote Sensing, Vol. 54, No. 6, Part 1, pp Hermanson, G., J. Hinchman, U. Rauhala, and W. Mueller (1993). Comparison of two terrain extraction algorithms: hierarchical relaxation correlation and global least squares matching. SPIE Spring Convention, Orlando, April. Kanade, T., and T. Nakahara (1992). A multiple-baseline stereo method. Proceedings of the 1992 DARPA Image Understanding Workshop, January. Li, M. (1991). Hierarchical multipoint matching. Photogrammetric Engineering & Remote Sensing, Vol. 57, No. 8, August, pp Marr, D. (1982). Vision, W. H. Freeman and Company, San Francisco, California. Medioni G., and R. Nevatia (1985). Segment-based stereo matching. Computer Vision, Graphics, and Image Processing, Vol. 31, pp Ohta, Y., and T. Kanade (1985). Stereo by intra- and inter-scanline search using dynamic programming. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-7, No. 2 (March), pp Okutomi, M., and T. Kanade (1992). A locally adaptive window for signal matching. International Journal of Computer Vision, 7:2, Rosenholm, D. (1987). Multi-point matching using the least squares technique for evaluation of three-dimensional models. Photogrammetric Engineering & Remote Sensing, Vol. 53, No. 6, pp Stevenson, P.J. (1995). The problem of data conversion. Geo Info Systems, February, pp Vosselman, G., M. Sester, and H. Mayer (2004). Basic computer vision techniques. Manual of Photogrammetry, Fifth Edition, American Society for Photogrammetry and Remote Sensing (edited by McGlone, J.C., Mikhail E.M., and J. Bethel). pp Zhang B., and S. Miller (1997). Adaptive automatic terrain extraction, Proceedings of SPIE, Volume 3072, Integrating Photogrammetric Techniques with Scene Analysis and Machine Vision (edited by D. M. McKeown, J. C. McGlone and O. Jamet). pp