DEVELOPMENT AND STATUS OF IMAGE MATCHING IN PHOTOGRAMMETRY

Transcription

1 The Photogrammetric Record 27(137): (March 2012) DOI: /j x DEVELOPMENT AND STATUS OF IMAGE MATCHING IN PHOTOGRAMMETRY Armin Gruen Institute of Conservation and Building Research, ETH Zurich, Switzerland (Based on a presentation at the Ian Dowman Retirement Symposium, entitled Progress and Prospects for Photogrammetry and Remote Sensing in a Changing World, held at University College London on 28th June 2010) Abstract Image and template matching is probably the most important function in digital photogrammetry and also in automated modelling and mapping. Many approaches for matching have evolved over the years, but the problem is still unsolved in general terms. This paper describes the development of image matching techniques in photogrammetry over the past 50 years, addresses the results of some empirical accuracy studies and also provides a critical account of some of the problems that remain. Although automated approaches have quite a number of advantages, the quality of the results is still not satisfactory and, in some cases, far from acceptable. Even with the most advanced techniques, it is not yet possible to achieve the quality of results that a human operator can produce. There is an urgent need for further improvements and innovations, be it through more powerful multi-sensor approaches, thereby enlarging the information spectrum, and/or through advancements in image understanding algorithms, thus coming closer to human capabilities of reading and understanding image content. Keywords: DSM generation, empirical tests, image matching, least squares, multi-image, multiple image features Introduction Professor Ian Dowman was among the first to propose the use of fully digital systems in photogrammetry (Dowman, 1984), in this case for topographic mapping from satellite data. In any of those systems image matching is a crucial function, upon which many other follow-up products will depend. Image matching is a key component of many tasks in photogrammetry, computer vision and image analysis; it is also crucial to a wide range of applications such as navigation, guidance, automatic surveillance, robot vision, medical image analysis and to the modelling and mapping sciences. For more than 50 years, image matching has been an issue of research, development and practical implementation in software systems. Nevertheless, a critical assessment of the current status of image matching shows that the problem has not yet been solved in general terms.. The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. Blackwell Publishing Ltd Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street Malden, MA 02148, USA.

2 The Photogrammetric Record This paper aims to describe the major lines of development and achievements as they can be traced in the mapping community. With a topic so important to many disciplines and the limited space provided, it is clear that not all developments can be described and given proper credit here. There is a large body of publications on image matching in the computer vision literature; however, no particular developments, except some basic interest operators or other image analysis algorithms, have transferred successfully into photogrammetric systems. Lately, PhotoSynth (Snavely et al., 2006) has become well known for its simultaneous and automated orientation of hundreds of non-calibrated (Internet) images and its associated derivation of sparse point clouds. While this is a quite interesting development, the focus of this approach is not on image matching for surface model generation; indeed no high-quality dense and complex surface models have been shown up to now. Some other recent developments, however, show some promising matching results (Hirschmueller, 2008; Vu et al., 2009), albeit that strictly controlled tests are still missing. This contribution aims at tracing the development in image matching in photogrammetry from the middle of the 1950s until the present day. Since the author and his group of researchers have worked in this field for about 30 years and contributed a great number of publications, there will be a certain focus on this work. The chronology in developments will be structured into the following periods: the Early Years (1960s and 1970s); the New Approaches (1980s); the Time of Consolidation and Extensions (1990s); and finally the Time of Acceptance (2000s). The discussion on the development and status of image matching must also take into consideration that this technique is used for a variety of different tasks, with different prerequisites and expectations. The prime, though not exclusive, applications of image matching in our fields are for surface generation, tie and control point measurement for orientation and triangulation, industrial quality control (targeted and non-targeted points), feature (edge) extraction and feature/object tracking. The type, size and quality of images used (satellite, aerial, terrestrial) and the expected accuracies of the results vary greatly. Accordingly, a critical analysis will lead to different results. Therefore, the arguments will be based on the achievable results and not necessarily on those required by a certain application. Research in this context is not only about developing some new methodology, but also about providing a clear understanding of its properties, which means pushing this methodology to its performance limits in order to gain an insight into its potential and limitations. Three Basic Matching Techniques Image matching has been a major research issue in computer vision and digital photogrammetry for many years; accordingly, many different approaches have evolved. Three basic matching techniques can be distinguished: (a) intensity-based; (b) feature-based; and (c) relational. In intensity-based matching the original, or slightly modified (enhanced), image data is used in the form of a matrix of grey values. The most prominent methods are cross-correlation and least squares matching (LS matching or LSM), which are also called area-based matching. They provide sub-pixel accuracy, in extreme cases 1/10 pixel and even better. LS matching is a highly non-linear process and therefore requires very good approximate values. The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 37

3 Gruen. Development and status of image matching in photogrammetry Feature-based matching requires, firstly, the extraction of basic image features, such as patches, corners, junctions, edges and so on. In a second step, matching is performed between these features. Features are sometimes, but not always, more stable with regard to reflectance characteristics. On the other hand, information that is lost during the feature extraction phase can no longer be recovered. Some methods provide for sub-pixel accuracy, but not at the level of the intensity-based methods. Feature-based matching has been performed with: (1) relaxation; (2) dynamic programming; (3) robust estimation; (4) cross-correlation; and (5) graph matching. The solution space may be reduced by constraints such as: (1) use of epipolar images; (2) use of more than two images; (3) limits on the magnitude of changes in parallax; (4) a priori modelling of objects (coarse description of object); (5) hierarchical coarse-to-fine strategies; (6) best-first strategies, using features sequentially, according to the relevance of their information content; (7) thin-to-thick or thick-to-thin strategies (either starting with just a few saved match points as a skeleton and densifying, or starting with a dense point field and thinning out by blunder detection); and (8) observation of behaviour of parallaxes (inflections are not allowed). Relational matching uses geometric or other relations between features and structures (combination of features). Correspondence is established by tree-search techniques. These methods are not very accurate but are usually robust; they do not require good approximations. Their use in digital photogrammetry for digital terrain model (DTM) generation is rather scarce. There are several more or less exhaustive descriptions of these various techniques available (Lemmens, 1988; Baltsavias, 1991, Chap. 3). The Early Years (1960s and 1970s) Image matching was first introduced in the early 1950s. It started as an analogue procedure using electrical circuits for solving the matching equations (see, for example, Hobrough, 1959). A good survey of the very early efforts of analogue cross-correlation is given in Hobrough (1965). It clearly shows that equipment manufacturers were the driving force behind the development, rather than university groups. A particularly successful and much discussed system was the Gestalt Photomapper GPM I and GPM II (Kelly et al., 1977; Alberich, 1985). In Mikhail et al. (1978), Fred Doyle reports the experiences of the USGS with the GPM. They used it only with small scale photography (1:80 000) for orthophoto production and reported a height accuracy of 5 m. Hobrough (1965) already lists two attempts at digital correlation (Williams, 1959; DeMeter, 1963). What is amazing is the great optimism that accompanied these developments. The automated mapping problem was considered to be practically solved; Hobrough (1965) states: Within the next three to five years many of the present automation programs should be completed and several of the conventional photogrammetric operations will probably be automated on a more or less routine basis. 38 The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd

4 The Photogrammetric Record By the middle of the 1970s, the increasing computational power of digital computers allowed for the fully digital treatment of the matching problem. Helava (1972) already discussed the epipolar principle in matching and a little later (Helava, 1976, 1978) presented an in-depth study of auto- and cross-correlation based on linear system theory. He listed 11 crosscorrelation principles and was critical of the traditional cross-correlation approach, which is insufficient for photogrammetric stereo applications and requires image data shaping to account for the phase differences between different image frequencies and within each frequency. The image power spectrum also reveals cases of multiple correlation peaks, whereby the highest peak does not necessarily represent the correct solution. Helava also referred to the new AS-11B-X system, an automated stereo mapper, which used parallel processing of neighbouring terrain profiles in order to perform the matching of one aerial model within 10 minutes, a speed that is dreamt of even today. For more technical details of the system, see Scarano and Brumm (1976). In an interesting variant, Masry (1974) used crosscorrelation on an analytical plotter with epipolar constraints for change detection. Dowman and Haggag (1977) also worked on this method. By the early 1980s, the literature on image analysis and matching had grown tremendously. Therefore, only very few publications can be referenced here, most of which possess an overview character. The approaches and achievements within the photogrammetric community of those early years are described in Makarovic (1980), Konecny and Pape (1981) and Baltsavias (1984). Computer vision scientists also had an early and significant impact on matching techniques (as an example, see Baker and Binford, 1981). Overviews and summaries of the state-of-the-art methods can be found in Aggarwal et al. (1977), Andrews (1978), Bernstein (1978) and Chellappa and Sawchuk (1985). It was obvious for those closely involved in matching problems that the existing approaches, which were mostly based on cross-correlation, had significant deficiencies. Helava (1976) states that the human operator is far superior and addresses with this remark the lack of suitable image understanding algorithms. The characteristics of cross-correlation were well understood by the early 1980s and its deficiencies were identified: (1) discrepancy between conjugate images, caused by geometrical distortions (terrain slope, height differences, positional and attitude differences of sensors), radiometric problems (illumination, reflectance, varying material properties) and imaging artefacts; (2) discretisation of trial step size; and (3) lack of good methods for the assessment of results, figure-of-merit (quality of match). The shortcomings of this class of image matching methods finally, after early excitement and false predictions, caused a slow-down in the development of operational automated correlation systems. This problem was addressed at a panel session at the annual ASP Convention in Denver, Colorado in March Five short articles in Photogrammetric Engineering & Remote Sensing (PE&RS, 1983) reflect a part of this discussion. Cross-correlation cannot respond appropriately to a number of facts that are inseparably related to stereo-images of three-dimensional and sometimes even two-dimensional objects. The conjugate images created under the laws of perspective projection might differ considerably from each other. Terrain slope, height differences, and positional and attitude differences of the sensors cause geometrical distortions. Illumination and reflectance conditions might distort the images radiometrically. Under certain circumstances, this may even trigger a geometrical displacement in the matching. Noise from the electrical components and the The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 39

5 Gruen. Development and status of image matching in photogrammetry sampling rate (pixel size) could also influence both the geometric and the radiometric correspondence of the images. Cross-correlation is very simple conceptually, easy to implement and computationally fast. The main problem with conventional cross-correlation is that it allows only for two shift parameters between template and patch. Rotations, scale and other deformations between template and patch cannot be accommodated. Therefore, the following rules should be observed when applying cross-correlation: use it only with epipolar images and use small patch sizes. Cross-correlation works well and is fast if the patches to be matched contain sufficient signal without too much high-frequency content, and if geometric and radiometric distortions are kept to a minimum. Both conditions are not often encountered in aerial and terrestrial images. On the other hand, in satellite images, due to their smaller image scale, these problems are less influential and cross-correlation is more likely to succeed. Therefore, some efforts were made very early on to design matching techniques that are more efficient than crosscorrelation. Concepts that were suggested by the artificial intelligence community included: first- and second-order derivative matching; relaxation methods; segmentation and graph structure matching; transform ( Hough transform ) matching; and feature (edge) matching. Thus, a tendency to switch from area-based to edge-based analysis could be observed. However, Rosenfeld (1984) remarked, in an excellent critical review, that these new methods also did not solve the problems referred to above. It is important to note that the concept of epipolar line matching, for the purpose of reduction of the matching solution space and thus the number of false matches, was already proposed by Helava (1972). Panton (1978) clearly expressed the need for epipolar line constraints, patch shaping, algorithmic tuning, reliability monitoring and even parallel processing, by showing the first implementations of these concepts. He demonstrated that parallel processing can result in amazing performance: 270 match points per second, 35 minutes for a full stereomodel consisting of points on a CDC 1700 minicomputer. Compared to the single processor mainframe computer CDC 6400 this gave an increase in speed by a factor of 34. In a much later study, Zheltov and Sibiryakov (1997) have shown that cross-correlation can be modified into a version that takes care of all six parameters of an affine transformation between template and patch, albeit at higher computational expense. It also has been shown that this modified version is equivalent to least squares matching. New Approaches (1980s) The 1980s saw a long period of very active development of new, more powerful, matching approaches. Originally, the driving force behind the new developments was the equipment industry. Now researchers from universities took charge. At the same time, the system manufacturers started to offer software solutions for digital matching that were incorporated in photogrammetric equipment, firstly in analytical plotters and later in digital stations. Probably the most significant contribution, the least squares matching technique (LS matching, LSM), was developed in the early 1980s. Due to its flexibility and accuracy, it has turned out to have a major impact on image matching, with many extensions, and is currently used in many digital photogrammetric matching tasks. Early investigations were reported by Förstner (1982), Ackermann (1984) and Pertl (1984). The author also investigated this technique in 1982 as adaptive least squares matching (ALSM). The method was called adaptive because it can be executed in a self-tuning mode, 40 The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd

6 meaning the parameter set to be estimated can be corrected automatically in order to obtain a most appropriate estimation model set-up with respect to the specific signal content of the patches to be matched. Early concepts and software were developed in a project funded by Helava Associates, Inc. Tests were performed with synthetic and real images. The method was found to be of great potential for a variety of image- and template-matching problems (Baltsavias, 1984; Grün, 1984; Gruen and Baltsavias, 1984). If properly used, least squares matching combines the advantages of area-based and edgebased matching. The basic equations are set up in the context of a statistical estimation model. The estimation itself is performed as least squares estimation. The familiar apparatus of the least squares approach with respect to parameter estimation and hypothesis testing can be favourably utilised. Precision and reliability measures are readily available and allow an assessment of the quality of the match in a better way than is feasible with other matching techniques. Algorithmic, computational and numerical aspects can also be studied in a wellknown environment. ALSM has great potential in different respects, as recognised from the very beginning and reiterated here: (a) high matching accuracy; (b) geometrical/stochastical constraints: stabilisation, reliability, speed; (c) multi-image matching (reliability); (d) simultaneous matching/point positioning; (e) multi-patch matching: neighbourhood conditions; (f) multispectral, multitemporal matching; (g) monitoring of quality (precision, reliability); (h) simultaneous image reshaping, radiometric adjustment; (i) combination of area-based and edge-based analysis; (j) usable in hierarchical mode ( coarse-to-fine ); (k) usable as derivative-operator-based matching procedure (first-order slope variables, second order); (l) rule-based matching: patch selection (good signal content); (m) incomplete data patches (for example, triggered by occlusions); (n) (o) (p) The Photogrammetric Record computational performance: parallel implementation possible; usable for pattern recognition (template matching), feature extraction, image feature measurement (fiducials, tie points, control points), change detection, line following; and general matching technique (beyond images): DTM/DSM analysis/co-registration, image/map registration. A comprehensive description of the basic algorithm and its multiphoto geometrically constrained (MPGC) extension and many results are given in Baltsavias (1991). The quality of a matching procedure depends mainly on the type and content of the image signal. Given the images of an object, there is not much room for improvement of the signal. Very often there is, however, additional information available that could support the matching. Important categories of information are geometrical and radiometric conditions. They relate to the imaging geometry of the sensor, orientation and positional data of the sensor, image feature radiometry and geometric constraints, and to object constraints. They have to be set up as linear or linearised observation equations in the least squares context and are, as such, added to the observation equations for the grey values of the pixels. The resulting hybrid system is of the combined adjustment type. It leads not only to a much improved matching procedure but, in The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 41

7 Gruen. Development and status of image matching in photogrammetry addition, provides for a simultaneous matching/object point positioning technique. Thus, the two-stage process of image measurement object positioning is replaced by a one-stage solution that is capable of utilising all available radiometric and geometric information at once. The MPGC technique offers considerable advantages with respect to precision and reliability. Because of the use of all the geometric information available and the internal consistency of the algorithm, the success rate increases and many problematic situations are signalised: (1) The use of the geometrical constraints increases the convergence radius and rate because the search is one-dimensional. The use of multiple scenes has the same effect. In addition, the search is constrained with respect to direction and step size, so that image patches subject to small displacements can support those that are subject to larger displacements. (2) In many cases, occlusions do not prevent correct convergence. The less occluded patches beneficially influence those that are more occluded. The quality measures of the algorithm allow for the detection of occlusions. (3) Multiple solutions are drastically reduced because of the conditional onedimensional search. Mismatches can be detected unless all image patches hit false maxima along epipolar lines simultaneously, which would be a very rare case. Other extensions are summarised in Gruen (1996b). In the following some modifications and extensions are addressed, which make this approach so powerful and usable in a variety of different forms. In particular, the following stages of algorithmic development are distinguished: (1) Stereo (two-image) adaptive least squares matching (ALSM) (Gruen, 1985b). (2) Multiphoto geometrically constrained (MPGC) matching (Gruen, 1985a; Gruen and Baltsavias, 1985, 1988b; Baltsavias, 1991): (a) collinearity constraint; (b) forward intersection constraint (interior and exterior orientations known; X, Y, Z estimated simultaneously); (c) epipolar constraint (interior and exterior orientations known; X, Y, Z derived in a separate step); and (d) bundle constraint (interior and exterior elements are simultaneously estimated with X, Y, Z coordinates). (3) Digital surface model (DSM) constraints: (a) XY constraint (Gruen, 1985a; interior and exterior elements known; given X, Y, only Z is estimated simultaneously; this also became known as the vertical line locus method); and (b) Z (contour) constraint (Gruen, 1985a; Gruen and Baltsavias, 1986; interior and exterior elements known; given Z, only X, Y are estimated simultaneously; this is equivalent to drawing contours from a stereomodel or a multi-image arrangement). (4) Image feature constraints: (a) edge constraint (Gruen and Stallmann, 1991). (5) Globally enforced least squares matching: (a) multiple patch matching; (b) 2D patches (Gruen, 1985a); (c) 3D (volume element) patches (Maas et al., 1994); (d) linear feature extraction with LS template matching (Gruen and Agouris, 1994); (e) linear feature extraction with LSB-snakes (Gruen and Li, 1997); 42 The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd

8 The Photogrammetric Record (f) object-space-oriented LSM (Gruen and Zhang, 2002; Zhang, 2005); and (g) neighbourhood constraints by stochastic relaxation (Gruen and Zhang, 2002; Zhang, 2005). (6) 3D surface matching: (a) 3D surface and space curve matching (Gruen and Akca, 2005; Akca, 2007). A particular type of image feature constraint is the image edge constraint. Whenever edges have to be measured, such as breaklines in DTMs, this may constitute an appropriate solution. Essentially MPGC is an area-based matching technique. For high accuracy edge matching, the method is transformed into a combination of an area-based and feature-based technique. This is achieved by introducing, as a reference template, a synthetic (or real) edge pattern, which is to be matched with the actual image edges. Compared to the conventional feature-based matching techniques, this method does not require the extraction of image edges, but matching is done directly by using the original grey value edges. For algorithmic details, see Gruen and Stallmann (1991). An efficient automatic measurement procedure can be realised via implementation of a tracking technique, which tracks the edges either in object or in image space. If larger image regions have to be processed by matching, image patches with low or no signal content pose a serious problem. For such cases the technique of globally enforced least squares matching has been developed. Here the basic idea is to establish geometrical neighbourhood conditions between adjacent patches in order to give stabilising support to the weak patches by the strong ones. Ideally, the weak regions would be bridged and a stable global solution would be obtained. The first such solutions were introduced as multipoint matching or multi-patch matching techniques. The introduction of neighbourhood constraints leads to a simultaneous solution for all patches. Thus, a full image format can be processed in one sweep. Early approaches to this concept have been presented by Gruen (1985a), Rauhala (1986), Rosenholm (1986) and Li (1989). Another globally enforced technique is object space oriented least squares matching as introduced by Wrobel (1987), Ebner and Heipke (1988) and Helava (1988b). It represents the most generalised approach to least squares matching. Due to its complexity with respect to implementation and handling, it has been used only occasionally and under laboratory conditions (Kempa and Schlueter, 1993). In the 1980s, image matching was mostly used in close-range applications. Examples include: camera calibration (Beyer, 1987, 1992); human face measurements (Gruen and Baltsavias, 1988a); object tracking (Baltsavias and Stallmann, 1990); and industrial quality control (Gruen and Stallmann, 1991). Often it was applied in the form of template matching (for example, Lue et al., 1987). Many more tests and applications are reported in Gruen (1988) and Baltsavias (1991), the latter noting that scientific investigations with aerial imagery were rather scarce. Otto and Chau (1988) demonstrated an early application to SPOT satellite stereoimages using the techniques of region growing. For the results of an ISPRS WGIII/4 test on image matching, see Guelch (1988). The many activities in algorithmic development were occasionally also accompanied by accuracy studies. Schewe and Förstner (1986) reported industrial car measurements on the analytical plotter Planicomp, with an accuracy of 0Æ05 to 0Æ2 pixels. Gruen and Baltsavias (1986) derived DTMs from 1:5300 scale aerial images with a height error of 0Æ02 to 0Æ04% of flying height for natural points. This compares with an accuracy of manual measurements of signalised points of 0Æ003% of flying height (Trinder, 1986). Rosenholm (1986) processed aerial images at scales between 1:4000 and 1: with an accuracy of 0Æ4 to 0Æ6 pixels. The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 43

9 Gruen. Development and status of image matching in photogrammetry In summary, the following accuracies have been attained with LSM: (1) laboratory test, targeted points: 0Æ01 to 0Æ02 pixels; (2) close-range applications, project conditions, targeted points: 0Æ1 to0æ2pixels; and (3) aerial photogrammetry, natural points: 0Æ3to0Æ5 pixels. In parallel to these many research and development efforts, commercial systems came onto the market. Correlators were implemented on analytical plotters, for example, by Kern in the form of the vertical line locus method (Bethel, 1986; Almroth and Hendriks, 1987) or least squares matching (Pertl, 1984) and on mono-comparators (Helava, 1988a, with least squares matching). From its very beginnings, image matching was not isolated from practice, but always developed with the aim of integrating it into a photogrammetric processing system. As explained earlier, these efforts go back to the early 1950s and have been pursued ever since. In addition, the ISP (International Society of Photogrammetry) had established a working group on Automated and Analytical Instruments in 1969, well before analytical plotters entered the civilian market. In 1976, at the ISP Helsinki Congress, this working group was split into two and a new working group entitled Automated Instruments and Systems was formed. The International Archives of Photogrammetry, Volume 24 (Proceedings of ISPRS Technical Commission II Symposium held in Ottawa in 1982) contains a number of papers addressing this issue. Of particular interest is Case (1982). He describes the concept of a fully digital station DSCC (digital comparator/correlator) and gives detailed numbers on expected performance parameters. This is the time when university groups were also investigating fully digital systems and a number of studies and implementations were revealed (Albertz and Koenig, 1984; Dowman, 1984; Gruen and Beyer, 1986, 1990; Haggren, 1986; Gruen, 1989), although not all were supported by image matching functions. A system for the processing of SPOT images was under development by Dowman et al. (1987). Four different image matching algorithms were tested, with least squares matching finally implemented (Otto and Chau, 1988). Many efforts in digital photogrammetry at that time were driven by the fact that the US Defence Mapping Agency had launched a large research and development programme with the goal of executing its mapping operations fully automatically and, thus, fully digitally by the year The first commercial photogrammetric digital station was presented as DSP1 by Kern at the ISPRS Congress in Kyoto in 1988, but at this time without a matching module (Cogan et al., 1988). Time of Consolidation and Extension (1990s) The idea of globally enforced matching was later generalised such that 3D image data-sets (voxel cuboids) could be matched. The related method was used for the measurement of laserinduced fluorescence flow fields in a technical chemistry application (Maas et al., 1994). A further modification was suggested for the extraction of linear features (Gruen and Agouris, 1994). This, again, was further developed into LSB-snakes, which combine the powerful tools of least squares estimation with the determination of energy-minimising functions (Gruen and Li, 1997). A particularly promising and successful implementation has been presented by Maas (1996). This approach is derived from the concept of MPGC matching. It is a multi-image matching technique, whereby features are searched for along epipolar lines. The computational effort grows exponentially with the number of images, but specific search strategies help to keep the computing times within reasonable bounds. Compared to MPGC matching, this is a linear procedure requiring no approximate values and no iterations. On the other hand, it is 44 The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd

10 The Photogrammetric Record essentially a searching technique for features whose locations are defined by epipolar lines. There is some room for improvement in the sense that feature attributes could be considered for matching and/or least squares area-based matching could be done as a last step. DTM Generation on Digital Workstations A large number of commercial digital systems emerged in the 1990s, but not all of them survived for very long. Only a few have made a commercial impact. Automated DTM generation, in particular for ortho-image production, is a major function of digital photogrammetric stations. Most systems and approaches work hierarchically with image pyramids. They use epipolar images rather than the original images and apply either crosscorrelation or feature-based techniques. The sampling mode is often on a regular grid in either object space or image space, or based on arbitrarily distributed points. In the last two cases, the data is often transformed into a regular DTM grid before being presented to the user. In the author s opinion, this is not an appropriate solution. DTM generation by image matching and DTM interpolation should be separated. Otherwise, the effects of both are inseparably combined and the user has no indication of the quality of either procedure. The most popular digital photogrammetric stations at that time had the following DTM software and approaches installed: (1) Leica/Helava DPW, Automated Terrain Extraction (AATE), cross-correlation (Miller and De Venecia, 1992; Zhang and Miller, 1997); (2) Zeiss PHODIS ST, Topo SURF (MATCH-T), feature based; (3) Intergraph ImageStation, MATCH-T, feature based (Krzystek, 1995); (4) PCI Geomatics, DEM extraction; (5) SOCET SET, BAE Systems digital DTM generation; and (6) VirtuoZo, cross-correlation with reshaping, global matching with probabilistic relaxation (Zhang et al., 1992, 1996). The results of studies comparing the performance of different digital stations with respect to DTM generation are referenced in Baltsavias et al. (1996), Gruen (1996a) and Smith and Smith (1996). In general, most high-end workstations delivered results of similar quality, although the underlying algorithms, strategies, robustness and ease of use of the software varied. Most systems required the user to set a large number of input parameters (up to 28 in a particular system). Even small changes in seemingly harmless parameters often led to significant alterations in the results. There was no logical visible or predictable connection between the change of parameters and the results. Major problems occurred in the case of homogeneous texture, shadows, dense vegetation (trees), dark and very steep slopes, water features, urban environments, and so on. In Gruen (1996a), the main problems were listed as: (1) lack of recognition of object edges and geomorphologically important features; (2) no bridging of regions with poor signal content; (3) insufficient handling of occlusions and shadow areas; (4) unreliable reduction of DSM to DTM; and (5) missing quality assessment; no good internal quality control. In another study (Gruen, 1999), it was shown by empirical tests that the accuracy results of automated DTM generation were worse by a factor of five or more compared with those obtained from analytical plotters. The results of yet another empirical accuracy study were published in Gruen et al. (2000). The matching software of three commercial systems was The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 45

11 Gruen. Development and status of image matching in photogrammetry tested with three different aerial image data-sets of different image scales. The rms errors achieved were worse by factors of 1Æ6 to 16 than the theoretical expectations for manual measurements. This was largely due to very large numbers of blunders. Gong et al. (2000) have also carried out an interesting assessment. The low-quality results were caused by the fact that the capabilities of existing matching algorithms had not been fully utilised. Significant improvements could be expected through: (1) use of more than two images for matching; (2) making use of all available geometrical constraints; (3) development of methods for internal quality control; (4) improvement of user interface; (5) explanation of functions of parameter settings; (6) integration of a priori knowledge (such as existing DTMs); and (7) integration of image understanding algorithms and use of multi-sensor data. Empirical, controlled testing of matching procedures and software, when applied to surface model generation, is not a simple task. The need to generate reference data-sets with an accuracy at least three times better than the expected system performance (a conservative number) indeed causes some headaches. If a system performance of 0Æ3 pixels is assumed then this translates, in the case of medium scale aerial images (20 cm footprint), into a required reference point accuracy of 2 cm. In the case of close-range applications with a footprint of 1 mm this value would be 0Æ1 mm. For satellite images there are not such stringent requirements, although when considering the latest generation with a footprint of 50 cm, problems with reference data generation are also faced. This situation is complicated by the fact that the matcher will generate hundreds of thousands, or even millions, of points per stereomodel, thus requiring a very large number of reference points. This is surely one of the reasons why little suitable empirical test data is available, even up to the present time. As a result it can be said that the matching procedures and software have developed not only without a solid theoretical basis (company approaches are usually not even published), but also without much empirical scientific control. It is up to the individual user to judge the suitability of his results not a very comforting situation. Time of Acceptance (2000 to Now) Automatic DTM/DSM generation through image matching has gained much attention in recent years. A wide variety of approaches have been developed and automatic DSM generation packages have, in the meanwhile, been commercially available on several digital photogrammetric workstations. At the turn of the century, it was still noted that commercial image matching software did not have the required performance in terms of high-quality results. This applies in particular to the processing of aerial images, but also to the recently available high-resolution stereo satellite images. Close-range applications are considered by most system manufacturers as niche markets. Therefore, no effort is made to provide suitable software and this area is left entirely to the activities of academic research groups. Although the algorithms and the matching strategies of commercial systems may differ from one another, the accuracy performance and the problems encountered are very similar in the major systems. Furthermore, the performance of commercial image matchers does not live up, by far, to the standards set by manual measurements (Gruen, 1996a, 1999; Gruen et al., 2000). The main problems in automated DTM generation are encountered with: 46 The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd

12 The Photogrammetric Record (a) little or no texture; (b) distinct object discontinuities; (c) local object patches are not sufficient approximations of planar faces; (d) repetitive objects; (e) occlusions; (f) moving objects, including shadows; (g) multi-layered and transparent objects; (h) radiometric artefacts such as specular reflections; and (i) reduction from DSM to DTM. In the year 2000, the author started development cooperation with Starlabo Inc., Tokyo. As part of a larger software package, a new matching approach and module was developed for an aerial three-line scanner (TLS) system. It was later modified to also be able to handle satellite images (SAT-PP) and terrestrial close-range cases (CLORAMA). The following refers to descriptions of the TLS system without, however, any restriction in the generality of the chosen algorithms. The TLS matcher aims to generate DSMs by considering specifically the problems (a) to (f) above. The matcher is described in detail in several publications (for example, Gruen and Zhang, 2002; Zhang, 2005). The raw level TLS images were used together with the given, or previously triangulated, orientation elements. After the generation of image pyramids, the matcher uses three kinds of image features, namely, general feature points, edge points and grid points. A triangular irregular network (TIN) based DSM is constructed from the matched points on each level of the pyramid, which in turn is used in the subsequent pyramid level for approximation and adaptive computation of the matching parameters. Finally, the modified MPGC matching is used optionally to achieve more accurate results for all the matched features. Among the usual matching techniques, area-based matching (ABM) and feature-based matching (FBM) are the two main ones applied in automatic DSM generation, but additionally relational matching is sometimes used. All basic matching techniques have advantages and disadvantages with respect to the problems presented above. The key to successful matching is an appropriate matching strategy, making use of all available and explicit knowledge concerning the sensor model, network structure and image content. However, even then the lack of an image understanding capability will lead to problems, whose impact must be judged by the project specifications. The matching approach is a hybrid method that combines ABM and relational matching. It uses a coarse-to-fine hierarchical strategy with a combination of several image matching algorithms and automatic quality control. ABM (both in the form of a modified crosscorrelation and least squares matching) is employed to match feature points and grid points (see also Hsia and Newton, 1999). Generally, the performance and success rate of ABM mainly depends on: the existence of sufficient image texture; the quality of the approximations and a set of matching parameters, such as the matching window size; the search distance; and the acceptance threshold for the correlation coefficient. How to select a set of correct matching parameters is problematic, because the requirements for these parameter values are conflicting. These matching parameters are functions of many factors, including terrain type, image texture, image scale, disparity variations and image noise. The TLS matcher uses a set of adaptively determined matching parameters. This is done by analysing the results of the higher level image pyramid matching and using them at the current pyramid level. The performance of ABM is not good if there is insufficient image texture, in the case of repetitive patterns, and at surface discontinuities. Unfortunately, these problems are very typical of large scale images as provided by TLS. In the first case, because of missing points, The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 47

13 Gruen. Development and status of image matching in photogrammetry the ABM may lead to holes in the DSM. To overcome this problem, a global image matching technique, based on probabilistic relaxation (Hancock and Kittler, 1990), is employed to match grid points in order to bridge the poor texture areas. This relational matching uses local smoothness constraints. In the last case, ABM generates smoothing effects at surface discontinuities. ABM is also employed to match the edges located on such discontinuities, but the matched edges are used as breaklines to control the weights of the surface smoothness constraints in the global image matching procedure. As such, they prohibit the smoothness constraints from crossing the edges. The quasi-epipolar curves derived from the TLS sensor model are used to restrict the search range to only one direction. The residual y parallax grid is used to compensate some of the errors in the raw image data. In summary, the matching approach can be characterised by the following aspects: (1) Multiple image matching and different matching algorithms. A new flexible and robust matching algorithm, the geometrically constrained cross-correlation (GC 3 ) method, has been developed in order to take advantage of multiple images. The algorithm is based on the concept of multi-image matching guided from the object space and allows the reconstruction of 3D objects by matching all available images simultaneously, without having to match all individual stereopairs separately and merge the results. Besides this special form of cross-correlation, LSM is also used as an option. (2) Matching with multiple primitives. More robust hybrid image matching algorithms have been developed by taking advantage of both ABM and FBM techniques and utilising both local and global image information. In particular, an edge matching method is combined with a grid point matching method through a probability relaxation-based relational matching process. The use of edges leads to the preservation of surface discontinuities, while grid points bridge areas with little or no texture. (3) Self-tuning matching parameters. The adaptive determination of the matching parameters results in a higher success rate and fewer mismatches. These parameters include the size of the correlation window, the search distance and the correlation threshold values. This is done by analysing the matching results at the previous image pyramid level and using them at the current level. (4) High matching redundancy. With this matching approach, highly redundant matching is achieved, so that points and edges can be generated. Highly redundant matching results are suitable for representing very steep and rough terrain and allow the terrain microstructures and surface discontinuities to be well preserved. Moreover, this high redundancy also allows for better automatic blunder detection. (5) Efficient surface modelling. The object surface is modelled by a TIN generated by a constrained Delauney triangulation of the matched points and edges. A TIN is suitable for surface modelling because it integrates all the original matching results, including points and edge features, without any interpolation. It is adapted to describe complex terrain types that contain many surface microstructures and discontinuities. (6) Coarse-to-fine hierarchical strategy. The algorithm works in a coarse-to-fine multiresolution image pyramid structure, and obtains intermediate DSMs at multiple resolutions. Matches on low-resolution images serve as approximations to restrict the search space and to adaptively compute the matching parameters for the subsequent levels. Results of Controlled Accuracy Tests Results, based on controlled tests, have been published at several times, such as in Gruen and Zhang (2002) for aerial images. In Eisenbeiss et al. (2005) and Lambers et al. (2007) the 48 The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd

14 The Photogrammetric Record results of a comparison of image matching from aerial UAV images (using SAT-PP) with terrestrial laser scan data showed no clear superiority of either technique. Image matching, however, turned out to be of greater generality and more flexible use. Recently, the German Society of Photogrammetry and Remote Sensing (DGPF) conducted a test on evaluation of digital photogrammetric aerial camera systems. As part of this test the accuracy of DSMs, derived automatically, was also investigated. Some of the results have been reported in Wolff (2009). The key problem with such tests, which were done with aerial images with footprints of 8 and 20 cm, is the generation of sufficiently good reference data. Tests in close range applications have been reported, for example, by Remondino et al. (2008, 2009) and Remondino and Menna (2008), although not always supported by accurate reference data. In some cases, terrestrial laser scanning was compared with image matching results, with no clear indication as to which technique would deliver the better results. The performance of the matching software SAT-PP for DSM generation has been verified extensively with several high-resolution satellite imagery data-sets, such as SPOT-5, IKONOS, QuickBird, ALOS/PRISM, Cartosat-1 and WorldView-1, over different terrain types; these include hilly and rugged mountainous areas, and rural, suburban and urban areas. A detailed analysis of the results of IKONOS was presented in Gruen et al. (2005) and Zhang and Gruen (2006). Other processing and evaluation results of IKONOS and SPOT-5 HRS/HRG can be found in Zhang and Gruen (2004), Poli et al. (2004), Baltsavias et al. (2006), Poon et al. (2005) and Crespi et al. (2008). ALOS/PRISM results are described in Gruen and Wolff (2007) whilst WorldView-1 results can be found in Poli et al. (2009). A general summary of satellite imagematching results is given in Wolff and Gruen (2008). In a special application, the performance of image matching with respect to the generation of 3D tree canopy models from aerial images was tested and compared to lidar data. Image matching results (computed with SAT-PP) turned out to give better results (Baltsavias et al., 2008). What has been observed in all the controlled tests is that an accuracy of 1 to 2 pixels can be achieved in DSM generation, but there are many blunders in the results. These blunders are usually not single spikes (which could be easily detected), but rather groups of gross errors that act like local systematic errors. Therefore, they are not easily detected automatically. In future research and development activities emphasis should be on the understanding of the underlying reasons for these blunders and on the development of techniques to avoid them. The main sources of errors are: (1) object features (edges, height differences, steepness of slopes, repetitive objects, role of vegetation); (2) illumination and reflectance properties (lack of image texture, shadows, specular reflections), image quality (signal-to-noise ratio, image artefacts); and (3) network problems (insufficient design, partial occlusions). Image matching must be seen in the context of automation. The potential advantages of automation are: (1) increased accuracy; (2) reduced equipment costs; (3) increased throughput; (4) faster availability of results (online capability); (5) new kinds of products; and (6) better quality products. The Photogrammetric Record Ó 2012 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd 49