Image matching, point transfer, DSM generation

Transcription

1 Image matching, point transfer, DSM generation Dr. Maria Pateraki Department of Rural and Surveying Engineering Aristotle University of Thessaloniki tel: , URL: Introduction Definition match: A suitable conjunction or pairing (Oxford English Dictionary) The matching problem is also referred to as the correspondence problem. The data sets can be: images maps object models GIS data What is image matching? Finding (measuring) automatically conjugate points in two or more images. Synonyms alternative terms Image matching ~ automatic stereo matching ~ correlation ~ correspondence problem 1

2 Introduction What does image matching needs? digital stereo imagery (>=2) grey value variations (good texture) smooth object surface, small slopes non-moving objects Matching strategy for DSM generation is similar to ARO / AAT, plus: dense surface description hierarchical procedure with 3D points as result integration of matching and surface interpolation Potential problem areas urban areas forests water bodies Introduction Where is image matching being used (in which procedures)? target measurements interior orientation relative and absolute orientation of steropairs aerotriangulation DSM generation image Fusion (or Registration). template search 2

3 Matching problematic areas Matching tools Area based matching High accuracy potential but smoothing effects Sensitivity to discontinuity Methods: a. Cross correlation b. Least squares matching Feature based matching Use of abstract image representation derived from feature extraction algorithm Avoid smoothing effects No subpixel precision Methods: a. points b. edges c. blobs Relational matching Based on relationship between objects (distances, angles, collinearity) 3

4 Area based matching Intensities are used as input to solve the correspondence problem Geometric transformation (object surface): a) Translation b) Translation rotation c) Affine (6 parameters - locally planar surface) d) Projective (8 parameters globally planar surface) e) Smooth (smooth surface without occlusions) f) Piecewise smooth (possibly with occlusions) Similarity measure (most frequently applied) a) Distance (Haussdorf, Euclidean) b) Sum of products (2 nd moments, covariance) c) Sum of squared differences (least squares) d) Sum of absolute differences e) Normalized cross correlation Algorithmic solution a) Sequential b) Heuristic c) Iterative d) Dynamic programming Area based matching - Cross Correlation Measure the similarity of the template with the matching window by computing the correlation factor. Highest cross correlation coefficient in the search image NCC = normalized cross correlation coefficient Std. Dev. of f Std. Dev. of g Covariance of f and g 4

5 Area based matching - Cross Correlation Geometric differences are modeled only by translation Radiometric differences exist only due to brightness and contrast No generalization for Multi-image matching No correlation between the two image patches NCC = 0 Identical image patches NCC = 1 Inverse correlation NCC = -1 (positive and negative) Area based matching - Cross Correlation Problematic cases (a) Little contrast low reliability of the match (b) Multiple solutions Repetitive patterns (c) Lower correlation than (a), But possibly better match a = flatness or the angle between the tangents to the parabola next to the maximum 5

6 Area based matching - Cross Correlation Different cases (Putockova 2004) Area based matching Multi-pass Cross Correlation 3 passes of matching using different parameters NCC as similarity measure improve coarse approximations for position Template Search search range x search range x search range x search range y search range y search range y Large patch! aims at reliability of the solution! less sensitive to noise, occlusions, multiple solutions Small patch! aims at precision of the solution! better preserves height discontinuities 6

7 Area based matching - LSM Minimize the grey level differences between template and search image. Position and shape of the matching window are parameters to be determined in the adjustment process Using Affine transformation (locally planar surface) 2 shifts, 2 rotations, 2 scales, 1 radiometric parameter template search search Precision: Can reach 1/25 pixel It requires GOOD initial approximations Area based matching - LSM 7

8 Area based matching - LSM Area based matching - LSM The matching is ideal if : f(x,y) = g (x,y) Template = Search Because of random effects (noise), a noise vector e(x,y) is added: f(x,y) e(x,y) = g (x,y) non-linear observation equation f(x,y) = vector of observations (in the template image) g(x,y) = function model Non linear system => linearization of the equations =>Approximations required => Iteration necessary Taylor linearization: 8

9 Area based matching - LSM Parameter vector: Least squares estimation (Gauss-Markov model) l - e = Ax l = f(x,y) - g0(x,y) e = error vector A = design matrix (coefficients of the parameters) = solution vector for transformation v = residuals (grey-values differences Template Search) Area based matching - LSM 9

10 Area based matching - LSM Convergence (e.g. the residual vector is close to 0) vs. divergence Possible cases in LSM: - FAST AND CORRECT CONVERGENCE - SLOWLY BUT CORRECT CONVERGENCE - CONVERGENCE TO FALSE POSITION - NO CONVERGENCE (singular matrix) Fast convergence Slow convergence, partial divergence Area based matching Multiphoto Geometrically Constrained LSM In general Geometrical Constraints are used to: strengthen matching improve computational stability convergency Pairwise or simultaneous using all available images. Geometrical constraints: Collinearity equations 10

11 Area based matching Multiphoto Geometrically Constrained LSM Pairwise or simultaneous using all available images. Geometrical constraints: Collinearity equations 1 Template and n Search images (Patches) n1 collinearity conditions for object point P Parameters to be estimated: Shape and position of the patch in each search image Object coordinates, Y, Z Area based matching Multiphoto Geometrically Constrained LSM Least squares estimation (Gauss-Markov model) solution vector residual vector for intensity observations residual vector for collinearity constraint observations variance factor 11

12 Area based matching NCC vs. LSM Positive and negative properties of NCC and LSM Feature based matching (FBM) Feature based matching uses symbolic descriptions for establishing correspondences The representation using features is invariant with respect to distortions (illumination, reflectance, geometry) Density of features is usually not sufficient. Usually FBM are combined with ABM methods to improve surface representation Features a) Interest points Type: corners, center of circular features Operators: contour-, intensity-based and parametric- model based Intensity - based methods: Easy to implement, less complex, better localization accuracy E.g. Foerstner, Harris, Moravec, Susan, etc. b) Edges Detecting edge pixels (or edgels) -> Linking edge pixels into edges -> Grouping edges Operators: gradient- based, surface fitting, model matching, Laplacian of Gaussian and momentbased algorithm 12

13 Feature extraction Edgel extraction - Canny extractor 1. Smoothing: using a gaussian smoothing operator 2. Gradient 3. Non-maximum suppression 4. Hysteresis threshold dx = dy = Feature extraction Canny Grad. Thresh 1 13

14 Feature extraction Points Foerstner Harris Susan Edgels Canny Grad. Thresh Susan Feature extraction All Edges Straight Edges Straight Edges > 10 pixels 14

15 Feature extraction Edgels All Edges Long straight Edges Feature extraction Points Edgels Comparison of operators (qualitative analysis) Foerstner Susan Harris Foerstner Susan Canny Grad. Thresh. Completeness Localization Distinct features 0-0 Reliability Time & memory performance - - Method Remarks: -Thresholds are not comparable - No quantitative analysis - Performance is related to implementation 15

16 Edgel matching example modified LSM Patch analysis derive interest values for signal ellipse (qa, qb ) direction (φ) Method A: - pre-rotation of patch with respect to the template - small patch size vs. standart LSM: less iterations, decrease of oscillations for scales and shears Method B: - pre-rotation of template and patch - thin rectangle patch (long dimension is aligned with the edge direction vs. standart LSM: false matching could be almost eliminated, as long as the derived edge directions in template and patch are reliable. Edge matching example modified LSM Patch dimension expanded to contain whole edge - Center of template at the middle point of the edge - Pixels off-the-edge assigned a small weight (0.1) Advantages of method: - fast convergence (3-7 iterations) - less good approximations are required - danger of multiple solutions is reduced - safer determination of rotation Nadir Backward Forward 16

17 Matching options according to features Hierarchical processing Use of image pyramids: small parallaxes small search range Use of doublets: - avoid propagation of mismatches - reduce processing time 17

18 Interior orientation determine parameters of a 2-D transformation between pixel and image coordinate system (usually affine) Example (a): approximate position of each fiducial use fiducial as template cross correlation Example (b): precisely locate fiducial centre cross correlation & subpixel fit (e.g. LSM) compute transformation parameters robust least squares adjustment UL UR LL LR Interior orientation Examples of problematic fiducials poor contrast (center background) poor contrast (fiducial image) scratches 18

19 Interior orientation Current status included in most DPWS autonomous, fast, accurate often manual measurements only automatic estimation is problematic (reliability) for images acquired from digital cameras Automatic Interior Orientation is not required, because the projection centre remains stable with respect to the image Relative orientation recovery of the position and orientation of one imaging system relative to another from correspondences between 5 or more ray pairs Option (a): tie point distribution in six von-gruber positions only most important areas for geometric stability similar to analytical relative orientation processing of relatively few data only - patches possibly unsuitable for matching - no full exploitation of available information Option (b): tie point distribution in the whole overlapping area avoids disadvantages of option (a) - higher computational cost (a) (b) 19

20 Relative orientation ideal tie point positions Point transfer search space step 1: transformation into object space step 2: computation of predicted position size of search space depends on: accuracy of initial values for surface height image orientations location of selected position 20

21 Point transfer search space - Main source of uncertainty results from the terrain surface Maximum Error Δpx in the predicted x-parallax in dependecy of : 1. Base to height ratio B/Z 2. focal length c 3. tolerance Δφ in the two angles φ of the left and the right camera, 4. the tolerance ΔZ/Z of the height Z related to Z, 5. the tolerance ΔB/B of the basis related to B Δpx = 2 c {1 1/4 (B/Z) } Δφ c (B/Z) ΔZ/Z c (B/Z) ΔB/B - In mountainous terrain the maximum error can reach nearly 6 cm. Increasing overlap, thus reducing B/Z, also reduces the prediction errors (80%-90%) Most approaches for point transfer contain module for generating a rough digital elevation model. Point transfer search space Example computation of search space Normal case Different overlap in and across base direction Z has largest influence on search space size (base direction only) influence of exterior orientation can be further reduced by GPS / IMU 21

22 Relative orientation tie point selection criteria located in stable areas (no vegetation, water, moving objects, shadows) the position of the points should be accurate enough for the final adjustment process evenly distributed across the whole image located in open, horizontal areas in order to be visible in both images lie in as many overlapping images as possible, for stability reasons, be distinct for supporting efficient matching, More robust selection when knowledge about the scene content and geometry exist (image orientation, object surface) Relative orientation 22

23 Tie points on moving objects Tie points on moving objects

24 Tie points and repetitive texture Tie points and repetitive texture 24

25 Relative orientation Current status various systems available autonomous black box systems accuracy: between 0.2 and 0.4 pixel large redundancy ( points per model) much faster than manual measurements Problems in Automatic Realtive Orientation and Point Transfer - large and unknown rotation differences in k (> 20 degrees) - large scale differences (> 1.5) - close range imagery, in particular convergent imagery Relative orientation accuracy, reliability and redundancy replace intelligence by redundancy (Ackermann 1996), at least partly compensates for lack of knowledge of scene content (not of geometry!) use of many features per image ( 100, high redundancy) low accuracy σ0 of a single observation can be compensated by high redundancy (large n) single measurement repeated measurements high reliability and easy methods for blunder detection (robust adjustment) 25

26 Automatic Aerial Triangulation Automatic Aerial Triangulation Matching Strategy tie point distribution tie point selection accuracy, reliability, and redundancy search space hierarchical processing (image pyramids) 26

27 Automatic Aerial Triangulation Matching Strategy pairwise matching in all combinations followed by tuple generation ABM and/or FBM methods available mismatches can often be identified at an early stage - ambiguous solutions across more than two images possible - generation of image tuples via forward intersection is a combinatorial search problem, computational complexity must be controlled Automatic Aerial Triangulation Multi image matching most accurate matching technique available combinatorial complexity is avoided a priori (ABM only) mismatches possibly distributed across all related images 27

28 Automatic Aerial Triangulation AAT system characteristics high redundancy, many tie points per image hierarchical image matching, followed by robust bundle adjustment sub pixel accuracy for tie pint coordinates, often through least-squares image matching some systems integrate determination of tie points and computation of orientation parameters Point transfer Generic approach compute image pyramids extract features in both images separately (attention to feature distribution) interest operators (Moravec, Förstner), edge operators (Canny, Deriche) 28

29 Point transfer Generic approach match features feature description (interest value, length, curvature, contrast,...) approximate epipolar constraint flight navigation data Point transfer Generic approach compute approximate orientation parameters robust least squares adjustment refine results through image pyramid 29

30 Strategies for point transfer Author Tsingas (1992) Schenk (1993) Ackermann (1995) Technique LSM FB points hierarchy (levels) 3 dense 3-4 Points/Image Local Strategy sequential all pairs simultaneous Local Constraints collinearity affinity smooth surface Local Accuracy [pel] < Selection areas Standard positions Footprints Footprints Strategies for point transfer Author GLOBAL STRATEGY Approximate Values orientation index DEM Block formation a. Sequential relat. orientation scale tranfer link of strips b. Parallel (constraints) local relat. Orientation local affine local smooth Rough DEM Exterior orientation (BA) Tsingas (1992) index maps no No Yes Schenk (1993) index maps no Yes Yes Ackermann (1995) GPS no No Yes 30

31 Strategies for point transfer Use of image pyramids in all approaches Small pull in range in LSM -> use all levels of the image pyramid Large pull in range in FBM -> 3-4 levels of the image pyramid Multi image matching in all approaches Tsingas relies on good navigational data and NOT too hilly terrain, Ackermann and Schenk determine footprints Tsingas and Ackermann use the selected and correctly matched feature points in the final triangulation (σ0 is a good indicator for the accuracy of the feature points). Schenk s tie points are iteratively determined in the local multi-image matching procedure and transferred from the reference surface via the digital elevation model to the image planes. (σ0 can be expected to be in the range comparable or superior to manual measurements) Point transfer (Schenk) 31

32 DSM generation Hierarchical processing Image pyramid Features DSM DSM Generation Matching modules exist in various commercial photogrammetric systems. Methods used are often based on cross-correlation, and match at a regular object or image grid. Better methods have been developed at research labs. Matching results, espec. with commercial systems, can vary a lot depending on the selection of the matching parameters (which have sometimes an unclear definition or at least effect). Below 3 automatically generated DSMs with DPW770, SocetSet. Left and right ATE, middle Adaptive ATE (effect of different matching strategies and matching parameters is clear) 32

33 DSM Generation (one of the methods developed in ETHZ) DSM Generation 33

34 Matching quality Example of some measures Normalized correlations coefficient (NCC) Second best NCC Change of NCC when using different masks Consistency in backmatching (left-right and right-left match) Local paralax consistecy check angle of dominant edge direction to the epipolar line residuals from forward intersection...lsm specific number of iterations changes in parameters 34