GIS Conflation using Open Source Tools

Transcription

1 GIS Conflation using Open Source Tools Dave Blasby, Martin Davis, Djun Kim, Paul Ramsey ABSTRACT We give an overview of geographic data conflation, examining the kinds of problems that arise and various approaches to solving them. We then describe a suite of open-source utilities for automated and human-assisted conflation, illustrating the capabilities of these tools by examining how they have been applied to the solution of a number of real-world problems. CONTENTS 1. INTRODUCTION OVERVIEW OF CONFLATION EXAMPLES CONFLATION TERMINOLOGY CATEGORIZATION OF CONFLATION PROBLEM AUTOMATED VS. HUMAN-ASSISTED CONFLATION CONFLATION WORKFLOW OPEN SOURCE TOOLS JTS TOPOLOGY SUITE JUMP UNIFIED MAPPING PLATFORM JCS CONFLATION SUITE (JCS) JCS ARCHITECTURE SOLVING CONFLATION PROBLEMS USING JCS FINDING GEOMETRY DIFFERENCES COVERAGE ALIGNMENT BOUNDARY ALIGNMENT COVERAGE CLEANING STREAM NAME MATCHING FURTHER DEVELOPMENT FINDING DIFFERENCES IMPROVED COVERAGE ALIGNMENT ALGORITHMS REFERENCES...32 Page 1

2 1. INTRODUCTION This paper describes a practical approach to solving real-world geographic data conflation problems. We describe a suite of open-source software tools that have been successfully applied to solve a number of challenging conflation problems. The largest section of this paper presents several of these solutions in detail to illustrate the variety of tools, techniques and approaches supported by our software. A variety of current research problems and areas for future exploration rounds out our discussion. The present section introduces conflation, motivating the problem by considering several examples. Next, we introduce terminology and a categorization of conflation problems. We conclude the introduction by considering workflow in typical conflation processes, in order to motivate the theme of human-assisted conflation, around which idea much of our software is designed. 1.1 OVERVIEW OF CONFLATION Conflation is the action of unifying two distinct datasets into a new dataset. This may be relatively easy to do or extremely difficult, depending on the complexity of representation and the size and quality of the datasets involved. The field of GIS presents many important and challenging conflation problems. The problem of validation, verification and QA/QC of spatial data is closely related to the conflation problem; many of the same issues arise and the same tools and techniques may often be applied. A few examples will help to illustrate the diverse nature of conflation problems. 1.2 EXAMPLES The following examples are intended to illustrate in a general way some of the issues typical of conflation problems. These examples are examined in greater detail in Section 3, Solving Conflation Problems using JCS Cadastral Coverage Vertical Alignment Organizations managing cadastral data typically have datasets of different kinds of cadastral features for the same area. For instance, a municipality may have both lot polygons and jurisdiction boundary polygons. It is often desirable to ensure that the polygon edges are aligned perfectly as possible between the two different coverages. This may be necessary for representational purposes, or for analytical purposes, to ensure that there are no false overlaps which cause erroneous answers to spatial queries Cadastral Coverage Boundary Alignment A related problem to the above is where two cadastral datasets of the same feature class are adjacent to one another. When forming a single seamless coverage it is desired that no erroneous gaps or slivers exist between the datasets, and that the resulting coverage is noded correctly. These differences can exist due to different provenance of the datasets, or even due to carrying out identical spatial processing or editing on the datasets independently. A horizontal conflation process needs to be carried out to remove gaps and slivers below a given tolerance, and to node adjacent polygons correctly. Page 2

3 1.2.3 Multi-scale Stream conflation In many organizations, geographic and spatial data collections have, over time, been extensively enhanced, supplemented and annotated starting from a base geographic dataset at a given scale. Improvements in mapping technology and more stringent requirements often make it necessary to work to newer geographic base datasets at larger scales. The problem is to migrate all of the added attributes and associated information to the more detailed, higher resolution maps. These examples show that conflation is a very general problem. Solving any specific instance requires a clear statement of the problem to be solved, in terms of the representations and constraints associated with the datasets. 1.3 CONFLATION TERMINOLOGY We will define particular meanings for terms used in this paper and a number of the software projects we shall describe. 'Matching' is the activity of identifying features or data elements that represent the same real-world entity. 'Alignment' describes to what degree two features or data elements have coincident geometry. 'Adjustment' is the alteration of geometry or attributes of matched features to make them align. The 'Reference' dataset is the one which is to be conflated to. It is usually of greater spatial accuracy than the subject dataset. The 'Subject' dataset is to be matched or adjusted. 1.4 CATEGORIZATION OF CONFLATION PROBLEM Yuan and Tao [Yua99] classify conflation problems into two types: horizontal and vertical. We add a third type: internal Horizontal Conflation Horizontal conflation is the process of eliminating discrepancies along the common boundary of datasets that are adjacent to one another. Typically the datasets contain data from the same feature classes. Examples are aligning the boundaries of adjacent coverages, or edge-matching neighbouring networks Vertical Conflation Vertical Conflation involves matching and/or eliminating discrepancies between datasets that occupy the same area in space. Examples include road network matching between two representations of roads in the same region. Two important kinds of vertical conflation are Version Matching and Feature Alignment. In the case of Version Matching the input datasets consist of different versions of the same features. The conflation process is intended to identify matching features. Attributes may be transferred between matched features, and unmatched features may be transferred in their entirety. An example of this is matching different versions of road networks for the same geographical area. In case of Feature Alignment the input data consists of features from two or more different feature classes that bear some defined relationship to each other. The conflation process is intended to remove discrepancies between the datasets that causes this Page 3

4 relationship to fail to hold. A common relationship is that of geometric alignment. An example of this is aligning the boundaries of different kinds of feature classes such as municipal districts and lot parcels Internal Conflation An internal conflation problem involves conflating or resolving features or elements within a single dataset. For example, in coverage cleaning we may need to remove overlaps in the polygons comprising a coverage. 1.5 AUTOMATED VS. HUMAN-ASSISTED CONFLATION Many conflation tasks have in the past been carried out in a manual process, with a human operator identifying matches or errors, determining alignments or corrective actions, and manually correcting geometry or attribution. Although various tasks within this process may have been carried out with the assistance of software, scripts and macros, the overall process would generally be characterized as 'computer assisted' manual conflation. Increasingly sophisticated software tools have recently enabled 'human-assisted' approaches to conflation, inverting the traditional paradigm 'computer-assisted' paradigm. Now, human expert judgment and intervention is required only in relatively rare situations within an essentially automated process. Such approaches, if they can be applied to a given problem, greatly improve the productivity and reliability of conflation projects. Obviously the goal is to automate the solution of conflation problems entirely. However, this may not always be possible, for two reasons: Algorithm Limitations. This is the case if there is no currently known algorithm capable of correctly identify all matches, or computing appropriate adjustments to eliminate discrepancies. In this case human input at different stages of the algorithm can increase the percentage of problems that can be fixed. Data Ambiguity. In this case there is not enough information contained in the input datasets to correctly identify matching features. It may not even be possible for a human operator to determine a correct course of action. Additional data attributes may need to be obtained in order to resolve the discrepancy. The first limitation may be overcome by providing for human input into the conflation process. This input can occur at several places in the conflation workflow, including Dataset Alignment, Feature Matching, and Geometry Alignment. Manual input into Dataset Alignment could include manually warping or shifting the data in order to reduce discrepancies to a level at which the available automated algorithm becomes effective. Manual Feature Matching requires that human be able to create or delete matches between features. Ideally this can be carried out in conjunction with automated matching in a iterative process. It is important that an appropriate method of visualizing both the input data and current set of matches be provided. Manual Geometry Alignment can take the form of simple geometry editing tools, as well as more specific tools such as Vertex Snapping. The User Interface should provide some method of determining that manual edits have correctly eliminated discrepancies. For all of the above situations, the effectiveness of human assistance is directly proportional to the quality and richness of the user interface. This includes factors such as execution speed, ease-of-use, variety of feedback, ability to visualize spatial data items, and the presence of tools such as transaction recorders, multi-level undo, etc. The design of an Page 4

5 effective user interface for human-assisted conflation is as much of a technical challenge as the design of the automated algorithms themselves. 1.6 CONFLATION WORKFLOW Workflow in a human-assisted conflation process generally can be broken down into the following common subtasks. Depending on the nature and quality of the data in a specific conflation problem some of these subtasks may be trivial or not required. Data Pre-processing. This step normalizes the input datasets to ensure that they are compatible. For instance, they must have the same coordinate system. This may also involve format translation and any other basic preparation of the datasets. Data Quality Assurance. Some conflation tasks require that datasets have a given level of internal consistency. For instance, coverage alignment algorithms require that the input datasets are in fact a clean coverage. During this step the internal consistency of the datasets is verified and if necessary improved. Dataset Alignment. In some cases datasets are sufficiently misaligned that an initial alignment process is required to allow more precise conflation to be carried out. This alignment is typically coarse grained in nature, not descending to the level of aligning individual features. This process may be either manual or automatic. Feature Matching. During this step common features between the datasets are matched. This may be done either in an automated fashion using one or more conflation algorithms, or via manually determined matches. After this phase has been performed the discrepancies between the datasets will have been identified. It is often useful to provide statistical summaries of data quality, or to visualize the discrepancies. Geometry Alignment and/or Information Transfer. Once features have been matched the match information can be used to improve the quality of one or both of the input datasets. Geometry Alignment removes discrepancies between geometries. Information Transfer involves updating one dataset with information from the other. This information can be either attributes or geometry to be added to an existing feature, or entire features to be added to the dataset. Page 5

6 Reference Dataset Data Preprocessing Data QA Dataset Alignment Subject Dataset Data Preprocessing Data QA Feature Matching Geometry Alignment / Information Transfer Conflated Dataset Figure Conflation Workflow 2. OPEN SOURCE TOOLS Various geographic, historical and economic factors have led the Province of British Columbia in Canada to the widespread adoption and heavy use of large-scale GIS and spatially-based resource management systems. As a large customer of GIS and spatial data and services, the Province of British Columbia was in an excellent position in 2001 to benefit from the availability, and widespread adoption of, a capable, reliable, and highly usable base of geomatics software upon which to build productive, highly automated, spatial software and data products. Recognizing this opportunity, Dr. Mark Sondheim and the Ministry of Sustainable Resource Management (Government of British Columbia) initiated a number of projects to create this geospatial software development base. We will describe three of these projects, developed by Vivid Solutions Inc. and Refractions Research Inc. The Java Topology Suite (JTS), JUMP, the Unified Mapping Platform, and JCS, the Java Conflation Suite have been under development for over two years, and are available under the Free Software Foundation's GNU Public License. 2.1 JTS TOPOLOGY SUITE The JTS Topology Suite is a 100% pure Java implementation of the OpenGIS Consortium OpenGIS Simple Features specification. [REF] JTS is fast, production quality code. It is reliable and very robust; the extensive test suite and test framework developed by Vivid is in itself a significant contribution to developers and maintainers of spatial or computational geometric base libraries. JTS is supports both fixed and floating precision models, with explicit precision, for numerical computations, up to the full precision allowed by Java s IEEE794 double floating point numbers. Page 6

7 2.1.1 JTS Topology Suite Geometry Model and Operations JTS defines points, linestrings, polygons, and collections of these. It provides spatial predicates over these basic objects using the dimensionally extended nine-intersection model (i.e., considering cases over all pairs of interior, boundary, exterior of two objects in binary predicates (e.g. equals, touches, crosses, within...) JTS further defines a range of geometric operations including Boolean operations, convex hull, centroid, buffer, etc. 2.2 JUMP UNIFIED MAPPING PLATFORM JUMP is a 100% pure Java, Open Source (GPL license) extensible platform for the visualization of geospatial data, development of spatial algorithms, and the interactive development of automated and human assisted spatial processes, It provide both a GUI environment and an extensive API. JUMP uses the best of modern software design practices to enable easy extensibility, providing developers of spatial processes direct access to the full capabilities of the Java platform JUMP API JUMP wraps geometric objects from the JTS in a spatial context, providing Features with geometry and attributes. These may be aggregated as Feature Collections. Spatial access methods including quadtrees, STR-Trees, and binary interval trees are provided. Spatial transforms including Warping, affine transforms, and bilateral interpolated triangulation are provided. Datasets may be read from and saved to a variety of sources (e.g., files, WMS), in many common formats including Well Known Text (WKT), GML, and ESRI Shapefiles JUMP Workbench The JUMP Workbench GUI provides a multi-document interface, with each window supporting multiple layers of spatial data. Users can apply a rich variety of styling options to optimize the visual presentation of datasets. Styling may be applied by attribute. Difficult problems such as non-overlapping rendering of text labels at variable scales are automatically handled. JUMP's GUI enables direct interaction with the JUMP API functions; flexible, efficient and intuitive geometry & attribute editing tools are available. The GUI is easily extensible via a plug-in framework. 2.3 JCS CONFLATION SUITE (JCS) Written in Java as a highly portable, highly efficient toolbox for performing conflation, the JCS Conflation Suite leverages the best of existing spatial tools, algorithms, and modern software development techniques. JCS was designed to solve real-world conflation problems; operational efficiency and ease of re-use have been strict requirements from the onset of the project. JCS is implemented as a collection of modular extensions ('plug-ins') for the JUMP Unified Mapping Platform. These extensions are supported by and build upon the basic geometric types, predicates, and operations provided by the JTS Topology Suite. JCS provides tools to visualize, explore, clean and prepare data in a variety of common GIS data formats and sources. The core of the JCS is a large number of plug-ins supporting the various stages of the workflow of automated or human-assisted conflation processes. Specialized functionality can be implemented by extending existing plug-ins or writing new Page 7

8 ones. The support provided by the JCS architectural framework makes this surprisingly easy. 2.4 JCS ARCHITECTURE The JCS Architecture has been designed to be flexible, modular and reusable. Core functionality (conflation algorithms and other spatial algorithms) is exposed as a Java API. This allows JCS functionality to be incorporated in batch processes, and easily reused in other applications. Functional components of JCS are identified in the following figure. Figure 1: JCS Functional Components 3. SOLVING CONFLATION PROBLEMS USING JCS The Java Conflation Suite addresses many types of conflation problems. Some users may require nothing more than Dataset QA, either as part of an automated process, or only occasionally to find out exactly why another GIS application complains or goes wrong on a given dataset. However, JCS has tools to help with every step of the conflation workflow, enabling users to go beyond dataset QA to the automated detection and resolution of data problems, to automated conflation with integrated tools for human-assisted conflation (manual edits) when necessary. This section presents a variety of examples illustrating how JCS has been applied. The goal is to provide sufficient detail to enable potential adopters of the JCS platform to understand how JCS might be used within their own applications. 3.1 FINDING GEOMETRY DIFFERENCES A common problem in spatial data processing is to find differences in the geometries in two datasets. Geometry differences can be caused by: Features being added or deleted Geometry linework being altered Page 8

9 Data format conversion or other processing altering the values of coordinates, e.g. by explicitly reducing the precision or by numerical roundoff errors A common situation is that one dataset is an updated version of the other, and it is required to determine the location and number of changes made to the original dataset. These changes can be detected by means of a spatial difference algorithm. Difference algorithms work by matching geometries between the two datasets, and reporting the geometries in each dataset which do not have a match in the other dataset. Matching can be carried out in two different ways: Exact Matching tests if geometries are the same on a point-by-point basis. This is useful if the datasets are known to be very similar and only small changes have occurred. Matching within a Distance Tolerance tests if each point in the geometries lies within a given distance of some other point on the other geometry. This has the effect of ignoring changes smaller than the distance tolerance. This is useful in situations where many points have been shifted by small amounts (e.g. by a change in precision caused by format conversion) and only large-scale differences are of interest Diff Geometry JCS provides a function called Diff Geometry to detect the differences between the geometry of features in two datasets. (The term Diff is a reference to a well-known Unix application for detecting the differences between two files. The JCS Diff Geometry function performs a similar function for spatial datasets.) The basic algorithm is: 1. For each geometry a in dataset A a. attempt to find a matching geometry b in dataset B. (Since matches can only occur if two geometries are within the distance tolerance on one another, a spatial index is used to improve performance of this query) b. If a matches b, mark both as being matched 2. Report all a in A and b in B which are not matched A slight variation on this algorithm involves matching components of geometries, rather than the geometries themselves. This allows ignoring cases where (for instance) a MultiPolygon in one dataset has be split into two separate Polygons in another dataset Geometry Matching The key algorithm in the Diff Geometry function is that of matching Geometries. Two algorithms have been developed: Exact Match and Match Within Distance Tolerance Exact Match Exact Matching tests whether the type and the contents of the Geometries are equal. Testing type equality is straightforward. To test equality of the contents, Exact Match compares the components and pointlists of the geometries in sequence. In order to eliminate trivial mismatches caused by the ordering of the points within rings and of components within collections, the components and pointlists can be normalized before Page 9

10 comparison. Normalization sorts components and rings according to an order function based on the common dictionary ordering for coordinates Match Within Distance Tolerance The fundamental idea behind matching with a distance tolerance is to be able to match geometries in spite of small differences between them. Small difference is an imprecise notion which is difficult to formalize. In practice we have found that a useful definition involves using the Hausdorff distance. We say that two Geometries match if the Hausdorff distance between both the Geometries and their boundaries is less than the distance tolerance. More formally, Geometry A matches Geometry B iff H(A, B) = d and H(boundary(A), boundary(b) = d We do not need to compute the Hausdorff distance explicitly. Instead, we need to solve the decision problem: is H(A, B) < d? This can be solved in a straightforward way by using buffering to test for an upper bound on the Hausdorff distance. We have the following relationship: H(A, B) = d if buffer(a, d) contains B and buffer(b, d) contains A This gives the following algorithm: Geometry A matches geometry B if A is contained in buffer(b, d) and B is contained in buffer(a, d) and boundary(a) is contained in buffer(boundary(b), d) and boundary(b) is contained in buffer(boundary(a), d) Table An example of using buffers to match within a distance tolerance Page 10

11 The two examples in Table 3-2 illustrate that both the geometry buffer test and the boundary buffer test are necessary; neither one alone is sufficient. In this case the geometry buffers are almost identical, but the linework is different. The boundary buffer containment test will detect this difference. In this case the boundary buffers are almost identical, but the geometries are different. The geometry buffer containment test will detect this difference. Table Examples showing that both boundary buffers and geometry buffers must be tested to properly detect geometry differences 3.2 COVERAGE ALIGNMENT Coverage Alignment is an example of vertical conflation. It consists of aligning the edges of a subject coverage to the edges of a reference coverage that partially or completely overlaps the Subject Suburb-Road/Lot Alignment The following images shows a situation involving two coverages for the same are. The yellow/red polygons are Suburb areas. The green and red polygons are a coverage of road and lot polygons contained in the suburbs. The edges of the suburb polygons are fairly close to the road/lot polygons, but not precisely aligned. Page 11

12 Figure Suburb and Road / Lot Coverages Figure An example of a discrepancy between the Suburb boundary and the Road/Lot boundaries Analysis: The conflation problem is to align the boundaries of the suburb coverage polygons to the road/lot coverage linework. The road/lot polygons are the Reference dataset, and the suburb polygons are the Subject dataset. The Subject dataset will be adjusted to align with the Reference dataset. The adjustments may include both moving and inserting vertices. A further conflation step is to insert vertices in the Reference dataset to ensure that it is noded exactly with the Subject dataset. Page 12

13 There are two options for noding when aligning Subject edges to matching Reference edges. If two contiguous Subject segments match a single Reference segment, so that a vertex exists in the Subject but not in the Reference, a choice can be made to either remove the vertex from the Subject (by merging the segments) or to insert the vertex into the Reference by splitting the segment. This choice can be made based on external requirements for how much change is allowable in each dataset. Although not visible from the images, the polygons in the road and lot coverage in fact contain gaps and overlaps. Before conflation can be carried out, these problems must be corrected so that the road and lot polygons form a correct coverage Discussion: Algorithms Key idea: Lines that are close should be identical Match Line segments Hausdorff distance < tolerance Relative angle < tolerance Adjust Vertices Snap close vertices Add or merge vertices if necessary Heuristics: Do not break topology Minimize change to existing vertices 3.3 BOUNDARY ALIGNMENT Problem Description Along common edges polygons may not meet exactly. There may be gaps or overlaps between adjacent polygons. Vertices may be present on one edge that are not present on its neighbour. The following example illustrates gaps, overlaps, missing vertices, and large gaps Page 13

14 Table Example of datasets requiring edge-matching Dataset Descriptions Input Data. A set of two (or more?) coverages (collections of non-overlapping polygons). Each input coverage is assumed to be clean. The coverages are intended to form a single larger coverage, with no gaps or overlaps at their common edges, and correctly noded along their common edges. Other terms. Gap Removal, Sliver Removal Assumption/Restriction. The process assumes that the discrepancies that exist between the datasets are small relative to the minimum segment size in the datasets, and to the minimum gap size between the two datasets. (If this is not the case, it becomes difficult to automatically decide whether differences are legal or are due to conflation problems). Output Data. Updated versions of the input coverages, edge-matched along their common edges. Conditions on Output Data 1. No existing vertices in either input dataset will be deleted. 2. No existing vertices other than boundary vertices will be moved in either dataset. 3. No existing vertices in the Fixed dataset will be moved Page 14

15 4. New vertices may be created along the common boundary in either dataset. Parameters Parameter Tolerance Description A Subject segment or vertex, which is within this distance of a Reference segment or vertex, will be adjusted to align to the Reference. Adjustments Performed Match A Subject vertex and Reference vertex are within the Tolerance A Subject segment has a Reference vertex within the Tolerance A Reference segment has a Subject vertex within the Tolerance Adjustment The Subject vertex is adjusted to the Reference vertex location. A new vertex is created on the Subject segment at the location of the Reference vertex. A new vertex is created in the Reference segment at the closest point to the Subject vertex (the projection of the Subject vertex on the reference segment). The Subject vertex is adjusted to the location of the new vertex. Note: matches are processed in order of priority. If two or more matches occur, the higher priority one determines the adjustment carried out. Issues 1. If a Subject boundary edge reverses direction at a vertex and both vertices are close to a Reference segment, the vertices will both be adjusted to the same segment. This will cause adjusted segments to self-intersect and the adjusted polygon to have invalid topology. This can be defined as a requirement for the boundary to be smooth relative to the tolerance value. This issue may be able to be solved by constraining matched segments to be in opposite directions (where opposite is defined as having slopes that are at least 135 degrees apart). Figure A case where adjusting vertices would cause self-intersection Page 15

16 2. If a boundary polygon completely overlaps a neighbour polygon, the segments of the boundary polygon may incorrectly match segments of the neighbour polygon that are not on its boundary Algorithm 1. Identify boundary edge lists that are close together. 2. In the Subject dataset, adjust vertices to the closest Reference vertex within the Tolerance. 3. In the Subject dataset, insert vertices along line segments where there is a Base vertex closer than the Tolerance to the segment. The inserted vertices will be inserted at the location of the Reference vertex. 4. In the Subject dataset, existing vertices that are within the Tolerance of a reference line segment will be moved to lie as close as possible to the Reference line segment. 5. In the Reference dataset, insert vertices along line segments where source vertices are within the Tolerance of the segment. 3.4 COVERAGE CLEANING JCS provides tools to do the following tasks involved in coverage cleaning: QA coverages to detect errors (gaps and overlaps) Automatically fix most or all errors in a coverage Allow manual fixing of errors not fixed automatically Gap Detection An automated tool has been developed to find gaps in coverages. Gap detection is based on detecting misaligned line segments within a certain distance tolerance. Output Data Layer Gap Size Gap Segments Description Shows the segments on either side of detected gaps Contains indicators showing the size of gaps Figure QA layers created by the Find Coverage Gap function Page 16

17 Figure Closeups of gaps showing Gap Segment indicators (red) and Gap Size indicators (blue) Overlap Detection JCS has a tool find overlaps in coverages. Overlap detection is based on the JTS capability to compute exactly whether the interiors of two polygons overlap. Output Data Layer Overlap Size Overlap Seg Overlaps Description Contains indicators showing the size of overlaps Shows the segments forming overlaps Contains copies of the features which overlap Figure Layers created by the Find Coverage Overlaps function Coverage Gap and Overlap Removal Automated Gap Correction JCS contains an algorithm to correct many gaps automatically. Since it is not always welldefined how a gap should be corrected without negatively impacting the topology of the surrounding geometry, the algorithm contains heuristics that help to prevent causing further topology errors. This means that not all gaps can be automatically fixed. However, tests run on many real-world datasets show that usually 95% or better of gaps can be fixed automatically. Page 17

18 Manual Gap Correction To correct gaps that cannot be fixed automatically JCS provides manual editing tools. The tools are Snap Vertices ( ) and Delete Vertex ( ). Figure Zooming to a gap using the Attribute View Zoom tool The following figure illustrate how a gap is manually fixed using the Snap Vertices tool. Draw a Fence around the gap with the Snap Vertices tool Shift-click to choose a point to snap all the vertices in the fence to The vertices in the fence are snapped together to close the gap Use Clean > Update Gap In Fence to update the gap indicators and verify that the gap has been fixed Figure Fixing a gap manually Page 18

19 Manual Overlap Correction Most small overlaps should have been detected and fixed using the Gap correction tools. Overlaps that remain may be due to two things: The Distance Tolerance used in gap cleaning may have been too small to correct the gap due to the overlap. If possible, increase the Distance Tolerance value to allow the gap to be corrected. However, do not increase the Distance Tolerance to the point at which valid spaces begin to be affected. Overlaps may be due to duplicate features in the datasets. In this case, it may be possible to simply remove one of the features. (The best way to do this is to use the Feature Info tool to view all features at a specified point. One of the features can be selected in the Feature Info view and deleted using Cut Selected Features) Overlaps may be present in the input dataset for a business reason. In this case it is necessary to manually decide how to deal with the overlap. 3.5 STREAM NAME MATCHING In order to preserve a large legacy database of various habitat and inventory information, the 1: Watershed Atlas (WSA) must be conflated with the 1: Corporate Stream Network (CSN) Matching The British Columbia stream conflation problem is a multi-scale conflation problem. An older stream network, at 1:50K scale, has been used for many years as the spatial basis for fisheries and habitat inventory databases. New requirements for spatial accuracy require a 1:20K stream network. The new network describes the same water features as the old network, but because the data were collected using different resolutions of input data, the interpretations of stream locations and network connectivity differ between the two products. The stream conflation problem is to identify those streams in the new network which are logically the same as the streams represented in the old network. Because the matching is against streams, some additional information can be used to enhance the matching process: The two networks tend to describe the same flow behaviours. The 1:20K network is generally a superset of the 1:50K network. Streams which are the logically the same have similar spatial locations. Page 19

20 These dark blue lines represent a small river network taken from the 1:50k BC Watershed Atlas. These red lines represent the same area but are from the 1:20k Stage 1 CSN. The dark blue (50k) data is layered on top of the red (20k) data. The matching process correlates the 20k river segments to the corresponding 50k rivers Algorithm The matched 20k segments are shown in purple, the unmatched segments are in red. The basis of the matching process is a JUMP application that provides: access to very specialized conflation algorithms, a process flow UI to ensure all problems are properly address, specialized UI tools to enable intricate corrections to be easily applied, and validation to ensure no errors are introduced. The driving principle behind the matching process is to ensure that the best geometric match is made. Metrics about the quality of the match are retained. The best geometric match means that the 20k path chosen to represent a 50k river is the one that is the least spatially distant. The basis for matching is to follow the hierarchy already inherent in the 50k river network not only the parent-child relationship, but also parent-side-channel relationships. Following these hierarchies ensures that connectivity and flow relationships are easy to Page 20

21 validate and preserve. The actual river matching process simply finds sets of possible paths in the 20k data and then chooses the best path. This path is tested to ensure that it is an accurate depiction of the 50k river and does not break any flow relationships Hierarchy Following Following the 50k hierarchy means that a tributary (child) of a river is not matched until its parent has been matched. The following figures present a hypothetical watershed and show the ordering of matching. The JUMP application provides the user with tools to quickly fix a parent stream that could not be automatically matched, and tools to subsequently match its children. Page 21

22 Attempt to match (Stem A) If A matches, attempt (Stem B) If B matches, attempt (Stem C) If A matches, attempt (Stem D) If D matches, attempt (Stem E) If D matches, attempt (Stem F) Path Formation, Walking, and Pruning The basis of the matching process is to create a set of possible 20k matches, then choose the best path. The creation of these paths is quite simple find 20k tributaries near the mouth of the 50k river, and start following the 20k rivers upstream. Incorrect tributaries will be quickly pruned away during the walking process. Page 22

23 The parent river has been matched (20k match shown in purple) so its children are being matched now. For the 50k tributary, the location of its mouth is determined, and all the 20k tributaries that are within 200m are found. Starting from each of the start points found, paths are found by walking up each of the streams. The best of all the paths is found and then verified against rules for minimally acceptable matches. Page 23

24 Unfortunately, finding all possible paths can create too many paths to be evaluated especially on large rivers. In order to be more efficient, a pruning process is used to quickly remove paths as soon as they diverge too much from the river they are trying to match. In this example, a naïve path finding approach would find over a hundred paths. On larger rivers, a full set would contain over 100,000 possibilities. In order to be more efficient, paths are pruned removed from consideration as soon as they diverge too much from the 50k river. The type coding of rivers needs to be obeyed too. In this example, the lower loop is a side channel. The main stem of a river is first matched it will not attempt to find paths that include a secondary flow segment. If the match is successful, all the side channels will be matched (in longest-channel first order to help with heavily braided rivers). During this match, only segments that are coded secondary flow will be considered. Page 24

25 The actual path formation is very simple start at the initial segments and find the upstream segments. The path is built by adding this segment to the already existing path. If there was a junction in the river, simply make two copies of the path one for each branch. This shows the path building method. The purple represents an already existing path that is being built. The 50k data has not been adequately matched, so the path must be extended. At this junction, we have two possible choices to follow adding segment A or adding segment B (or creating two new paths). If segment B meets an accuracy criterion, a new path will be created from the original by adding segment B. This extended path will continue to be investigated and segment C and/or segment D will be added in the next step. Typically, several possible paths are produced. The best of all the possible paths is chosen. If it passes the match-is-good-enough criterions, the path is accepted. Page 25

26 Active stack stack of all the paths currently being investigated Match list list of paths that might be matches to the 50k river while (paths in active stack) pop a path from the stack find all the segments upstream from this segment if there are no more upstream segments, add this path to the match list for each of these upstream segments if this segment passes the criteria: * coded properly (i.e. Primary flow) * does not diverge too far from 50k river then add it to the current path and push the new path on the stack Path Evaluation The basic unit of information driving the matching process is the measurement of how well a particular 20k segment matches a 50k river. A very simple sampling technique is used to make judgements about nearness and shape. The 20k segment is sampled every 25m along it length. For each of these sample points, the nearest corresponding point on the 50k river is found. Using this information, the nearest, furthest, and average distance between the two lines is determined and used to determine the line s nearness. Shape similarity is determined at the same time as nearness by watching the movement of the corresponding nearest-distance matches on the 50k data. If the shapes are very similar, the corresponding 50k points should move the same distance (along the segment) as the sample points on the 20k segment. If the shapes are dissimilar, the corresponding distance on the 50k river will be either too small or too large. This principle is used to estimate similarity of the shapes. Page 26

27 A 20k segment is sampled every 25m along its length. Each of these points is evaluated against the 50k river. The results of each of these point matches are summarized into a segment match. This picture shows the details of the point matches made. The closest point, on the 50k river, to points A and B are found. The distance, along the 20k data, between the two points is 25m. The distance between the corresponding points, on the 50k data, is also recorded. In this case, the corresponding distance match is only 19m. Once the set of paths has been made, and each segment in the path has been compared to the 50k data, the best path needs to be found. This is done by pair-wise comparing paths in a transitive 1 manner. This process involves looking at: o o o How well the shape of the path matches the 50k river How close the path is to the 50k river How much of the 50k river is close to the 20k path The best path may still not be a close enough match to consider a true match. The final match is evaluated to ensure that it really does follow the shape and location of the 50k river to a degree that is satisfactory. In this context, the better than function means that if A better than B and B better than C then A better than C. Page 27

28 Human Assisted Matching The automated matching process provides an excellent initial matching of the 50k and 20k data. Unfortunately, not all matches can be automatically detected, and some matches may need to be verified as correct. In general, this type of manual process can be extremely time consuming. By providing the user with workflow control and powerful tools, the time required to do this can be dramatically reduced. The underlying bases of the majority of these tools already exist in the algorithms used for the automated process. Three examples of how a user could quickly solve simple problems are given below: Due to differences in interpretation between the 20k and 50k data, the river mouths are about 1250m apart. The automated process rejected the match due to the discrepancy. During the human-assisted process, this situation would be presented to the user. The user would simply start the automated matching process with the correct initial segment (labelled Start ). Once the match is complete, the river s children can also be automatically matched. This problem is solved in a few simple mouse clicks. Page 28

29 In this example, the 20k data prematurely ends (relative to the 50k interpretation). The user decides that this side channel should be matched. They automatically build a new route by click at the start and end of the desired path (the two Click here points in the picture). The match is made, evaluated, and logged. The children are then matched. This 50k river does not have a corresponding 20k equivalent. The automated process could not find a match and the user can very quickly verify that there really is none. The workflow control and validation routines ensure that all problems are satisfactorily addressed, and the specialized tools allow each of the problems to be solved by a human user with just a few mouse clicks. The application would record and log all the changes so that they could be re-applied if there are any changes to the underlying CSN 20k dataset. The end result is an application in which a GIS professional can be trained to competency in as little as one day. This application controls the entire matching process, providing logs, presenting problems, facilitating correction, and determining the quality of each match Assumptions The following assumptions are important to the algorithm: Both (20k and 50k) networks are topologically clean before matching begins. Both networks are can be built into upstream/downstream networks using digitization direction as a consistent guide. The attributed 50k network (BCWSA) includes a hierarchical watershed code used to determine parent-child relationships. Both networks have the BCWSA feature coding applied to their segments. These codes are used to distinguish between main flow and secondary flow matches. Differences in interpretation between the two datasets might cause problems to be investigated by the human-assisted application. Waterbodies (i.e. marshes and lakes) will be matched by another process. Page 29

30 4. FURTHER DEVELOPMENT 4.1 FINDING DIFFERENCES In practice, the spatial difference (Spatial Diff) algorithm works well in finding differences between geometry datasets. However, there are pathological examples which are reported as matches by the algorithm but which would probably not be considered as matching by a human. An example is shown below. It would be interesting to experiment with some other matching algorithms to reduce the likelihood of invalid matches. As a precursor to this, it may be necessary to categorize the kinds of differences which are considered as matches or mismatches in particular application areas. Table A example of different Geometries reported as matching by the buffer-based algorithm 4.2 IMPROVED COVERAGE ALIGNMENT ALGORITHMS The current algorithms used in the JCS Coverage Alignment module are oriented towards handling relatively smooth boundary geometry (such as occurs in man-made features). The algorithms will perform less well on more complex geometry (such as occurs in boundaries that follow natural features like rivers and heights of land) It is of interest to develop effective algorithms to handle other kinds of linework. A motivating example for this problem is discussed below Alignment To Natural Features In British Columbia the boundaries of many Administrative Areas follow watershed outlines (height-of-land) for portions of their length. Since the Admin boundaries were originally digitized the accuracy of the base watershed outlines has been improved. There is a need to align the Admin boundaries with the more accurate watersheds. The image shows Admin boundaries in red and watershed outlines in blue. It clearly shows Admin boundary sections that follow the watersheds and sections, which follow surveyed lines. Page 30

31 Table Admin Boundaries and Watersheds Analysis: The conflation problem is to align the boundaries of the Admin polygons with the boundaries of the watershed polygons. The watershed polygons are the Reference dataset, and the Admin boundaries are the Subject dataset. One of the challenges in this problem is that only some portions of the Admin boundaries are intended to follow the watersheds. The non-watershed sections need to be distinguished from sections that are intended to be aligned with watershed boundaries. The non-watershed sections should not be altered by the conflation process. Page 31

32 5. REFERENCES [Yua99] Shuxin Yuan, Chuang Tao. Development of Conflation Components [Aqu02a] Jonathan Aquino. JUMP Workbench User s Guide [Aqu03a] Jonathan Aquino. JUMP Developer s Guide [Davis03a] Martin Davis. JUMP Unified Mapping Platform Technical Report [Davis03b] Martin Davis. JTS Technical Specifications (v1.3). March [Davis03c] Martin Davis. JCS Conflation Suite User s Guide [Davis03d] Martin Davis. JCS Conflation Suite Technical Report [Blasby03] David Blasby. Stream Matching User s Guide Page 32