Esri Best Practices: QA/QC For Your Geodata. Michelle Johnson & Chandan Banerjee

Transcription

1 Esri Best Practices: QA/QC For Your Geodata Michelle Johnson & Chandan Banerjee

2 Overview Understand Document Implement

3 Understanding QA/QC

4 Understanding QA/QC Fit For Use - support your GIS applications? - support your Business Systems? Decision-making - Good data good decisions save time & money - Bad data bad decisions more time & more money Reporting - Accuracy & Accountability

5 Understanding QA/QC GIS Technician Manager Executive Knowledge Workers Frees up time Less rework Focus elsewhere Effective data stewardship GIS productivity Increased GIS usage Confidently make decisions Reduce financial risk Optimize performance Confidence in GIS Increase efficiencies

6 Understanding QA/QC Daily Maintenance Data Conversion/ Migration Acquisition of Data (Free or Purchase) Annual Data Revisions

7 Understanding QA/QC Quality Assurance Processes or methods to help prevent errors from being introduced into the data. - Examples: - Data Model, Industry-specific Editing Templates, Attribute Assistant, data-specific editing tools Quality Control Processes or tools to identify errors that are already in the data. - Examples: - GP tools (Select by Attribute, Select by Location), GP Model, Data Reviewer Batch Jobs

8 6 Elements of Data Quality Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Usability The accuracy of the position of features within a spatial reference system. The presence and absence of features, their attributes and relationships. The quality of the temporal attributes and temporal relationships of features Classification correctness related to features and their attributes Adherence to logical rules of data structure, attribution, and relationships (documented in data product specifications). The data adhering to the user requirements for its intended use (ISO-19157:2013 Geographic Information Data Quality)

9 Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Usability Name the Data Quality Issue

10 Which Element of Data Quality? Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Issue - Street centerline data contains street names, but no address ranges - Impact: Not enough information for geocoding - Inappropriate scale - Impact: Cannot be used for intended purpose Used to route emergency vehicles Used to map National Parks Usability

11 Which Element of Data Quality? Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Issue - Land use polygon coded as residential instead of commercial - Impact: Business was not granted a timely permit - River captured as a single line instead of a polygon - Impact: River is not symbolized as being navigable, when it should be Swimming pool captured as wetland Usability

12 Which Element of Data Quality? Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Issue - Missing building footprints - Impact: Planned construction on top of existing building - Water mains containing blank or null diameter attribute values - Impact: Water utility crews may not have proper equipment for the job Neighborhood with missing building footprint Usability

13 Which Element of Data Quality? Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Issue Misplaced or misaligned features Impact: Can cause construction problems Inaccurate coordinate values Impact: Could navigate to incorrect address Lake feature has been shifted Usability

14 Which Element of Data Quality? Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Issue - Incorrect time or date for discrete events - Hydrant status not updated - Impact: Could affect firefighters response time - Navigational charts - Impact: Plane could fly into closed airspace Outdated chart showing open runway Updated chart should show closed runway Usability

15 Which Element of Data Quality? Positional Accuracy Completeness Temporal Quality Thematic Accuracy Logical Consistency Issue Reducer connects two pipes with same diameter Impact: Could affect hydraulic modeling results Road surface type attributed incorrectly Impact: Could affect routing and drive times Highway with road surface type gravel Usability

16 Documenting QA/QC

17 Documenting QA/QC Purpose & Scope Roles & Responsibility Testing Environment Requirements & Acceptance Criteria QC Workflow Processes & Tools Reporting & Addressing Errors (high-level)

18 Understanding QA/QC GIS Applications, Business Systems - Required fields, feature connectivity, topological relationships What can be automated? What will need to be checked manually/visually? - Data Reviewer Sampling - Systematic QC using grids

19 Industry Examples

20 Industry: Hydrography Rule: Lakes and/or ponds must be individual features Tool: Data Reviewer Multipart Polygon Check Rule: Rivers must flow downhill Tool: Data Reviewer Monotonicity Check 4ft 5ft 6ft 8ft 3ft 10ft 11ft 12ft

21 Industry: Water Utilities Rule: Tee fitting must be connected to 3 water lines Tool: Data Reviewer Valency Check Select Junction by Edge Count (Water Utility Network Reporting toolbar) Rule: Hydrants must be connected to hydrant laterals Tool: Data Reviewer Geometry on Geometry Check Find Disconnected Features in Geometric Network GP Tool

22 Industry: Facility Management Rule: Building Facility Site Identifier must be unique Tool: Data Reviewer Unique ID Check Frequency GP tool Rule: Buildings must be greater than a certain size Tool: Data Reviewer Evaluate Polygon Perimeter and Area Check Select by Attribute

23 Industry: Land Use Planning Rule: Flood zones should not be slivers Tool: Data Reviewer Polygon Sliver Check Database Topology Rule: Zoning district polygons should not be duplicates Tool: Data Reviewer Duplicate Geometry Check Find Identical GP tool

24 Implementing QA/QC

25 Implementing QA/QC Data model Industry-specific editing templates Attribute assistant Data-specific editing tools

26 Feature Templates & Attribute Assistant

27 Implementing QA/QC Automated Data Validation - Geoprocessing Tools & Models (Toolbox) - Python Scripts - ArcGIS Data Reviewer Manual Data Validation - ArcMap Document - ArcGIS Data Reviewer - Checklist

28 Implementing QA/QC Daily or when editing - Step in a workflow - Current extent Periodically (weekly or monthly) - Scheduled, automated - Full database New or Updated Dataset - When received - Full dataset

29 Automated Data Validation

30 Leverage GP model to perform data validation Business rule: Single family parcels must not contain more than one address Single Family Home Apartment Complex Single Family Home

31 Leverage Python Script to perform data validation Business rule: Compare Features

32 Summary QA/QC is important & perform regularly Create a QA Plan & define QC requirements Develop a repeatable, automated process

33 Resources LGIM data model Industry-related Editing Templates - Tools - Attribute assistant Industry-related Data Reviewer Templates - Sample Reviewer Batch Jobs

34 Please Take Our Survey on the App Download the Esri Events app and find your event Select the session you attended Scroll down to find the feedback section Complete answers and select Submit

35