Make Your GIS Work for You, First Step: Data Quality

Transcription

1 Make Your GIS Work for You, First Step: Data Quality Presented by: Scott Sumners, GIS Manager, City of Brentwood James McCord, GIS Analyst, City of Brentwood Gerardo Boquin, GISP /CH2M

2 Brentwood GIS CAD to GIS Aerial Photography Topographic base maps Schema Changes Added new layers Added subtypes and domains New Imagery GIS Dept. (Staff and Interns) Water Billing Asset Mang / Vueworks

3 Topics Why is Data Quality important? Brentwood GIS Case Study Introduction to Data Reviewer Initial Findings Data Quality Assurance Plan (Methodology) Running Data Reviewer (Lessons Learned) Geometric Networks (Lessons Learned) Fixing the data The results The unperceived benefits

4 Why is Data Quality Important? Data needs to be: - Accessible - Reliable - Spatially accurate - Descriptive As a result, water, wastewater, and stormwater utilities are now heavily focusing on quality assurance (QA) and quality control (QC) to ensure that their GIS data truly meets their needs. ESRI White Paper: GIS Data Quality Best Practices for Water, Wastewater and Stormwater Utilities July 2011

5 Why is Data Quality Important? Summary of tools described in paper: - Geodatabases - A good data model - Versioned Environment (multiple editors) - Geometric Networks - Model Builder/ Python - ArcGIS Data Reviewer - Production Mapping Workflow Manager ESRI White Paper: GIS Data Quality Best Practices for Water, Wastewater and Stormwater Utilities July 2011

6 Why is Data Quality Important? Positional Accuracy Topological Logic Geometric Data Considerations Projections and Coordinate Systems Attribute and Data Structure

7 Why is Data Quality Important? It is very expensive to have dirty-bad or incomplete data - Data is not reliable - Tools don t work correctly - Creating custom tools is very time consuming - Trouble shooting data errors

8 Brentwood GIS Case Study GIS Workflow - Multiple editors - Typical QA/QC consisted on visual checks - No issues = no problems Until - CH2M ran a courtesy check on sewer GIS data using ArcGIS Data Reviewer - Overall GIS health was good. - Data Reviewer pointed out areas where business needs were not met 100%.

9 Shocking introduction to Data Reviewer in 2015 Consultant calls to say: I ran Data Reviewer on your data and discovered that YOUR data has lots of ISSUES

10 Initial Findings Issues Found (52,218) Sewer Dataset only: - Non compliant domains - Duplicate geometries - Multipart geometries - Duplicate ID s

11 Initial Findings Geometric Network

12 Initial Findings Missing assets 2- Pipes had wrong flow direction 3- Pipes connected to wrong asset Disconnected assets Issue Missing Assets Solution Add Taps and Tee s Flow Connectivity Trim/Extend Pipe Snap features

13 Data Quality Assurance (Methodology) Established a Data Quality Plan - Consisted on 4 pre-configured batch files addressing: 1. Invalid geometries 2. Duplicate ID s 3. Duplicate Geometries 4. Domain and Subtype Validation - Geometric Network 1. Checking all features 2. Customized for critical features Assets 1Flow Monitors 2Grease_Interceptors 3GrinderPumps 4LiftStation 5Manholes 6SewerAirReleaseValve 7SewerControlValve 8SewerFitting 9SewerGravityPipe 10SewerLateralPipe 11SewerPressurizedPipe 12SewerService 13SewerServiceValve

14 Data Quality Assurance (Methodology) CH2M provided hands on Data Reviewer training City of Brentwood self performed data clean up City of Brentwood submitted copy of the Sewer and Water feature datasets every 3-4 weeks for review CH2M Reviewed all datasets consistently and compiled information for presentation

15 Running Data Reviewer (Lessons Learned) 1. Always perform a check for invalid geometries prior to perform other checks. 2. Create batch files containing custom checks 3. Create multiple batch files and group them logically 4. Create separate database to store QAQC results

16 Running Data Reviewer (Lessons Learned) More on Invalid Geometries: - In general, invalid geometries are the worst error to fix. - It can cause tools to crash or produce incomplete results. Does this sound familiar?

17 Geometric Networks (Lessons Learned) Does not support features with M or Z values - This might require a data model change 10.2 geometric networks didn t accept versioned features - Created database replicas to perform geometric network fixes 10.3 accepts versioned features

18 Fixing the data Systematically mining the data by ObjectID Organizing the data

19 Fixing the data

20 Number of Features Number of Errors Results GIS Data Vrs. Domain/ Subtype Errors Number Of Features Validation Base R1 R2 R3

21 Results Schema changes - Removed unnecessary fields - Added new domain values to fill in the Nulls Used Data Reviewer - Identify non complaint values and fixed them - Sorted fields Ascending (typed values were on top) Tackle one attribute problem at a time - Worked on resolving Invalid values first - Address Null values second

22 Number of Features Number of Errors Results GIS Data Vrs. Duplicate ID Errors Number Of Features DuplicateIDs Base R1 R2 R3 0 Created a Python Script to assign new ID s automatically

23 Results Unique ID Check helped identify Backflow Control Valves that didn t have an ID assigned to it in the backflow control database Both Backflow control Engineer and GIS Analyst were aware of the issue. GIS Analyst followed up with Engineer to assign ID s. Brentwood gets audited once a year by the state and not having unique ID s that correlated to backflow control tables would have cause a reason to fail the audit.

24 Number of Features Number of Errors Results GIS Data Vrs. Geometric Errors Number Of Features GeometricErrors Base R1 R2 R

25 The unperceived benefits 1. Integrated systems work correctly a) Data that is part of a business process needs to be properly maintained 2. Robust datasets a) Source of truth b) Better reporting and access to more spatial analysis capabilities 3. QA/QC Industry Best Practices 4. Faster tool development a) Standardized data = less time spent developing workflow to help trouble shoot analysis work flows

26 Conclusions Data is dynamic and maintaining Data Quality is a continuous process Its expensive to have dirty / incomplete data ArcGIS Data Reviewer is easily customizable to: - Check Data (Batch or single check) - Data Health Reports A combination of visual and computerized checks is the best formula to maintain Data Quality

27 Thank you for your time. Scott Sumners: James McCord: Gerardo Boquin: