EEOS 381 -Spatial Databases and GIS Applications

Transcription

1 EEOS 381 -Spatial Databases and GIS Applications Lecture 2 Data Sources Data Development and Data QA/QC

2 Guiding Principle Good quality GIS data make up the foundation of all maps and spatial analyses and applications EEOS 381 Spring 2015 Lecture 2 2

3 Questions to Ask Where do I get GIS data? What do I do with it once I get it? EEOS 381 Spring 2015 Lecture 2 3

4 Data Sources Basic choices: Use existing data (Data Transfer) best choice to get started on analysis, mapping Create your own data (Data Capture) time-consuming expensive EEOS 381 Spring 2015 Lecture 2 4

5 Data Sources Existing Data: Start with general Web search google.com, yahoo.com Search for place, data category, (e.g. Massachusetts roads GIS data ) ArcGIS Online entry point for accessing many data providers view data and add to ArcMap or web map EEOS 381 Spring 2015 Lecture 2 5

6 Data Sources Existing Data (cont.): Government agencies local, municipal regional, county state federal Look for links pages on any site for other related sites Data may be free via download, or cost money to download or receive on tape/cd/dvd, based on public records law EEOS 381 Spring 2015 Lecture 2 6

7 Data Sources Existing Data (cont.): common gov t. sites: MassGIS - mass.gov/mgis Datalayers, OLIVER (also in lab s SDE) USGS EROS Data Center NOAA NRCS EEOS 381 Spring 2015 Lecture 2 7

8 Data Sources Existing Data (cont.): common gov t. sites: U.S. Fish & Wildlife Service Census Bureau (TIGER, American FactFinder) Where you go depends on what type of data you need (specific to application) EEOS 381 Spring 2015 Lecture 2 8

9 Data Sources Existing Data (cont.): GIS data portals, clearinghouses: Data.Gov (formerly GeoSpatial One Stop) National Spatial Data Infrastructure - National Atlas - National Map - EEOS 381 Spring 2015 Lecture 2 9

10 Data Sources Existing Data (cont.): Commercial sources NavTeq, TeleAtlas (TomTom), GIS DataDepot (data.geocomm.com), GIS Lounge, TopoZone (Trails.com), TopoQuest, SPOT imagery, others Some sample data may be free via download; most sites charge money for download or data on media EEOS 381 Spring 2015 Lecture 2 10

11 Data Sources Existing Data (cont.): Data may come with software: ESRI Data & Maps Ask colleagues, teachers Call/write to agency directly See Chapter 9 in textbook Make the search effort before developing data yourself EEOS 381 Spring 2015 Lecture 2 11

12 Data Sources Creating data (Data Capture): Can account for up to 85% of the cost of a GIS Primary - direct measurement of objects for use in GIS Secondary - captured for other purposes and then converted for use in a GIS EEOS 381 Spring 2015 Lecture 2 12

13 Primary Data Capture Raster (imagery) Remote sensing (e.g. satellites, airplanes) Focus on resolution -» Spatial - pixel size» Spectral - what part of EM spectrum? (e.g. visible, IR)» Temporal - frequency of image capture Normally, hire a company to capture data, put out RFR/RFP, with spec. (specifications) Ex.: MassGIS ortho imagery and DTMs EEOS 381 Spring 2015 Lecture 2 13

14 Primary Data Capture Vector (points/lines/polygons) Surveying capture coordinates on the ground, using benchmarks, angles, distances The coordinates can then be used to create vector (point/line/polygon) features Can use Parcel Editor toolbar in ArcMap EEOS 381 Spring 2015 Lecture 2 14

15 Primary Data Capture Vector (points/lines/polygons) GPS Global Positioning System capture coordinates using satellites Coordinates then processed and converted to vector features see pages in textbook (section 5.8) EEOS 381 Spring 2015 Lecture 2 15

16 Primary Data Capture Raster and Vector LIDAR LIght Detection And Ranging Raster and Vector products Surfaces ( bare earth, last return ) Building footprints 3-D buildings See and EEOS 381 Spring 2015 Lecture 2 16

17 Secondary Data Capture Raster (imagery) Scanning maps aerial or satellite photos Images need to be registered (georeferenced) for use in GIS (use GeoreferencingToolbar in ArcMap) May be used as background GIS layer or for conversion to vector data (see next slide) EEOS 381 Spring 2015 Lecture 2 17

18 Secondary Data Capture Vector Manual digitizing - tablet with paper map Heads-up digitizing - use registered image on-screen Vectorization conversion of raster to vector automated or semi-automated methods R2V, ArcScan, GRID/Spatial Analyst or ArcToolbox tools EEOS 381 Spring 2015 Lecture 2 18

19 Secondary Data Capture Vector (cont.) Address matching (geocoding) locate points using other data sources (roads, parcels, ZIP Codes, structures, etc.) use tabular data Photogrammetry - use stereoplotters; DTM to DEM, contours COGO coordinate geometry enter bearings, distances, convert to X,Y coordinates Using a stereoplotter EEOS 381 Spring 2015 Lecture 2 19

20 Once You Get the Data Data Manipulation define/change projection (e.g. Geographic to state plane) append tiles clip out portion for project extent add/modify attributes dissolve, simplify overlay operations EEOS 381 Spring 2015 Lecture 2 20

21 Once You Get the Data Data Conversion change formats (SHP to COV, text to raster,.e00, zips, exes, etc.) import into existing database (e.g. SHP to SDE) Use your software s tools (e.g. ArcToolbox, Geoprocessing menu) EEOS 381 Spring 2015 Lecture 2 21

22 Example of Data Capture MassGIS schools data layer, initial capture Receive dbase files from MA DOE Geocode/Address Match using TIGER roads from Census Bureau Refine locations using other sources ortho imagery, USGS Quad imagery, street atlases, calling schools add fields documentation EEOS 381 Spring 2015 Lecture 2 22

23 Example of Data Capture Update to MassGIS schools data layer download new text files from MA DOE site convert CSV files to DBF add fields, calc. school type, modify attributes compare to existing layer determine which schools are new, which need to be deleted (use MS Access and ArcGIS) Address match new schools to NavTeq roads and refine locations documentation EEOS 381 Spring 2015 Lecture 2 23

24 Example of Data Capture MassGIS OpenSpace data layer Sent out base maps to each town Someone in the town added OS parcels Received base maps back Digitized (with tablet) or scanned maps (and then heads-up digitized) Added attributes MassDEP Wetlands See diagram at mass.gov/itd/wetdep EEOS 381 Spring 2015 Lecture 2 24

25 Once You Get the Data Data QA/QC Quality Assurance / Quality Control QC while data is being developed QA after you receive the data A planned and systematic pattern of all actions necessary to provide adequate confidence that the product optimally fulfills customers' expectations, i.e. that it is problem-free and well able to perform the task it was designed for. - The Free On-line Dictionary of Computing, Denis Howe EEOS 381 Spring 2015 Lecture 2 25

26 Data QA/QC What is QA/QC? VERY IMPORTANT STEP - the results of your analysis depend on the quality of your data (i.e. Garbage In, Garbage Out ) EEOS 381 Spring 2015 Lecture 2 26

27 Data QA/QC Expectations and Perspectives When designing a QA procedure, consider the following questions: What is the quality of the source data being used? How is the data being generated? What established benchmarks can the data be compared against? What can one reasonably attempt to test for? How will the QA be implemented? By whom? EEOS 381 Spring 2015 Lecture 2 27

28 Data QA/QC Source Data Quality - What is the quality of the source data being used? Consider: Paper vs. Digital sources Physical Condition / Data Accuracy Availability Scale Currentness EEOS 381 Spring 2015 Lecture 2 28

29 Data QA/QC Data Generation - How is the data being generated? Automation vs. Manual Procedures Software/OS Platforms Data Mining and Collection Techniques Hopefully the source data has METADATA! EEOS 381 Spring 2015 Lecture 2 29

30 Data QA/QC Testing Accuracy - Against what established benchmarks can the data be compared? Base Map Information Previous Incarnations Official Documents Formal Standards (e.g. National Map Accuracy Standards, MassGIS Level 3 Parcels) No Benchmarks = Indefensible QA EEOS 381 Spring 2015 Lecture 2 30

31 Data QA/QC What do you test? Geography - spatial accuracy Compare to your most accurate layer (Spatial Co-existence), aka ground truthing Linework accuracy, edgematching, shift, dangles, overshoots, gaps, overlaps, etc. Validate topology (geodatabase) Projections Use standard Pan/Zoom, print checkplots, pt-in-poly identity, etc. EEOS 381 Spring 2015 Lecture 2 31

32 Data QA/QC Example - QA/QC of roads data Does this look right? Bring in ortho image EEOS 381 Spring 2015 Lecture 2 32

33 Data QA/QC Example - QA/QC of roads data Compare with similar data set EEOS 381 Spring 2015 Lecture 2 33

34 Data QA/QC Correct geometry Overshoot (Dangle) Undershoot Edgematching problems between 2 tiles of appended data EEOS 381 Spring 2015 Lecture 2 34

35 Data QA/QC What do you test? Attributes - Consider: Required Fields Names, Types, Sizes, Order Missing? Uniqueness (constraints, unique IDs) Standardized? Range/Domain Values & Null Values Common Sense Values (e.g. 3 for a date field) Relationships (joins, links) EEOS 381 Spring 2015 Lecture 2 35

36 Data QA/QC QA/QC Procedure Considerations: Geographic Limitations e.g. check polygons over a certain size, check a certain number of features in each region Computing Limitations how long will it take to run on sample (test 10 features, extrapolate to 10,000 features) Time = Money Develop automated procedures - can t check every single feature/record manually spot check EEOS 381 Spring 2015 Lecture 2 36

37 Data QA/QC Requirements: ATTENTION TO DETAIL Knowledge of data - what am I supposed to see? What do I expect to see? (based on specification) Knowledge of appropriate software, tools (often includes programming) Know requirement of application of the data Time, staff Document everything (can add COMMENTS field) Don t be afraid to send data back or reject EEOS 381 Spring 2015 Lecture 2 37

38 Data QA/QC -Examples Is it valid? EEOS 381 Spring 2015 Lecture 2 38

39 Data QA/QC -Examples Drawing and labeling two roads layers for comparison EEOS 381 Spring 2015 Lecture 2 39

40 Data QA/QC -Examples Example of good, expected point location EEOS 381 Spring 2015 Lecture 2 40

41 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 41

42 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 42

43 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 43

44 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 44

45 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 45

46 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 46

47 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 47

48 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 48

49 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 49

50 Data QA/QC -Examples EEOS 381 Spring 2015 Lecture 2 50

51 QA needed after ambiguous geocoding match EEOS 381 Spring 2015 Lecture 2 51

52 This town hall is on Route 28, according to the ADDRESS_1 field, but is located on Old Main St., plus the point is not on a building. EEOS 381 Spring 2015 Lecture 2 52

53 Data QA/QC -Examples NAME field needs to be longer to complete the word Building. A few other records are affected as well. EEOS 381 Spring 2015 Lecture 2 53

54 Data QA/QC -Examples Fall River is a city the name of the building probably should have City Hall instead of Town Hall in it? EEOS 381 Spring 2015 Lecture 2 54

55 Data QA/QC -Examples Should Town Hall be included in the name? Municipality name is all caps, unlike other records. Also same for NEWTON and PEABODY. Does the city hall name include the word Election? The city s web site just calls it Quincy City Hall. EEOS 381 Spring 2015 Lecture 2 55