Lecture 11. A Nationwide Infrastructure for Water Observations

Transcription

1 Lecture 11 A Nationwide Infrastructure for Water Observations GIS in Water Resources Spring 2015 Acknowledgement Many slides of this lecture were prepared by: David Maidment David Tarboton Kim Schreuder Jeff Horsburgh Ilya Zaslavsky David Valentine U of Texas Utah State Utah State Utah State San Diego Supercomputer Center San Diego Supercomputer Center 1

2 How is new knowledge discovered? By deduction from existing knowledge By experiment in a laboratory By observation of the natural environment Deduction Isaac Newton Deduction is the classical path of mathematical physics Given a set of axioms Then by a logical process Derive a new principle or equation In hydrology, the St Venant equations for open channel flow and Richard s equation for unsaturated flow in soils were derived in this way. Three laws of motion and law of gravitation (1687) 2

3 Experiment Louis Pasteur Experiment is the classical path of laboratory science a simplified view of the natural world is replicated under controlled conditions In hydrology, Darcy s law for flow in a porous medium was found this way. Pasteur showed that microorganisms cause disease & discovered vaccination Foundations of scientific medicine Observation Charles Darwin Observation direct viewing and characterization of patterns and phenomena in the natural environment In hydrology, Horton discovered stream scaling laws by interpretation of stream maps Published Nov 24, 1859 Most accessible book of great scientific imagination ever written 3

4 Conclusion for Hydrology Deduction and experiment are important, but hydrology is primarily an observational science discharge, climate, water quality, groundwater, measurement data collected to support this. Great Eras of Synthesis 2020 Scientific progress occurs continuously, but 2000 there are great eras of 1980 synthesis many 1960 developments happening at once that 1940 fuse into knowledge 1920 and fundamentally change the science 1900 Hydrology (synthesis of water observations leads to knowledge synthesis) Geology (observations of seafloor magnetism lead to plate tectonics) Physics (relativity, structure of the atom, quantum mechanics) 4

5 CUAHSI Hydrologic Information System (HIS) team University of Texas at Austin David Maidment, Tim Whiteaker, James Seppi, Fernando Salas, Harish Sangireddy, Jingqi Dong San Diego Supercomputer Center Ilya Zaslavsky, David Valentine, Tom Whitenack, Matt Rodriguez Utah State University David Tarboton, Jeff Horsburgh, Kim Schreuders, Justin Berger University of South Carolina Jon Goodall, Anthony Castronova Idaho State University Dan Ames, Ted Dunsford, Jiri Kadlec CUAHSI Program Office Rick Hooper, Yoori Choi HIS Goals Data Access providing better access to a large volume of high quality hydrologic data; Hydrologic Observatories storing and synthesizing hydrologic data for a region; Hydrologic Science providing a stronger hydrologic information infrastructure; Hydrologic Education bringing more hydrologic data into the classroom. 5

6 Component 1: Desktop Hydrologic Information System Modeling Observations Weather and Climate GIS Remote Sensing Component 2: Services Oriented Architecture for Water Data Catalogs Metadata Search Data Servers Users 6

7 Crossing the Digital Divide Discrete spatial objects with time series Observations Continuous space time arrays Weather and Climate GIS Remote Sensing These are two very different data worlds Focus on Water Observations Data Modeling Observations Weather and Climate GIS Remote Sensing We have focused on water observations data 7

8 Water Observations Data Measured at Gages and Sampling Sites Water quantity Rainfall Soil water Water quality Time series of observations at point locations Meteorology Groundwater Water Data Web Sites We need a process of archive web enablement.... discovering, accessing, and synthesizing data from the internet 8

9 How does the internet work? This is how it got started.. Web servers Text, Pictures in HTML Mosaic browser..this is how it works now Three key components linked by services and a common language Google, Yahoo, Bing Catalogs Web servers Servers Text, Pictures in HTML Firefox, Internet Explorer Users 17 What has CUAHSI Done? Taken the internet services model.. Catalogs Servers Users..and implemented it for water observations data HIS Central Time series data HydroServer, Agency Servers HydroDesktop, HydroExcel,... in WaterML 9

10 CUAHSI HydroDesktop A Hydrologic Information System Searching and Graphing Time Series 10

11 CUAHSI Network Observations Model NWIS Daily Values Data Service NWIS Sites Network GetSites San Marcos River at Luling, Tx Discharge, stage (Daily or instantaneous) Sites Variables 18,700 cfs, 3 July 2002 GetSiteInfo GetVariableInfo GetValues Observation {Value, Time, Qualifier} A data source operates an observation network A network is a set of observation sites A site is a point location where one or more variables are measured A variable is a property describing the flow or quality of water A value is an observation of a variable at a particular time A qualifier is a symbol that provides additional information about the value Terrain flow information model. The way that data is organized can enhance or inhibit the analysis that can be done Raw DEM Pit Removal (Filling) Flow Field Channels, Watersheds, Flow Related Terrain Information 11

12 Observation Data Model for hydrologic and environmental measurements The way that data is organized can enhance or inhibit the analysis that can be done Streamflow Groundwater levels Precipitation & Climate Water Quality Soil moisture data Flux tower data Why an Observations Data Model Provides a common persistence model for observations data Syntactic heterogeneity (File types and formats) Semantic heterogeneity Language for observation attributes (structural) Language to encode observation attribute values (contextual) Publishing and sharing research data Metadata to facilitate unambiguous interpretation Enhance analysis capability 24 12

13 Scope Focus on Hydrologic Observations made at a point Exclude Remote sensing or grid data. Primarily store raw observations and simple derived information to get data into its most usable form. Limit inclusion of extensively synthesized information and model outputs at this stage. What are the basic attributes to be associated with each single data value and how can these best be organized? Value DateTime Variable Location Units Interval (support) Accuracy Offset OffsetType/ Reference Point Source/Organization Censoring Data Qualifying Comments Method Quality Control Level Sample Medium Value Type Data Type 13

14 CUAHSI Observations Data Model A relational database at the single observation level (atomic model) Stores observation data made at points Metadata for unambiguous interpretation Traceable heritage from raw measurements to usable information Standard format for data sharing Cross dimension retrieval and analysis Precipitation & Climate Streamflow Water Quality When V i t What Variables, V Time, T Groundwater levels Soil moisture data Flux tower data A data value v i (s,t) s Where Space, S Data Values indexed by What where when Time, T When t A data value v i (s,t) s Where What V i Space, S Variables, V 14

15 Data Values Table Time, T t v i (s,t) s V i Space, S Variables, V Data Series Metadata description End Date Time, t 2 Time There are C measurements of Variable V i at Site S j from time t 1 to time t 2 Count, C Begin Date Time, t 1 Site, S j Space Variable, V i Variables 15

16 Data Storage Relational Database Values Value Date Site Variable Sites Site Name Latitude Longitude Value Name Date Latitude Site Variable Longitude 4.5 Cane3/3/2007 Creek Streamflow Cane3/4/2007 Creek Streamflow Town3/3/2007 Lake Temperature Town3/4/2007 Lake Temperature Site Name Latitude Longitude 1 Cane Creek Town Lake Simple Intro to What Is a Relational Database Why Use a RDBMS Mature and stable technology Structured Query Language (SQL) Sharing of data among multiple applications Data integrity and security Access by multiple users at the same time Tools for backup and recovery Reduced application development time 16

17 CUAHSI Observations Data Model Horsburgh, J. S., D. G. Tarboton, D. R. Maidment and I. Zaslavsky, (2008), A Relational Model for Environmental and Water Resources Data, Water Resour. Res., 44: W05406, doi: /2007wr At last 2 1 Work from Out to In And don t forget 4 CUAHSI Observations Data Model 17

18 Simplified ODM Structure Sites Table SiteID SiteCode SiteName Latitude Longitude LatLongID 1 AcmeP1 Backyard Pond AcmePR2 Mill River gage Station Spatial References Table SpatialReferenceID SRSID SRSName 0 Unknown NAD NAD83 Units Table UnitsID UnitsName UnitsAbbreviation 12 parts per million ppm 23 cubic feet per second cfs What are the basic attributes to be associated with each single data value and how can these best be organized? Value DateTime Variable Location Units Interval (support) Accuracy Offset OffsetType/ Reference Point Source/Organization Censoring Data Qualifying Comments Method Quality Control Level Sample Medium Value Type Data Type 18

19 Site Attributes SiteCode, e.g. NWIS: SiteName, e.g. Logan River Near Logan, UT Latitude, Longitude Geographic coordinates of site LatLongDatum Spatial reference system of latitude and longitude Elevation_m Elevation of the site VerticalDatum Datum of the site elevation Local X, Local Y Local coordinates of site LocalProjection Spatial reference system of local coordinates PosAccuracy_m Positional Accuracy State, e.g. Utah County, e.g. Cache Variable attributes Cubic meters per second Flow m 3 /s VariableName, e.g. discharge VariableCode, e.g. NWIS:0060 SampleMedium, e.g. water ValueType, e.g. field observation, laboratory sample IsRegular, e.g. Yes for regular or No for intermittent TimeSupport (averaging interval for observation) DataType, e.g. Continuous, Instantaneous, Categorical GeneralCategory, e.g. Climate, Water Quality NoDataValue, e.g

20 Scale issues in the interpretation of data The scale triplet a) Extent b) Spacing c) Support quantity quantity quantity length or time length or time length or time From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p. The effect of sampling for measurement scales not commensurate with the process scale (a) spacing too large noise (aliasing) (b) extent too small trend (c) support too large smoothing out From: Blöschl, G., (1996), Scale and Scaling in Hydrology, Habilitationsschrift, Weiner Mitteilungen Wasser Abwasser Gewasser, Wien, 346 p. 20

21 ValueAccuracy A numeric value that quantifies measurement accuracy defined as the nearness of a measurement to the standard or true value. This may be quantified as an average or root mean square error relative to the true value. Since the true value is not known this may should be estimated based on knowledge of the method and measurement instrument. Accuracy is distinct from precision which quantifies reproducibility, but does not refer to the standard or true value. ValueAccuracy Accurate Low Accuracy Low Accuracy, but precise Data Types Continuous (Frequent sampling fine spacing) Sporadic (Spot sampling coarse spacing) Cumulative Incremental Average Maximum Minimum V(t) Constant over Interval Categorical t 0 Q( )d V ( t) Q ( t) t V(t) t t t Q( )d 21

22 Independent of, but can be coupled to Geographic Representation ODM Arc Hydro Feature Observations Data Model Sites SiteID SiteCode SiteName Latitude Longitude 1 1 OR CouplingTable SiteID HydroID 1 1 HydroPoint HydroID HydroCode FType Name JunctionID ComplexEdgeFeature HydroEdge HydroID HydroCode ReachCode Name LengthKm LengthDown FlowDir FType EdgeType Enabled * Waterbody HydroID HydroCode FType Name AreaSqKm JunctionID * HydroNetwork SimpleJunctionFeature 1 HydroJunction 1 HydroID HydroCode NextDownID LengthDown DrainArea FType Enabled AncillaryRole Watershed HydroID HydroCode DrainID AreaSqKm JunctionID NextDownID * EdgeType Flowline Shoreline Stage and Streamflow Example 22

23 Water Chemistry from a profile in a lake Loading data into ODM Interactive OD Data Loader (OD Loader) Loads data from spreadsheets and comma separated tables in simple format Scheduled Data Loader (SDL) Loads data from datalogger files on a prescribed schedule. Interactive configuration SQL Server Integration Services (SSIS) Microsoft application accompanying SQL Server useful for programming complex loading or data management functions OD Data Loader SDL SSIS 23

24 Managing Data Within ODM ODM Tools Query and export export data series and metadata Visualize plot and summarize data series Edit delete, modify, adjust, interpolate, average, etc. HydroServer Goals A platform for publishing space time hydrologic datasets that is: Self contained fully documented with local control of data Makes data universally available Combine spatial data and observational data Autonomous e.g., functional independent of the rest of HIS 24

25 Ongoing Data Collection Point Observations Data Internet Applications Historical Data Files ODM Database GIS Data GetSites GetSiteInfo GetVariableInfo GetValues WaterML WaterOneFlow Web Service HydroServer Data presentation, visualization, and analysis through Internet enabled applications

26 Syntactic Heterogeneity Multiple Data Sources With Multiple Formats Excel Files Text Files Access Files Data Logger Files ODM Observations Database From Jeff Horsburgh Semantic Heterogeneity General Description of Attribute USGS NWIS a EPA STORET b Structural Heterogeneity Code for location at which data are collected "site_no" "Station ID" Name of location at which data are collected "Site" OR "Gage" "Station Name" Code for measured variable "Parameter"? c Name of measured variable "Description" "Characteristic Name" Time at which the observation was made "datetime" "Activity Start" Code that identifies the agency that collected the data "agency_cd" "Org ID" Contextual Semantic Heterogeneity Name of measured variable "Discharge" "Flow" Units of measured variable "cubic feet per second" "cfs" Time at which the observation was made " " " :00:00" Latitude of location at which data are collected "41 44'36" " " Type of monitoring site "Spring, Estuary, Lake, Surface Water" "River/Stream" a United States Geological Survey National Water Information System ( b United States Environmental Protection Agency Storage and Retrieval System ( c An equivalent to the USGS parameter code does not exist in data retrieved from EPA STORET. From Jeff Horsburgh 26

27 Overcoming Semantic Heterogeneity ODM Controlled Vocabulary System ODM CV central database Online submission and editing of CV terms Web services for broadcasting CVs Variable Name Investigator 1: Temperature, water Investigator 2: Water Temperature Investigator 3: Temperature Investigator 4: Temp. ODM VariableNameCV Term Sunshine duration Temperature Turbidity From Jeff Horsburgh Dynamic controlled vocabulary moderation system ODM Data Manager ODM Website ODM Tools XML ODM Controlled Vocabulary Moderator Local ODM Database Local Server ODM Controlled Vocabulary Web Services Master ODM Controlled Vocabulary From Jeff Horsburgh 27

28 HydroServer Implementation in WATERS Network Information System National Hydrologic Information Server San Diego Supercomputer Center 11 WATERS Network test bed projects 16 ODM instances (some test beds have more than one ODM instance) Data from 1246 sites, of these, 167 sites are operated by WATERS investigators ICEWATER A Regional HIS ICEWATER INRA Constellation of Experimental WATERsheds Coalition of 8 universities WA OR ID MT WY Point Observations Stream gages Water quality sampling Weather stations Soil moisture Snow monitoring Groundwater level/quality Spatially Distributed Data Land use/cover Terrain Hydrography AK CA NV UT AZ CO NM 28

29 Sustainability Principles Servers maintain their own complete data and metadata. Local control of data that is complete and self describing Adherence to standards Open Source Minimize custom programming Maintain syntactic and semantic consistency Data repositories are required Proposed HydroServer Access Control User Authentication HydroServer Services Security Service Data consumer provides credentials Security service returns a token WaterOneFlow Web Service Data Store (ODM) Data consumer calls GetValues using the token User Authorization and Data Access The token is evaluated to see if the consumer is authentic and authorized True (Authorized), False (Not Authorized), or Error (Token Not Found) Data returned to consumer Data Access Logged True Get DataValues from the data store Returns DataValues 29

30 Why Access Control Significant feedback from academic users: Control who can download data How, when, and if data go from private to public Publish papers before data are released Track who is downloading their data Have and use a data use/access agreement Only expose the best or highest quality data Integrate data organization, management, and publication Some say that they will not publish their data using the CUAHSI HIS until they have access control An online collaborative environment centered on the sharing of hydrologic data and models Simple and easy to use Find, create, share, connect, integrate, work together online Leverage existing online sharing and collaboration platforms Hydro value added 30

31 Purpose Facilitate collaboration Provide a place for HydroDesktop users to simply upload and publish data Support immutable archive data collections as well as transient work in progress data sharing Support seeing inside data collections to facilitate integration and synthesis across datasets An example From Tim Whiteaker 31

32 CUAHSI Online Data analysis and publication use case 1 Observer or instrument 2 HydroServer 3 5 HydroDesktop 4 10 HIS Central Web Interface (HUBzero) Web Services (WaterML) 8 Data Storage (irods) 7 Present HIS CUAHSI Online Publishing an ODM Water Data Service Utah State University University of Florida University of Iowa Assemble Data From Different Sources Ingest data using ODM Data Loader WaterML Observations Data Model (ODM) USU ODM UFL ODM Load Newly Formatted Data into ODM Tables in MS SQL/Server UIowa ODM Wrap ODM with WaterML Web Services for Online Publication 32

33 WaterML as a Web Language USGS Streamflow data in WaterML language Discharge of the San Marcos River at Luling, TX June 28 July 18, 2002 This is the WaterML GetValues response from NWIS Daily Values Publishing a Hybrid Water Data Service WaterML USGS Metadata are Transferred to CUAHSI HIS Central USGS METADATA Metadata From: Data Dump from USGS to CUAHSI HIS Central USGS Water Data Service USGS DataValues Get Values from: USGS WaterML Web Service Web Services can both Query the HIS Central for Metadata and use a USGS WaterML Web Service for Data Values Calling the WSDL Returns Metadata and Data Values as if from the same Database 33

34 Data managed independently at each site and ASCII files sent to a national CZO portal at SDSC Published in WaterML NCDC Integrated Station Hourly Data Hourly weather data up to 36 hours ago 13,628 sites across globe 34 variables Published by National Climate Data Center and populated with weather observations from national weather services 34

35 USGS Instantaneous Data Real time, instantaneous data over the last 60 days sites, nationally for the US 80 variables Published by USGS National Water Information System Corps of Engineers Water Observations Time series at Corps gages 2210 sites, mainly in Mississippi Basin 80 variables 4954 series Published by Corps of Engineers, Rock Island District to support their WaterML plugin to HEC DSS 35

36 Reynolds Creek Experimental Watershed 1 data service 84 sites 65 variables 372 series 17.8 million data Published by USDA ARS as part of an Idaho Waters project Iowa Tipping Bucket Raingages Data Manager: Nick Arnold, IIHR 72 36

37 The CUAHSI Water Data Catalog 57 services 15,000 variables 1.8 million sites 9 million series... All the data is accessible in WaterML 4.3 billion data Values 73 What have we learned? Three core patterns Centralized data services using ASCII file ingestion; ODM based data services at a university Water agency data services from USGS, EPA, NWS,. The metadata describing these water agency services is huge and is difficult to ingest and manage centrally 37

38 Data Searching Search multiple heterogeneous data sources simultaneously regardless of semantic or structural differences between them Searching each data source separately NWIS request return request return request return NAWQA request return request return NAM 12 request return request return request return NARR Semantic Mediation GetValues Searching all data sources collectively GetValues NWIS GetValues generic request GetValues GetValues GetValues NAWQA GetValues GetValues NARR HODM 38

39 Hydrologic Ontology CUAHSI HIS: We are doing this now We ve built a very large scale prototype..we ve discovered that simple but general patterns exist GetSites GetSiteInfo (WaterML) HydroServer (ODM) HIS Central GetValues (WaterML) GetSeriesCatalogForBox (XML) GetWaterOneFlowServiceInfo (XML) GetOntologyTree (XML) HydroDesktop All these services are custom programmed.... we can transition to using OGC web service standards 39

40 Open Geospatial Consortium Web Services Sensor Observation Service Web Processing Service Web Coverage Service Web Feature Service Web Map Service Remote Sensing Using an OGC standards based approach we can cross the digital divide OGC Sensor Web Enablement Image from Arne Broering, 52North 40

41 Sensor Observations Service: Get Observation Observed Property := Wind_Speed Result Sampling Time Feature of Interest 23 m/s :45 uom Procedure (ID := DAVIS_123 ) Observation Slide adapted from Arne Broering, 52North Data Access Workflow Get Services Query for matching Services from HydroCatalog A national water portal? Narrow OGC Catalog Services for the Web Get Metadata Metadata in space Query for matching Series from each HydroServer Narrow OGC Web Feature Service Get Data Observations in time Better water science!! Get Values from each HydroServer Narrow Produce the final Theme WaterML and future OGC WaterML2 standard 41

42 Observations Metadata Web Feature Service USGS Streamflow and Nexrad Rainfall in CAPCOG region Tropical Storm Hermine, 8 Sept 2010 An ArcGIS map service in space Tropical Storm Hermine CRWR Map service Tropical Storm Hermine CRWR Layer Package 42

43 Summary CUAHSI has constructed a very large scale prototype A services oriented architecture with distributed data and centralized metadata This performs syntactic mediation (unity of format in WaterML) and semantic mediation (unity of meaning using concept ontology) The patterns revealed by the prototype show that the same functions can be performed using OGC and ESRI map services supported by a time series services for the observations values Same pattern that CUAHSI has developed can be applied in different application contexts (HydroDesktop, ESRI,..) Can continue with centralized metadata for water research servers, but need to have distributed metadata for water agency servers OGC Services are the key to making a services oriented architecture for water data Summary HydroServer provides a self contained autonomous data publication system Local control of data, but universally accessible Downloadable user (data publisher) configurable software stack that contains: ODM and associated tools WaterOneFlow web services Geographic data sharing using WFS, WCS, WMS from ArcGIS server Time Series Analyst ArcGIS server based web map application HydroServer Capabilities web service that publishes metadata about regions and services (observational and spatial) Registering with HIS Central makes your data searchable 43