Steps towards a Web Data Laboratory: data analysis for the 21 st Century

Transcription

1 Steps towards a Web Data Laboratory: data analysis for the 21 st Century M. Benno Blumenthal International Research Institute for Climate and Society Columbia University

2 Data Analysis as a Service The Data Library's open data model and ability to create networks of virtual web pages and other web resources leads to some powerful applications As datasets become more complicated and difficult to handle, systems that hide that complexity and facilitate analysis become more essential Metadata and its transforms are essential Archived but accessible data are ever more important

3 Complexity pervades (*)

4 Specialized Data Tools Overview Maproom Generalized Data Tools Data Viewer Data Language IRI Data Collection Dataset Dataset Dataset Variable ivar ivar multidimensional URL/URI for data, calculations, figs, etc

5 IRI Data Collection Ocean/Atm geolocated by lat/lon multidimensional spectral harmonics equal-area grids GRIB grid codes climate divisions Economics Public Health geolocated by entity IRI Data Collection Dataset Dataset Dataset Variable ivar ivar multidimensional GIS geolocation by vector object or projection metadata

6 IRI Data Collection GRIB netcdf images binary Servers OpenDAP THREDDS spreadsheets IRI Data Collection Dataset Dataset Dataset Variable ivar ivar Database Tables queries shapefiles images w/proj

7 IRI Data Collection GRIB netcdf images binary Calculations virtual variables Servers OpenDAP THREDDS spreadsheets IRI Data Collection Dataset Dataset Dataset Variable ivar ivar Database Tables queries shapefiles images w/proj images graphics descriptive and navigational pages Data Files netcdf binary images Clients OpenDAP THREDDS Tables OpenGIS WMS/WCS KML

8 IRI General Data Tools Data Page

9 IRI General Data Tools Data Viewer

10 IRI General Data Tools Calculations: svd ( documentation (link: svd ( dataset (link: svd results ( svdview (link:

11 svd program

12 IRI General Data Tools Calculations: Cluster Analysis ( fn (link: k-means ( dataset (link: cluster results ( view (link: cluster

13 IRI General Data Tools ( page (link:figure WMS and KML: land cover

14 IRI General Data Tools ( page (link: figure WMS and KML: precipitation

15 IRI Map Room

16 IRI Map Room Malaria Early Warning System Front page illustrates most recent dekadal rainfall estimates ( RFE (FEWS Administrative and epidemiological overlays available Change dates to view different time periods Click and drag box across map to zoom

17 IRI Map Room MEWS Time Series Analyses STEP 1: Select size of domain for analysis Administrative District OR Box 11km, 33km, 55km, 111km STEP 2: Select location for which analysis will be created

18 IRI Map Room MEWS Time Series Analyses

19 MEWS tool transparently interrelates the three geospatial models Dekadal precipitation (longitude, latitude, time) District outlines Time series for districts (generated on-the-fly) from first two

20 ( link ) Faceted Search

21 Models, Crosswalks, and Objects in a single RDF/OWL framework

22 RDF Data Model Exchange Standard metadata schema RDF Standard Metadata Schema RDF RDF Standard Metadata Schema RDF RDF Tools Users RDF Datasets RDF RDF Tools Users RDF Datasets RDF Standard Metadata Schema Standard Metadata Schem Standard Metadata Schema RDF Tools Users RDF Datasets RDF Tools Users RDF Datasets RDF Tools Users RDF Datasets

23 Search Interface Data Servers IRI RDF Architecture Ontologies MMI JPL bibliography Sesame Start Point RDF/XML-Schema Crawler XSLT/GRDDL ingest XML Schema to OWL translation Owl Semantics SWRL Rules SeRQL CONSTRUCT Search Queries Standards Organizations Location Canonicalizer Time Canonicalizer

24 Semantic Crosswalk for metadata translation

25 Semantic metadata translation: maproom to GCMD DIF

26 Sample GCMD DIF-CD Record

27 OpenDAP CF to WCS Service

28 Function Documentation (*)

29 Function Semantics Used to generate function documentation Basis for more extensive function semantics Eventually would like to use to generate workflows Currently working with SSWAP to insure that it can describe these workflow steps: e.g. variable to transformed variable and variable to figure to image file.

30 SSWAP Simple Semantic Web Architecture and Protocol A way of providing a service that semantically describes its domain and range to advertise it. To invoke it, both domain and range are restricted. Traditionally we specify of chain of processing steps, and provenance documents that effort. SSWAP specifies an object by constraining it you could specify its provenance to get it traditionally, or some other quality.

31 Multiplicity of Data Representations RDF provides a unifying framework to simultaneous hold and deliver dataset metadata according to multiple standards Models, Crosswalks, and Objects organizes that framework clarifying the semantic distance spanned bidirectional XML Schema to OWL translation enables delivery of inferred metadata to existing XML-based systems Persistence with inference/transform is the underlying technology Semantic Service Framework could extend this framework to semantically-informed workflow generation

32 Data Flow based Analysis with explicit semantics data analysis data Results Semantic Web Data analysis Data

33 21 st Century data analysis Definitive web-accessible data archives Cloud data analysis services based on those archives Semantic descriptions of datasets Semantic descriptions of analysis steps Semantic assembly of workflow pipelines Science is about reproducibility, as are virtual dataset services. This means access to the data, access to the analysis methods, and commitment to archives.

34 Other Maproom Examples

35 I. Food Security: Application At the request of the UN FAO, a web-based tool was created to support Desert Locust management and control Eliminates NDVI-based error for identification of locust habitat Adds daily and 10-day CMORPH rainfall estimates for identification of potential breeding areas Michael Bell, Benno Blumenthal

36 I. Human Health: Application MODIS images: composite and NDVI are now available through IRI Health Maproom Ministry of Health in Eritrea follows NDVI indices on regular basis and provides warnings to the sub-districts Michael Bell, Benno Blumenthal, John del Corral, Emily Grover-Kopec

37 Fire Management Michael Bell, Benno Blumenthal, Joshua Qian, Andy Robertson, Michael Tippett Presentation of the tool to CARE and Ministry of Environment (Indonesia). Improvement and publications are in progress

38