he Community Data Portal and the WIS Presented by: Michael Burek Progress, metadata architecture, and collaboration in the future
Acknowledgments NCAR - Luca Cinquini, Rob Markel, Nathan Willhelmi, Don Middleton partners - Jeremy andy, Jurgen Seib, Guillaume Aubert
Introduction to the CDP UCAR wide, uniform, community resource for accessing the data resources from a wide variety of data holders, and active campaigns MSS deep store is 1-2PB, access to ~8000 dataset collections across the organization Search and Browse services -- Support for free or structured queries to find data, boolean combinations, and Keyword, controlled vocabularies, hierarchical browse Data Publication Services -- data access control (local/remote) controlled by groups, uploading services, metadata editor, administered by data provider Visualization Services -- LAS, Unidata Integrated Data Viewer (IDV), upcoming KML interfaces Metrics Services -- tracking of data downloads, uploads, services Data delivery Services -- HP File access, aggregation, subsetting, Mass Store, OPEnDAP, DS, SRM, SRB, connections to remote FP, HP data Metadata federation services -- OAI support for DC, DIF, HREDDS, ISO Remote access of HREDDS catalogs that are exposed on HP, or DB HREDDS interfaces
Progress he CDP is a DCPC in the system and provides metadata records and references to datasets (collections) to GISCs with a URL link back to CDP dataset collections he CDP produces prototype ISO19139 records, currently v0.5, for approximately 7000 dataset collections. he extensions are built upon ISO19139 standards and are a small extension of a large standard. ISO Records are now automatically generated from HREDDS records using an XSL stylesheet during the CDP data ingest workflow he OAI (Open Archives Initive) protocol is used to export ISO19139/ records from the CDP (DCPC) and GISCs Selected ISO19139 records have been made available with an OAI server at NCAR, a mixture of (now obsolete) v0.2 and newer v0.5 prototype records he v0.2 records are harvested by SIMDA with OAI automatically, and are included in the SIMDA demo
Progress(2) he CDP harvests DWD ISO19139 DWD metadata, translation into HREDDS, DWD metadata is discoverable and browsable on the CDP he OAI server/provider software, joai, has been updated to allow incremental harvests, and more flexible sets. his software is freely available. joai supports compression during metadata transfers to facilitate metadata transfers on bandwidth limited links
CDP Distributed Metadata Usage
CDP OAI Architecture Hierarchical CDP HREDDS CDP Indexer "Flat" " HREDDS XSL xform OAI Server OAI Clients DWD, SIMDA, others... Populates Catalog Search engine req Catalog Resolver req Browser + Remote Metadata Store XML HML URL from catalog Browser req CDP Browse Populates remote URL from HREDDS catalog HML CDP Harvester Client Extension + ISO -> HREDDS xsl Writes ISO 19139 and Flat hredds records CDP External Data Center Link to data source catalog CDP OAI Harvster Remote OAI Server Remote Metadata Store Remote Data Catalog
CDP Data Delivery Architecture Extracts Headers NcML extractor Creates NCML NCML NCML NCML Uses NcML is a XML representation of the header Information Browser Downloads Extracts Data LAS/Ferret Uses OPEnDAP Subsetting server CDP Aggregation Server Data Visualizations (output) Aggregations/subsetting Any File Format Subsetting of individual files (output) Aggregations/ subsetting with OPEnDAP Access Control for restricted datasets proxy Any Client IDV (native) GRIB Code ASCII... HP Server WCS BD Web service GRIB GRIB GRIB GRIB2 GRIB2 GRIB2 DS server HREDDS HREDDS HREDDS HREDDS HREDDS HREDDS HREDDS xfrm KML KML KML KML KML GE Client
Future Work Complete work on specifying and implementing a profile of the information contained in records Complete OAI interface Extend the HREDDS to ISO19139 stylesheet to include extensions Publish schema metadata records into WIS GISCs Write ISO19139/ to HREDDS translator Harvest ISO19139 and metadata via OAI and make the results available on the CDP Write new metadata available notification extension for joai, to create a push model in the OAI framework
Possible future cooperation Further define and implement web services for data, including subsetting, aggregation and decimation that are compatible with workflows, starting wit sualization interface for GE that interfaces to WIS datasets Propose that WIS consider as a standard supported format so that aggregation and subsetting services can be leveraged Further integration of CDP user tools into workflow, such as editors, publication tools as appropriate.
Demo CDP interface to /DWD metadata http://cdp.ucar.edu/browse/browse.htm?uri=http://dataportal.ucar.edu/ metadata/wmo/wmo.thredds.xml GE Metars + CDP dataset extents
Questions/Discussion?