HMA for Science - Evolution. Final Presentation

Transcription

1 HMA for Science - Evolution Final Presentation Y. Coene, Spacebel s.a. Tri Tran Minh, Spacebel s.a. Minh Nguyen Quang, Spacebel s.a. 16 October 2014 ESRIN, Frascati Page 1

2 Outline Introduction Science Services Integration OpenSearch Other SSE Portal Conclusion and Future Work Demo Page 2

3 Introduction - What is HMA A collaborative project started in Europe and Canada by the Ground Segment Coordination Body (GSCB) in 2005 with the objective to: Guarantee a seamless and harmonised access to heterogeneous EO datasets from multiple mission ground segments, including national missions and ESA missions. Standardise the ground segment interfaces of the satellite missions for easier access to EO data. Provide interoperability for coordinated data access enabling the interactions with services or Value Adders and EO Contributing Missions. Page 3

4 Introduction HMA for Science FedEO Contributions to standardisation Identity Management specification Metadata specification Product discovery specification Collection discovery specification Ordering specifications Processing specifications Feasibility Analysis specification FedEO is a system providing a brokered discovery (and access) capability to European (and Canadian) EO missions data based on HMA interfaces Page 4

5 Architecture Page 5

6 Architecture Main results: Service endpoints for external clients Option 1: OpenSearch Dataset Series, Dataset Discovery and download: OpenSearch Gateway Option 2: OGC Dataset Series Discovery OGC Dataset Discovery OGC Ordering (incl. OGC ) Portal clients Page 6

7 Outline Introduction Science Services Integration OpenSearch Other SSE Portal Future Work Demo Page 7

8 Architecture OpenSearch Interface Dataset series, dataset (and services) discovery Applied for heterogeneous metadata: EOP O&M (OGC r3 and r4) metadata DC or ISO metadata RDF metadata ISO19139 metadata (as per OGC r1 and OGC r2) Interface : OpenSearch OGC r8 Geo and Time Extension OGC r4 Extension for Earth Observation Aligned with OASIS searchretrieve 1.0 conventions OpenSearch SRU Extension RESTful interface being prepared. Page 8

9 OASIS Search Web Services (SWS): SearchRetrieve searchretrieve <<Part 1>> Abstract Protocol Definition <<Part 7>> Explain <<Part 2>> Binding for SRU 1.2 <<Part3>> Binding for SRU 2.0 <<Part 4>> Binding for OpenSearch Page 9

10 OpenSearch Specification Tree searchretrieve <<Part 1>> Abstract Protocol Definition <<Part 7>> Explain <<Part 2>> Binding for SRU 1.2 <<Part3>> Binding for SRU 2.0 <<Part 4>> Binding for OpenSearch SRU Extension Parameter Extensiion <<OGC r8>> Geo and Time Extensions <<OGC r4>> EO Extension Page 10

11 Architecture - OSDD Recursive or multi-step search: Step 1: return dataset series metadata and OSDD to be used for dataset search inside the dataset series Step 2: return dataset (i.e. product/granule) metadata Approach: RFC 6573, The Item and Collection Link Relations. <Url> for step 1: Response: <dc:identifier> in response identifies dataset series (OGC ) <link rel="search" href=".." <Url> for step 2: type="application/opensearchdescription+xml"/> {eo:parentidentifier} identifies dataset series to be searched. Page 11

12 Use Case 1 : (One-Step) Search HTTP GET OpenSearch Description Document ID HTTP Response ID HTTP GET Request parentidentifier = name = frascati startdate = T00:00:00Z Spatial Extent Temporal Extent Metadata Link [Search Link] [Data Link] Spatial Extent Temporal Extent Metadata Link [Search Link] [Data Link] HTTP GET parentidentifier=eop%3aegeos%3aegeos%23ikonos&startdate= T00:00:00Z&cloudCover=10]&endDate= T00:00:00Z&name=frascati&radius=1000 Page 12

13 Use Case 2 : Two-step search HTTP GET Parent OSDD Child OSDD HTTP GET Request HTTP GET Request HTTP Response Parent ATOM Feed HTTP Response Child ATOM Feed Atom Entry 1 Atom Entry 2 Atom Entry 1 Atom Entry 2 Metadata Link Metadata Link Metadata Link Metadata Link Search Link Search Link [Data Link] [Data Link] Spatial extent Spatial extent Spatial extent Spatial extent Temporal extent Temporal extent Temporal extent Temporal extent Page 13

14 FedEO Two-step Search Search Client FedEO Partner Mission GS Collection Partner Mission Mission Catalogs GS GS Step 1: Collection Search Get OSDD OSDD for collection search Perform collection search Atom response with collections and link to OSDD for each Create OSDD based on backend capabilities. Collect responses and # of results from backend collection catalog(s). Compute first/next/previous/last links. Partner Mission GS Granule Partner Mission Mission Catalogs GS GS Return matching collections and number of hits Step 2: Granule Search Extract location of OSDD from Atom response Get OSDD for granule search Perform granule search Atom response with granules and link to thumbnail, browse, product Create OSDD based on backend capabilities. Note that atom:link may contain non-fedeo address. Return matching granules and number of hits Page 14

15 Two-Step Search Dataset Series (Collections) Described with Collection Metadata Searchable Collections Collections with metadata Collections that are searchable Collection with collection metadata but not searchable (e.g. GSCDA) Collection supporting 2-step Search Searchable Collection not yet described with collection metadata (e.g. Copernicus, ASF, ) Page 15

16 Architecture - Dataset Series Dataset Series hierarchy: Top-level: parallel search in 4 (TBC) catalogs Search inside a selected dataset series (eo:parentidentifier). Page 16

17 EOP:ASF EOP:VITO:VGT_S10 ATS_TOA_1P MSG_SEVIRI Dataset Series metadata EOP:ESA:FEDEO:COLLECTIONS EOP:ESA:GPOD-EO : : : : Dataset metadata EOP:VITO:VGT_S10 MSG_SEVIRI Metadata Legend: ISO OGC RDF Page 17

18 Architecture Dataset Series Search target: {eo:parentidentifier} Can be used in dataset series search and in dataset search. Default value is EOP:ESA:FEDEO Can be logical name for one or more endpoints No new naming conventions: CDS EO-DAIL collection identifiers kept. urn:ogc: optional VA4, GPOD, M2CS and CWIC collection identifiers kept. Allows for same collection identifier in two catalogs (use prefix) parentidentifier = EOP:ESA:GPOD-EO:ASA_IM 0P Page 18

19 Architecture Dataset Series EOP:ESA:FEDEO EOP:ESA:FEDEO:COLLECTIONS EOP:ESA:GPOD-EO EOP:ESA:EO-VIRTUAL-ARCHIVE4 EOP:VITO:VGT_S10 EOP:ASF EOP:SPOT:ALL MSG_SEVIRI ATS_TOA_1P ASA_IM 0P Queryables in OSDD differ per dataset series: fier={eo:parentidentifier} fier=eop:esa:fedeo:collections fier=eop:dlr:eoweb:geohazard.supersite.terrasar-x_ssc Page 19

20 Parameter Extension (*) Additional constraints on input parameters (in OSDD): For example (OGC Connector): fier=eop:dlr:eoweb:geohazard.supersite.terrasar-x_ssc (*) Draft with known issues (See conclusion) and competing with WADL ( and RAML (RESTful API Modeling Language) etc. and partly overlapping with searchretrieve Explain (OASIS) capabilities. Page 20

21 Architecture - Queryables Queryables differ per dataset series: «time» and «geo» namespace OGC r8 «eo» namespace OGC r4 Mapping to dataset series metadata ISO19139 (step 1) {eo:parentidentifier}, {eo:organisationname}, {eo:producttype}, {eo:instrument}, {eo:platform} Mapping to dataset metadata OGC r4 (step 2) «dc» namespace 35 queryables e.g. {eo:instrument}, {eo:cloudcover}, E.g. {dc:title}, {dc:type} (service, collection,..), {dc:publisher},.. «semantic» namespace {semantic:classifiedas} : search dataset series with (SKOS) concept URI Page 21

22 Architecture - Queryables dc:type allows for searches for dataset series, datasets or services e.g. in ISO AP, I15 EP catalogs. Both dc:type and eo:parentidentifier have expected defaults. Page 22

23 Architecture - Parameters url= httpaccept=application/atom%2bxml& query={searchterms?}& recordschema={sru:recordschema?}& startrecord={startindex?}& maximumrecords={count?}& bbox={geo:box}& Parameter conventions: searchretrieve: Part 3. searchretrieve Operation: APD Binding for SRU 2.0, Version 1.0, OASIS Standard, 30 January 2013 searchretrieve: Part 4. APD Binding for OpenSearch Version 1.0, OASIS Standard, 30 January 2013 OpenSearch Extensions SRU 1.0 Draft 1 OGC r8, OpenSearch Geo and Time Extensions Page 23

24 OpenSearch (Example with Gazetteer) Parameters: Request: Media type: Atom httpaccept Collection: IKONOS {eo:parentidentifier} Less than 10% cloud cover {eo:cloudcover} Area of interest: 1 km from Frascati {geo:name},{geo:radius} Time frame: {time:*} er=eop%3aegeos%3aegeos%23ikonos&startdate= T00:00:00Z&cloudCover=10]&endDate= T00:00:00Z&name=frascati&radius=1000 Use as client. Page 24

25 OpenSearch (Example Google Maps) Page 25

26 OpenSearch (Example - Firefox) Same result visualised directly by Firefox Web Browser Page 26

27 Architecture - Responses OGC r4 is flexible: Metadata can be embedded or be external (atom:link). FedEO Client can select recordschema (and Media Type): {sru:recordschema} : allows client to decide which metadata is to be embedded as foreign markup (iso, om, om11, dc). Give me the best metadata you have: «server-choice» By default, atom:link are included instead. Page 27

28 Architecture - Responses Access to other media types and metadata formats (atom:summary) Page 28

29 Responses - HATEOAS Hypermedia as the Engine of Application State Standardised Media types Standardised Link Relations <link rel= > (RFC 5988) first, previous, self, next and last : Navigation search : Two step search enclosure : Product download payment : Product order page (future work - TBC) Page 29

30 Responses Media RSS Use of Media RSS as per OGC r4 Access to cloud mask, thumbnail, browse image Page 30

31 Responses Media RSS FedEO Media RSS understood by third party applications Page 31

32 Architecture - Responses Improved Atom response in case of ISO collection metadata with semantic annotations (OGC r2) Page 32

33 Architecture - Exceptions Exception handling: HTTP error codes as per OGC r8 ows:exceptionreport as per OGC r9 (OGC Web Services Common Standard) Page 33

34 Architecture - Explain Explain OASIS SearchRetrieve (Part 7) Provides additional metadata about endpoint E.g. response schemas Possible values for parameters e.g. {eo:parentidentifier} Default values for search parameters etc. Available at Page 34

35 Documents - Client Partner Guide Audience: Client developers, service consumers General description of the FedEO environment and its relation to other systems Details about the FedEO query interface based on OpenSearch (e.g., principles, the search parameters, search response format, etc...) Description of two cases: one starting from a dataset series catalogue, and a second one accessing immediately the dataset catalogue Details on the catalogue connectors and how their corresponding dataset series and dataset metadata can be discovered Page 35

36 Documents Software Configuration File Software Configuration File Target audience: administrator OpenSearch Gateway. Page 36

37 Software Configuration Endpoint configuration done via configuration file(s) <dcat:dataset>.. </dcat:dataset> <dcat:dataset>.. </dcat:dataset> <dcat:dataset>.. </dcat:dataset> <dcat:dataset>.. </dcat:dataset> CSW Capabilities File (OGC ) OSDD <url> </url> CSW Capabilities File (OGC ) OSDD <url> </url> CSW Capabilities File (CWIC) Page 37

38 Documents Software Configuration Endpoint configuration done via configuration file: dc:identifier, accessurl, type (protocol). Page 38

39 Documents Software Configuration Note: File format to change in next release Page 39

40 Deployment (*) (*) (*) include series metadata Page 40

41 Deployment Architecture Connectors implemented supporting: OGC (CSW EOP EP) OGC (OpenSearch) OGC (CSW ISO AP) OGC (CSW CIM EP) OGC (CSW I15 EP) CWIC (CSW) ASF (API) Page 41

42 Science Services Integrated (Oct. 2014) Dataset Series / Collection Search OSDD ISO RDF FedEO OpenSearch OSDD OSDD OSDD OSDD OSDD OSDD ISO ISO ISO I15 (2) REF (3) GPOD (4) VA4 (5) synch (gscda) ISO ISO synch (cwic) synch/convert (fedeo) IDN ECHO (6) OSDD is collection-specific. Page 42

43 Science Services Integrated (Oct. 2014) Dataset / Product Search OSDD EOP O&M ISO DC RDF FedEO OpenSearch EOP O&M CDS DAIL SMOS M2CS GPOD VA4 DISS HARM EOP O&M (tar) ISO DC FedEO CCM CWIC VITO DLR EUM CNES ASF ECHO CCM SPIRIT SPOT CHARTER OSDD is collection-specific. Page 43

44 Science Services - examples Alaska Satellite Facility m%2bxml&parentidentifier=eop:asf&startdate= T11:59:59Z&endDate= T11:59:59Z&maximumRecords=10&frame=[290,310] Page 44

45 Science Services - examples NASA ECHO tion/atom%2bxml&parentidentifier=eop:nasa:echo&query =vegetation 1_STD_FRAME_L0&bbox=-80.0,- 90.0,80.0,90.0&startPage=1&maximumRecords=10 Page 45

46 Science Services - examples CNES SPIRIT, Disasters Charter catalogs OP:CNES:SPIRIT&query=Russia OP:CNES:CHARTER&query=Germany ESA DISSHARM Landsat 5 etc. =LANDSAT_5_TM&startDate= T00:00:00Z&endDate= T00:00:00Z Page 46

47 Science Service - DISSHARM Page 47

48 Science Services - Metrics Collections accessible through FedEO (26/09/2014) Page 48

49 Outline Overall Architecture Science Services Integration OpenSearch Other SSE Portal Conclusion and Future Work Demo Page 49

50 Discovery and Ordering Gateway Page 50

51 Discovery and Ordering Gateway OGC and integration: VITO DLR 1 VITO collection added to OGC and OpenSearch 23 DLR collections added to OGC and OpenSearch OGC integration (DLR test server). EUMETSAT 13 Collections added to OGC and OpenSearch Page 51

52 Outline Introduction Science Services Integration OpenSearch Other SSE Portal Conclusion and Future Work Demo Page 52

53 SSE Portal Instance of DAIL/SSE Portal with additional «clients» Output of Evolution framecontract Page 53

54 SSE Portal OGC Client Pre-existing client (DAIL/SSE) HMA-SE collections added to CDS EO-DAIL collection list... Page 54

55 SSE Portal - Ordering Page 55

56 SSE Portal OGC OpenSearch Client New HMA-SE client. Main purpose: test client for FEDEO Gateway Page 56

57 SSE Portal Page 57

58 SSE Portal CWIC product search Page 58

59 SSE Portal ASF Product Search Page 59

60 Outline Introduction Science Services Integration OpenSearch Other SSE Portal Conclusion and Future Work Demo Page 60

61 Conclusion Synergies with HMA-S FedEO OpenSearch implementation pre-dated HMA-S specifications and provided inputs/feedback to HMA-S Tasks 3 and 4 (generalisation, ISO19139, MediaRSS, etc.). FedEO OpenSearch implementation more general than HMA-S OpenSearch specification. FedEO implementation exploits OGC r4 EOP O&M specification (Task 3): CDS EO-DAIL, M2CS, ASF product metadata in EOP O&M available now. G-POD, SMOS, VA4 available on test platform. FedEO implementation relies on software components from HMA-S and raised NCR against these components. I15 EP Catalog Implementation (Task 5) OGC r8 Catalog Implementation (Task 3) Page 61

62 FedEO Components OpenSearch Gateway Being reused in ESA CDS v3 (OpenSearch to OGC ). Open-Source Components available at: Documentation available on ESA HMA Wiki. Page 62

63 Conclusion I/F Architecture exploits OGC r4 flexibility: Mapping to EOP O&M Metadata (OGC r4) Mapping to ISO19139 Metadata Two-step and/or one step approach Heterogeneous metadata types either: Embedded also ngeo approach By reference (atom:link) CEOS Recommendation. Complemented with existing OpenSearch extensions: "sru" "semantics" Can be extended with additional media types (RSS, SRU, RDF, HTML, JSON ) Can be extended with additional link types (HATEOAS), e.g. for ordering, metadata editing,... Page 63

64 Conclusion - Issues Implementation/Specification Issues <atom:link rel= alternate type= gml+xml href= /> Links to metadata record of a certain type. No media types defined to distinguish EOP O&M 1.0 (OGC r3) from EOP O&M 1.1 Solution such as OGC (Addition of a profile parameter to GML MIME type) is needed. Parameter extension Should allow <Parameter/> in global scope: to allow for shared Parameter definitions for multiple <Url>. name attribute should be optional and value attribute should be mandatory. Is currently the opposite. Should allow to define a default value. Page 64

65 Conclusion - Issues Implementation/Specification Issues OGC r4 has all its queryables under eo: namespace. Some should be dc: instead. Conformance Testing Conformance Tests from HMA-S for OpenSearch Procedural Not Tester-friendly: unsupported optional parameters flagged as errors. Interpretation of empty BBOX should be a point. Flag error if EOP O&M not embedded inside response: fails if a derived type (opt:, sar: etc.) or if included as atom:links or if other metadata type (ISO, ) Who should contact whom when backend-systems not operational or failing. Page 65

66 Future work Outline Metadata completion Proposed interface changes CNES RESTo query analyzer integration Interface harmonisation (homogeneous metadata formats) RESTful and Linked Data interfaces Page 66

67 Future work Collection Metadata Much remaining work to populate Dataset Series metadata catalogs. E.g. additional metadata in EOP:ESA:FEDEO:COLLECTIONS R&parentIdentifier=EOP:ESA:FEDEO:COLLECTIONS&type=col lection Has positive effect on #collections that support 2-STEP search in FedEO. 511 Page 67

68 Future work Interface changes: Implement additional CEOS OpenSearch recommendations Improve compliance with INSPIRE Download Service Technical Guidance. Harmonise Atom response messages from different backends Have 2 nd -step OSDD link point to FEDEO in all cases. Start counting results from 1 in all cases. Support ISO and EOP O&M metadata format for more backends (GPOD, VA4 etc.) Page 68

69 Future work Interface changes (cont d): Replace dc:subject by eo:parentidentifier consistently. Reduce supported parameters for dataset search in principal OSDD to those required by CEOS. Refine OSDD generated for OGC Connector: e.g. different parameters in case of Optical or Radar collection. Requires this to be discoverable on CDS side. Support «virtual collections» and efficient parallel search with correct paging for product search. Integration with Ordering via HATEOAS Could more seamlessly integrate OpenSearch Gateway with HMA OGC ordering Page 69

70 Query Analyzer Integration Integration with CNES RESTo Query Analyzer. Simplied OSDD understood by Web Browser search I/F <?xml version="1.0" encoding="utf-8"?> <OpenSearchDescription xmlns=" xmlns:dc=" xmlns:eo=" xmlns:geo=" xmlns:semantic=" /opensearch/extensions/semantic/1.0/" xmlns:sru=" xmlns:time=" <ShortName>FEDEO</ShortName> <Url template=" ataset&query={searchterms?}&startrecord={startindex?}&maximumrecords={count? } &recordschema={sru:recordschema?}" type="application/atom+xml"/> </OpenSearchDescription> Page 70

71 Query Analyzer Integration Gateway obtains platform, instrument and the other parameters from the CNES RESTo Query Analyzer Search Terms FedEO Gateway (P3) List of OpenSearch parameters RESTo Query Resolver (P7) Example: Search Terms: Images acquired today by Spot HRV List of OpenSearch parameters returned by RESTo Query Analyzer: "eo:platformshortname":"spot% "eo:instrument":"hrv%" "time:start":" t00:00:00 "time:end":" t23:59:59" Page 71

72 Query Analyzer Integration Integration with CNES RESTo Query Analyzer: type=dataset&query=images acquired today by Spot HRV acquired yesterday by radarsat sar Page 72

73 Future work - Harmonisation Harmonisation Collection Metadata format (OGC r1) Harmonisation Product Metadata format (OGC r4) Page 73

74 Future work - Harmonisation Harmonise Collection Metadata Format ion/atom%2bxml&query=mer_frs_1p CURRENT NEW Page 74

75 Future work - Harmonisation Harmonise Product Metadata Format ntidentifier=mer_frs_1p&startdate= t00:00:00z&enddate= T00:00:00Z NEW CURRENT Page 75

76 Future Work Faceted Search Faceted Search allowing users to explore a collection of information by applying multiple filters (*) E.g. provide links to (dataset series) results per group (e.g. FedEO, CWIC, ECHO, VA4, GPOD, CSCDA etc.) or other classification. In roadmap of CNES and NASA implementations (Source: WGISS #38 Moscow) Can be integrated in Atom-based responses using rel=: Advertise number of items (if known): thr:count=123 ( Other approaches at and Addressed in OASIS searchretrieve specifications. (*) Page 76

77 Future Work FedEO Dataset Series Server / ISO support Current implementation HMA-S (Task 5) I15 EP Implementation with OpenSearch front-end. Possible alternative Derived from HMA-S (Task 3) EOP O&M Catalog Server. Advantages: Lightweight/fast. More native OpenSearch and scalable. Based on SolR/Lucene -> generate Parameter extension / facet information - from index. Less dependent on metadata format, e.g. future support for ISO and -3. Page 77

78 Future work - Semantic Annotations HTML Microdata: Recommended by CEOS. May be indexed by major search engines: Google, Yahoo, Bing Support in HTML response of FedEO OpenSearch Gateway </li> <li itemscope itemtype=' <span itemprop='alternatename'>doi: /ornldaac/1</span> <span itemprop='version' >1</span> <span itemprop='provider' >ORNL_DAAC</span> <div itemprop='description' >ABSTRACT: USGS 15 minute stream flow data for Kings Creek on the Konza Prairie</div> <span itemprop='temporal' > T00:00:00Z/</span> <span itemprop='spatial' > </span> Page 78

79 Future work Additional RESTful binding RESTful requests: Resources Description../series - GET Dataset series search../series/{id} - GET Dataset series with given ID../series/{id}/datasets - GET Dataset search../ series/{id}/datasets/{id} - GET Dataset with given ID Response type controlled by: Explicit parameter: httpaccept (current implementation) Resource extension:.atom,.rss,.rdf (CEOS Best Practice) Content negotiation: Atom or RSS or RDF or HTML etc... (Linked Data requirement) Page 79

80 Future work Linked Data Open Linked Data: Additional media types: RDF media type for dataset series. Make available FedEO metadata to the Linked Open Data (LOD) community. W3C RDF, DCAT, LDP 1.0 ESA FedEO Page 80

81 Future work Linked Data CEOS? eo:instrument eo:platform URI isni.org? organisation URI GEMET keyword URI loc.gov Language URI ESA FedEO dataset series dataset URI NASA KMS Keyword Mgt System loc.gov Country URI ESA RARE Multidomain thesaurus Page 81

82 Outline Introduction Science Services Integration OpenSearch Other SSE Portal Conclusion and Future Work Demo Page 82

83 FedEO Demo Examples of two-step approach: collections and granule search Page 83

84 FedEO Demo Page 84

85 Additional information Further reading: ESA_TM-21_Heterogeneous_Missions_Accessibility Page 85

86 Spare slides Page 86

87 FedEO Client Applications News reader: SharpReader ( Test file fedeo-public.opml Page 87

88 FedEO Client Applications CNES Mapshup Client GSCB Federation Pilot #3 (June 2014) Page 88

89 FedEO Client Applications Android App SmartHMA Page 89

90 FedEO Client Applications Android App Proof of Concept Page 90

91 ISO Metadata Landscape Page 91