Enabling BigData Workflows in Earth Observation Science

Transcription

1 Enabling BigData Workflows in Earth Observation Science A RESEARCH NETWORKING PERSPECTIVE Jakob Tendel Maria Minaricova DFN GÉANT

2 Outline Research and Education Networking National Research and Education Networks GÉANT R&E Services COPERNICUS Collaboration with ESA Sentinel Data Hubs on GÉANT International contract services Collaboration with EUMETSAT Multicast across network domains Single point of contact services R&E Services Relevant to Individual Researchers

3 Research and Education Networking

4 Research and Education Networks Dedicated Internet networks Serve publicly funded research and academic education organisations Architected and tuned to best performance for scientific data Most countries have one designated or de facto national R&E network Strong international R&E community enabling cross-border services

5 National R&E Network Founded in 1984 Non-government, non-profit Association revenue mainly service fees no government subsidies since member organisations 236 higher education, 84 government/public, 12 industry 701 sites served with DFNInternet Vast majority of academic researchers and students in Germany

6 International R&E Network Collaboration Reformed in 2014 Membership Association of almost all European NRENs International connectivity between NRENs Recipient and coordinator of EC grant funds under the series of GNx projects Project management Service development and provision Partner and connector in global R&E networking moving 1Pb/day

7 R&E Services Connectivity High-velocity, high-volume data transfer Low latency access to off-site resources Wide-area IP backbone Point-to-point light-path services Connectivity to commercial upstream networks ( The Internet ) For incidental internet use, not sustained high-volume science data traffic Authentication AAI single-sign-on infrastructure PKI identity certificate services Support for research projects Testing and troubleshooting connectivity Coordinating international connectivity setups Contracting and project management support

8 COPERNICUS SUPPORTING THE DISTRIBUTION OF COPERNICUS DATA OVER R&E NETWORKS

9 COPERNICUS Data Product Categories: Sentinel Sensor (Level1) data products Copernicus Services (Level2) data products In-situ and Contributing Missions data Data Provision Responsibilities: ESA: Sentinel products, Copernicus Services data products Sentinels Scientific Data Hub scihub.copernicus.eu Copernicus Services Data Hub cophub.copernicus.eu Sentinels International Access Hub inthub.copernicus.eu Sentinel-1 Collaborative Data Hub colhub.esa.int EUMETSAT: Sentinel 3-6 Marine, Climate, and Atmosphere products EUMETCast eumetsat.int/website/home/data/datadelivery/eumetcast Online Data Access eumetsat.int/website/home/data/datadelivery/onlinedataaccess

10

11 R&E Support For ESA COPERNICUS R&E users are a significant target audience Tend to be regular, high-volume data consumers Are already served by a dedicated network infrastructure How Best to disseminate to R&E users over the R&E networks? The European R&E networking community is represented by GÉANT GÉANT maintains links to worldwide R&E networks Implement COPERNICUS dissemination via GÉANT

12 ESA COPERNICUS Dissemination Data Processing Hubs Pickup Point Satellite Downlink COPERNICUS WAN Product Generation Archives Frankfurt a.m., Germany

13 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives

14 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives

15 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives DFN NREN A GÉANT NREN B

16 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives DFN NREN A GÉANT NREN B

17 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives DFN NREN A GÉANT NREN B

18 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives DFN NREN A GÉANT NREN B

19 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives DFN NREN A GÉANT NREN B

20 ESA COPERNICUS Dissemination Satellite Downlink Data Processing COPERNICUS WAN Product Generation Hubs Pickup Point T The Internet Archives DFN NREN A Option: Availability of 9K Jumbo frames across R&E networks Much lower protocol overhead, greater effective data throughput GÉANT NREN B

21 ESA COPERNICUS Contract Services ESA data hub in Germany DFN for GÉANT access from Germany ESA EO unit in Italy Cross-border contract issues Contract GÉANT Data Contract DFN SOLUTION: GÉANT signs agreement with ESA GÉANT subcontracts DFN DFN implements connection in DE ESA ESRIN ITALY Data Data Hubs GERMANY

22

23 What R&E has done for them EUMETCast data distribution using IP multicast from EUMETSAT HQ in Germany to 31 countries in Europe, using GÉANT and the NRENs networks A Single point of contact for EUMETSAT connecting them with the whole community In the absence of a formal SLA: Implementation of added-value services that support an operational-like data provision Monitoring, reporting and escalation processes

24 Service portfolio Feasibility Analysis in respect to connecting individual end-points Single Point of Contact and activity coordinator Technical Service Support and escalation processes Web based near real-time monitoring of EUMETCast terrestrial multicast at GÉANT OC

25 Web based near real-time Monitoring In-house developed portal Escalation procedures based on monitoring critical section Multicast beacon matrix

26 EUMETCast Terrestrial Footprint Accessibility mainly within Europe to all member states Prerequisite to accessibility is: eligibility to access the dissemination via NREN Currently 8 sites are registered with the multicast stream and receive data in near real-time in Europe and 3 sites overseas. Outreach to Asia - Korea (KMA) via cooperation between GEANT and our Asian partner networks Outreach to USA (NOAA) via cooperation - GEANT and Internet2 Outreach to Australia (NCI) via cooperation GEANT-TEIN and Asia-Pacific partner networks Feasibility of outreach to selected organisations located in Africa planned for 2016

27 R&E Services FOR RESEARCHERS AND PROJECTS

28 R&E Services for Researchers AAI, federated identity Single-Sign-On Access to e-infrastructures Access to cloud service providers Custom connectivity solutions: VPNs Light-paths Transfer performance optimisation Help with project Data Management Plans Contacts: Dr. Jakob Tendel DFN Maria.Minaricova GÉANT NREN Resources enlightenyourresearch.net geant.org/services services.geant.net/clouds fasterdata.es.net e-infrastructures eudat.eu prace-ri.eu egi.eu rd-alliance.org

29 Q?? Q

30 IP Multicast Multiple Unicast Multicast Source Branch Destination