Seismology and EPOS Integrated Core Services

Transcription

1 Seismology and EPOS Integrated Core Services Daniele Bailo (INGV) and the EPOS IT team ORFEUS meeting September 22, Bucarest

2 What is EPOS? EPOS is a long-term plan for the integration of research infrastructures for solid Earth Science in Europe EPOS integrates the existing (and future) advanced European facilities into a single, distributed, sustainable infrastructure taking full advantage of new e-science opportunities

3 EPOS Architecture Governance Financial Work Technical Implementation Communication Harmonization

4 EPOS IP project Timeline Implementation Validation Pre-operation

5 Communities and services

6 Communities and services

7 Communities in numbers MAP OF: - Seismic/GPS stations - Laboratories -- etc. Research Infrastructure LIst 244 Research Infrastructures 138 Institutions 22 countries 2272 GPS receivers 4939 seismic stations 464 TB Seismic data PB Storage capacity (seismology) PB Storage capacity (GNSS DC) 828 instruments in 118 Laboratories Diversity in data type and formats

8 The challenge of integration

9 Storage / curation challenge Yesterday Today

10 Data exchange challenge Yesterday Today

11 Format / standard challenge ASCII files Binary files Local metadata quakeml No metadata SEED Telegrafic Format SAC Yesterday IASPEI Seismic Format (ISF) Today stationml SEED Dost, B., Zednik, J.,et al. (2012): Seismic Data Formats, Archival and Exchange. - In: Bormann, P. (Ed.), New Manual of Seismological Observatory Practice 2 (NMSOP-2), Potsdam : Deutsches GeoForschungsZentrum GFZ, p

12 Future challenges: do you know Who (name and surname) and how is using your data? Can you measure the impact on society? Which is the value of your data? Can I use it for commercial purposes? Can I make computation without downloading waveforms? Will the data / software survive you?

13 Future challenges: do you know How to increase the usage of your data? How to increase the citation of your data/dc/publications? How to improve system performance? How to bridge with other disciplines (e.g. seismic data from volcanologist)? the answer is.

14 INTEGRATION

15 Advantages of integration USERS: access to an environment where heterogeneous data and tools available: Multidisciplinary science made simple [1] More papers, more citations Data and tool availability DATA PROVIDERS: Great diffusion of data (EU funds attraction?) Track usage of data and services steering capacity EU open data policy do not reinvent the wheel [1]: [Chiaraluce, L. (2012). Unravelling the complexity of Apenninic extensional fault systems : A review of the 2009 L Aquila earthquake Journal of Structural Geology. ]

16 HOW TO achieve (technical) INTEGRATION for solid Earth data/services? A) By means of discussions, conferences, meetings B) Promoting interoperability of systems C) By setting up a new international standardization body D) Promoting participation to European projects

17 Interoperability Promoting interoperability of systems What does it mean? 1.Connect several systems (bricks) 2.By means of interfaces (APIs or web services) 3.No need to know system interal specifics (black box concept)

18 Interoperability: example THIN LAYER (or mediator, or EPOS integrated services or ) FDSN station FDSN dataselect FDSN events GSAC services Seismic Waveforms Seism Events GPS wavef. Interoperability implies usage of standards: For metadata For data formats For communication protocols

19 Future Platform

20 Scientist!

21 Switch to advanced search ENVISAT_ _ Sicily North East Coast Accelerometric Data Lipari Islands M: add show info Sicily North East Coast M: add show info Sicily North East Coast M: add show info SAR Data ENVISAT_ _ add show info ENVISAT_ _6 add show info square square Geodetic DAta Rynex ad add show info Rynex ad add show info Rynex ad add show info

22 Sicily North East Coast Latest Earthquakes Lipari Islands M: Info Sicily North East Coast M: Info Sicily North East Coast M: Info Shakemaps Sicily North East Coast Download Sicily sea Download Hystorical Earthquakes Siracusa Aea - Imax: 11MCS Infos Documents & other resources Civil protection: Sicily hazards Link Hystory of Sicily Hearthquakes Link

23 Bulk Actions Accelerometric Data: SAR Data: Accelerometric Waveforms Filter Hi-Pass Low-Pass Band-Pass Accelerometric Data MI : AQV : AQG : AQA : AQK : PCB : AQV : AQM : AQU : PCB : GSA : PCB : Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show Remove Show SAR Data ENVISAT ASA_IM 0P ENVISAT ASA_IM 0P ENVISAT ASA_IM 0P

24 Scientist at work Then what can the scientist do? Integrate use of SAR, GPS, Accelerometric Data, etc. Use different codes and languages (python, fortran, any other ) Perform heavy processing online (use of HPC resources) Compare results (e.g. focal mechanism catalogues) Compare different data Save data in personal area and more (even download the data)

25 Generic Features 1. Download 2. Visualization 3. Processing 4. Modeling

26 EPOS demonstrator SEISMIC DATA DOWNLOAD GPS DATA DOWNLOAD Collaboration with VERCE

27 IT Architecture & Design

28 Architecture Integrated Core Services (ICS) Interoperability layer Thematic Core Services (TCS)

29

30 TCS generic architecture EPOS ICS API / web service TCS system Data/metada ta catalogue National network National repository Local HPC

31 Principles Co-development (ICS and TCS) The development of the ICS depends on the requirements and provided data, services, resources and users of the TCS. The TCS are at different stages of maturity. For some with little infrastructure to date - the adoption of the EPOS architecture is straightforward. For others with several years of infrastructural investments behind - a jointly agreed evolutionary plan to converge to interoperability with ICS will be adopted. The EPOS approach in this context is neither top-down nor bottom up: the main idea is to use the general architecture and follow a cooperative approach in the designing and development of the software to build the compatibility layer, which is the place where harmonisations and communications are achieved.

32 Principles Do not reinvent the wheel a. Reuse local technologies The co-development philosophy maximises re-use of existing software services, data availability and resources. (e.g. VERCE, EUDAT) b. Do not build a supercomputer: build ICS-D EPOS will have its own computing resources mainly to provide the uniform view over EPOS entities. For the massive processing EPOS will provide subject to authorisation access to appropriate computing facilities including HPC (High Performance Computing) machines for modelling and simulation, and HTC (High Throughput Computing) clusters for data intensive applications such as data mining. These facilities are known collectively as ICS-D (Integrated Cores Services Distributed) c. EU Initiatives Harmonization: EGI, ENVRI, EUDAT, RDA

33 Principles Microservices approach The Microservices architecture approach envisages small atomic services dedicated to the execution of a specific class of tasks, which have high reliability. EPOS will either take existing software services and wrap them for EPOS use or build new services complying with EPOS architectural standards. The aim is to have atomic services with defined interfaces which can be composed together to form a software stack able to address unpredictable user request.

34 Principles Clear long-term technical goals, but iterative short-term approach The overall architecture of EPOS is clearly defined and agreed through EPOS-PP. However, its implementation in EPOS-IP would require a step-by-step approach to build a reliable system environment to meet the requirements of end-users and their communities

35 Principles lessons learnt Working in phase (ICS and TCS) Communicating across different domain (IT, scientists, lawyers..etc) Collaborating and building communities

36 Thank You WebSite Newsletter R.I.D.E. Epos Social

37 IT Architecture & Design

38 Architecture Integrated Core Services (ICS) Interoperability layer Thematic Core Services (TCS)

39 ICS - Integrated Core Services ICS-C and ICS-D ICS-C ICS-D ICS-Central Hub Orchestrates external resources (ICS-D) EPOS member states competition for hosting it ICS-Distributed services Provide IT resources not built by EPOS ICS-C Computational resources Visualization resources Others

40 ICS-C EPOS GUI EPOS APIs METADATA catalogue System manager AAI system Workflow Engine ICS-D TCS Data set TCS API Servic es Softw are Data set TCS API Servic es Softw are Other TCSs National Layer IT Seismic net FR Seismic net PT GPS net NO GPS net

41 Elements of EPOS ecosystem Researchers, research managers, innovators, media User Model User interaction with the system: AAAI, interaction workflow, multlinguality Processing Model Describes functions of processing environment etc. Describes research data and ancillary info (instituions) Data Model Representing ICT resources Resource Model Complete ICT environment for research

42 Web portal, Spatio-Temporal Search Discovery (DC) and (CKAN, egms) Contextual (CERIF metadata model) Detailed (community specific) domain specific data (TCS or Naational Ris) Metadata model - 3-layers - Manage the complexity (see previous slide) - CERIF (formal syntax, declared semantics) Functions -Discovery, selection -Impact assessment -Manage interoperability Issues (selection of) - heterogeneity - Many standards to be mapped from TCS - Ontologies - Complex work

43 Contextual (CERIF metadata model) ( Common European Research Information Format

44 EPOS Interface Human interface (GUI) Website or portal Machine interface - API or Web service - which execute scripts or queries - Returns data/metadata in a given standard

45 EPOS Interface: interoperability Metadata avaialble in different formats: RDF export ( old ENVRI) OAI-PMH, CKAN, opensearch (EUDAT) other standards CERIF XML APIs RDF OAI-PMH EPOS APIs

46 Located in Italy Ontologies Trieste Is Born Michelini Is Director of Has hobby Sailing use Boat use sea Machine readable and Machine understandable How does it work? 1. Connects concepts 2. Needs vocabulary CNT Is section of INGV Gresta Is president of Issues Many ontologies exist Vocabulary Mapping CERIF supports it

47 AAI system (federeated & distributed) Purpose - Authenticate users - Authorize users - Transparent access to TCS and ICS-D data & services - X509 - EPOS IdP - TCS need guidelines - Accept all auth. mechanisms Issues - Delegation - Many system, sometimes non interoperable - EPOS-EGI-CC

48 Comments - Interoperability among various workflow engines - VERCE contribution - E-FLOW by EGI - CERIF?? Workflow engines Purpose - Tracks data transformation - Allows versioning - Allows reproducibility

49 How to register/cite data or publications? PID system Purpose - DO / publication can be uniquely referenced - Assign a PID at data creation times Issues - Need for a simple mechanism to implement it - EUDAT can help - Not at ICS level (only for

50 TCS envisaged architecture EPOS ICS API / web service TCS system Data/metada ta catalogue National network National repository Local HPC

51 ICS-TCS common work Metadata Definition of community, standardbased metadata Match & Map: Match/Mapping of community metadata within the Metadata Catalogue APIs Definition of APIs for data discovery and for data access/retrieval