EIDA European Integrated Data Archives

Transcription

1 EIDA European Integrated Data Archives Marcelo B. de Bianchi Joachim Saul IT Coordination Meeting KNMI / Utrecht

2 Data acquisition Seismometer / Accelerometer Orthogonal (continuous) ground displacement data A seismometer is an instrument that measures the ground movement at a fixed interval (normally times per second). The ground movement is decomposed into three perpendiculars components, each of these are named one channel or stream of data.

3 Channel Names attribution GE APE GE APE GE APE Network Station Location BHZ Channel BHN BHE The streams of data are given codes. The codes are grouped by location. Location by stations and stations by networks. The FDSN is responsible to attribute non overlapping network codes to institutions/persons running a seismic station.

4 Sharing the metadata & data Sharing this data can be a real challenge when... every country/institutions has its own stations and interests in the end people wants to receive acknowledgment by their work and money invested

5 Multiple countries & data producers makes it hard to find all the data available BE FR DK WM BS IC FN GR EE NZ TM ES C CH EI CN II CZ GE CX CU KC GT SX SK SJ MS TW JP PS TE G AU PA IG HT NL NU AF GB NS MY HU NO PL IQ IA OE BW NA PM IS CR MN MK TT CK IU AD KO HL RD RO EB HE US?

6 European Integrated Data Archives (EIDA) Inside EIDA each node has a list of all that exists in EIDA, and can like lenses redirect the user to the correct node. EE II DD BE FR DK WM BS IC FN GR EE NZ TM ES C CH EI CN II CZ GE CX CU KC GT SX SK SJ MS TW JP PS TE G AU PA IG HT NL NU AF GB NS MY HU NO PL IQ IA OE BW NA PM IS CR MN EIDA also tries keep each data center / producers interests MK TT CK IU AD KO HL RD RO EB HE US AA

7 The EIDA driven engine: ArcLink 7 nodes are currently part of EIDA: GFZ (Germany), ETH (Switzerland), KNMI/ODC (Holland), BGR (Germany), LMU (Germany), IPGP (France), INGV (Italy) Each node has an ArcLink server, that in the end fulfills the requests Each node synchronize its network, station, location, channel metadata everyday On top of the ArcLink each node has build its on infrastructure to access the data, taking advantage of the infra-structure established

8 ArcLink Is a technology that is used to query a server for seismological data in a certain time window and region. The ArcLink can refer to the protocol, or just to the server (computer program) that is included in SeisComp3 that implements the protocol. It is the technology behind EIDA today T12:00:11 to T00:00:00 GE APE BHZ ArcLink or Inventory

9 How it works User Time Windows per net/station and region Client Requests ASCII FLOW Server Waveforms, QC & Inventory The clients defines how the user interact with the ArcLink server, some client examples are: arclinktool, arclink_fetch, webdc.eu portal, Breqfast (in seiscomp3), ObsPy, scolv (picker) and so on... Products: XML Inventory Routing, QC FullSeed Dataless Seed MiniSeed ArcLink uses a tracking Id (Similar to DHL, Fedex) for each request, allowing the client to break the connection while waiting for the data.

10 Basic Arclink: Wildcard & Routing Server A Server B Server C Inventory Request GE APE 00 BHZ GE APE 00 BHN IU STA 10 HHZ 2011,1,1 2011,1,2..GE APE BH? * 2011,1,1 2011,1,2..IU STA HH? *

11 Basic Arclink: Wildcard & Routing Server A Server B Server C Inventory Request Routing Request GE APE 00 BHZ Server B GE APE 00 BHN Server B IU STA 10 HHZ Server C 2011,1,1 2011,1,2..GE APE BH? * 2011,1,1 2011,1,2..IU STA HH? *

12 Basic Arclink: Wildcard & Routing Server A Server B Server C Inventory Request Routing Request Data Request GE APE 00 BHZ Server B GE APE 00 BHN Server B IU STA 10 HHZ Server C 2011,1,1 2011,1,2..GE APE BH? * 2011,1,1 2011,1,2..IU STA HH? *

13 ArcLink Protocol hello ArcLink v1.2 ( ) Bianchi request waveform ERROR showerr authentication required user OK request routing OK 2011,1,1,00,00, ,1,2,00,00,00 GE APE BH? * END 1597 status 1597 <?xml version="1.0"?> <arclink> <request id="1597" type="routing" label="" args="" encrypted="false" size="545"... <volume id="routing" dcid="bia" status="ok" size="545" encrypted="false" message=""> <line content="2011,1,1,00,00, ,1,2,00,00,00 GE APE BH? *" status="ok"... </volume> </request> </arclink> END purge 1597 OK bye The communication with the ArcLink server can be tested using telnet and the set of commands specified on the protocol. hello user request status institution label showerr purge download bdownload end bye Command Set

14 ArcLink Request Types REQUEST <TYPE> <Attributes> <start time> <end time> <Network> <Station> <Channel> <Location> <constraints>... END Request Type ArcLink WAVEFORM seed / c format = FSEED * / MSEED compression = none * / bzip2 Constraints : Yes WildCards on: Location/Channel Optional field: Location. If not added the default is to use only empty location codes

15 ArcLink Request Types REQUEST <TYPE> <Attributes> <start time> <end time> <Network> <Station> <Channel> <Location> <constraints>... END ArcLink Request Type WAVEFORM format = FSEED * / MSEED compression = none * / bzip2 seed INVENTORY / c compression = none / bzip2 Constraints : Yes instruments = true / false WildCards xml on: / clocation/channel modified_after = <ISO DATE> Optional field: Location. If not added the default is to use only empty WildCards on: all fields location codes Optional fields: Station, Location, Channel those not added to the request are not returned expect Location that defaults for <Empty>

16 ArcLink Request Types REQUEST <TYPE> <Attributes> <start time> <end time> <Network> <Station> <Channel> <Location> <constraints>... END ArcLink Request Type WAVEFORM format = FSEED * / MSEED compression = none * / bzip2 seed INVENTORY / c compression = none / bzip2 Constraints : Yes instruments = true / false WildCards xml RESPONSE on: / clocation/channel compression = none * / bzip2 modified_after = <ISO DATE> Optional field: Location. If not added the default is to use only empty WildCards seed on: / nc all fields location codes Optional fields: Station, Location, Channel those not added to the WildCards on:station / Location / Channel request are not returned expect Location that defaults for <Empty> Optional field: Location. If not added the default is to use only empty location codes

17 ArcLink Request Types REQUEST <TYPE> <Attributes> <start time> <end time> <Network> <Station> <Channel> <Location> <constraints>... END ArcLink Request Type WAVEFORM format = FSEED * / MSEED compression = none * / bzip2 seed INVENTORY / c compression = none / bzip2 Constraints : Yes instruments = true / false WildCards xml RESPONSE on: / clocation/channel compression = none * / bzip2 modified_after = <ISO DATE> Optional field: Location. If not added the default is to use only empty WildCards seed ROUTING on: / nc all fields compression = none / bzip2 location codes Optional fields: Station, Location, modified_after Channel = <ISO those DATE> not added to the WildCards xml on:station / nc / Location / Channel request are not returned expect Location that defaults for <Empty> Optional field: Location. If not added the default is to use only empty WildCards on: all fields location codes Optional fields: Station, Location, Channel. Channel and Location are ignored.

18 ArcLink Request Types REQUEST <TYPE> <Attributes> <start time> <end time> <Network> <Station> <Channel> <Location> <constraints>... END ArcLink Request Type WAVEFORM format = FSEED * / MSEED compression = none * / bzip2 seed INVENTORY / c compression = none / bzip2 Constraints : Yes instruments = true / false WildCards xml RESPONSE on: / clocation/channel compression = none * / bzip2 modified_after = <ISO DATE> Optional field: Location. If not added the default is to use only empty WildCards seed ROUTING on: / nc all fields compression = none / bzip2 location codes Optional fields: Station, Location, modified_after Channel = <ISO those DATE> not added to the WildCards xml on:station / QC nc / Location compression / Channel = none / bzip2 request are not returned expect Location that defaults for <Empty> Optional field: Location. If outages not added = true the / default false is to use only empty WildCards on: all fields location codes xml / nc logs = true / false Optional fields: Station, parameters Location, = Channel. Coma separated Channel and Qc parameters Location are ignored. WildCards on: all fields Optional fields:none

19 ArcLink Request Types REQUEST <TYPE> <Attributes> <start time> <end time> <Network> <Station> <Channel> <Location> <constraints>... END ArcLink Request Line constraints Request Type WAVEFORM format = FSEED * / MSEED compression = none * / bzip2 seed INVENTORY / c compression = none / bzip2 Constraints : Yes instruments = true / false WildCards xml RESPONSE on: / clocation/channel compression = none * / bzip2 modified_after = <ISO DATE> Optional field: Location. If not added the default is to use only empty WildCards seed ROUTING on: / nc all fields compression = none / bzip2 location codes Optional fields: Station, Location, modified_after Channel = <ISO those DATE> not added to the WildCards xml on:station / QC nc / Location compression / Channel = none / bzip2 request are not returned expect Location that defaults for <Empty> Optional field: Location. If outages not added = true the / default false is to use only empty WildCards on: all fields location codes xml Line / nc Constraints logs = true / false Optional fields: Station, parameters Location, = Channel. Coma separated Channel and Qc parameters Location are ignored. WildCards sensortype on: all fields = VBB, BB, SM, OBS or combined with '+' Optional latmin fields:none = <Value> latmax = <Value> lonmin = <Value> lonmax = <Value> permanent = true / false restricted = true/ false Wildcards on: All No Optional items

20 EIDA/ArcLink use cases Metadata Request Obtain instruments response from more than one data center at the same request Waveform Request Obtain miniseed / FullSeed data from one / multiple data centers Find what channels are available on a certain region, epoch by type and permissions Qc Request Obtain pre-computed quality control parameters for certain streams of data. Those QC parameters could be used together with inventory to return better datasets automaticaly. Routing Request Maps the channel name to ip address of arclink server capable of delivering data in the specified time period Obtain Ip address of seedlink server capable of delivering real-time data

21 Web Portals Web Portals try to offers more sophisticate ways to select data based normally on earthquake catalog and predicted arrival times at certain stations. While each webportal is not really part of the EIDA infrastructure they all use the EIDA infrastructure behind to collect the data from multiple data centers and return the data to the user. They are portals to EIDA.

22 Near Future Improvements ArcLink Encryption: Will secure the restricted dataset downloads Each user would need (1) to register his with the data center responsible for the restricted data and (2) a password key to decrypt the received volumes ArcLink Proxy: Will allow access to nodes with firewall problems using the GFZ backbone. This would also make possible to write simple clients (Without Routing/Inventory Expanssion) to fetch data from many nodes at the same time ArcLink Authoritative Tables: Will allow a more precise synchronization procedure to be run where an authoritative table would guide the full synchronization process. A master file would guide the arclink server on deciding which server is authoritative for which network. EIDA heartbeat : ArcLink monitoring system based on NAGIOS A NAGIOS setup to continuously monitor EIDA and ensure the correct operation of EIDA

23 Future developments

24 Thank you

25 Archive Synchronization On ArcLink: Inclusion of MD5 computation based only on the amplitude part from miniseed records inside Arclink protocol. Ant gap filler: would enable efficient development of an Ant Gap researcher and filling robot that could slowly work to keep data archives complete and in sync. Archive A T12:00:11 to T00:00:00 GE APE BHZ c35cc44e980cea0f97f7302bfc20e37b ArcLink Server B