EBAS (a name not an acronym)

Transcription

1 EBAS (a name not an acronym) EBAS is a database developed to host data from a range of «frameworks and programmes». Now considered as an electronic research infrastructure. NILU has more than 40 years experience in operating stateof-the-art observations at local regional global scale (+1000 stations) NILU is also a data user with expertice in modeling and assessment of results NILU also operate database solutions for satelite CALVAL actvities on contract by ESA, and related to urban air quality measurements on contract by EU, local municipalities etc.

2 Acid rain History technological development of EBAS OECD LRTAP; a new European network of sites was established and data collected at NILU, data mainly reported on paper -> LRTAP of acid rain proven and formed basis for EMEP EMEP; data reported on paper and later as flat files on floppy disk EBAS; a relational database, developed around 1995 (EMEP/AMAP) EUSAAR; funded the web interphase, later developments funded by ACTRIS EBAS-Online; funded an updated relational database model with a comprehensive metadata description EBAS- Online OECD CREATE/EUSAAR/ACTRIS EBAS GAW-WDC EMEP (HELCOM, OSPAR, AMAP)

3 History use of «EBAS» in supporting «frameworks» Frameworks/programs OECD; EMEP; 79-> HELCOM; 1989-> OSPAR; 1991-> AMAP; > CREATE/EUSAAR/ACTRIS 2002-> GAW-WDCA; > GAW-WDCRG; > Complexity (user expectations) Major inorganics in air and precipitation (~1972->) Surface ozone, NOx, VOCs (~1977->) Heavy metals in air and preipitation (~1973->) POPs in air and precipiation (~1992->) Aerosol physical and optical properties (~1999->) Tracers, and long-lived species (~2000->) Mainly surface based measurements made at rural/background sites (but not only)

4 Some statisics The term dataset is used to define one timeseries of one variable having identical core meta-data (i.e. an SO2 time series of daily samples over 40 years may be counted as one dataset). EBAS contains (fall 2016) nearly datasets. Data from 975 different locations - 69 different countries 48 frameworks/programmes 29 different matrix types and 101 different instrument types 631 different measurands (variables/components) Data back to 1971 The most complex/comprehensive database for atmospheric composition existing!

5 Who funds EBAS efforts?

6 QAQC info: results from intercomp etc. Institution/PI Authorised body Manual or automated Official repositories Data review: - CCC/NILU - External - user Site offering ~1-600 variables Formats Standards Units Flags Tools etc. EBAS Hardware Software Licences Security IPR, Branding, Versioning, Tracking use, tracking Impact, DOIs. Validated data NRT prelim. data Aggregates, plots, reports, maps, assessments User support Meta data exchange Stakeholders Secondary portals (WIGOS, GEOSS, IASOA, ACTRIS ++++ Complicating factors: many stakeholders, distributed (and limited) funding..

7 Data management is one part of a chain.

8 basic concepts EBAS is a Sybase relational database (data stored in tables, linked by meta-data) EBAS mainly represents a Primary repository (many reasons to avoid secondary (nonofficial) storage) The same data can be affilitated to several different programmes/framworks Data providers defines the affiliations as part of their meta data in the submission These definitions are used to manage the data (data policies, acknowledgements etc) Several data reporting levels is supported Level 0: raw data, all info at instrument level (for processing), native time resolution Level 1: data processed to final parameter Level 1,5: Aggregated to hourly values, etc, auto-prosessed (NRT) Level 2: fully processed and full quality assurance (regular reporting)

9 Data import format Data need to be formatted in the EBAS NASA-Ames format by the data provider (see reasoning behind this setup). Definitions are made in a process involving the providers and users Comprehensive documentation of meta-data (concept is to describe what aspects may make data not comparable, and help user understand what measurements represent) Data flags by data originator or by data centre Templates for data reporting

10 Tools for data reporting Templates for data reporting Open source scripts (Python) Data format checker User support - ticket system EBAS data quality feedback system Training courses

11 Templates - Reactive gases submit.nilu.no/submit- Data/Data- Reporting/Templates/Ca tegory/trace- Gases/NOx/level2

12

13

14

15

16

17 Ebas-submit.nilu.no (currently under revision)

18

19 EBAS Support:

20 EBAS team Rita Larsen Våler: data management, level I, Anne Hjellbrekke: development and data management (EMEP), Paul Eckhardt: development, Richard Rud: development, Ann Mari Fjæraa: data management, level II/III data, EVDC Markus Fiebig: GAW-WDCA, aerosols, interoperability, Cathrine Lund Myhre: ACTRIS, GHGs, Wenche Aas: QAQC, EMEP, HMs, POP, Sverre Solberg: GAW-WDCRG, photoox and precursors, Kjetil Tørseth: EMEP, GAW

21 Deadlines and data review/quality assurance Framwork/programmes define reporting deadlines EMEP, OSPAR, HELCOM, ACTRIS; July 31 the following year WMO-GAW; Dec 31 the following year A) As is (OSPARCOM) B) Reference lab and data audits NILU is Chemical Coordinating Centre for EMEP and is responsible for data audits (but always jointly with data originators) C) External review ACTRIS (NOx with NILU/DWD/DO, and VOC with NILU/EMPA/DO) GAW-WDCA The submission publication process takes time, i.e. there is delays from data are received untill they are available in EBAS Some frameworks may have data release procedures In Europe, often the same data

22

23 «EBAS User Feedback tracker»

24

25

26 How to find and access data in EBAS Search and download through Web interface ( Batch mode using scripts internally within NILU for supporting users operational exports to external users Validated data Near-real-time data for services Meta data export to specific portals: GAWSIS, ACTRIS, IASOA Meta-data server OAI-PMH

27 Demo - how to use EBAS-WEB:

28 Scripts inside firewall

29 Data for services, COPERNICUS

30 Interoperability, WIS-DCPC Open Archives Initiatives-Protocol Metadata Harwest (OAI-PMH) server established (WMO WIS Data Collection Production Centre compliant) query

31

32 User statistics

33 Flextra air mass trajectories operational Flexpart products on demand

34 Dealing with Data Policies Today: Zoo of individually defined data policies, user often needs to contact every data provider when using a self-defined data collection. Data policies ARE NOT legally binding violations can t really be sanctioned. Data use is very difficult to quantify (beyond download statistics). Possible Future: Easier use of readily available data a system similar to that for scientific articles: A type of widely used license is attached to the data (e.g. Creative Commons) License is easy to understand for user and legally binding abuse has legal consequences. Data are tagged with PIDs, i.e. DOIs Quoting / Use of data DOIs is counted similar to scientific articles.

35 Existing Puzzle Pieces of Future: Creative Commons Licenses Creative Commons Licenses General Features: Copyright stays with originator Allow non-commercial use. Users have to credit originators for their work. Users have to keep copyright notices intact. Additional license components: Attribution (CC BY): users may distribute / build upon work, but have to give credit for original work. Attribution-Share Alike (CC BY-SA): as CC BY, but requires the same license for the product. Attribution-NoDerivs (CC BY-ND): redistribution allowed, but work may not be changed / built upon. Attribution-NonCommercial (CC BY-NC): users may build upon work, but only noncommercially Attribution-NonCommercial-ShareAlike (CC BY-NC-SA): as CC BY-NC, but requires the same license for the product. Attribution-NonCommercial-NoDerivs (CC BY-NC-ND): only download and sharing allowed

36 Plans for ACTRIS Data Centre Nodes 1. All ACTRIS DC nodes need to identify DOI registrar for them to use, and clarify details around machine-based DOI assignment (API). 2. Figure out suitable granularity for issuing primary DOIs 3. Figure out how to include DOI assignment into the data curation workflow. Plans at EBAS: Use newly established Norwegian DOI registration node. Granularity: one DOI per 1 year of data per instrument at a station (1 DOI per submitted data file). Assign DOI on import of data file into database.

37 ACTRIS QAQC efforts related to NOX/VOC

38 Claude et al: VOCs and NOx in ACTRIS

39

40

41

42

43