Quality control service for Inspire data

Transcription

1 Quality control service for Inspire data INSPIRE Conference 2014 Aalborg Anders Foureaux Process Manager Data feeds Data hosting Environmental Informations Systems Unit Reasearch and Assessment Department Swedish EPA Naturvårdsverket Swedish Environmental Protection Agency 1

2 Naturvårdsverket Swedish Environmental Protection Agency 2

3 Background In Sweden, the environmental monitoring is done by many different contractors. Each contractor shall deliver data to a datahost, for publishing the data We have datahosts Naturvårdsverket Swedish Environmental Protection Agency 3

4 Problem It takes too long from the collection of data until anyone can use it Time consuming manual QA-process It is hard for the end user to find relevant quality information at the same place were data is stored or made available Naturvårdsverket Swedish Environmental Protection Agency 4

5 Solution Last year, the Swedish EPA developed a validation service for environmental data. It is a pilot project under development. Naturvårdsverket Swedish Environmental Protection Agency 5

6 The goal: 1. Speed up the flow of data from collection to presentation and usage of data. 2. Check the data quality against agreed quality standards and quality measures. 3. Create metadata "on-the fly" for performed quality control. 4. Gather all the requirements for data quality into a Data Product Specification. 5. Save time so that the contracted experts can use their time to more advanced analyzes of data instead of correcting incomplete data files. 6. To the extent possible, use the same OGC- and ISO standards for environmental data, as the INSPIRE community uses for Geodata. 6

7 One of 150 manuals for collecting data Naturvårdsverket Swedish Environmental Protection Agency 7

8 Data Produkt Specification ISO We have tried to use a range of ISO standards to describe the quality requirements we impose on the data we collect. We use the same standards used in the INSPIRE - directive We specify all requirements in one document for each data type. The document is called Data Product Specification in accordance with ISO We're not done yet...

9 Data produkt specification for Metals and Contaminants in sediment Naturvårdsverket Swedish Environmental Protection Agency 9

10 Data produkt specification for Contaminants in biota Naturvårdsverket Swedish Environmental Protection Agency 10

11 Data produkt specification for Air Quality Measurements Naturvårdsverket Swedish Environmental Protection Agency 11

12 Reporting Template Metals, persistent organic pollutants in sediment Naturvårdsverket Swedish Environmental Protection Agency 12

13 Reporting Template Air quality in cities Naturvårdsverket Swedish Environmental Protection Agency 13

14 Cloudbased Validation Service Dataspecification Datamodel Codelists Translation Validation Data Specificaton compliance Data host Metadata Datafile in xls is transformed into O&M (xml) Naturvårdsverket Swedish Environmental Protection Agency 14

15 Cloudbased Validation Service Dataspecification Datamodel Codelists Translation Validation Data Specificaton compliance Data host Metadata Datafile in xls is transformed into O&M (xml) Naturvårdsverket Swedish Environmental Protection Agency 15

16 Cloudbased Validation Service? Dataspecification Datamodel Codelists Translation Validation Data Specificaton compliance Data host Metadata Datafile in xls is transformed into O&M (xml) Naturvårdsverket Swedish Environmental Protection Agency 16

17 The validation service need other look-up services for example At JRC? Naturvårdsverket Swedish Environmental Protection Agency 17

18 The validation service need other look-up services for example At EEA? At JRC? Naturvårdsverket Swedish Environmental Protection Agency 18

19 The validation service need other look-up services for example At EEA? At JRC? Our Station registry WFS-T Naturvårdsverket Swedish Environmental Protection Agency 19

20 Two pilots Validation Service (FME) Registry of monitoring stations (WFS-T) Two ongoing pilot project together with Metria and Sweco (consultants) and the Data hosts and other experts Naturvårdsverket Swedish Environmental Protection Agency 20

21 Validate and deliver: Hazardous substances 21

22 Validate and deliver: Hazardous substances XLS (t ex) 22

23 Feedback by to the provider if it went well. The Data host gets the file with data, metadata according to ISO 19139, compliance report Compliance report Original file (xls) Delivery note Datafile transformed according to O&M (xml) 23

24 Feedback by to the supplier if it is NOT going well. No data gets to the Data host. 24

25 Feedback by to the supplier if it is NOT going well. Information about the errors that were found in the file. Wrong! The station is missing in Station Registry which right now is a pilot, developed by Sweco as a WFS service that responds to calls from the validation service. WFS = Web Feature Service (an OGC and ISO standard 25

26 The quality report in XML ISO which is the same as ISO "Inspire standard for metadata" 26

27 Naturvårdsverket Swedish Environmental Protection Agency 27

28 Why a Cloud-based Validation Service? Describe the quality of the data in a standardized and traceable way. Let the service automatically check the data instead of people. (This is one of the main reasons why the data does not come out on the web quick enough and at a time expected by users) To create a stronger link between exams / manuals and validation.

29 Advantages? Gives the Swedish EPA (or other financers) a tool to facilitate and ensure the quality of environmental monitoring data collection. To invest in a stand-alone service that can be used by many must be cheaper than if each data host builds its own service Going for a joint service will promote coordination around the terminology, concepts, code lists Expandable to any data type, or format, for example, emissions

30 Applied standards. (Some of them) Data Product Specification ISO structure structure according to INSPIRE requirements Data model based on ISO Observations and measurements, making data models compliant with INSPIRE data theme (EF), the station registry and official codelists Quality control done in accordance with ISO The results from the quality control is presented as a metadafile in XML according to ISO 19139

31 Conclusions: If data will be useful the quality of the data should be well described. One must be able to refer to a requirement, a quality measure. One must also be able to demonstrate how the quality has been secured The one that is financing the monitoring activities reasonably determines which quality data should have We save time and effort to use the same ISO- and OGC standards that is used within INSPIRE Naturvårdsverket Swedish Environmental Protection Agency 31

32 Thanks for listening! Naturvårdsverket Swedish Environmental Protection Agency 32