INSPIRE roadmap and architecture: lessons learned INSPIRE 2017

Transcription

1 INSPIRE roadmap and architecture: lessons learned INSPIRE 2017 Stijn Goedertier GIM Thierry Meessen GIM Jeff Konnen ACT Luxembourg Patrick Weber ACT Luxembourg 1

2 Administration du cadastre et de la topographie (ACT) Luxembourg Responsible for: National cadastral plan Differential GPS network Aerial photo s and maps Open data portal: data.public.lu Geoportal: geoportail.lu INSPIRE: catalog.inspire.geoportail.lu 2

3 #1 Make a plan ACT is responsible for the INSPIRE data harmonisation for the whole of Luxembourg 253 datasets from 22 government entities Made a roadmap Tasks, roles & responsibilities, estimated timing and effort 3

4 #2 Review the application architecture Components were replaced by their open-source counterparts (where feasible) Data harmonisation FME Desktop / FME Server 2017 Metadata & discovery service View service Download service Data store File store Validation GeoNetwork GeoServer GeoNetwork (ATOM/OpenSearch) Postgres/PostGIS AWS S3 (via udata) ETF-WebApp, GDI-DE Testsuite, INSPIRE validator 4

5 #3 Keep things as simple as possible 5

6 #3 Keep things as simple as possible e.g. Metadata: one language only: English 6

7 #3 Keep things as simple as possible e.g. Metadata: one language only: English e.g. Metadata: only mandatory metadata elements 7

8 #3 Keep things as simple as possible e.g. Metadata: one language only: English e.g. Metadata: only mandatory metadata elements e.g. Open Data: no access restriction means no authentication and authorisation 8

9 #3 Keep things as simple as possible e.g. Metadata: one language only: English e.g. Metadata: only mandatory metadata elements e.g. Open Data: no access restriction means no authentication and authorisation e.g. Service capability document: only minimal extension (scenario 1) 9

10 #4 Go for a file download service 10

11 #4 Go for a file download service Minimally a predefined dataset download service: i.e. a file download service ATOM/OpenSearch or WFS2.0 stored queries) The user must download the entire dataset as a file Requires no effort from data publisher (when ATOM/OpenSearch generated from metadata) 11

12 #4 Go for a file download service Minimally a predefined dataset download service: i.e. a file download service ATOM/OpenSearch or WFS2.0 stored queries) The user must download the entire dataset as a file Requires no effort from data publisher (when ATOM/OpenSearch generated from metadata) Where practicable a direct access download service (i.e. a WFS2.0 query service) The user can query and get the feature(s) of interest Implementations like GeoServer App-Schema have many limitations QoS (availability, response, throughput) is more difficult to guarantee 12

13 #4 Go for a file download service Minimally a predefined dataset download service: i.e. a file download service ATOM/OpenSearch or WFS2.0 stored queries) The user must download the entire dataset as a file Requires no effort from data publisher (when ATOM/OpenSearch generated from metadata) Where practicable a direct access download service (i.e. a WFS2.0 query service) The user can query and get the feature(s) of interest Implementations like GeoServer App-Schema have many limitations QoS (availability, response, throughput) is more difficult to guarantee 13

14 #4 Go for a file download service predefined dataset download service: Implementation: GeoNetwork was enhanced to support local atom feeds generated automatically on the basis of the service and dataset metadata records. Conformance testing: Implementation was tested with the GDI-DE test suite. Benefits: No additional effort required from data publisher (just correct metadata records) QoS easy to guarantee 14

15 #4 Go for a file download service direct access download service: Implementation: GeoServer Application Schema. Conformity: good results with ETF-Webapp Drawbacks: Requires transformation of source database Not possible to distinguish multiple datasets (only feature types) (multiple virtual WMS not possible with App-Schema) QoS more difficult to guarantee 15

16 #5 Speed up deployment with Docker Dockerfiles / docker images for: FME Server PostgreSQL, GeoServer, GeoNetwork, ETF-Webapp NGINX INSPIRE dashboard. Building images and starting-up containers in concert using Docker-compose or Rancher 16

17 #6 Automate conformance testing Metadata: INSPIRE Geoportal Validator Data: ETF-Webapp (ARE3NA ETS repository) Network Services: GDI-DE TestSuite 17

18 #7 Automate the workflow 1. Harmonise dataset udata API udata Data Portal 2. Upload harmonised dataset 3. Configure view service 4. Update metadata records 5. Test conformance of data, metadata, and services FME ReST API CSW-T API ReST API GeoServer View (WMS) and download (WFS) services GeoNetwork Data Catalog INSPIRE Validation service 18

19 #7 Automate the workflow - 1. Harmonise data with FME XML Schema: write GML according to 34 built-in schemas or your own supports INSPIRE complexities: multiple geometries, nested features, object referencing, repeating elements, nillable elements, grids 19

20 #7 Automate the workflow - 1. Harmonise data with FME XML Schema: write GML according to 34 built-in schemas or your own supports INSPIRE complexities: multiple geometries, nested features, object referencing, repeating elements, nillable elements, grids How to map features into the right schema? 20

21 #7 Automate the workflow - 1. Harmonise data with FME What is the SchemaMapper? restructures the existing schema (data model) of incoming features to a new schema, based on mappings defined in an external lookup table. very useful when the mappings are potentially complex, when they need to be maintained by someone unfamiliar with FME. 21

22 #7 Automate the workflow - 1. Harmonise data with FME source_feature_type target_feature_type: mappings of features types. source_attr_name target_attribute_name: mappings of attributes. Attribute filters: new attribute-value combinations (filter_target_attribute_name filter_target_attribute_value), conditional to the presence of attribute-value combinations in the source feature (filter_source_attribute_name filter_source_attribute_value). source_feature_type target_feature_type source_attr_name target_attr_name UNESCO_vieux_quartiers_fortifications_LUX_2zones ProtectedSite UNESCO_vieux_quartiers_fortifications_LUX_2zones UNESCO_vieux_quartiers_fortifications_LUX_2zones UNESCO_vieux_quartiers_fortifications_LUX_2zones UNESCO_vieux_quartiers_fortifications_LUX_2zones UNESCO_vieux_quartiers_fortifications_LUX_2zones Id inspireid.identifier.localid UNESCO_vieux_quartiers_fortifications_LUX_2zones UNESCO_vieux_quartiers_fortifications_LUX_2zones Lien_Link1 UNESCO_vieux_quartiers_fortifications_LUX_2zones Lien_Link2 gml_descriptionreference.xlink_href UNESCO_vieux_quartiers_fortifications_LUX_2zones Nom_Name sitename{0}.geographicalname.spelling{0}.spellingofname.text UNESCO_vieux_quartiers_fortifications_LUX_2zones UNESCO_vieux_quartiers_fortifications_LUX_2zones 22 UNESCO_vieux_quartiers_fortifications_LUX_2zones Type_Typ

23 #7 Automate the workflow - 1. Harmonise data with FME A simple workspace looks as follows (sometimes more operations required) 23

24 #7 Automate the workflow - 2. Upload dataset with FME Upload datasets to a data portal using an Application Programming Interface (API) s/create_resource 24

25 #7 Automate the workflow - 2. Upload dataset with FME import requests url = ' headers = {'X-API-KEY': 'eyjhb'} data = { 'title': 'Dataset title', 'description': 'Dataset abstract', 'tags': 'keywords', 'private': 'true', 'spatial': {'geom': 'null', 'granularity': 'other', 'zones': []} } response = requests.put(url,headers=headers,json=data) print(response.text) 25

26 #7 Automate the workflow - 3. Configure view service with FME Use the GeoServer ReST API to create workspaces, stores, layers, styles, and WMS services. 26

27 #7 Automate the workflow - 3. Configure view service with FME HTTPCaller: general-purpose ReST client 27

28 #7 Automate the workflow - 3. Configure view service with FME HTTPCaller: general-purpose ReST client 28

29 #7 Automate the workflow - 4. Update metadata records Data and service metadata records are managed with GeoNetwork. Standards: ISO19139, CSW, INSPIRE TG metadata & discovery service 29

30 #7 Automate the workflow - 4. Update metadata records Use Xquery update statements to update the metadata record Use CSW-T API to post the updated metadata record to GeoNetwork Also for service metadata (e.g. <srv:coupledresource> <srv:operateson>) 30

31 #7 Automate the workflow - 5. Test conformance ETF-webapp validator: FME HTTPCaller used to execute tests 31

32 Conclusion: lessons learned 32

33 Conclusion: lessons learned #1 Make a plan 33

34 Conclusion: lessons learned #1 Make a plan #2 Review the application architecture 34

35 Conclusion: lessons learned #1 Make a plan #2 Review the application architecture #3 Keep things as simple as possible 35

36 Conclusion: lessons learned #1 Make a plan #2 Review the application architecture #3 Keep things as simple as possible #4 Go for a file download service 36

37 Conclusion: lessons learned #1 Make a plan #2 Review the application architecture #3 Keep things as simple as possible #4 Go for a file download service #5 Speed up deployment with Docker 37

38 Conclusion: lessons learned #1 Make a plan #2 Review the application architecture #3 Keep things as simple as possible #4 Go for a file download service #5 Speed up deployment with Docker #6 Automate conformance testing 38

39 Conclusion: lessons learned #1 Make a plan #2 Review the application architecture #3 Keep things as simple as possible #4 Go for a file download service #5 Speed up deployment with Docker #6 Automate conformance testing #7 Automate the publication workflow 39