Schema transformations in mass data harmonization process. Yohann Ly 8/9 Octobre 2013 Marne la Vallée

Transcription

1 Schema transformations in mass data harmonization process Yohann Ly 8/9 Octobre 2013 Marne la Vallée

2 Mass data harmonization process Problematic : Multiple data providers Multiple storage format Multiple data schema Amount of data INSPIRE format GMLs Data Provider 1 Oracle Central organization Data Provider 2 Data Provider Post GIS Enterprise GIS Centralized database Access SQL Server GeoMedia INSPIRE schema

3 What is Enterprise GIS? Enterprise GIS (EGIS) Desktop application built on top of GeoMedia as well as an EGIS Service that allows the enterprise capabilities to be accessed from Geospatial Portal. Deployed in Poland, Netherlands, Nigeria, Spain, Austria and Sweden. It has a SDK, English documentation for users, administrators and developers Intergraph Corporation 3

4 Sophisticated access control SSO based on the Windows domain Open architecture to add custom authentication method (e.g. with LDAP). Traditional login-password authentication method available. Integration with third-party identity providers implementing Web SSO with SAML 2.0 protocol available. This is a Windows domain SSO example. EGIS can utilize Web SSO with SAML 2.0.

5 Sophisticated access control User interface to manage Roles and groups and users Users can be placed inside some organization structure. 5

6 Sophisticated access control Access rights for Feature classes / Attributes / Spatial Extent 6

7 Data AK7 Server Hides the database-level complexity to the end user. Enterprise GIS will give end user access to data he s allowed to have, and easy-to-use interface to work with it. Different data sources are treated as they would be only one 7

8 Diapositive 7 AK7? niezwiązane w warsztatami. To jest techniczna ciekawostka, raczej na koniec, jeżeli wogóle Alina Kmiecik; 01/10/2013

9 Categories tree End user doesn t work on raw database tables (e.g. SKJZ_L), he works on friendly names (e.g. Roads) placed in a tree structure.

10 Feature properties Names of classes and attributes can be changed in easy way

11 Relations You can define relations between objects. Three types of relations: 1. Attribute. 2. Spatial. 3. Dependency table. 10

12 Relations You can define relations between objects. 11

13 Analysis It is possible to prepare various dynamic analysis (even analysis on analysis), register them in the system to work on them: Attribute query Attribute selection Join Union Buffer Zone Functional Attributes Analytical Merge Aggregation Spatial query Spatial Intersection Spatial difference

14 Analysis It is possible to prepare various dynamic analysis (even analysis on analysis), register them in the system to work on them: Data Provider 1 Oracle Central organization Data Provider 2 Data Provider Post PostGIS GIS Enterprise GIS Centralized database Access SQL Server Dynamic queries definitions INSPIRE schema

15 Data exchange Map different data models and exchange data. Translate dictionary entries, force values or convert geometry on-the-fly 14

16 Data exchange Different update policy: none, update and append, update only, append only or synchronize.

17 Class mapping For each class mapping there is attributes mapping defined.

18 Attribute conversion

19 Data exchange Here s an example of what can we do along with transferring data. Source: MS Access Table name: LAKE Destination: Oracle Table name: GREAT_LAKES Column Type Column Type ID Autonumber ID INTEGER NAME Text Untrimmed values NAME VARCHAR DESC Text DESCRIPTION VARCHAR DETAILS GEOMETRY BLOB Area GEOM SDO_GEOMETRY Compound IMPORTED_BY VARCHAR Problem Table names don't match. Copying ID values may lead to unique constraint violation. Source attribute NAME may have leading and trailing white chars. Different column names with feature description. Lack of DETAILS column in the destination database. Different geometry type. Lack of IMPORTED_BY column in the source database. Solution Translate. Renumber. Trim. Translate. Skip. Convert. Force value. 18

20 Data exchange Data exchange can: Test schema Test import Import data

21 Import test

22 Import test Error Warning Ok

23 Import test

24 Data exchange report

25 Schema transformations First schema transformation is made on the Data Exchange level Source databases schemas Target database (INSPIRE application schema) Second schema transformation is made on GML files INSPIRE application schema GML Exporter (XLST transformation) INSPIRE complaint GML files

26 GML Exporter Feature classes to export Spatial filter Theme (INSPIRE) Output folder Output coordinate system

27 Why such a tool is needed? mass-data cadastral parcels 32 mln records many diffuse sources transportation network: source datasets, many different models transportation network: - 15 source application schemas

28 Thank you