How I learned to love XML. Peter Laulund National Survey and Cadastre

Transcription

1 How I learned to love XML Peter Laulund National Survey and Cadastre

2 Agenda KMS and INSPIRE About XML/GML Writing XML with FME Templates Schema mapping, semantic Schema mapping, geometry Workflow design

3 INSPIRE INSPIRE is a European initiative to create a common SDI Specific datasets has to be available in a harmonized way KMS is the National contact point KMS has five Annex 1 datasets I will talk about XML not INSPIRE

4 KMS datasets Transport Networks Hydrography Cadastral Parcels Geographical Names Administrative Units Documentation - Warning: getting through the PDF and the related XSD(s) is a tough read! Inspire-foss

5 FME and XML

6 XML XML: extensible Markup Language Syntax used to describe data GML: Geography Markup Language XML dialect for describing geography GML 2, GML 3.1.1, GMLSF, GML XSD: XML Schema Definition XML dialect for describing the contend of xml files

7 XML - example <?xml version="1.0" encoding="utf-8"?> <!-- oprettet af Pel, kms, 3. august > <venner xmlns:p=" xmlns:d=" <p:person id="345"> <p:navn>peter Laulund</p:navn> <p:adresse>sognegårds alle 54</p:adresse> <p:født> <d:dato> <d:dag>3</d:dag> <d:måned>maj</d:måned> <d:år>1957</d:år> <d:klokken/> </d:dato> </p:født> <p:telefon type="fastnet"> </p:telefon> <p:telefon type="mobil"> </p:telefon> <p:giftmed href="#445"/> <p:arbjedsgiver/> </p:person> </venner>

8 FME and XML FME reads and writes XML/GML Converts geometry to gml XMLSampleGenerator XMLTemplater XMLValidator XMLFormater

9 XMLTemplater An XML template is an XML document with XQuery functions <gn:text>{fme:get-attribute("name")}</gn:text> <au:geometry> {fme:get-xml-attribute("gml_geom")} </au:geometry> {fme:get-xml-list-attribute("level{}.xml")} <gml:featuremembers> {fme:process-features("feature")} </gml:featuremembers>

10 XMLTemplater The document may be loaded from an attribute a file entered into the transformer We use a file that is loaded into an attribute

11

12 Templates Use the XMLSampelGenerator to create the template Edit the template in a text editor Delete Add XQuery function calls Use XMLValidator to evaluate the result Use XMLFormater to make it look pretty

13 Writing INSPIRE GML

14 Challenges Five datasets some with more than one feature type Data for download from ftp don with FME WFS with Snowflake WMS with? All data in one Oracle database

15 Dataflow in KMS Oracle *.GML Read Write Schema Mapping - semantik Sql FME tools Tcl Schema Mapping - geometry Aggregate SetTraits OGCGeometry Transform to xml XMLTemplate XMLValidate XMLFormater

16 Schema mapping Schema mapping are basic FME functionality Add or remove attributes Change feature types Alter domain values All our data are in an Oracle database we will therefore use sql for schema mapping

17 Schema mapping FeatureType : oldattribute : newattribute REGION : REGIONKODE : nationalcode REGION : DAGI_ID : localid REGION : FEAT_ID : localidgeom REGION : TIMEOF_CRE : beginlifespanversion REGION : FEAT_TYPE : nationallevelname REGION : REGIONNAVN : name REGION : DQ_RESPONS : sourceofname FeatureType : newattribute : value REGION : namespace : dk.kms.au REGION : namespacegeom : dk.kms.au.geom REGION : country : DK REGION : gmltemplate : AdministrativeUnit REGION : nationallevel : 2ndOrder

18 Schema mapping Europavej Primærvej Sekundærvej Anden vigtig vej Større lokalvej Lokalvej Indkørselsvej Anden Read vej Hovedsti Cykelsti langs vej Sti, diverse Trafikvej-Gennemfart Trafikvej-Fordeling Lokalvej-Primær Lokalvej-Sekundær Lokalvej-Tertiær Ikke tildelt? mainroad firstclass secondclass thirdclass fourthclass fifthclass sixthclass seventhclass eighthclass ninthclass Write

19 inspireid All features must have an inspireid, it is a complex type made of A namespace - <country>.<organisation>.<dataset> Id the features database id Version null, sequence or timestamp <cp:inspireid> <base:identifier> <base:localid> {fme:get-attribute("localid")} </base:localid> <base:namespace> {fme:get-attribute("namespace")} </base:namespace> <base:versionid> {fme:get-attribute("beginlifespanversion")} </base:versionid> </base:identifier> </cp:inspireid> <base:identifier> <base:localid>595944</base:localid> <base:namespace>dk.kms.tn.roadnode</base:namespace> <base:versionid> t18:09:26</base:versionid> </base:identifier>

20 Example - geometry Coordinate System: ` EPSG:25832' Geometry Type: IFMEPoint Number of Geometry Traits: 1 GeometryTrait(string): `gml_id' has value `dk.kms.tn.roadnode ' Coordinate Dimension: 3 GML_3.2.1, gml_geom) <net:geometry> {fme:get-xml-attribute("gml_geom")} </net:geometry> <net:geometry> <gml:point gml:id="dk.kms.tn.roadnode srsname="epsg:25832" srsdimension="3"> <gml:pos> </gml:pos> </gml:point> </net:geometry>

21 gml:id gmi:id is mandatory Unique within the document Must start with a letter In FME default is an UUID Build the same way as inspireid <namespace>.<id>.<timestamp>

22 gml:id NAVNE - ID - FraDato - Navn X MONTAGE - ID - FraDato - Geometri <gn:geometry> <gml:multicurve gml:id="dk.kms.gn " srsname="epsg:25832" srsdimension="2"> <gml:curvemember> <gml:linestring gml:id="dk.kms.gn.geom t085805"> <gml:poslist> </gml:linestring> </gml:curvemember> <gml:curvemember> <gml:linestring gml:id="dk.kms.gn.geom t085805"> <gml:poslist> </gml:linestring> </gml:curvemember>... </gn:geometry>

23 FME script design

24 Design When we are designing an FME script we should reflect on Schema mapping FME or database Generic or specific Design of dataflow design patterns Pre- and post processing Existing system architecture

25 Dataflow design

26 Dataflow design

27 Dataflow design

28 Conclusion design Testing shows that design #3 is best Design #1 will not work with big datasets because of the list It use 10 to 15 percent less memory than #2 The timing is identical to #2 We can validate individual features

29 Conclusion After the first tests it been easy to work with templates Schema mapping in Oracle with sql A generic solution based on Design #3 Templates as individual files read into an attribute Only problem is FME can t handle big datasets ( features) yet

30 Peter Laulund Rentemestervej 8 DK-2400 Copenhagen NV Denmark Phone: pelau@kms.dk Questions?