An Introduction to Data Interoperability

Transcription

1 Esri International User Conference San Diego, California 2012 Technical Workshops July 24/25 An Introduction to Data Interoperability Bruce Harold - Esri Dale Lutz Safe Software

2 Background Why Data Interoperability?

3 What is the Data Interoperability Extension? ESRI & SAFE Co-Development based on FME Format Support Adds support for 130+ data formats Custom Formats Direct use in ArcGIS Desktop Quick Data Translation Data Interoperability Tools Toolbox Convert between data formats Spatial ETL (Extract, Transform, Load) Schema Mapping (may be Dynamic) Transformation of Geometry Logic Controls Data Flow Per Feature

4 ArcGIS 10.0 Data Interop Common Tasks Data Integration - Bring multiple datasets together Data Migration - Process of moving data between systems Data Cleanup / Data Manipulation - Clean up errors in the data and streamline data preparation Data Distribution - Distribute data to different systems Change Detection - Determine changes between two different files Data Validation - Verify and validate spatial data Managing Databases Spatial types in a DBMS ETL

5 Data Interoperability Differentiators Not Just About Schema Manipulation 3D PDF Writing - Share data with the world Vertical Datum Handling - When working with water, transmission XML Documents as Features or Attributes JSON Documents as Attributes HTTP interaction GeoRSS interaction Cross-platform SQL support - SQLite temporary workspace Parallel Processing

6 Basics: OGC Support What you get just by installing

7 OGC GML Simple Feature Format Support Support for GML-SF in ArcGIS Desktop GML simple features WFS optionally non-cached for auto refresh Note: Installation of the Data Interoperability extension is required

8 OGC Demo: GML & WFS Bruce Harold

9 Demo Summary: GML & WFS Comes Free - But you have to install - Simple Features Only Better with the extension licensed - Non-simple parameters supported Direct Read of GML, consume WFS Important in many jurisdictions

10 Using Formats Direct Read of any format

11 Direct Read of Formats Formats automatically recognized by Data Interop. - File-based, by extension, like.tab,.mid,.skp,.sl3. - Visible in Catalog window browse tree - May be extended with Custom Formats Directly usable in ArcGIS - Save time converting data - ArcPy Features, Tables, SearchCursors Consume external data - ArcPad AXF - Other application formats - Web feeds

12 Easy Format Selection Browse or Search

13 When to Browse and when to Connect Browse & Direct Read uses format defaults Many formats have optional parameters - Select subsets, set conditions, passwords, filters Connections give you access to format parameters Connections let you set Coordinate Systems

14 Interoperability Connections Persist between application sessions Convenient to access in browse tree Retain your settings Off-disk data - Web Feeds - Google Spreadsheets Share with colleagues -.FDL file created - C:\Users\<user>\AppData\Roaming\Safe Software\Interoperability

15 Formats Demo: Direct Read Bruce Harold

16 Demo Summary: Direct Read Direct Read Summary - Directly read, inspect and browse within ArcGIS - Works like native ESRI formats - Access directly in ArcMap, Geoprocessing - No development or customization required Just works!

17 Quick Translation Quick Import & Quick Export GP Tools

18 Quick Data Import/Export Extension provides Geoprocessing Tools: Quick Export - To Any Format Quick Import - To Staging Geodatabase

19 Automation and Quick Translations Use Quick Tools in ModelBuilder - Get any format into a temporary File GDB - Handle Multiple Formats In/Out

20 Quick Translate Demo: Quick Export Bruce Harold

21 Demo Summary: Quick Import/Export Quick GP Tools - Useful in ModelBuilder - Also Scriptable in Python - Easy interaction with others - Departments - Contractors - 3 rd Party Software Users

22 Leveraging Formats Custom Formats

23 What if you need your own format? Save time by canning a custom format workflow Commonly handled text-based sources - CSV - XML Build format recognition workflow once Re-use the persisted workflow any time Lets you develop a file format for sharing

24 Leveraging Formats: Build a Custom Format Bruce Harold

25 Demo Summary: Custom Formats Based on an existing format May be shared -.FDS file in profile directory for each format - C:\Users\<user>\Documents\FME\Formats Editable later

26 The Power of Transformation Spatial ETL

27 The Full Power of Spatial ETL Formats are only part of the story Often the format does not change * - But the schema does Transformation is key - Many categories of transformer - Feature by Feature or Dataset by Dataset - Geometry or Attribute transformations Many Inputs Many Outputs is OK *like shapefile to shapefile

28 Workbench Application Graphically Design Data Flows READ TRANSFORM WRITE

29 It s a Pipeline Readers send features into the workspace - Geometry X:Y:Z:M - Attributes with a schema Transformers do something - May operate per feature and pass it immediately - May work on sets or all features together - Some offer parallel processing Writers send features to the destination DATA

30 Some Transformers Are All You Need This example derives Adds, Deletes and NoChanges in one step - Esri Community Maps contributor data is processed this way

31 The Power of Transformation Using Spatial ETL 100 s of Transformers - Categorized Browse Search Popup in Workbench Navigate to in Workbench Don t see what you need? - Ask for it!

32 Authoring Workbenches Readers, Transformers, Visualizers, Writers Add Spatial ETL tools into a ToolBox - Right Click context menu Wizard to stub out your Workbench Refine the workspace Publish Parameters - Inputs are automatic Run the workspace - In Workbench - As a GP Tool

33 Authoring Workbenches Readers, Transformers, Visualizers, Writers Add Spatial ETL tools into a ToolBox - Right Click context menu Wizard to stub out your Workbench Refine the workspace Publish Parameters - Inputs are automatic Run the workspace - In Workbench - As a GP Tool

34 Authoring Workbenches Readers, Transformers, Visualizers, Writers Add Spatial ETL tools into a ToolBox - Right Click context menu Wizard to stub out your Workbench Refine the workspace Publish Parameters - Inputs are automatic Run the workspace - In Workbench - As a GP Tool

35 Authoring Workbenches Readers, Transformers, Visualizers, Writers Add Spatial ETL tools into a ToolBox - Right Click context menu Wizard to stub out your Workbench Refine the workspace Publish Parameters - Inputs are automatic Run the workspace - In Workbench - As a GP Tool

36 Full ETL Demo: Some LiDAR Workflows Bruce Harold

37 Demo Summary: Spatial ETL Complements Core Functionality Analogous to ModelBuilder, except - Feature by Feature granularity - Feature geometry manipulable - Non-Esri data sources are OK

38 Wrapping Up More on use cases

39 A Word on XML XML is a Document, not a Format XMLTemplater - Easily use an XML file to stub out XML - Populate attributes with XQuery expressions XQuery Extract, Update, Explode <feederobject objectid="{fme:get-attribute("objectid")}"> <feedername>{fme:get-attribute("feedername")}</feedername> <feederno>{fme:get-attribute("feederno")}</feederno> <ealoc>{fme:get-attribute("ealoc")}</ealoc> </feederobject>

40 Organizational Interaction Regional / County National GIS State ETL Federal

41 ArcGIS Server and Data Interoperability 64bit Non-Native formats in Map Services Publish ETL as GP services Esri WFS services more flexible - Non-cached streaming option - Setting in interop connection

42 New in 10.1 More Python-based formats are coming in SP1 Based on latest FME platform FME Improved performance - Improved FME Workbench user experience New Formats - ArcPad AXF (requires SQL Compact) - AIXM5 - Esri Mapping Specification for CAD - Google Fusion Tables - Point Cloud (ASCII XYZ, LAS, LAZ) - R Statistical Data New Transformers - FeatureReader spatial and aspatial queries - SQLCreator create feature per result row - Stylers symbology in DWG, DGN, KML, PDF

43 Resources: Learn More How do I try it? - Request software evaluation (60 days): - roperability/evaluate How do I learn it? - Two free courses (new for-fee courses coming): - ebcoursedetail&courseid= ebcoursedetail&courseid=1588 Who do I talk to? - Visit the Analysis Island in the Esri Showcase or the SAFE Software booth (#2400) in the exhibit hall.

44 Steps to evaluate UC sessions My UC Homepage > Evaluate Sessions Choose session from planner OR Search for session

45 Thank you for attending Have fun at UC2012 We are now open for Questions Please fill out the evaluation: First Offering (Tuesday) ID: 1164 Second Offering (Wednesday) ID: 752

46