EEOS Spatial Databases and GIS Applications

Transcription

1 EEOS Spatial Databases and GIS Applications Lecture 6 Introduction to Enterprise Geodatabases and ArcSDE

2 What is ArcSDE? Technology for multiuser geodatabases integrated into Esri s ArcGIS for Desktop and ArcGIS for Server products Spatial Database Engine Manages spatial data within an enterprise RDBMS Gateway ( middleware ) between GIS clients and an enterprise RDBMS Enables you to store, access, and manage spatial data within an RDBMS such as Oracle, Microsoft SQL Server and SQL Server Express, IBM DB2, IBM Informix (Commercial) PostgreSQL (Open Source) EEOS Spring 2015: Lecture 6 2

3 What is ArcSDE? Typically installed on a server (for Enterprise GDBs) for Windows, Unix and Linux Desktop and Workgroup GDBs can be installed on a server or local PC Multiuser GDBs have all the functionality of file and personal GDBs, including all the application logic EEOS Spring 2015: Lecture 6 3

4 What is ArcSDE? Critical when you need to manage long transactions and versionedbased workflows such as Support for multiuser editing Distributed editing Managing historical archives EEOS Spring 2015: Lecture 6 4

5 Notes to Keep in Mind RDBMS physically stores relational tables vector, raster, tabular data (expanding functionality of RDBMS), metadata, ArcSDE interprets the contents for use in GIS software and clients ArcGIS, ArcIMS, ArcView 3.x (via Database Access extension) and ArcInfo Workstation, non-esri and OpenSource too (like MassGIS OLIVER) EEOS Spring 2015: Lecture 6 5

6 ArcSDE as Part of ArcGIS 3-tier architecture Clients ArcSDE Technology RDBMS Prior to ArcGIS 9.2, Esri sold ArcSDE as a core product. At ArcGIS 9.2, ArcSDE is no longer a separate product. It is now ArcSDE Technology. EEOS Spring 2015: Lecture 6 6

7 Benefits of ArcSDE Centralized spatial database Large volume of data Seamless, non-tiled layers Rapid spatial searching Storage and integration of vector, raster, tabular data Database and application portability Concurrent access for MANY users Versioning/Multi-user editing Clients connect over TCP/IP network (e.g. WEB) Direct connect also available EEOS Spring 2015: Lecture 6 7

8 Benefits of ArcSDE Use of robust RDBMS environment Security, backup and recovery Scalability, Indexing Referential and data integrity Can access GIS attribute data at the db level via SQL*Plus, OSQL, or in MS Access with ODBC connection, etc. Link to data in existing RDBMS Supports long transactions and Views SDE administrators may be assisted by system administrators and RDBMS administrators EEOS Spring 2015: Lecture 6 8

9 Benefits of ArcSDE Spatial queries directly at SQL prompt if RDBMS supports spatially-enabled data types Oracle Spatial, DB2 Spatial Extender, Informix Spatial DataBlade, SQL Server 2008 spatial data type Use C, Java API for custom app. dev. Fully integrated with ArcGIS Can batch data import/export and other processes using Administrator Commands EEOS Spring 2015: Lecture 6 9

10 Drawbacks of ArcSDE Requires separate purchase of RDBM$ (but PostgreSQL is free) Need a DBA or be trained in your RDBMS software for backups, tuning (require time, money, expertise) Can take a while to master intricacies, work around RDBMS errors EEOS Spring 2015: Lecture 6 10

11 Types of ArcSDE (ArcSDE GDBs are Scalable) Enterprise Geodatabase Workgroup Geodatabase Desktop Geodatabase (known as Personal SDE at 9.2) Application Scenario Large-scale enterprise applications Small-to medium-sized departmental applications Small teams or a single user who requires the functionality of a multiuser geodatabase Supported RDBMS DB2, Informix, Oracle, PostgreSQL, SQL Server SQL Server Express SQL Server Express Management Interface ArcCatalog, RDBMS, ArcSDE command line ArcCatalog ArcCatalog Storage Capacity Depends on the server 4 GB 4 GB Licensing Availability Supported OS Platform Number of Concurrent Users Network Application ArcGIS Server Enterprise ArcGIS Server Workgroup ArcGIS Engine, ArcInfo, ArcEditor Any platform Windows Windows Unlimited editors and readers 10 editors and readers 1 editor and 3 readers Intranet and Internet Intranet and Internet Desktop and local network use Differentiating Characteristics Supports versioning and multiuser editing; Supports spatial types; integration with enterprise IT Supports versioning and multiuser editing Supports versioning N E 9. 2 EEOS Spring 2015: Lecture 6 11

12 Accessing ArcSDE in ArcGIS Desktop and Workgroup Enterprise EEOS Spring 2015: Lecture 6 12

13 Accessing Enterprise ArcSDE Double-click Enter the parameters as shown at right. Note: the Password is the same as the User Name To connect to your data (to edit), you would enter your Oracle username and password Click OK. These parameters get saved to a.sde file in your profile (on the C: drive). You can copy this file to H: for use on any PC. The password for DATAREADER is DATAREADER1 You can also use the connection in S:\ge381_s15\ EEOS Spring 2015: Lecture 6 13

14 ArcSDE Feature Classes GDB container for vector data One geometry type (point, line, polygon, annotation, etc.) per feature class, and attributes Owned by RDBMS users (i.e. Oracle accounts, like MGISDATA) Stored as series of tables in the RDBMS Joined at query time so that the client (i.e. ArcCatalog) sees one object (layer) in the geodatabase One row is one feature (single or multi-part) Spatial column identifies geometry SHAPE field in RDBMS, actual coordinates stored in BLOB (binary long object) or long raw formats shape + attributes = feature EEOS Spring 2015: Lecture 6 14

15 ArcSDE Feature Classes ArcSDE manages integrity of tables (i.e. when a layer is renamed, deleted, edited) users should never alter these tables schemas i.e. add columns to layers in ArcCatalog or ArcMap only!! You CAN use SQL*Plus or MS Access to edit business table attributes (and to add rows to non-feature class tables but be careful! EEOS Spring 2015: Lecture 6 15

16 ArcSDE Feature Classes Types of storage ST_GEOMETRY Base (business) table with attributes and spatial column, and spatial index table Default at version 10 SDELOB (binary geometry type) Business (attributes), F (spatial), and S (spatial index) tables Each type is linked to a configuration keyword as defined in the SDE.DBTUNE table. Choose the keyword when creating data. For more details see: EEOS Spring 2015: Lecture 6 16

17 Feature class tables in the RDBMS (SDELOB storage) Business table stores attributes name in RDBMS is feature class name (e.g. TOWNS_POLY), as seen in ArcGIS. Its actual appearance depends on the RDBMS: Oracle: OWNER.TABLENAME Ex: MGISDATA.TOWNS_POLY SQL Server: DATABASE.OWNER.TABLENAME Ex: SDE.MGISDATA.TOWNS_POLY EEOS Spring 2015: Lecture 6 17

18 Feature class tables in the RDBMS (SDELOB storage) Feature table ( F table) stores geometry in binary column (X,Y,Z,M values), plus envelope (bounding rectangle) for each feature, area & length named F<layer_id> in RDBMS (layer_id is primary key in SDE.LAYERS table) one-to-one relationship between business and F table, via unique feature identifier ( OBJECTID, always indexed) *** With ST_GEOMETRY storage the business table stores the geometry, and there is no F table EEOS Spring 2015: Lecture 6 18

19 Feature class tables in the RDBMS (SDELOB storage) Spatial index table ( S table) stores grid tiles and envelope (X/Y extent of each tile) named S<layer_id> in RDBMS one-to-many relationship between F and S table used during spatial searches (pan/zoom) for optimum data retrieval *** With ST_GEOMETRY there is also a spatial index table for each feature class EEOS Spring 2015: Lecture 6 19

20 Feature class tables in the RDBMS Adds table ( A table) stores added features during ArcMap versioning edit session Deletes table ( D table) stores deleted features during ArcMap versioning edit session aka Delta (change) tables created only when a feature class is Registered as Versioned Named A<registration_ID> and D<registration_ID> (registration_id is primary key in SDE.TABLE_REGISTRY table) EEOS Spring 2015: Lecture 6 20

21 Attribute Indexes Column indexes may be built on attributes in feature class business table Improves performance when using joins, relates, and relationship classes. EEOS Spring 2015: Lecture 6 21

22 Spatial Indexes Clients use spatial filter to reduce query results and speed up data retrieval Ex.: Only fetch visible features for display ArcSDE uses spatial index to reduce I/O Eliminates need for full table scan Based on standard feature of RDBMS Format of the spatial index is determined by the RDBMS being used Grid tile or R-tree (Oracle Spatial) structure EEOS Spring 2015: Lecture 6 22

23 Spatial Indexes ArcSDE for Oracle uses grid tile structure Spatial index built by applying a grid to the feature class ArcSDErecords which features fall within each grid cell in an index table ( S table), based on feature envelope A feature that falls within many cells is listed in each Grid cells with no data are not listed in table A layer may have 1 to 3 index grids, though usually one is sufficient When you create a vector feature class, ArcGIS automatically determines the optimum grid cell size (in the units of the data, e.g. feet, meters) and the number of grids. The feature class is overlaid by grid cells to create the spatial index. EEOS Spring 2015: Lecture 6 23

24 Spatial Indexes Manage in Indexes tab in the Feature Class Properties box (you must be the data owner) Can also use: EEOS Spring 2015: Lecture 6 24

25 Loading Data into ArcSDE Use ArcToolbox Conversion Tools > To Geodatabase Right-click spatial database connection and choose Import Supported data types or XML docs Right-click existing SDE layer and choose Load > Load Data Create new empty FC with desired schema (can import from a table) and Load in features from any data source Right-click non-sde layers and choose Export > To Geodatabase Use SDE commands shp2sde, cov2sde, tbl2sde (for tables) Use special tools/dialogs for labels and annotation In all cases, you must specify a configuration keyword (your SDE administrator will tell you if you should use DEFAULTS or another keyword EEOS Spring 2015: Lecture 6 25

26 After Loading Data into ArcSDE Right-click and Assign privileges to data way to control access Enter db user or role Check SELECT, UPDATE, INSERT, DELETE Analyze Updates DBMS statistics Check/clear components to analyze Rename layers? Loaded data named the same as existing data will have _1 in new name Take a look at the data!!! EEOS Spring 2015: Lecture 6 26

27 Raster Data in the RDBMS >> ArcSDE/GDB raster layers are known as raster datasets EEOS Spring 2015: Lecture 6 27

28 Raster Data in the RDBMS Benefits RDBMS security, data management and retrieval Common data format Supports large seamless images (mosaics), raster catalogs, and mosaic datasets Compression LZ77 lossless or lossy JPEG and JPEG 2000 lossy Pyramids stored in database for improved display performance EEOS Spring 2015: Lecture 6 28

29 Raster Data in the RDBMS Loading ArcCatalog / ArcToolbox (all supported formats) Use sderaster command (for 1 to 3-band TIFF, BSQ, or ArcSDE raster) Then: Build Pyramids Calculate Stats Analyze Pyramids enable fast display at any scale EEOS Spring 2015: Lecture 6 29

30 Raster Data in the RDBMS SDE.RASTER_COLUMNS table Stores information on every raster dataset in the database, one entry per dataset Each layer has unique ID - RASTERCOLUMN_ID - assigned by ArcSDE For more details see: storage_types_in_an_arcsde_geodatabase/002n v000000/ EEOS Spring 2015: Lecture 6 30

31 Raster Data in the RDBMS and a series of many tables Raster layer Business table (OWNER.IMAGE_NAME as seen in ArcGIS) Link between supporting tables and RASTER_COLUMNS table Raster layer Block Table (SDE_BLK_<RASTERCOLUMN_ID>) Stores actual pixel data Raster layer Metadata Table (SDE_RAS_<RASTERCOLUMN_ID>) contains raster description Raster layer Band Table (SDE_BND_<RASTERCOLUMN_ID>) Information about each raster band Raster layer Auxiliary table (SDE_AUX_<RASTERCOLUMN_ID>) Stores colormaps and statistics EEOS Spring 2015: Lecture 6 31

32 ArcSDE Uses Client/Server Model All data stored in RDBMS accessed over a TCP/IP network All data is retrieved through SQL (Structured Query Language) queries SELECT statements fetch data (see pages in textbook) Query filters limit the rows returned Attribute and Spatial Clients display data Two connection methods Application Server Direct Connect EEOS Spring 2015: Lecture 6 32

33 ArcSDE Application Server Connection Traditional ArcSDE connection method Available for all supported RDBMS GIS client requires no additional software Server requires ArcSDE instance giomgr spawns one dedicated gsrvr process per connection, using RDBMS authentication Performance considerations Decreases client load Decreases network load Increases server load Link between application and database Processes client requests Sends data from client to server Updates & maintains ArcSDE metadata Active until user disconnects Listens for, monitors, and maintains connections, spawns gsrvr processes ArcSDE giomgr ArcGIS TCP/IP network 5151 ArcSDE gsrvr RDBMS server Data EEOS Spring 2015: Lecture 6 33

34 ArcSDE Direct Connect Client connects directly to RDBMS Client does its normal job AND performs function of the gsrvr, using a software library called a Direct Connect driver No ArcSDE instance used No gsrvr or giomgr processes required on server Performance considerations Increases client load Increases network load Decreases server load (most appropriate for robust clients on hearty computers) More stable than App Server connection ArcGIS client (with Direct Connect Driver) RDBMS This is the default at 10.1 For more details start with: _connection_to_a_geodatabase_in_oracle/002n / EEOS Spring 2015: Lecture 6 34

35 Services File Stores port number (5151) used by ArcSDE for communication between client and RDBMS Add the following line: esri_sde 5151/tcp to the SERVICES file in C:\WINDOWS\SYSTEM32\DRIVERS\ETC or C:\WINNT\system32\drivers\etc or other similar location. EEOS Spring 2015: Lecture 6 35

36 TNSNAMES.ORA File Text file that stores information that defines Oracle database addresses for establishing connections to them. This file normally resides in the %ORACLE_HOME%\NETWORK\ADMIN folder. Required for direct connection to SDE from ArcGIS (and to use SQL*Plus) Syntax: <addressname> = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(Host = <hostname>)(port = <port>)) ) (CONNECT_DATA = (SERVICE_NAME = <sid>) ) ) EEOS Spring 2015: Lecture 6 36

37 SDE.DBTUNE Table RDBMS table (owned by SDE user) that stores storage tuning parameters in an enterprise ArcSDE geodatabase Based on keywords DEFAULTS used for all feature classes loaded without keyword specified Different keywords can be set up by the SDE administrator for vector data, raster data, topology, and for different users and editors Control where data are stored - i.e. which tablespace (Oracle) or file group (SQL Server) for feature tables, indexes. If you want users to be able to write to (i.e., store data in) certain parts of a database, the SDE administrator would set up a keyword (like ##STUDENTS), and in ArcGIS dialogs that have a dropdown or textbox to specify a configuration keyword, the user would enter STUDENTS. EEOS Spring 2015: Lecture 6 37

38 ArcSDE Repository Administrative information Feature class name Unique feature class ID, etc. AKA ArcSDE system tables sde.raster_columns sde.table_registry sde.layers LAYER_ID TABLE_NAME... A few of the many tables owned by the sde user See system tables diagram at: Full details at: tem_tables/002n m000000/ EEOS Spring 2015: Lecture 6 38

39 ArcSDE Administration Tools Command-line utilities for managing the ArcSDE server Executed at operating system prompt May be used remotely Command syntax found in the ArcSDE Developer Help See /admincmdref.htm Benefits Easy to format Easy to debug Reusable Record of work (can write output to log files) Unattended execution (can batch) EEOS Spring 2015: Lecture 6 39

40 ArcSDE Administration Tools Example: the sdemon command Purpose: ArcSDE server monitor, used to check connection statistics. The sytax is: sdemon -o info -I <{users config stats locks }> [-I <instance>] Required parameter sdemon -o info Optional parameter [ ] Substitute a value < > Choose a value from list for the <instance> argument: -I <{users config stats locks }> Others: sderaster, cov2sde, shp2sde, sdeexport, sdeimport, sdelayer, sdetable,... EEOS Spring 2015: Lecture 6 40

41 Versioning ArcSDE Geodatabase mechanism to allow multiple users to simultaneously edit the same database, including the same layers and features, without explicitly applying locks to prohibit other users from modifying the same data Part of multi-user editing capability of SDE geodatabases No need to extract or make copies of data for editing EEOS Spring 2015: Lecture 6 41

42 Versioning What is a version?: In databases, an alternative state of the database that has an owner, a description, a permission (private, protected, or public), and a parent version. Versions are not affected by changes occurring in other versions of the database. - From ESRI GIS Dictionary You can think of a version as your own doorway to the database, allowing you to make changes that don t affect the database as a whole. Also see ESRI Help: is_a_version/ q000000/ k_tour_of_versioning/ r000000/ EEOS Spring 2015: Lecture 6 42

43 Versioning Use versioning to: Manage alternative engineering designs Solve complex what-if scenarios without impacting the production database Create point-in-time representations of the database Conflict resolution dialog allows DBA to choose proper version EEOS Spring 2015: Lecture 6 43

44 Versioning Versions are not copies of the database or layers Only store edits made to database in A and D tables Versions are named default version (always present) is SDE.DEFAULT syntax: OWNER.VERSION_NAME AKA a named state of the database Maintain parent-child relationships (see chapter 7 in Modeling Our World and chapter 10 in Building A Geodatabase) EEOS Spring 2015: Lecture 6 44

45 Versioning Isolates user's work across multiple edit sessions, allowing the user to edit without locking features in the production version or immediately impacting other users. Supports undo/redo capability Multiple users can access a version A user may access multiple versions What the user can do is based on version permissions Contains access to all the datasets in the geodatabase EEOS Spring 2015: Lecture 6 45

46 Versioning Versions have permissions: Public - any user may view the version and modify features Protected - Any user may view version, only the owner may edit Private - Only the owner may view the version and modify features Notes: Version owner can rename, delete, change access SDE user can see and delete all versions regardless of their permissions EEOS Spring 2015: Lecture 6 46

47 Versioning How tables react to edits: Insert (add) feature record (OBJECTID) added to A table Delete feature record (OBJECTID) added to D table Update feature records (OBJECTIDs) added to A and D tables EEOS Spring 2015: Lecture 6 47

48 Versioning Process Step 1 Data Owner In ArcCatalog: 1.Register data with geodatabase, if necessary 2.Register data as Versioned (makes data editable) A & D tables added to feature class: A<registration_ID> stores adds D<registration_ID> stores deletes 3.Grant SELECT, UPDATE, INSERT, DELETE privileges to all editors To register an object as versioned, right-click it in ArcCatalog and choose Register As Versioned. You must be connected as the owner of the dataset. EEOS Spring 2015: Lecture 6 48

49 Versioning Process Step 2 - Editor 1. Create version, set permission (ArcCatalog) 2. Change version in your spatial database connection and add feature class (or feature dataset) to ArcMap 3. Make and save edits in ArcMap 4. From Versioning Toolbar in ArcMap: Reconcile with parent (other edits in parent seen by child) Post changes to parent (changes in child seen by parent) parent owner needs to do this if parent version is not public parent makes version of child and posts EEOS Spring 2015: Lecture 6 49

50 Versioning Process Editing SDE.DEFAULT (PROTECTED) SCOTT.OS_DBA (Protected) MTRUST.MYEDITS (protected) Layer edited and saved JKERRY.MYEDITS (protected) SCOTT.MTRUST_QA (protected) SDE.READY_FOR_DEFAULT EEOS Spring 2015: Lecture 6 50

51 Versioning Process Step 3 SDE user 1. Compressing the Database Done as SDE user, with Compress button added to a toolbar from Customize box in ArcCatalog, or with ArcToolbox tool Moves records in A & D tables into base (business, F, & S) tables Removes all redundant rows and states not referenced by a version Improves performance Two types: Full compress: When editing is done and all reconciling and posting is complete, after deleting all versions Partial compress: done anytime, but locks may prevent deletion of certain states 2. Analyze data after compression Note: Compact is procedure for PGDBs EEOS Spring 2015: Lecture 6 51

52 Versioning Process - Compressing SDE.DEFAULT (PROTECTED) Compress Geodatabase SCOTT.OS_DBA (Protected) Start the process all over again MTRUST.MYEDITS (protected) Layer edited and saved JKERRY.MYEDITS (protected) SCOTT.MTRUST_QA (protected) SDE.READY_FOR_DEFAULT EEOS Spring 2015: Lecture 6 52

53 Non-versioned Editing Editing of source data directly Features and attributes The last saved edit is final No ability to undo/redo changes The only way to undo back to last edit is to stop editing without saving changes The object (feature class or table) does not have to be registered as versioned) To turn on non-versioned editing, in ArcMap, go to the Editor toolbar, click on the Editor menu dropdown, and choose Options. Then, in the Versioning tab, uncheck the Edit a version checkbox. EEOS Spring 2015: Lecture 6 53

54 GDB Replication Allows you to create copies of data across two or more geodatabases such that changes to the data may be synchronized. Requires ArcInfo or ArcEditor For more information, start with Understanding distributed data in ArcGIS Help for full details: rstanding_distributed_data/ / EEOS Spring 2015: Lecture 6 54

55 GDB Archiving Provides the functionality to record and access changes made to all or a subset of data in a versioned geodatabase. The mechanism for capturing, managing, and analyzing data change ( historical versions ). See Geodatabase archiving for more details: atabase_archiving/ / EEOS Spring 2015: Lecture 6 55

56 Views A virtual table in a database that displays certain information in the database, based on a view definition Not a table itself; the definition is stored and the view appears as a table when accessed by a user Advantages Can limit rows (use where clause ) and can include only certain fields Simplicity and security Can assign alias (more intuitive) field names Can include joins to other tables When SHAPE field is included in an SDE view, the view appears as an ArcSDE Feature Class ( Spatial view ) Use the sdetable o create_view admin command See: EEOS Spring 2015: Lecture 6 56