Spatially Enable Your DBA PUG Miles Phillips LandWorks, Inc Feb 27, 2008

Transcription

1 Spatially Enable Your DBA PUG 2008 Miles Phillips LandWorks, Inc Feb 27, 2008

2 Audience Experienced DBAs assigned to support ArcSDE Experienced ArcGIS users that need to understand their DBA can do for them

3 Geodatabase Foundation of ArcGIS Server Stores and manages spatial data Scales from personal to large enterprise databases In your chosen container Online ArcGIS Server Help

4 Geodatabase Types Personal Geodatabase (MS Access) File Geodatabase (File system folders) ArcSDE SQLServer 2005 Express Personal / Workgroup SQLServer, Oracle, DB2, Informix Enterprise (multiuser RDBMS) ArcGIS Server Help Types of geodatabases

5 What is ArcSDE really? The Blind Men and an Elephant

6 ArcSDE Perspectives Casual GIS user ArcGISTech ArcSDE Administrator Spatial Data Administrator System Administrator DBA

7 ArcSDE to the GIS users

8 Fundamental Datasets in GDB Feature classes (AKA SDE Layers) Features (points, lines, polygons, annotation) Raster datasets Tables Registered to GDB

9 ArcSDE Through ArcCatalog

10 ArcSDE as seen by the DBA Tables Indexes Triggers Views Special accounts / privileges

11 ArcSDE DBMS Accounts ArcSDE Administrator AKA: SDE account User Accounts Data Viewer Data Editor Data Creator GDB Administrator

12 ArcSDE System Tables Owned by SDE admin LAYERS TABLE_REGISTRY Others VERSIONS, STATES, MVTABLES_MODIFIED, RECONCILED_STATES, METADATA, LOCATOR, RASTER_COLUMNS, GEOMETRY_COLUMNS, SPATIAL_COLUMNS

13 GDB System Tables Owned by SDE admin GDB_FEATURECLASSES GDB_FEATUREDATASET GDB_FIELDINFO GDB_OBJECTCLASSES GDB_USERMETADATA Others.

14 ArcSDE System Tables SDE admin account GDB System Tables GDB_FEATURECLASSES GDB_OBJECTCLASSES SDE System Tables LAYERS TABLE_REGISTRY DBTUNE

15 ArcSDE Admin User Permissions See ArcGIS Server Help User Permissions

16 ArcSDE Admin User Permissions Oracle (side notes) The SDE user always owns the master GDB User with privs can create own GDB SQLServer dbo user can be the ArcSDE Admin if using the single GDB model; otherwise uses SDE DB2 uses the SDE account Informix uses the SDE account

17 User Permissions Data Viewer Oracle CREATE SESSION SELECT on database objects SQLServer Informix CREATE TABLE (not needed for session-based log files) SELECT on database objects CONNECT DB2 CONNECT to database SELECT on database objects ArcGIS Server Help User Permissions

18 User Permissions Data editors Oracle CREATE SESSION SELECT, INSERT, UPDATE, DELETE on other user s data SQLServer Informix DB2 CREATE TABLE CREATE STORED PROCEDURE SELECT, INSERT, UPDATE, DELETE on other user s data RESOURCE CONNECT to database CREATETAB in database CREATEIN, ALTERIN, DROPIN for necessary schema CONTROL, ALTER, DELETE, INSERT, SELECT, UPDATE REFERENCES ArcGIS Server Help User Permissions

19 User Permissions Data creators Same as Data Editors but also includes Oracle SQLServer Informix DB2 CREATE PROCEDURE CREATE SEQUENCE CREATE TRIGGER CREATE VIEW (if using views) CREATE TABLE CREATE VIEW (if using views) No additional CONNECT to database CREATETAB in database CREATEIN, ALTERIN, DROPIN for necessary schema CONTROL on database objects ArcGIS Server Help User Permissions

20 ArcSDE User Tables Depends on feature geometry storage types ArcSDE Compressed Binary Spatial Type (Oracle) For complete list, see ArcGIS Desktop Help An Overview of Feature Geometry and Raster Data Storage

21 ArcSDE User Tables ArcSDE Compressed Binary Three tables per layer Business Table User Attributes 1 COUNTIES SHAPE NUMBER OBJECTID NUMBER NAME VARCHAR POP2000 NUMBER F table geometry 1 S table spatial index 1 M F1201 FID NUMBER S1201 SP_FID NUMBER

22 ArcSDE LAYERS Table SDE.LAYERS table Select Layer_ID, Owner, Table_Name from SDE.LAYERS; 1201 LAND COUNTIES 1503 LAND TOWNSHIP

23 ArcSDE User Tables Spatial Type Two tables per layer Business Table User Attributes COUNTIES SHAPE ST_GEOMETRY OBJECTID NUMBER NAME VARCHAR POP2000 NUMBER S table spatial index S05_IDX$ SP_ID ROWID

24 User Business Tables DBMS table with shape column (layer) Limited data type support ArcGIS Desktop Help Data types in the DBMS

25 Delete trigger on business table ArcSDE Compressed Binary TRIGGER SPCOL_DEL_CASCADE_<layer> AFTER DELETE OR UPDATE OF <SPATIAL_COL> ON business_table IF DELETING THEN DELETE FROM F<layer> WHERE F<layer>.fid = old.<spatial_col> DELETE FROM S<layer> WHERE S<layer>.fid = old.<spatial_col> IF UPDATING AND new.<spatial_col> IS NULL THEN DELETE FROM F<layer> WHERE F<layer>.fid = old.<spatial_col> DELETE FROM S<layer> WHERE S<layer>.fid = old.<spatial_col> IF UPDATING AND new.<spatial_col>!= old.<spatial_col> THEN raise_application_error (-20013,'Invalid update of SDE Spatial Column.')

26 Part #2 The Whine List

27 Whine #1

28 Update DBMS Indexes Rebuild indexes Analyze spatial data Search support.esri.com on How can ArcSDE performance be improved

29 Updating statistics Through ArcCatalog

30 Updating statistics Through ArcSDE command line sdetable o update_dbms_stats t LW_LEGAL_DESCRIPTION_KEY i esri_sde s gis_server u bound p?? -N option, no prompt -q option, no messages Search support.esri.com on Updating statistics on an ArcSDE Enterprise geodatabase

31 Through SQL Select 'Alter Index ' Index_Name ' Rebuild;' From User_Indexes; Select Distinct 'Analyze Table Table_Name ' Compute Statistics;' From User_Indexes;

32 Whine #2

33 ArcSDE and DBMS Initialization Settings See ArcSDE Tuning Guide for your DBMS Example SDEHOME\etc\dbinit.sde init.ora parameter settings Oracle block size (8k? 16k?)

34 Tuning an ArcSDE GDB Separate frequently used files to minimize disk I/O - Business tables, S-tables, F-tables, Indexes Set initialization parameters - Oracle - SGA must not swap (not larger than 2/3 physical RAM) - Avoid excessive paging ( SGA too large. ) - Configure enough virtual memory (4 x RAM) - Use explicit quotas on tablespaces - SQL Server - Limit SQL Server s memory consumption (fixed or max memory size) - Boost SharedSection in the registry (more than 50 connections) - Use lightweight pooling for Windows NT fibers Search support.esri.com on An Overview of tuning ArcSDE geodatabase

35 ArcSDE Layer Storage Controls SDE.DBTUNE table KEYWORD PARAMETER_NAME CONFIG_STRING sdedbtune.exe -o import f dbtune.txt u sde -o export f dbtune.txt u sde

36 DBTUNE Default Entry ##DEFAULTS A_INDEX_RASTER "PCTFREE 0 INITRANS 4 NOLOGGING " A_INDEX_ROWID "PCTFREE 0 INITRANS 4 NOLOGGING " A_INDEX_STATEID "PCTFREE 0 INITRANS 4 NOLOGGING " A_INDEX_SHAPE "PCTFREE 0 INITRANS 4 NOLOGGING " A_INDEX_XML "PCTFREE 0 INITRANS 4 NOLOGGING " ATTRIBUTE_BINARY "BLOB" AUX_STORAGE "PCTFREE 0 INITRANS 4" B_INDEX_ROWID "PCTFREE 0 INITRANS 4 NOLOGGING " B_INDEX_TO_DATE "PCTFREE 0 INITRANS 4 NOLOGGING " D_INDEX_DELETED_AT "PCTFREE 0 INITRANS 4 NOLOGGING " BND_STORAGE "PCTFREE 0 INITRANS 4" BND_INDEX_ID "PCTFREE 0 INITRANS 4 NOLOGGING " BND_INDEX_COMPOSITE "PCTFREE 0 INITRANS 4 NOLOGGING "

37 DBTUNE Keywords Common Keywords GEOMETRY_STORAGE ATTRIBUTE_BINARY RASTER_STORAGE A_STORAGE F_STORAGE S_STORAGE AUX_STORAGE ArcGIS Server Help DBTUNE configuration Parameters for complete list

38 Updating DBTUNE Defining tablespace example 1. sdedbtune o export 2. edit SDEHOME\etc\dbtune.txt ##LANDGRID UI_TEXT For Storage of LANDGRID A_STORAGE TABLESPACE LANDGRID_A F_STORAGE TABLESPACE LANDGRID_F S_STORAGE TABLESPACE LANDGRID_S. END 3. sdedbtune o import

39 Accessing Custom DBTUNE Config_string ##DEFAULTS ##user defined Configuration Referenced by SDE command line executables and ArcCatalog GUI sdelayer o create K <config> sdetable o create K <config>

40 Accessing DBTUNE Configuration through ArcCatalog

41 ArcSDE System Tables SDE.LAYERS table (SDE.SDE.SDE_LAYERS) Select Layer_ID, Owner, Table_Name from SDE.LAYERS; 1201 LAND COUNTIES 1503 LAND TOWNSHIP SDE.TABLE_REGISTRY table Select OWNER, TABLE_NAME, CONFIG_KEYWORD from SDE.TABLE_REGISTRY; LAND COUNTIES DEFAULTS LAND TOWNSHIP LANDGRID

42 Whine #3

43 Problem with User s LOGFILE Table used to storage primary key values of selected features See ESRI tech support for help

44 Whine #4

45 sdeimport o create l jeff_sect,shape SDE admin account GDB System Tables GDB_OBJECTCLASSES (before) LAND user account SDE System Tables LAYERS

46 sdeimport o create l jeff_sect,shape SDE admin account GDB System Tables GDB_OBJECTCLASSES (after) LAND user account JEFF_SECT F1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT S1301

47 Registering to GDB (before) SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND user account JEFF_SECT F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

48 Registering to GDB (after) SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT LAND user account JEFF_SECT F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

49 Delete a Layer If not registered with GDB, then either Delete with ArcCatalog Delete with SDE command line utilities sdelayer o delete sdetable o delete If registered with GDB, then must delete with ArcCatalog

50 sdelayer o delete l jeff_sect,shape (before) SDE admin account LAND user account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT JEFF_SECT F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

51 sdelayer o delete l jeff_sect,shape SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT (after) LAND user account SDE System Tables LAYERS

52 Delete from ArcCatalog (before) SDE admin account LAND user account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT JEFF_SECT F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

53 Delete from ArcCatalog (after) SDE admin account LAND user account GDB System Tables GDB_OBJECTCLASSES SDE System Tables LAYERS

54 Drop Table JEFF_SECT; (before) SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT LAND user account JEFF_SECT F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

55 Drop Table JEFF_SECT; (after) SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT LAND user account F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

56 Drop User LAND Cascade; (before) SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT LAND user account JEFF_SECT F1301 S1301 SDE System Tables LAYERS 1301 LAND JEFF_SECT

57 Drop User LAND Cascade; SDE admin account GDB System Tables GDB_OBJECTCLASSES LAND JEFF_SECT (after) SDE System Tables LAYERS 1301 LAND JEFF_SECT

58 More on Registered Tables Must change columns with ArcCatalog Careful manipulating the table structure directly with SQL Take care when migrating registered objects

59 Whine #5

60 Ghost Features See the feature but cannot touch it Demo Causes SPCOL_DEL_CASCADE_<layer> missing/disable Business table tablespace full Cleanup delete orphaned F table records Delete From F<layer> f Where Not Exists (Select 1 From COUNTIES c Where c.shape = f.fid)

61 Bulk delete tips Don t delete on a versioned layer/table Don t use SQL TRUNCATE Use DELETE without a where clause sdelayer o truncate sdetable o truncate

62 Whine #6

63 Register with GDB

64 Register with GDB Adds OBJECTID column Populates the GDB metadata

65 Whine #7

66 Register as Versioned Adds A and D tables per layer Could need compressed

67 Whine #8

68 Unsupported Data Type Provide a view, converting invalid data type ArcGIS Desktop Help Data types in the DBMS

69 Whine #9

70 No access to layer User does not have select access Grant through ArcCatalog Grant through sdelayer o grant Layer has been deleted / corrupted

71 Whine #10

72 Issues with moving data ArcCatalog drag and drop if tables/layers all registered XML workspace document (with or without data) Uncompressed (potential very large files) GDB aware sdexport/sdeimport Not database aware Must register SDE layers after move

73 Issues with moving data (continued) DBMS Export / Import User level export + SDE admin export ok only if new instance does not have SDE yet No merge of SDE metadata ORA10DEV instance ORA10PROD instance SDE.LAYERS (110) LANDTEST.JEFF_SECT SDE.LAYERS (102) WELLS.WELL_BORE

74 Other Issues

75 Check DBMS Version or ArcSDE patch level

76 Random Tipbits Don t store layers in SDE account Discourage adding unregistered tables in an account with GDB registered tables

77 Maintaining a ArcSDE GDB Database backup and recovery Compress versioned geodatabases Data synchronization for replication Search support.esri.com for Maintaining an ArcSDE geodatabase

78 ArcSDE Admin Commands See ArcGIS -> ArcSDE -> Command References

79 Connection Choices Direct connect client connects directly no ArcSDE service 3 tier architecture ArcSDE service DBMS Client connects to service

80 Spatial Views sdetable o create_view t?? T?? c?? Alter the view with SQL Search support.esri.com on Using database views

81 Setting up test environments Same instance Different accounts Same server, different instance Advantage that account can have the same name / different instance Different servers

82 ArcSDE Data Access Model ArcSDE Developer SDK Help