Best Practices with ArcSDE Technology. Katja Krivoruchko CA/NV/HI Technical Marketing ESRI

Transcription

1 Best Practices with ArcSDE Technology Katja Krivoruchko CA/NV/HI Technical Marketing ESRI

2 Agenda What is ArcSDE Maintaining ArcSDE performance Geodatabase Toolset (GDBT) Upgrading ArcSDE Capacity Planning Performance and scalability Principles of planning for capacity Multiuser Editing Versioning Considerations Diagnostic Tools ArcSDE logs RDBMS logs RDBMS tracing

3 What is ArcSDE Software that spatially enables RDBMS Part of ArcGIS Server (Basic Enterprise)

4 ArcGIS Server Basic Edition Client Application ArcSDE Network

5 Why ArcSDE (More) Efficient data access Security and scalability High availability options Intelligent features simplify data management

6 Maintaining Performance with ArcSDE

7 Maintaining Performance with ArcSDE ArcSDE dependant on RDBMS Tune RDBMS per vendor recommendations ArcSDE requires some additional changes

8 Maintaining statistics ArcSDE does not maintain statistics; DBA responsible Describe data for database optimizer Critical for maintaining performance Updating statistics Depends on editing activity Before and after database compress Use Analyze Components dialog DBMS statements ArcSDE command line

9 Maintaining indexes Inserting and deleting causes fragmentation Insertions introduce non-contiguous entries Deletions cause skewed index Fragmentation leads to increased I/O ArcSDE does not coalesce or rebuild after compress Rebuild, coalesce, or defragment indexes Maintain index statistics Use DBMS statements ArcSDE command line

10 Geodatabase Toolset (GDBT) No-cost extension to ArcGIS Provides ability to: see versioning information see RDBMS statistics information use refresh data timer/fetch data timer start and stop RDBMS trace Provides a window into ArcSDE system read-only look at repository tables does not change any settings in RDBMS, ArcSDE, GDB, etc. up to ArcSDE administrator to analyze/act upon information

11 Demonstration: GDBT

12 Upgrading ArcSDE: Approach to Change Control Development The sandbox where new software, functionality, and configurations are learned. Test The clone of the production environment.. A test here means that things happen exactly as they will in production. Production Production environment. No change is applied that has not been through development and test.

13 Upgrade components Back up your database (see installation guide) Export customizations DBTUNE and SDE_SERVER_CONFIG Back up ArcSDE software Execute sdesetup -o o upgrade command or use Post Installation wizard Updates ArcSDE repository ArcSDE service Need to stop, delete, and recreate service Geodatabase

14 Client and server compatibility (9.2 and below) ArcSDE clients are backward compatible Must be same release for full geodatabase support Client and geodatabase must be at same release for direct connects Upgrade procedure Upgrade clients and servers concurrently Or upgrade clients first 9.1 Server DC Geodatabase 9.2 Client DC 9.2 Server Geodatabase DC = Direct Connects

15 Keys to Success: The Test System Test and Production are separate environments. Use Test Systems to: 1. Test all critical workflows on the new release(s). 2. Get buy-in from stakeholders. 3. Create an upgrade process which is proven to work for your system. 4. Determine the resources and timing required for the production upgrade.

16 Stories: Recovery Plan Customer More than 50 users Many ArcGIS users Many custom applications (home grown) ArcSDE, ArcGIS Server, ArcIMS, etc. Change Control Procedures A development environment No test environment. Did have recovery/rollback plan Change Control Problem After upgrades, system did not allow critical workflows. Solution Rolled back, created a test environment, resolved the problem there, then re-upgraded production.

17 Stories: Total Disaster Customer 5 main GIS users ArcSDE Mostly ArcGIS A couple of ArcIMS sites Change Control Procedures No test environment; no back-up. Change Control Problem After upgrades, system did not allow critical workflows. Solution Had to abandon in-place upgrade and rebuild on target release from scratch.

18 Stories: Un-Sung Heroes Customer 5 main ArcGIS users Lots of ArcGIS customizations and 3 rd party solutions ArcSDE, ArcIMS Change Control Procedures Upgrade efforts are motivated by business needs for specific functionality. Development, Test, and Production environments for any sort of system change (customization, 3 rd party app, upgrade, etc). Upgrade process testing. Critical workflow testing Change Control Problem Very few Solution Years of successful upgrades; valuable software quality feedback to ESRI.

19 Capacity Planning

20 High Performance Versus High Capacity Performance: : what is the client response time in single- user benchmark? Capacity: : how many concurrent users can the server support without performance degradation If single-user benchmark is not fast enough,, adding capacity (e.g., more server cores) will not solve the problem.

21 Capacity Planning: Bottlenecks Server Hardware Server O/S must be adequately configured Network Easily becomes the bottleneck DBMS Must be adequately tuned specifically for GDB ArcSDE Minor tuning required Geodatabase Design an efficient data model ArcGIS client Using proper techniques ArcObjects Customization ArcObjects delivers extremely rich set of functionality Too much functionality delivered to client

22 Planning for Capacity Gather load requirements number of concurrent users client software ArcMap, ArcGIS Explorer, web browser, etc. frequency of map refreshes requirements for server-side side geoprocessing Size of data (per map draw)

23 Capacity Planning Tool User workflows. Productivity Performance Configs Peak Users Network Traffic Selected Platforms Terminal Server Sizing Web Server Sizing Map Server Sizing Geodatabase Server Sizing

24 Demonstration: Capacity Planning tool

25 Multiuser Editing

26 Multiuser Editing: Versioning GIS Edit session lasts up to hours Need strategy to manage locks Versioning allows Many editors to simultaneously edit same data Many representations of the database (what if, proposed, as built) Starting at 9.2, non-versioned editing also supported Limited to simple features only Easy to run into dead-locks

27 Versioning: Concepts Edits stored in supporting tables (called delta tables) A for Adds (inserts) D for Deletes Updates are an add and a delete at the same time Main Street Business table (base table) SHAPE OBJECTID NAME 1 1 Main St. Supporting vector tables Feature table (F) Spatial Index table (S) Supporting delta tables Adds table (A) Deletes table (D)

28 Versioned editing implications Benefits Can edit all geodatabase data Versions can help isolate work Conflict detection and resolution between editors and versions Undo/redo ability Distributed editing and archiving Considerations Non-ESRI applications do not understand edits in delta tables Caveat: Access through multiversioned views or move to base DBMS behavior is restricted Some schema changes not allowed 3-28

29 Versioned performance Performance can degrade over time Workflow can often generate unnecessary versions Versions may just hang-out pinning state tree Example: Replica versions Response time increases with volume of states In production, number of rows can increase significantly Table growth is a function of edits Performance impacted by stale DBMS stale statistics Database might choose a sub-optimal execution plan Performance impacted by fragmented DBMS indexes 14-29

30 Multiple versions Versions can be based on projects Tasked with maintaining specific entity or project Example: Forest cut-blocks Tasks may be logically divided based on resource or location Example: Forest cut-blocks in western half of Province Supports complex projects Supports batch reconcile/ post process When are versions B and C posted to version A? When is A posted to DEFAULT? Daily? Weekly? Monthly? Upon project completion? B DEFAULT A C

31 Compressing the database 0 DEFAULT 0 49 QA 49 Land_mgmt After Parcels_mnt 54 Before 78 Zone_mnt Maintains performance Moves common rows from delta tables into base tables Reduces depth of state tree Removes rows and states no longer referenced by a lineage

32 Diagnostic Tools

33 Diagnostic Tools Many times, underlying error message displayed in client application Oracle SQL Server ArcSDE has own diagnostics (log files) Certain errors found in RDBMS logs only If attempting to troubleshoot problem (not error) use RDBMS tracing

34 ArcSDE diagnostic files ArcSDE logs stored in SDEHOME\etc giomgr_<service>.log Client connection information sde_<service>.log Application Server error messages sdedc_sql_server.log Direct Connect error messages upgrade.log Error messages for updating (database objects) Setting environment variables in dbinit.sde Error Logging SET SDEVERBOSE=TRUE (enable for short durations)

35 RDBMS Logs Oracle Alert log: Location: C:\oracle oracle\product\10.2.0\admin\sde\bdump\alert_<sid>.log Mon Jul 17 11:12: ORA-00060: Deadlock detected. More info in file c:\oracle oracle\product\10.2.0\admin\ora10\udump\ora10_or a_856.trc. SQL Server ERRORLOG: Location: C:\Program Files\Microsoft SQL Server\MSSQL.1 MSSQL.1\MSSQL\LOGLOG :01:55.54 Backup BACKUP LOG WITH TRUNCATE_ONLY or WITH NO_LOG is deprecated. The simple recovery model should be used to automatically truncate the transaction log.

36 RDBMS Tracing Database tracing used for performance tuning execution plans database time spent to fetch data Remember that database is one part of ArcSDE system Bottleneck can be in the client or network Use GDBT or similar tool to investigate system performance Can also be used for troubleshooting e.g. cannot connect to ArcSDE, no meaningful errors in logs

37 Demonstration: Error logging

38 Resources Project Center Website support.esri.com ESRI Enterprise Advantage Program ESRI Professional Services

39 ESRI Training ESRI offers a variety of training options Free or low cost Virtual Campus Training Instructor-led (classroom) training Client-site training Custom training Multiple options for developing a training plan Work with your Training Coordinator (Kerri Manorek) and account manager

40 ArcSDE Instructor-Led Classes Introduction to Multiuser Geodatabase (2 days) Data Management in the Multiuser Geodatabase (3 days) At ESRI Sacramento March ArcGIS Server Enterprise Configuration and Tuning for SQL Server (2 days) At ESRI Sacramento March ArcGIS Server Enterprise Configuration and Tuning for Oracle (2 days)

41 Thank You Katja Krivoruchko ESRI 380 New York Street Redlands, California USA Phone: Fax: