Administering your Oracle Geodatabase

Transcription

1 Esri International User Conference San Diego, California Technical Workshops July 25, 2012 Administering your Oracle Geodatabase Travis Val Jim McAbee

2 Agenda Administration Topics - System Requirements and Configuration Administration Options Client and Connection Differences - Installation, Migration and Upgrade Scenarios - Managing Performance in 11g - Some Notes on Advanced Configurations - Loading, Backup and Spatial Types Pre-Requisites - Administration of Geodatabase Intro - Basic knowledge of database terminology - Working knowledge of ArcSDE Technology Basics

3 System Requirements and Configuration

4 Major Themes 9.x/10.0 to 10.1 System Requirements - 9.3/10 to 10.1 biggest impact on client configuration 32 vs. 64 bit, instant client, etc Administration Automation - Command Line Tools: Oracle runs on Unix/Linux so they may be used ArcSDE command line tools are separate install GDB Administrative interface and new GP tools Upgrade/Migration Datatypes (should be migrating out of LONG RAW to other types) Oracle issues - Datapump imports into or , tends not to have issue (GDB_Items, etc..) ORA possible

5 ArcGIS and Oracle Versions Oracle Editions Enterprise - Standard One Standard ArcGIS Version Backwards Compatibility ArcGIS Version Oracle Versions bit only (64 bit) (32 & 64) (32 & 64) 9.3/ (64 bit) (64 bit) (32 & 64) (32 & 64) (32 &64)

6 System Requirements: OS for Oracle 11g r2 Check Operating System also.including patch level requirements

7 DBMS_PIPE, DBMS_LOCK, and other packages What are they? - DBMS_PIPE: allows sessions to communicate through memory - DBMS_LOCK: Oracle s api to create locks in DB from SQL For ArcSDE granted execute to public Remove privileges from public? - Tech Article #35408 Can the public privilege be removed from Oracle s dbms_pipe and dbms_lock packages? GRANT EXECUTE ON dbms_pipe TO public; GRANT EXECUTE ON dbms_lock TO public; GRANT EXECUTE ON dbms_lob TO public; GRANT EXECUTE ON dbms_utility TO public; GRANT EXECUTE ON dbms_sql TO public; GRANT EXECUTE ON utl_raw TO public; GRANT EXECUTE ON dbms_crypto TO sde;

8 Oracle configuration parameters Oracle Memory: - System Global Area (SGA) and Program Global Area (PGA) - Memory areas used for - Interpreting SQL statements, Fetching data and Submitting edits to database Use Automatic Shared Memory Management (ASMM) - Set SGA_TARGET through Oracle Enterprise Manager (OEM) Cursor Configuration: - Cursor: Points to rows returned by a SQL statement - Used to avoid re-parsing similar statements with bind variables - Stored in PGA and used extensively by ArcSDE - open_cursors = 2000 (or larger to avoid ORA-1000 error) - Tech Article #27024 What is an appropriate value for the Oracle init.ora parameter open_cursors

9 Controlling Storage Tablespace Management based on - Backup requirements - Activity - Size of segment SDE.DBTUNE table - Specifies configuration keyword, storage type, storage parameters To create a keyword: - Export DBTUNE table to a file, edit it and import back sdedbtune o alter

10 10.1 Clients and Connections

11 ArcGIS Geodata Management technology stack ArcGIS clients (ArcObjects/API s) Geodatabase Behaviors & Complex Features ArcSDE (ArcSDE libraries & stored procedures) Oracle (RDBMS) Spatially enabled database Database Versioning & Distributed Data Transaction Management, Spatial Types, Indexing

12 Connection Architectures Direct Connect Recommended Method ArcSDE libraries Oracle Client 1521 Geodatabase Application Server ArcSDE Libraries Database Server giomgr 5151 gsrvr Geodatabase Application Server - standalone ArcSDE Libraries giomgr Database Server 5151 gsrvr Oracle Client 1521 Geodatabase

13 10.1 and Oracle Clients Administrator Runtime Instant (now supported at 10.1) Remember ArcGIS Server now 64 bit at 10.1 ArcGIS for Desktop bit Oracle Client ArcGIS for Server bit Oracle Client 64 bit Oracle 64 bit Oracle

14 10.1 Connection Functionality Geodatabase Database Geodatabase Administrative Schema Spatial Data ST_Geometry* SDO_Geometry *ST_Geometry SQL function support only available for supported Operating Systems (where library compiled) Clients Query Layers Clients 10.1

15 Connecting at

16 10.1 Oracle Instance Connection string server name/service name (or ID) IP address of server/service name (or ID) server name:port/service name (or ID) IP address of server:port/service name (or ID) URL of server dbsrvr/orcl 10:10:10:10/orcl dbsrvr:60000/orcl 10:10:10:10:60000/orcl or [6543:e04:0:1:f587:1249:12f9 :w3ud]:60000/orcl //dbsrvr:60000/orcl server name only (if listener configured to point to default instance) If instance not listening on default port number, must use connection syntax that includes port number.

17 Demo: Setting Up Instant Client

18 10.1 Administration

19 Administrative and Maintenance Task more capabilities from desktop for GIS manager and staff Administrative task automation more options at more functionality in Desktop and Server - SDE command line tools separate install & 64 bit (need 64 bit client)

20 Inclusion of Geodatabase Toolset

21

22 Multiple Geodatabase Configuration Multiple Geodatabase for Different Tasks - At least two Geodatabases should be present typically production and test - Other reasons for multiple geodatabases - Editing and publishing (web) - Production and test/development - Different application needs - Separate versioning management - Methods for configuring multiple Geodatabases in Oracle - Multiple instances on same server - Multiple instances on different servers - Project/Schema Geodatabase Test/Dev. Production Publishing

23 Oracle Schema Geodatabase Admin An option for multiple Oracle Geodatabases Multiple Oracle Geodatabases multiple instances or within same instance Geodatabase contained in specific User Schema SDE master geodatabase present with Schema based Geodatabase in instances table Rules - a user can own only one geodatabase - that user is the geodatabase administrator and requires those database privileges - that user can only own data in that geodatabase - users can only own data in one geodatabase

24 Notes on Upgrades and Migration

25 Administration Topics : Planning for New Configuration What are some of the more advanced issues? - Long Raw to BLOB - Geodatabase Administrative schema changes - Differences between Oracle versions Default Oracle settings are a good starting point except: - Database Cache / Memory, Max number of cursors a session can use, Data Storage Estimate or measure load and storage: - Need to define a sample data, MXD and operations when possible - Oracle Enterprise Manager (OEM), Oracle Remote Diagnostic Agent (RDA)

26 Upgrade Considerations Always perform any upgrades in a test environment Backup any custom files in %SDEHOME%\etc directory - dbinit.sde - dbtune.sde Upgrade OS and Oracle first, if needed, then test Upgrade ArcSDE, then test Backup, Backup, Backup! - It is NOT a Backup unless it has been tested to make sure it works. Check existing GDB for any errors or redundancies Clean DBMS_PIPE - Values in the database pipe can cause connection problems Upgrade to 10.x requires use of ArcObjects GP tool - New Geodatabase Administrative schema changes

27 Upgrading Schema based Geodatabase Must upgrade master first and not simultaneously Must make a backup of the entire database; creating a backup of only a user's schema does not include the user-defined types and functions Stopping the ArcSDE service on the master geodatabase disables service connections to user-schema geodatabases

28 Migration Tool LONG RAW data type being deprecated at Oracle 11g Migrate Storage GP tool - Easily convert GIS datasets from a different storage format to ST_GEOMETRY in Oracle - Changes data in place, no new datasets are created - Must be the data owner to execute - Supports versioned & archived datasets

29 Upgrade Geodatabase from ArcCatalog

30 Upgrade Geodatabase Requires sde user upgrade permissions Requires a direct connection to geodatabase Pre-requisite check determines if geodatabase is upgradable - See Preparing to upgrade a geodatabase in Oracle topic for a full list of requirements # Process: Upgrade Geodatabase arcpy.upgradegdb_management(<connection file>, "PREREQUISITE_CHECK", "UPGRADE")

31 Installation, Migration, Upgrade Summary Start with recommended minimum parameters for New and monitor Existing Cursors and Memory (OPEN_CURSORS, SGA, etc..) Check connections parameter in server_config table via sdeconfig export/import command or sql Monitor usage, or ask DBA s to monitor and adjust as necessary Establish a performance baseline so future growth and changes can be measured OEM, mxdperfstat, Windows performance monitor, ASH

32 ESRI KB and Help Articles for Parameters/Upgrade FAQ: What is an appropriate value for the Oracle init.ora parameter 'open_cursors'? Error: ORA-01000: maximum open cursors exceeded HowTo: Identify a cursor leak in Oracle SQL Statements using st_geometry operators experience decreased performance in Oracle 11g Oracle Parameters help topics cle_ htm

33 Spatial Types

34 ESRI Spatial Type ST_GEOMETRY User Defined Type (UDT) used to store geographic features Allows access to spatial data through SQL functions Efficiency - Automatic geometry validation Conforms to ISO and OGC standards Available since ArcSDE 9.2

35 Using SQL with ST_GEOMETRY Needs an external dll Extproc needs to be set up for - Executing SQL commands on a ST_GEOMETRY column - For using ST_GEOMETRY operators - Definition and label queries inside ArcMap With SQL and ST you can - Create tables with a spatial attribute - Read and Analyze the spatial data - Insert, update and delete simple geometry data Geodatabase behavior not supported through SQL

36 Spatial Types and Functions Creation of Features - through SQL Analysis - Buffering Geometry - Convex Hull - Difference of Geometries - Intersection of Geometries - Symmetric Differences - Union - Minimum Distance - Aggregates Feature Information

37 ST_Geometry spatial type configuration st_shapelib library extproc.ora (11g) or listener.ora/tnsnames.ora configuration

38 Setting up the ExtProc via Listener and Tnsnames tnsnames.ora file: EXTPROC_CONNECTION_DATA = (DESCRIPTION = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) (CONNECT_DATA = (SID = PLSExtProc) (PRESENTATION = RO)) ) listener.ora file: LISTENER = (DESCRIPTION_LIST = New Entry (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST=AKLAP.esri.com)(PORT=1521)) IPC)(KEY = EXTPROC1)) ) (ADDRESS = (PROTOCOL = TCP)(HOST=AKLAP.esri.com)(PORT=1521)) ) ) ) Original Entry Key must match

39 Setting up the ExtProc (10g) listener.ora file: LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1)) (ADDRESS = (PROTOCOL = TCP)(HOST=AKLAP.esri.com)(PORT=1521)) ) ) SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = C:\oracle\product\10.2.0\db_1) (PROGRAM = extproc) (ENVS="EXTPROC_DLLS=C:\Program Files\ArcGIS\ArcSDE\ ora10gexe\bin\st_shapelib.dll") ) )

40 11g Located in ORACLE_HOME\hs\admin directory Windows EXTPROC_DLLS=ONLY:C:\\mylibraries\\st_shapelib.dll Unix/Linux SET EXTPROC_DLLS=ONLY:/user/esrilibs/libst_shapelib.so

41 Oracle ExtProc - for St_geometry Modify the listener.ora and tnsnames.ora files - Oracle Documentation - Webhelp topic - Configuring the Oracle listener to access the geodatabase with SQL - Technical articles: Error: ORA-28595: Extproc agent : Invalid DLL Path Error: ORA-06522: libsg.so: cannot open shared object file: No such file or directory Bug: ORA-06522: (path)\st_shapelib.dll: cannot open shared object file: No such file or directory Bug: Configure ST_GEOMETRY for use in Oracle 11G Release 2 on Windows How to: Check if the Oracle extproc is set up correctly for direct ST_GEOMETRY SQL queries

42 Demo: Configuring ST_Geometry

43 Few notes on SDO_Geometry Set following in SDE.DBTUNE table when working with data stored in SDO_Geometry - UNICODE_STRING set to FALSE (if VARCHAR vs. NVARCHAR is desired) - GEOMETRY_STORAGE set to SDO_Geometry - SDO_SRID (if same SRID is always used) - other parameters

44 Managing Performance and Troubleshooting

45 Managing Performance in 11g : Statistics Table statistics - The distribution and contents of rows - What the optimizer uses to make execution plans Index statistics - Information about the rows stored in IOTs, and other index metadata System statistics - Internal object statistics

46 Oracle table statistics Describe segment characteristics for optimizer and DBA - Critical for maintaining performance - Current statistics increase Oracle IO performance Update when data changes significantly: - Loading - Major editing Analyze through ArcGIS Create statistics on attributes, geometry, versioning delta tables and historical archive

47 Oracle 11g Automatic Maintenance Optimizer Statistics Collection statistics collection Segment Advisor segment reorganization SQL Tuning Advisor attempts to tune high-load SQL Tasks Automated at 11g

48 Managing Performance in 11g : Statistics Oracle automatic statistics gathering 10g vs. 11g Utilize optimizer_dynamic_sampling (KB32005) Other options - ArcCatalog and GP Tools - sdetable o update_dbms_stats t roads m estimate Temporary Tablespace - increase size for compute - consider increasing the size of your temporary table space to compute statistics rather than estimate them because it provides more accurate statistics for the optimizer.

49 11g Auto Statistics Gathering Setting Stale Percentage

50 Other Configuration Tips D-Table indexes - need to be checked for errors and redundancy in the _IDX (KB31081) and _PK (KB24925) indexes A-Table Index - Performance gains by altering the Adds table indexes (KB32184) Cache the lobs - Most lobs can be cached to reduce trips to disk ST(KB33428), Network(34485), Raster(35521) Index rebuild - Sweeper Indexes need to be rebuilt to reduce their number of blocks (KJ35409) ST_GEOMETRY Spatial index shrink - Reduce the number of blocks used by the spatial index (KB33341) Re-gather Stats - Do not build stats on the logfile tables Note: Scripts to perform these tasks are included at the end of slide deck

51 Diagnostic Files Log files are ArcSDE output about what SDE is doing Useful when encountering a problem such as - Service will not start, log file will tell you why ArcSDE logs stored in SDEHOME\etc - giomgr_<service>.log Client connection info - sde_<service>.log Application Server error messages - sdedc_oracle.log Direct Connect error messages Oracle log files, contain DBMS logged information - System alert log: alert_<sid>.log - Background process logs

52 Oracle Trace Output using tkprof The output contains SQL execution paths, row counts and wait events OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS call count cpu elapsed disk query current rows Parse Execute Fetch Misses in library cache during parse: 0 Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited Waited SQL*Net message to client SQL*Net message from client SQL*Net more data to client direct path read log file sync

53 Capturing Metrics: Session Information Opened cursors cumulative 3468 Opened cursors current 34 Check Performance Metrics Review Statistics Search Sessions Select the Correct Session sde.process_information v$session, v$process KB30394 HowTo: Enable extended Oracle tracing for an ArcGIS connection

54 Measuring and Monitoring Memory Tools available in 11g Manager Advisor Central - Memory Advisors

55 SDE Logfiles Three types - giomgr_<service>.log - giomgr process information, startup/shutdown, connections - sde_<service>.log - gsrvr process information, event, errors - Setting %SDEVERBOSE% to true increase the amount of information logged, but slows performance - sde_setup.log - Information about the setup process To enable for direct connections, set %SDEHOME%, or check the user %TEMP% directory

56 SDE DC Log Information that is similar to the sde_<service>.log file. - This file is only present during direct connections This file will automatically be generated and placed in the %SDEHOME%/etc or %TEMP% directories.

57 SDE Trace - This logfile lists all the calls that are made to the ArcSDE client API. - Not all of these calls result in a round trip to the server - To enable set %SDETRACELOC% to the path name. Files are named %SDETRACELOC%.001, %SDETRACELOC%.002

58 SDE Intercept A list of all calls between the SDE client and the SDE sever. - If using DC, will have two files that mirror each other. To enable set: - %SDEINTERCEPT% to the required values - set SDEINTERCEPT=cwrfT - %SDEINTERCEPTLOC% to the path name - Files are named %SDEINTERCEPTLOC%.001, %SDEINTERCEPTLOC%.002

59 Oracle DBMS Log Oracle Alert Log, - The Alert file is a log file that records information about internal errors and administrative activities, such as backups. 2/e17766/intro.htm#i10817 Automatically enabled, location can be found by issuing the following SQL - SQL> select value from v$parameter where name = 'background_dump_dest'

60 Oracle SQL Trace Description of every SQL statement that was executed per session - Very useful in analysis of slow running queries and operations, TKProf results much more readable Activate by altering session - As DBA user, tracing a different session: - SQL> DBMS_SYSYEM.SET_ENV(<session id>, <session serial>, 10046, 12, ) - Tracing the current session, additional privileges required: - SQL> alter session set events '10046 trace name context forever, level 12'

61 Summary Log Type 2 Tier 3 Tier Location SDE Logfiles set SDEHOME Active by Default %SDEHOME%/etc / SDE Trace SDE Intercept set SDETRACELOC set SDEINTERCEPT and set SDEINTERCEPTL OC Set SDETRACELOC set SDEINTERCEPT and set SDEINTERCEPTL OC %SDETRACELOC % %SDEINTERCEP TLOC% SDE DC Log Not Available Active %TEMP% Oracle DBMS Log Oracle SQL Trace Active Active Oracle Alert Log DBMS_SYSYEM. SET_ENV or alter session set events DBMS_SYSYEM. SET_ENV or alter session set events Oracle User Dump Dest

62 Performance and Troubleshooting Demo Spatial Clustering and Reorg

63 Advanced Configuration Topics

64 Advanced Configuration Topics Some not formally supported must test External Authentication Direct Connect - OS Authentication - Advanced Security - Encryption - Must use Direct Connect Oracle RAC clusters - Must use Direct Connect Oracle DataGuard replication - Physical stand by - Logical

65 Oracle RAC Use Direct Connect architecture Extproc configuration for ST_Geometry spatial type - Windows ST_ShapeLib.dll is dependent on dlls found in the Visual C++ Redistribution Package. - Installed by SDE setup program automatically, or can be installed using redistribution package Cursors - Cursors don t failover, reconnect either must be done or takes place depending upon application and configuration.

66 Oracle Dataguard Physical standby database commonly used Logical RDBMS replication - rare in Geodatabase configurations - Users can connect to standby database, but should be treated as read-only. - Not supported, and configuration complex due to SQL exclusion required.

67 Views, Spatial Views and Types Spatial Functions can be used in Spatial Views - Can provide dynamic feature classes on map CREATE VIEW san_berdoo_quakes_v AS SELECT a.objectid, a.location, b.name FROM quakes4 a, st_counties b WHERE b.name = 'San Bernardino' AND sde.st_intersects(a.location,b.boundary); Also Materialized Views May require registration with ArcSDE and Geodatabase - Depending upon creation method used - sdetable o create_view vs. SQL - Depending upon intended use - Display in arcmap or for sql query only, Query Layer (10.x)

68 Advanced View Customization

69 Advanced View Customization select /* FIRST_ROWS */ service_point_id, shape, ckt_id, job_id, poly_id, prev_energ_poly_id, lock_status, xfr_tag, point_type, account_priority, customer_count, replace_flag, clue_cd, pri_call_type, priority, topcall, symbol From (select g.*, gc.clue_cd, m.pri_call_type, ROW_NUMBER() OVER (PARTITION BY g.service_point_id ORDER BY m.priority) topcall, m.priority, m.symbol from gator_service_points g, gator_calls gc, master_lut m Where g.service_point_id = gc.service_pt_id And CASE when g.job_id is null then 0 else 1 end = m.assign And gc.clue_cd = m.clue and g.account_priority = m.accnt and (CASE When ROUND((sysdate - gc.last_chng_datetime) * 1440) = 0 then '0' When ROUND((sysdate - gc.last_chng_datetime) * 1440) > 0 and ROUND((sysdate - gc.last_chng_datetime) * 1440) <= 120 then '120' When ROUND((sysdate - gc.last_chng_datetime) * 1440) > 120 and ROUND((sysdate - gc.last_chng_datetime) * 1440) <= 240 then '240' When ROUND((sysdate - gc.last_chng_datetime) * 1440) > 240 then '241' Else '9999 End) = m.age_class) where topcall = 1 and pri_call_type IN ('PF','LD','LO')

70 Spatial Functions Example select c.objectid, sde.st_intersection(c.shape, b.shape) shape from contour_l c, brokentopv b where sde.st_intersects(c.shape, b.shape)=1

71 Bulk Loading Considerations Exception to the rule Versioning - Do a reconcile and post of the version to default before you load - Highly recommended that you do a compress before the load - No other editing during a load - Indexes - Bulk loads are faster without indexes - Put feature classes in load only mode (drop indexes) before loading - Do not forget to take them out again of load only mode (recreate indexes)

72 Why to export/import using RDBMS tools? Copying non-versioned data without GDB functionality Copying the complete database to a test instance Logical Rules: Target has SDE without GDB functionality Target Has SDE with GDB functionality Target Doesn t have SDE Source Has SDE Use database tools Use GDB replication Use database tools

73 What to Export? No GDB functionality : SDE.ST_GEOMETRY and SDO_GEOMETRY data, or tables without spatial data, with no GDB functionality (Domains, Relationship Classes, Feature Dataset) - Only need to export the table - Drop SDE.ST_Geometry spatial prior to export GDB functionality need the SDE schema. - The User, all related data and SDE schema - May be easier to export entire Oracle Instance

74 How to export? Data pump (Oracle), SDEExport (ArcSDE) - Oracle tools, work well with ST_Geometry starting at 9.3 Simple data only. Advanced object types have to be exported using the ArcGIS, GP tools, python or ArcObjects. KB34342 Index: Working with ST_Geometry and Oracle export/import

75 Backup Options Backup all schemas, including SDE Make sure you test your recovery scripts/backups before you need to use them. On restore, may have to complie SDE schema packages. - DBMS_UTILLiTY.COMPILE_SCHEMA( SDE ) Note: It is worthwhile to backup dbinit file and dbtune settings whenever they change.

76 Steps to evaluate UC sessions My UC Homepage > Evaluate Sessions Choose session from planner OR Search for session

77 Thank you for attending Have fun at UC2012 Open for Questions Please fill out the evaluation:

78 Display Index Grid Stats