Session: E01 E02 What Every DBA Should Know About Dynamic SQL

Transcription

1 Session: E01 E02 What Every DBA Should Know About Dynamic SQL Suresh Sane DST Systems, Inc. Mon, Oct 2, :15 am -12:15 pm Mon, Oct 2, :30 pm 2:30 pm Platform: z/os A DBA depressed with Dynamic SQL? Do you manage PeopleSoft, SAP or Siebel? Want to know how caching really works? What tuning knobs do you have at the system and application level? Are there holes in your implementation of security? How do you prevent run-away queries before they create havoc? We will cover these and a wide array of challenges you must face when dealing with Dynamic SQL. This session will empower you with all the tools you need to tame the Dynamic SQL beast. In this session, which includes the latest V8 information, you will learn not only how to cope with Dynamic SQL, but how to exploit it fully. 1

2 Suresh Sane About the Instructor Co-author-IBM Redbooks SG , May 2002 SG , March 2004 SG , July 2006 Educational seminars IDUG Orlando, Denver and Toronto Various presentations at IDUG North America and IDUG Asia Pacific IDUG Solutions Journal article Winter 2000 Numerous DB2 courses at various locations IBM Certified Solutions Expert for both platforms for Application Development and Database Administration 2 Suresh Sane works as a Database Architect at DST Systems in Kansas City, MO. He provides strategic direction for deployment of database technology at DST and has overall responsibility for the DB2 curriculum for about 1,500 technical associates. Contact Information: sssane@dstsystems.com Suresh Sane DST Systems, Inc Broadway Kansas City, MO USA (816)

3 About DST Systems DST is a leading provider of computer software solutions and services. Highlights Financial Services Mutual Fund shareholder recordkeeping Corporate Securities & Processing 100M+ accounts 28M accounts Automated Work Distributor 85K workstations Output Solutions Customer Management Literature distribution Software, billing, output for cable, telephone, satellite and utilities 2.1B packages annually 40% of US cable households 3 If you have ever invested in a mutual fund, have had a prescription filled, or are a cable or satellite television subscriber, you may have already had dealings with our company. DST Systems, Inc. is a publicly traded company (NYSE: DST) with headquarters in Kansas City, MO. Founded in 1969, it employs about 12,000 associates domestically and internationally. The three operating segments - Financial Services, Output Solutions and Customer Management - are further enhanced by DST s advanced technology and e-commerce solutions. 3

4 Session Outline What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 4 We will first build a business case for dynamic SQL and introduce various challenges you must face. Then, we will deal with the DBA tuning knobs to make dynamic SQL hum. 4

5 Disclaimer Content Warning! While every effort has been made to provide a balanced view, this presentation contains facts and opinions. Your mileage may vary! 5 While the opinions are based on research, some personal bias most likely exists. Your shop s environment is different and the suggested actions may not be appropriate for your needs. 5

6 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 6 Let us begin with an elementary discussion of what dynamic SQL is. It turns out that it is not that easy to define. 6

7 What is Dynamic SQL? Coding View Static SQL SQL statement or SQL statement string is hard-coded in an application program Text may be stored in a program variable Text may be coded in a DECLARE CURSOR statement Statement is visible in the program When SQL changes, program must be compiled and linked Dynamic SQL Statement text is provided or assembled at execution time User enters statement (e.g. QMF, SPUFI) User marks fields on a screen Statements are built and prepared at runtime Allows many different statements to be executed without having to compile and link-edit 7 Looking at dynamic SQL from a coding perspective. 7

8 What is Dynamic SQL? Execution View Static execution mode Pre-compiler generates DBRM DBRM is bound into an executable package or plan Statement strings and related access paths Package or plan is executed without additional preparation Dynamic execution mode SQL statements not bound before execution, but prepared at runtime No plan or package exists All statements executed by callable interfaces (e.g. ODBC, JDBC, REXX) are considered dynamic SQL and are prepared by DB2 at execution time Even if hardcoded in the program 8 And looking at it from an execution perspective. 8

9 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 9 We now present a business situation where the use of dynamic SQL is justified along with some of the design considerations and challenges that accompany it. 9

10 When is Dynamic SQL Appropriate? Example Training record keeping system where the user can retrieve training history based on at least one, or a combination of, the following pieces of information: Last name First name Employee ID Department number Supervisor ID Course ID Date range 10 A simple example of a generic search screen where a user may enter any or all of the search criteria. 10

11 When is Dynamic SQL Appropriate? Alternatives Static SQL with cursors for each of 127 combinations (without considering sort order) 7 with exactly one value entered 21 with exactly two values entered etc. Requires: 127 DECLAREs 127 OPENs 127 FETCHes 127 CLOSEs 11 (7 variations with just 1 criterion, 21 with 2, 35 with 3, 35 with 4, 21 with 5, 7 with 6 and 1 with all 7). In general, n optional search fields lead to about 2**n (two the power n) variations a number that grows exponentially, not linearly. With 16 search criteria, we are up to 32K! If you take the various ordering of data that may be requested, the number increases even more dramatically. Only a small number of optional fields, therefore, leads to an astronomical number of variations that would need to be managed. 11

12 When is Dynamic SQL Appropriate? Alternatives Static SQL with generic search criteria SELECT... FROM TRAINING_HISTORY WHERE LASTNAME BETWEEN :LO-LAST AND :HI-LAST AND FIRSTNAME BETWEEN :LO-FIRST AND :HI-FIRST AND EMPLOYEE_ID BETWEEN :LO-EMPID AND :HI-EMPID AND DEPT_NBR BETWEEN :LO-DEPT AND :HI-DEPT AND SUPERVISOR_ID BETWEEN :LO-SUPV AND :HI-SUPV AND COURSE_ID BETWEEN :LO-COURSE AND :HI-COURSE AND CLASS_DATE BETWEEN :LO-DATE AND :HI-DATE 12 Sequencing the data needs to be handled with another process, or not handled at all. Host variables are populated based on what the user enters. If the user enters a specific department number, then :LO-DEPT and :HI-DEPT both contain that value, thereby making the predicate an equal to condition. All of the other host variables would be set to permissible minimum and maximum values for the respective data type and column size. Typically, performance of such a query is poor, except in cases where the user specifies the columns that correspond to the index chose by the optimizer (at bind time!) or when REOPT(VARS) is specified. 12

13 When is Dynamic SQL Appropriate? Mix and Match? Combine static and dynamic SQL in the same program Consider coding static SQL for the most common SQL requests Default to dynamic SQL for all other unpredictable requests Best of both worlds!! 13 Using the rule, we can get the performance of static SQL without giving up the flexibility of dynamic SQL. 13

14 When is Dynamic SQL Appropriate? A Few Things to Consider Flexibility Complexity Performance Non-uniform distribution statistics (NUDs) Security Access path determination Preventing a runaway query Impact analysis REOPT(VARS)/REOPT(ONCE) (V8) Catalog statistics Auditing Portability 14 Some of the issues we need to consider. We will discuss each of these in the following slides. 14

15 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 15 Let us step into the developer s shoes for a brief moment. We will use a simple COBOL example to illustrate. This session is not about coding, but you will likely deal with programmers who turn to you for help! 15

16 A Glimpse of Dynamic SQL Development Variations of Dynamic SQL Statements Non-SELECT without parameter markers Non-SELECT with parameter markers Fixed-list SELECT without parameter markers Fixed-list SELECT with parameter markers Varying-list SELECT without parameter markers Varying-list SELECT with parameter markers 16 From the perspective of developing applications containing dynamic SQL, you need to be aware of three basic variations: non-select statements (e.g. INSERT, UPDATE, DELETE), fixed-list SELECT statements (i.e. the list of selected columns is known) and varying-list SELECT statements (i.e. the list of selected columns is not known). For each basic variation, the presence or absence of parameter markers (loosely corresponding to host variables in static SQL) requires some additional steps, leading to the 6 variations shown above. 16

17 A Glimpse of Dynamic SQL Development SELECT LASTNAME,FIRSTNME,SALARY FROM DSN8710.EMP WHERE LASTNAME LIKE? AND SALARY >? Variable number of Columns selected Variable number of predicates 17 Let s follow the programming tasks involved in preparing a dynamic SQL statement shown above with an arbitrary number of columns selected (3 in this case) and an arbitrary number of parameter markers (2 in this case). This is the most complex of the 6 variations discussed before. 17

18 A Glimpse of Dynamic SQL Development Varying-list SELECT with Parameter Markers Include SQLCA Build SQL statement Yes Prepare into SQLDA Combine prepare and describe? Acquire storage to hold 1 row Place storage addresses in SQLDA Populate host variables corresponding to parameter markers Open cursor using host variables No Prepare statement Describe statement Include SQLDA to receive values Declare cursor Fetch cursor End of cursor? Yes Close cursor No Process row Handle errors 18 The steps involved in making this happen. 18

19 A Glimpse of Dynamic SQL Development PARM-SQLDA as populated by the program SQLDAID SQLDABC SQLN SQLD SQLDAID Varltype, Not Null *100 Length Max Actual SQLTYPE SQLLEN SQLDATA SQLLIND SQLNAMEL SQLNAMEC Pointer Pointer N/A N/A Pointer Pointer N/A N/A Decimal Nullable 256 * (precision 9, scale 2) 19 SQLDABC: This must be 16 bytes (header) + 44 bytes for each occurrence based on SQLN. SQLN: Max number of parameter markers in this case, we assume 100. SQLD: Actual number of parameter markers in this case, 2. SQLTYPE: See appendix C of SQL Reference for details. SQLLEN: Length for decimal columns it must contain the precision and scale as shown. SQLDATA: Pointer to the host variable SQLLIND: Pointer to the indicator variable 19

20 A Glimpse of Dynamic SQL Development RECEIVING-SQLDA as populated by DB2 SQLDAID SQLDABC SQLN SQLD SQLDAID SQLTYPE SQLLEN SQLDATA SQLLIND SQLNAMEL SQLNAMEC Pointer Pointer 8 LASTNAME Pointer Pointer 8 FIRSTNME Pointer Pointer 6 SALARY 20 In addition to the variables populated as shown on the last slide, SQLNAMEL and SQLNAMEC contain the column length and name. SQLDATA and SQLIND now must point to where you want to fetch the data. 20

21 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 21 The most important consideration, from a performance perspective, is caching. We will discuss in detail the different types of caching, how you implement them and what tuning knobs you turn to benefit from it fully. 21

22 Dynamic Statement Caching Confused by the Caching Conundrum? 22 This section will cut through the hype and present a clear explanation of how caching works. 22

23 Dynamic Statement Caching Levels of Caching Four different levels of dynamic statement caching: No caching Local caching Global caching Full caching (= Local + Global) 23 Four levels of caching. A vast majority of shops will likely use global caching (global only, not local) for internally developed applications but full caching if using one of the ERP products like PeopleSoft. 23

24 Dynamic Statement Caching No Caching Program A PREPARE S EXECUTE S SQLCODE=0 SQLCODE=0 Full Thread A Prepare Prepared statement S EXECUTE S Program B PREPARE S EXECUTE S COMMIT EXECUTE S PREPARE S EXECUTE S SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE= -514 or -518 SQLCODE=0 SQLCODE=0 Full Thread B Prepare Prepared statement S Invalidating prepared statements Prepared statement S Full Prepare 24 No sharing of statements across threads (each must do full prepare before using). Within a thread, a statement invalidated after a commit and must be prepared again. SQLCODE -514 (SQLSTATE 26501) for a cursor. SQLCODE -518 (SQLSTATE 07003) for a non-cursor THE CURSOR cursor-name IS NOT IN A PREPARED STATE -518 THE EXECUTE STATEMENT DOES NOT IDENTIFY A VALID PREPARED STATEMENT 24

25 Dynamic Statement Caching Local Caching Program A PREPARE S SQLCODE=0 Full Thread A Prepare Prepared statement S EXECUTE S EXECUTE S Program B PREPARE S EXECUTE S COMMIT EXECUTE S SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 Full Thread B Prepare Prepared statement S Invalidating prepared statements Statement text preserved Prepared statement S Full Prepare (Implicit) 25 The only real benefit is the ability to issue an EXECUTE without issuing a PREPARE (DB2 will take care of this for you) you still pay the price. Cache allocated in storage of each thread in DBM1 address space. Controlled by KEEPDYNAMIC bind option. A thread can issue a prepare once and use it across commits. After commit, implicit prepare by DB2. Application must be able to recognize that execute can get errors that normally occur during prepare. For DRDA clients, KEEPDYNAMIC(YES) causes early close of cursor. 25

26 Dynamic Statement Caching Global Caching Program A PREPARE S EXECUTE S EXECUTE S Program B PREPARE S EXECUTE S COMMIT EXECUTE S PREPARE S EXECUTE S SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE=0 SQLCODE= -514 or -518 SQLCODE=0 SQLCODE=0 Thread A Prepared statement S Thread B Prepared statement S Invalidating prepared statements Prepared statement S Full Prepare Short Prepare Short Prepare EDM Pool SKDS S 26 The main benefit comes from the ability to reuse a statement that has been prepared by another thread. Cache within EDM Pool of DBM1 address space. Activated by CACHEDYN=YES zparm. Can be in dataspace via zparm EDMDSPAC. Skeleton copy of prepared statement in cache. Unless local caching is active, directly issuing an execute is not possible must issue prepare first. 26

27 Dynamic Statement Caching Full Caching Program A PREPARE S SQLCODE=0 EXECUTE S SQLCODE=0 EXECUTE S SQLCODE=0 Program B PREPARE S SQLCODE=0 EXECUTE S SQLCODE=0 COMMIT SQLCODE=0 EXECUTE S SQLCODE=0 Thread A Prepared statement S Thread B Prepared statement S No effect on prepared statements Full Prepare Short Prepare Avoided Prepare EDM Pool SKDS S 27 This combines the two benefits (ability to issue an EXECUTE without PREPARE small impact) + (the ability reuse the statement prepared by another thread big impact). Activated by KEEPDYNAMIC(YES) bind parameter and zparm MAXKEEPD > 0 and zparm CACHEDYN=YES. No need to issue a prepare after a commit. Local cache uses FIFO algorithm. Global cache uses LRU algorithm. 27

28 Dynamic Statement Caching Different Prepare Types When using dynamic statement caching, four different types of prepare operations can take place: Full prepare No skeleton copy in global cache Short prepare Skeleton copy copied from global to local cache Avoided prepare Bypass need of short or full prepare since statement text and prepared statement found in local cache Implicit prepare Application using local caching issues execute, but prepare cannot be avoided (no global cache or gone from global cache or gone from local cache) DB2 issues prepare on behalf of the application (short or full) 28 Relative costs: Full = 100; Short = 1; Avoided = 0. Remember what IBM s Prem Mehra once said? The most efficient SQL is one that is not executed at all!. The avoided prepare is just that! 28

29 Dynamic Statement Caching Restrictions for Reuse Dynamic statement cache is used only when: Statement given to DB2 is 100% the same At prepare time, same ATTRIBUTES have been used Statement does not come from private protocol source Auth ID used to prepare statement and cache it is the same auth ID that retrieves the statement from the cache for reuse Plan or package is bound with identical parameters Special registers (e.g. CURRENT DEGREE, CURRENT RULES, etc.) are the same at prepare time Declared cursor characteristics match Parser options (e.g. APOST vs. QUOTE) are the same This matters when DYNAMICRULES(RUN) and (BIND) are run on the same system Parallelism is disabled by RLF for thread that cached the statement, and parallelism is disabled for thread that is searching the cache 29 Most of these do not change from one execution to the next, but two that are worth emphasizing are the text and the auth ID. The text must match exactly (trailing blanks are not removed!). However, beware of program like DSNTEP2 that strip trailing blanks. In itself, this is good, but if you purposely add a few blanks to force a new access path, you will not get one since the statement will end up with an identical text. When using secondary auth IDs, the auth ID seen by DB2 is usually a common ID. But remember that a query run by user A that is later run by user B will not benefit from caching if the auth IDs are different. 29

30 Dynamic Statement Caching Invalidation Cached statements are invalidated or removed from either global cache or local cache. Invalidated from global cache when: No free pages are available in the EDM pool DROP, ALTER, or REVOKE is executed for object(s) on which the plan depends RUNSTATS is executed for object(s) on which the plan depends 30 Also see section on V8 for latest enhancements a new option of RUNSTATS. 30

31 Dynamic Statement Caching Invalidation Invalidated from local cache when: MAXKEEPD is exceeded Statement ID is reused by a prepare operation DROP, ALTER, or REVOKE is executed for object(s) on which the plan depends ROLLBACK and re-signon are executed RUNSTATS is executed for object(s) on which the plan depends 31 The issue of reusing the statement ID seems trivial but is easy to overlook. For example, an application that always prepares a statement as S1 will never benefit from local caching. Each prepare will destroy the cache from the previous execution. 31

32 Dynamic Statement Caching Impact of Bind Options Bind options that behave differently when dynamic statement cache is active: REOPT(VARS) applications can run in CACHEDYN=YES systems, but their statements are not cached. REOPT(VARS) is incompatible with KEEPDYNAMIC(YES) and produces a bind error. RELEASE(DEALLOCATE) does not apply to dynamic SQL statements on systems where statement caching is not used, or only partially used. Locks acquired by dynamic SQL statements are released at commit. RELEASE(DEALLOCATE) applies to dynamic SQL statements if full caching is used. 32 Important considerations for bind options. In general REOPT(VARS) and caching are in conflict with each other (i.e. how can you have a custom-made access path and still not pay to have it built just for you?). 32

33 Dynamic Statement Caching Caching in a Distributed Environment If no prepare after commit, Package at the DB2 for z/os server must be bound with KEEPDYNAMIC(YES). If both requestor and server are DB2 for z/os, KEEPDYNAMIC value of the requestor is honored. Possible KEEPDYNAMIC performance implications for DRDA clients using WITH HOLD cursors: KEEPDYNAMIC(NO) A separate network message is required when the DRDA client issues a CLOSE for a WITH HOLD cursor. KEEPDYNAMIC(YES) The DB2 for z/os server automatically closes the cursor when SQLCODE +100 is detected. When a distributed thread has touched any package that is bound with KEEPDYNAMIC(YES), the thread cannot become inactive. 33 Impact of KEEPDYNAMIC in a distributed environment. 33

34 Dynamic Statement Caching Dynamic Statement Caching Summary CACHEDYN NO YES KEEPDYNAMIC NO YES No skeletons cached in EDMP Only full prepares No prepared statements kept across commits (note 1) No statement strings kept across commits (note 3) No skeletons cached in EDMP Only full prepares No prepared statements kept across commits (note 1) Stmt strings kept across commits implicit prepares Skeletons cached in EDMP 1st prepare full; others short (note 2) No prepared statements across commits (note 1) No statement strings kept across commits (note 3) Skeletons cached in EDMP 1st prepare full; others short (note 2) Prepared stmts across commits Stmt avoided strings prepares kept across (note 4) commits implicit prepares 34 The differences show up as: Is skeleton cached? What is the type of PREPARE? Is the prepared statement kept across a commit? Is the statement string kept across a commit? Note 1: unless a cursor WITH HOLD is open Note 2: unless invalidated or flushed out due to LRU Note 3: SQLCODES 514 or 518 result Note 4: assuming MAXKEEPD > 0 34

35 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 35 In this section we discuss some important ZPARMS to manage the system resources including the dynamic statement cache and the resource limit facility (RLF) - which is covered later. 35

36 Controlling System Resources Statement Caching Controlling local cache storage Amount of local storage used by locally cached dynamic statements can be controlled by the following DSNZPARMs: MAXKEEPD Applies to full caching FIFO managed Is approximate (actual number may exceed) CONSTOR Provides management of local thread DBM1 storage MINSTOR Provides management of fragmentation of local thread DBM1 storage Performance vs. memory utilization 36 Parameters to manage size and operation of the local cache: MAXKEEPD: Local cache can take up approx. 10 KB per statement per thread. Applies to statements kept only when full caching is in effect. For applications with infrequent commits, actual max can be greater. Default 5,000; max 65K Value = 0 does not mean no local caching. CONSTOR: When set to YES, DB2 looks at each thread at commit to see if contraction can occur based on either of: # of commits > 50 since last contraction Total thread storage > 2 MB MINSTOR: Controls fragmentation, consider setting to YES if needed. 36

37 Controlling System Resources Statement Caching Global cache storage in EDM pool which contains: Cursor tables (CTs), package tables (PTs) and database descriptors (DBDs) in use Skeleton cursor tables (SKCTs) for the most frequently used applications Skeleton package tables (SKPTs) for the most commonly used applications Database descriptors (DBDs) referred by these applications Cache for plans that use cache Those bound with CACHESIZE > 0 Skeletons (SKDS) for the most frequently used dynamic SQL statements 37 The EDM pool contains several other components, and a strategy to manage the EDM pool must also focus on these aspects (which should be part of normal monitoring and tuning without dynamic SQL). 37

38 Controlling System Resources Statement Caching Controlling global cache storage (EDM pool) Sizing the global statement cache See notes for estimation formula Using dataspace for global statement caching Consider only if DBM1 is close to 2GB limit (thru V7) See section on V8 no longer in V8 Controlling the DBD size Keep it a manageable size EDMBFIT Best fit vs. best performance 38 Cachesize can be estimated as: # of plans with dynamic SQL * average size of prepared stmt * number of stmts used repeatedly + Average size of prepared stmt * # of ad hoc stmts used only once Here, the average size of prepared stmt is generally 1.8 KB for the first table and 0.2 KB for each additional table. (Use these as a rule-of-thumb only.) Consider dataspaces only if DBM1 is close to 2 GB limit. DBD size based on # of tables (e.g. 50 tables with 200 columns = 150 KB). Run REORG and MODIFY frequently. EDMBFIT: Choice of first fit (better performance) or best fit (better storage optimization). 38

39 Controlling System Resources Other DSNZPARMs Resource Limit Facility (RLF) Parameter defaults to control the RLF at installation time: RLF Auto start (Yes or No) RLFAUTH Auth ID RLFTBL Table suffix (DSNRLSTnn) RLFERR Default action for local connections (nolimit, norun, nn) RLFERRD Same for remote connections 39 Parameters that control the start up options for the Resource Limit Facility (RLF). See the section on Governing for details on the structure and contents of the RLF. 39

40 Controlling System Resources Other DSNZPARMs Default DSNZPARM for DYNAMICRULES: When DYNRULES(YES), the following defaults are affected: DECIMAL POINT STRING DELIMITER SQL STRING DELIMITER MIXED DATA DECIMAL ARITHMETIC When DYNRULES(NO): Values of precompiler options are used 40 Remember that setting the value of the DYNRULES ZPARM to YES impacts the programming options. These may be different from the options specified in the precompiler. 40

41 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 41 In this section, we focus on those aspects of concurrency that are related to dynamic SQL. These span various issues such as caching and the impact of isolation levels on ambiguous cursors. 41

42 DBD Locks Locking and Concurrency Without global statement caching DML SQL statements acquire an S-lock on the database descriptor (DBD) With global statement caching No database descriptor (DBD) lock is acquired for short PREPAREs 42 Impact of caching on DBD locks. Not only does caching improve performance, it also improves the concurrency by not requiring an S-lock on the DBD. 42

43 Locking and Concurrency Ambiguous Cursors Definition A cursor is considered ambiguous when DB2 cannot determine if the cursor is read-only or updateable. The presence of any dynamic SQL statement results in an ambiguous cursor because DB2 cannot detect subsequent logic in the program. CURRENTDATA Local access Remote access 43 Local: When CURRENTDATA(YES), current row cannot be changed; it is ignored when parallelism is active Remote: impacts isolation levels RR, RS, CS when CURRENTDATA(NO) block fetching is on and current row can be changed; UR always uses block fetching. 43

44 Locking and Concurrency Lock Avoidance Factors Cursor Type ISOLATION and CURRENTDATA UR and N/A READ N/A UPDATE --- AMBIGUOUS --- CS and YES CS and NO RS and N/A RR and N/A Avoid locks on returned data Avoid locks on rejected data 44 RETURNED data = data qualified (row selected) REJECTED data = data not qualified (row not selected) This discussion applies to static as well as dynamic SQL. 44

45 Locking and Concurrency RELEASE(DEALLOCATE) Normally applies to static SQL only Applies to dynamic SQL if global caching is used Demotion of a gross lock occurs if: DB2 acquires lock due to lock escalation Application issues a LOCK TABLE statement Application issues a mass delete Demotion cannot occur if: Tablespace LOCKSIZE is tablespace Segmented tablespace LOCKSIZE is table Gross locks acquired due to isolation level RR Gross lock acquired on tablespace started for read-only Lock was held for static SQL 45 Locks on tablespaces, tables and partitions (called gross locks) generally apply to static SQL only, but they also apply to dynamic SQL when global caching is used. DB2 tries to demote such locks at commit but cannot in some cases, as shown above. Note: RELEASE(DEALLOCATE) applies to all packages containing static and dynamic statements regardless of KEEPDYNAMIC value. So, a copy of the package table (PT) is not released from the EDM pool and must be reloaded each time after a commit. This is generally not an issue, but can cause performance issues when using AUTOCOMMIT(YES) which commits each transaction in a JDBC/ODBC environment. 45

46 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security What is new with V8? Summary 46 In this section we will discuss how you can control the resources consumed by dynamic SQL. By using the Resource Limit Facility (RLF) wisely, you can prevent runaway queries and be able to do so in a proactive manner if you choose. We will cover the structure of the RLF and provide realistic business examples. 46

47 Governing RLF (Resource Limit Facility) AUTHID Auth ID PLANNAME Name of plan ASUTIME Processor service units LUNAME Name of originating location RLFFUNC How row is used RLFBIND Bind permitted? RLFCOLLN Collection ID RLFPKG Program name RLFASUERR Error threshold (category A only) RLFAUWARN Warning threshold (category A only) RLF_CATEGORY_B Category (action for category B) Activate by issuing: START RLIMIT ID = nn 47 Since this table contains a variety of different things in it, it is hard to understand its purpose without looking at some examples (see next few slides). Note that LUNAME value of blank does not apply to all locations but local requests only. A value of PUBLIC applies to all locations. ASUTIME does not need to be changed after an upgrade since a more powerful processor means less CPU seconds but the same amount of work. 47

48 Governing Reactive RLFFUNC ASUTIME blank 2 null nnnnn 0 or <0 Reactive governing of S/I/U/D statements by plan name Reactive governing of S/I/U/D statements by package or collection No limit Service units Disable all dynamic S/I/U/D statements 48 Other values of RLFFUNC: 1 Reactive governing of BIND 3 Disable query I/O parallelism 4 Disable CPU parallelism 5 Disable sysplex parallelism 48

49 Governing Predictive RLFFUNC RLFASUERR RLFASUWARN 6 7 nnnnn null 0 or <0 nnnnn null 0 or <0 Predictive governing of S/I/U/D statements by plan name Predictive governing of S/I/U/D statements by package or collection Service units threshold for category A No error threshold All dynamic S/I/U/D statements receive SQLCODE 495 (SQLSTATE 57051) Service units threshold for category A No warning threshold All dynamic S/I/U/D statements receive SQLCODE +495 (SQLSTATE 01616) 49 For category B - RLF_CATEGORY_B values: blank = allow (default) Y = allow N = disallow, return 495 W = prepare and issue +495; then application decides 49

50 Governing Predictive Prepare Cost Cat=A? N Y SQLCODE -495 Y Cost > W N RLF_CAT_B RLFASUERR? N Y or Blank SQLCODE +495 (Appl Decides) Y Cost > RLFASUWARN? SQLCODE +495 Appl Decides Execute SQLCODE The cost category is A when DB2 can estimate the cost accurately (statement contains literals only) and B when it cannot (host variables, no RUNSTATS, UDFs, triggers or referential integrity). 50

51 Governing Combining Predictive and Reactive Prepare Cost Cat=A? N SQLCODE -495 Y Y Cost > RLFASUERR? W RLF_CAT_B Blank or Y N SQLCODE +495 Appl Continues Y Cost > RLFASUWARN? SQLCODE +495 Appl Cont s Execute SQLCODE -495 SQLCODE -905 Y Cost > ASUTIME? N Continue 51 This behaves just like predictive governing discussed before with the additional check during execution to see that the threshold is not exceeded. 51

52 Governing Qualifying Rows in the RLF DB2 attempts to find the closest match to find applicable limit Search order: Exact match Auth ID Plan name, or collection ID + package name LU name No match If none or error, use RLFASUERR value. 52 If you plan to use multiple rows, it is strongly recommended that you use unique values for ASUTIME (e.g. instead of 3 rows with a value of (say) 800 make them unique like 800, 801, 802, 803 etc). This helps pinpoint which row caused the action to be taken. Also note that wildcarding is not permitted (e.g. entering PAY% in the program name will not govern all payroll programs like PAYPGM01, PAYPGM02 etc.). In the examples that follow, we will assume we are using packages. We will also assume a value of 20,000 as the number of service units per CPU second. (This value depends on your processor speed. See z/os V1R1.0 MVS Initialization and Tuning Guide, SA for details, or just ask your SYSPROG!) Recommendation: Do not attempt to fine tune the ASUTIME value. The estimate can vary widely, and it should be used only to control a true runaway query. 52

53 Governing Case 1 Predictive Governing Suresh - When executing any package in the HR collection, produce warning if estimated cost exceeds 5 CPU seconds. Do not allow to run at all if estimated cost exceeds 8 seconds. (Do not allow any host variables.) 1 SURESH - 7 HR N # AUTHID PLANNAME RLFPKG ASUTIME LUNAME RLFFUNC RLFCOLLN RLFASUERR RLFASUWARN RLF_CATEGORY_B 53 This table shows how the business requirements specified can be implemented using the Resource Limit Facility (RLF). Suresh, when executing any program in the HR collection, gets: SQLCODE 0 if the estimated cost is less than service units SQLCODE +495 (SQLSTATE 01616) if the estimated cost exceeds service units but less than service units SQLCODE 495 (SQLSTATE 57051) if the estimated cost exceeds service units (Note that any host variables will cause a -495 since RLF_CATEGORY_B is set to N.) 53

54 Governing Case 2 Reactive Governing Kathy - When executing package PGM1 in the PAY collection, no involvement before execution. If actual execution takes more than 10 CPU seconds, execution should be terminated. 2 KATHY PAY PGM1 - - # AUTHID PLANNAME RLFPKG RLFASUERR ASUTIME RLFASUWARN LUNAME RLFFUNC RLF_CATEGORY_B RLFCOLLN 54 This table shows how the business requirements specified can be implemented using the Resource Limit Facility (RLF). Kathy, when executing program PGM1 in the PAY collection, gets: SQLCODE 0 if the actual cost is less than service units SQLCODE -905 (SQLSTATE 57014) if the actual cost exceeds service units 54

55 Governing Case 3 Combined proactive + reactive Dave - When executing package PGM2 in the GL collection, produce warning if estimated cost exceeds 5 CPU seconds. Do allow to run at all if estimated cost exceeds 8 seconds. (Allow use of host variables.) If actual execution take more than 10 seconds, execution should be terminated. 3 DAVE - 7 GL PGM W 4 DAVE GL PGM2 - - # AUTHID PLANNAME RLFPKG ASUTIME RLFASUERR LUNAME RLFASUWARN RLFFUNC RLF_CATEGORY_B RLFCOLLN 55 This table shows how the business requirements specified can be implemented using the Resource Limit Facility (RLF). Dave, when executing program PGM2 in the GL collection, gets: SQLCODE 0 if the estimated cost is less than service units SQLCODE +495 (SQLSTATE 01616) if the estimated cost exceeds service units but less than service units SQLCODE 495 (SQLSTATE 57051) if the estimated cost exceeds service units (Note that any host variables will cause a +495 since RLF_CATEGORY_B is set to W.) SQLCODE 905 (SQLSTATE 57014) if the actual cost exceeded regardless of how it was executed (no warning, warning followed by application deciding to continue) 55

56 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 56 In this section, we will discuss how you can obtain information about the access path of an SQL statement and what metrics are important to collect and analyze to make sure that the dynamic SQL is running properly. This includes monitoring how well caching is working. We will keep the discussion generic in nature without referring to a specific monitor from a specific vendor. Almost all monitors that I have worked with provide the information you need only the format is different. 56

57 EXPLAINing, Tracing and Monitoring Using Visual Explain 57 Input can be free-form SQL, package or plan as well as a cached dynamic SQL statement (V8). When explain from cache, it provides the access path currently in use (i.e. it does not re-explain the statement which could generate a different access path). 57

58 EXPLAINing, Tracing and Monitoring Monitoring Thread Detail Information relevant to dynamic SQL Total DML (SELECT, INSERT, UPDATE, DELETE) Prepare (QXPREP number of explicit PREPARE requests) Describe Describe table Open Fetch Close Note: EXECUTE IMMEDIATE is not counted towards total PREPAREs 58 At the thread level, the important counters to note are the number of PREPAREs and DESCRIBEs issued. Note that even though an EXECUTE IMMEDIATE issues a PREPARE under the covers, it is not reported under the PREPARE count. 58

59 EXPLAINing, Tracing and Monitoring Monitoring Thread Detail for Caching Counters related to caching include: QXSTNFND Not found in cache QXSTFND Found in cache QXSTIPRP Implicit PREPAREs QXSTNPRP PREPAREs avoided QXSTDEXP Statement invalid (MAXKEEPD) QXSTDINV Statement invalid (DDL) 59 Some of the counters to look out for. Ideally a low number should be present for not found in cache and the number of invalidations should be zero. 59

60 EXPLAINing, Tracing and Monitoring Monitoring EDM Pool Ratios to monitor DBD hit ratio (%) CT hit ratio (%) PT hit ratio (%) Counters to monitor QISEDSC Pages used for dynamic SQL cache QISEDPGE Pages in EDM pool data space QISEDFRE Free pages in data space free chain QISEDFAL Failures due to data space full 60 The ratios measure the percentage of times a request could be specified using the EDM Pool. Ideally all ratios should be close to 100%, but a realistic target is about 95% to 99%. Failures should ideally be zero. 60

61 EXPLAINing, Tracing and Monitoring Monitoring SQL Number of PREPARE requests Number of full PREPAREs Number of short PREPAREs Global cache hit ratio (%) = Number of short PREPAREs Number of short PREPAREs + Number of full PREPAREs x The global cache hit ratio ideally close to 100% (95%-99% is realistic goal). Besides the size of the cache, excessive use of literals or a security scheme that uses individual auth IDs could be the reason for a low number (see section on dynamic statement caching). 61

62 EXPLAINing, Tracing and Monitoring Monitoring SQL Implicit PREPAREs Due to local caching Avoided PREPAREs Due to full caching Statement invalid (MAXKEEPD) Number of times discarded due to max Statement invalid (DDL) Number of times discarded due to DDL or RUNSTATS Local cache hit ratio (%) = Number of avoided PREPAREs x 100 Number of avoided PREPAREs + Number of implicit PREPAREs 62 A similar number for local caching. The application must be cognizant of local caching and must not issue a second prepare. If it does, this ratio is likely to be low. 62

63 Monitoring sample - EZDBA Panel Hyperlink to Dynamic SQL Statement Cache Hyperlink to EDM Pool Application, or; Direct hyperlink to Dynamic SQL Cache Application 63

64 Monitoring sample - EDM Pool Status View Hyperlink to Dynamic SQL Statement Cache Application PF8 Scrolls to next detail display on the EDM Pool 64

65 Monitoring sample - EDM Pool Status View Page 2 65

66 Monitoring sample - Dynamic SQL Cache Details Got here from hyperlink on STEMPD for Dynamic SQL Cache Hyperlink to Statement Cache Analysis PF8 to second page of detail 66

67 Monitoring sample - Dynamic SQL Cache Details Page 2 67

68 Monitoring sample - Dynamic SQL Cache Menu Got here from hyperlink on SQL Cache Statement Analysis on STCACHED Hyperlink to Statements In the Cache PF8 to second page of detail 68

69 Monitoring sample - Dynamic SQL Cache Statement Summary Hyperlink on Time Cached takes you to detail information on this statement in the cache Two clicks gets you to this panel which lists all SQL statements currently in the cache 69

70 Dynamic Monitoring SQL sample Cache - Statement Detail You can hyperlink from here to BMC s Common Explain to analyze the actual statement 70

71 EXPLAINing, Tracing and Monitoring Monitoring Dynamic SQL Cache Detail information about statement cache is available only to online monitoring programs. Information cannot be externalized to SMF or GTF To obtain information, IFI program must request IFCID 316. IFCID 316 only contains first 60 bytes of statement text IFCID 317 can be used to obtain full SQL statement text IFCIDs are available when monitor trace class 1 is active. 71 To obtain the statement text, you must activate IFCIDs 316 and 317. More on this in the section on V8. 71

72 EXPLAINing, Tracing and Monitoring Analysis of Timeouts/Deadlocks 72 DSNT375I/DSNT376I accompanied by a DSNT501I message. Shows culprit, victim and resource on which they encountered contention. Normal remedies like row-level locking, processing in same order, lowering commit frequency, etc. apply. 72

73 EXPLAINing, Tracing and Monitoring Identifying Users to Track Usage DRDA allows four types of data to be sent to DB2 for z/os to help identify a user or an application. Client accounting string String to identify client application String to identify client workstation name String to identify client user ID For identification only Not a DB2 authorization ID 73 In a DRDA architecture, you have various choices to pass to DB2 a string to identify the user or application. This is strictly for identification security does not depend on this. 73

74 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing EXPLAINing, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 74 In this section, we will focus on the security aspects of dynamic SQL. This is a particular challenge in a distributed environment that allows free-form ad hoc SQL. 74

75 Managing Security Security Exposure STATIC UPDATE EMP SET SALARY = SALARY + :HV-RAISE WHERE EMPNO = :HV-EMPNO DYNAMIC UPDATE EMP SET SALARY = SALARY +? WHERE EMPNO =? For static SQL, user needs execute authority on plan/package only Not underlying update authority For dynamic SQL, user needs update authority, also Assuming default DYNAMICRULES(RUN) Therefore, with dynamic SQL: Authorizations must be maintained Users can bypass application and use ad hoc facilities such as SPUFI and QMF 75 Managing security for dynamic SQL poses new challenges that are not faced with static SQL. 75

76 Managing Security DYNAMICRULES(BIND) Causes dynamic SQL to behave like static from a security perspective Uses authorization of plan/package owner Good use: Application in server-only environment When generated SQL is controlled Caution: Very dangerous in an ad hoc or client/server environment since owner of ODBC/JDBC package is usually SYSADM 76 DYNAMICRULES(BIND) is a life-saver in the mainframe-only environment, but can be extremely dangerous in a client/server environment. 76

77 Managing Security Using Stored Procedures Dynamic SQL is converted to embedded static SQL in a stored procedure May not always be possible without losing flexibility Needs execute authority on stored procedure package only Does not need authority for underlying operation Complex when stored procedure itself contains dynamic SQL Covered later 77 Using stored procedure simplifies the security issue since the SQL within the stored procedure is generally static. 77

78 Managing Security ODBC/JDBC Applications Embedded: ODBC/JDBC package is not usable outside the application No security issues Generic: ODBC/JDBC package is usable by any application Using DYNAMICRULES(BIND) on a common package is dangerous Generally too high level of security (SYSADM) Security across applications One ODBC/JDBC package per application possible Use appropriate collection name during bind Use appropriate collection during run via ODBC keyword COLLECTIONID or CURRENT PACKAGESET 78 ODBC/JDBC applications present special challenges, especially when the SQL can be generic. 78

79 Managing Security Security without an Application Server Free- Form SQL? User Auth for Objects? DYNAMIC RULES Note N N BIND Most secure; access only via SQL processor N N RUN Meaningless since no access w/in or outside SQL processor N N Y Y BIND RUN Security hole since access same as owner w/in SQL processor, but no control outside processor Security hole since access controlled w/in SQL processor, but no control outside processor Y N BIND Same as owner w/in SQL processor, no access outside. Secure, but too permissive via processor? Y N RUN Meaningless since no access w/in or outside SQL processor Y Y Y Y BIND RUN Same as owner w/in SQL processor, security hole when outside. Too permissive via processor? Security hole since access controlled w/in SQL processor, but no control outside processor 79 The table above shows how each combination of type of SQL, user authorization to the objects, and the setting for DYNAMICRULES affects the security. No free-form SQL and no JDBC/ODBC DYNAMICRULES(BIND) is likely the best option. This model is similar to that used for static SQL. If free-form SQL or JDBC/ODBC, no fully secure option exists. That is why you will need to deploy an application server in this case. 79

80 Managing Security Security with an Application Server User 1 z/os Environment User 2 User 3 Application Server 1 App-server ID1 + pwd DB2 Subsystem User 4 User 5 D D F Grant execution on DB2 packages to app-server ID1 & app-server ID2 User 6 User 7 Application Server 2 App-server ID2 + pwd User 8 80 In a 3-tier architecture, the application server takes on the primary task for validating security and connecting to the z/os server via a common authorization ID. As long as this ID/password combination is kept secure and the package bind is controlled, this leads to a very secure environment. This scheme is typically used by ERP vendors. 80

81 Managing Security Network Facilities to Control Database Access Three-tier architecture Only application server needs access to database server Only application needs port number and IP address Firewalls guard against intruders 81 By using a common authorization ID and by allowing only the application server to connect to z/os, various options for preventing unauthorized access can be deployed. Using SNA, even more options are available. (This is beyond our current scope.) 81

82 Managing Security Dynamic SQL Security Kathy? Terry? Suresh? Auth ID? Dave? SURESH SET CURRENT SQLID = KATHY PACKAGE OWNER = TERRY SP PACKAGE OWNER = DAVE Default qualifier? Current SQLID applies? Grant/ revoke, etc. allowed? It Depends! 82 The answer to all 4 of these questions is defined by the dynamic statement behavior. This behavior depends on the DYNAMICRULES bind option and the runtime environment (see next slide). The definition of this statement behavior is shown on the slide after that. DYNAMICRULES values of BIND and RUN are commonly known. The other four choices are less known and could be deployed to meet your needs. 82

83 Managing Security DYNAMICRULES + Run-time Environment DYNAMICRULES value BIND RUN DEFINEBIND DEFINERUN INVOKEBIND INVOKERUN Stand-alone program environment behavior Bind Run Bind Run Bind Run Store procedure or UDF environment behavior Bind Run Define Define Invoke Invoke = Statement Behavior 83 How the dynamic SQL statement behavior is determined. Note that a program that is a stored procedure or UDF, or one that has a stored procedure or UDF somewhere in the calling sequence as a parent, is considered to be in a stored procedure/udf environment. 83

84 Managing Security Definitions of Statement Behaviors Attribute Environment Auth Id Default Qualifier Current SQLID GRANT, etc. allowed? Bind Run Define Invoke Plan/ package owner Current SQLID SP/UDF owner Auth ID of invoker (see note 1) Bind owner or qualifier Current SQLID SP/UDF owner Auth ID of invoker N/A Applies N/A N/A No Yes No No 84 This table defines how the dynamic SQL statement behavior affects 4 key areas: Authorization ID Default qualifier Current SQLID allowed? Ability to issue GRANT, REVOKE, CREATE, ALTER, DROP and RENAME Note 1: If the invoker is the primary auth ID of the process or the CURRENT SQLID value, secondary auth IDs will also be checked. Otherwise only the ID of the invoker is checked for the required authorization. 84

85 Where Are We? What is dynamic SQL? When is dynamic SQL appropriate? A glimpse of dynamic SQL development Dynamic statement caching Controlling system resources Locking and concurrency Governing Explaining, tracing and monitoring Managing security DB2 Connect ivity issues What is new with V8? Summary 85 In this section we will focus on connectivity issues in a distributed environment. 85