See the mechanics of how to do this for a cycle-driven process with a high degree of usability and easy job output management.

Transcription

1 Abstract: When concurrency is not needed for warehouse applications it is possible to use standard z/os tools to load a Db2 Analytics Accelerator without sample programs or 3rd party tools. See the mechanics of how to do this for a cycle-driven process with a high degree of usability and easy job output management. 1

2 2

3 3

4 CDC being the Change Data Capture feature for capturing inserts/updates in near real-time from the Db2 logs, and sending them directly to an Accelerator for usage. This requires usage of the data capture changes (none is the default) parm at the table level to write a complete row to the logs for all updates. Note you may have to enlarge your logs if you newly activate this for a large number of tables. 4

5 Note here is an example of the provided C-sample program job for loading 1 table, if you do have a C compiler but don t have the resources to implement a custom load process: //LOADFULL EXEC PGM=IKJEFT01,DYNAMNBR=20,COND=(4,LT) //* PARAMETER #1 FOR ACCELERATOR NAME //AQTP1 DD * IDAAENV1 /* //* PARAMETER #2 FOR LOCK_MODE //AQTP2 DD * NONE /* //* PARAMETER #3 FOR LOAD CONTAINING LOAD SPECIFICATION //AQTP3 DD * <?XML VERSION="1.0" ENCODING="UTF-8"?> <AQT:TABLESETFORLOAD XMLNS:AQT=" VERSION="1.1"> <TABLE SCHEMA="DBQENV1" NAME="TAAA_TRN_CNF" FORCEFULLRELOAD="TRUE"/> </AQT:TABLESETFORLOAD> /* //SYSTSPRT DD SYSOUT=* //SYSPRINT DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SYSTSIN DD * DSN SYSTEM(DB2A) RUN PROGRAM(AQTSCALL) PLAN(AQTSCALL) - PARMS('LOADTABLES') END 5

6 In V5 the new ACCEL_SYNCHRONIZE_SCHEMA stored procedure eliminates the need to remove/re-add/reload Accelerator tables when a source table has a new column added to it. Alternatively, some vendor tools have a TSO and/or batch interface to invoke maintenance stoprocs. For example, in CA s RCQuery tool we can use the following command syntax in a batch job to perform Accelerator synchronization tasks (for V4 usage only):.call PROCEDURE ACCEL SYSPROC.ACCEL_REMOVE_TABLES.DATA NAME(IDAAENV1) SSID(DB2A) TABLE("DBQENV1"."TXYZ_PTY_SRV_FCT").ENDDATA.CALL PROCEDURE ACCEL SYSPROC.ACCEL_ADD_TABLES.DATA NAME(IDAAENV1) SSID(DB2A) TABLE("DBQENV1"."TXYZ_PTY_SRV_FCT").ENDDATA 6

7 A simple DSNTIAUL step to generate the Accelerator driver file: //STEP1 EXEC PGM=IKJEFT01,DYNAMNBR=20 //SYSTSIN DD DSN=DBQENV1.JCL.LIB(TIAUL#Q),DISP=SHR //SYSIN DD * SELECT 'NONE,' CREATOR ',' NAME ' ' FROM SYSACCEL.SYSACCELERATEDTABLES WHERE ENABLE = 'Y' WITH UR; /* //SYSREC00 DD DSN=DBQENV1.IDAA.LOADPARM,DISP=OLD //SYSPRINT DD SYSOUT=* dsntiaul runparms genned tb parm output Use the Parse command to remove junk bytes before and after the needed parm values in each record. 7

8 Sample jcl to generate the driver file, and invoke rexx program to load an Accelerator: //STEP1 EXEC PGM=IKJEFT01,DYNAMNBR=20 //SYSTSIN DD DSN=DBQENV1.JCL.LIB(TIAUL#Q),DISP=SHR dsntiaul runparms //SYSIN DD DSN=DBQENV1.SQL.LIB(GENPARM),DISP=SHR sysaccel extract sql //SYSREC00 DD DSN=DBQENV1.IDAA.LOADPARM,DISP=OLD genned tb parms //SYSTSPRT DD SYSOUT=* //* //STEP2 EXEC PGM=IKJEFT01,PARM='IDAALODR DB2A,IDAAENV1' invoke rexx //SYSPROC DD DSN=DBQENV1.CLIST.LIB,DISP=SHR //SYSTSIN DD DSN=DBQENV1.IDAA.LOADPARM,DISP=SHR use driver file //SYSTSPRT DD SYSOUT=* 8

9 9

10 Every table will have either one of these Load messages, or a failed load message. Since most most of the time the loads succeed the amount of output generated is reasonable, and either way is easy to search on. 10

11 The +462 is technically a warning from the sql call of the stoproc: +462 EXTERNAL FUNCTION OR PROCEDURE name (SPECIFIC NAME specific-name) HAS RETURNED A WARNING SQLSTATE, WITH DIAGNOSTIC TEXT text To see the details of the Sqlstates for this sql code search on IDAA error handling : Stored procedures that do not return result sets The successful execution of a stored procedure that does not return result sets ends in SQL code 0 and SQL state If an error or warning is reported, the stored procedure returns a SQL code of +462 and one of the following SQL states: 01H08 If one or more errors have been encountered. There might be warnings as well. 01H04 If one or more warnings, but no errors have occurred. 11

12 Tables Processed corresponds to the number of records in the driver file. If a table has not changed (per Real-Time Stats) since the last load it is not re-loaded to the Accelerator. Real-Time Stats are an important requirement for using the Accelerator. If a driver file is not genned, but fixed table lists are used instead, there may be a need to periodically comment tables out, so these are reflected in the last metric. Note- All tables counted in the load failed counter are listed below the total box for easy searching & diagnosis. 12

13 The form of the Parse to get the step level EXEC-parms is the Parse Arg command, from the Rexx manual: PARSE ARG - parses the string or strings passed to a program or internal routine as input arguments. As part of housekeeping, before the main loop begins, I use these 2 Rexx commands: LdType='Chgd' /* init to changed-data by default */ PARSE ARG DBSSID ', AccelId ', LdType /* get 1x per job */ Instead of relying on the changed data parm being set in the jcl, I set the value as a default, before pulling the jcl value into the program. Note the comma literals are set as delimiters for the retrieved values. 13

14 The form of the Parse command to get the input file records is the Parse Pull command - see slide 20 for usage of this in the main loop routine. From the Rexx manual: PARSE PULL - parses the next string from the external data queue. If the external data queue is empty, PARSE PULL reads a line from the default input stream (the user's terminal), and the program pauses, if necessary, until a line is complete.. See also PULL for the PULL instruction. PULL and PARSE PULL read from the data stack. In TSO/E, if the data stack is empty, PULL and PARSE PULL read from the: Input stream, which is SYSTSIN (TSO/E background). In non-tso/e address spaces, if the data stack is empty, PULL and PARSE PULL read from the input stream as defined by the file name in the INDD field in the module name table (see Module name table). The system default is SYSTSIN. If SYSTSIN has no data, the PULL and PARSE PULL instructions return a null string. 14

15 Lock mode options, from IBM V4 online documentation: Controls the protection level while accelerator-shadow tables are being loaded. The protection level ensures a consistent state of the data with respect to the specified entity. Valid values: TABLESET - Protects the DB2 source tables of the tables to be loaded against changes during the load operation. TABLE - Protects just the DB2 source table of the table that is currently being loaded. PARTITIONS - Protects the DB2 table-space partition containing the part of the of the source table that is currently being loaded. With this setting, an unpartitioned table is always locked completely. NONE - No locking at all. However, only committed data is loaded into the acceleratorshadow tables because the DB2 data is unloaded with isolation level CS and SKIP LOCKED DATA. ROW - Protects at least the row or page that is being loaded against updates. Whether a row, a page, or an entire table is locked is determined by the LOCKSIZE defined for the table space of the source tables. 15

16 The XML string options are very flexible- this means the XML syntax can get very complex. I chose to keep the XML syntax & format as simple as possible, which is why we only specify 1 schema & 1 table per 1 tablesetforload XML string. See the doc for these options under : Accelerator 4.1 / Stored Procedures / SYSPROC. ACCEL_LOAD_TABLES Note by using the detectchanges tag as our default we can also avoid special xml coding for partitions. If a partition has changed data it is loaded, if no changes it is not reloaded. Please note this warning: Do not combine the detectchanges= DATA attribute with an explicit partition list. If you do, the partition list will be ignored. 16

17 XML tag syntax can be extremely particular, and easy to misuse, especially when strings are nested within strings. In the case above we are also concatenating Rexx variables within the XML tags. The good news is you don t have to know what the XML components mean or why the literals ( 2011?) are what they are. The main point is if the tag is constructed in a way the stored procedure accepts as valid- it works. Fortunately Rexx is great for building strings from variables and string parts all concatenated together. A way to deconstruct the above string is this: Variable = big xmlstring literal..until I get to this subtag where I append this string, '<table name= /* double quotes are a literal needed in this string part */ variable-name-in-red ', /* need to end the string with double-quotes, also for XML */ /* the comma is for rexx string continuation */ Rexx concatenates these pieces together into 1 valid XML string, while using the variable-value. 17

18 The Message XML return string is required, and has no nested variables. Note that this is only used for output, though the proc parm is defined as InOut. The MSG area is defined in the stoproc as a 64k clob, for V4.1. The MessageIndicator =1 means data has been returned. 18

19 The housekeeping routines do several 1-time items: - Get the EXEC step level parms, including the Db2 subsys - Create the Rexx environment & connect to Db2 - Initialize counters & variables The main routine (yes this is a procedural language, not object oriented) gets the driver record, parses out the schema & table name into var names and calls the load proc using the XML string literals. The XML strings include the vars parsed from the driver file. When a load is successful there are useful statistics returned in the message variable as a large XML string. These stats are parsed out and accumulated using literals. Note the XML literals used for parsing could change with upgrades, but this is a known risk that will be included as an upgrade task item. Error parsing and formatting takes several looping routines, Rexx comments are used to make these less cryptic. 19

20 The Parse Var form of the parse command is extremely useful for pulling apart strings based on a delimiter such as a blank or a comma. In this example the string within variable InData is broken into pieces based on the commas in the input record. The period after Tabname end is a placeholder for any comments that may be present, basically any remaining string portion. Here is the description from the Rexx manual: PARSE VAR name - parses the value of the variable name. The name must be a symbol that is valid as a variable name (that is, it cannot start with a period or a digit). Note that the variable name is not changed unless it appears in the template, so that for example: PARSE VAR string word1 string removes the first word from string, puts it in the variable word1, and assigns the remainder back to string. Similarly PARSE UPPER VAR string word1 string in addition translates the data from string to uppercase before it is parsed. 20

21 21

22 Db2 Rexx Language Support includes an API to send sql statements to Db2. The ADDRESS DSNREXX command is how this API is used. Note the double-quotes used for the exec-sql string, this tells Rexx an external environment is being used, and this string should not be processed as a Rexx string. Also note the form of a case-statement in Rexx is the SELECT/when command. This is very useful for error code checking. 22

23 The +462 stoproc warning code is always treated here as a load-proc failure. Note the OR condition in the When clause: the vertical bar for the 2 sqlstate conditions. 23

24 Note the usage of the Position built-in function as a way to find a substring based on its position in a string. A zero returned means it was not found. If the good path sqlstate message (AQT10000I) is not found then throw an error, and use the Parse Var command to deconstruct the error literal in the xml string. 24

25 This is an example of the unformatted XML error string returned when a load fail occurs: <?xml version="1.0" encoding="utf-8"?><dwa:messageoutput xmlns:dwa=" version="1.0"> <messag e severity="error" reason-code="aqt10102i"><text>the value 'TABC_CRG_MMB'; is incorrect for an identifier of the TABLE type. The error condition is: NOT EXISTS. The identifier can take on a maximum length of 128.</text><description>This error occurs when an input parameter of a stored procedure contains an invalid identifier value. This happens, for example, if the accelerator name is too long. The incorrect value can be a database name, an accelerator name, a schema name, a location name, a table name, a column name, a tablespace qualifier, a tablespace name or a path in a hierarchical file system.</description><action>specify a correct identifier value.</action></message></dwa:messageoutput> 25

26 Two useful but different built-in functions: WORDS(string) returns the number of blank-delimited words in string. Here are some examples: WORDS('Now is the time') -> 4 WORDS(' ') -> 0 WORD(string,n) returns the nth blank-delimited word in string or returns the null string if fewer than n words are in string. The n must be a positive whole number. Here are some examples: WORD('Now is the time',3) -> 'the' WORD('Now is the time',5) -> '' 26

27 There are 4 variable types in Rexx, two can be used for Arrays: compound symbols & stems. From a Rexx manual: A compound symbol permits the substitution of variables within its name when you refer to it. A compound symbol contains at least one period and at least two other characters. It cannot start with a digit or a period, and if there is only one period in the compound symbol, it cannot be the last character. The name begins with a stem (that part of the symbol up to and including the first period). This is followed by a tail, parts of the name (delimited by periods) that are constant symbols, simple symbols, or null. The derived name of a compound symbol is the stem of the symbol, in uppercase, followed by the tail, in which all simple symbols have been replaced with their values. A tail itself can be comprised of the characters A Z, a z, 0 9, # $.!? and underscore. The value of a tail can be any character string, including the null string and strings containing blanks. For example: stem.taila=99 stem.tailb=stem.taila say stem.tailb /* Displays: 99 */ These are compound symbols: FRED.3 Array.I.J AMESSY..One.2. Before the symbol is used (that is, at the time of reference), the language processor substitutes the values of any simple symbols in the tail (I, J, and One in the examples), thus generating a new, derived name. This derived name is then used just like a simple symbol. That is, its value is by default the derived name, or (if it has been used as the target of an assignment) its value is the value of the variable named by the derived name. The substitution into the symbol that takes place permits arbitrary indexing (subscripting) of collections of variables that have a common stem. Note that the values substituted can contain any characters (including periods and blanks). Substitution is done only one time. 27

28 A short example of trace output: first the line of code prefixed by the line number, then the next line has a marker of >>>, followed by the variable values or condition result (0=false, 1=true). 594 *-* do cnt=1 until cnt >= wrdcnt >>> "0" 595 *-* if length(newstr wordz.cnt) > 68 >>> "0" 602 *-* else 603 *-* newstr = newstr wordz.cnt >>> " The value TABC_CRG_MMB is incorrect" 604 *-* end 594 *-* do cnt=1 until cnt >= wrdcnt >>> "0" 595 *-* if length(newstr wordz.cnt) > 68 >>> "0" 602 *-* else 603 *-* newstr = newstr wordz.cnt /*note the concatenated string */ >>> " The value TABC_CRG_MMB is incorrect for" 604 *-* end 28

29 The ARG function is retrieving the passed string from variable MSG_TEXT, when this routine is called. This passed value contains all of the useful load statistics. It is assigned to variable longtext for parsing out & deconstructing the literals and values from the XML MSG area in the Parse Var command. The ARG function is similar to the Parse Arg command. 29

30 This program does include a call to the stoproc Accel_Set_Tables_Acceleration to guarantee tables are Enabled/online after a successful load. This could be an extraneous or optional step however if all the tables you are load are already Enabled as part of their selection criteria in the load generator sql. 30

31 Use the Rexx program return code to determine whether the load report step should send the report, or whether an to resolve a load failure should be sent to support staff instead. 31

32 32

33 Speaker Bio: I've worked with Db2 since v1.3, and have been a Db2 DBA for over 20 years. 33