G09. Davy Goethals Arcelor Technologies. Platform: DB2 UDB for z/os

Transcription

1 G09 Unloading and Loading LOB data in DB2 for z/os becomes reality Davy Goethals Arcelor Technologies Wednesday 4 October :00 am 12:00 am Platform: DB2 UDB for z/os Recently IBM introduced an enhancement to DB2 V7 and V8 to allow LOB data bigger than 32K to be unloaded and loaded using the standard DB2 Unload and Load Utilities. In this presentation, this new function will be explained in depth, based on experiences gathered during a 5 month IBM beta program in which the presenter participated. 1

2 Overview Implementation of LOBs in DB2 Unloading LOB data using the new enhancements of UNLOAD Loading LOB data using the new enhancements of LOAD Lessons learned ; recommendations Conclusions 2 How are LOBs (Large objects) implemented in DB2 for z/os and what were the current restrictions. What exactly are the new features introduced by apar PK10278 in DB2 V7 and DB2 V8. How to unload LOB data using the enhanced UNLOAD features. How to load LOB data using the enhanced LOAD features. Benefits to other techniques. Remark : every time we will use the terms DB2 in this presentation, we mean DB2 UDB for z/os 2

3 Unloading and Loading LOB data Implementation of LOBs in DB2 3 How are LOBs (Large objects) implemented in DB2 for z/os and what were the current restrictions. 3

4 Large objects (LOBs) in DB2 Before DB2 V6 : max character field was 32 Kb SBCS character (EBCDIC or ASCII) BIT bit format Larger strings mostly stored as multiple row 4 Before DB2 V6, character string data could only be stored in CHAR or VARCHAR columns with a maximum length of 32 Kb. The same was true for bit string data. When character string data was defined as SBCS (single byte character set), it could be stored in EBCDIC or ASCII format. The CCSID is used during conversion between machines with different encoding schemes. When defined as BIT string, no conversions are done between machines with different encoding schemes. Strings larger then 32 K could only be stored and retrieved in an application using multiple rows with a sequence number column.. 4

5 Large objects (LOBs) in DB2 >= DB2 V6 : support for large objects till 2 Gb CLOB character (EBCDIC, ASCII) Since DB2 V7 also in UNICODE BLOB bit format LOBs < 32K are called SLOBs can be used to store text documents, images, audio and video objects 5 Since DB2 V6 character string data and bit string data can also be stored as a large object (LOB) with a maximum length till 2 GB. CLOBs are used for storing character string data, BLOB s are used for bit string data. Since DB2 V7 CLOB s can also be stored in UNICODE format. DCLOB s are used to store DBCS data (double byte character set) for languages like Japanese with double byte characters. LOBs can be used for storing documents and multimedia objects like images, audio and video. These objects have a size which is mostly bigger than 32Kb. A colored image has a typically size of 25 Mb, a video of 1 hour is about 1 Gb.. 5

6 Lob support in DB2 : architecture Base table space base table key col col2 col3 rowid lob col key a data a data a rowid a - key b data b data b rowid b - auxiliary index Lob table space rowid rowid a rowid b auxiliary table lob data lob data a lob data b 6 Although a lob column is conceptually part of the base table, it is physically stored in a separate table which is called the auxiliary table for that lob column. This auxiliary table resides in a separate table space which is called the lob table space for that lob column. Both base table space and lob table space must reside in the same database. If the base table space is a partitioned table space, each lob column has a separate auxiliary table in a separate lob table space for each partition. So if I have a partitioned table space with a base table in it with 5 partitions and 2 lob columns, I will end up with 10 auxiliary tables in 10 lob table spaces. The link between the base table and auxiliary table is done by a ROWID column. For quickly finding the lob data in a lob table space, a special index called the auxiliary index must be created on the auxiliary table. There are 2 main reasons for separating the lob data from the data of the other columns : performance (think about table space scans on the base table) and the fact that the maximum row size of a classic table is 32K. 6

7 Lob support in DB2 : DDL CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. CREATE LOB TABLESPACE IMAGELOB IN DATABASE USING STOGROUP SYSDEFLT.. BUFFERPOOL BP1 ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) IN DATABASE. TSIMAGE LOG LOCKSIZE DSSIZE NO LOB 8 G CREATE UNIQUE INDEX A.AUX_IX ON A.IMAGES_AUX USING STOGROUP SYSDEFLT... CREATE AUXILIARY TABLE A.IMAGES_AUX IN DATABASE. IMAGELOB STORES A. IMAGES COLUMN IMAGE ( PART. ) 7 BASE TABLE : the base table is created in a normal database and table space and can have multiple indexes as a normal table. (DDL for the base table space is not shown here). However it is not allowed to create an index on a LOB column. LOB TABLE SPACE : each auxiliary table must be created in it s own lob table space. Different storage groups and bufferpools may be considered.. AUXILIARY TABLE : each lob column must have an associated auxiliary table where the actual lob data will be stored. The auxiliary table is created in a lob table space and refers to a lob column of the base table (or lob column of a base table partition using the PART clause). AUXILIARY INDEX : the auxiliary table must have one AUXILIARY INDEX (no columns defined). The implicit index definition consists of a 2 key value (rowid + lob version). 7

8 Loading Lob data in DB2 for z/os Cobol key col col2 col3 rowid lob col key a data a data a rowid a key b data b data b rowid b host appl file DB2 table drda - - lobs z/os host ftp client appl Java or UDB import client 8 In practice, most lob data will be created on a client platform Conceptually, there are 2 ways for loading this data into a DB2 table on the mainframe z/os server. 1) send the data with a file transfer to the host and LOAD the data in the DB2 table with a host application (written in COBOL,PL/1 ) or DB2 utility. 2) send the data via drda to the host DB2 table with a client application (written in C, Visual Basic, Java ) or a UDB import utility 8

9 Lob support in DB2 : LOAD utility LOAD utility: restrictions before PK10278 Only SLOBs allowed Always load of base table (not auxiliary table) Lob columns treated as normal data columns Normal data and lob data in same input record with a limit of 32K (MVS limit for sequential data set) Not sufficient in most cases! No cross loading (INCURSOR) before PQ90263 To load data from another DB2 table 9 Before PK10278 it was only possible to load lob data from a sequential file when the total input record size, containing both normal columns and lob columns, was smaller than 32 k. In most cases this 32k limit is not enough as it limits the use of Lobs to Slobs. It is also possible to convert input CHAR or VARCHAR data to LOB columns. You can also apply CCSID conversions if the CCSID of your input data set and table space are different. No conversion is applied is the subtype is BIT. Support for the cross loader was introduced with PQ90263 (may 2005) 9

10 Lob support in DB2 : LOAD utility LOAD LOG Tablesp LOG What logged Tablesp status yes yes all data - no no control info - no yes nothing copyp no no nothing copyp 10 If you specify LOG YES, the amount of data that will be logged depends on the LOG parameter of the LOB table space. If the lob table space has the value LOG NO, only control information will be logged that is needed for recovery. If you specify LOG NO nothing is logged and both the base table space and lob table will be put in the COPYP status. You then have to take a full image copy to remove the restricted state (or use the NOCOPYPEND option if recoverability is not important) 10

11 Lob support in DB2 : Loading Other techniques to LOAD Lob data from sequential files : Writing a COBOL or PL/I program Using host variables and locator variables Using static or dynamic SQL Be aware of performance Look at ISV offerings Use a workstation application to directly populate the table using DRDA or UDB IMPORT command 11 11

12 Lob support in DB2 : UNLOAD utility UNLOAD utility: restrictions before PK10278 Only SLOBs allowed Lob columns treated as normal data columns Normal data and lob data in same output record with a limit of 32K (MVS limit for sequential data set) TRUNCATE keyword can be used to limit Not sufficient in most cases! 12 Before PK10278 it was only possible to unload lob data to a sequential file when the total output record size, containing both normal columns and lob columns, was smaller than 32 k. 12

13 Lob support in DB2 : Unloading Other techniques to UNLOAD Lob data to a sequential file : DSNTIAUL : also truncates on 32 K in DB2 V7 & V8 Writing a COBOL or PL/I program Net.Data macro to unload to HFS files ISV offerings Use a workstation application to directly export the data using DRDA or UDB EXPORT command 13 Before PK10278 unloading lob data was also not a simple task. It was possible to write an own application to create one or more sequential data sets with the lob data contained in some private layout spread over multiple records.. In V7 and V8 the DSNTIAUL sample program also limits the output data set to the first 32K of the data. If you had a Net.Data environment you could use a Net.Data macro to unload the lob records to HFS files. Maybe you had some ISV utilities to help you. 13

14 Unloading and Loading LOB data Unloading LOB data using the new enhancements of UNLOAD 14 How to unload LOB data using the enhanced UNLOAD features.. 14

15 Unload Lob enhancements PK10278 Unload lob data to separate file instead of the normal unload file (with max 32K record length) Member of a pds or pdse Hfs file Store file name in the normal unload file as char or varchar field 1 separate file per lob value 15 UNLOAD is changed to store the value of a LOB column in a separate file and record the name of the file in the unloaded record of the base table. 15

16 Unload Lob enhancements PK10278 Implemented through : New keywords in the field specification for char and varchar (file reference variables) BLOBF template name CLOBF template name DBCLOBF template name New keywords in TEMPLATE definition : DIR DSNTYPE (&UNIQ) or /&UNIQ. automatically appended 16 The new file reference variables BLOBF,CLOBF and DBCLOBF specify that the output field in the normal unload file will contain the name of a file where the actual LOB value will be stored (in CHAR or VARCHAR format) For BLOBF the BLOB will be stored without CCSID conversions. For CLOBF and DBCLOBF the CLOB or DBCLOB will be stored with any required CCSID conversions specified in the UNLOAD clause If you use a TEMPLATE to dynamically allocate the output lob files the new keyword DSNTYPE allows you to specify the data set type LIBRARY, PDS or HFS. If you omit the DSNTYPE keyword the file will be allocated as DSORG=PO and using the default SMS rules active. The keyword DIR can be used to specify the number of directory blocks for a new PDS (default = number of records divided by 20) The member name for PDS,PDSE or data set name for the HFS directory is based on the value of the new template variable &UNIQ. or &UQ.. It is a unique 8 characters string derived from the system clock beginning with an alpha followed by 7 alpha or numeric characters.. 16

17 Unload Lob enhancements PK10278 CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. TEMPLATE TSYSPUN DSN('WRK.PUNCH') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(MOD,CATLG,CATLG) UNLOAD TABLESPACE IMAGES.IMAGES UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN) ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) IN IMAGES.IMAGES DSNU1218I - DSNUULIA - LOGICAL RECORD LENGTH OF OUTPUT RECORD EXCEEDED THE LIMIT FOR TABLE A.IMAGES DSNU012I DSNUGBAC - UTILITY EXECUTION TERMINATED, HIGHEST RETURN CODE=8 17 Let s for example take a table which contains a BLOB column with a maximum length of 100 Mb. Normal unload of the table space will fail because the 100 Mb exceeds the maximum record length of 32K for the unload data set. 17

18 Unload Lob enhancements PK10278 CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. TEMPLATE TSYSPUN DSN('WRK.PUNCH') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(MOD,CATLG,CATLG) UNLOAD DATA FROM TABLE A. IMAGES UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN) ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) IN IMAGES.IMAGES DSNU1218I - DSNUULIA - LOGICAL RECORD LENGTH OF OUTPUT RECORD EXCEEDED THE LIMIT FOR TABLE A.IMAGES DSNU012I DSNUGBAC - UTILITY EXECUTION TERMINATED, HIGHEST RETURN CODE=8 18 In the same example, normal unload of the table will also fail for the same reason 18

19 Unload Lob enhancements PK10278 CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) IN IMAGES.IMAGES TEMPLATE TSYSPUN DSN('WRK.PUNCH') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(MOD,CATLG,CATLG) UNLOAD DATA FROM TABLE A. IMAGES IMAGE_ID, IMAGE_TYPE, IMAGE BLOB) UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN) DSNU1218I - DSNUULIA - LOGICAL RECORD LENGTH OF OUTPUT RECORD EXCEEDED THE LIMIT FOR TABLE A.IMAGES DSNU012I DSNUGBAC - UTILITY EXECUTION TERMINATED, HIGHEST RETURN CODE=8 19 In the same example, normal unload of the table specifying the column names will also fail for the same reason 19

20 Unload Lob enhancements PK10278 CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) TEMPLATE TSYSPUN DSN('WRK.PUNCH') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(MOD,CATLG,CATLG) UNLOAD DATA FROM TABLE A. IMAGES IMAGE_ID, IMAGE_TYPE, IMAGE BLOB (30000) TRUNCATE ) UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN) IN IMAGES.IMAGES 20 Before PK10278 the only thing possible here in the UNLOAD utility was the TRUNCATE keyword to truncate the lob data to fit in an unload data set with a maximum record length of 32K. 20

21 Unload Lob enhancements PK10278 CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) IN IMAGES.IMAGES TEMPLATE TSYSPUN DSN('WRK.PUNCH') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSLOB DSN( WRK.SYSLOB ) DSNTYPE(LIBRARY) DISP(MOD,CATLG,CATLG) UNLOAD DATA FROM TABLE A. IMAGES IMAGE_ID, IMAGE_TYPE, IMAGE VARCHAR BLOBF TSYSLOB ) UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN) 21 Members : AYPPXJRD AYPPXJ3H AYPPXJ6K. With PK10278 you can instruct the UNLOAD utility to unload the blob data to a separate data set. In this example a PDSE was chosen using the BLOBF file reference variable and a template TSYSLOB. As a result a new PDSE with DSNAME=WRK.SYSLOB will be dynamically allocated and for each lob value a new member with a unique name will be created for storing the lob value. 21

22 Unload Lob enhancements PK10278 CREATE TABLE A. IMAGES (IMAGE_ID VARCHAR(30) NOT NULL, IMAGE_TYPE VARCHAR(10), VERSION VARCHAR(30),. ROW_ID ROWID NOT NULL GENERATED ALWAYS, IMAGE BLOB(100M) ) TEMPLATE TSYSPUN DSN('WRK.PUNCH') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(MOD,CATLG,CATLG) TEMPLATE TSYSLOB DSN(/u/davy) DSNTYPE(HFS) UNLOAD DATA FROM TABLE A. IMAGES IMAGE_ID, IMAGE_TYPE, IMAGE VARCHAR BLOBF TSYSLOB ) UNLDDN(TSYSREC) PUNCHDDN(TSYSPUN) IN IMAGES.IMAGES 22 Files /u/davy/ AYPPXJRD /u/davy/ AYPPXJ3H /u/davy/ayppxj6k. In this example a HFS directory was chosen using the BLOBF file reference variable and a template TSYSLOB. As a result for each lob value a new file with a unique name will be created in this directory for storing the lob values. 22

23 Unload Lob enhancements PK10278 Others : A null lob will be represented by a null output file name (null indicator byte) Unloading lob data from image copies is not allowed HFS directory must exist and template must specify full path name (i.e. not relative) Use uppercase when using template variables Unloading to sequential files is not possible (DSORG = PO or HFS) 23 23

24 Unloading and Loading LOB data Loading LOB data using the new enhancements of LOAD 24 How to load LOB data using the enhanced LOAD features. 24

25 Load Lob enhancements PK10278 Load lob data from separate file instead of the normal sysrec input file (with max 32K record length) Sequential file Member of a pds or pdse Hfs file File name present in the input file as char or varchar field 1 separate file per lob value 25 LOAD is changed to allow an input field value to contain the name of a file containing a LOB column value. The LOB column will be loaded from that file. 25

26 Load Lob enhancements PK10278 Implemented through : New keywords in the field specification for char and varchar (file reference variables) BLOBF CLOBF DBCLOBF Ccsid conversions apply for CLOBF and DBCLOBF NULL input file name results in null lob value 26 The new file reference variables BLOBF,CLOBF and DBCLOBF specify that the input field in the normal input file contains the name of a file (in CHAR or VARCHAR format) where the actual LOB value is stored. For BLOBF the BLOB will be loaded without CCSID conversion. For CLOBF and DBCLOBF the required CCSID conversions will be done during the load. 26

27 Load Lob enhancements PK10278 CREATE TABLE A. DOCUMENTS (DOC_ID VARCHAR(30) NOT NULL, DOC_TYPE VARCHAR(10), VERSION VARCHAR(30),. ROW_ID ROWID NOT NULL GENERATED ALWAYS, DOCUMENT CLOB(100M) ) TEMPLATE TSYSREC DSN('WRK.SYSREC') DISP(OLD,KEEP,KEEP) LOAD DATA INDDN TSYSREC LOG NO REPLACE INTO TABLE A.DOCUMENTS (DOC_ID POSITION( 1 : 31) VARCHAR, DOCUMENT POSITION( 122:378) VARCHAR CLOBF NULLIF(00121)=X'FF' ) IN DOCUMENTS.DOCUMENTS 27 Let s for example take a table which contains a CLOB column with a maximum length of 100 Mb to store text documents. The actual documents are loaded from a file which name is contained in the normal input file as a VARCHAR field 27

28 Load Lob enhancements PK DOC001 WRK.SYSLOB(AYPPXJRD) DOC002.WRK.SYSLOB(AYPPXJ3H) DOC003 WRK.SYSLOB(AYPPXJ6K) NULLIF(00121)=X'FF 28 The normal input file for the LOAD utility will contain a file name where the actual lob data can be found. In this example it will be a member of a PDS. If the indicator variable contains a -1 (null value) the lob will be null. 28

29 Load Lob enhancements PK DOC001 /u/davy/ayppxjrd DOC002./u/davy/AYPPXJ3H DOC003 /u/davy/ayppxj6k NULLIF(00121)=X'FF 29 The normal input file for the LOAD utility will contain a file name where the actual lob data can be found. In this example it will be a file on a HFS directory. 29

30 Xload Lob enhancements PQ90263 Cross load : new V7 feature of the LOAD utility input is not a sequential file but the result set of a dynamic SQL query source data in query are local or remote DB2 table(s) source can be on any drda server or the result of Websphere Information Integrator Replaces unload, file transfer, load 30 Websphere Information Integrator was formerly called Datajoiner 30

31 Xload Lob enhancements PQ90263 EXEC SQL DECLARE c1 CURSOR FOR select query ENDEXEC LOAD DATA INCURSOR c1 INTO TABLE creator.table 31 31

32 Xload Lob enhancements PQ90263 CREATE TABLE A. DOCUMENTS (DOC_ID VARCHAR(30) NOT NULL, DOC_TYPE VARCHAR(10), VERSION VARCHAR(30),. ROW_ID ROWID NOT NULL GENERATED ALWAYS, DOCUMENT CLOB(100M) ) IN DOCUMENTS.DOCUMENTS EXEC SQL DECLARE C1 CURSOR FOR SELECT DOC_ID,VERSION,..DOCUMENT FROM DB2R.A.DOCUMENTS WHERE DOC_ID. ENDEXEC LOAD DATA INCURSOR C1 LOG NO REPLACE INTO TABLE A.DOCUMENTS DSNU1178I - DSNUULIA - SUM OF INPUT COLUMN LENGTHS FOR CURSOR C1 IS TOO LARGE DSNU012I DSNUGBAC - UTILITY EXECUTION TERMINATED, HIGHEST RETURN CODE=8 32 Before PQ90263 it was impossible to cross LOAD data with LOB columns when the LOB column caused the total record length to exceed

33 Xload Lob enhancements PQ90263 Architectural limit of bytes bypassed by using a separate buffer for LOB columns above 16Mb line Only 8 bytes stored per lob column in byte buffer Cross loading now limited by available memory above 16Mb (region size) Message DSNU1178I explanation changed 33 With PQ90263 the explanation text of message DSNU1178I has been changed to reflect the fact that cross loading of Lob columns will only be limited by the available memory above the 16Mb line. If the user receives this message then increasing the region size will likely result in successful execution of the LOAD utility. 33

34 Unloading and Loading LOB data Lessons learned ; recommendations 34 34

35 Lessons learned ; recommendations PK10278 code delivered by PK22910 UK13720 for DB2 V7 (12/05/2006) UK13721 for DB2 V8 (12/05/2006) Also recommended : PK22949 : abend 04e rc00e20005 during load Informational apar II13767 : recommended maintenance lobs 35 35

36 Lessons learned ; recommendations Generate LOAD statements using UNLOAD syspunch file Generate UNLOAD statements using rexx or equivalent UNLOAD DATA FROM TABLE With BLOBF,CLOBF,DBCLOBF From sysibm.syscolumns type in ( blob, clob, dbclob ) From sysibm.sysdatatypes Sourcetype if type = distinct 36 36

37 Lessons learned ; recommendations Be prepared for recovery of LOB data Use TEMPLATES and LISTDEF to backup all at once and to create a common recoverable point of consistency after LOAD LISTDEF MYLIST INCLUDE TABLESPACES TABLE creator.table ALL INCLUDE INDEXSPACES COPY YES TABLE creator.table ALL COPY LIST MYLIST FULL YES SHRLEVEL REFERENCE PARALLEL COPYDSN(TSYSCOPY) 37 It is recommended to take an image copy of the base table space and lob table spaces after LOAD using SHRLEVEL REFERENCE to create a consistent backup. Because a non-partitioned or partitioned base table can contain many auxiliary objects (one lob table space and one auxiliary index per lob column and per partition) it is recommended to use a LISTDEF to generate a list of all these objects. Use the lob indicator keywords BASE, LOB or ALL if you want to include the base objects only, the auxiliary objects only or both in your list. Use the INDEXSPACES COPY YES keyword if you do not want to include the base table indexes for which the COPY YES attribute is disabled. Use the PARALLEL keyword to take the copies in parallel. If you specify a LISTDEF with SHRLEVEL REFERENCE, all copy data sets will have the same START_RBA value in the DB2 catalog table SYSIBM.SYSCOPY resulting in a good consistency point. 37

38 Lessons learned ; recommendations RACF problem Unload lob files are created by DBM1 addres space Under authority of DBM1 adres space userid Not with userid of utility job Fixed for PDS/PDSE in PK10278 base ptf Problem still exists for HFS files (PK22061?) 38 We found that the lob unload data sets were created under the userid of the DBM1 started task and not under the userid of the utility job submitted by the enduser. The problem could be bypassed by giving RACF ALTER authority on the data set profile to the userid associated with the DBM1 started task. Normally for a data set dynamically allocated with a TEMPLATE only the userid associated with the utility job needs the proper RACF authorities and not the DBM1 started task. The implication of this problem in a big organization would be that the DBM1 started task userid would become a kind of superuser entitled to update all possible data sets a common user comes up with in his utility TEMPLATE statements. This problem was solved for PDS/PDSE in the official ptf for PK22910 but still remains for HFS data sets. Currently open APAR PK22061 might solve this issue. 38

39 Lessons learned ; recommendations SMS ACS routine for db2 utility output data sets WHEN (&DSORG = 'PO') SET &DATACLAS='NODATACL' OTHERWISE SET &DATACLAS='STRIPEP' 39 We had to adjust our SMS ACS routines for DB2 utility output data sets to take into account that a LOB unload file can be a PDS or PDSE (before always sequential data set that was forced to be striped) 39

40 Lessons learned ; recommendations Unload lob file allocated as DSORG =PO RECFM=V,LRECL=0,BLKSIZE=27998 Restrictions : Cannot be allocated using ispf 3.2 Cannot be tersed PK27029 : UK16672 (V7) & UK16673 (V8) 40 The unload file is allocated with a LRECL=0 which is surprising. This is supposed to be solved by PK27029 (LRECL=27994) 40

41 Lessons learned ; recommendations Space abends on unload data sets IEC032I E37-04 IEC217I B14-0C File is not freed by DBM1 adres space Unable to delete, rename, etc. STOP/START DB2 to free 41 We also found following problem during our testing: when we unload a table with LOB columns to a PDS or PDSE using a LOB unload file and the unload fails with a space abend, then sometimes the unload file is being kept allocated by the DBM1 adress space. The only way to free the file is by rescheduling the DB2 member (terminate util is not enough). We got this problem during different test cases like when the directory of the unload PDS is full or when the unload PDSE reaches its maximum of 123 extents This is very annoying from an operational point of view because it is impossible to reallocate, rename or delete the data set as long as the DB2 member has not been rescheduled. 41

42 Lessons learned ; recommendations What if lob data sets in SYSREC do not exist? LOAD abends : Abend S04E RC00E40350 in dsnurwbx+5d3e reg2 = if the pds does not exist reg2 = if the pds member does not exist RC 00E40350 : error has been returned from LOB MANAGER insert. More user friendly messages expected (new SQLCODE=-452 in DB2 V9) 42 42

43 Lessons learned ; recommendations LOBVALA to small during LOAD Max amount of storage per user for storing lobs in dataspace or memory in KB storage RC 00C900D1 with LOBVALA = Kb if biggest LOB is 888K Wait for PK

44 Lessons learned ; recommendations TEMPLATE variable &UNIQ. or UQ. Generate unique name of 8 chars based on system clock starting with alphanumeric Can be used as data set qualifier 44 44

45 Lessons learned ; recommendations REORG versus UNLOAD/RELOAD REORG Improves prefetch performance of lob data Make lob pages as contiguous as possible Reclaim defragmented space (defragmentation) Reduce the number of allocated chunks to the minimum corresponding to orgratio = 100 No reclaim of physical space except delete of unused secondary data sets A002,A003,. 45 The REORG utility run on a lob table space is completely different from a reorg on a normal table space. The main objective is to improve read I/O. Because LOG NO lob table spaces should be consistent after rollback operations, each time a lob is updated a new version of the lob is created allocating new chunks of data. The space of old versions of lobs is only reusable after the rollback or commit of the logical-unit-of-work. When afterwards new versions of lobs are inserted, fragmentation occurs and on average more chunks have to be read to retrieve the lob data. Concurrent SQL access from applications is not allowed. 45

46 Lessons learned ; recommendations rowid1 Auxiliary rowid2 rowid points to first lob map page of lob index rowid3 rowid4 16 pages 16 pages 16 pages 16 pages 16 pages chunk1 chunk2 chunk6 chunk7 chunk13 First lob map page of lob with rowid1 points to chunk1 (partially allocated from page 1 with 6 pages) and chunk2 (partially allocated from page 6 with 7 pages) 46 DB2 allocates space for LOB data in chunks. A chunk is 16 contiguous pages (16 immediately adjacent pages). A single LOB value can be stored in multiple chunks which will be fully or partially allocated to that LOB. When an application wants to access a certain lob, the rowid stored in the base table will be used to locate the first lob map page via the auxiliary index. This lob map page contains a table with all the chunks that are fully or partially allocated to this lob. A lob table space is considered to be perfectly organized when the number of chunks allocated to a lob is minimal (no extra chunks needed). This is expressed by the orgratio factor which is 100 in the perfect case. In the above example the lob pointed by rowid1 is not perfectly organized because it is allocated to 2 chunks (6 pages in chunk 1 and 7 pages in chunk2) where one chunk (13 out of 16 pages) would be sufficient. PQ96460 introduced a new definition of ORGRATIO : it is now the percentage of organization in the LOB table space.a value of 100 indicates perfect organization of the LOB table space. A value of 1 indicates that the LOB table space is disorganized. A value of 0.00 indicates that the LOB table space is totally disorganized. 46

47 Lessons learned ; recommendations REORG versus UNLOAD/RELOAD REORG Improves prefetch performance of lob data Rebuild lob table space in place (no unloadreload) No delete/define cluster : recover utility was needed to change priqty & secqty allocated values and number of extents 47 The REORG TABLESPACE utility cannot be used to change the allocated space because the underlying VSAM data sets are not deleted and redefined during reorg. The only way before to do this was using the RECOVER utility. Now UNLOAD/RELOAD can be used as an alternative 47

48 Lessons learned ; recommendations TABLE changes that require UNLOAD/RELOAD Column changes that required drop/create table were painful if the table also contained lob columns : Must also recreate all auxiliary tables and indexes How to unload/load the lob data if lob columns > 32 K? Idem for ddl changes to the base table space that required a recreate 48 48

49 Lessons learned ; recommendations TABLE changes that require UNLOAD/RELOAD Before PK10278 : use shadow table : Create identical shadow table Copy lob data using insert Repopulate after drop/create with insert from shadow table With PK10278 : use unload/reload Support in DB2 Admin Tool to be expected end The easiest way to unload/load lob data if no proper tool available was to keep the lob data in DB2 format by copying it to an identical shadow table. 49

50 Unloading and Loading LOB data Conclusions

51 Unloading and Loading LOB data PK22910 introduced File Reference Variables BLOBF,CLOBF & DBCLOBF to allow LOB data to be loaded and unloaded from a separate file PQ90263 introduced improved LOB handling for the CROSSLOADER 51 51

52 Unloading and Loading LOB data Use unload/reload instead of shadow tables when altering tables that require drop/recreate Use unload/reload instead of recover to redefine the underlying VSAM data sets Wait for implementation in DB2 tools 2007 Look out for DB2 V9 for much more enhancements on LOBs 52 52

53 Reference material DB2 V7, V8 and V9 manuals for z/os APAR text PQ90263 and PK22910 IBM redbooks SG Large Objects with DB2 for z/os and OS/390 (V7) New! SG LOBs with DB2 for z/os: Stronger and Faster (V9) 53 53

54 54 54

55 Session G09 Unloading and Loading LOB data in DB2 becomes reality Davy Goethals Arcelor Technologies 55 55