V9 Migration KBC. Ronny Vandegehuchte

Transcription

1 V9 Migration KBC Ronny Vandegehuchte

2 KBC Configuration 50 subsystems (15 in production) Datasharing (3 way) 24X7 sandbox, development, acceptance, production

3 Timings Environment DB2 V9 CM DB2 V9 NFM Sandbox 01/02/2 15/04/2 Development 17/03/2 20/05/2 Acceptance 31/03/2 29/05/2 Production 16/05/2 13/06/2

4 V9 is using more 32K for SORTing Development subsystem(s) : 4K sort files isolated in BP3 3 files, each size of 3000 cyls each 3000 buffers BP Thresholds : VP SEQUENTIAL=100 DEFERRED WRITE = 90 VERTICAL DEFERRED WRT = 85, 0 32K sort files isolated BP32K3 1 file, 100 cyls 50 buffers BP thresholds Defaults Rarely used..

5 V9 is using more 32K for SORTing After migrating development systems to CM SQLcodes K resource DSNP007I -SDTC DSNPXTN0 - EXTEND FAILED FOR 995 SSYSDTC.DSNDBD.SWDSDTC.DSN32K01.I0001.A001. RC=00D70027 CONNECTION-ID=BATCH, ID CORRELATION-ID=PD0Y1ODO, ID LUW-ID=BEBBCM01.A2SDTC.C3E6CF9D5628=4763 Increased BP32K3 size (50->500->2500) Increased to 3 workdatasets, 3000cyls each Changed BP32K3 thresholds to BP3 thresholds Got problem on 32K sort tables, not accessable, no locks, no use Stop/start didn t work, need to recycle DB2 system

6 V9 is using more 32K for SORTing DSNPI -SDTC DSNPXTN0 - REQUIRED USER-DEFINED DATA 931 SET NOT AVAILABLE- SSYSDTC.DSNDBC.SWDSDTC.DSN32K02.I0001.A002. RC=00D70002 CONNECTION-ID=DB2CALL, CORRELATION-ID=CIIT0OD, LUW-ID=BEBBCM01.A2SDTC.C4120A6B41FE= DSNP007I -SDTC DSNPXTN0 - EXTEND FAILED FOR 932 SSYSDTC.DSNDBD.SWDSDTC.DSN32K02.I0001.A002. RC=00D70002 CONNECTION-ID=DB2CALL, CORRELATION-ID=CIIT0OD, LUW-ID=BEBBCM01.A2SDTC.C4120A6B41FE= Defined the sort table spaces, cyls(3000,0) which is more then 2 Gb for primary When we filled 2 Gb (usually 2913 cyl), we tried to extend to the second pageset which hadn't been defined. Defined sortfiles as 2900cyls! Due to create of 32K3 spaces, they become segmented!

7 V9 is using more 32K for SORTing DPC1 600, , , , Millions A Wrts 200, , ,00 0 1/03/2 4/03/2 7/03/2 10/03/2 13/03/2 16/03/2 19/03/2 22/03/2 25/03/2 28/03/2 31/03/2 3/04/2 6/04/2 9/04/2 12/04/2 15/04/2 18/04/2 21/04/2 24/04/2 27/04/2 30/04/2 3/05/2 6/05/2 9/05/2 12/05/2 15/05/2 18/05/2 21/05/2 24/05/2 BP3 GETP BP32 GETP BP3 ASYC_W BP32 ASYC_W

8 Unicode improvements Date/time/timestamp are stored as UTF-8 in a unicode table Pre V9 : V9 : Conversions to UTF-16 are done by calls to z/os conversions services,.conversions cost Conversions to UTF-16 are done by DB2 it self

9 Unicode improvements Test case: Table CBTUNIDD has 39 Date Cols, and 40K rows. 1,8 1,6 Unicode improvements Date from UTF-8 to UTF-16 Select all rows and all 1,4 columns into PIC N(10), 1,2 1 via an COBOL pgm. 33% CPU decrease 0,8 0,6 CPU Conversions are done in DSNXRTIM instead of CUNMUNI 0,4 02 0,2 0 V8 V9

10 Rebind BEFORE migration We did get Abend0c4 in program s, on some acceptance&production p systems in packages that where bound on Pre V8 and even V8 (UK44259) rebind solved all cases Wrong query results!! in packages that where bound on Pre V8 and even V8 rebind solved all cases To avoid problem s after migration : rebind before migrating to V9

11 Rebind AFTER migration Invalid SPROC s Invalid Sproc DPC1 SDPC Millions 02/04/2 06/04/2 10/04/2 14/04/2 18/04/2 22/04/2 26/04/2 30/04/2 04/05/2 08/05/2 12/05/2 16/05/2 20/05/2 24/05/2 28/05/2 01/06/2 05/06/2 09/06/2 13/06/2 17/06/2 21/06/2 25/06/2 29/06/2

12 Rebind AFTER migration Invalid SPROC s A pgm uses 19% more CPU Millions CPU SQL_CALLS

13 EDM Pool Structure V9 (no Rebind) EDM_SKELETON_POOL BP8K0 SKPT BP0 2GB PT SPT01 DBM1 EDMPOOL

14 EDM Pool Structure V9 (no Rebind) EDM_SKELETON_POOL : DSNZPARM based on calculations, 6389KB From DB2 stats : Max(QISESKPT+QISESKCT) KB EDMPOOL = KB down to 40000KB WHAT IS/WAS THE IMPACT? Nothing. Or HIGHer CPU usage in DBM1 HIGHer CPU usage in the application HIGHer I/O rates LOWer BP8K0 hitratios

15 EDM Pool Structure V9 (no Rebind) Statistics interval 10min is to big!! EDM Pool From DB2 Statistics EDMPAGES CT-PT Pge EDMFULL :10:46 1:10:46 2:10:46 3:10:46 4:10:46 5:10:46 6:10:46 7:10:46 8:10:46 9:10:46 10:10:46 11:10:46 12:10:46 13:10:46 14:10:46 15:10:46 16:10:46 17:10:46 18:10:46 19:10:46 20:10:46 21:10:46 22:10:46 23:10:46

16 EDM Pool Structure V9 (Rebind) EDM_SKELETON_POOL BP0 BP8K0 SKPT PT 2GB PT SPT01 DBM1 EDMPOOL

17 Increased Logging Activity Load table Resume No Replace Log No Part 1 Table has 5 NPI s s, logging occurs for these NPI s Same # of rows (no changes of the datamodel) V8 : GB -> V9: 825GB 8.25 PMR to IBM This is caused by the structure modification of the NPI (i.e. delete leaf page). This in terms is a secondary effect of a new algorithm we use to 'build' the NPI during load part replace operation in DB2 9. APAR PK83683 UK50265

18 Access Path Changes NO mass rebind, risk to high Look at what happens Solve it case by case

19 Access Path Changes Static SQL Bad access path after first bind in DB2 9 Runstats t + REBIND : NOT ok Advisory runstats OSC + REBIND : Problem solved RUNSTATS TABLESPACE DF2BR01.SF23060 TABLE(PIBBRP.TF2BR3060) SAMPLE 5 COLUMN(CMED,RVERS) COLGROUP(CMED) FREQVAL COUNT 10 SORTDEVT SYSDA SORTNUM 4 INDEX(PIBBRP.XF2BR30601 HISTOGRAM NUMCOLS 1 NUMQUANTILES 20, PIBBRP.XF2BR30600 KEYCARD)

20 Access Path Changes Static SQL In most cases better performing SQL Less Sorting Nested Loop + sort -> Hybrid Join (no sort) In some cases use of other IXs Other access sequence More Multiple IX Access : Sometimes simple rebind solves the problem In other cases Advisory runstats OSC + REBIND

21 Access Path Changes Dynamic SQL DB2 V8 and Tivoli Directory Server (LDAP): Use of fdynamic statement cache Reduce dynamic overhead => Trying to guarantee stable performance No regular collection of DB2 statistics Unique collection of DB2 statistics with specific colgroup stats => Trying to avoid accesspath changes

22 Access Path Changes Dynamic SQL DB2 V9 CM and LDAP Bad LDAP performance => new collection of DB2 stats with colgroup stats (cfr V8) did not help => collection of stats adviced by OSC Statistics Advisor helped in certain cases, but due to data skewing still some problems we bound DSNCLINF with reopt(always) => initial CPU overhead 10% - no use of dynamic cache anymore - every stmt is reoptimized but stable performance! but worse than in V8

23 Access Path Changes Dynamic SQL DB2 V9 NF and LDAP We tried to bind DSNCLINF with reopt(none) => again bad accesspaths We rebound DSNCLINF with reopt(always) p( y) => performance OK but initial CPU overhead 10% + day after day increase of LDAP DB2 CPU (open PMR) Ultimately DSNCLINF was bound with reopt(once) => performance OK => LDAP CPU usage down to the CPU levels of DB2 V8

24 Access Path Changes Dynamic SQL LDAP Resource Usages per Month CPU Seconds s M illio n s SQLCALLS GETPAGE INDB2_CPU INDB2_TIME

25 Access Path Changes Dynamic SQL Support IBM Difficult to address Access Path problems/changes Other access path changes? YES (data warehousing) Use OSC statistics advisor!

26 Remove BUILD2 Phase in Reorg DB2 V9 removes the need for a BUILD2 phase, by reorganizing gthe whole of the NPI. The reorganized NPIs are then switched in the SWITCH phase along with the other data sets. The only outage is during the relatively short SWITCH phase Relieve of massive GBP writes, short on storage Our benefit. None got worse We run reorg part x jobs in parallel.. Less Concurrency between individual concurrent REORG Part job s (DSNU180I) Critical reorg s jobs do not run..

27 Remove BUILD2 Phase in Reorg Users should be aware of the following implications (V9 Technical Overview SG247330) DB2 will need additional temporary disk storage for the shadow data sets for each NPI. The cost of the REORG will increase, as the build of the entire NPI will require more CPU than the updates that occurred in the BUILD2 phase. As the entire NPI will be built and then switched, it is no longer possible to run reorganizations of different parts in separate jobs in parallel. Jobs that ran in parallel will need to be changed. We REORG on a Partition Level Need to change to REORG flow

28 Other problems OSC problem: LOB-tablespaces FULL First tried to drop those tablespaces But EXPLAIN failed Problem solved by sheduling clean-up procedures UK54574

29 System Monitoring CPU I/O/ Memory

30 CPU DBM1_TCB DBM1_SRB MSTR_TCB MSTR_SRB 26/06/2 14/06/2 02/06/2 21/05/2 09/05/2 27/04/2 15/04/2 3/04/2 22/03/2 10/03/2 26/02/2 14/02/2 2/02/2 21/01/2 9/01/2 28/12/2 16/12/2 4/12/2 22/11/2 10/11/2 29/10/2 17/10/2 5/10/2 23/09/2 11/09/2 30/08/2 18/08/2 6/08/2 25/07/2 13/07/2 1/07/2

31 300,00 250,00 200,00 150,00 100,00 50,00 0,00 I/O (Sync ASync) SYNC_R ASYNC_IO Millions 1/07/2 25/07/2 13/07/2 6/08/2 18/08/2 30/08/2 11/09/2 23/09/2 5/10/2 17/10/2 29/10/2 10/11/2 22/11/2 4/12/2 16/12/2 28/12/2 9/01/2 21/01/2 2/02/2 14/02/2 26/02/2 10/03/2 22/03/2 3/04/2 15/04/2 27/04/2 09/05/2 21/05/2 02/06/2 14/06/2 26/06/2

32 I/O (Prefetching) REQ_LIS REQ_SEQ REQ_DYN 26/06/2 14/06/2 02/06/2 21/05/2 09/05/2 27/04/2 15/04/2 3/04/2 22/03/2 10/03/2 26/02/2 14/02/2 2/02/2 21/01/2 9/01/2 28/12/2 16/12/2 4/12/2 22/11/2 10/11/2 29/10/2 17/10/2 5/10/2 23/09/2 11/09/2 30/08/2 18/08/2 6/08/ /07/2 13/07/2 1/07/2 0 Millions

33 Memory Memory /07/20 11/07/20 21/07/20 31/07/20 10/08/20 20/08/20 30/08/20 9/09/20 19/09/20 29/09/20 9/10/20 19/10/20 29/10/20 8/11/20 18/11/20 28/11/20 8/12/20 18/12/20 28/12/20 7/01/20 17/01/20 27/01/20 6/02/20 16/02/20 26/02/20 8/03/20 18/03/20 28/03/20 7/04/20 17/04/20 27/04/20 07/05/20 17/05/20 26/05/20 05/06/20 15/06/20 25/06/20 AUX_FR REAL_FR

34 IFCID 225 Note that IFCID 225 is moved from record type 102 to 100

35 Lessons learned Rebind BEFORE migrating to V9 Rebind AFTER migrating to V9 Optimizer needs more detailed statistics Size of EDM SKELETON pool! Adjust the 32K workdatasets Be aware of changed REORG NPI behaviour