Db2 V12 Gilbert Sieben

Transcription

1 Db2 V12 Gilbert Sieben

2 Agenda 1. Time line 2. Premigration checks 3. Migration to V12 4. Measurements 5. New Features 6. Lessons learned Company 2

3 1. Time line Project of 1 year, 300 Mandays, described in a Project Charter, monthly reporting to steering committee. Migration in phases (starting from V11 FFM) TREL Nov 2017 : Db2 tools TREL Jan 2018 : early code TREL May 2018 : V12R1M100 : 1/3 Dev, 15/3 Acpt, 5/5 Prod TREL June 2018 : V12R1M500 : 17/5, 3/6 Acpt, 16/6 Prod Sandbox3 : to M100 and M500 : todo Company 3

4 1. Time line : Db2s to migrate # subsyst. 2-way DS 4-way DS Sandbox 4 2 DEV 14 2 ACPT PROD Company 4

5 2. Premigration checks Cfr. Installation & Migration Guide for checklist Release incompatibilities : simple TS, explain tables in old format, Packages that access the catalog BIND/REBIND Packages/Plans? DBAs were involved They got output of premigration checks They schedule (re)binds in applicative releases All products/tools (>40) that use/access Db2 were checked for V12 tolerance/compatibility were upgraded if needed at TRELs Remark : >6 months elapsed, a lot of effort! Company 5

6 3. Challenges Planned outage because of CATMAINT job? In V11 we did, both at CM and FFM Window between 01:00 and 03:00 including migration and testing by DBA s and Business In V12 we didn t 20 sec for M100 for our largest catalog But (to be continued ) Company 6

7 4.1. Migration to M100 on DEV DB2 dumps when executing query in V12R1M100 : Usermod (applied immediatly), APARFIX (with M500), UI55366 Query : select count(1) from (select NAME,(case when EXISTS(select 1 from sysibm.sysdummy1) then '1' else '2' end) from sysibm.systables where NAME = 'X') DBM1 creates a 0C : UI55990 Company 7

8 4.2. Migration to M100 on ACPT During migration Catmaint V12R1M100 fails with REASON 00C90101 ERQUAL 5020 (OMG!) PMR opened : looks like PI93894 : bad value in the SCA record for table space DSNDB08.D8TS1 before and after Db2 start. Fallback to V11 for 2 non-data sharing Db2s (scenarios were in place and tested before!). We didn t migrate the 3 others. Solved via usermod 10 days later : migrate to M100 on non-data sharing Db2s (IBM Dev worked around the clock to deliver AI94553) UI55070 Remark : -TERM UTIL needed if catmaint is not restarted (not documented) Be sure all utilities are terminated in V11 (no V11 UTIL can be terminated under V12 so fallback to V11 needed) Company 8

9 4.3. After migration to M100 on ACPT Issues with UR : UR? : We always use UR, also for reading in our programs, to have a minimal impact to online workload regarding locks. Impacted : all UNLOAD utilities with SHRLEVEL CHANGE ISOLATION UR Not all records are read (= records altered by 1 subsystem of a Data Sharing group and thus registered in the Coupling Facility) Workaround : mass update UNLOAD jobs with SHRLEVEL CHANGE ISOLATION UR REGISTER YES 1100 jobs in PROD APAR PI99075 / UI57005 Company 9

10 4.3. After migration to M100 on ACPT S0C1 dump when executing query in V12R1M100 : workaround : add rtrim function to query + alter hidden zparm. APARFIX PI95129 applied. Afterwards UI55397 DIST address space dumps after migrating to V12R1M100 : CASE expression : UI alter hidden zparm DB2 supplied java stored procedures address space loops : workaround : set the NUMTCB to 1 Resource unavailable for EDM DYNAMIC STATEMENT SPACE dynamic statement cache cleanup mechanism is not working anymore Increased size to 4G but not sufficient workaround: disable the dynamic statement cache (zparm ACPT warehouse) Solution : PI91054 / UI57563 Company 10

11 4.4. Migration to M100 on PROD Executed on Saturday evening Sunday morning : issues with impact during 3 hours PMR Sev 1 opened lock requests (the vast majority) are issued for Space Map pages Db2 tries to store the V0 OBDREC info in a system page (new V12 feature and retrofitted to V11) and that the page set partition does not have enough space to store this info. online reorgs of the UTS PBG TS during the morning solved the issue Company 11

12 4.4. Migration to M100 on PROD Monday morning : issues with P-locks, severe impact! PMR Sev 1 opened 3 online reorgs of the PBG UTS TS during the morning to solve the issue 46 online reorgs during the evening + testing by Business V12 : UI57257 (TREL1811) Company 12

13 4.4. Migration to M100 on PROD Page p-locks in batch jobs Very high impact Cause : overflow rows Bypass : reorg the involved TS and ALTER TS PCTFREE FOR UPDATE (to reserve space) Solution : usermod (TREL1806) APARfix (TREL1811) Company 13

14 4.4. Migration to M100 on PROD Excessive getpages after initialisation of FTB. Very high impact Cause :? Not reproducable re-enabling FTB is not an option Bypass : turn of FTB for this particular subsystem Solution : UI53014 for other FTB issue Company 14

15 4.4. Migration to M100 on PROD Data inconsistency : 00C90101 : data of 1 index is corrupt High impact Cause : regression on spacemap page Bypass : rebuild IX Solution : UI57598 (TREL1811) Company 15

16 4.4. Migration to M100 on PROD Select in batch gets 811 but there are no duplicates High impact Cause : wrong leaf page when FTB enabled Bypass : disable FTB (was not an option for us) Solution : UI53014 (TREL1806) Company 16

17 4.5. Migration to M500 on DEV No issues Company 17

18 4.6. Migration to M500 on ACPT All load utilities on PBG UTS that result in an extra partition : resource unavailable with resource type and no name filled in Also called DBD puffing' First work around : ALTER TABLE...ADD PARTITION (but generates unneccesary partitions) Second workaround : ALTER TS PRIQTY Our solution : ALTER ALL PBG and PBR UTS TS SEQ QTY TS on PROD! UI56192 Company 18

19 4.7. Migration to M500 on PROD Issues with medium impact : problem with reorg job Hung remote thread consumes a lot of cpu runstats ends with 04E-00C90101 Db2 12 message DSNI011I PAGE TEMPORARILY UNAVAILABLE DBM1 creates a 0C Company 19

20 4.8. Problem solving Follow-on PMRs : each TREL we roll out the most recent maintenance On the planning : customer profiling Sending Catalog/Directory and certain queries to IBM A workshop with experts from IBM and KBC to explain our environment We asked IBM if it is possible to collect Db2 usage info and share that (so, if you get that question, please respond) Company 20

21 5. Measurements : expectations System and OLTP : - 2-3% CPU reduction without Index In-Memory feature % CPU reduction using Index In-Memory feature Query performance : % without changing access path % with new access path A baseline measurement is done upfront Company 21

22 Measurements : results o V12R1M100 (TREL1805 on 5/5) and V12R1M500 (TREL1806 on 16/6) We didn t see any effect on average TRX CPU usage o 26/5, 23/6 and 25/8 : RAC on PROD : o more effect on average TRX CPU usage o o A very hard interpretation of the numbers because of a lot of issues after M100 and all necessary workarounds and (temporary) maintenance 18/08 : TREL1808 on PROD o o Extra maintenance because Performance issues Dynamic Statement Cache issues (went to 4Gig and was not cleared)

23 Measurements : results

24 Measurements : results

25 Measurements : Total CPU

26 Measurements : ziip

27 Measurements : DBM1 ziip

28 Measurements : real memory

29 Measurements : Invalid SPROCs

30 Measurements : RID pool

31 Measurements : RID pool o = internal Record Identifier o Expected increase of 40% because of extension from 5 to 7 bytes. o Only visible after M500 o With TREL1808 ZPARM MAXRBLK on PROD increased (with 60% on PROD)

32 Measurements : RID pool

33 Measurements : latch contention relief Latch contention classes : in TYPE 100 records LC14 buffer manager latch : QVLSLC14 LC19 log latch : QVLSLC19 : removed?? LC23 page latch timer : QVLSLC23 LC24 EDM latch : QVLSLC24

34 LC19 log latch After M100 : still (but less) LC19 (so not totally removed )

35 LC14 buffer manager latch Remark. : decrease after M100 on May 5 Remark : increase LC14 buffer manager latch : QVLSLC14 since M500! (no explanation yet) 35

36 LC23 page latch timer

37 LC24 EDM latch Remark : strong decrease after M100 on May 5 = normal because EDM pool is on other location and therefore handled by Storage Manager

38 6. New features : We start by using defaults and out-of-the box coding New features are subject of study (not in project scope!) In-memory bufferpools (PGSTEAL(NONE) option of ALTER BP) In-memory index optimization using index Fast Traverse Block (FTB) Application related features (study by DBAs): SQL performance enhancements Implementation only at TREL or RAC Still an open question : how to deal with Continuous Delivery Company 38

39 7 Summary Company 39 Hardest upgrade ever for (just the four of) us Had to open 4 SEV1, 12 SEV2 and 7 SEV3 PMRs for a lot of new issues We had to convince all management levels to proceed the upgrade to M500 We were backed up by the management of our department A lot of (technical) explaining to do, even to our business Great cooperation between system engineers and DBAs, the involved development teams, the colleagues of our department Good cooperation with IBM Issues picked up immediatly Got an advocate (Bart) assigned Conf call and meeting with Jeff Josten + follow up of status of PMRs

40 8. Lessons learned: Have fallback scenarios ready and tested Be current on maintenance Allthough some PTF which caused problems were already retrofitted in V11 (would have popped up during our half year maintenance) Have a good relationship with advocate Bart Company 40

41 Questions? Company 41