DB2 Stop & Start Deep Dive

Transcription

1 Isaac Yassin - IYC Session Code: B09 Wednesday, 10 November :45 10:45 Platform: z/os DB2 Stop & Start Deep Dive

2 Isaac Yassin IBM certified Solution Expert -2- The presentation material, code examples and timing measurements are given as-is, without any warranty of any kind, either expressed or implied. The author shall not be liable for any loss or damage caused directly or indirectly by using the examples given. Isaac Yassin is an IBM Gold Consultant and IBM Information Champion with 35 years of experience. Isaac works mainly as a consultant to many organizations, both in Israel and Europe, covering wide aspects of performance tuning, troubleshooting, system design and education. Isaac has been working closely with DB2 since it's debut and followed up all versions and is an IBM certified solution expert and IBM Certified Database Administrator for V8, 9 & 10. Isaac is a well known lecturer and conducts many courses dealing with computer science (mainly in the DBMS area). Isaac holds a bachelor degree in Economics and computer Science from BAR-ILAN University and holds several other titles from MAMRAM computer center at the IDF. Isaac serves as Israel RUG co-leader and serves on IDUG EMEA-CPC since 2005 IBM and DB2 are trademarks of IBM corporation. Copyright 2010 Isaac Yassin DBMS & IT Consulting Services 2

3 3 Agenda DB2 system Restart/Shutdown Checkpoint Commit Improving DB2 restart performance Speed up DB2 shutdown process 3

4 4 Checkpoint Longer time between checkpoints leads to more processing in DB2 when it needs to process the log 4

5 5 Taking Checkpoints At DB2 startup time When CHKFREQ value is reached When DB2 switches Active Log data set When DB2 is shutting down When we issue one of the following commands: -SET LOG LOGLOAD(0) -SET CHKTIME(0) -SET LOG SUSPEND in a non-data sharing environment 5

6 6 What does DB2 do at Checkpoint? Externalizes all changed data (except workfiles) to a GBP or tablespace / indexspace on DASD. Records the pageset status information in the log, including database pending writes. Detects candidate data sets for R/O switching. Records pageset exception status. Records not-completed UR status The status of all URs is recorded at system checkpoint time. Updates the LOGONLY recovery log scan starting point. Updates the data sets down level detection IDs. 6

7 7 CHKFREQ Every m minutes or n records Should be coordinated with PCLOSEN PCLOSET 7

8 8 Checkpoint penalty Checkpoint should not be done more than once a minute It is a disturbing process Prefer CHKFREQ of 3-5 minutes over CHKFREQ every m log records Example: CHKFREQ = 5 PCLOSEN = 12 PCLOSET = 60 In DB2 10 we should be able to use a combination of the lower of the two (minutes / log records) 8

9 9 Two Phase Commit DB2 supports the two-phase commit protocol 9

10 10 UR Unit of Recovery A unit of recovery (UR) is generated only for updating transactions But NOT for read-only transactions. Start UR DB2 writes a begin phase-1 log record and starts to perform phase-1 processing. If DB2 ends before it completes its phase-1 processing and writes the end phase-1 log record, the transaction is considered to be in inflight status. At restart, DB2 rolls back all the updates done by this inflight transaction to its last commit point. Log Inflight Inflight Start UR Begin phase-1 End phase-1 10

11 11 UR Unit of Recovery If DB2 stops after the end commit phase-1 log record is written, but before the begin phase-2, then the UR is considered to be in indoubt status. Start UR Log Log Start UR Inflight Begin phase-1 Inflight End phase-1 Indoubt Begin phase-2 11

12 12 UR Unit of Recovery The UR will be in-commit status as soon as DB2 writes the begin phase-2 log record. The UR is ended as soon as the end commit phase-2 log record is written after phase-2 is completed. Start UR Log End UR Log Inflight Inflight Start UR Begin phase-1 End phase-1 Indoubt Begin phase-2 Incommit End phase-2 12

13 13 The 5 phases of DB2 restart processing 1. Log Initialization - the start-up phase. 2. Current Status Rebuild - the analysis phase. DB2 is trying to determine the transaction status when it was last stopped. 3. Forward Log Recovery - performs redo operations for all database updates that have not been written to GBP or DASD. 4. Backward Log Recovery - performs undo operations for all database updates that are inflight and in-abort URs. If we have long running URs, this phase can run for a long time. 5. End Restart - finish processing. DB2 can start working. DB2 takes a system wide checkpoint at the end of this phase. 13

14 14 Phase 1 Log Initialization During log initialization, DB2 locates the last log RBA that was written before termination Logging continues at the next log relative byte address (RBA) after that. 14

15 15 Phase 1 Log Initialization (1) Starts the DB2 address spaces (MSTR & DBM1) and loads DB2 load modules Starts Internal Resource Lock Manager (IRLM) Open the BSDS and active log data sets. After the BSDS and active log data sets are opened, DB2 scans the log to determine the actual endpoint of the log. 15

16 16 Phase 1 Log Initialization (2) DB2 uses a pseudo log endpoint stored in the BSDS as the log scan starting point. The pseudo log endpoint value is updated when one of the following occur: The log output buffer is ready to wrap-around. The active log data sets is filled. The system checkpoint is taken. DB2 is terminated normally. 16

17 17 MSTR messages STC16099 DSNR001I #DB2P RESTART INITIATED STC16099 DSNR003I #DB2P RESTART...PRIOR CHECKPOINT RBA=0034C

18 18 Phase 2 Current Status Rebuild (1) During CSR phase, DB2 determines the status of objects and units of recovery (UR) at the time of termination. CSR phase also includes the forward log scan pass that is used to determine the transaction status and the attributes and the restart REDO log scan points for all the R/W pending Pagesets or Partitions. 18

19 19 Phase 2 Current Status Rebuild (2) DB2 scans the log, starting from the Begin Checkpoint log record for the last completed DB2 system checkpoint. The maximum number of log records that will be scanned is bounded by the DSNZPARM CHKFREQ value. This phase also rebuilds the Data Base Exception status Table (known as the DBET). DBET information is recorded in every system checkpoint. DBET contains, for example, information about RECP status In a non-data sharing environment, DB2 reconstructs the status information and build the list of URs from the log. In a data sharing environment, DB2 gets information from the shared communication area (SCA) structure in a coupling facility. 19

20 20 Phase 2 Current Status Rebuild (3) If the SCA is damaged then DBET log records will be used to rebuild the DBET. At the end of the CSR phase, console message DSNR004I is displayed with all the transaction status information showing how many transactions are inflight, how many are indoubt, etc. DB2 also determines how far it has to go back in the log to start the forward scan, which is the Phase 3 (Forward Log Recovery) of restart. 20

21 21 MSTR messages STC16099 DSNR001I #DB2P RESTART INITIATED STC16099 DSNR003I #DB2P RESTART...PRIOR CHECKPOINT RBA=0034C STC16099 DSNR004I #DB2P RESTART...UR STATUS COUNTS IN COMMIT=2, INDOUBT=0, INFLIGHT=1, IN ABORT=0, POSTPONED ABO STC16099 DSNR007I #DB2P RESTART...STATUS TABLE T CON-ID CORR-ID AUTHID PLAN S URID DAY B BATCH BATCHXXXXXX XXXXXX PLAN1 F 0034BE S CICS01 POOLXXXXXXX XXXXXX PLAN2 C 0034C16A624F S CICS02 POOLYYYYYYY XXXXXX PLAN3 C 0034C16A

22 22 Building the CSR Pseudo Log End BSDS Last Checkpoint Checkpoint Log end Log Log 22

23 23 Phase 3 Forward Log Recovery (1) During the FLR phase, DB2 completes the processing for all committed changes and database write operations. FLR phase starts processing at the checkpoint preceding the first log records for the oldest indoubt UR, or the oldest redo logpoint for any pageset, and proceeds forward in time from there. 23

24 24 Phase 3 Forward Log Recovery (2) DB2 does a REDO operation on all pending writes During FLR DB2 reads the log records (using fast log apply) and accesses the data pages to see if the changes actually were written to the database. This means that DB2 waits for the I/O DB2 compares the LRSN / RBA in the page header and the LRSN / RBA of the log record to find if the database update was successfully externalized to GBP or DASD. Changed pages are written back (asynchronously) 24

25 25 Phase 3 Forward Log Recovery (3) During phase 3 of restart in a data sharing environment, DB2 also purges the retained page p- locks since they are no longer needed to protect the unwritten data. DB2 also writes log records to change each in-commit UR to complete UR status. Fast Log Apply also reads and sorts log records before applying them. It also applies updates in parallel to the pagesets involved. 25

26 26 FLR example Committed transaction modified page P4, with 300 as the LRSN value of the log record. During restart, DB2 reads the log for the transaction that updated page P4. LRSN = 300 Next, It reads page P4 from the database and finds in the header that LRSN = 100 This means that the update to page P4 was committed, but the write back to DASD had not completed before DB2 went down. DB2 has to write the updated page to the database on DASD. For pages P6 and P9, no changes to the database are needed because the page LRSN values are the same as the log LRSN values. DB2 need to read the pages from DASD before it can decide whether or not any changes must be applied. 26

27 MSTR messages STC16099 DSNR001I #DB2P RESTART INITIATED STC16099 DSNR003I #DB2P RESTART...PRIOR CHECKPOINT RBA=0034C STC16099 DSNR004I #DB2P RESTART...UR STATUS COUNTS IN COMMIT=2, INDOUBT=0, INFLIGHT=1, IN ABORT=0, POSTPONED ABO STC16099 DSNR007I #DB2P RESTART...STATUS TABLE T CON-ID CORR-ID AUTHID PLAN S URID DAY B BATCH BATCHXXXXXX XXXXXX PLAN1 F 0034BE S CICS01 POOLXXXXXXX XXXXXX PLAN2 C 0034C16A624F S CICS02 POOLYYYYYYY XXXXXX PLAN3 C 0034C16A STC16099 DSNR005I #DB2P RESTART...COUNTS AFTER FORWARD RECOVERY 229 IN COMMIT=0, INDOUBT= STC16099 DSNR018I #DB2P RESTART...BACKWARD RECOVERY PROCESSED FROM RBA 0034C166D574 TO RBA 0034BE STC16099 DSNR006I #DB2P RESTART...COUNTS AFTER BACKWARD RECOVERY 275 INFLIGHT=0, IN ABORT=0, POSTPONED ABORT= STC16099 DSNG007I #DB2P DB2 CATALOG LEVEL (910) CODE LEVEL (910) MODE (N) STC16099 DSNR002I #DB2P RESTART COMPLETED STC16099 #DB2PRECOVER POSTPONED STC16099 DSNV434I #DB2P DSNVRP NO POSTPONED ABORT THREADS FOUND STC16099 DSN9022I #DB2P DSNVRP 'RECOVER POSTPONED' NORMAL COMPLETION 27 27

28 28 Phase 4 Backward Log Recovery During BLR, all DB2 changes that were done for in-flight or in-abort UR are backed out. In-abort UR is generated when DB2 fails during a transaction rollback. In-flight UR occurs if a transaction has not committed or DB2 has not completed its phase-1 commit process when DB2 ended. During BLR DB2 scans the logs backwards for all log records belonging to the in-flight and in-abort transactions and backs out any changes that had been applied to the database, while writing another log record to compensate the (uncommitted) change made earlier. 28

29 29 BLR example The in-abort UR that updated page P6 has a log LRSN value of 500. DB2 writes a compensation log record for this in-abort UR at log LRSN value 800 and changes the P6 page LRSN value from 500 to 800 Depending on whether DB2 consistent restart (later) is being used, DB2 restart might end before all UNDOs have been finished. Compensation Log Record 29

30 30 Phase 5 End Restart Processing Any indoubt transaction causes DB2 to issue a write claim on the involved tablespace so utilities will not take over until the indoubt is resolved. No utility can be run until indoubt URs are resolved. During the restart process, DB2 opens only the data sets that have restart pending activities. DB2 does not open the objects for read-only table spaces or R/W table spaces without a log record for that object. 30

31 31 Phase 5 End Restart Processing Prior to V9 DB2 tried to close SYSLGRNX entries For R/W object without a log record, For an object without inflight URs or outstanding uncommitted updates In DB2 V9, this processing is done at the first system checkpoint after the restart Allows a faster restart. All data modified during the BLR phase is externalized to GBP/DASD DB2 also purges all remaining retained locks and than triggers a system checkpoint to record the current status of the DB2 subsystem. After the End Restart phase, DB2 is ready for work. 31

32 32 DB2 restart methods DB2 can restart in several various methods. DB2 group restart DB2 conditional restart DB2 consistent restart DB2 restart light 32

33 33 DB2 group restart DB2 Group restart happens when: The restarting member determines during the DB2 Initialization phase that the SCA or the DB2 lock structure must be recovered None of the other members are active, If at least one member is active that can access both structures successfully, no group restart takes place. Recommendation Use manual restart It is faster Data availability is better as retained locks that are held for non-starting members are released faster 33

34 34 Important (group restart) A group restart means that the SCA or DB2 lock structure are recovered from the logs of the members of the data-sharing group by those members that are restarted. It does not mean that all members of the datasharing group are automatically restarted. 34

35 35 DB2 conditional restart (CRCR) A conditional restart is a special process, activated by a conditional restart control record (CRCR) in the BSDS, which is used when you need to skip some portion of the log processing during DB2 restart (perhaps because of a damaged log data set). Performing a conditional restart: While DB2 is stopped, run the Change Log Inventory Utility (DSNJU003) using the CRESTART control statement to create a new conditional restart control record in the BSDS. Restart DB2. The types of recovery operations that take place are determined by the current CRCR. 35

36 36 DB2 conditional restart (CRCR) Choosing this point of consistency carefully is very important. A useful approach is to run the DSN1LOGP utility and review a summary report of the information contained in the log before setting the conditional restart. Important Although conditional restart is faster than normal restart (because it skips some portion of the log processing), data in the associated objects can become inconsistent if a conditional restart skips any database change log record. Any subsequent attempt to process them for normal operations may cause unpredictable results. 36

37 37 DB2 consistent restart (1) This function allows DB2 to start up more quickly after a subsystem failure by limiting the backout activities for long running applications that issue very infrequent or no commits. The DSNZPARM LBACKOUT and BACKODUR parameters can be modified based on your workload characteristics. For non-data-sharing DB2 marks the entire object as Restart Pending and unavailable, so those objects touched by the long running URs will not be available until the long running URs are backed out. 37

38 38 DB2 consistent restart (2) For data-sharing: Pages updated by long-running transactions are protected by retained locks. Retained locks are not purged End Restart They are held until the backout is finished. Retained locks are on page or row level and not on pageset level. Long-running jobs that issue infrequent commits, delays DB2 start Using consistent restart allows DB2 to access all other objects except those affected by long running non committing jobs. 38

39 39 BACKODUR (Backout Duration) (1) During restart, backward-log processing continues until: All inflight and in-abort URs with update activity against the catalog or directory are backed out. All inflight and in-abort URs that only update temp tables are backed out. The number of log records processed is equal to the number specified in BACKODUR multiplied by the number of log records per checkpoint (CHEKFREQ). If the checkpoint frequency type is minutes, the default value of 50,000 log records is used to calculate the number of log records to process. (way too low) 39

40 40 BACKODUR (Backout Duration) (2) Inflight and in-abort URs that are not completely backed out during restart are converted to postponed-abort status. They are put into restart-pending (RESTP). This state blocks all access to the object except The RECOVER POSTPONED command state Automatic back-out processing performed by DB2 if LBACKOUT=AUTO. A table space can be in restart-pending mode, while the associated index spaces are not Happens if a postponed-abort UR updates only non-indexed columns of a table in a table space. Index-only queries can still be executed 40

41 41 LBACKOUT (Limit Backout) NO means that DB2 backward-log processing handles all inflight and inabort units of recovery (URs) before DB2 completes its startup YES postpones back-out processing for some units of work until we issue the command RECOVER POSTPONED. AUTO postpones some backout processing, but automatically starts the back-out processing when DB2 restarts and begins acceptance of new work. Some backout work might be delayed and will be triggered automatically after DB2 comes up. With YES or AUTO, back-out processing runs concurrently with new work. Datasets or partitions with pending back-out work are unavailable until their back-out work is complete. 41

42 42 DB2 restart light (1) Use LIGHT (YES) for a data-sharing to quickly bring up a DB2 member solely to recover retained locks. When DB2 is started with LIGHT(YES), if indoubt URs exist at the end of restart recovery, DB2 will remain up and running so that the indoubts can be resolved Either automatically through re-synch processing with the commit coordinators Or manually through the -RECOVER indoubt operator command. After all the indoubt URs have been resolved, the LIGHT(YES) DB2 member self-terminates. APAR PK29791 introduced a new LIGHT(NOINDOUBTS) option on the -START DB2 command. 42

43 43 DB2 restart light (2) LIGHT(NOINDOUBTS) This option specifies that DB2, during a restart light, does not wait for indoubt units of recovery to resolve before it terminates. Restart Iight mode is intended only for a cross-system restart of a system that does not have adequate capacity to sustain the DB2 and IRLM in the event of a failed z/os system. A data sharing group member started with the LIGHT option is not registered with the Automatic Restart Manager (ARM). ARM does not automatically restart a member that has been started with LIGHT(YES). 43

44 44 DB2 shutdown (1) DB2 terminates normally in response to the -STOP DB2 command. If DB2 stops for any other reason, the termination is considered abnormal. 44

45 45 DB2 shutdown (2) At normal system shutdown, DB2: Takes a shutdown checkpoint and updates the BSDS. Quiesces all transactions Flushes out all changed pages from the local bufferpools to a GBP (for shared objects) or to DASD. Closes all its open data sets. All URs are completed, except of indoubt URs (if exist). If indoubt URs do exist, DB2 records the indoubt UR status in the Shutdown Checkpoint log record. 45

46 46 DB2 shutdown types -STOP DB2 MODE (QUIESCE) The default (work in process is quiesced) -STOP DB2 MODE (FORCE) Rollback (work in process is aborted) 46

47 47 DB2 Abnormal Termination Happens when DB2 does not, or cannot, terminate in an orderly way leaves data in an inconsistent state: Units of recovery might be interrupted before reaching a point of consistency. Committed data might not be written to external media. Uncommitted data might be written to external media. 47

48 48 Improving DB2 Shutdown (1) Lower DSMAX leads to faster shutdown time Lower DSMAX means more pseudo-close and physical close during execution A good balance is needed 40K open page sets closes in ~50 seconds (z/os 1.10) Each pseudo-closed dataset is physically closed on next PCLOSET/PCLOSEN if not written into meanwhile 48

49 49 Improving DB2 Shutdown (2) DDCONS NO (for not consolidating SMF 30 information) is faster than DDCONS YES CASTOUT (NO) in Data-Sharing Speeds shut down process Retains IX mode P-lock on last updated page sets in that member To be used only for short time member shut down Not to be used if shutting down more than one member Not Data-Sharing? Put DB2 datasets in the SYSTEMS exclusion resource name list (RNL) Changes the ENQ scope to one system 49

50 50 Improving DB2 Startup (1) DB2 9 new command -ACCESS DATABASE Example: -ACCESS DATABASE (DB000001) SPACENAM (SP00001) MODE(OPEN) -ACCESS DATABASE (DB000001) SPACENAM (*) MODE(OPEN) Using SELECT command is much faster (60% of access database) 50

51 51 Improving DB2 Startup (2) Fast Log Apply default is 100MB since V8 Avoid long running UR URCHKTH Display long-running UR on console Default 0 (never) Recommend every 5 checkpoints URLGWTH Display long-running INFLIGHT UR on console Default 0 (never) Recommend every 100,000 log records YMMV 51

52 52 Improving DB2 Startup (3) More active logs (93 in DB2 9) Have at least a full working day on active logs, preferably two One long-running looping UR can easily fill that Archive to DASD and migrate (HSM) If not enough DASD,use more for ACTIVE (up to max.) Archive one copy to DASD other to Tape/VTS Reading Archived logs from tape at startup can be a real killer 52

53 53 Improving DB2 Startup (4) Have bigger GBP GBP are accessed in microseconds (DASD in milliseconds) Most datasets will be accessed faster Does not apply to GBP-dependent objects as they are forced-at-commit For non-gbp-dependent objects or uncommitted transactions we need to read the logs and examine whether updates were being externalized to GBP or DASD. 53

54 54 Session: B09 DB2 Stop & start Deep Dive Isaac Yassin 54