DB2 UDB Monitoring with Scripts

Size: px

Start display at page:

Download "DB2 UDB Monitoring with Scripts"

Silvester David Dennis
6 years ago
Views:

Platform: DB2 UDB for Linux, UNIX, and Windows DB2 UDB Monitoring with Scripts Thomas Horner Senior DBA, S1 Corporation Session: D11 May 25, 2005 at 3:30 PM This presentation will cover in detail ten

1 Platform: DB2 UDB for Linux, UNIX, and Windows DB2 UDB Monitoring with Scripts Thomas Horner Senior DBA, S1 Corporation Session: D11 May 25, 2005 at 3:30 PM This presentation will cover in detail ten shells that I have implemented to monitor our DB2 UDB production databases on AIX. These shells are quick and easy to implement, and for our DB2 installations give us warning of potential or real problems. They include full backups, log backups, update statistics and reorg checks, get status, and file cleanup. This seminar will go through each script in some detail, showing what they do and how to implement them in an AIX/UNIX environment.

2 Overall Objectives To prevent the unnecessary late night page or cell phone call by using these scripts A quick and easy way to set up administration for both test and production servers To be proactive in monitoring database space and other items in order to avoid users of your database suffering an outage 2

3 Presentation Overview Purpose of these database administration shells Overall design philosophy of these shells Details of the shells Limitations of these shells Questions and comments Please note that these shells work only on AIX and UNIX 3 Original bullet points (very similar to slide s): 1. Purpose of these database administration shells 2. Overall design philosophy of these shells 3. Details of each shell 4. Limitations of shells 5. Questions and comments

4 Purpose of these Shells These shells enable us to automate basic database administration simply They perform daily database maintenance full backups, log backups, and update statistics low They perform weekly database maintenance update statistics high with a reorg check A scheduled job is run to gather database statistics and do snapshots to warn of potential issues A daily file cleanup shell archives information gathered by the other shells to be used for trending analysis There are 12 shells, 3 flat files, and a README file 4

5 Overall Design Philosophy of Shells Four calling shells are run on a scheduled basis These calling shells use the databases file to call the four main functional shells The databases file is a flat file that lists all the databases that need to be maintained by these shells It may not include all the databases in the system This file also controls what each function is to do for each database by passing certain parameters There is one other scheduled shell that runs once per day from cron to clean up and archive results files 5

6 Overall Design Philosophy of Shells (cont d) There are three other shells that do a quick status, a quick diagnostic check (a work in progress), and run special log backups and update statistics using the specials file All four functional shells have the no notification ability in them (see next slide) that enables the use of these shells for test environments where notifications may not be wanted Results of shells are stored in daily results directories Last week s results are stored in oldresults directories Archived results are stored in arcresults directories 6

7 Overall Flow of Primary Shells CRON or Schedule Program Backup_app.sh Once per day Logbackup_app.sh Once or twice per hour Updstats_app.sh Once per day Getstatus_app.sh Twice per hour Cleanup_files.sh Once per day RUN_BACKUP BACKUP_LOGS RUN_STATS GET_STATUS 7

8 No Notification Option At the beginning of each shell the alarm.nomail file is searched for, and if there, no notifications are wanted MAILFLAG shell variable is then set to on or off Before every statement where notification is to be sent, the MAILFLAG variable is looked at If MAILFLAG is off, the DBA or notification of the data center is not done If on, send and (if critical) notify data center You can simply remove the alarm.nomail file to start having notifications sent 8

9 Installations Installed on the following DB2 versions: DB2 UDB Version 7.2, on AIX 5.1 and 4.3 DB2 UDB Version 8.1 with FixPak 2 (don t ask) and FixPak 6, on AIX 5.1 and 5.2 These are installed on seven production servers and several test servers running over 40 production databases Requires or other notification means for the DBA team and data center (such as HP OpenView) Our data center is manned 24 hours per day, but the shells can page an on-call support person in lieu of that 9

10 Scheduled Runs of Calling Shells Four calling shells, which in turn call the actual functional shells, are run from scheduling tool like cron or Maestro The full backup is run once a day, and the log backups are run hourly or every half hour depending on database usage (server) The update statistics shell is run daily The gather status shell is run every half hour The cleanup file shell which cleans up and archives output files is run nightly and has no database access The four calling shells use the databases control file to call the functional shells, in sequence for each database 10

11 The databases Control File This flat file controls what the four major functions will do for each database it drives the shells It contains the name of each database, the instance owner, the database owner, the diagnostic file path, and options for the backup, logbackup, getstatus, and updstats functional shells With this control file, the same four functional shells can be used for single and multiple instance database servers This file can be changed any time to reflect production status (as databases are added or dropped, for example) 11

12 An Example of the databases Control File db20130 sdb20130 db2inst1 /home/db2inst1 prod yes yes low db2283p sdb2283a db2inst1 /home/db2inst1 prod yes yes low db40091 sdb40091 db2inst1 /home/db2inst1 prod yes yes low db4560 sdb4560 db2inst1 /home/db2inst1 test no no low db4902 sdb4902 db2inst1 /home/db2inst1 prod yes yes highck db4912a sdb4912a db2inst1 /home/db2inst1 prod yes yes low db81019p sdb81019 db2inst1 /home/db2inst1 prod yes yes low db81082 sdb81082 db2inst1 /home/db2inst1 no no yes highnock 12

13 Example Detail Flow of Scheduled Job Cron or Scheduling Tool databases file Getstatus_app.sh GET_STATUS run for each entry Results (output and log) stored in proper day under the results directory 13

14 Full Backup Shell Overview Up to two parameters can be passed to the program Name of database Type of backup Test to create a special backup in a unique directory Prod to create a regular daily backup in a production directory Offline to create an offline backup for a database without recovery logging, in a unique directory Can be a scheduled run, or run from the command prompt If run from command prompt, the shell will ask for the database name and type of backup (unless supplied by the user in the command) 14

15 Full Backup Shell Design If the type of backup is test or prod, an online backup is first attempted Most of our databases use RECOVERY logging If the online backup fails: An offline backup is attempted If the offline backup also fails (database is active, for example), the DBA team and data center is notified If the type of backup is offline, only an offline backup is attempted If this fails (database is active), the DBA team and data center is notified 15

16 Full Backup Shell Design (cont d) If the type of backup is test or offline : Backup is stored in a uniquely named directory Name of the directory is derived from the database name, process ID, and timestamp of run If the type of backup is prod : Backup is stored in a production backup directory named for the database This backup is overwritten each time another prod backup is performed The shell defaults to test type of backup 16

17 Log Backup Shell Overview Up to two parameters can be passed to the program Name of database Instance owner Can be a scheduled run, or run from the command prompt If run from command prompt, the shell will ask for the instance owner and then the database name 17

18 Log Backup Shell Design Shell determines which logs are still active and which ones can be archived (moved to an archive log backup directory) This is done using snapshot command to determine first active log number and using the log numbers imbedded in the log file names Shell moves the inactive logs to archive log directory and copies the active logs to a backup log directory Shell purges logs older than 6 days from the archive log directory The DBA team is notified about any issues that arise Inactive (or incorrect) database High percentage of log directory being used 18

19 Update Statistics Shell Overview Four parameters (or none) can be passed to the program Name of database Instance owner Database owner Type of update statistics Low perform run_stats on table with distribution (default) Highck perform run_stats on table with distribution and detailed indexes all shrlevel change, and when done with all tables do reorg check command: db2 reorgchk current statistics on table all Highnock perform run_stats on table with distribution and detailed indexes all shrlevel change, but do not do reorg check Highck is always used on Saturday so that there is at least one reorgchk per week 19

20 Update Statistics Shell Design Can be a scheduled run, or run from the command prompt If run from command prompt, the shell will ask for all four parameters from the user (unless supplied in the command) Shell validates the input parameters Shell gets the list of tables to run stats on, using the database owner supplied in the parameters, and places them in a file Only one owner is assumed for all the tables in the database Shell performs appropriate run_stats command using the database owner and table names from the file created above 20 db2 runstats on table ${DBowner}.${TableName} with distribution (low) db2 runstats on table ${DBowner}.${TableName} with distribution and detailed indexes all shrlevel change (high)

21 Update Statistics Shell Design (cont d) Once this is completed, shell checks for errors or warnings in output files (SQL2314W is ignored) If highck parameter was passed, the reorg check command is run using current statistics If Saturday, run the reorg check command regardless of the parameter value, but is run only once that day After the reorg check is completed, grep commands are run on the output file to count how many tables and indexes need reorganization and how many do not Report these results via to the DBA team 21 SQL2314W is warning message that indexes also need to be reorg d ( out of sync with table).

22 Get Status Shell Overview Four parameters (or none) can be passed to the program Name of database Instance owner Database owner Location of DB2 diagnostic and notify logs Can be a scheduled run, or run from the command prompt If run from command prompt, the shell must be supplied with all four parameters (no user input is requested) Shell validates input parameters as much as possible 22

23 Get Status Shell Design Several DB2 commands are run: db2 list active databases db2 get dbm cfg db2 get db cfg for ${DBNAME} db2 get snapshot for database on ${DBNAME} db2 list applications db2 list tablespaces show detail db2 list tablespace containers The more important information from the outputs of these commands are placed in the results file What is more important was a design decision 23

24 Get Status Shell Design (cont d) Shell then starts to check for possible issues Database not active Secondary logs being used (control file used) Large gap between first active log and first log in log directory Unusually large percentage of log space being used A high number of agents waiting for locks A high percentage of instance agents being used A high percentage of database agents being used Tablespaces running low (control file used) Tablespace(s) with unusual status flags (0x0800 is ignored) Possible long transaction agent holding onto a log for an unusually long time (control file used) 24 Status flag 0x0800 means tablespace is being backed up.

25 Get Status Shell Design (cont d) Shell uses control (flat) files to store information between executions to determine if a notification is needed Lowest percent space free for each database Number of secondary logs being used for each database Application ID of user holding log for each database The lowest space free file also prevents duplicate space warnings Sent only once per day unless percent free space goes even lower Shell uses DB2 commands to get tablespace details while determining amount of free space remaining 25

26 Get Status Shell Design (cont d) Shell does a df k to show overall file system usage Shell lists out the last 100 lines of the DB2 diagnostic log for quick review if necessary Results are stored in the appropriate results subdirectory for review (subdirectory name is based on day of week) The DBA team is notified of any issues via , and the data center is notified of any critical issues so that they can contact on-call support 26

27 Cleanup File Shell Overview This shell is all AIX/Unix commands no database access It moves the output files and logs (logs of shell results, not the database logs) from the results directory to the oldresults directory, and the oldresults information to the appropriate arcresults directory, based on the day it is being run There are 7 results directories, one for each day of the week Sun, Mon, Tue, Wed, Thu, Fri, Sat There are 7 oldresults directories, for each day of the week Sun, Mon, Tue, Wed, Thu, Fri, Sat 27

28 Cleanup File Shell Overview and Design There are 52 arcresults directories One for each week of the year (01 through 52) Note that the results and oldresults directories need to be set up before using this shell Design of Shell: The next day s results files (which are now 6 days old) are moved to the corresponding oldresults directory If it is Friday, before moving Saturday s files to oldresults, archive the existing files in oldresults/sat to arcresults by week number, but only move the last file of each type If the diagnostic and/or notify log is getting too large, start new one(s) after archiving old one(s) 28

29 Layout of results, oldresults, arcresults /support results oldresults arcresults Mon Tue Wed Thu Fri Sat Sun Mon thru Sun 01 thru 52 29

30 Special Shell and specials File The special_runs shell is run from cron every 10 minutes to do special log backups, update statistics low, or get status It uses the specials flat file (same format as the databases file) to control what to do Usually there is only one database in the specials file due to, for example, a migration to a new application version that has a large update Database backups are not done by this shell It uses the same functional shells that the calling shells use 30 Update statistics is only run on the half-hour (if asked for), and after log backup is completed Get status is only run on the hour (if asked for), and after log backup is completed

31 Diagnostic Shell and Quick Status Another team wrote the diagnostic shell, but I did make changes and corrections to it It can be used only with DB2 version 8.x, since it utilizes the snapshot tables It provides information on what could be causing an issue with the database, such as poor performance This is a work in progress, but thought I would include it here so that you might make use of it Finally, the get_quick_status shell calls the getstatus_app shell (same one that is in cron) to obtain a current status 31

32 Limitations of these Shells These shells are run on a timed basis, not on a demand basis You may need to change how often they are run based on your installation. For example, log backups may not run often enough for a heavy load or massive update If you do not schedule them right, functions may overlap which may cause an issue Certain thresholds and names are hard-coded in the shells and will need to be changed for your installation Update statistics shell looks for only one database owner Once we go to FixPak 7+ (version 8.2) with all its new features for logs, etc., some of the shells may change 32

33 Review Shells are run from cron or other scheduling tool (such as Maestro) which in turn call the actual functional shells The databases control file Full and log backup shells Get status shell Update statistics shell Cleanup file shell Special shell and specials file Diagnostic shell is a work in progress but may help you Some limitations of these shells 33

34 Questions and Comments? To get a copy of these shells, me at thorner@s1.com. I can package the files and send them to you via . Objective here was to be proactive in monitoring the DB2 databases and prevent that call or page at 3 AM. Thank you all for your attention. I hope that these shells enable you to keep better informed about the status of your production systems. 34

35 DB2 UDB Monitoring with Scripts Session: D11 Thomas Horner S1 Corporation 35

Lesson 9 Transcript: Backup and Recovery

Lesson 9 Transcript: Backup and Recovery Slide 1: Cover Welcome to lesson 9 of the DB2 on Campus Lecture Series. We are going to talk in this presentation about database logging and backup and recovery.