Session: D12 Case Study: Home-Grown DB2 Monitoring and Related Tools Jayesh Thakrar The Warranty Group May 10, 2007 09:20 a.m. 10:20 a.m. Platform: DB2 for Linux, Unix and Windows 1
Topics 1. Presentation Objectives 2. DB2 Monitoring - Needs and Gap Analysis 3. Buy versus Build Solution 4. Discuss a Home-Grown Solution 2 2
Topics 1. Presentation Objectives 2. DB2 Monitoring - Needs and Gap Analysis 3. Buy versus Build Solution 4. Discuss a Home-Grown Solution 3 3
Presentation Objectives Empower / enable you to - Perform DB2 monitoring needs & gap analysis Survey & compare available packaged solutions Decide between buy versus build Walk-through a sample home-grown solution - Objectives & needs Design and packaging Deployment Highlight DB2 LUW built-in monitoring features 4 4
Topics 1. Presentation Objectives 2. DB2 Monitoring - Needs and Gap Analysis 3. Buy versus Build Solution 4. Discuss a Home-Grown Solution 5 5
Needs Analysis - Background Evaluate Current Environment - Hardware (servers), software and applications List of RDBMS platforms & versions to support (If supporting more than just DB2 LUW) Count of databases & platforms that need monitoring (e.g. 100 Production, 60% of Test & UAT, 20% of Dev) Evaluate scope of responsibilities - Databases or application systems System resources (database, storage, etc.) Backup and recovery Change control, security, etc. 6 6
Needs Analysis - Operational Non-technical People, skill-sets & day-to-day needs Process and change management rigors Organizational structure (Development & production DBAs, OnCall, etc.) Peer groups for tiers above and below Formal / informal service level agreements Budget and resource allocation The unspoken / implicit needs & objectives Promote standardization Promote rigor, structure and change control Promote documentation Promote visibility into infrastructure & environments 7 7
Needs Analysis - Technical Monitor what? (Document details & specifics) RDBMS and databases Database jobs and batches Database resources System resources Communication, reporting, security and auditing For status = ok/healthy For status = problem Integration needs Alerting, systems management, escalation, etc. 8 8
Gaps = Needs - Current Categorize the needs Needs that are absolutely necessary Needs that serve as job aids, tools and automation Good to have Categorize the gaps Critical to your operations and service delivery Critical to bottom line Critical to corporate alignment Determine timeframe and budget to meet the gaps 9 9
Topics 1. Presentation Objectives 2. DB2 Monitoring - Needs and Gap Analysis 3. Buy versus Build Solution 4. Discuss a Home-Grown Solution 10 10
Before you Buy or Build Corporate introspection and survey Corporate standards Solutions available or used in-house Best of breed versus comprehensive solution Budgeting, cost allocation, etc Dove-tail monitoring solution into - Corporate objectives and priorities Corporate / enterprise application needs Prepare TCO (Total Cost of Ownership) and ROI (Return On Investment) Marshall executive sponsorship 11 11
Buy versus Build Considerations - Consider real cost, time & benefits for both!!!! Buy - Cost = buy + maintenance + infrastructure + training + time + Time = procurement + customization + training + deployment + operational procedures +.. Benefits = jump-start (time savings) + proven solution + support + best practices + documentation +.. Personal observations Cost = usually 1.5 x original estimates (or more) Time = usually 2.0 x original estimates (or more) 12 12
Buy versus Build Considerations - Build - Cost = Time + time + time +. People + people + people + Time = Proof-of-concept (build & throw-away) + Actual development (sufficient rigor and project management controls needed) Other needs and considerations - Risks Support and maintenance Documentation Corporate standards Other 13 13
Surveying available solutions Features Create matrix (absolutely needed features and others) Include Infrastructure and other needs, monitoring solution foot-print, scalability, manageability, security Cost Freebies included e.g. - Training and implementation Best practices Other Best of breed versus comprehensive & integrated Product and vendor quality and viability Product and vendor ranking & support 14 14
Topics 1. Presentation Objectives 2. DB2 Monitoring - Needs and Gap Analysis 3. Buy versus Build Solution 4. Discuss a Home-Grown Solution 15 15
Why We Chose Home-Grown Solution Needs 24 x 7 monitoring (no previous monitoring in place) Quick, effective, proven solution for DBAs only Good to have - integration (HP Openview + Netbackup) (Netbackup = Veritas Netbackup from Symantec Corp) Cost Skill-set and existing scripts Spare time! Database tasks completed prior to application tasks Challenge & Opportunity 16 16
Home Grown Solution Topics 1. Overview and Key Features 2. In-Depth Look 3. Toolkit Deployment 4. Next Steps.. 17 17
Home Grown Solution 1. Overview and Key Features 2. In-Depth Look 3. Toolkit Deployment 4. Next Steps.. 18 18
Overview Scripts to monitor database availability and capacity Scripts for 30-60% of regular, time-consuming, DBA administration & maintenance tasks Saves money & time and helps productivity Cost (license, training and deployment) Time (procurement, testing, deployment, day-to-day tasks) Integration (HP OpenView, Netbackup) Tried and tested at Fortune 500 Financial Institution Data Center Outsourcing Company The Warranty Group 19 19
Key Features Implemented using shell scripts Support for multiple DB2 instances and databases Individual configuration for each database Modular and easily configurable Email (or any set of actions) on exception Scripts tested on AIX, Solaris and RedHat Linux Single script for comprehensive monitoring Single script for DB2 admin & maintenance tasks Other misc. scripts 20 20
Key Features HP Openview Integration Alerts from DB2 Monitoring appear on HP OpenView Console and message browser windows 21 21
Toolkit Contents opc_db2_mon.ksh - Availability and capacity monitoring script db2utils.ksh - General utilities script monitor-system.ksh Ad hoc OS and DB2 performance data collection script 7 SQL scripts Performance monitoring related db2_discovery.sql Comprehensive HTML report 15 crontab entry scripts Sample deployment import-audit-data.ksh - Audit data extract, load & purge README.txt Readme for db2utils.ksh 22 22
Home Grown Solution 1. Overview and Key Features 2. In-Depth Look 3. Toolkit Deployment 4. Next Steps.. 23 23
In-Depth Look Availability and Capacity Monitoring Performance Monitoring Administration Utilities Security and Audit 24 24
Availability & Capacity Monitoring Implemented by - opc_db2_mon.ksh Single, standalone, light-weight script Monitoring of - Instance availability (db2sysc) DB2 TCP/IP Listener availability (db2tcpcm) Database availability (connectivity) Tablespace states (normal) Filesystem space (> 85% used) Action on exception HP OpenView event & email 25 25
Availability & Capacity Monitoring 1. Check for $HOME/sqllib/db2profile 2. Check for $HOME/.profile 4. Check for filesystem free space (> 15% free) 3. Check for db2sysc and db2tcpcm processes 5. Check for database and tablespace states 6. Check for available log space (> 64 MB) 7. Connect to the database 26 26
Performance Monitoring Implemented as - 7 SQL scripts Not for ad hoc performance trouble-shooting Data collection for profiling and trending Uses DB2 v8 snapshot functions (monitor switches ON at instance level) Daily data collection 5 weeks retention and automatic purge of older data Cumulative & differential values for metrics (e.g. cumulative and differential logical reads) Collects data across all partitions 27 27
Administration Utilities Implemented by db2utils.ksh Database Backup Online or offline, full or incremental To disk, tape or third-party software Configuration Backup db & dbm cfg db & node directory Delete old objects log archives backups (on disk or Netbackup) Files 28 28
Security and Audit Implemented by import-audit-data.ksh Standard db2audit data extract and import into database (recommend dedicated database) Actual data collected as per your audit configuration Audit data configuration options in DB2 Administration Guide Implementation Interactive script run on a periodic basis - Stops auditing (momentarily) - Extracts audit data - Imports into audit database - Prunes DB2 audit data files 29 29
Home Grown Solution 1. Overview and Key Features 2. In-Depth Look 3. Toolkit Deployment 4. Next Steps.. 30 30
Toolkit Contents (repeated) - opc_db2_mon.ksh - Availability and capacity monitoring script db2utils.ksh - General utilities script monitor-system.ksh Ad hoc monitoring & performance script 7 SQL scripts Performance monitoring related db2_discovery.sql Comprehensive HTML report 15 crontab entry scripts Sample deployment import-audit-data.ksh - Audit data extract, load & purge README.txt Instructions and other details 31 31
Deployment 1.. Copy and untar the tar file Untar creates directory DB2INSTANCE_HOME/scripts Become familiar with the contents of the directory Read README.txt 32 32
Deployment 1..2.. Copy instance-database.env to <instance>-<database>.env Edit each.env file Sample <instance>-<database>.env file #-------------- Start of Customization --------------. $HOME/.profile export SCRIPTS_DIR="/home/db2inst1/scripts" export AUDIT_LOG_DIR="/home/db2inst1/scripts" export REPORTS_DIR="/home/db2inst1/scripts/reports" export OLD_BACKUP_RETENTION=7 export DB_BACKUP_DIR="/applications/Database/backups/bocmsdev" export NOTIFICATION_EMAIL= name@domain.com" export alert_level_1_email= name@domain.com" export alert_level_2_email= name@domain.com export alert_level_3_email= name@domain.com" export APPLICATION_NAME= Finance Data Warehouse(Development)" #---------------- End of Customization -------------- 33 33
Deployment 1..2..3.. Edit performance data setup scripts, if necessary (DB2INSTANCE_HOME/scripts/db2mon) Run the performance data setup scripts 01-db2mon_create_tablespace.sql 02-db2mon_create_tables.sql 03-db2mon_get_snapshot_data.sql 04-db2mon_get_object_data.sql 34 34
Deployment 1..2..3..4.. Edit monitoring script, if necessary (scripts/opc_db2_mon.ksh) Create /etc/db2.txt with entries for all databases across all instances Create monitored object(s) in HP OpenView or appropriate Systems Management tool Validate exception action Provide the script to admin (root) for monitoring 35 35
Deployment 1..2..3..4..5.. Test db2utils.ksh and the monitoring scripts (see 01-*.ksh through 15*.ksh in sample crontab.txt) Create crontab entries (scripts/crontab.txt for sample) ##### # Edit History # 12/22/2006 (Jayesh Thakrar) - Created initial crontab entries ##### # 03 * * * * $HOME/scripts/01-check-filesystem-space.ksh >/dev/null 21 * * * * $HOME/scripts/01b-check-database-availability.ksh >/dev/null 03 22 28 * * $HOME/scripts/03-db2-rotate-diag-log-files.ksh >/dev/null 41 22 28 * * $HOME/scripts/04-db2utils-rotate-audit-log-file.ksh >/dev/null 01 05 * * 0 $HOME/scripts/06-db2-runstats-all.ksh >/dev/null 13 16 * * 0 $HOME/scripts/07-db2-reorgchk.ksh >/dev/null 27 17 * * 0 $HOME/scripts/08-db2-schema-extract.ksh >/dev/null 03 22 * * * $HOME/scripts/09-db2-delete-old-archived-logs.ksh >/dev/null 31 18 * * 3,6 $HOME/scripts/10b-db2-delete-old-backups.ksh >/dev/null 31 19 * * 3,6 $HOME/scripts/10-db2-online-full-backup.ksh >/dev/null 16 18 * * * $HOME/scripts/11-db2-daily-snapshots.ksh >/dev/null 22 23 * * * $HOME/scripts/12-db2mon-get-snapshots.ksh >/dev/null 32 23 * * 0 $HOME/scripts/13-db2mon-get-object-data.ksh >/dev/null 42 23 * * * $HOME/scripts/14-db2mon-analyze-snapshot-data.ksh >/dev/null 52 23 * * 0 $HOME/scripts/15-db2mon-purge-old-data.ksh >/dev/null 23 06 * * * /usr/bin/crontab -l > $HOME/scripts/crontab-l-$USER-$HOSTNAME.txt 36 36
Deployment 1..2..3..4..5..go Deployment Ready Take-off!! 37 37
Home Grown Solution 1. Overview and Key Features 2. In-Depth Look 3. Toolkit Deployment 4. Next Steps.. 38 38
Possible Future Enhancements Use db2pd instead of snapshots (Why? - lightweight) Monitor local and remote databases (authentication info?) Single repository database for all monitored databases Configurable threshold monitoring and alerting Collect operating system performance data Automated analysis and feedback on capacity planning Port to Java, Perl or Python AND Windows GUI front-end (HTML, web-services, XML?) 39 39
Comparison with DB2 s built-in Health Monitor Health monitor available out of the box Health monitor depends upon the availability of DAS (DB2 Administration Server) DB2 fault-monitor also available out of the box however occasional problems (performance/cpu, respawning rapidly, etc.) Health monitor does not maintain historic data Health monitor interval not configurable Toolkit scripts customizable and extensible 40 40
Questions? 41 41
Session: D12 Case Study: Home-Grown DB2 Monitoring and Related Tools Jayesh Thakrar The Warranty Group Jayesh.thakrar@us.thewg.com 42 42