Maximizing DB2 Audit Logging Facility Without Compromising Application Performance. James P O Leary Maria Shanahan HSBC North America

Size: px

Start display at page:

Download "Maximizing DB2 Audit Logging Facility Without Compromising Application Performance. James P O Leary Maria Shanahan HSBC North America"

Erin Lucas
6 years ago
Views:

1 Session: D10 Maximizing DB2 Audit Logging Facility Without Compromising Application Performance James P O Leary Maria Shanahan HSBC North America May 21,2008 1:30 p.m. 2:30 p.m. Platform: DB2 for Linux, UNIX, Windows Utilizing the built-in Audit Facilities of DB2 UDB an appropriate level of auditing can be established without impacting database application performance. This presentation will illustrate how to establish a viable methodology in order to define, setup, implement, and monitor database auditable activities. Also included will be an explanation of the various audit event types, establishment of DB2 audit event tables and steps needed to extract audit information for reporting purposes. 1

2 Maximizing DB2 Audit Logging Facility Without Compromising Application Performance Establishment of UDB audit logging at the appropriate level for a particular application Definition, setup, implementation and utilization of DB2 UDB Audit Logging Facility How to create audit logging tables, population of tables, and establishment of maintenance jobs for audit logging facility Understanding of different audit event types How to utilize tools (Perfmon, LoadRunner) to assist in determining overhead of audit logging on a DB2 database 2 Various options and customization allow for different levels of auditing. Application need and DB2 performance overhead need to be taking into consideration before activating logging facility. 2

3 Vision To utilize the built-in Audit Facilities of DB2 UDB to establish an appropriate level of auditing without impacting database application performance. 3 As with any tool, overhead risks need to be weighed against benefits. To avoid the possible increased risk to performance, due diligence needs to be exercised to ensure the implications are determined upfront. Analysis of real-time simulations must be complete to avoid surprises in production activation. 3

4 Goals and Objectives: Define Methodology Establish DB2 Audit Environment Establish Testing Environment Perform Testing Comparative Analysis of Tests Results and Conclusions 4 4

5 Define Methodology Application Audit Requirements Application Simulation Tool Criteria DB2 Audit Facility Requirements Testing Scenarios Performance Needs Best Auditing Capability Criteria 5 All projects can be better managed when divided into separate components. Project plan should include requirements gathering, audit facility configuration, simulation tool setup, system testing (baseline and performance), performance analysis, and final recommendation. 5

6 Define Methodology Application Audit Requirements Audit needs Reporting requirements Audit record retention Application Audit Requirements will have direct correlation to DB2 Audit Facility requirements 6 Overhead can be minimized by only auditing necessary components. Decide on type of events necessary first then whether only successful or failed events should be logged. Successful event logging significantly increases overhead. Error handling can be configured to ignore audit facility errors and only send errors associated with the operation being performed. ERRORTYPE in db2audit command. Reporting requirements will assist in determining what information is necessary to capture to provide data for reports. Data retention requirements may necessitate offloading audit records to auxiliary repository. Reports retention may differ from raw data retention. 6

7 Define Methodology Application Simulation Tool Criteria Is one needed Is it scaleable Does it have performance reporting capabilities 7 Application Simulation tool will assist with volume/stress testing. Necessary to emulate application especially if complex transaction detail and volume. This will facilitate testing in a production like environment so that there will be no surprises once logging is activated. 7

8 Define Methodology DB2 Audit Facility Requirements Level required (full, partial or both) Based on reporting needs are we offloading or using audit tables Purge of audit events records 8 Full audit logging may be necessary if you need to associate a group of events back to a single database operation. If requirements can be met with partial audit logging overhead considerations may be minimal. Audit needs may require on demand access to audit tables which would mean easy access to audit data. Data may need to be offloaded and then purged to avoid space issues. 8

9 Define Methodology Testing Scenarios Baseline Normal processing activity and volume Partial - Same as baseline with all but operation context logging on Full - Same as partial with operation context logging on 9 Baseline testing will allow for normal processing both volume and detail. Partial audit logging test will be same as baseline with DB2 partial audit logging activated. Full audit logging test will be same as baseline and partial with DB2 full (Operation context) audit logging activated. 9

10 Define Methodology Performance Needs Acceptable performance impact level to application and DB2 Different level of auditing based upon application impact Purchase additional hardware if needed 10 Criticality of application response time and other acceptable performance levels should be determined in order to successfully evaluate impact. Auditing level can be tailored to meet requirements and application overhead associated with that level Additional memory and / or DASD may assist in offsetting impact 10

11 Define Methodology Best Auditing Capability Criteria Application response time (through put) CPU consumption DISK I/O 11 Monitor response time to see existing baseline values. Application response time should be addressed with business to establish acceptable levels. CPU and DISK I/O should be evaluated to determine optimal scenario. 11

12 DB2 Audit Environment Setup Database Audit Event Types Create UDB Audit Database Objects Configure and Activate Audit Facility Implement process to offload Audit data and purge UDB tables 12 Audit environmental settings need to be completed before audit is activated. Decisions made in the methodology phase will be implemented here, and used later in performance phase. In review, the following questions should be taken into consideration: 1. Level of audit requirement 2. Reporting requirements make sure necessary values are being captured (Flat files or DB2 tables) 3. Audit records retention duration (purge process) 5. SMS or DMS for audit database DMS used here 12

13 DB2 Audit Environment Database Audit Event Types AUDIT CHECKING OBJMAINT SECMAINT SYSADMIN VALIDATE CONTEXT 13 Audit records are produced whenever the following events take place: Audit - Generates records when audit settings are changed or when the audit log is accessed Authorization checking - Generates records during authorization checking of attempts to access or manipulate DB2 objects or functions Object maintenance - Generates records when creating or dropping data objects Security maintenance - Generates records when granting or revoking object or database privileges or DBADM authority. Records are also generated when the database manager security configuration parameters SYSADM_GROUP, SYSCTRL_GROUP, or SYSMAINT_GROUP are modified System administration - Generates records when operations requiring SYSADM, SYSMAINT, or SYSCTRL authority are performed User validation - Generates records when authenticating users or retrieving system security information Operation context Generates records to show the operation context when a database operation is performed. This category allows for better interpretation of the audit records. When used with the log's event correlator field, a group of events can be traced back to a single database operation that can provide needed context when analyzing audit results. 13

14 DB2 Audit Environment UDB Audit Database Objects CREATE SCHEMA AUDIT CREATE BUFFERPOOL BPAUDIT_R CREATE REGULAR TABLESPACE AUDIT_DATA_TS CREATE TABLE AUDIT 14 CREATE SCHEMA AUDIT; SET CURRENT SCHEMA = 'AUDIT'; CREATE BUFFERPOOL BPAUDIT_R ALL NODES SIZE 4192 PAGESIZE 4096 NOT EXTENDED STORAGE; CREATE REGULAR TABLESPACE AUDIT_DATA_TS IN NODEGROUP IBMDEFAULTGROUP PAGESIZE 4096 MANAGED BY DATABASE USING( FILE 'D:\AUDIT\Audit_data_ts.ts1' ) EXTENTSIZE 16 PREFETCHSIZE 32 BUFFERPOOL BPAUDIT_R OVERHEAD 8.30 TRANSFERRATE 0.18 DROPPED TABLE RECOVERY ON; CREATE TABLE AUDIT (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, STATUS INTEGER, USERID VARCHAR(1024), AUTHID VARCHAR(128)) IN AUDIT_DATA_TS; 14

15 DB2 Audit Environment UDB Audit Database Objects CREATE TABLE CHECKING CREATE TABLE OBJMAINT 15 CREATE TABLE CHECKING (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, STATUS INTEGER, DATABASE CHAR(8), USERID VARCHAR(1024), AUTHID VARCHAR(128), NODENUM SMALLINT, COORDNUM SMALLINT, APPID VARCHAR(255), APPNAME VARCHAR(1024), PKGSCHEMA VARCHAR(128), PKGNAME VARCHAR(128), PKGSECNUM SMALLINT, OBJSCHEMA VARCHAR(128), OBJNAME VARCHAR(128), OBJTYPE VARCHAR(32), ACCESSAPP CHAR(18), ACCESSATT CHAR(18), PKGVER VARCHAR(64))IN AUDIT_DATA_TS; CREATE TABLE OBJMAINT (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, STATUS INTEGER, DATABASE CHAR(8), USERID VARCHAR(1024), AUTHID VARCHAR(128), NODENUM SMALLINT, COORDNUM SMALLINT, APPID VARCHAR(255), APPNAME VARCHAR(1024), PKGSCHEMA VARCHAR(128), PKGNAME VARCHAR(128), PKGSECNUM SMALLINT, OBJSCHEMA VARCHAR(128), OBJNAME VARCHAR(128), OBJTYPE VARCHAR(32), PACKVER VARCHAR(64))IN AUDIT_DATA_TS; 15

16 DB2 Audit Environment UDB Audit Database Objects CREATE TABLE SECMAINT CREATE TABLE SYSADMIN 16 CREATE TABLE SECMAINT (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, STATUS INTEGER, DATABASE CHAR(8), USERID VARCHAR(1024), AUTHID VARCHAR(128), NODENUM SMALLINT, COORDNUM SMALLINT, APPID VARCHAR(255), APPNAME VARCHAR(1024), PKGSCHEMA VARCHAR(128), PKGNAME VARCHAR(128), PKGSECNUM SMALLINT, OBJSCHEMA VARCHAR(128), OBJNAME VARCHAR(128), OBJTYPE VARCHAR(32), GRANTOR VARCHAR(128), GRANTEE VARCHAR(128), GRANTEETYPE VARCHAR(32), PRIVAUTH CHAR(18), PKGVER VARCHAR(64))IN AUDIT_DATA_TS; CREATE TABLE SYSADMIN (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, STATUS INTEGER, DATABASE CHAR(8), USERID VARCHAR(1024), AUTHID VARCHAR(128), NODENUM SMALLINT, COORDNUM SMALLINT, APPID VARCHAR(255), APPNAME VARCHAR(1024), PKGSCHEMA VARCHAR(128), PKGNAME VARCHAR(128), PKGSECNUM SMALLINT, PKGVER VARCHAR(64))IN AUDIT_DATA_TS; 16

17 DB2 Audit Environment UDB Audit Database Objects CREATE TABLE VALIDATE CREATE TABLE CONTEXT 17 CREATE TABLE VALIDATE (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, STATUS INTEGER, DATABASE CHAR(8), USERID VARCHAR(1024), AUTHID VARCHAR(128), EXECID VARCHAR(1024), NODENUM SMALLINT, COORDNUM SMALLINT, APPID VARCHAR(255), APPNAME VARCHAR(1024), AUTHTYPE VARCHAR(32), PKGSCHEMA VARCHAR(128), PKGNAME VARCHAR(128), PKGSECNUM SMALLINT, PKGVER VARCHAR(64), AUDIT_DATA_TS; PLUGINNAME VARCHAR(32))IN CREATE TABLE CONTEXT (TIMESTAMP CHAR(26), CATEGORY CHAR(8), EVENT VARCHAR(32), CORRELATOR INTEGER, DATABASE CHAR(8), USERID VARCHAR(1024), AUTHID VARCHAR(128), NODENUM SMALLINT, COORDNUM SMALLINT, APPID VARCHAR(255), APPNAME VARCHAR(1024), PKGSCHEMA VARCHAR(128), PKGNAME VARCHAR(128), PKGSECNUM SMALLINT, STMTTEXT CLOB(2M), PKGVER VARCHAR(64))IN AUDIT_DATA_TS; 17

18 DB2 Audit Environment Configure and Activate Audit Facility Configure the audit buffer Configure auditing type Start / Stop auditing 18 db2 update db2 cfg using AUDIT_BUF_SZ 256 db2stop db2start db2audit configure scope all status both errortype normal db2audit configure scope audit, checking, objmaint, secmaint, sysadmin, validate status both errotype normal db2audit start db2audit stop 18

19 DB2 Audit Environment Offload Audit Data and Purge UDB tables Export audit records Load audit tables Db2audit prune all 19 Schedule daily script audit_load.ksh which performs the following functions: 1. Flush the buffer db2audit flush 2. Create load ready extract tables from the binary file db2audit.log db2audit extract delasc 3. Prune the db2audit.log table (remove data that used to create load ready files) 4. Load the UDB audit tables db2 connect to db2audit db2 import from audit.del of del modified by chardell0xff insert into audit.audit db2 import from checking.del of del modified by chardell0xff insert into audit.checking db2 import from objmaint.del of del modified by chardell0xff insert into audit.objmaint db2 import from secmaint.del of del modified by chardell0xff insert into audit.secmaint db2 import from validate.del of del modified by chardell0xff insert into audit.validate db2 import from sysadmin.del of del modified by chardell0xff insert into audit.sysadmin db2 import from context.del of del modified by chardell0xff insert into audit.context 19

20 Establish Testing Environment Tools Performance Monitor Application scripts Visual Illustrations Testing Scenarios Baseline Partial Audit Logging Full Audit Logging 20 20

21 Establish Testing Environment Tools Application Simulation tool - LoadRunner Performance monitor Perfmon Visual Illustrations / Charts EXCEL 21 LoadRunner application simulation tool utilized to emulate normal production activity and scaling of the application workflow to stress machine capacity thresholds. Perfmon Windows Performance monitor utilized to capture CPU utilization, memory utilization and DB2 processing. EXCEL - Chart wizard utilized to graph statistics extrapolated from testing 21

22 Establish Testing Environment Testing Scenarios Baseline Starting point for comparative analysis Define scripts in LoadRunner to emulate normal activity Turn on Perfmon to capture statistics Execute via LoadRunner workflow 22 Baseline - Is used to establish normal application workflow for a processing day without DB2 Audit Facility enabled. The baseline will the be compared against partial audit and full audit logging test results Define workflow in LoadRunner via scripts Determine normal activity (number of concurrent users, frequency of scripts) Configure Perfmon to capture DB2 services for CPU and memory utilization along with File I/O 22

23 Establish Testing Environment Testing Scenarios Partial Audit Logging Turn on partial audit logging (audit, authorization checking, object maintenance, security maintenance, system administration and user validation) Turn on Perfmon to capture statistics Execute via LoadRunner workflow 23 Partial Audit Logging Is used to determine the minimal audit requirements that are available to the application. Utilization of the scenarios will require additional research as to what occurred in a particular audit event on the database. Additional tools will be required such as application logs (if available), DB2 archive logs etc. 23

24 Establish Testing Environment Testing Scenarios Full Audit Logging Turn on full audit logging (audit, authorization checking, object maintenance, security maintenance, system administration, user validation, and operation context) Turn on Perfmon to capture statistics Execute via LoadRunner workflow 24 Full Audit Logging - Is used to establish maximum capabilities of DB2 audit logging. Using this scenario additional research may not be required when analyzing an audit event. However, the Risk vs. Reward increase in CPU, memory and thru put may be taken into consideration. 24

25 Comparative Analysis of Test Data gathered through Perfmon for each test exported to Excel Excel chart wizard to graph results Maximum CPU percentages utilized for all testing 25 25

26 Comparative Analysis of Test Maximum CPU utilization with full auditing was 96% as compared to 80% and 83% for the baseline and partial tests respectively. Maximum CPU processor time for DB2 with full auditing was 75% as compared to 28% and 36% for the baseline and partial tests respectively

27 Comparative Analysis of Test Maximum CPU utilization % Utilization Baseline Partial Full DB2 CPU 27 27

28 Comparative Analysis of Test 100 Baseline % Disk Time % Processor Time 28 Observations: As transaction flow increases so does cpu utilization. Full audit logging has high cpu utilization from the start Partial audit logging not only consumes less cpu but stays close to normal application cpu usage 28

29 Comparative Analysis of Test 100 Partial Audit Logging % Disk Time % Processor Time 29 29

30 Comparative Analysis of Test Full Audit Logging % Disk Time % Processor Time 30 30

31 Comparative Analysis of Test 100 CPU% All Tests CPU Percent Time Baseline Partial Logging Full Logging 31 31

32 Results and Conclusions Full audit logging consumes a significant amount of O/S resources Partial audit logging Biggest Bang for your Buck Very important to develop methodology based on application audit needs Risk and Reward must be taken into consideration 32 32

33 Resources DB2 UDB security, Part 5: Understand the DB2 audit facility, harticle/dm-0603wasserman/index.html IBM DB2 Administration Guide Planning V9 - Chapter 9. Auditing DB2 Activities SC

34 Session D10 Maximizing DB2 Audit Logging Facility Without Compromising Application Performance James P O Leary Maria Shanahan HSBC North America James.P.O Leary@us.hsbc.com Maria.Shanahan@us.hsbc.com 34 James P. O Leary Over twenty five years data processing experience with special expertise in the areas of data warehousing, database administration, database system programming, operating system software, and applications systems support. Maria Shanahan Over twenty five years experience in Information Technology with special expertise in database administration and project management. 34

Introduction to IBM DB2

Introduction to IBM DB2 Architecture Client-server system Server: SERVEDB, servedb.ing.man 10.17.2.91 Client: IBM Data Studio: graphical DB2 Command Window: command line 2 Architecture Servers, instances,