IBM Tivoli Risk Manager, Version 4.1 Warehouse Enablement Pack, Version 1.1 Implementation Guide

Transcription

1 , Version 4.1 Warehouse Enablement Pack, Version 1.1 Implementation Guide SC

2 Note: Before using this information and the product it supports, read the information in "Notices" on page 39. Second Edition (October 23) This edition applies to version 4, release 1 of IBM Tivoli Risk Manager and to all subsequent releases and modifications until otherwise indicated in new editions. Copyright IBM Corporation 23. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. ISO 91 Certification This product was developed using an ISO 91 certified quality system. Certification has been awarded by Bureau Veritas Quality International (BVQI) (Certification No. BVQI / A). BVQI is a world leader in quality certification and is currently recognized by more than 2 accreditation bodies. 2

3 Contents, Version Warehouse Enablement Pack, Version 1.1 Implementation Guide About this document Related documentation Tivoli Enterprise Data Warehouse IBM DB2, DB2 Data Warehouse Center, and DB2 Warehouse Manager Overview Overview of Tivoli Enterprise Data Warehouse Overview of warehouse enablement pack Detailed mapping Installing and configuring Prerequisites Supported hardware and software Database sizing considerations Data sources and targets Pre-installation procedures Installation procedure Post-installation procedures Maintaining Backing up and restoring Pruning Central data warehouse Data mart Miscellaneous utilities hrm_cdw_classcat_data.sql hrm_cdw_reset_data.sql hrm_mart_reset_data.sql and hrm_cdw_reset_etl2_extctl.sql hrm_cdw_drop_data.sql ETL processes HRM_c5_Load_CDW_Process HRM_c5_s1_Extract HRM_c5_s2_Transform HRM_c5_s3_Load_Comp HRM_c5_s4_Load_Msmt Exception handling HRM_m5_Build_Mart_Process HRM_m5_s1_Pre_Extract...2 3

4 5.2.2 HRM_m5_s2_Extract HRM_m5_s3_Load HRM_m5_s4_Rollup HRM_m5_s5_Prune HRM_m5_s6_Run_Report Central data warehouse schema implementation Component configuration Component type (table CompTyp) Component (table Comp) Component relationship type (table RelnTyp) Component relationship rule (table RelnRul) Component relationship (table CompReln) Attribute type (table AttrTyp) Attribute rule (table AttrRul) Attribute domain (table AttrDom) This table is not used by Tivoli Risk Manager Component attribute (table CompAttr) Component measurement Measurement group type (table MGrpTyp) Measurement group (table MGrp) Measurement group member (table MGrpMbr) Measurement unit category (table MUnitCat) Measurement unit (table MUnit) Time summary (table TmSum) Measurement source (table MSrc) Measurement type (table MsmtTyp) Component measurement rule (table MsmtRul) Measurement (table Msmt) Helper tables Event category descriptions (table HRM.ClassCatDesc) Exception tables Exception log (table HRM.Exception_Log) Error messages (table HRM.Error_Message) Incremental extraction Data mart schema information Star schemas HRM hourly Tivoli Risk Manager event star schema Fact table HRM.F_Event_Hour HRM daily Tivoli Risk Manager event star schema Fact table HRM.F_Event_Day HRM weekly Tivoli Risk Manager event star schema Fact table HRM.F_Event_Week HRM monthly Tivoli Risk Manager event star schema Fact table HRM.F_Event_Month Metric dimension tables HRM.D_Event_Metric Dimension tables

5 7.3.1 Dimension table HRM.D_CLASSCAT Dimension table HRM.D_DST_HOST Dimension table HRM.D_SRC_HOST Data marts and reports Tivoli Risk Manager data mart Reports Events by destination host last 3 days Events by destination and category last 3 days Access/authentication events last 3 days Service compromise events last 3 days Infection events last 3 days Policy/configuration events last 3 days Events by destination subnetwork last 3 days Notices Trademarks...4 5

6 1 About this document This document describes the warehouse enablement pack for IBM Tivoli Risk Manager. It covers the following topics: Installing and configuring the warehouse pack The data flow and data structures used by the warehouse pack With this warehouse pack, you can load Tivoli Risk Manager event data into the central data warehouse. 1.1 Related documentation You can access many Tivoli publications online using the Tivoli Information Center, which is available on the IBM Web site: The following sets of documentation are available to help you understand, install, and manage this warehouse pack: (optional) Tivoli Enterprise Data Warehouse IBM DB2, DB2 Data Warehouse Center, and DB2 Warehouse Manager The following sections list and briefly describe these libraries The following Tivoli Risk Manager documents are useful for understanding the data that is generated and loaded into the Tivoli Enterprise Data Warehouse: Release Notes, GI Provides the most up-to-date information about Tivoli Risk Manager, and lists hardware requirements and software prerequisites. User s Guide, SC Provides complete information about Tivoli Risk Manager administration and usage, including installation, configuration, operation, and database usage. Chapter 5, Database, describes in detail the database schema and database management used by Tivoli Risk Manager. Developer s Guide, SC Provides guidelines for developing new adapters that forward events to a Tivoli Risk Manager server, where the events are correlated and eventually stored in the database for future extraction to the warehouse. Adapters Guide, SC Provides instructions to install, configure, and manage Tivoli Risk Manager adapters. Adapters are software programs that generate Risk Manager events that will eventually be stored in the Risk Manager archive and then loaded into the central data warehouse (CDW). Problem Determination Guide, SC Provides problem determination processes and scenarios to assist in determining why Tivoli Risk Manager is malfunctioning The following Tivoli Risk Manager documents are useful for understanding the data that is generated and loaded into the Tivoli Enterprise Data Warehouse: 6

7 Release Notes, GI Provides the most up-to-date information about Tivoli Risk Manager, and lists hardware requirements and software prerequisites. Administrator s Guide, GC Provides information on how to configure, and manage Tivoli Risk Manager. This guide also provides an overview for each Tivoli Risk Manager component. Installation Guide, GC Provides information on planning your product deployment, including topics such as network topology and installing prerequisite software and describes how to install and configure the Tivoli Risk Manager product and components. Adapters Guide, GC Provides instructions to install, configure, and manage Tivoli Risk Manager adapters. Adapters are software programs that generate Risk Manager events that will eventually be stored in the Risk Manager archive and then loaded into the central data warehouse (CDW). Problem Determination Guide, GC Provides problem determination processes and scenarios to assist in determining why Tivoli Risk Manager is malfunctioning. Command Reference, GC Provides information on commands used to administer Tivoli Risk Manager Tivoli Enterprise Data Warehouse The following Tivoli Enterprise Data Warehouse documents are available on the Tivoli Enterprise Data Warehouse Documentation CD: Tivoli Enterprise Data Warehouse Release Notes, GI Provides late-breaking information about Tivoli Enterprise Data Warehouse and lists hardware requirements and software prerequisites. Installing and Configuring Tivoli Enterprise Data Warehouse, GC Describes how Tivoli Enterprise Data Warehouse fits into your enterprise, explains how to plan for its deployment, and gives installation and configuration instructions. It provides an introduction to the built-in program for creating and running reports, and contains maintenance procedures and troubleshooting information. Enabling an Application for Tivoli Enterprise Data Warehouse, GC Provides information about connecting an application to Tivoli Enterprise Data Warehouse. This book is for application programmers who use Tivoli Enterprise Data Warehouse to store and report on their application s data, data warehousing experts who import Tivoli Enterprise Data Warehouse data into business intelligence applications, and customers who use their local data in the warehouse. The following Tivoli Enterprise Data Warehouse Redbook is available from the IBM Redbooks Web site ( ): Introduction to Tivoli Enterprise Data Warehouse, SG Provides a broad understanding of the Tivoli Enterprise Data Warehouse. Some of the topics that are covered in this redbook are: concepts, architecture, installation, tips for using the Reporting Interface, ETL, best practices in creating Data Marts, integration with major OLAP tools (such as, Brio, Business Objects and Cognos PowerPlay), multi-customer 7

8 (Service Provider) environments, operational considerations, and troubleshooting. Most of the topics are explained using real customer implementations IBM DB2, DB2 Data Warehouse Center, and DB2 Warehouse Manager The DB2 library contains important information about the database and data warehousing technology provided by IBM DB2, DB2 Data Warehouse Center, and DB2 Warehouse Manager. Refer to the DB2 library for help in installing, configuring, administering, and troubleshooting DB2. The DB2 library is available on the IBM Web site: After you install DB2, its library is also available on your system. The following DB2 documents are particularly relevant for people working with Tivoli Enterprise Data Warehouse: IBM DB2 Universal Database for Windows Quick Beginnings, GC Guides you through the planning, installation, migration (if necessary), and setup of a partitioned database system using the IBM DB2 product on Microsoft Windows. IBM DB2 Universal Database for UNIX Quick Beginnings, GC9-297 Guides you through the planning, installation, migration (if necessary), and setup of a partitioned database system using the IBM DB2 product on UNIX. IBM DB2 Universal Database Administration Guide: Implementation, SC Covers the details of implementing your database design. Topics include creating and altering a database, database security, database recovery, and administration using the Control Center, a DB2 graphical user interface. IBM DB2 Universal Database Data Warehouse Center Administration Guide, SC Provides information on how to build and maintain a data warehouse using the Data Warehouse Center. IBM DB2 Warehouse Manager Installation Guide, GC Provides the information to install the following Warehouse Manager components: Information Catalog Manager, warehouse agents, and warehouse transformers. IBM DB2 Universal Database and DB2 Connect Installation and Configuration Supplement, GC Provides advanced installation considerations and guides you through the planning, installation, migration (if necessary), and set up a platform-specific DB2 client. After the DB2 client is installed, you then configure communications for both the client and server, using the DB2 GUI tools or the Command Line Processor. This supplement also contains information on binding, setting up communications on the server, the DB2 GUI tools, DRDA AS, distributed installation, the configuration of distributed requests, and accessing heterogeneous data sources. IBM DB2 Universal Database Message Reference Volume 1, GC and IBM DB2 Universal Database Message Reference Volume 2, GC A list of the messages and codes issued by DB2, the Information Catalog Manager, and the Data Warehouse Center, and describes the actions you should take. 8

9 2 Overview The following sections provide an overview of Tivoli Enterprise Data Warehouse and the warehouse pack. 2.1 Overview of Tivoli Enterprise Data Warehouse Tivoli Enterprise Data Warehouse provides the infrastructure for the following: Extract, transform, and load (ETL) processes through the IBM DB2 Data Warehouse Center tool Schema generation of the central data warehouse Historical reporting As shown in Figure 1, Tivoli Enterprise Data Warehouse consists of a centralized data store where historical data from many management applications can be stored, aggregated, and correlated. Figure 1. Tivoli Enterprise Data Warehouse overview The central data warehouse uses a generic schema that is the same for all applications. As new components or new applications are added, more data is added to the database; however, no new tables or columns are added in the schema. A data mart is a subset of a data warehouse that contains data tailored and optimized for the specific reporting needs of a department or team. The central data warehouse ETL reads the data from the operational data stores of the application that collects it, verifies the data, makes the data conform to the schema, and places the data into the central data warehouse. The data mart ETL extracts a subset of data from the central data warehouse, transforms it, and loads it into one or more star schemas, which can be included in data marts to answer specific business questions. A program that provides these ETLs is called a warehouse enablement pack, or warehouse pack. 9

10 The ETLs are typically scheduled to run periodically, usually during non-peak hours. If an ETL encounters data that it cannot correctly transform, it creates an entry in an exception table. Exception tables are described in Exception tables on page Overview of warehouse enablement pack The warehouse pack aggregates event data from the Tivoli Risk Manager Archive Table (see the database chapter in the Administrator s Guide). The warehouse pack aggregates events based on the same three key attributes (archive table column names in parentheses) used in Tivoli Risk Manager correlation: Event category (CLASS_CAT) Event source token (SRC_TOKEN), hostname (SRC_HOSTNAME) or IP address (SRC_IPADDR) Event target hostname (DST_HOSTNAME) or IP address (DST_IPADDR) Every Tivoli Risk Manager event contains a value for the three attributes listed above. Each event is mapped to a host component in the central data warehouse, which has a parent-child relationship to an event component. Each event component has a measurement consisting of the number of events that occurred for that event against the parent host in a given hour. The host component for an event is determined from the DST_HOSTNAME and DST_IPADDR archive columns. The event component is determined from the combination of CLASS_CAT and SRC_TOKEN. The event measurement is determined from the REPEAT_COUNT archive column, plus one, and then summed for all events matching on host component and event component over a full hour. In addition, host and event components are assigned attributes from the hostname and IP address values associated with each host and event. The Tivoli Enterprise Data Warehouse data model for Tivoli Risk Manager events can be summarized as follows: Host Component Event Component IP_HOST Attributes: LAST_IP_ADDRESS or HRM_EVENT Attributes: HRM_CLASSCAT HRM_SRC_HOSTNAME HRM_SRC_IPADDR IP_INTERFACE Attributes: HRM_DST_HOSTNAME PCHILD relationship or HRM_HOST 1

11 2.2.1 Detailed mapping Tivoli Risk Manager elements created in the central data warehouse tables are listed in Section 6, Generic schema implementation. Refer to that section while reading the following narrative description of how events are mapped into the central data warehouse: 1. If the target hostname is a fully qualified hostname, then it maps to a warehouse host component of type IP_HOST, with the component name coming from the archive column DST_HOSTNAME. Column DST_IPADDR, if not null, is assigned to attribute LAST_IP_ADDRESS. a. If the above condition is not true, but the target IP address is a syntactically valid IP address, then the destination host maps to warehouse host component type IP_INTERFACE, with the component name coming from archive column DST_IPADDR. Column DST_HOSTNAME, if not null, is assigned to attribute HRM_DST_HOSTNAME. b. If neither of the above conditions is true, but the target hostname is a syntactically valid short hostname, then it maps to warehouse host component type HRM_HOST, with component name coming from archive column DST_HOSTNAME. 2. For each unique host component in the warehouse, one or more event components of type HRM_EVENT are created, with the component name derived from the combination of CLASS_CAT and SRC_TOKEN columns in the archive table. SRC_TOKEN is simply the value of SRC_HOSTNAME, if not null; otherwise, the value of SRC_IPADDR. In addition, the columns CLASS_CAT, SRC_HOSTNAME and SRC_IPADDR, if not null, are assigned to HRM_EVENT attributes. The uniqueness of a HRM_EVENT component is determined from its component name (CLASS_CAT + SRC_TOKEN), and the parent host component ID in the Comp_Corr_ID column of the component record. 3. Finally, for each HRM_EVENT component, one or more measurements are created, which measure the number of times the event was targeted at a single host component in a single hour. 11

12 3 Installing and configuring For detailed instructions to install this warehouse pack see the installing data warehouse section of the IBM Tivoli Risk Manager 4.2 Installation Guide or the Tivoli Risk Manager Data Warehouse Readme file in this warehouse pack. Also, become familiar with section Installing warehouse packs from Installing and Configuring Tivoli Enterprise Data Warehouse (GC32-744). 3.1 Prerequisites IBM DB2 Universal Database 7.2, plus the following fixes: IBM DB2 UDB 7.2 Fix Pack 6 Tivoli Enterprise Data Warehouse 1.1 Patch 2 (1.1-TDW-2) which replaces DB2 files 4.1 or 4.2, Event Server component Tivoli Enterprise Data Warehouse 1.1, plus the following fixes and patches: Tivoli Enterprise Data Warehouse 1.1 Fix Pack 2 (1.1-TDW-FP2) 3.2 Supported hardware and software Warehouse Enablement Pack, Version 1.1, supports, Version 4.1 and 4.2. It supports specific versions of DB2, Microsoft SQL Server, Oracle, and Sybase database products as documented in the IBM Tivoli Risk Manager 4.2 Administrator s Guide and 4.1 and 4.2 Release Notes. For information about the hardware and software requirements of Tivoli Enterprise Data Warehouse, see the Tivoli Enterprise Data Warehouse Release Notes. 3.3 Database sizing considerations Ensure that you have sufficient space in the central data warehouse (CDW) database for the historical data collected by this warehouse pack. The amount of data in the CDW database can vary greatly from one Tivoli Risk Manager installation to another. The greatest impact on how much data you will collect is the number and variety of Tivoli Risk Manager adapters feeding into your Tivoli Risk Manager server. The more events from several different adapters will mean more measurements in the database. To help determine database capacity requirements, first estimate the following parameters: The number of unique hosts in your enterprise, for example all the distinct dst_hostname and dst_ipaddr values in your archive table. These values will map to new or existing IP_HOST/IP_INTERFACE components in the CDW. The upper limit for this number should be the number of managed hosts in your enterprise, as in the number of hosts with Risk Manager adapters monitoring events. The number of unique combinations of class_cat + src_token, for each destination host/ip in your archive table. These values will map to HRM_EVENT components in the CDW. As an example, if all the adapters in your enterprise are antivirus adapters, then the event category (class_cat) for every event will be the same: VIRUS. The source host (src_token) for each VIRUS event will be the same as the destination host, which means that for each destination host in the archive table, there will be at most one unique combination of class_cat + src_token. If you have a Web server with a Web IDS adapter generating Tivoli Risk Manager events, then you would have an additional event category (WEB) to consider. In this case, the destination host would usually be the same for every event, but the number of unique source hosts for those Web events would depend on how many different hosts access this Web server. For a departmental Web server, the number would be relatively small (<2); for a public Internet server, the number is potentially very large (>1). Therefore, you would need to add the number of estimated source hosts for this Web server to the HRM_EVENT parameter estimate. 12

13 The number of hours in the day when an event combining a unique destination host, source host and event category occurs. These values will map to measurements in the CDW. The range for this number is between and 24. For the antivirus adapter events, this number is likely to be small (1 or 2) because most viruses are detected in a single hour of the day when a virus scan is performed. For Web IDS events, the number would likely be higher because Web requests (and potential IDS events) occur throughout the day. Use the values you found above in the table below to arrive at the total number of components and measurements stored in the database. The values given in the table below are sample values for an enterprise with six managed hosts, all with antivirus adapters, and one host with a Web IDS adapter. Parameter Value Components Measurements per Day Tivoli Risk Manager destination hosts 6 6 (IP_HOST, IP_INTERFACE, HRM_HOST) N/A Tivoli Risk Manager event category plus source host combinations (one for each applicable destination host) Hours in the day (-24) when a unique event occurs 6 (VIRUS) + 1 (WEB) 2 (VIRUS) 22 (WEB) 16 (HRM_EVENT) N/A N/A 12 (VIRUS) + 22 (WEB) Totals To translate the above numbers to bytes of database storage, use the following multipliers: For each destination host: 15 bytes (Comp table) + 15 (CompAttr table) For each event category + source host combination: 15 bytes (Comp table) + 3 (CompAttr table) + 1 (CompReln table) For each daily measurement: 1 bytes (Msmt table) * Number of days of storage Using the sample table above, the total storage required would be: Components (including attributes, relationships): 6 * * 55 = 6 KB Measurements (for 6 months of data): 2212 * 1 * 6 * 3 = 39,816 KB For practical purposes, the storage required for measurements far exceeds (several orders of magnitude) the storage required for components. The total database capacity required for the sample numbers given above would be approximately 4 MB. 3.4 Data sources and targets The data source for the Tivoli Risk Manager warehouse pack is the Tivoli Risk Manager Archive Table. The table is named rm_t_arc41 and is typically created in database tec. Note that the table and database name is in lowercase, which is significant only if the source database platform is Sybase. For the other database platforms, you may specify the table and database in uppercase or lowercase. The Tivoli Risk Manager warehouse pack accesses the Tivoli Risk Manager Archive Table through an ODBC data source name (DSN). The pack requires that the DSN be named RMDB. If the Tivoli Risk Manager Reports (Crystal Reports) component is installed on the same system as the Tivoli Risk Manager warehouse pack, you may use the same RMDB DSN 13

14 as that defined for the reports. If you need ODBC drivers, it is best to use the ones provided by the database vendor with the client software for the version of the database you are using. 3.5 Pre-installation procedures 1. Make sure that the Tivoli Enterprise Data Warehouse is installed. For instructions on installing the Tivoli Enterprise Data Warehouse, refer to Installing and Configuring Tivoli Enterprise Data Warehouse. 2. Make sure that all required product fix packs and patches are applied. See the 4.2 Installation Guide or the Tivoli Risk Manager Data Warehouse Readme file in this warehouse pack for details. 3. Make sure that is installed, and the RMDB data source is available. 3.6 Installation procedure For general warehouse pack instructions, read the section Installing warehouse packs from Installing and Configuring Tivoli Enterprise Data Warehouse. For deviations from the general instructions and more specific instructions to install the Tivoli Risk Manager Warehouse Enablement Pack, refer to the 4.2 Installation Guide or the Tivoli Risk Manager Data Warehouse Readme file in this warehouse pack. 3.7 Post-installation procedures Read the section about getting started with Tivoli Enterprise Data Warehouse in the Installing and Configuring Tivoli Enterprise Data Warehouse guide to become familiar with the steps required to configure and schedule reports for a warehouse enablement pack. Before trying to run any warehouse pack processes, perform the following steps: 1. Change the database user ID and password for your source and target databases. From the Data Warehouse Center user interface, expand the Warehouse Sources and Warehouse Targets nodes in the left pane. You will see a list of HRM Source and Target databases. For each one: Right click the source or target, and select Change User ID and Password. Change the user ID and password, if necessary, and then click OK. 2. Schedule the Tivoli Risk Manager ETL processes to run, by following the instructions in the 4.2 Installation Guide or the Tivoli Risk Manager Data Warehouse Readme file in this warehouse pack. Also see the section on scheduling your ETL steps in the Installing and Configuring Tivoli Enterprise Data Warehouse guide. See the 4.2 Installation Guide or the Tivoli Risk Manager Data Warehouse Readme file in this warehouse pack for step-by-step post-installation procedures. 14

15 4 Maintaining 4.1 Backing up and restoring When backing up the central data warehouse database (TWH_CDW), be sure to back up the following warehouse pack tables: HRM.ClassCatDesc HRM.Cust_Lookup HRM.Centr_Lookup HRM.Extract_Latency HRM.Error_Message HRM.Exception_Log 4.2 Pruning For a complete discussion of pruning issues, see the section on removing old data from the central data warehouse in Installing and Configuring Tivoli Enterprise Data Warehouse Central data warehouse Pruning of old measurements in the central data warehouse (TWH_CDW) is performed using the following enterprise data warehouse process: CDW_c5_PurgeMsmt_Process You must schedule this process to run in order to prune old data. The process is not scheduled by default. The age of Tivoli Risk Manager data to be pruned is determined by an entry in the TWG.Prune_Msmt_Control table in the TWH_CDW database. To display the current pruning threshold, execute the following SQL against the TWH_CDW database: select pmsmtc_age_in_days from TWG.Prune_Msmt_Control where MSrc_Cd = HRM and TmSum_Cd = H ; The value returned is a string that represents a date duration whose format is yyyymmdd. The default value for the Tivoli Risk Manager warehouse pack is 1, which represents one year. To update the pruning threshold to 6 months, for example, execute the following SQL command: update TWG.Prune_Msmt_Control set pmsmtc_age_in_days = 6 where MSrc_Cd = HRM and TmSum_Cd = H ; Data mart Pruning of old measurements in the data mart (TWH_MART) is performed automatically during the Tivoli Risk Manager load data mart process: HRM_m5_Load_Mart_Process The age of the Tivoli Risk Manager data to be pruned is determined by entries in the HRM.Prune_Mart_Control table in the TWH_MART database. The table contains one entry for each time-based fact table (hourly, daily, weekly, monthly). To display the current pruning thresholds, execute the following SQL against the TWH_MART database: select * from HRM.Prune_Mart_Control; The resulting output, with default values, should be similar to the following: 15

16 TABLE_NAME PMARTC_DURATION HRM.F_EVENT_HOUR 3 HRM.F_EVENT_DAY 1 HRM.F_EVENT_WEEK 1 HRM.F_EVENT_MONTH 3 The value returned in the PMARTC_DURATION column is a string that represents a date duration whose format is yyyymmdd. The default prune control thresholds are: hourly 3 months, daily 1 year, weekly 1 year, monthly 3 years. To update the pruning threshold for hourly data to 6 months, for example, execute the following SQL command: update HRM.Prune_Mart_Control set PMartC_Duration = 6 where Table_Name = HRM.F_EVENT_HOUR ; Table HRM.Prune_Mart_Log can be examined to see when and how many facts were pruned from each of the fact tables. 4.3 Miscellaneous utilities A set of miscellaneous database utilities is provided with this warehouse pack. These utilities are SQL command scripts that are located in the subdirectory <TWH_TOPDIR>/apps/hrm/v11/misc, where <TWH_TOPDIR> is the data warehouse installation directory. These utilities are not required for proper operation of the Tivoli Risk Manager warehouse enablement pack, but they are provided to help with database administration tasks. Each script runs against either the central data warehouse (CDW) or the data mart database. To execute a script, you must first open a DB2 command window. From the Windows Start menu, select Programs -> IBM DB2 -> Command Window. Next, you must connect to the desired database (TWH_CDW or TWH_MART), using the following command (user ID and password may be different): db2 connect to TWH_CDW user db2admin using password To execute the script, enter the following command from the DB2 command window: db2 tvf <script-file-name> You will see each statement of the script displayed in the window, followed by a DB2 message indicating success or failure of the statement. If you see any errors, refer to IBM DB2 Universal Database Message Reference, Volume 1. A description of each utility is contained in the following sections hrm_cdw_classcat_data.sql This script initializes the event category description table (HRM.ClassCatDesc) in database TWH_CDW. It is not necessary to run this script during normal operations because the table will be populated as events are loaded into the CDW. However, normal operations will create entries only for event categories contained in the loaded events, which might not include all known categories. This script initializes the description table with all of the predefined Tivoli Risk Manager event categories. You may also modify and run this script if you want to use non-english descriptions, or alternate English descriptions, for the event class categories. Do not modify the event category name field (CC_Name) in the script or you will encounter errors hrm_cdw_reset_data.sql This script resets the central data warehouse database (TWH_CDW) by removing all Tivoli Risk Manager generated data, including measurements, component attributes, component relationships and HRM_EVENT and HRM_HOST components. Only the IP_HOST and IP_INTERFACE components that came from the Tivoli Risk Manager source database are not removed, as these cannot be uniquely associated with Tivoli Risk Manager and might have originated from another Tivoli 16

17 Enterprise Data Warehouse application. This script also returns the extract control parameters for the CDW to the initial, postinstallation state. Use this script when you are testing the operation of the warehouse process HRM_c5_Load_CDW_Process hrm_mart_reset_data.sql and hrm_cdw_reset_etl2_extctl.sql The first script resets the data m art database (TWH_MART) by removing all Tivoli Risk Manager generated data, including facts, dimension data, and all staging data. The second script resets the extract control parameters for the data mart ETL back to the initial, post-installation state. This script must be run against the CDW database (TWH_CDW) because that database is where the extract control parameters are stored. Running these two scripts returns the data mart database to the initial, postinstallation state. Use these two scripts when you are testing the operation of warehouse process HRM_m5_Build_Mart_Process hrm_cdw_drop_data.sql This script removes all Tivoli Risk Manager static data from the central data warehouse (TWH_CDW). It puts the CDW into a state that existed before the Tivoli Risk Manager warehouse pack was installed. To reinitialize the Tivoli Risk Manager static data in the CDW, execute script hrm_cdw_data.sql in subdirectory <TWH_TOPDIR>/apps/hrm/v11/cdw/dml, where <TWH_TOPDIR> is the data warehouse installation directory. 17

18 5 ETL processes This warehouse pack has the following processes: HRM_c5_Load_CDW_Process loads the central data warehouse (CDW) with component and event data from the Tivoli Risk Manager archive table. HRM_m5_Build_Mart_Process builds a data mart with four star schemas based on the Tivoli Risk Manager event counts aggregated hourly, daily, weekly and monthly. 5.1 HRM_c5_Load_CDW_Process This process performs the complete central data warehouse ETL, loading all new Tivoli Risk Manager event data measurements into the central data warehouse. The process runs in four steps that are described below. The process is configured automatically to start with the first step and run each succeeding step if and only if the preceding step succeeds. Run this process once after installing the warehouse pack and on a regular scheduled basis, typically once a day, during offpeak hours. This process has the following steps: HRM_c5_s1_Extract Source: RMDB archive table rm_t_arc41 Target: TWH_CDW staging table HRM.Extract_Archive This step transfers Tivoli Risk Manager event data from the Tivoli Risk Manager source database archive table (rm_t_arc41) to the central data warehouse (CDW) staging table, HRM.Extract_Archive. This step uses table TWG.Extract_Control to limit transfer of events to those events generated later than the time of the last extract. Table HRM.Extract_Latency is used to adjust the extract window so that the most recent events are not extracted if they fall within the latency value, in hours. The archive table columns that are copied to the warehouse include: timestamp32, class_cat, class_cat_desc, src_token,src_hostname, src_ipaddr, dst_hostname, dst_ipaddr, repeat_count. After events are transferred to the staging table, select columns are normalized: All leading and trailing blanks are removed from the event category, token, hostname and IP address fields All hostname and IP address fields are set to NULL if they contain an empty string or the value N/A Dst_IPAddr field is set to NULL if it contains the value... Src_Token field is assigned the value of Src_Hostnam e or Src_IPAddr, whichever is non-null HRM_c5_s2_Transform Source: TWH_CDW staging table HRM.Extract_Archive Targets: TWH_CDW permanent table HRM.ClassCatDesc, staging tables HRM.Transform_Archive, HRM.Transform_Hosts, HRM.Transform_Events, HRM.Event_Msmts This step converts staged Tivoli Risk Manager event data from HRM.Extract_Archive into several staging tables in preparation for loading into the CDW schema. The target staging tables are the following: 18

19 HRM.Transform_Archive contains one record for each raw Tivoli Risk Manager event, with new columns added to store the values of computed target and source component names, plus the date and hour of the event. HRM.Transform_Hosts contains one record for each distinct destination host in all the processed Tivoli Risk Manager events. Each record maps to an IP_HOST, IP_INTERFACE or HRM_HOST component. Also stores the host component name and attributes as computed by this step. HRM.Transform_Events contains one record for each distinct event category and source host in all the processed Tivoli Risk Manager events. Each record maps to a HRM_EVENT component. Also stores the source component name and attributes as computed by this step. HRM.Event_Msmts contains one record for each new measurement to be added to the CDW. Measurements are aggregated by source and target components, as well as by hour. In addition, this step updates the permanent event category description table (HRM.ClassCatDesc see 6.3.1) with any new class categories that may have been received since the last time this step was executed. During this step, events that contain insufficient data to store in the CDW are flagged and stored in the exception log table. Those same events are excluded from further processing and are output in the log file created by this step HRM_c5_s3_Load_Comp Source: TWH_CDW staging tables HRM.Transform_Hosts, HRM.Transform_Events Target: TWH_CDW tables TWG.Comp, TWG.CompReln, TWG.CompAttr This step loads all staged component data into the CDW component tables. Each insert uses 'where not is' logic to ensure that duplicate components or attributes are not inserted HRM_c5_s4_Load_Msmt Source: TWH_CDW staging table HRM.Event_Msmts Target: TWH_CDW table TWG.Msmt This step loads all staged measurement data into the CDW measurement table: TWG.Msmt. If successful, it drops the staging table and updates the Extract Control table so that the next execution of this process will start with subsequently generated events Exception handling Occasionally, Tivoli Risk Manager events might contain insufficient information in the DST_HOSTNAME or DST_IPADDR columns to classify the target host component of an event. In these cases, there is no way to assign a measurement, so these events are handled as exceptions. The ETL process writes key fields of the invalid events into the exception log, HRM.Exception_Log, and then removes the events from staging table, HRM.Extract_Archive. Analyze the data in table HRM.Exception_Log and take corrective actions, if possible. Column Error_Code in the table indicates how the destination host attributes failed processing. One of 3 values is possible: NULL_DST_HOST when the DST_HOSTNAME and DST_IPADDR columns are both NULL INV_DST_HOST when the DST_IPADDR column is NULL and DST_HOSTNAME column does not begin with an alphabetic character INV_DST_IP when the DST_HOSTNAME column is NULL and the DST_IPADDR column has an invalid IP address: N/A,..., non-numeric or not containing any decimal points. 19

20 The timestamp32 column in the exception table contains the latest value of timestamp32 in the Tivoli Risk Manager archive table for all invalid events that match the same event category, source, and destination host/ip parameters. There might be many source events that match these parameters, but they are not all stored in the exception table, although they are all deleted from the staging table, HRM.Extract_Archive. If it is possible to modify the DST_HOSTNAME or DST_IPADDR columns in the source archive table, rm_t_arc41, then modify these columns so that they contain valid host component values. Contact your database administrator for assistance in performing this function. In the long run, it is advisable to correct the Tivoli Risk Manager adapter/sensor that is generating the invalid destination host data. 5.2 HRM_m5_Build_Mart_Process This process performs the complete data mart ETL. It extracts new Tivoli Risk Manager measurements from the central data warehouse (CDW) and converts the measurements to facts in the Tivoli Risk Manager data mart database. The process also extracts new Tivoli Risk Manager host and event components, plus their attributes, from the CDW and converts the data to dimensions in the data mart. Finally, it prunes old facts from the data mart and runs any prepackaged reports that have been configured to run automatically. The process runs in six steps that are described below. The process is configured to start with the first step and run each succeeding step if and only if the preceding step succeeds. This process is linked to the previous process (HRM_c5_Load_CDW_Process) through a warehouse shortcut, so that it will automatically start if the previous process completed successfully. Therefore, scheduling HRM_c5_Load_CDW_Process is sufficient to causing this process to run at the appropriate time. This process has the following steps: HRM_m5_s1_Pre_Extract Source: TWH_CDW tables TWG.MsmtTyp, TWG.MSrc Target: TWH_CDW staging table HRM.Event_Metric This step reinitializes staging table HRM.Event_Metric in the CDW, creating a row for each Tivoli Risk Manager measurement type. The staging table has the same structure as the metric dimension table in the data mart database. This step by default enables minimum, maximum, average and total metrics for all measurements. In the hourly fact table, all measurement values for any hour are the same, but in the daily, weekly and monthly fact tables, the minimum, maximum and average values might be different from the total value HRM_m5_s2_Extract Source: TWH_CDW staging table HRM.Event_Metric, private table HRM.ClassCatDesc, warehouse tables TWG.Msmt, TWG.Comp, TWG.CompAttr Target: TWH_MART translation dimension tables HRM.T_Event_Metric, HRM.T_Dst_Host, HRM.T_Src_Host, and staging tables HRM.Stage_ClassCat, HRM.Stage_F_Event_Hour This step adds new Tivoli Risk Manager event data from the central data warehouse (CDW) to staging tables and translation dimension tables in the Tivoli Risk Manager data mart. The staging tables are dropped and re-created each time this step is run. The translation dimension tables are permanent and have a structure exactly the same as their corresponding data mart dimension tables, but in addition contain information to identify the CDW (in a multi-cdw environment) where the data originated, as well as the original component and measurement IDs. Translation dimension tables are used to track IDs from the original CDW into the star schema so you can tell where the data actually came from when looking at a star schema. The target staging and translation dimension tables are the following: 2

21 HRM.Stage_ClassCat contains all Tivoli Risk Manager event category descriptions defined in CDW table HRM.ClassCatDesc. HRM.T_Event_Metric contains all Tivoli Risk Manager measurement types expressed as data mart metrics. Currently, Tivoli Risk Manager has only one metric Event Count. HRM.T_Dst_Host contains one record for each distinct destination host in all the processed Tivoli Risk Manager events. Each record maps from an IP_HOST, IP_INTERFACE or HRM_HOST component in the CDW. Each record also stores the hostname and IP address values from the LAST_IP_ADDRESS or HRM_DST_HOSTNAME attributes, if they exist. Finally, the table contains, for each host, the IP address represented as an integer, plus strings for the subnets represented by each portion of the dotted decimal IP address. Three source database (CDW) views and several staging/intermediate tables are used in this step to populate this table. HRM.T_Src_Host contains one record for each distinct source host in all the processed Tivoli Risk Manager events. Each record maps from a HRM_EVENT component, and its corresponding HRM_SRC_HOSTNAME or HRM_SRC_IPADDR attribute, in the CDW. HRM.Stage_F_Event_Hour contains one record for each new measurement added to the CDW. Each record contains foreign key integer fields pointing to the metric, event category, destination, and source host translation dimension tables. Extract control is achieved by using the component and measurement IDs from the CDW tables. The ExtCtl_To_IntSeq value is set to the maximum ID currently in the CDW, as defined by an appropriate view. After the extraction is complete, this value is copied into ExtCtl_From_IntSeq to be used for the next extraction. The following table shows which tables and ID columns are used for extract control. Target Tables (TWH_MART) Source Tables (TWH_CDW) Source Table Views (TWH_CDW) Source Table Columns used for Extract Control (TWH_CDW) Source Table Extract Control Views (TWH_CDW) HRM.T_Event_Metric HRM.Event_Metric HRM.VD_Event_Metric Metric_ID HRM.VE_Event_Metric HRM.T_Dst_Host TWG.Comp, TWG.CompAttr HRM.VD_Host_Dst_IP, HRM.VD_Host_Dst_NoIP, HRM.VD_Host_Dst_Name Comp_ID HRM.VE_Dst_Host HRM.T_Src_Host HRM.Stage_F_Event_Hour TWG.Comp, TWG.CompAttr TWG.Msmt, TWG.Comp, TWG.CompAttr HRM.VD_Host_Src Comp_ID HRM.VE_Src_Host HRM.VF_Stg_Evt_Hour Msmt_ID HRM.VE_Event_Hour HRM_m5_s3_Load Source: TWH_MART staging tables HRM.Stage_ClassCat, HRM.Stage_F_Event_Hour; translation dimension tables HRM.T_Event_Metric, HRM.T_Dst_Host, and HRM.T_Src_Host 21

22 Target: TWH_MART fact table HRM.F_Event_Hour; dimension tables HRM.D_ClassCat, HRM.D_Event_Metric, HRM.D_Dst_Host, HRM.D_Src_Host This step loads the dimension tables and hourly fact table with Tivoli Risk Manager data from the translation dimension tables and a staging fact table. Translation dimension tables are used to track IDs from the original CDW into the star schema so you can tell where the data actually came from when looking at a star schema. Each insert uses 'where not is' logic to ensure that duplicate dimensions or facts are not inserted HRM_m5_s4_Rollup Source: TWH_MART tables HRM.Stage_F_Event_Hour, HRM.F_Event_Hour; TWH_MD table IWH.STARSCHEMA Target: TWH_MD table RPI.SSUPDATED; TWH_MART tables HRM.F_Event_Day, HRM.F_Event_Week, HRM.F_Event_Month This step rolls up new hourly facts from the staging fact table into the daily, weekly, and monthly fact tables. Timestamps for daily/weekly/monthly facts are converted from GMT to local time to comply with the expectation of the Tivoli Enterprise Data Warehouse reporting interface (RPI). If new data is rolled up for any of the above tables, the appropriate star schema (hourly, daily, weekly, monthly) is indicated as updated in table RPI.SSUPDATED, which enables any prepackaged reports based on the updated schema to be automatically regenerated HRM_m5_s5_Prune Source: TWH_MART table HRM.Prune_Mart_Control Target: TWH_MART tables HRM.F_Event_Hour, HRM.F_Event_Day, HRM.F_Event_Week, HRM.F_Event_Month, HRM.Prune_Mart_Log This step deletes facts from the hourly, daily, weekly, and monthly fact tables if the fact date is older than the prune control values. Prune control values are held in table HRM.Prune_Mart_Control and can be adjusted by the customer as desired. For the hourly fact table, adjustment is made for GMT but the other fact tables are in local time. The number of records deleted from each fact table is stored in permanent table HRM.Prune_Mart_Log HRM_m5_s6_Run_Report Source: TWH_MART table HRM.Stage_F_Event_Hour Target: TWH_MD table RPI.SSUPDATED This step runs a user-defined program named runreport.sh, which is installed by Tivoli Enterprise Data Warehouse. The runreport.sh program looks for records inserted by the rollup step into the RPI.SSUPDATED table. If a record exists that identifies a star schema that has been updated, then the report execution engine runs all scheduled reports that have been created from that star schema. You can remove this step if you do not want the scheduled reports to run. 22

23 6 Central data warehouse schema implementation Before reading this section, read about the generic schema for the Tivoli Enterprise Data Warehouse central data warehouse, which is described in Enabling an Application for Tivoli Enterprise Data Warehouse. That document defines the content of each table and explains the relationships between the tables in this document. Shaded columns in the following tables contain data that is translated. Installing and Configuring Tivoli Enterprise Data Warehouse contains instructions for installing support for additional languages. 6.1 Component configuration Component type (table CompTyp) CompTyp_Cd CHAR(17) CompTyp_Parent_Cd CHAR(17) CompTyp_Nm * VARCHAR(12) CompTyp_Strt_DtTm CompTyp_End_DtTm IP_HOST NULL IP Host IP_INTERFACE NULL IP Interface HRM_HOST NULL Unqualified Tivoli Risk Manager Host HRM_EVENT NULL Tivoli Risk Manager Event Component (table Comp) Comp _ID INTEG ER CompTy p_cd CHAR (17) Centr _Cd CHAR (6) Cust_ID INTEGE R Comp_ Corr_ID INTEGE R Comp_ Nm VARCH AR (254) Comp_Corr _Val VARCHAR (254) Comp_Strt_ DtTm Comp_End_ DtTm Comp_Ds VARCHAR (254) 1 IP_HOST CDW 1 joe.ibm. com IP_INTER FACE 3 HRM_HO ST CDW CDW 1 ted HRM_EV ENT CDW 1 1 WEBbob.ibm.com HRM_EV ENT CDW 1 2 DOSbob.ibm.com

24 Comp _ID INTEG ER CompTy p_cd CHAR (17) Centr _Cd CHAR (6) Cust_ID INTEGE R Comp_ Corr_ID INTEGE R Comp_ Nm VARCH AR (254) Comp_Corr _Val VARCHAR (254) Comp_Strt_ DtTm Comp_End_ DtTm Comp_Ds VARCHAR (254) 1 HRM_EV ENT CDW 1 2 TROJjim HRM_EV ENT CDW 1 2 VIRUSsally HRM_EV ENT CDW 1 1 DOSfred.ibm.com Component relationship type (table RelnTyp) RelnTyp_Cd CHAR(6) PCHILD RelnTyp_Nm * VARCHAR(12) Parent Child Relation Component relationship rule (table RelnRul) CompTyp_Source_Cd CHAR(17) CompTyp_Target_Cd CHAR(17) RelnTyp_Cd CHAR(6) RelnRul_Strt_DtTm RelnRul_End_DtTm IP_HOST HRM_EVENT PCHILD IP_INTERFACE HRM_EVENT PCHILD HRM_HOST HRM_EVENT PCHILD Component relationship (table CompReln) CompReln_ID Comp_Source_ID Comp_Target_ID RelnTyp_Cd CHAR(6) CompReln_ Strt_DtTm CompReln_ End_DtTm PCHILD PCHILD PCHILD