Performance Tuning for MDM Hub for IBM DB2

Size: px

Start display at page:

Download "Performance Tuning for MDM Hub for IBM DB2"

Phyllis Lucas
5 years ago
Views:

Performance Tuning for MDM Hub for IBM DB2 2012 Informatica

1 Performance Tuning for MDM Hub for IBM DB Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation.

2 Abstract Performance tuning for MDM Hub for IBM DB2 ensures increased performance and improved data throughput. This article is a performance tuning guide that contains procedures and configuration recommendations that help increase Hub performance. Supported Versions MDM Multidomain Edition for DB Table of Contents Overview The IBM DB2 Configuration Settings Recommended Settings for IBM DB2 Database Configuration Parameters Setting the IBM DB2 Database Configuration Parameters Recommended Settings for IBM DB2 Database Manager Parameters Setting the Database Manager Parameters Recommended Settings for IBM DB2 Registry Variables Setting the IBM DB2 Registry Variables Improving Match API Performance Important Considerations About REORGCHK... 6 Performing a REORGCHK on a STRP Table Performing a REORG Benefitting from I/O Parallelism Determining the Path of each Tablespace and the Database Transaction Logs Moving a Tablespace or Changing the Location of a Transaction Log Overview Performance tuning for your MDM Hub installation is important to ensure optimal Hub performance. This article outlines multiple steps that you can take to increase the performance of Informatica MDM Hub for IBM DB2. This article provides instructions for configuration of the IBM DB2 database, the database layout, and the database maintenance procedures to optimize Hub performance. The IBM DB2 Configuration Settings DB2 configuration settings are applied automatically if the MDM Hub store database was created with the script db2ant. You must manually change the DB2 configuration settings if the MDM Hub store database was created manually. You must optimize the settings for the list of parameters discussed in this section to increase the performance of MDM Hub for IBM DB2. 2

3 Recommended Settings for IBM DB2 Database Configuration Parameters The following table lists the IBM DB2 database configuration parameters that should be optimized, along with their recommended settings, and a description of the impact of these settings: Configuration Parameter Recommended Setting Description LOCKLIST AUTOMATIC Various operations in MDM processes use row level locking to complete tasks and support concurrency. Depending on the number of rows being processed during these operations, DB2 can acquire a high number of row level locks. If the incoming volume differs greatly in size, optimizing the setting for LOCKLIST is difficult, unless LOCKLIST is set to a very high number. You should set this parameter to AUTOMATIC to allow the database manger to determine the appropriate value. A conservatively tuned LOCKLIST value can cause lock escalations, which may cause some of the MDM operations to fail due to lock timeouts. MAXLOCKS AUTOMATIC MAXLOCKS limits the percentage of the LOCKLIST that a single application can consume. Most MDM operations are executed under the scope of a single application. Such single application can acquire a high number of row level locks, which may occupy a large amount of LOCKLIST. Predicting an appropriate value for the incoming volume that varies greatly is difficult unless MAXLOCKS is set to a very large value. You should set this parameter to AUTOMATIC to allow the database manager to choose the appropriate value. A conservatively tuned MAXILOCKS value can cause lock escalations, which may cause some of the MDM operations to fail due to lock timeouts. CATALOGCACHE_SZ or higher The memory for this parameter is allocated from the database shared memory. DB2 stores system catalog information in the catalog cache. Informatica MDM is comprised of many dynamic SQLs that reference a number of its metadata objects. A larger catalog cache allows DB2 to retain information for some of these objects from the system catalogs in memory. This results in quicker compilation of the subsequent dynamic SQLs that require these same objects. Informatica MDM hub consists of many metadata tables which are accessed frequently so CATALOGCACHE_SZ should be set to or higher. LOGBUFSZ 4096 or higher Most of the operations in MDM are logged through db2 transaction logs. These log records are buffered in log buffer before they are written out to the disk. Larger log buffer will allow the log records to be written out less frequently. This will make log related disk IO much more efficient. Default value for this database configuration parameter is not sufficient for an average MDM environment. The recommendation is to tune this parameter to 4096 pages or larger. LOGFILSIZ or higher A single MDM transaction can contain many DML queries. Some of these queries work with high number of rows depending on the configured batch size for each phase of the MDM workflow. Hence, a single transaction can cause very large number of log records that may span across many logs. Having a large log file size avoids the need to create new log files very frequently. Creating new log files frequently can be expensive, especially on IO bound systems. Size of the total log space for a database equals to total number of logs (primary + secondary) multiplied by the log file size. It is very important to have adequate log space. This ensures MDM transactions do not fail running out of log space. Such situation may require significant time rolling back MDM transactions. Setting LOGFILSIZ to is the first step in avoiding this situation. Please refer to our recommendations on number of primary and secondary logs for complete equation of log space LOGPRIMARY LOGSECOND All primary log files are created upon activation of the database. Secondary log files are created one at a time as needed only after primary log space is exhausted by an uncommitted transaction. Secondary logs act as a backup in cases where long transactions can exhaust the entire primary log space. 3

4 Configuration Parameter Recommended Setting Description Log file creation can be expensive based on the size of the log file. The recommended settings for LOGPRIMARY and LOGSECOND assure enough primary log space is pre-allocated with database activation. Having sufficient primary log space increases transaction performance because frequent secondary log file creation is avoided. The 100 secondary logs can cover any unexpected long transaction caused by large incoming volumes, especially during the batch operations. The sum of the values of LOGPRIMARY and LOGSECOND should be 200. PCKCACHESZ Informatica MDM hub has many dynamic SQLs. Every dynamic statement will have a compiled package associated with it. DB2 caches these packages in package cache memory. Properly configured cache will avoid package cache overflows. Package cache overflows can cause performance degradation. Begin with value for this parameter and monitor during different phases of MDM. Tune this parameter further if frequent package overflows are observed. STMTHEAP AUTOMATIC Statement heap is used for compiling a query. Limited statement heap may prevent optimizer from evaluating all possible access plans for a query. This may result in sub optimal plan causing performance degradation. Setting this memory to automatic will allow optimizer to cover all possible access plans during the compilation before concluding the best one. SORTHEAP AUTOMATIC Inadequate sort heap can cause large sorts to spill from memory to disk. As disk IOs are much slower compared to memory, such sort spills can cause queries to run longer. The performance degradation becomes a lot more obvious with larger spills. Setting this value to automatic can avoid sort spill as sort memory can grow on demand. Informatica MDM hub performs many sorts in all its phases. SHEAPTHRES_SHR AUTOMATIC This parameter sets an upper limit to the total amount of database shared memory that can be allocated to all the sorts at one time. It is recommended to set this parameter to automatic if SORTHEAP is set to automatic. UTIL_HEAP_SZ or higher During certain batch operations, Informatica MDM hub uses DB2 LOAD utility to move data between tables. DB2 LOAD utility uses the utility heap to complete the data movement process. The size of this heap can impact the performance of the LOAD operation. Setting this parameter to an appropriate value will provide better throughput for these MDM operations. Setting the IBM DB2 Database Configuration Parameters You can manually configure the IBM DB2 database configuration parameters. u Run the following DB2 command for each database configuration parameter that you need to configure. db2 update db cfg for <Database_Name> using <Parameter_Name> <Parameter_Value> 4

5 Recommended Settings for IBM DB2 Database Manager Parameters The following table lists the IBM DB2 database manager parameters that you should optimize, along with their recommended settings, and a description of the impact of these settings: Configuration Parameter Recommended Setting Description INTRA_PARALLEL YES DB2 can split a single query into multiple sub-sections. These sub-sections can be executed in parallel in multiprocessor (SMP) systems. To exploit this capability in MAX_QUERYDEGREE DFT_DEGREE (database configuration parameter) ANY ANY DB2, INTRA_PARALLEL must be set to YES. Informatica MDM hub has many complex queries that can take advantage of this feature in DB2. It is best to let DB2 choose the degree of parallelism by setting MAX_QUERYDEGREE to ANY and DFT_DEGREE to ANY. Most of the INSERT statements in Informatica MDM hub use SELECT queries. Such inserts can also take advantage of intraparallelism in DB2. SHEAPTHRES 0 Setting this parameter will make all sorts shared. The overall shared shot heap threshold will be controlled by SHEAPTHRES_SHR variable. This will also allow the tuning of SHEAPTHRES_SHR to be set to automatic. Setting the Database Manager Parameters You can manually configure the IBM DB2 Database Manager parameters. u Run the following DB2 command to configure each Database Manager parameter. db2set <Parameter_Name>=<Parameter_Value> Recommended Settings for IBM DB2 Registry Variables The following table lists the IBM DB2 registry parameters that require optimization, their recommended settings, and a description of the impact of these settings: Registry Variable Recommended Setting Description DB2_INLIST_TO_NLJN NO Under certain circumstances, the DB2 SQL compiler may rewrite IN list predicate as a join. Such rewrites may provide better performance if there is index defined on the joined columns. When the optimizerdoes not have accurate statistics information, it may not be able determine the best join method for the rewritten join. Setting this variable to NO will prevent the optimizer from favoring nested loopjoin in such cases. DB2_ANTIJOIN YES MDM Hub has queries that use NOT EXISTS subqueries. This setting will allow DB2 to look for possibilities to transform NOT EXISTS subqueries into anti-joins which can be processed more efficiently by DB2. DB2_REDUCED_OPTIMIZATION REDUCE_LOCKING This setting favours NLJOIN over MSJOIN whenever possible to reduce the amount of locking on the outer table. 5

6 Setting the IBM DB2 Registry Variables You can manually configure the IBM DB2 registry variables. u Run the following DB2 command to configure each registry variable. db2set <Parameter_Name>=<Parameter_Value> Improving Match API Performance Routinely reorganize each STRP table (<base_object>_strp) to increase the performance of Match API queries. In the MDM Hub, there is generally one STRP table for every base object. The following table lists the tasks necessary for reorganizing each STRP table: Task Perform a REORGCHK on each STRP table to determine if a REORG is necessary. Reference Performing a REORGCHK on a STRP Table on page 7 Perform a REORG of each STRP based on the Primary Key. Performing a REORG on page 8 Manually perform a second REORGCHK to update the STRP table statistics, or automatically perform RUNSTATS on each STRP table after an automatic REORG. Performing a REORGCHK on a STRP Table on page 7 Important Considerations Reorganizing STRP table based on SSA_KEY (primary key) column causes some performance degradation during batch match operation. A few queries in the batch match operation search the STRP table based on ROWID_OBJECT. These queries degrade in performance if the STRP table is organized based on SSA_KEY. The amount of degradation in batch match operation must be tested to compare with the benefits to the Match API. If all tokens are regenerated before batch match operation, MDM truncates the STRP table and regenerates tokens for all base object records. This causes the STRP table to be organized based on ROWID_OBJECT. Hence, batch match operation will not degrade in performance. After completion of batch match operation, the STRP table should be reorganized based on SSA_KEY to ensure good performance of the Match API. In the case of partial token generation, the STRP table must be reorganized based on ROWID_OBJECT before batch match operation. This ensures good performance for batch match operation. After the completion of the batch match operation, once again, the STRP table must be reorganized based on SSA_KEY column to ensure good performance of the Match API. About REORGCHK The CLUSTERRATIO value of the primary key displayed in the REORGCHK results determines if a STRP table REORG is required. Perform a REORGCHK of the STRP tables frequently so the STRP tables can be reorganized before Match API performance decreases. For optimal performance, the CLUSTERRATIO should be as close to 100 as possible. Establish when to reorganize the STRP table by noting the CLUSTERRATIO value when degraded Match API performance is observed. Reorganize the STRP table before the CLUSTERRATIO approaches the value noted during degraded Hub Match performance. 6

7 The CLUSTERRATIO is displayed in column F4 of the REORGCHK results. In the following example, the primary key index, CMX_ORS.SVR1_546, has a CLUSTERRATIO value of 0, indicating the STRP table requires a reorganization based on that primary key: Performing a REORGCHK on a STRP Table The IBM DB2 REORGCHK command determines if the STRP table needs to be reorganized. You do not need to perform a REORGCHK if you plan to perform the REORG as a scheduled task. Perform the REORGCHK frequently for each STRP table. 1. Run the following command from the IBM DB2 command prompt: db2 REORGCHK UPDATE STATISTICS ON TABLE CMX_ORS.<STRP_TABLE_NAME> The STRP table statistics are displayed. 2. In the STRP table statistics, note the CLUSTERRATIO value under column F4. If the CLUSTERRATIO is close to 100 and not approaching the value associated with degraded Match API performance, a REORG on the STRP table is not required. If the CLUSTERRATIO is approaching the value associated with degraded MDM Hub Match performance, go to step Note the Primary Key name for the STRP table. 4. Perform a REORG on the STRP table. 7

8 Performing a REORG Perform a REORG when the CLUSTERRATIO is approaching a value associated with degraded MDM Hub Match performance. The REORG command increases query performance by clustering the data according to the primary key of the STRP table. 1. Run the following command from IBM DB2: db2 REORG TABLE CMX_ORS.<base_object_name>_STRP INDEX CMX_ORS.<SCHEMA_INDEX_VALUE> The STRP table is reorganized. 2. Update the statistics so the DB2 Optimizer can use the table layout generated by the REORG to create the access plan for queries against the reorganized table. If the REORG was launched manually, perform a REORGCHK to update statistics. You can also confirm that the CLUSTERRATIO is now an acceptable value. See About REORGCHK on page 6. If the REORG was launched in an automated process, the statistics should also be updated as part of the automated process by running the following RUNSTATS command: db2 RUNSTATS ON TABLE CMX_ORS.<base_object_name>_STRP AND INDEXES ALL Benefitting from I/O Parallelism Informatica MDM Hub creates many IBM DB2 tablespaces to organize metadata, indexes, and temporary tables. The following table describes these tablespaces and their functions: Tablespace Type of Tablespace Description CMX_REPOS Data Contains all tables that store Hub metadata. CMX_DATA Data Contains all the user tables and their related metadata tables. CMX_INDX Data Contains all the ORS index objects. CMX_TEMP Temporary Contains all the shared temporary tables created during batch operations. CMX_USER_TEMP Temporary Contains temporary user data for declared temporary tables and their indexes. During many of the HUB batch operations, the Hub moves data between temporary tables and regular tables. Many of these transactions are logged, particularly when Batch API Interoperability is enabled. Moving data between tables and logging transactions requires a significant amount of I/O. If all the I/Os occur on a single disk drive, I/O wait times can increase, which degrades Hub performance. A single CPU can drive many disks at one time. A CPU is not fully utilized if processes keep getting blocked for I/Os. Increasing I/O parallelism reduces the amount of blocked I/Os. You can achieve I/O parallelism by keeping data tablespaces, temporary tablespaces, and transaction logs on different physical disk drives. Batch operations can access all the paths in parallel, which increases the throughput by reducing the I/O waits. If more physical disk drives are available, you can increase I/O parallelism further by extending parallelism to the container level. To increase I/O parallelism, place all containers for a tablespace on different physical disks. This allows various DB2 prefetchers and I/O cleaners to access these containers in parallel without blocking each other, providing an even bigger increase in throughput. 8

9 Determining the Path of each Tablespace and the Database Transaction Logs Determine the path of each tablespace and its containers to ensure the data tablespaces, temporary table spaces, and database transaction logs are located on different disks. 1. Run the following command to display the list of tablespaces. db2 list tablespaces A list of tablespaces is displayed along with the tablespace IDs. 2. Note the tablespace IDs of the CMX_REPOS, CMX_DATA, CMX_INX, CMX_TEMP, and CMX_USER_TEMP tablespaces. 3. For each tablespace ID noted, run the following command to determine the path of each tablespace container. db2 list tablespace containers for <TABLESPACE_ID> The "Name" field in the output indicates the path for each tablespace container. 4. Run the following command to determine the path of your transaction log. db2 get db cfg for <DATABASE_NAME>. The value of the parameter "Path to log files" indicates your current transaction log path. 5. Compare the paths of the data tablespaces, the temporary tablespaces, and the database transaction logs. If the data tablespaces, the temporary tablespaces, and the database transaction logs are kept on separate physical disks, the disks are optimized for I/O parallelism. If any of the data tablespaces, the temporary tablespaces, and the database transaction logs are not kept on separate physical disks, move them. Moving a Tablespace or Changing the Location of a Transaction Log Use the db2relocatedb command to move a tablespace or change the location of a transaction log. See the IBM DB2 documentation for more information on the db2relocatedb command. Authors Anirudh Patel Ivan Begg Joon Wong The MDM documentation team. 9

IBM DB2 LUW Performance Tuning and Monitoring for Single and Multiple Partition DBs

IBM DB2 LUW Performance Tuning and Monitoring for Single and Multiple Partition DBs Day(s): 5 Course Code: CL442G Overview Learn how to tune for optimum the IBM DB2 9 for Linux, UNIX, and Windows relational