IBM Tivoli Identity Manager Performance Tuning Guide Version 4.6

Transcription

1 IBM Tivoli Identity Manager Performance Tuning Guide Version 4.6 Issue Date: 2005 June 30 First Edition Publication Number: SC

2 Copyright Notice Copyright IBM Corporation All rights reserved. May only be used pursuant to a Tivoli Systems Software License Agreement, an IBM Software License Agreement, or Addendum for Tivoli Products to IBM Customer or License Agreement. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any computer language, in any form or by any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise, without prior written permission of IBM Corporation. IBM Corporation grants you limited permission to make hardcopy or other reproductions of any machine-readable documentation for your own use, provided that each such reproduction shall carry the IBM Corporation copyright notice. No other rights under copyright are granted without prior written permission of IBM Corporation. The document is not intended for production and is furnished as is without warranty of any kind. All warranties on this document are hereby disclaimed, including the warranties of merchantability and fitness for a particular purpose. U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation. Trademarks IBM, the IBM logo, Tivoli, the Tivoli logo, AIX, IBM DB2, IBM Tivoli Identity Manager and WebSphere Application Server are trademarks or registered trademarks of International Business Machines Corporation or Tivoli Systems Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others. Notices References in this publication to Tivoli Systems or IBM products, programs, or services do not imply that they will be available in all countries in which Tivoli Systems or IBM operates. Any reference to these products, programs, or services is not intended to imply that only Tivoli Systems or IBM products, programs, or services can be used. Subject to valid intellectual property or other legally protectable right of Tivoli Systems or IBM, any functionally equivalent product, program, or service can be used instead of the referenced product, program, or service. The evaluation and verification of operation in conjunction with other products, except those expressly designated by Tivoli Systems or IBM, are the responsibility of the user. Tivoli Systems or IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, New York , U.S.A.

3 Table of contents Table of contents...1 About this guide...3 Who should use this guide Introduction Vital tunings Initial tunings Resource allocation Memory CPU Disk space J2EE application servers IBM WebSphere Application Server Java Virtual Machine (JVM) size Java virtual machine (JVM) options Sun Solaris IBM HTTP Server Connections IBM WebSphere MQ IBM Tivoli Identity Manager application Database servers IBM DB Quick guide for setting the IBM DB2 tuning parameters Bufferpools Connections Tablespaces Transaction logs Indexing Runstats Lock tuning Lock type Lock timeout Microsoft SQL Server Oracle Database Server Configuration Updating Statistics Directory servers IBM Tivoli Directory Server Quick guide for setting the IBM Tivoli Directory Server tuning parameters Bufferpools Connections Transaction logs Indexing Runstats Sun ONE Directory Server All IDs Threshold Value Indexing Database cache size Operating Systems Red Hat Best practices Troubleshooting Log files Identifying bottlenecks Specific problems IBM Tivoli Identity Manager application...27 IBM Tivoli Identity Manager Performance Tuning Guide Page 1

4 8.3.2 IBM WebSphere MQ IBM Tivoli Directory Server Sun ONE Directory Server IBM DB2 performance monitoring Enable monitoring Snapshots Bufferpool hit ratio IBM DB2 Statement Monitor Other resources Appendix A scripts and files perftune_runstats.sh...34 Page 2 IBM Tivoli Identity Manager Performance Tuning Guide

5 About this guide This guide identifies some ways to tune your IBM Tivoli Identity Manager system to improve performance. Who should use this guide Use this guide if you are responsible for installing or maintaining an IBM Tivoli Identity Manager system. The following competencies are recommended: Familiarity with basic database and directory design principles. General knowledge of system resource management (memory, disk, and processor) including your operating system s memory model. Without knowledge of other resource requirements on the system it is possible to negatively impact performance by over tuning. Understanding of how to configure and administer your directory and database servers. You may need to have your local database administrator or directory administrator perform these tunings for you. IBM Tivoli Identity Manager Performance Tuning Guide Page 3

6 1 Introduction The IBM Tivoli Identity Manager product solves the complex problem of identity management. Due to the complexity of the problem, itcan be challenging to optimize the use of resources by IBM Tivoli Identity Manager or in other words, to tune. This tuning guide provides a system administrator with the information needed to tune the application for your environment. Other individuals (such as IBM DB2 or IBM Tivoli Directory Server administrators) in your organization might offer differing advice. In our experience, system administrators know your environment better and their advice may be more accurate for your environment than this tuning document. The IBM Tivoli Identity Manager product can be divided into four major pieces: Java 2 Platform Enterprise Edition (J2EE) application servers, the IBM Tivoli Identity Manager application, database servers, and directory servers. We will address each of these separately in this document. The IBM Tivoli Identity Manager server can be installed as either a single server or clustered servers. A clustered environment can be considered a group of single servers with regard to tuning. This document is a working document. As more information is gathered settings may be added, removed or changed in future editions. It is recommended that you check the IBM web site for the most recent version. To find the most recent version, go to Select Search technical support, type ITIM Tuning Guide in the search box, and click submit. IBM Tivoli Identity Manager configurations vary significantly by platform, operating system, middleware applications, and hardware being used. You must tailor your configuration settings for your deployment. Configurations and settings used in this document include AIX, Windows, and Solaris in single server and cluster configurations. 1.1 Vital tunings There are several thousand different parameters that you can modify to tune J2EE application servers, the IBM Tivoli Identity Manager product, directory servers, and database servers. This tuning guide discusses a small subset of these parameters that have proven effective during performance testing. If you are setting up an acceptance or production environment, read each section and perform the applicable tunings for your systems. If you are setting up a test environment and want to get started as quickly as possible, focus on these areas: IBM DB2 - Indexing Indexing the following columns in the ITIM database was found during performance testing to significantly improve workflow performance. After adding these indexes you ll need to run the runstats command on these tables to make sure the IBM DB2 optimizer knows to use them. Check for the existence of these indexes using the following commands: db2 describe indexes for table ENROLE.PROCESS db2 describe indexes for table ENROLE.PROCESSDATA db2 describe indexes for table ENROLE.SCHEDULED_MESSAGE Rrun the following IBM DB2 runstats commands: db2 create index PROCESS_SUB_X on ENROLE.PROCESS ( SUBMITTED DESC, PARENT_ID ASC) db2 create index PROCESS_ID_ST on ENROLE.PROCESS ( ID ASC, STATE ASC) db2 create index PROCESSDATA_ID_DEF on ENROLE.PROCESSDATA ( PROCESS_ID ASC, DEF_ID ASC, VALUE_LAST_MODIFIED ASC) db2 create index SCH_MSG_SVR on ENROLE.SCHEDULED_MESSAGE ( SERVER ASC) Page 4 IBM Tivoli Identity Manager Performance Tuning Guide

7 Runstats IBM DB2 - Bufferpools IBM WebSphere Application Server - Java Virtual Machine (JVM) size The IBM DB2 - Indexing Indexing the following columns in the ITIM database was found during performance testing to significantly improve workflow performance. After adding these indexes you ll need to run the runstats command on these tables to make sure the IBM DB2 optimizer knows to use them. Check for the existence of these indexes using the following commands: db2 describe indexes for table ENROLE.PROCESS db2 describe indexes for table ENROLE.PROCESSDATA db2 describe indexes for table ENROLE.SCHEDULED_MESSAGE Rrun the following IBM DB2 runstats commands: db2 create index PROCESS_SUB_X on ENROLE.PROCESS ( SUBMITTED DESC, PARENT_ID ASC) db2 create index PROCESS_ID_ST on ENROLE.PROCESS ( ID ASC, STATE ASC) db2 create index PROCESSDATA_ID_DEF on ENROLE.PROCESSDATA ( PROCESS_ID ASC, DEF_ID ASC, VALUE_LAST_MODIFIED ASC) db2 create index SCH_MSG_SVR on ENROLE.SCHEDULED_MESSAGE ( SERVER ASC) Runstats tunings is a vital part of the IBM Tivoli Identity Manager product performance when IBM DB2 is used as the database server. See the Initial tunings section to tune this parameter in a newly deployed environment. 1.2 Initial tunings Most of these tunings can be implemented in a newly deployed environment or an environment that is already deployed. When tuning IBM DB2 in a newly deployed environment it is important to prime your database statistics using the IBM DB2 runstats command. The runstats command creates information in the IBM DB2 statistics tables based on information currently in the data tables. However, in a pristine environment, those tables are empty and the statistics generated from runstats on these tables are useless to IBM DB2. Without adequate information on the distribution of data within these tables, IBM DB2 makes poor choices on how to fulfill the queries used by the IBM Tivoli Identity Manager application. IBM Tivoli Identity Manager initializes the table statistics at configuration time so that IBM DB2 will make the best choices for a small startup configuration. As transactions are processed and the database grows, these initial statistics will no longer be enough to ensure good performance. The solution to this problem is to prime your database by doing a small number of transactions, running the runstats command, and then running the full transaction. The IBM DB2 - Indexing Indexing the following columns in the ITIM database was found during performance testing to significantly improve workflow performance. After adding these indexes you ll need to run the runstats command on these tables to make sure the IBM DB2 optimizer knows to use them. Check for the existence of these indexes using the following commands: IBM Tivoli Identity Manager Performance Tuning Guide Page 5

8 db2 describe indexes for table ENROLE.PROCESS db2 describe indexes for table ENROLE.PROCESSDATA db2 describe indexes for table ENROLE.SCHEDULED_MESSAGE Rrun the following IBM DB2 runstats commands: db2 create index PROCESS_SUB_X on ENROLE.PROCESS ( SUBMITTED DESC, PARENT_ID ASC) db2 create index PROCESS_ID_ST on ENROLE.PROCESS ( ID ASC, STATE ASC) db2 create index PROCESSDATA_ID_DEF on ENROLE.PROCESSDATA ( PROCESS_ID ASC, DEF_ID ASC, VALUE_LAST_MODIFIED ASC) db2 create index SCH_MSG_SVR on ENROLE.SCHEDULED_MESSAGE ( SERVER ASC) Runstats section discusses how to use the runstats command. Failing to prime the database in this manner can result in poor performance or transaction rollbacks. Example: If the goal is to load 20,000 users into the IBM Tivoli Identity Manager product using a DSML feed, load the first 1000 people using the DSML file first. After the load completes, run the IBM DB2 runstats command and then load the remaining users with the DSML file. 1.3 Resource allocation Tuning values are more complex to manage when more than one piece of middleware is running on a given system. For example having the IBM Tivoli Identity Manager server, IBM DB2, and IBM Tivoli Directory Server all on the same machine. Regardless of configuration, it is important to calibrate the following resources so that they are not over-allocated Memory All middleware components allow you to adjust how much memory they will use. When calculating how to allocate memory to middleware components, keep these considerations in mind: Configuring middleware memory settings too high can result in the operating system swapping memory out to disk if the physical memory is exceeded. This will result in extremely poor performance and should be avoided. After setting up or changing the memory values for the middleware, monitor the memory and swap space used to ensure that nothing is being swapped out to disk. If it is, adjust your memory settings to compensate. 32-bit processes can only allocate 2GB (or AIX) to 4GB (on Solaris) of RAM. If a 32-bit process is configured to allocate more than the OS-specific limit on process memory, the application (such as IBM Tivoli Directory Server) could halt or unexpectedly fail. IBM Tivoli Directory Server has internal caches in addition to the IBM Tivoli Directory Server process. Ensure that the size of your caches plus the size of your IBM Tivoli Directory Server process does not exceed 2GB. When this 2GB limit is reached, the IDS server will refuse new connections and fail. The entry cache size limit determines the number of entries in the cache, not the size of the cache. The size of each cache entry will vary based on the IBM Tivoli Identity Manager product configuration and any extensions to the base IBM Tivoli Directory Server schema. The default cache values may exceed the 2GB limit in rare cases. Although the bufferpools account for a large amount of the memory used by IBM DB2, the application control heaps, the sort heaps, and the statement heaps also use memory. Do not overlook these sizes when computing how much memory to allocate to IBM DB2. See the IBM DB2 section. A large part of the WebSphere Application Server s memory usage is the JVM size. However, the size of the JVM does not set an upper bound on the amount of memory that the WebSphere Page 6 IBM Tivoli Identity Manager Performance Tuning Guide

9 Application Server may use. See the J2EE application servers - IBM WebSphere Application Server section CPU All the components of the IBM Tivoli Identity Manager product (IBM Tivoli Identity Manager application, J2EE application server, database server, and directory server) are CPU intensive. This document groups the J2EE application server and the IBM Tivoli Identity Manager application together because it is difficult to isolate their CPU usage. Keep in mind the following considerations: In IBM Tivoli Identity Manager 4.6 role evaluation, policy enforcement and reconciliation have been enhanced by distributing work across multiple server threads and multiple TIM cluster members to complete these activities much faster than in previous TIM versions. Both IBM Tivoli Directory Server and IBM DB2 are multi-threaded (multi-process in DB2 s case) applications and will perform best on a multi-processor machine. Even in a well-tuned environment the system bottleneck(s) may vary between the CPU, memory and disk on the IBM Tivoli Identity Manager server, the directory server, and/or the database server. For this reason, the IBM Tivoli Identity Manager server, the directory server, and the database server can benefit by being deployed on separate machines. If that is not possible, put the database and directory server on the same machine with a high performance disk configuration using multiple disks and a high performance RAID configuration to provide fast read and write capacity Disk space Each of the middleware components uses different amounts of disk space for various purposes. The J2EE application server and the IBM Tivoli Identity Manager application use disk space beyond their installation size because of log files (such as the msg.log and trace.log files) and WebSphere MQ queues. Adjust the number of archives and size of the msg.log and trace.log files in the enrolelogging.properties file. Make sure that WebSphere MQ has enough disk space for its processing logs (not error logs) to grow. The IBM Tivoli Identity Manager server pushes many entries onto the queues during large provisioning changes, causing the queues to grow. IBM Tivoli Directory Server uses disk space both from the IBM Tivoli Directory Server process (log files like ibmslapd.log) and the IBM DB2 database. IBM Tivoli Directory Server uses system managed space (SMS) tablespaces that allow the system to manage the amount of disk space used. SMS tablespaces do not allow the specification of an upper bound so it is important to monitor the amount of disk space used to ensure that the drive does not become full. In addition to the tablespaces for the database data, IBM DB2 uses disk space for the transaction logs. When configuring the transaction logs (discussed in the IBM Tivoli Directory Server Transaction logs and IBM DB2 Transaction logs sections) ensure that you have enough disk space for the log files. The IBM Tivoli Identity Manager DB2 database uses directory managed space (DMS) tablespaces that require pre-allocating disk space for the database to use. If the tablespace becomes full, IBM DB2 will be unable to continue and will return an error in the msg.log and trace.log files. If this occurs, add more tablespace containers. As a general rule, for every 160,000 transactions that occur, 1GB of tablespace is needed to archive the historical transaction information. A policy change that makes modifications to 5,000 users is 5,000 transactions from a IBM Tivoli Identity Manager perspective. See the IBM DB2 Tablespaces section for additional information. IBM Tivoli Identity Manager Performance Tuning Guide Page 7

10 2 J2EE application servers Regardless of the installation type (single machine or cluster), the IBM Tivoli Identity Manager server can be thought of as two components: the J2EE application server running the application (WebSphere Application Server or BEA WebLogic Server) and the IBM Tivoli Identity Manager application itself. Both components need to be tuned. 2.1 IBM WebSphere Application Server WebSphere Application Server allows you to use a variety of settings to tune your environment. This document discusses the Java virtual machine (JVM) size, the maximum HTTP connections and Java Messaging Service queue size Java Virtual Machine (JVM) size By default, WebSphere Application Server sets the maximum JVM size to 256MB. This value is too small for the IBM Tivoli Identity Manager product to run beyond a basic concept test and should be increased to a minimum of 1GB. If your machine has adequate available RAM increase this value to as much as 1.5GB. For large reconciliations or role and policy evaluations the default values will not be enough memory to complete these tasks. Setting the maximum memory value too high can cause memory allocation problems in JAVA when 2GB is reached. The maximum JVM size used in this value is not the actual maximum allocated size of the JAVA heap, and as much as 15% is allocated to a portion of the heap for the system s use. IBM recommends that you not use a value higher than 1.5GB even if your system has the available memory. Do not set the JVM heap size to be larger than the physical RAM. The WebSphere Application Server suffers significant performance degradation if the operating system swaps out the JVM to swap space. Consequences of this include very slow UI performance, transaction rollbacks, timeouts, and high disk utilization. The WebSphere Application Server sets the initial JVM size to 0GB and IBM recommends using a value of 256MB. While previous JVM levels have worked well with the initial JVM size set the same as the maximum JVM size, setting these the same with version of the JVM (the version embedded in WebSphere Application Server 5.1) can cause memory fragmentation resulting in poor performance. initial_jvm_heap_size = The initial size of the JVM heap in megabytes. Recommended value: 256. max_jvm_heap_size = The maximum size of the JVM heap in megabytes. Recommended value: 1GB to 1.5GB. 1) Open the WebSphere Administration Console. 2) Expand the Servers list in the navigation pane. 3) Select Application Servers in the navigation pane. 4) Select the server to manage. 5) Select Process Definition from the Additional Properties pane at the bottom. 6) Select Java Virtual Machine from the Additional Properties pane at the bottom. 7) Set the Initial Heap Size to initial_jvm_heap_size. 8) Set the Maximum Heap Size to max_jvm_heap_size. Page 8 IBM Tivoli Identity Manager Performance Tuning Guide

11 9) Repeat this procedure for each IBM Tivoli Identity Manager server Java virtual machine (JVM) options Sun Solaris Note: This section applies specifically to WebSphere Application Server V5.1 on Sun Solaris, not AIX or Windows. WebSphere Application Server V5.1 for AIX ships with IBM s implementation of Java whereas WebSphere Application Server V5.1 for Sun Solaris ships with Sun s implementation of Java IBM s Java and Sun s Java have two different garbage collection algorithms and need to be tuned differently. If you are using WebSphere Application Server on AIX or Windows, skip this section. Sun s JVM on Sun Solaris supports two different JVM types: client and server. The client JVM is designed to have a faster startup time and is tuned for shorter-running applications than the server JVM. The server JVM is designed for longer-running applications and therefore sets different initial values to some JVM tunings. By default the WebSphere Application Server uses the client JVM. IBM Tivoli Identity Manager runs better under the server JVM than under the client JVM. Sun s JVM uses a generational garbage collection algorithm. How this algorithm works is outside the scope of this document, but we will give a short summary. See the Other resources section at the end of this document for links to more information. The following is a simplified view of how the generational garbage collection algorithm works. The JVM divides the Java heap size into two major components: the young generation, which includes Eden and two Survivor pools, and the old generation (sometimes called the tenured generation). When objects are allocated inside IBM Tivoli Identity Manager, they are created inside Eden. After Eden becomes full, a partial garbage collection occurs and any objects that are still being used (live objects) are moved to the first Survivor pool. Eden is cleared and object allocation begins again inside Eden. After Eden becomes full again, any live objects inside Eden and the first Survivor pool are moved to the second Survivor pool. Eden is cleared and object allocation begins again inside Eden. This process repeats with the objects being switched between the two survivor pools until objects die or are eventually tenured to the old generation. When there is no more space inside Eden or the Survivor pools, all remaining live objects are moved to the old generation. Eden is cleared and object allocation begins again. When the old generation becomes full, a full garbage collection occurs. Every object within the heap is analyzed and heap compaction occurs within the old generation. When partial or full garbage collections occur everything within the JVM stops. This causes small pauses within IBM Tivoli Identity Manager during garbage collections. Partial garbage collections finish very quickly, usually in less than half a second. Full garbage collections take longer to compact the entire heap and can take 10 seconds or more to finish, depending on the size of the heap. Because of this, it is important to minimize the number of garbage collections, particularly full garbage collections. We do this by adjusting the size of the Java heap, the new generation, and the survivor pools. A diagram illustrating the JVM heap layout: Eden Survivor pools Tenured Generation max_new_size survivor_ratio max_jvm_size new_ra tio jvm_type = The type of JVM to use, either -client or -server. Recommended value: -server max_new_size = The maximum size of the new generation within the JVM in megabytes. Set this IBM Tivoli Identity Manager Performance Tuning Guide Page 9

12 to 1/3 of your maximum heap size. new_ratio = The ratio of the new generation to the old generation. Set this to 2, thereby setting the size of your new generation to 1/3 of your heap size. survivor_ratio = The ratio of a single survivor pool to Eden. Set this to 3, thereby setting the size of each survivor pool to 1/5 of your new generation (note this is not 1/4 because there are two survivor pools). target_survivor_ratio = A target for the amount of survivor space to be used after a garbage collection. This value is used by the JVM to calculate for how many iterations an object will stay in the survivor pools before being tenured to the old generation. Set this to 75 to allow more time for objects to die in the survivor pool before being tenured. With a total JVM heap size of 512MB, the recommendations above would yield the following heap configuration: Eden 170MB Survivor pools 512MB Tenured Generation 342MB 102MB 34MB Calculating the values argument_string = jvm_type -XX:MaxNewSize=max_new_sizem -XX:NewRatio=new_ratio -XX:SurvivorRatio=survivor_ratio -XX:TargetSurvivorRatio=target_survivor_ratio Note: Ensure the MaxNewSize value has a trailing m specifying that the value is in megabytes. Example: For a JVM size of 512MB, use the following argument string: argument_string = -server -XX:MaxNewSize=170m -XX:NewRatio=2 -XX:SurvivorRatio=3 -XX:TargetSurvivorRatio=75 1) Open the WebSphere Administration Console. 2) Expand the Servers list in the navigation pane. 3) Select Application Servers in the navigation pane. 4) Select the server to manage. 5) Select Process Definition from the Additional Properties pane at the bottom. 6) Select Java Virtual Machine from the Additional Properties pane at the bottom. 7) Append argument_string to Generic JVM arguments 8) Repeat this procedure for each IBM Tivoli Identity Manager server IBM HTTP Server The WebSphere Application Server uses the IBM HTTP Server as a front-end server in a single-server installation and as a load balancer between nodes in a cluster installation. The default configuration parameters for the IBM HTTP Server work well for small and medium configurations but may need to be increased in environments with a large number of concurrent users. Page 10 IBM Tivoli Identity Manager Performance Tuning Guide

13 Connections The IBM HTTP Server can be configured to limit the number of connections that it will accept at one time. This value may be too small if the servers experience a large number of concurrent users. Connections are maintained to the TIM server until a user selects log-out (which doesn t always happen) or until WebSphere clears the connection after 10 minutes. The IBM HTTP Server supports the HTTP/1.1 KeepAlive request that allows a client to make multiple HTTP requests through a single persistent connection. There is a limit on the number of KeepAlive requests that a single connection will accept. After this limit is reached, the connection is closed and another connection must be established. The default value may be too small for some external provisioning processes such as a JNDI feed. ibmhttp_home = The home directory of the IBM HTTP Server, such as /usr/ibmhttpserver. max_connections (ThreadsPerChild on Windows) = The maximum number of connections that can be made to the HTTP server at one time. This should be set to the maximum number of concurrent users you expect on your system. Recommended value: 200. This may be increased to as high as 600 for customers with a large number of concurrent users. Edit the ibmhttp_home/conf/httpd.conf file and update the following entries: MaxClients max_connections Stop and restart the IBM HTTP Server for these changes to take effect IBM WebSphere MQ The WebSphere Application Server uses an embedded copy of the IBM MQSeries messaging service to manage Java Message Service (JMS) messages. The IBM Tivoli Identity Manager server configures several persistent queues (the exact name and number can vary slightly from version to version). The IBM Tivoli Identity Manager server uses these queues during various provisioning actions. During large provisioning policies it is possible to reach the limit on the number of uncommitted messages, so this value should be increased. queue_manager_name = The name of the queue manager, such as WAS_vanexel_jmsserver. You can find a list of valid queue manager names by running dspmq. max_uncommitted_messages = The maximum number of uncommitted messages. Default value: Recommended value: ) Start the MQ Script interpreter: runmqsc queue_manager_name 2) Alter the maximum uncommitted messages size: alter qmgr maxumsgs(max_uncommitted_messages) 3) Confirm the change: display qmgr maxumsgs 4) Exit the MQ Script interpreter: end IBM Tivoli Identity Manager Performance Tuning Guide Page 11

14 3 IBM Tivoli Identity Manager application The IBM Tivoli Identity Manager application includes several configuration files that provide an area for tuning various parts of the application s performance. These are in the data/ directory under the IBM Tivoli Identity Manager product home directory. Page 12 IBM Tivoli Identity Manager Performance Tuning Guide

15 4 Database servers The IBM Tivoli Identity Manager product supports three different database systems: IBM DB2, Microsoft SQL Server, and Oracle Database. Each database requires slightly different tunings. Each database machine should have at least one processor and 1GB of RAM. It can reside on a single processor machine by itself or share a multi-processor machine with other applications, but the requirements of 1GB of RAM per one CPU is the minimum for the database server. Tuning the database is one of the most important tunings for the IBM Tivoli Identity Manager product. 4.1 IBM DB2 Tuning IBM DB2 to run with the IBM Tivoli Identity Manager product involves adjusting the bufferpools, modifying the number of connections, modifying internal database values, adding tablespace, adjusting logs, indexing, and running runstats Quick guide for setting the IBM DB2 tuning parameters This section includes the commands needed to tune the IBM DB2 parameters. This section assumes you have been through the tunings before, know what the desired values are, and just need a reminder of how to set them. As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool enrolebp size enrolebp_npages db2 "ALTER TABLESPACE ENROLE_DATA ADD ( \ FILE '/database_home/database_name/node0000/sql00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" db2 ALTER TABLESPACE ENROLE_INDEXES ADD ( \ FILE '/database_home/database_name/node0000/sql00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP db2 ALTER TABLESPACE TEMP_DATA ADD ( \ FILE '/database_home/database_name/node0000/sql00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP db2 update db cfg for itim_database using logsecond logs_secondary db2 update db cfg for itim_database using logfilsiz logs_size db2set DB2_RR_TO_RS=YES Bufferpools IBM DB2 bufferpools are the secondary buffer for IBM Tivoli Directory Server. These bufferpools should be large enough that most table searches can be read directly from memory instead of using the disk. This value can be measured by looking at the hit ratio for the bufferpools. See the IBM DB2 performance monitoring section for information about calculating the bufferpool hit ratio. The IBM Tivoli Identity Manager database has two bufferpools: IBMDEFAULTBP and ENROLEBP. IBMDEFAULTBP is used as a buffer for tablespaces with small extent sizes (4k) and ENROLEBP is used as a buffer for tablespaces with large extent sizes (32k). Most of the tables within the IBM Tivoli Identity Manager database use the tablespace with a large extent size and thus will use the ENROLEBP bufferpool. Use a 1:3 memory ratio between IBMDEFAULTBP and ENROLEBP. First, determine the following values for your system: total_mem_avail = The total amount of physical memory in bytes. mem_for_itimdb = The amount of memory in bytes to allocate to the IBM Tivoli Identity Manager IBM Tivoli Identity Manager Performance Tuning Guide Page 13

16 database. This value should be small enough that it will reside in physical memory and not be swapped out to disk. Recommended value: (500MB) or greater. Next, calculate the following values: ibmdefaultbp_npages = (mem_for_itimdb / 4096) * 0.25 enrolebp_npages = (mem_for_itimdb / 32768) * 0.75 As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool enrolebp size enrolebp_npages Connections The IBM Tivoli Identity Manager server uses JDBC connections from the J2EE application server to communicate with the database. The number of connections from the application server to the database depends on the needs of the application. Both application servers allow you to set maximum connection values. Performance testing did not show any need to increase the maximum connection values from their default value of 50. The IBM DB2 variable MAXAPPLS must be set to a value greater than the maximum possible connections from all nodes. The default value for MAXAPPLS is AUTOMATIC, which is the recommended setting, however if you need to set it lower, we recommend setting MAXAPPLS to five more than the total maximum possible connections. itim_database_name = The name of your IBM Tivoli Identity Manager database, such as itim. num_connections = Maximum number of connections. See the J2EE application servers section to calculate the maximum number of connections and add five to that value. As the database administrator, connect to the database and run the following command: db2 update db cfg for itim_database_name using maxappls num_connections Tablespaces The IBM Tivoli Identity Manager product uses a Database Managed Space (DMS) tablespace to store its data. This type of tablespace gives better performance than System Managed Space (SMS) tablespaces but requires you to set aside disk space for the database to use. The default size of the IBM Tivoli Identity Manager tablespace is 512MB for ENROLE_DATA, 256MB for ENROLE_INDEXES, and 256MB for TEMP_DATA. This is frequently not large enough and should be increased by adding more containers to the tablespace. Add more tablespaces by using the IBM DB2 alter tablespace command. If possible, add files that reside on another physical drive because IBM DB2 performs better if a tablespace has multiple containers on multiple drives. The creation and adoption of these altered tablespaces is not immediate. Examine the output of the alter tablespace command as it is executing and rerun the command if the database is busy altering a tablespace database_home = the home directory of your database administrator, such as /home/db2inst1. database_name = the name of the database such as db2inst1. This is also a subdirectory under database_home. container_name = the name of the file to create to hold the tablespace container, such as enrole_data2. Page 14 IBM Tivoli Identity Manager Performance Tuning Guide

17 num_pages = the number of 32k pages to add to the tablespace. To calculate the number of pages from the amount of disk space, divide the size in megabytes by A 512MB tablespace would be pages. As the database administrator, connect to the database and run the following command: db2 "ALTER TABLESPACE ENROLE_DATA ADD ( \ FILE '/database_home/database_name/node0000/sql00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" db2 "ALTER TABLESPACE ENROLE_INDEXES ADD ( \ FILE '/database_home/database_name/node0000/sql00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" db2 "ALTER TABLESPACE TEMP_DATA ADD ( \ FILE '/database_home/database_name/node0000/sql00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" Transaction logs IBM DB2 keeps logs during transaction processing. During large transactions the default log number and sizes might be too small and cause transaction rollbacks. Increasing the size and number of log files available to IBM DB2 solves this issue. IBM DB2 has two types of log files: primary and secondary. Primary logs are allocated when the database is started and remain allocated until the database is stopped. Secondary logs are allocated as needed after the primary logs are full and are released after they are no longer needed. Increase the number of secondary logs to be prepared when large transactions occur. The default size of the log files is k pages, or 4MB. Increase this to k pages, or 40MB. Note that this will increase the size of both your primary and secondary log files. itim_database = The name of the IBM Tivoli Identity Manager database, such as itim. logs_secondary = The number of secondary logs. Recommended value: 12. logs_size = The size of the primary and secondary logs in 4k pages. Recommended value: As the database administrator, connect to the database and run the following commands: db2 update db cfg for itim_database using logsecond logs_secondary db2 update db cfg for itim_database using logfilsiz logs_size Indexing Indexing the following columns in the ITIM database was found during performance testing to significantly improve workflow performance. After adding these indexes you ll need to run the runstats command on these tables to make sure the IBM DB2 optimizer knows to use them. Check for the existence of these indexes using the following commands: db2 describe indexes for table ENROLE.PROCESS db2 describe indexes for table ENROLE.PROCESSDATA db2 describe indexes for table ENROLE.SCHEDULED_MESSAGE Rrun the following IBM DB2 runstats commands: IBM Tivoli Identity Manager Performance Tuning Guide Page 15

18 db2 create index PROCESS_SUB_X on ENROLE.PROCESS ( SUBMITTED DESC, PARENT_ID ASC) db2 create index PROCESS_ID_ST on ENROLE.PROCESS ( ID ASC, STATE ASC) db2 create index PROCESSDATA_ID_DEF on ENROLE.PROCESSDATA ( PROCESS_ID ASC, DEF_ID ASC, VALUE_LAST_MODIFIED ASC) db2 create index SCH_MSG_SVR on ENROLE.SCHEDULED_MESSAGE ( SERVER ASC) Runstats In order to efficiently execute queries, IBM DB2 requires information on the number of rows in the tables and what indexes are available. Run the runstats command on all tables in the database before and after large DSML loads and reconciliations. If you experience high CPU usage or poor IBM DB2 performance, run the runstats command on all the tables in the database. IBM DB2 reorgchk does not update index statistics and is not a replacement for runstats. To update index statistics, run the runstats command on each table individually. A script (perftune_runstats.sh) that detects the version of IBM DB2 and runs the runstats command against all tables for a given schema in a database is provided in Appendix A scripts and files. If the runstats command is run in a working environment, allow the connected applications to continue to write to the database. Use the "allow write access" option to allow users to write to a database while runstats is running. Use the runstats command on an idle or lightly-used database because it requires update locking on the system statistics table to update the database statistics. A database with a heavy load might experience transaction rollbacks due to the system acquiring locks on the tables that are used by the database optimizer to fulfill queries. Use the runstats command on every table in the ENROLE schema. As the database administrator, connect to the database and use the following command to get a listing of all tables in the schema: db2 list tables for all grep ENROLE For each table in the ENROLE schema, run the following IBM DB2 runstats command: db2 runstats on table ENROLE.table_name with distribution and \ detailed indexes all allow write access Lock tuning Lock type The IBM Tivoli Identity Manager application requires that RR_TO_RS locking be enabled. This setting allows select statements that involve rows that are deleted but not yet committed to be fulfilled without waiting for the transaction to complete. More information on this environment setting is available at: There are no values to be determined for this section. Run the following command as the IBM Tivoli Identity Manager database administrator: db2set DB2_RR_TO_RS=YES Page 16 IBM Tivoli Identity Manager Performance Tuning Guide

19 Stop and restart the database instance for this change to take effect Lock timeout The IBM Tivoli Identity Manager application can generally be classified an online transaction processing (OLTP) application. As such it is normally recommended that the lock timeout (locktimeout) value within IBM DB2 be decreased from the default value of -1, which represents infinity. However, it is strongly encouraged that you do not modify this from the -1 value. Not all components of the application recover from a lock timeout and making such a change can result in transaction rollback errors. 4.2 Microsoft SQL Server Steps to tune Microsoft SQL Server will be included here when those settings and their values are determined. 4.3 Oracle Database Server Configuration The default Oracle configuration uses a low performance setting, increasing the configuration to the middle performance setting provides faster Oracle performance. Consult with an Oracle DBA, Oracle administrator, or Oracle documentation for more information on tuning of the Oracle server Updating Statistics At regular intervals from 1 week to 1 month on a production IBM Tivoli Identity Manager system, or after a large amount of processing has been completed, database statistics should be gathered and updated. These statistics are used by Oracle to make query decisions on locating information that have a dramatic impact on how fast Oracle can return requests. The DBMS_STAT package must be installed to issue the following DBMS_STAT commands. database_instance = The name of the database instance such as enrole. Create a file named Oracle_dbms.stat_cmds.txt containing the following lines: exec dbms_stats.gather_schema_stats(ownname => 'database_instance', cascade => true); As the system user, execute sqlplus and run the following Oracle_dbms.stat_cmds.txt Generating statistics may take from several minutes to several hours for a large database. Execution should be issued during off-peak times. IBM Tivoli Identity Manager Performance Tuning Guide Page 17

20 5 Directory servers The IBM Tivoli Identity Manager product supports two different directory servers: IBM Tivoli Directory Server and Sun ONE Directory Server (formerly known as Netscape iplanet Directory Server). 5.1 IBM Tivoli Directory Server With respect to tuning, IBM Tivoli Directory Server can be divided into two parts: the IBM Tivoli Directory Server process and IBM DB2. In a well-tuned system, the IBM Tivoli Directory Server process and the IBM DB2 processes consume approximately the same amount of CPU. In a poorly tuned system, IBM DB2 can max out the CPU usage trying to fulfill queries in a poor manner. Both IBM Tivoli Directory Server and IBM DB2 have caches that speed up data retrieval. Optimizing your available memory is the key to tuning IBM Tivoli Directory Server. When a read request comes in to IBM Tivoli Directory Server, it checks the filter cache to see if has seen that search filter before. If it has, the results are pulled from the cache, otherwise the query goes to IBM DB2. After the search filter is evaluated, IBM Tivoli Directory Server pulls the entries that match the search filter from the entry cache. If the values are not in the entry cache it queries IBM DB2. For each request to IBM DB2, IBM DB2 checks to see if the data resides in a bufferpool. If not, it reads the value from the disk. Ideally, all requests to the directory server register an IBM Tivoli Directory Server cache hit or a IBM DB2 bufferpool hit for the quickest response. Queries that require disk access can be very slow Quick guide for setting the IBM Tivoli Directory Server tuning parameters This section includes the commands needed to tune the IBM Tivoli Directory Server parameters. This section assumes you have been through the tunings before, know what the desired values are, and just need a reminder of how to set them. As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool ldapbp size ldapbp_npages db2 update db cfg for ldap_database using logsecond logs_secondary db2 update db cfg for ldap_database using logfilsiz logs_size Stop IBM Tivoli Directory Server, edit ibmslapd.conf and update the following configuration options: ibm-slapddbconnections: 30 Stop and restart IBM DB2 and the IBM Tivoli Directory Server server for these changes to take effect Bufferpools IBM DB2 bufferpools are the secondary buffer for IBM Tivoli Directory Server. These bufferpools should be large enough that most table searches can be read directly from memory instead of using the disk. This value can be measured by looking at the hit ratio for the bufferpools. See the IBM DB2 performance monitoring section for information about calculating the bufferpool hit ratio. The IBM Tivoli Directory Server database (ldapdb2 by default) has two bufferpools: IBMDEFAULTBP and LDAPBP. IBMDEFAULTBP is used as a buffer for tablespaces with small extent sizes (4k) and LDAPBP is used as a buffer for tablespaces with large extent sizes (32k). Most of the tables in the ldapdb2 database use the tablespace with a small extent size and use IBMDEFAULTBP. Use a 3:1 memory ratio between IBMDEFAULTBP and LDAPBP. Page 18 IBM Tivoli Identity Manager Performance Tuning Guide

21 Allocate enough memory to the IBM DB2 bufferpools so the bufferpool hit ratio is greater than 95% and allocate the rest of the memory to the IBM Tivoli Directory Server process and the caches. The following formula can help in determining the bufferpool values for your machine. First, determine the values for your system: ldap_database = the name of the IBM Tivoli Directory Server database, such as ldapdb2. mem_for_ldapdb2_bps = the amount of memory in bytes to allocate to the ldapdb2 bufferpools. This value should be small enough that it will reside in physical memory and not be swapped out to disk. Recommended value: (500MB) or greater. Next, calculate the following values: ibmdefaultbp_npages = (mem_for_ldapdb2_bps / 4096) * 0.75 ldapbp_npages = (mem_for_ldapdb2_bps / 32768) * 0.25 As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool ldapbp size ldapbp_npages Connections There are two connection values to modify for IBM Tivoli Directory Server. The first value is the number of connections that IBM Tivoli Directory Server should make to the IBM DB2 backend. Set this to 30. The second value only applies to Windows systems. Windows allows up to 64 sockets in a select call for each thread. This limits the number of inbound connections to 64. To overcome this problem set the SLAPD_OCHANDLERS environment variable to specify the number of connection threads, each one supporting 64 connections. ldap_database_name = The name of your IBM Tivoli Directory Server database, such as ldapdb2. num_ldap_connections = the number of connections from IBM Tivoli Directory Server to the IBM DB2 backend. Recommended value: 30 (from the IBM Tivoli Directory Server Tuning Guide). For Windows systems, also determine the following values: max_num_ldap_connections = the maximum number of IBM Tivoli Directory Server connections (enrole.connectionpool.maxpoolsize) from the enrole.properties file. The IBM Tivoli Identity Manager application default is 100. itim_nodes = the number of IBM Tivoli Identity Manager nodes in your cluster. This is the sum of the WF and the UI components. oc_handlers = (max_num_ldap_connections * itim_nodes) / 64 To set the number of connections the IBM Tivoli Directory Server server will make to the IBM DB2 server, stop IBM Tivoli Directory Server, edit ibmslapd.conf, and find the stanza: dn: cn=directory, cn=rdbm Backends, cn=ibm Directory, cn=schemas, cn=configuration Change the following attribute: ibm-slapddbconnections: num_ldap_connections For Windows systems, also set the SLAPD_OCHANDLERS values. Stop IBM Tivoli Directory Server, edit ibmslapd.conf and find the stanza: IBM Tivoli Identity Manager Performance Tuning Guide Page 19