Workload Control Center Performance Tuning

Transcription

1 Workload Control Center Performance Tuning This document includes recommendations and best practices for performance tuning for CA Workload Control Center (WCC), for the following releases: Unicenter Workload Control Center r1 SP4 Unicenter Workload Control Center r11 The document includes general recommendations, feature-specific recommendations, and release-specific information, as well as procedures and checklists for the recommendations. General Recommendations General recommendations for performance tuning are as follows: 1. Make sure that the WCC server machine meets the recommended specifications identified in the README. The WCC readme includes details of hardware requirements in the System Requirements section. 2. Turn off unused features If you are not using features such as Host Access or Event Console, you can disable these services on the WCC server. 3. Remove unused portlets from the workplace Review the default pages to ensure that portlets in the pages are actually being used. The default pages should contain the portlets that are the most heavily used. 4. Modify the default maxthreads and maxmemory settings for each Tomcat server (including Portal) The maxthreads parameter is located in installation_root\unicenter Workload Control Center\*Server\conf\server.xml. This parameter should be set to 2x the maximum expected number of concurrent users for the feature. The maxmemory parameter should be to a maximum of 1024 to 1200 on Windows. You must ensure that there is adequate memory on the machine for the settings; the JVM will not load if maxmemory cannot be allocated. This parameter is located in installation_root\unicenter Workload Control Center\*Server\conf\wrapper.conf. Job Status Console has a slightly different location: installation_root\unicenter Workload Control Center\JobStatusConsoleServer\JobSServer\conf\wrapper.conf Note: Windows 2003 Server edition does not support more than 4 GB RAM. To utilize more than 4 GB of RAM, you must use Windows 2003 Enterprise. 5. Limit logging to Error/Info levels

2 Workload Control Center includes extensive logging for all components. By default, logging levels are set to INFO or ERROR. To optimize performance, ensure that you do not set logging values to DEBUG. The common components (specifically JBoss for UWCC r11 and etrust Directory) also have more extensive logging than is needed for normal operations. To optimize performance, ensure that you have modified these logging parameters as well. 6. Check the network as well as machine usage, especially if a subset of users have problems Occasionally, there will be a problem with network bandwidth in an area of the enterprise. If there is a subset of users who are having problems, a simple test is to perform a ping to see what the transit time is between the user s machine and the WCC server. 7. Resolve resource contention issues for virtual servers and shared servers. If WCC is installed on a shared server or virtual server, ensure that there are no resource contention problems. For example, if WCC is installed on a virtual server that hosts multiple environments and you experience periodic performance drops, verify that the other environments are not causing resource utilization spikes. 8. Ensure there are no database connection limitations WCC utilizes database connections for CA-7 and for AutoSys 4.x. For AutoSys 4.x, ensure that any limits to the number of database connections are not being violated. For CA-7, you must ensure that ACBs are sized appropriately. We recommend that you work with CA Support to set this up. An approximate number is: Number of concurrent users + connection pool + CCI terminals for batch processing 9. Modify the initial and maximum heap sizes on client computers (Job Flow Monitoring only) Feature-specific Recommendations Each feature contains an overview of how the cache is constructed, plus specific recommendations for optimizing performance for that feature. Job Editor Job Editor Cache The initial expansion of server/instance fetches object names. This data goes into user-based cache on the WCC server. o In r11, there are some properties cached on the WCC server, but these are minimal (group, application)

3 Selecting an object fetches the properties of that object. The property data goes into the client-based cache. Saving an object refreshes the properties of the object in the cache. Executing a Refresh on the server will refresh the object names on the user s server-based cache and will flush the property data from the cache. Job Editor Recommendations Follow the General recommendations For CA 7, do the following: o Set the Connection Pool Size to an appropriate size, based on the number of expected concurrent users who will be using the connection pool. o Ensure that the number of ACBs on the mainframe is set to an appropriate value Job Flow Monitoring Job Flow Monitoring Cache Job flows are processed on the server. Each job flow s data is maintained separately. All users share the same data for a job flow that is on production. User ID: o CA-7 and AutoSys r11: the Job Flow is activated under the monitor ID (CA 7 and AutoSys r11). o AutoSys 4.x: the Job Flow is activated under the database user ID. Status data and structure updates (dependency changes) are fetched under the user ID, and are updated across all users. For the following actions there is additional load because the user s credentials are utilized, rather than the common monitoring ID: o Executing job actions (Start, Force Start, Requeue, Cancel, etc.) o Accessing the Requirements View o Accessing the Properties View Job Flow Monitoring Recommendations Follow the General recommendations Modify the following parameters, as follows: o Increase the Number of Threads. The Number of Threads should be set to at least the number of Job Flows on production. o Increase the Number of Caches. o Reduce Session Idle Timeout o Lengthen Update Interval values (AutoSys and CA 7) o CA-7 Increase Connection Pool Size Flow Rendering only: o Reduce the Job Flow Maximum Job Number, Job Flow Maximum Number of Levels, and Job Flow Maximum Cutoff Level Client Machine Recommendations Modify the initial and maximum heap sizes on client computers to improve flow rendering. Reduce the Monitoring Refresh Interval to provide a faster refresh rate for the applet, for specific users.

4 Job Status Console Job Status Console Cache The Job Status Console cache is user ID-based (the userid that is used to connect to the job manager server rather than the login user ID). If data is available in the cache, the 2nd user accessing the data will get the cached data rather than a new fetch. Cache is automatically refreshed when a user hits Refresh in a view. More complex/more options than JFM or Job Editor o Configuration/Environment/Job Status Console Cache Properties Properties o Caching for AutoSys - Enable Cache Size o Set to # views + expected # of concurrent users of JSC o No downside to setting it higher rather than lower. Cache Validity Period o Set higher to use cache more. Low validity period = very fresh data, but will cost more in memory usage. o Recommended is the refresh rate on your views. Cache Timeout o Set to validity period. Maintenance Interval runs check to remove. Job Status Console Recommendations Follow the general recommendations, including optimizing the maxthreads and maxmemory settings The Global Session is for the Views portlet. Utilizing this makes the Views portlet much more efficient. You must provide a valid Global User ID/password if you enable the Global Session. Alerts DB o For AutoSys only enterprises, alerts may not be required o o Decrease Alert Deletion Threshold value. Set Alert Deletion Threshold to 0 and restart JSC to clean up all the existing alerts View Size Calculation o Cost per job, cost per user ID. To accommodate more views, decrease the Session Inactivity Timeout value. Consider increasing the Portlet Refresh Interval and update interval for views on Development/Test instances. To accommodate more concurrent users, increase the following parameters: o Pool Size: set to Connection Pool Size o Connection Pool Size: set at 5 to 10. o Model Worker Pool Size controls threading. Increase this value above the default (5) if the WCC server is NOT CPU-bound.

5 Release-specific Recommendations Unicenter Workload Control Center r1 SP4 Ensure that the Connection Pool Maximum Size parameter in the Portal properties is set to an adequate level for the number of concurrent users in the enterprise. Modify etrust Directory logging settings, as described in the Procedures section. Unicenter Workload Control Center r11 Modify JBoss logging settings, as described in the Procedures section. Modify etrust Directory logging settings, as described in the Procedures section. Scheduling Engine Recommendations AutoSys 4.5 If you are experiencing problems with the number of concurrent users, please check with AutoSys Support. There is a known issue with RCS and concurrent users for which there is a fix. AutoSys r11 AutoSys r11 utilizes an application server to manage secure communication with the AutoSys database. The application server is typically installed on the AutoSys Scheduler machine. If the machine where the application server is installed is heavily used, the machine may become CPU-bound. To maximize performance, install the instance of the application server that is used by WCC to a lower-load machine. There is a known issue in the AutoSys API that affects both Job Editor and Job Flow Monitoring. Please check with AutoSys support for this fix. CA 7 (all releases) Ensure that the number of ACBs on the mainframe is set to an appropriate value. Ensure that CPS parameters are set appropriately. Typical values for these parameters are: o SET SLIMIT=50 -- Maximum Active Sessions o SET SESSTOUT=60 -- Session Timeout (in minutes) no activity o SET PASSMODE=DORM -- CPS Pass Ticket Processing (PASS, DORM, FAIL)

6 Procedures The following section details procedures referenced above. Increase the initial and maximum heap sizes on a client computer If you are experiencing problems with loading very large flows in Job Flow Monitoring, you can increase the initial and maximum heap heap size in the Java browser plug-in. 1. Click Start, Settings, Control Panel. The Control Panel window appears. 2. Double click the Java Plug-in icon. The Java Plug-in Control Panel window appears. 3. Click the Advanced tab of the Java Plug-in Control Panel. 4. In the Java Runtime Parameters field, enter the following: -Xms256m -Xmx400m 5. Click Apply. The initial and maximum heap sizes are updated. Note: The Xmx parameter can cause problems since the Java Virtual Machine will fail to load if it can t get enough contiguous memory from the OS to satisfy the amount requested in the Xmx param. This is typically a problem on Windows when the Xmx is set to 512m or higher. Set the maxthreads parameter for the Tomcat servlet engine The maxthreads parameter is located in server.xml in installation_root\unicenter Workload Control Center\*Server\conf\server.xml. This parameter should be set to 2x the maximum expected number of concurrent users for the feature. <Connector acceptcount="100" connectiontimeout="20000" debug="0" disableuploadtimeout="true" enablelookups="false" maxsparethreads="75" maxthreads="150" minsparethreads="25" port="10129" redirectport="8443"/> Set the maxmemory parameter for the Tomcat servlet engine The maxmemory parameter is located in wrapper.conf in installation_root\unicenter Workload Control Center\*Server\conf. Job Status Console has a slightly different location in installation_root\unicenter Workload Control Center\JobStatusConsoleServer\JobSServer\conf. On Windows, this parameter should be set to a maximum of 1024 to However, you must ensure that there is adequate memory on the machine for the

7 settings. The Java Virtual Machine (JVM) will not load if maxmemory cannot be allocated. #******************************************************************** # Wrapper Properties #******************************************************************** # Java Application wrapper.java.command=../../jre/bin/java wrapper.debug=false wrapper.java.initmemory=16 wrapper.java.maxmemory=1024 Modify the Connection Pool Maximum Size This procedure only applies to the Portal release (Portal 4.71) embedded in the UWCC r1 SP4 release. To modify this parameter, do the following: 1. Login to Workload Control Center as the Portal admin user. When the workplace opens, click My Profile (upper right-hand corner of page). The Profile page opens. 2. Click Portal Administration. A new window for Portal Administration parameters opens. 3. In the left pane menu, scroll down to Properties, then click Database. The Database properties open in the right pane. 4. Modify Connection Pool Maximum Size to an appropriate value, then click Update Changes. The change is registered. Reduce JBoss logging This procedure prevents JBoss from logging informational messages, restricting logging to only error messages. This procedure only applies to UWCC r Modify JBOSS_HOME/server/default/conf/log4j.xml 2. Find "<root>" 3. Add the following tag to <root> <priority value="error"/> Reduce etrust Directory logging This procedure prevents etrust Directory from creating daily summary logs in the <install drive>\program Files\CA\eTrust Directory\dxserver\logs directory. 1. Navigate to <install drive>\program Files\CA\eTrust Directory\dxserver\config\logging 2. Open itechpoz-machinename.dxc. 3. Insert a # at the beginning of the line set summary-log = "logs/$s.log" to comment out the logging command. The snippet will appear as follows: # Daily summary log #set summary-log = "logs/$s.log"; 4. Save Open itechpoz-machinename.dxc

8 5. Open itechpoz-machinename-router.dxc. 6. Insert a # at the beginning of the line set summary-log = "logs/$s.log" to comment out the logging command. The snippet will appear as follows: # Daily summary log #set summary-log = "logs/$s.log"; 7. Save Open itechpoz-machinename-router.dxc. Note: You may also want to remove or archive any log files in the <install drive>\program Files\CA\eTrust Directory\dxserver\logs directory. Set the JSC Global Session Parameters 1. Open the WCC Configuration Manager. 2. Click the Environment tab, then click Job Status Console Server. The Job Status Console Server page opens. 3. From the Use Global Session drop-down list, choose True. 4. In the Global User ID field, enter the ID for the Job Status Console global user. 5. In the Global User Password field, enter the password for the Job Status Console global user. 6. In the Global Confirm User Password field, enter the password for the Job Status Console global user. 7. Click Save. 8. Click the Deploy tab. The Deploy page opens with the Deploy link selected by default. 9. In the Deploy Comments field, type appropriate text to indicate your changes. 10. Click Deploy. The comment you typed now appears in the Last Deploy Comments field. 11. Click the Services link. The Services page opens. This page will indicate what services need to be restarted following the deploy. Restart the indicated services. Important! If you do not plan to use the Global Session, make sure that you change the Use Global Session parameter to False. By default, the value for this parameter is True. Note: The CA 7 user must have list and FOJOB access. The AutoSys user can have read-only access. Checklists General Recommendations 1 Make sure that the WCC server machine meets the recommended specifications identified in the README. 2 Turn off unused features 3 Remove unused portlets from the workplace 4 Modify the default maxthreads and maxmemory settings for each Tomcat server (including Portal)

9 5 Limit logging to Error/Info levels for UWCC features as well as common components 6 Check the network as well as machine usage, especially if a subset of users have problems 7 Resolve resource contention issues for virtual servers and shared servers. 8 Ensure there are no database connection limitations Job Editor 1 Modify the default maxthreads settings for the Job Editor Tomcat instance (AdminServer) 2 Modify the default maxmemory settings for the Job Editor Tomcat instance (AdminServer) 3 Ensure there are no database connection limitations 4 Limit logging to Error/Info levels 5 CA 7 Only: Set the Connection Pool Size to an appropriate size, based on the number of expected concurrent users who will be using the connection pool 6 CA 7 Only: Ensure that the number of ACBs on the mainframe is set to an appropriate value Job Flow Monitoring Base Job Flow Monitoring 1 Modify the default maxthreads settings for the Job Flow Monitoring Tomcat instance (MonitorServer) 2 Modify the default maxmemory settings for the Job Flow Monitoring Tomcat instance (MonitorServer) 3 Monitoring Parameters: Increase Number of Threads 4 Monitoring Parameters: Increase Number of Caches 5 Monitoring Parameters: Reduce Session Idle Timeout 6 Monitoring Parameters: Lengthen Update Interval values (AutoSys and CA 7) 7 Monitoring Parameters, CA 7 Only: Set the Connection Pool Size to an appropriate size, based on the number of expected concurrent users who will be using the connection pool Job Flow Monitoring - Flow Rendering Only 1 Monitoring Parameters: Reduce the Job Flow Maximum Job Number 2 Monitoring Parameters: Reduce the Job Flow Maximum Number of Levels 3 Monitoring Parameters: Reduce the Job Flow Maximum Cutoff Level

10 4 Client Machines: Modify the initial and maximum heap sizes for the client Java browser plug-in 5 Client Machines: Reduce the Monitoring Refresh Interval for specific users Job Status Console 1 Modify the default maxthreads settings for the Job Status Console Tomcat instance (JobStatusConsoleServer) 2 Modify the default maxmemory settings for the Job Status Console Tomcat instance (JobStatusConsoleServer) 3 Enable the Global Session and Global User 4 Alerts Database: For AutoSys only enterprises, determine if alerts are needed, or if alarms meet the needs of the enterprise 5 Alerts Database: Decrease Alert Deletion Threshold value 6 Alerts Database: Set Alert Deletion Threshold to 0 and restart JSC to clean up all the existing alerts 7 View optimization: Decrease the Session Inactivity Timeout 8 View optimization: Consider ncreasing the Portlet Refresh Interval and Update interval for views on Development/Test instances 9 Concurrent user optimization: Set Pool Size to the value of the Connection Pool Size 10 Concurrent user optimization: Set Connection Pool Size to 5 to Concurrent user optimization: Model Worker Pool Size (controls threading) - increase this value above the default (5) if the WCC server is NOT CPU-bound UWCC r1 SP4 1 Portal: Modify the Connection Pool Maximum Size 2 etrust Directory Component Modify logging UWCC r11 1 etrust Directory Component Modify logging 2 Modify JBoss logging CA 7 (all releases) 1 Set ACBs to an appropriate size AutoSys Apply patch for RCS concurrent users

11 AutoSys r11 1 Ensure that the machine on which the AutoSys Application Server instance utilized by UWCC is installed is not CPU-bound. If it is, consider relocating this Application Server instance. 2 Apply ASAPI fix (affects Job Editor, Job Flow Monitoring)