Overview of Cúram JMX Statistics

IBM Cúram Social Program Management Overview of Cúram JMX Statistics Document Version 1.0 IBM Cúram System Test contributors: - Patrick Cadogan, System Test Engineer, pcadogan@ie.ibm.com - Patrick Curry, Senior Performance & Scalability, pat.curry@ie.ibm.com - Daniel Moraru, Software Architect, daniel.moraru@ro.ibm.com - William W. Walsh, Senior Software Engineer, wwalsh@us.ibm.com 1

(c) Copyright International Business Machines Corporation 2015. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. 2

CONTENTS Table of Contents Contents...3 1 Introduction...4 2 Why JMX statistics are important...5 2.1 Evaluating the performance of your Cúram customizations...5 3 JMX statistics in Cúram...6 3.1 JMX statistics available in Cúram...6 3.2 How JMX statistics can be collected...7 4 Configuring JMX statistics for collection...8 4.1 Configuring on-demand JMX statistics collection...8 4.2Configuring timer-based JMX statistics collection...9 4.2.1 WebSphere Application Server...10 4.2.2 Batch processes...11 4.3 Visualizing JMX statistics collected with the timer...12 4.4 Enabling JMX statistics for outbound web service calls...12 5 Collecting Cúram JMX statistics...13 5.1 On-demand collection of JMX statistics...13 5.2 Timer-based collection of JMX statistics...13 6 Analyzing Cúram JMX statistics...14 6.1 JMX statistics Use Cases...14 6.2 The Cúram JMX Report Generator...15 7 JMX Transaction Tracing...16 Appendices...17 Appendix A: Use Case: Investigating Hung Threads...17 Appendix B: JMX statistics properties...21 Appendix C: app_batchlauncher.sample.xml...23 References...26 3

1 Introduction This article provides an overview of JMX statistics in Cúram to help you to get started. You can use JMX statistics in various ways to assess the overall health of your system and to troubleshoot specific performance issues. A fully deployed Cúram system is typically made up of many components, which can include: a firewall, load-balancer, web server, application servers running the Cúram applications, database, operating system, public and private networks, and various supported hardware components. To produce the high performance system that users expect, all of these components must be correctly configured and tuned. When performance problems do occur, it is essential that administrators are able to detect these issues early, and resolve them quickly. To do this effectively, all components of the system (end to end) should be monitored for potential performance issues. To help monitor the performance of the Cúram applications and batch processes, Cúram has implemented support for JMX statistics. JMX statistics provide an insight into how the Cúram application is performing. Typically, when performance issues occur, JMX statistics can provide an excellent way to assess the performance of the application and help identify performance issues in areas of the system like the database. 4

2 Why JMX statistics are important JMX statistics in Cúram help you to monitor your overall system's performance and to troubleshoot issues. The following are key reasons for using JMX statistics in Cúram: To evaluate the performance of your Cúram customizations. To investigate a performance problem. To actively monitor performance aspects of the Cúram application in the following situations: To take snapshots at regular intervals to compare performance between intervals. To take a snapshot of Cúram JMX statistics at a point in time to review key performance indicators. To monitor performance in real time (for example, using Jconsole). To compare performance / load-balancing between application servers. To include data when creating a PMR (Problem Management Report) to send to IBM Support. 2.1 Evaluating the performance of your Cúram customizations Plan to assess the performance impact of your Cúram customizations. Cúram products are highly customizable so that you can adapt Cúram to your specific needs. In fact, it is very common to make many customizations to the base Cúram product. Cúram customization generally involves the addition of: Business Process Objects SQL statements Application screens Customized code can impact overall performance, so it is vital that all customizations are measured for performance to determine its acceptability. Customer customizations generally inherit already JMX instrumented Cúram code so no additional JMX instrumentation work is needed. When evaluating the performance of customizations, it is a good idea to take a snapshot of the system performance before the customization, so that you can then compare performance before and after the customizations are made. 5

3 JMX statistics in Cúram Now that you understand the importance of JMX statistics let's dig deeper into the various types of JMX statistics that are collected in Cúram. Let's look at the types of JMX statistics that are collected, and then relate these JMX types to the Cúram applications/components. 3.1 JMX statistics available in Cúram The JMX statistics, their relationship to the Cúram application layers, and whether they are enabled by default (or not) can be confusing so we clarify this here. The following table provides an overview of the various types of JMX statistics that are provided in Cúram. The second column identifies the J2EE application layer that relates to each JMX statistics type and the third column identifies whether it is enabled by default or requires an administrator to enable it. JMX Type Application Layer / Description URL Servlet layer Yes BPO (Business Process Objects) EJB layer Yes SQL Data Access Layer Yes Transaction Tracing Table 1: JMX types supported in Cúram From Servlet, EJB and Data Access layers Enabled by default? No, due to potential performance impacts For the Servlet, EJB and Data Access layers, the following types of statistics are collected: Aggregated counters. In-flight (a snapshot of what is running at the time the JMX statistics were taken). JMX statistics are also collected for: Caches. Exceptions. Configuration properties. Outbound Web Service calls (disabled by default). End-User (disabled by default). 6

3.2 How JMX statistics can be collected JMX statistics can be collected by the following means: 1. On-Demand - this is possible only when the application is running in an Application Server. On-Demand is not supported if the application is running as a stand-alone application such as Cúram batch. 2. Timer-Based - via JVM properties a frequency and location for statistics files are specified, which are generated in the background. These are specifically for nonapplication server contexts; that is, Cúram batch programs. 3. Using Standard JMX monitoring tools like JConsole (not covered in this Getting Started guide). Next, let's map the relationships between Cúram and the JMX collection methods, and in addition identify which are enabled by default. The following table represents these relationships: Cúram Application Type Application Name Internal Curam.ear Supported. Portal CitizenPortal.ear, MDTWorkspace.ear, CPMExternalNS.ear, CPMExternalS.ear On-Demand collection Application property Enable Statistics Download must be set to true to enable. Only supported when these applications are installed in the same application server as Cúram. Application property Enable Statistics Download must be set to true to enable. SearchServer SearchServer.ear Not supported Other (BIRT, Spellchecker) CuramBIRTViewer.ear, SpellChecker.ear Not supported Timer-Based collection Supported. In some versions of Cúram the statistics may be incomplete, so ondemand statistics are preferred. Not supported Web services CuramWSWebServices.ear Not supported Not supported Batch processes Various batch programs Not supported Supported Table 2: Relationship of Cúram application types to JMX collection 7

4 Configuring JMX statistics for collection Because of their negligible performance impact, enable JMX statistics in the application server environment and for batch processes. Here we will cover configuring ondemand and timer-based collection. 4.1 Configuring on-demand JMX statistics collection First, access the JMX property settings in Cúram as as follows: 1. In your browser, go to https://hostname:port/curam and log in to Cúram as the sysadmin user. 2. Select System Configuration > Shortcuts > Application Data. 3. In the Property Administration window, select the drop-down arrow to the right of the Category fields; from the list of options select "Infrastructure - JMX parameters"; next select the Search button. 4. A list of JMX parameters are displayed under Properties. 5. Verify that the property Enable Statistics Download is set to the value true. If not, set it to "true" and publish the properties. 6. The property Authorized User for Statistics Download defaults to the Cúram sysadmin user and if you choose to change this you must chose to be a valid Cúram user. 7. If you change any properties, you must republish them to ensure they are reflected within the application server environment. Note for IBM WebSphere: The "ALL AUTHENTICATED" special subject must be assigned the "Monitor" and "Operator" roles in order to be able to collect and download Cúram JMX statistics. From the WebSphere administrative console, click the Users and Groups menu and from the Administrative group roles add the "Monitor" and "Operator" roles to the special subject "ALL AUTHENTICATED". 8

4.2 Configuring timer-based JMX statistics collection You can enable timer-based collection of JMX statistics for Cúram by completing the following steps. The following table identifies the JVM properties that relate to Timer-Based JMX statistics and their default value: Property name Default value Description curam.jmx.output_statistic s_timer_enabled curam.jmx.output_statistic s_timer_folder curam.jmx.output_statistic s_timer_period Table 3: Timer properties FALSE The value of the Java java.io.tmpdir property. Enables the collection of Cúram JMX statistics using a timer. Specifies the folder where statistics files are placed. 300000 (5 minutes) Choose a value that is appropriate for your volume and timespan of processing. If using in an application server environment consider using on-demand statistics to first become familiar with usage and cycles in your environment in order to specify a useful value here. Depending on the length of the batch run, consider changing this timer to correspond to the length of the run. Considerations to balance include: A small timer value might, depending on the length of the run, create more statistics files than are reasonable to review. A small timer value might cause many trace files to be created, causing significant disk space to be used by the statistics files. A large timer value might create less granular captures and might cause the end of processing to be missed. For long runs, this granularity might not be significant. 9

4.2.1 WebSphere Application Server You configure collection of timer-based collection of JMX statistics by specifying the properties above to the JVM where the application or batch process is running. For WebSphere, do this as follows: 1. Log in to the WebSphere administrative console 2. Navigate to: Servers > Server Types > WebSphere Application Servers > Server 3. Under 'Server Infrastructure' click 'Process Definition' 4. Under 'Additional properties' click 'Java virtual machine' 5. Add these parameters to 'Generic JVM Arguments', for example: -Dcuram.jmx.output_statistics_timer_enabled=true -Dcuram.jmx.output_statistics_timer_folder=/opt/JMXStats -Dcuram.jmx.output_statistics_timer_period=300000 (5 mins) 6. Save the changes to the master configuration. 7. Repeat the steps for all relevant servers (consider a different folder for each server). 8. Restart the servers. With this method of generating Cúram JMX statistics, the collected statistics are written to the file JMXStats.xml, and are saved to the folder defined in the curam.jmx.output_statistics_timer_folder property. 10

4.2.2 Batch processes Always configure Cúram JMX statistics, including when you run Cúram batch jobs. To enable collection of timer-based JMX statistics for Cúram batch processes requires that you modify the app_batchlauncher.xml script in $CURAMSDEJ/bin to add these Ant tasks to the <java> task that runs the curam.util.impl.batchlauncher program. For example:  <jvmarg value="-dcuram.jmx.output_statistics_timer_enabled=true" />  <jvmarg value="- Dcuram.jmx.output_statistics_timer_folder=/opt/JMXStatsForBatch" />  <jvmarg value="-dcuram.jmx.output_statistics_timer_period=300000" />  <jvmarg value="-dcuram.jmx.ws_outbound_statistics_enabled=true" /> For the curam.jmx.output_statistics_timer_folder property use a unique value per process and node to make reviewing the results easier. See Appendix C: app_batchlauncher.sample.xml for an example of these changes with additional properties to ease setting the output folder name to more easily identify output types and setting garbage collection trace. 11

4.3 Visualizing JMX statistics collected with the timer The JMX statistics files are written periodically (see the curam.jmx.output_statistics_timer_period property) into the specified folder (see the curam.jmx.output_statistics_timer_folder property). In this location are the statistics in.xml files and a related XML style sheet, JMXStats.xsl. When any of the XML files that are opened with a browser, they are automatically formatted based on the style sheet. For instance, right-click any of the XML files, select Open with, and choose your preferred browser: Illustration 1: Viewing JMX statistics output 4.4 Enabling JMX statistics for outbound web service calls If a batch process or online application makes outbound web services calls, it is important to have insight into the performance that is associated with them. JMX monitoring for web service calls can be enabled by using the property: curam.jmx.ws_outbound_statistics_enabled=true 12

5 Collecting Cúram JMX statistics Collecting Cúram JMX statistics varies depending on whether you have configured ondemand or timer-based statistics. 5.1 On-demand collection of JMX statistics The download of JMX statistics for the Internal Cúram application (Curam.ear) must be enabled by setting the application s configuration property Enable Statistics Download to true. The default user allowed to download JMX statistics is defined by the Authorized User for Statistics Download property, which defaults to: sysadmin. Steps for downloading on-demand statistics: 1. Using a browser specify this URL: https://appserver-hostname:port/curam/jmxstats.do 2. You are then prompted to log in; log in as user sysadmin. 3. Depending on your browser s settings you are then prompted to save the file JMXStats.zip; save it to your hard disk. Then to view the statistics: Unzip the JMXStats.zip file. Two files are extracted: JMXStats.xml and JMXStats.xls. Open the XML file with any browser that supports rendering with XSL files (all Cúram-supported browsers will work for this). See also the The Cúram JMX Report Generator section on additional ways to view the collected statistics. 5.2 Timer-based collection of JMX statistics See Visualizing JMX statistics collected with the timer. 13

6 Analyzing Cúram JMX statistics This section provides some examples of how you can use JMX statistics to analyze your system's performance. 6.1 JMX statistics Use Cases Cúram JMX statistics can be used in various ways to assess the health of your system and to troubleshoot specific performance issues; the following are some examples for how you can use JMX statistics to: Provide a general health check of the application. Create a performance baseline for the application. Find out why the application is slow. Investigate transaction timeouts or hung threads. Investigate the performance of a specific transaction. Assess the performance of deferred transactions. In Appendix A, we look in detail at the use case Investigating Hung Threads and show how you can use the Cúram JMX statistics to investigate this issue and narrow down the area of investigation. 14

6.2 The Cúram JMX Report Generator Analyzing JMX statistics directly from the JMXStats.xml and JMXStats.xsl files can be difficult as the output can be large and cumbersome to navigate. To remedy this, we have delivered a tool called the Cúram JMX Report Generator, which takes one or more JMXStats.xml files as input and generates an HTML output file (Summary.html). The Summary.html file makes it easier to analyze the collected JMX statistics. Cúram JMX Report Generator (CuramJMXReportGenerator.zip) is a Java-based tool that processes one or more JMXStats.xml files. It highlights significant performance indicators; allows for basic comparison of multiple files, and provides the ability to sort and filter statistics. Click here to access the Cúram JMX Report Generator tool. The following table describes the types of statistics you see, in addition to the system summary, when you open Summary.html. JMX statistic types In-flight BPO Stats In-flight SQL Stats In-flight URL Stats BPO Method Stats (aggregate) SQL Stats (aggregate) URL Stats (aggregate) Cache Stats GlobalCache Stats Exception Stats SQL Batch Stats Cúram Web Services Outbound Stats Examples of statistics (counters) collected Avg Elapsed Time, Invocations, Max Elapsed Time Avg Elapsed Time, Executions, Rows Returned Avg Elapsed Time, URL, Username Avg Elapsed Time, Invocations, Max Elapsed Time Avg Elapsed Time, Executions, Rows Returned Avg Elapsed Time, Invocations, Max Elapsed Time Hits, Misses Evictions, Hits, Last access time, Layer, etc. Occurance, Username Avg Elapsed Time, Executions, Max Statements, etc. Avg Elapsed Time, Invocations, Max Elapsed Time, etc. Table 4: JMX statistics in the Summary.html file 15

7 JMX Transaction Tracing Cúram JMX transaction tracing can be a very useful mechanism when investigating performance issues. However, because it can have a significant impact on performance, if not configured correctly, transaction tracing is off by default. You should only to use transaction tracing when investigating issues, and only for a short period. You can enable JMX statistics for Transaction Tracing as follows: 1. Login in the specific Cúram Application server as sysadmin. 2. Click System Configuration. 3. Open the shortcuts sidebar, and click the Application Data; click the Property Administration link. 4. Search for the following property and change to value specified: Property name Set value to: Description Enable Transaction Tracing TRUE Whether transaction tracing is enabled or not in the application. We hope to cover more on transaction tracing in a future article. 16

APPENDICES Appendix A: Use Case: Investigating Hung Threads Hung threads can be a difficult issue to diagnose. Typically, there are two tools that we find to be effective in this area, the IBM Thread and Monitor Dump Analyzer and the in-flight data from the Cúram JMX statistics. The IBM Thread and Monitor Dump Analyzer is used to analyze thread dumps. Ideally, you should aim to take regular thread dumps during the period when the issue is visible, for instance 1 every 20 seconds for a 2 or 3 minute period. This allows you to monitor the activity in the threads over a period of time. Illustration 1: IBM Thread and Monitor Dump Analyzer The thread dump shows you the status of the various threads, whether they are running, blocked or parked, and so on, and if a thread is blocked by another thread, it shows you the blocking thread and provides a full stack trace for each thread. When looking for the threads that are associated with a Cúram application, running on WebSphere Application Server, you should focus on the 17

SIBJMSRATHREADPOOL threads and the WEBCONTAINER threads. SIBJMSRATHREADPOOL contains threads that are used for asynchronous transactions like deferred processes and workflows. The WEBCONTAINER thread pool contains the threads that are used for the online transactions. The in-flight data from the Cúram JMX statistics shows a similar perspective but provide information that is not available in the Thread Dumps. The in-flight data from the Cúram JMX statistics is a snapshot of the transactions that are running inside the application at the time the JMX statistics were taken. They show what URLS, BPO Methods, and SQL were being executed and for how long they have been running. Ideally, you should aim to take several snapshots over a period of time, similar to the Thread Dumps. For maximum effectiveness, you should coordinate the capture of the Cúram JMX statistics with the thread dumps as this allows you correlate data and activities between the two and build a complete picture of what activity is occurring in the application. In the illustrations below, I have shown what the in-flight data looks like and illustrated how the data can be used to map the different parts of a transaction together to help you narrow down the problem area. Illustration 2: InFlightURLStats from Cúram JMX statistics Illustration 2 shows the InFlightURLStats, these are the URLs, or servlets, that were trigger by the users and were still in progress when the JMX statistics were captured. Two URLs are present in this test case, the first is the URL that was used to capture the JMX statistics and the second is the URL that was triggered by a user to display a screen with a long list of records. Illustration 3: InFlightBPOMethodStats from Cúram JMX statistics 18

Illustration 3 shows the InFlightBPOMethodStats, these are the server-side transactions that were being executed when the JMX statistics were captured. In this test case, there is only one server transaction running. Illustration 4: InFlightSQLStats from Cúram JMX statistics Illustration 4 shows the InFlightSQLStats, these are the SQL statements that were being executed when the JMX statistics were captured. In this test case, there is only one SQL statement being executed. Each of the targets being executed has a unique identifier and can also have a parent unique identifier. By using the unique identifiers, we can map how a transaction performs across the layers in the application. Illustration 5 shows how the two transactions in our test case can be mapped. Focusing on the Test_largeListPage.do in the URL statistics we can see that the target has been executing for 250000ms. By taking its UUID and mapping it to a parent UUID in the InFlightBPOMethodStats we can see what server code was called by Test_largeListPage.do. We can also see how long that server code has been running for, in this instance it is 249900ms. Similarly, we can take the UUID of the BPO method target, curam.test.listscreen.readlonglist, and map it to a parent UUID in the InFlightSQLStats. In this test case, it shows us that curam.test.listscreen.readlonglist triggered an execution of the SQL statement, SELECT * FROM VERYLARGETABLE, and that SQL statement has been running for 249700ms. Illustration 5: Mapping of in-flight JMX data 19

By getting a snapshot of the transaction at the different layers, URL, BPO and SQL, and also having the execution times that we can determine in what layer the issue is arising in and narrow down the problem space. In this instance, the user TESTER1, has reported that is taking a long time to display the screen Test_largeListPage.do. From looking at the in-flight data, we can see that the URL target has been running for 250000ms, the BPO method that it triggered has been running for 249900ms, and the SQL statement that was triggered from the BPO method has been running for 249700ms. Overall, we can conclude that out of the 250000ms it was taking to return a response, 249700ms was spent executing a single SQL statement from the application. This narrows the problem space significantly. The investigation can now focus on what is happening between the data access layer in the application and the database and what is causing such a large execution time for a single SQL statement. The in-flight data from the Cúram JMX statistics offers a real insight into what is happening inside the application and are an invaluable tool when investigating issues like hung threads. Together with thread dumps they can be used to narrow down the problem space and focus investigations in the right areas of the system or application. 20

Appendix B: JMX statistics properties Name Value Description Authorized User for Statistics Download EJB Container MBeans List sysadmin curam.util.internal.mbean. CuramServerCoreStats, curam.util.cache.internal. mbean.curamcachestats, curam.util.configuration.i nternal.mbean.curamconfigu rationstats, curam.util.jmx.internal.ex tpoints.mbean.curamextensi onpointsstats, curam.util.webservices.int ernal.mbean.curamwebservic esoutboundstats The username of the user who is allowed to download the JMX statistics. The list of MBeans configured in the EJB container. Enable End-User Statistics FALSE Whether or not end user statistics collection is enabled. Enable In-Flight Transaction Statistics TRUE Whether or not statistics about inflight transactions are collected. Enable JMX Monitoring TRUE Whether JMX monitoring is enabled or not in the application. Enable SQL Statement Statistics TRUE Whether or not SQL statement statistics collection is enabled. Enable Statistics Download TRUE Whether or not the download of JMX statistics is allowed. Enable Transaction Tracing FALSE Whether transaction tracing is enabled or not in the application. End-User Statistics - Enable Browser Display End-User Statistics - Upload Delay Time End-User Statistics - User Filter Transaction Tracing - Maximum Recorded Threads FALSE Whether or not the end user statistics are displayed in the browser. If true, the statistics for the current page are displayed in the top left corner of the page. 7 The delay in seconds between the page reporting being loaded and the moment the statistics are uploaded. Regular expression that selects users for which end user statistics are collected. 5 The maximum number of threads for which transaction tracing data is collected. Note that at any one moment there could be more than this number of threads in the transaction tracing data but a significant amount of entries will only be preserved for this number 21

Name Value Description Transaction Tracing - Maximum Recorded Threads - Checking Interval Transaction Tracing - Maximum Recorded Threads - Maximum Thread Idle Time Transaction Tracing - URL Filter User Statistics Filter Web Container MBeans List of threads. 10 The period of time, in seconds, between checks to ensure that only the number of threads specified in curam.jmx.transaction_tracing_ma x_recorded_threads are preserved in the transaction tracing data. 600 The maximum amount of time, in seconds, a thread is allowed to be idle before its transaction tracing data can be cleared. curam.omega3.mbean.curamcl ientcorestats, curam.util.jmx.internal.we b.enduser.mbean.curamendus erstats Regular expression to identify URLs for which transaction tracing data is collected. Regular expression to identify users for which individual statistics are collected. The list of MBeans configured in the WEB container. 22

Appendix C: app_batchlauncher.sample.xml By default the app_runbatch.sample.xml file will generate JMX timer statistics and outbound web service statistics. The JMX timer statistics are generated every 60 seconds to a folder, C:\JMXStatsFor<batchtype>. Where: <batchtype> is the value of the batch.type property you pass to the batch invocation to differentiate a chunker and various stream invocations; for example: -Dbatch.type=Chunker. These Ant properties are available for controlling JMX timer statistics: To turn off JMX timer statistics pass -Djmx.timer.enabled=false on the command line. To turn off outbound web service statistics pass -Djmx.ws.outbound.enabled=false on the command line. To change the frequency of statistics collection, pass the jmx.timer.ms property on the command line with a value in milliseconds. For instance, to set the timer to 5 minutes use: -Djmx.timer.ms=300000. To change the location of statistics generation, pass the jmx.timer.dir property on the command line. app_batchlauncher.sample.xml <?xml version="1.0" encoding="utf-8"?>   <project name="app_batchlauncher" default="main">    <import file="./app_properties.xml" />  <import file="./app_utilities.xml" />    <target name="main" description="start the Batch Launcher"> 23

<antcall target="dispmsg" inheritall="false"> <param name="prm.msg" value="starting batchlauncher"/> </antcall>  <tstamp><format property="tim.start.sort" pattern="yyyymmddhhmmss"/></tstamp>  <mkdir dir="${dir.bld.log}"/> <record name="${dir.bld.log}/batchlauncher${tim.start.sort}.log" loglevel="info" />  <condition property="java.extra.jvmargs" value="-xnoargsconversion" > <os family="z/os"/> </condition> <property name="java.extra.jvmargs" value="-dfake.property=1"/> <property name="batch.program" value=" "/> <property name="batch.parameters" value=" "/> <property name="batch.username" value=" "/> <property name="disable.database.callback" value=" "/> <property name="batch.base.dir" value="${dir.bld.log}"/> <property name="curam.disable.dynamic.properties" value="false"/> <property name="force.disable.in.standalone" value=" "/> <tstamp> <format property="time.start" pattern="mmddyyyy-kkmmsss"/> </tstamp> --> -->   <property name="jmx.timer.enabled" value="true"/>  <property name="jmx.timer.dir.prefix" value="c:/jmxstats"/> <property name="jmx.timer.dir.suffix" value="/${batch.type}-$ {time.start}"/> <property name="jmx.timer.dir" value="${jmx.timer.dir.prefix}$ {jmx.timer.dir.suffix}"/>  <property name="jmx.timer.ms" value="60000"/>  <property name="jmx.ws.outbound.enabled" value="true"/> <mkdir dir="${jmx.timer.dir}"/>  <property name="gc.dir.prefix" value="c:/gc"/> <property name="gc.dir.suffix" value="/${batch.type}-${time.start}"/> <property name="gc.dir" value="${gc.dir.prefix}$ {gc.dir.suffix}"/> <property name="gc.file" value="gc.log"/>  <property name="gc.option" value="-xverbosegclog:" /> <property name="gc.log.arg" value="${gc.option}${gc.dir}/${gc.file}" /> 24

<property name="gc.arg" value="-xgcpolicy:gencon" /> <mkdir dir="${gc.dir}"/>  <java dir="${batch.base.dir}" classname="curam.util.impl.batchlauncher" classpathref="j.cp" fork="${java.fork}" failonerror="${java.failonerror}" maxmemory="${java.maxmemory}" taskname="batchlauncher" > <sysproperty key="curam.disable.dynamic.properties" value="$ {curam.disable.dynamic.properties}"/> <jvmarg line="${java.jvmargs}" /> <jvmarg value="-xms${java.maxmemory}" /> <jvmarg line="${java.extra.jvmargs}" /> <jvmarg value="-dconsole.encoding=${java.console.encoding}" /> <jvmarg value="-dfile.encoding=utf-8" /> <jvmarg value="-xbootclasspath/p:${jars.allxml}" />  <jvmarg line="${gc.arg}" />  <jvmarg line="${gc.log.arg}" />  <jvmarg value="-dcuram.jmx.output_statistics_timer_enabled=$ {jmx.timer.enabled}" /> <jvmarg value="-dcuram.jmx.output_statistics_timer_folder=${jmx.timer.dir}" /> <jvmarg value="-dcuram.jmx.output_statistics_timer_period=$ {jmx.timer.ms}" /> <jvmarg value="-dcuram.jmx.ws_outbound_statistics_enabled=$ {jmx.ws.outbound.enabled}" /> <arg value="${batch.program}" /> <arg value="${batch.parameters}" /> <arg value="${batch.username}" /> <arg value="${disable.database.callback}" /> <arg value="${force.disable.in.standalone}" /> </java> <antcall target="dispmsg" inheritall="false"> <param name="prm.msg" value="ending batchlauncher"/> </antcall> </target> </project> 25

REFERENCES Cúram Tech Notes relating to JMX: Collecting Data for Cúram Social Program Management issues: Collecting Data for IBM Cúram Social Program Management performance issues using JMX statistics Collecting Data for IBM Cúram Social Program Management batch performance issue For details on how to customize JMX statistics, see the Cúram JMX Configuration Guide. For details on how to add additional counters to Cúram JMX statistics, see the Cúram JMX Developer Guide. 26

Copyright IBM Corporation 2015 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation 4205 South Miami Boulevard Research Triangle Park, NC 27709 U.S.A. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. If you are viewing this information softcopy, the photographs and color illustrations may not appear. 27

Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at http://www.ibm.com/legal/copytrade.shtml. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Other company, product, or service names may be trademarks or service marks of others. 28