An article on collecting IBM InfoSphere CDC Subscription health matrices in a Java program using the Monitoring API

Transcription

1 An article on collecting IBM InfoSphere CDC Subscription health matrices in a Java program using the Monitoring API Aniket Kadam(anikadam@in.ibm.com)

2 Prerequisites Before going through the article reader should have the basic knowledge of How to configure the instance (datastore) of the InfoSphere CDC for source and target databases. How to add the datastores in the Management Console How to create a Subscription in Management Console and start data replication Java programming language Important note: The Monitoring API described herein requires the api.jar available in Management Console IF3 or later. Java code calling these monitoring classes and methods will not compile or run with earlier versions. Overview InfoSphere Change Data Capture (CDC) Management Console provides a view called Monitoring View. In this view the user can monitor the subscription activities and view the event log which gets populated with the messages depending on the subscription activities. Inside Monitoring View there is a tab called Performance. This view provides a more advanced view of what is happening on the selected subscription. Following are the figures showing different views seen in the Management Console. 1)Subscription and table mapping view 2)Performance View

3 3)The list of available matrices in Performance View 4)Selected Matrices for capturing 5)Captured data after DML operations on source table While the performance view gives the brilliant visual display of various matrices attributed to the subscription in the form of graphs,a customer might be more interested in collecting these matrices in a Java program. As the performance view has a limited scope for a single subscription,an environment where there are numerous subscriptions to track,calls for a more efficient approach of making these tasks automated. One of the advantages it has is that it removes all the manual intervention required to select the subscription and the matrices we want to capture. Once captured the user can export them to a file in a desirable format. Moreover,the user can put some logic depending on the business requirement in his Java program which makes efficient use of these matrices. InfoSphere CDC Management Console provides an enriched API to make this provision. This article introduces the user to some of the newly added classes which have extended set of utility methods that help the

4 user accomplish the task of developing such a Java program. These are collectively called the Monitoring API. At the end of the article a template Java program is provided which extracts some of the performance matrices for a subscription. Monitoring API Busy table statistics Performance statistics Refresh statistics Summary statistics This introduces four classes depending on what kind of statistics user is looking for. brief Each class provides a list of utility methods. Some of them which are important are described here in Busy table statistics Class BusyTablesStatisticData Methods -getbusytables() Returns a set of busy tables in an ArrayList,BusyTableStatisticData.BusyTable Performance statistics -getbytes(string tableowner, String tablename) Returns the number of bytes processed for a table -gettimestamp() Returns the timestamp for the statistic snapshot of the busy tables. Class PerformanceStatisticData There are numerous performance statistics provided for a subscription. We need to initialize performance statistics object by adding the required statistics to the dataset. An interface StatisticConstants maintains a list of statistic definitions each assigned a byte/int value. By referring to this interface it becomes easy to locate all the statistics at one place and user can conveniently add them to the data set. Once these definitions are added to dataset the user can start collecting the statistics by calling different methods. Performance statistics are collected over a period of time. Methods adddefinition(statisticdefinition definition) Adds a statistic definition to the data set. getaverage(int id) Returns the average value for the entire data set for the specified statistic definition id. getcumulativevalue(int id) Returns value last received from the agent for the statistics with the specified definition id. getmaximum(int id) Returns the maximum value for an id

5 getminimum(int id) Returns the minimum value for an id. getvalus(int id) Returns the value from the last row for statistics with the specified definition id. Refresh statistics Class RefreshStatisticData This data structure stores statistic data for refresh activity on a subscription. The current snapshot of the refresh activity is available through the Table object which contains the name of the table, start/end time, and source/apply record counts, as available. Methods gettables() Returns the set of tables under refresh in an ArrayList RefreshStatisticData.Table gettimestamp() Returns the timestamp for the statistic snapshot of the refresh operation. iscomplete() returns true if refresh operation for all the tables under subscription is complete Class RefreshStatisticData.Table Provides certain methods which are table specific. They help to track the status of a table that is under a refresh operation. Methods Summary statistics Class getapplyrecords() Returns the number of records applied on the target for completed tables. getduration() Returns table's refreshing time if refresh completed. getstarttimestamp() Returns the source engine time a table started refreshing for completed or current tables. getendtimestamp() Returns the source engine time a table completed refreshing for completed tables. iscompleted() Returns true if the table is completed. iscurrent() Returns true if the table is currently refreshing. ispending() Returns true if the table has not started refreshing.

6 SummaryStatisticData This data structure stores subscription summary statistics. These statistics mainly include replication state and status of the subscription. User can get problem and warning thresholds set for the subscription for the particular latency values. When the replication method is mirror scheduled end,the user can request for the scheduled end timestamp. Methods getlatencyproblemthreshold() Returns the latency problem threshold, in seconds getlatencyvalue() Returns the latency value. getlatencywarningthreshold() Returns the latency warning threshold, in seconds. getscheduledendtimestamp() Returns the timestamp for scheduled end. getstate() Returns the replication state of the subscription. getstatus() Returns the replication status of the subscription. Template Java Program This is a sample Java program to demonstrate how to initialize PerformanceStatisticData object and extract the required statistics. It also lists all the available statistics associated with the datastore depending on whether it is a source or a target. The sample assumes that a subscription is created with the name SUB1 in the Management Console and the data replication is started on it. The program is run when there are DML operations occurring on the source table. The Performance statistics seen in the figure no.4 are selected and added in the dataset and then the data collection is enabled. Following are the parameters which are used in the program to complete the entire operation. Source datastore CDC_DB2 Target datastore CDC_ORACLE Subscription Name SUB1 Access Server localhost Access Server user name admin Access Server Password admin12 / Licensed Materials - Property of IBM IBM InfoSphere Change Data Capture 5724-U70 (c) Copyright IBM Corp All rights reserved. The following sample of source code ("Sample") is owned by International Business Machines Corporation or one of its subsidiaries ("IBM") and is copyrighted and licensed, not sold. You may use, copy, modify, and distribute the Sample in any form without payment to IBM. The Sample code is provided to you on an "AS IS" basis, without warranty of any kind. IBM HEREBY EXPRESSLY DISCLAIMS ALL WARRANTIES, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow for the exclusion or limitation of implied warranties, so the above

7 limitations or exclusions may not apply to you. IBM shall not be liable for any damages you suffer as a result of using, copying, modifying or distributing the Sample, even if IBM has been advised of the possibility of such damages. / import java.util.list; import com.datamirror.ea.api.apiexception; import com.datamirror.ea.api.datasource; import com.datamirror.ea.api.defaultcontext; import com.datamirror.ea.api.performancestatisticdata; import com.datamirror.ea.api.replicationrole; import com.datamirror.ea.api.statisticcategory; import com.datamirror.ea.api.statisticconstants; import com.datamirror.ea.api.statisticdefinition; import com.datamirror.ea.api.statisticorigin; import com.datamirror.ea.api.toolkit; import com.datamirror.ea.api.publisher.publisher; import com.datamirror.ea.api.publisher.subscription; import com.datamirror.ea.api.subscriber.publication; import com.datamirror.ea.api.subscriber.subscriber; / Connects to the source and target datastores, prints out the supported performance statistics, and requests a sample of those statistics. / public class MonitorPerformanceSample public static String ACCESS_SERVER_HOST_TOKEN = "-host"; public static String ACCESS_SERVER_PORT_TOKEN = "-port"; public static String ACCESS_SERVER_USER_TOKEN = "-user"; public static String ACCESS_SERVER_PASSWORD_TOKEN = "-password"; public static String SOURCE_DATASTORE_TOKEN = "-source"; public static String TARGET_DATASTORE_TOKEN = "-target"; public static String SUBSCRIPTION_NAME_TOKEN = "-subscriptionname"; public static String PUBLICATION_NAME_TOKEN = "-publicationname"; private DataSource accessserver; / Runs the host the host name for access port the port number of access server in the host user the login user password the login sourcedatastore the name of the source subscriptionname the name of the subscription on the targetdatastore the name of the target publicationname the name of the publication on the ApiException if an error occurred. / public void run( String host, String port, String user, String password, String sourcedatastore, String subscriptionname, String targetdatastore, String publicationname) throws ApiException try connectaccessserver(host, port, user, password); if (sourcedatastore!= null && sourcedatastore.length() > 0) ReplicationRole datastore = null; PerformanceStatisticData statistics = null; try datastore = connectdatastore(sourcedatastore, true); statistics = initializedefinitions(datastore, true); Subscription subscription = ((Publisher) datastore).getsubscription(subscriptionname); if (subscription!= null) subscription.requestperformancestatistics(statistics); printcurrentresults(statistics); finally disconnectdatastore(datastore); if (targetdatastore!= null && targetdatastore.length() > 0) ReplicationRole datastore = null; PerformanceStatisticData statistics;

8 try datastore = connectdatastore(targetdatastore, false); statistics = initializedefinitions(datastore, false); Publication publication = ((Subscriber) datastore).getpublication(publicationname); if (publication!= null) publication.requestperformancestatistics(statistics); printcurrentresults(statistics); finally disconnectdatastore(datastore); finally disconnectaccessserver(); / Prints the current statistics the statistics. / public void printcurrentresults(performancestatisticdata statistics) int id; String pad = " "; for (StatisticDefinition definition : statistics.getdefinitions()) System.out.print((definition.getName() + pad).substring(0, 30) + " "); for (int i = 0, cnt = statistics.getdefinitions().size(); i < cnt; i++) System.out.print(" "); for (StatisticDefinition definition : statistics.getdefinitions()) id = definition.getid(); System.out.print((statistics.getCumulativeValue(id) + pad).substring(0, 30) + " "); / Collects the statistics for the datastorename the name of the name the subscription or publication source true to request source statistics, false to request target ApiException if an error occurred. / public PerformanceStatisticData initializedefinitions(replicationrole datastore, boolean source) throws ApiException PerformanceStatisticData statistics = new PerformanceStatisticData(); // Get the supported performance metrics. StatisticCategory[] categories = datastore.getstatisticdefinitions(); for (int i = 0; i < categories.length; i++) // Only include custom categories (>CATEGORY_CUSTOM). if (categories[i].getid() < StatisticConstants.CATEGORY_CUSTOM) continue; boolean printcategory = true; int id; List<StatisticDefinition> definitions = categories[i].getdefinitions(); if (definitions!= null) for (StatisticDefinition definition : definitions) if (source && definition.getorigin()!= StatisticOrigin.TARGET)

9 if (printcategory) System.out.println("Category [" + categories[i].getid() + "] " + categories[i].getname()); printcategory = false; System.out.println(" Definition [" + definition.getid() + "] " + definition.getname()); id = definition.getid(); switch (id) case StatisticConstants.SOURCE_PREFILTER_INSERT_OPERATIONS: case StatisticConstants.SOURCE_PREFILTER_UPDATE_OPERATIONS: case StatisticConstants.SOURCE_PREFILTER_DELETE_OPERATIONS: statistics.adddefinition(definition); break; if (!source && definition.getorigin()!= StatisticOrigin.SOURCE) if (printcategory) System.out.println("Category [" + categories[i].getid() + "] " + categories[i].getname()); printcategory = false; System.out.println(" Definition [" + definition.getid() + "] " + definition.getname()); id = definition.getid(); switch (id) case StatisticConstants.TARGET_APPLY_INSERT_OPERATIONS: case StatisticConstants.TARGET_APPLY_UPDATE_OPERATIONS: case StatisticConstants.TARGET_APPLY_DELETE_OPERATIONS: statistics.adddefinition(definition); break; return statistics; / Connects to a replication datastorename the name of the source true to request source statistics, false to request target ApiException if an error occurred. / public ReplicationRole connectdatastore(string datastorename, boolean source) throws ApiException if (accessserver == null!accessserver.isopen()) throw new ApiException("Connection to Access Server is not established"); ReplicationRole datastore; if (source) datastore = accessserver.getpublisher(datastorename); else datastore = accessserver.getsubscriber(datastorename); if (datastore == null) throw new ApiException("Failed to locate a datastore: " + datastorename); if (!datastore.isconnected()) try System.out.println("Connecting to " + datastorename + "..."); datastore.connect(); System.out.println("Connected."); catch (ApiException e) throw new ApiException("Failed to connect to datastore " + datastorename + ". " + e.getmessage());

10 return datastore; / Disconnects from the datastore the ApiException If an error occurred / public void disconnectdatastore(replicationrole datastore) throws ApiException boolean disconnecting = false; if (datastore!= null && datastore.isconnected()) System.out.println("Disconnecting from " + datastore.getname() + "..."); disconnecting = true; datastore.disconnect(); if (datastore!= null && datastore.isconnected()) throw new ApiException("Failed to disconnect from a datastore " + datastore.getname()); else if (disconnecting) System.out.println("Disconnected."); / Connects to access host the host name for access port the port number of access server in the host user the login user password the login ApiException if an error occurred. / public void connectaccessserver(string host, String port, String user, String password) throws ApiException try System.out.println("Connecting to Access Server..."); accessserver = Toolkit.getDefaultToolkit().createDataSource(); DefaultContext eaaccesscontext = new DefaultContext(); eaaccesscontext.setstring(datasource.user, user); eaaccesscontext.setstring(datasource.password, password); eaaccesscontext.setstring(datasource.hostname, host); eaaccesscontext.setint(datasource.port, Integer.parseInt(port)); accessserver.connect(eaaccesscontext); System.out.println("Connected."); catch (ApiException e) throw new ApiException("Failed to connect to " + host + "@" + port + " as " + user); / Disconnects from access ApiException if an error occurred. / public void disconnectaccessserver() throws ApiException boolean disconnecting = false; if (accessserver!= null && accessserver.isopen()) System.out.println("Disconnecting from Access Server..."); disconnecting = true; accessserver.close(); if (accessserver!= null && accessserver.isopen()) throw new ApiException("Failed to disconnect from AccessServer"); else if (disconnecting) System.out.println("Disconnected."); / Returns an example of arguments for the the help string.

11 / public static String gethelp() String help = "java -classpath \"pathtosample and api.jar\" " + "MonitorPerformanceSample\n" + " [-host accessserverhost -port accessserverport]\n" + " -user accessserveruser -password accessserverpassword\n" + " [-source sourcedatastorename -subscriptionname subscriptionname]\n" + " [-target targetdatastorename -publicationname publicationname]\n\n" + "If -host or -port are not specified, \"localhost\" and \"10101\" " + "will be used.\n" + "-source/-subscriptionname and/or -target/-publication are required."; return help; / Returns the named argument's args the command line name the name of the command line defaultvalue the default value. If null is specified, then an exception is thrown if the parameter does not value for the IllegalArgumentException if the argument is missing. / private String getparameter(string[] args, String name, String defaultvalue) throws IllegalArgumentException String parameter = null; for (int i = 0; i < args.length - 1; i++) if (args[i].equalsignorecase(name)) parameter = args[i + 1]; break; if (parameter == null) if (defaultvalue == null) throw new IllegalArgumentException("Parameter " + name + " is missing."); else parameter = defaultvalue; return parameter; public static void main(string[] args) try MonitorPerformanceSample sample = new MonitorPerformanceSample(); // Access server connection information. String host = sample.getparameter(args, ACCESS_SERVER_HOST_TOKEN, "localhost"); String port = sample.getparameter(args, ACCESS_SERVER_PORT_TOKEN, "10101"); String user = sample.getparameter(args, ACCESS_SERVER_USER_TOKEN, "admin"); String password = sample.getparameter(args, ACCESS_SERVER_PASSWORD_TOKEN, "admin12"); // Source information. If the datastore is specified, then the subscription // name must be as well. String sourcedatastore = null; String subscriptionname = null; sourcedatastore = sample.getparameter(args, SOURCE_DATASTORE_TOKEN, "CDC_DB2"); if (sourcedatastore!= null && sourcedatastore.length() > 0) subscriptionname = sample.getparameter(args, SUBSCRIPTION_NAME_TOKEN, "SUB1"); // Target information. String targetdatastore = null; String publicationname = null; targetdatastore = sample.getparameter(args, TARGET_DATASTORE_TOKEN, "CDC_ORACLE"); if (targetdatastore!= null && targetdatastore.length() > 0) publicationname = sample.getparameter(args, PUBLICATION_NAME_TOKEN, "SUB1"); if (sourcedatastore.length() == 0 && targetdatastore.length() == 0) throw new IllegalArgumentException("Parameter "

12 + SOURCE_DATASTORE_TOKEN + " and/or " + TARGET_DATASTORE_TOKEN + " is required."); sample.run( host, port, user, password, sourcedatastore, subscriptionname, targetdatastore, publicationname); catch (IllegalArgumentException e) System.out.println(MonitorPerformanceSample.getHelp()); catch (Exception e) System.out.println(e.getMessage()); e.printstacktrace(); Output Connecting to Access Server... Connected. Connecting to CDC_DB2... Connected. Category [ ] Datastore Definition [1001] Time check missed count Definition [1002] Free memory - bytes Definition [1003] Maximum memory - bytes Definition [1004] Total memory - bytes Definition [1005] Garbage collections - count Definition [1006] Garbage collection CPU time - milliseconds Definition [1007] Global memory manager - bytes Definition [2401] Network errors - count Category [9001] Database Workload Definition [1103] Total transactions - count Definition [1301] In scope transactions - count Definition [1310] Transaction histogram (0 to 1/2 X) - bytes Definition [1311] Transaction histogram (1/2X to X) - bytes Definition [1312] Transaction histogram (X to 2X) bytes Definition [1313] Transaction histogram(2x to 4X) - bytes Definition [1314] Transaction histogram (4X to 8X) - bytes Definition [1315] Transaction histogram (8X and larger) - bytes Category [9002] Log Reader Definition [101] Source database bytes processed Definition [1101] Physical bytes read - bytes Definition [1104] Thread CPU - milliseconds/second Category [9003] Log Parser Definition [1201] Disk writes - bytes Definition [1202] Disk reads - bytes Definition [1203] Disk size - bytes Definition [1303] Thread CPU - milliseconds/second Category [9004] Single Scrape Definition [1401] Disk writes - bytes Definition [1402] Disk reads - bytes Definition [1403] Disk size - bytes Category [9005] Source Engine Definition [102] Source engine bytes processed Definition [104] Source pre-filter inserts Definition [105] Source pre-filter updates Definition [106] Source pre-filter deletes Definition [107] Source post-filter inserts Definition [108] Source post-filter updates Definition [109] Source post-filter deletes Definition [118] Latency value (seconds) Definition [1501] Rows evaluating derived columns - count Definition [1502] Rows calling source database - count Definition [1503] Rows calling user exits - count Definition [1504] Thread CPU - milliseconds/second Definition [1507] MBCS conversions - bytes Definition [1601] Source engine bytes processed Definition [1602] Source pre-filter inserts Definition [1603] Source pre-filter updates Definition [1604] Source pre-filter deletes Category [9006] Communications Definition [103] Source communications bytes processed Definition [2402] Source network latency (millisecond) Definition [2404] Source missing round trip response - count Definition [2406] Keep alive sent - count Definition [2408] Bytes sent - count Source pre-filter inserts Source pre-filter updates Source pre-filter deletes

13 Disconnecting from CDC_DB2... Disconnected. Connecting to CDC_ORACLE... Connected. Category [ ] Datastore Definition [1001] Time check missed count Definition [1002] Free memory - bytes Definition [1003] Maximum memory - bytes Definition [1004] Total memory - bytes Definition [1005] Garbage collections - count Definition [1006] Garbage collection CPU time - milliseconds Definition [1007] Global memory manager - bytes Definition [2401] Network errors - count Category [9006] Communications Definition [201] Target communications bytes processed Definition [2403] Target network latency (millisecond) Definition [2405] Target missing round trip response - count Definition [2407] Keep alive received - count Definition [2409] Bytes received - count Category [9008] Target Engine Definition [202] Target engine bytes processed Definition [204] Target "source" inserts Definition [205] Target "source" updates Definition [206] Target "source" deletes Definition [2101] Thread CPU - milliseconds/second Definition [2103] Rows evaluating expressions - count Definition [2104] Rows calling target DB - count Definition [2105] Rows calling user exits - count Definition [2106] MBCS conversions - bytes Category [9009] Target Apply Definition [207] Target apply inserts Definition [208] Target apply updates Definition [209] Target apply deletes Definition [216] Latency value (seconds) Definition [1701] Target database bytes processed Definition [1702] Target apply inserts - count Definition [1703] Target apply updates - count Definition [1704] Target apply deletes - count Definition [1705] Target apply slow database operations - count Definition [2301] Thread CPU - milliseconds/second Definition [2303] CDR conflicts - count Definition [2304] Adaptive Apply changes - count Definition [2305] Apply commits - count Definition [2306] Average array size - count Definition [2307] Data errors ignored - count Definition [2308] Slow database operations - count Target apply inserts Target apply updates Target apply deletes Disconnecting from CDC_ORACLE... Disconnected. Disconnecting from Access Server... Disconnected. The statistics collected are the same as seen in the figure no.5 Resources Management Console Help and Admin Guide Management Console API resources