IBM FileNet Content Manager 5.2 April 2013 IBM SWG Industry Solutions/ECM IBM FileNet Content Manager 5.2 Asynchronous Event Processing Performance Tuning Copyright IBM Corporation 2013 Enterprise Content Management www.ibm.com No part of this document may be reproduced in any form by any means without prior written authorization of IBM. This document is provided as is without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranty of merchantability or fitness for a particular purpose. This document is intended for informational purposes only. It could include technical inaccuracies or typographical errors. The information herein and any conclusions drawn from it are subject to change without notice. Many factors have contributed to the results described herein and IBM does not guarantee comparable results. Performance numbers will vary greatly depending upon system configuration. All data in this document pertains only to the specific test configuration and specific releases of the software described.
IBM FileNet Content Manager 5.2 Page 2 CONTENTS Introduction...3 Event Processing Overview...3 Performance Considerations...4 Judiciously increase worker threads...4 Ensure optimal batch size...4 Optimize LOB retrieval...4 Maximizing QUEUEITEM table concurrency...5 Consult the IBM FileNet P8 v5.2 InfoCenter performance tuning topics...5 Conclusion...6 References...7 Author Information...8
IBM FileNet Content Manager 5.2 Page 3 Introduction IBM FileNet Content Manger provides the ability to execute user-implemented actions based on system or custom events. Subscriptions are used to associate an event action with a Content Platform Engine object, such as a document class. By creating subscriptions to certain events, such as document creation, an event handler is fired and the event action is performed. Event actions are typically custom Java classes and when a subscribed event occurs, the action handler is fired and the custom code is executed. The IBM FileNet Content Manager 5.2 release includes multiple enhancements, such as an optimized event processing infrastructure as well as increased configuration flexibility. In addition to discussing the event processing enhancements, this document will outline general performance considerations and best practices for optimizing event processing throughput. Event Processing Overview Subscriptions can be defined to execute an event action synchronously or asynchronously. A synchronous event action is executed in the same transaction as the activity that triggered the event action, such as document creation. If the event action fails for any reason, the entire transaction, including the document creation is rolled back. It is important to note that the subscription processor will remain blocked from processing further events until the event action completes. Additionally, a user-implemented action called a change preprocessor is available. Change preprocessors run synchronously and are preferable to synchronous events in certain situations. An asynchronous event action is not executed in the same transaction as the originating activity. The event action is placed in a queue and processed by a separate execution thread. It is important to understand business requirements when deciding between using event actions and change preprocessors as there could be a considerable performance impact between the two types. This document will focus on the performance tuning options available when using asynchronous events. For more information regarding events, event action handlers, subscriptions and change preprocessors, please consult the IBM FileNet P8 Information Center.
IBM FileNet Content Manager 5.2 Page 4 Performance Considerations It is important to protect critical system resources such as the backend database(s) from saturation by ensuring that the IBM FileNet Content Platform Engine (CPE) event processing mechanism is properly tuned. As with most performance tuning parameters, it is recommended to start with safe values and increase gradually while monitoring the system. The system default values are considered safe values that should not be changed unless evidence suggests tuning is necessary. Multiple performance considerations to maximize event processing rates are outlined in detail below. Note: It is critical to plan a structured performance benchmarking process with well defined performance baselines in order to clearly determine whether a performance tuning change has had the desired effect on the system. Judiciously increase worker threads It is important to keep in mind that the number of worker threads configured applies to each IBM FileNet CPE instance. Therefore, the larger the deployment of the IBM FileNet CPE, the smaller the incremental increase in worker threads must be in order to prevent unexpected database impact. As the number of worker threads is increased, it is critical to monitor the impact of fetching work from the database at an increased rate. Excessive QUEUEITEM table activity may require an increase in the batch size to further increase throughput. For IBM FileNet Content Manager installations of less than 6 CPE instances, an increment of no larger than 5 worker threads is recommended. For larger installations, an increment of 1 worker thread is recommended until optimal throughput is achieved. Ensure optimal batch size The default batch size of 100 is sufficient for most applications. If high volume event processing is required, increasing the batch size may be necessary to ensure that event processing worker threads are not starved for work. Take care to avoid increasing this parameter too aggressively. Processing of the QUEUEITEM table should remain steady, provided there is a sufficient backlog or a high rate of incoming events. Inconsistent processing rates may be an indication that the batch size is not sufficient to keep all available worker threads busy and increasing the batch size should be considered. It is also possible that I/O may be limiting the processing rate. See the section below regarding optimizing LOB retrieval for suggestions to optimize the QUEUEITEM table I/O for event processing. Optimize LOB retrieval A common bottleneck that prevents maximum event processing throughput is contention on the LOB column SERIALIZED_DATA of the QUEUEITEM table. The event action handler Java object is serialized and stored in this column, defined as a BLOB. The retrieval of this BLOB is necessary to instantiate the event action handler object for processing. Most databases have limitations as to the amount of data that can be stored inline. For example, IBM DB2 s limitation is just under the total database page size (32KB for IBM FileNet CPE). Consult your database vendor s documentation for more details around limitations for inline BLOBs. The size of this BLOB is directly associated with the size of the serialized Java object. If the event action handler document class has a significant amount of meta-data, this object can easily become too
IBM FileNet Content Manager 5.2 Page 5 large to store inline. Storing a LOB out of column makes the retrieval more expensive for the database due to increased I/O. It is possible to leverage database features to minimize the impact of LOBs, such as LOB column caching. However, depending upon the size of the QUEUEITEM table, caching may actually negatively impact performance since the cached record will be expunged from the cache by the time the event is processed. Another possibility is leveraging database compression to reduce the size of the LOB allowing for inline storage. IBM FileNet CPE only supports compression when using DB2. For other database vendors, please consult with IBM support. Consult your database vendor s documentation for instructions regarding how to determine the average size of the LOB column and how to estimate potential space savings from compression. Databases typically leave a portion of the data block unused. This is done to accommodate for future updates to the data in order to minimize fragmentation. The LOB column will never be updated once the event is in queue. Therefore, it is possible to further minimize I/O by making more efficient use of the data blocks and not leaving unused space unnecessarily. This can be achieved by setting a database table column attribute. For example, with DB2 set the PCTFREE attribute to 0 rather than the default of 20. Maximizing QUEUEITEM table concurrency During high volume event processing, the QUEUEITEM table can be subject to many concurrent requests. It is important to ensure parameters that control concurrent access to table resources are tuned appropriately. In many cases, ensuring I/O requests are as efficient as possible will ensure multiple requests for data from the QUEUEITEM table do not block. Improving LOB retrieval efficiency as described in the previous section in addition to using high performance SANs are ways to ensure requests can be satisfied as quickly as possible. Database tuning parameters exist that may also help improve table concurrency, such as INITRANS. Consult with your DBA to ensure maximum table concurrency for the QUEUEITEM table is configured optimally. Consult the IBM FileNet P8 v5.2 InfoCenter performance tuning topics It is always suggested to refer to the P8 performance tuning topics in the IBM FileNet P8 5.2 Information Center early in the planning and testing phase of a new deployment rather than after the environment is in production. Please see the References section for direct links.
IBM FileNet Content Manager 5.2 Page 6 Conclusion The IBM FileNet Content Platform Engine s event processing infrastructure is capable of processing millions of events each hour. There is virtually no limit on the number of worker threads that can be concurrently executing and as the worker thread limit is increased, the batch size that determines how much work to obtain from the database can be adjusted as well to minimize thread starvation. As the event processing rate is increased, experience has shown the rate at which data can be retrieved from the database is a common limiting factor of achieving maximum throughput. Efficient LOB retrieval, maximum QUEUEITEM table concurrency along with a balance of worker threads to batch size are all necessary to process events at the fastest rate possible. When faced with a system that is not meeting performance expectations, it is suggested to review the system architecture and ensure basic tuning is applied, such as adequate JVM heap space, an optimal garbage collection policy and sufficient thread pool sizes. Most importantly, system resources like CPU, memory and disk I/O capacity need to be properly sized for the intended workload. It is crucial to ensure sufficient hardware resources have been allocated to meet the intended workload and to work with your IT department to increase any resource capacity that remains saturated after performance tuning has been implemented.
IBM FileNet Content Manager 5.2 Page 7 References 1. IBM FileNet P8 Software: http://www.ibm.com/software/data/content-management/filenet-p8- platform 2. IBM FileNet P8 5.2 Information Center: http://publib.boulder.ibm.com/infocenter/p8docs/v5r2m0/index.jsp a. Administration Subscribing to events: http://pic.dhe.ibm.com/infocenter/p8docs/v5r2m0/topic/com.ibm.p8.ce.admin.tasks.doc/p8 pcc038.htm b. Java/.NET Developer s Guide Action Handlers Restrictions and Best Practices http://pic.dhe.ibm.com/infocenter/p8docs/v5r2m0/topic/com.ibm.p8.ce.dev.ce.doc/server_c oncepts.htm c. Java/.NET Developer s Guide Change Preprocessor Concepts http://pic.dhe.ibm.com/infocenter/p8docs/v5r2m0/topic/com.ibm.p8.ce.dev.ce.doc/changep reprocessor_concepts.htm d. Performance Tuning Topics: http://publib.boulder.ibm.com/infocenter/p8docs/v5r2m0/topic/com.ibm.p8.performance.d oc/p8ppt000.htm e. Monitoring IBM FileNet P8: http://publib.boulder.ibm.com/infocenter/p8docs/v5r2m0/topic/com.ibm.p8.sysmgr.admin. doc/overview_monitoring_p8.htm f. Troubleshooting the Content Platform Engine http://pic.dhe.ibm.com/infocenter/p8docs/v5r2m0/topic/com.ibm.p8.ce.trouble.doc/p8pct00 0.htm
IBM FileNet Content Manager 5.2 Page 8 Author Information Michael Bordash ECM Server System Test Engineer mailto:mbordash@us.ibm.com Contributors Matthew Vest ECM Server System Test & Performance Engineering Senior Manager Dave Royer ECM Performance Architect, Senior Software Engineer Special thanks to the following members of the ECM CE development team: Joseph Raby David Skinner Haibing Qiao Disclaimer The information in this publication is not intended as a substitution of the IBM FileNet product documentation provided by IBM. Please see http://www.ibm.com/software/data/content-management for more information about what publications are considered to be product documentation. References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product, program or service. Information in this publication was developed in conjunction with use of the equipment specified, and is limited in application to those specific hardware and software products and levels. The information contained in this publication was derived under specific operating and environmental conditions. While IBM has reviewed the information for accuracy under the given conditions, the results obtained in your operating environments may vary significantly. Accordingly, IBM does not provide any representations, assurances, guarantees, or warranties regarding performance. Any information about non-ibm ("vendor") products in this document has been supplied by the vendor and IBM assumes no responsibility for its accuracy or completeness. IBM, IBM FileNet Content Manager, DB2, WebSphere, AIX, Rational, and Tivoli are trademarks or registered trademarks of IBM Corporation in the United States, other countries, or both. Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a trademark of The Open Group. Windows is a registered trademark of Microsoft Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. Copyright IBM Corporation 2013 Produced in the United States of America All Rights Reserved The e-business logo, the eserver logo, IBM, the IBM logo, IBM Directory Server, DB2, FileNet, FileNet Content Manager and WebSphere are trademarks of International Business Machines Corporation in the United States, other countries or both. The following are trademarks of other companies: Solaris, Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. Windows and Windows 2008 Enterprise Edition are trademarks of Microsoft Corporation in the United States and/or other countries Oracle 9i and all Oracle-based trademarks and logos are trademarks of the Oracle Corporation in the United States, other countries or both. Other company, product and service names may be trademarks or service marks of others. INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. Information in this paper as to the availability of products was believed accurate as of the time of publication. IBM cannot guarantee that identified products will continue to be made available by their suppliers. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice.
IBM FileNet Content Manager 5.2 Page 9 Any references in this document to non-ibm web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY, USA 10504-1785
Copyright IBM Corporation 2013 IBM 3565 Harbor Boulevard Costa Mesa, CA 92626-1420 USA Printed in the USA 01-07 All Rights Reserved. IBM and the IBM logo are trademarks of IBM Corporation in the United States, other countries, or both. All other company or product names are registered trademarks or trademarks of their respective companies. The IBM home page on the Internet can be found at ibm.com