IBM SWG Enterprise Content Management

Transcription

1 IBM FileNet Content Manager 5.2 High Volume Scalability White Paper February 2014 IBM SWG Enterprise Content Management IBM FileNet Content Manager 5.2 High Volume Scalability Copyright IBM Corporation 2014 Enterprise Content Management This document is provided as is without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranty of merchantability or fitness for a particular purpose. This document is intended for informational purposes only. It could include technical inaccuracies or typographical errors. The information herein and any conclusions drawn from it are subject to change without notice. Many factors have contributed to the results described herein and IBM does not guarantee comparable results. Performance numbers will vary greatly depending upon system configuration. All data in this document pertains only to the specific test configuration and specific releases of the software described.

2 IBM FileNet Content Manager 5.2 High Volume Scalability Page 2 CONTENTS IBM FileNet Content Manager 5.2 High Volume Scalability...1 Introduction...3 Executive Summary Test environment description...4 Architectural Overview...4 Test Bed Population Population to 5 billion documents...6 Methodology...6 ECM Workload High volume transactional workloads...11 IBM FileNet Content Manager workload scalability...11 IBM Content Navigator workload stability Performance considerations and recommendations...13 Database storage...13 File Systems / File Stores...13 JVMs...13 Indexing...13 Covering index for sweeps...14 Consult the IBM FileNet CM Information Center performance tuning topics...14 Conclusion...15 References...16 Appendix...17 Detailed Software Information...17 Detailed Hardware Information...17 Detailed Tunings...17

3 IBM FileNet Content Manager 5.2 High Volume Scalability Page 3 Introduction IBM FileNet Content Manager (FileNet CM) is capable of managing enterprise volumes of business content. Many customers currently rely on FileNet CM to handle daily ECM workloads with repositories well into the hundreds of millions of documents, with several exceeding one billion documents in production. This paper demonstrates that FileNet CM, running on IBM POWER/AIX, DB2, GPFS and WebSphere Application Server, is capable of scaling to five billion documents and beyond while maintaining the industry leading performance that customers have come to expect. It also provides configuration best practices and performance guidance as the basis for confidence in achieving similar results in production systems. Executive Summary The IBM FileNet CM product has matured over the past 15 years of continuous improvement, becoming an industry leader in Enterprise Content Management. FileNet CM is in production at thousands of the world s largest companies, including many who are storing hundreds of millions of documents per object store, concurrently accessed by thousands of users. The stability and scalability of the FileNet CM platform has been well established, although concerns around the database and file storage layers when populations reach into billions of documents prompted further investigation. This paper reaffirms the ability of both the database and file storage to scale a FileNet CM object store into the billions of documents. IBM FileNet CM achieved unprecedented levels of data and workload scalability during this study. The FileNet CM repository successfully scaled to 5 billion document objects, consuming upwards of 120 terabytes of IBM XIV storage. Various operations were performed during pre-defined population checkpoints, demonstrating the scalability of the FileNet CM object store. Checkpoint operations included typical database maintenance tasks, the execution of custom property queries, FileNet CM sweeps, as well as IBM Content Navigator and API-driven workloads. The sweep framework is a new feature in IBM FileNet CM 5.2, providing the ability to traverse a database table and perform a pre-defined action on content elements that meet specific criteria. Two common types of sweeps were performed in order to demonstrate linear scalability. First, a sweep on the root document class, effectively sweeping over the entire multi-billion row Document table. Second, a more targeted sweep, specific to a single document class was executed. Overall system performance and scalability was demonstrated through the execution of a full mix of ECM operations driven by a custom Java API-based test driver. The ECM Workload was executed regularly throughout the population, scaling to over 120,000 transactions per minute. System reliability was demonstrated by running a sustained workload via IBM Content Navigator (ICN). IBM Rational Performance Tester was used to drive a complex workload for 72 hours and successfully demonstrated system stability when using a large content repository. The performance results reported in this paper present data and workloads run on an isolated network on specific operating environments and system configurations. Actual performance in real customer environments with production workloads may vary depending on many factors such as system configuration, workload characteristics, and data volume. The results presented here are not guaranteed to be repeatable in other systems.

4 IBM FileNet Content Manager 5.2 High Volume Scalability Page 4 1. Test environment description Architectural Overview The test environment was comprised of a single IBM Power (Model 9179-MHD) server with 5 native LPARs for the various software components. The Tivoli LDAP server was hosted on a local Microsoft Windows virtual machine. All file storage and database data files were located on an IBM XIV Generation 3 high performance storage device with SSD cache. All systems communicated through a private 10 Gbps VLAN. The IBM FileNet CM application was deployed in a three-node IBM WebSphere Network Deployment cluster, and each LPAR (node) contained a single FileNet CM Java Virtual Machine (JVM). See the Appendix at the end of this paper for more detailed environment information. Figure 1 Overall system architecture diagram

5 IBM FileNet Content Manager 5.2 High Volume Scalability Page 5 Test Bed Population The IBM FileNet object store was populated with 120 custom document classes, each consisting of between custom properties in total. Custom metadata was created to simulate typical customer needs. Figure 2 describes the custom properties and their values, which were used for query performance validation as the data population increased. Section 2, custom property validation, will show the results of the query s performance. Custom Property Type Date Time String (default 128 character length) Custom Property Values Timestamp at time of document creation Unique FileNet document ID (GUID) Integer low cardinality Normal distribution ( Bell curve ) between 1-10 Integer high cardinality Random number between 1-1,000,000 ID Float Multi-value String Multi-value String (256 character length) Content Unique FileNet document ID (GUID) 2 random float values per document Figure 2 Custom property descriptions. 2 random string values per document (out of 6 total) ~300 byte long String (stored in the LOB space) The ECM Workload (see section 2, ECM Workload) population was comprised of 10 million documents ranging from 10 KB to 5 MB distributed across multiple document classes and consisting of multiple versions. The remainder of the large population consisted of approximately 1 billion 30 KB documents, 2 billion 8 KB documents and 2 billion zero-byte content elements. Custom objects The ECM Workload utilizes FileNet CM custom objects. A total of 10 million custom objects were populated with roughly 1 million filed into folders. Folders The ECM Workload relies on an exact number of documents to be filed into specific folders during browse and query operations. A total of 1.3 million folders were created with a total of 7 million documents filed into these folders, with a varying number of containees ranging from 50 to 500. LDAP The IBM Tivoli Directory Server was pre-populated with 152,000 unique users and 5,000 groups. Each group contained a minimum of 20 members. The documents populated for the ECM Workload contained an additional 6 non-default ACLs comprised of 4 grant and 2 deny privileges.

6 IBM FileNet Content Manager 5.2 High Volume Scalability Page 6 2. Population to 5 billion documents Methodology The primary objective of these tests was to confirm that the IBM FileNet Content Platform Engine (CPE) would scale as the data population increased. At each population checkpoint, the following operations were performed to validate system performance: Custom property index creation and gathering updated database table statistics Execution of queries against various custom properties (cold and warm database) ECM Workload; full mix of operations performance test Execution of operations that sweep across the entire document repository Population checkpoints occurred at various stages of the 5 billion document population. During the initial phase to reach 1 billion documents, checkpoints were performed every 250 million documents. The system remained very stable up to the 1 billion milestone and checkpoints were increased to every 500 million documents until 3 billion documents were populated. The final checkpoint was performed at the 5 billion document milestone. Ingestion Document ingestion was performed using a custom IBM FileNet CPE Java API client application over the Java EJB (IIOP) protocol. The population was performed by 4 population clients (JVMs), each executing 4 Java threads, for a total of 16 concurrent population threads. Each population client created documents for a specific document class. Database Maintenance Operations Index creation was performed without gathering statistics. After all custom property indexes were created, basic table statistics were gathered, followed by detailed statistics with default sampling. Figure 3 demonstrates the linear scalability of updating database statistics on a multi-billion row database table. Figure 3 Database table statistics (RUNSTATS) scalability results.

7 IBM FileNet Content Manager 5.2 High Volume Scalability Page 7 Custom Property Validation The following custom property queries were manually executed by a stand-alone client application using the FileNet CPE Java API. SELECT TOP 100 low_card_prop FROM DOCUMENT WHERE low_card_prop = [INT]; SELECT high_card_prop FROM DOCUMENT WHERE high_card_prop = [INT]; SELECT datetime_prop FROM DOCUMENT WHERE datetime_prop >= [TIMESTAMP] AND datetime_prop <= [TIMESTAMP] ORDER BY datetime_prop; SELECT string_prop FROM DOCUMENT WHERE string_prop = {GUID} ; SELECT id_prop FROM DOCUMENT WHERE id_prop = {GUID}; SELECT id_prop, string_prop FROM DOCUMENT WHERE id_prop = {GUID} OR string_prop = {GUID} ; SELECT Id FROM DOCUMENT WHERE [FLOAT] IN float_prop_mv; SELECT TOP 100 Id FROM DOCUMENT WHERE ( [COLOR] IN string_prop_mv AND high_card_prop = [INT]); SELECT long_string_prop FROM DOCUMENT WHERE high_card_prop = [INT]; Custom property validation was performed twice for each query, once when the database was cold and then immediately afterwards when the database was warm. In addition to restarting the FileNet CM applications between property validations, the database was deactivated as well to ensure all memory buffers were cleared. Figure 4 As the database tables became larger, cold execution times gradually increased.

8 IBM FileNet Content Manager 5.2 High Volume Scalability Page 8 Figure 5 As the database tables became larger, warm execution times remained stable as content was already loaded into the database bufferpool. ECM Workload The ECM Workload is a Java API driven workload comprised of various create, delete, update, browse, query and retrieval operations with documents ranging in size from 10KB to 5MB. Figure 6 shows the distribution of operations in the workload. All content was stored in the file store. Each test execution was performed by a single test driver with 2,000 virtual users (client threads), with an average think time of 3 seconds, resulting in an effective workload of 40,000 transactions per minute (TPM). Operation Type Percentage of total workload Create/Delete 19% Update 14% Browse 14% Query 25% Retrieve 28% Total 100% Figure 6 ECM Workload work mix operation breakdown

9 IBM FileNet Content Manager 5.2 High Volume Scalability Page 9 Figure 7 demonstrates the stability of each type of operation for the ECM Workload executed at various population intervals. Figure 7 ECM Workload response time stability as population increased.

10 IBM FileNet Content Manager 5.2 High Volume Scalability Page 10 Sweep Execution A full Document table (DocVersion) sweep was performed along with a sweep based on a specific document class. The document class sweep was roughly ¼ of the total population for a given interval, hence the smaller duration. Figure 8 shows how sweep execution scaled linearly as the population increased. Both sweeps were executed in preview mode to demonstrate raw performance capability without modifying the test bed. Figure 8 Preview mode document sweep scalability results

11 IBM FileNet Content Manager 5.2 High Volume Scalability Page High volume transactional workloads IBM FileNet Content Manager workload scalability In addition to validating that the IBM FileNet Content Manager repository is able to maintain stable performance characteristics as a population grows to several billion documents, further tests were performed to evaluate the scalability of the IBM FileNet CM Java API at multiple checkpoints from 1 billion to 3 billion documents. Good response times were observed up to 120,000 transactions per minute, after which the test client became the bottleneck, preventing accurate reporting of operation response times. Figure 9 Demonstrates the scalability characteristics of the ECM Workload

12 IBM FileNet Content Manager 5.2 High Volume Scalability Page 12 IBM Content Navigator workload stability The IBM Content Navigator (ICN) component was configured against the 5 billion document IBM FileNet Content Manager object store. A 6,000 transaction per minute (TPM) full-mix workload was performed over 72 hours successfully demonstrating the stability of ICN against a large FileNet CM object store. The ICN driven workload included workflow and ICN Teamspace operations in addition to standard content based operations. Operation Type Percentage of total workload Create/Delete Document 15% Update 10% Browse 10% View Property 15% Search 10% Retrieve 10% Workflow 20% Social Features 5% Teamspace 5% Total 100% Figure 10 ICN work mix operation breakdown

13 IBM FileNet Content Manager 5.2 High Volume Scalability Page Performance considerations and recommendations The performance topics below were instrumental in achieving the goal of this study and should be strongly considered when attempting to scale a FileNet CM object store into the billions of documents. Many concepts discussed in this section can be leveraged by smaller scale deployments. IBM recommends strict performance benchmarks be performed in a highly controlled test environment before implementing any of the suggested tunings below in a production environment. Database storage IBM suggests enabling DB2 table and index compression on any table that will grow to a significant number of rows. Compression allows for more efficient use of buffer pools along with improved performance of table access in addition to general space savings with minimal CPU overhead. IBM recommends DB2 databases be created with automatic storage and dedicated table spaces for data, index and large objects (LOBs) for each object store. Ensuring that all table spaces are created on storage groups with multiple storage paths is crucial for optimal performance. Utilizing multiple storage paths will allow for parallel pre-fetching of table space data along with more efficient access to underlying storage devices by allowing multiple I/O paths (if available) to be accessed concurrently. File Systems / File Stores For large populations, IBM recommends that file store(s) use the large storage area structure. Multiple storage areas should be used to prevent too many documents from being stored in a single subdirectory of the underlying file system, which may adversely affect file system performance. Consult the IBM FileNet CM Information Center for more information and recommendations. IBM s General Parallel File System (GPFS) was used to provide the shared file system for each CPE instance. GPFS recommends that each file system should not exceed 1 billion documents. It is also important to ensure that the GPFS clients have sufficient cache available. For metadata-heavy workloads, consider creating dedicated NSDs for GPFS file system metadata. Please see the Appendix for more details on the specific tunings applied to this environment and the GPFS Wiki for more information about GPFS. JVMs Ample JVM heap space is critical for optimal application performance. Closely monitoring the heap usage during the system validation phase of a new deployment, as well as periodically during the lifecycle of your application will help ensure that excessive garbage collection is not impacting application response times. An inadequately sized heap can also create unnecessary load on server(s) due to frequent garbage collections, potentially increasing IT infrastructure costs. IBM suggests enabling verbose garbage collection to monitor JVM garbage collection activity. Indexing Ensuring sufficient SORTHEAP is available for in-memory sort operations is critical when creating large indexes. DB2 will perform a maximum of 6 concurrent sorts by default. Therefore, ensure that the SHEAPTHRESH_SHR value is at least 6 times that of the SORTHEAP value. Inability to perform sorts in memory will result in sort overflows into the temporary table space(s). Disk access is far slower than RAM, so these overflows can have a considerable impact on the performance of index creation.

14 IBM FileNet Content Manager 5.2 High Volume Scalability Page 14 In addition to sort overflows, temporary tables will be heavily utilized as sorting operations are completed and results are written to disk prior to the final sort merge. Therefore, having the temporary table space(s) spread across multiple high performance storage paths is strongly recommended. Covering index for sweeps Minimizing the impact of sweep execution on the database s resources with the presence of a proper covering index is crucial. Having a covering index in place will result in index-only access and minimize physical I/O reads during the sweep execution. A sufficiently sized bufferpool will help minimize physical I/O as well. Please consult the IBM FileNet CM Information Center for more information on creating sweep indexes. Consult the IBM FileNet CM Information Center performance tuning topics It is always suggested to refer to the P8 performance tuning topics in the IBM FileNet CM Information Center early in the planning and testing phase of a new deployment rather than after the environment is in production.

15 IBM FileNet Content Manager 5.2 High Volume Scalability Page 15 Conclusion Enterprise organizations are experiencing content growth at ever-increasing rates. For more than a decade, IBM FileNet Content Manger (FileNet CM) has been continually refined and improved upon to handle large amounts of business-critical data, securely and efficiently. IBM FileNet CM has been proven to meet the needs of Enterprise Content Management customers worldwide with repositories populated well into the hundreds of millions, with several customers exceeding a billion documents. The traditional guidance of spreading large populations across multiple object stores adds unnecessary complexity and administrative overhead to a company s IT organization. Many valid reasons do exist for utilizing multiple object stores in a FileNet CM system, such as segmenting the operations of varying business units or to meet IT backup and restore policies, to name a few. This paper demonstrates that even for document populations well into the billions, it is not necessary to utilize multiple object stores in order to achieve and maintain industry leading performance. Provided that the best practices and guidance outlined above are adhered to, it is possible to scale an IBM FileNet CM object store to five billion documents and beyond.

16 IBM FileNet Content Manager 5.2 High Volume Scalability Page 16 References 1. IBM FileNet Content Manager Software: 2. IBM FileNet P8 5.2 Information Center: a. Performance Tuning Topics: oc/p8ppt000.htm b. Monitoring IBM FileNet P8: doc/overview_monitoring_p8.htm c. Troubleshooting the Content Platform Engine 0.htm 3. IBM DB2 V10.1 Information Center: 4. IBM GPFS Wiki: lang=en#!/wiki/general%20parallel%20file%20system%20%28gpfs%29

17 IBM FileNet Content Manager 5.2 High Volume Scalability Page 17 Appendix Detailed Software Information IBM FileNet P8 Content Manager GA IBM Content Navigator GA IBM WebSphere Network Deployment IBM JDK (PPC64) IBM DB2 v LUW Advanced Enterprise Server Edition Storage Optimization Feature Enabled IBM GPFS v IBM AIX PPC IBM Tivoli Directory Server v6.3 Detailed Hardware Information IBM AIX POWER M/T 9179-MHD 128 CPU cores (SMT disabled) 2TB RAM 8 Gbps fiber cards (12 I/O paths configured) 10 Gbps Ethernet cards All XIV hdisks queue_depth increased to 256. All XIV hdisks max_transfer increased to 100,000. IBM XIV Generation 3 15 modules full stack 80x2TB SAS hard disks 6 TB SSD cache Detailed Tunings IBM FileNet Content Manager CPE o 8 GB min/max heap size o 4 GB nursery o Gencon garbage collection policy (WAS 8.x default) o Object Store JDBC data source pool max threads: 500 o Global Configuration Database (GCD) JDBC data source pool max threads: 10 o ORB thread pool max threads: 100 o IBM DB2 JDBC JCC driver v IBM Content Navigator 8 GB min/max heap size IBM GPFS pagepool: 8GB maxfilestocache: 5000 block size: 256KB 15 file systems with dedicated NSD for metadata each IBM DB2 INSTANCE_MEMORY: 256GB (fixed) Bufferpools o Data tablespace: 64GB fixed o Index tablespace: 32GB fixed Automatic maintenance tasks disabled Instance parameters o DB2_WORKLOAD=FILENET_CM o DB2_MINIMIZE_PREFETCH=YES (included by default in the above workload in future fix packs of DB2)

18 IBM FileNet Content Manager 5.2 High Volume Scalability Page 18 Author Information Michael Bordash ECM Server System Test Engineer Contributors Matthew Vest ECM Server System Test & Performance Engineering Senior Manager Dave Royer ECM Performance Architect, Senior Software Engineer Special thanks to the following members of the IBM FileNet CM development team: Mike Winter, ECM Distinguished Engineer Joseph Raby, ECM Development, Manager David Skinner, ECM Development Haibing Qiao, ECM Development

19 IBM FileNet Content Manager 5.2 High Volume Scalability Page 19 Disclaimer The information in this publication is not intended as a substitution of the IBM FileNet product documentation provided by IBM. Please see for more information about what publications are considered to be product documentation. References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product, program or service. Information in this publication was developed in conjunction with use of the equipment specified, and is limited in application to those specific hardware and software products and levels. The information contained in this publication was derived under specific operating and environmental conditions. While IBM has reviewed the information for accuracy under the given conditions, the results obtained in your operating environments may vary significantly. Accordingly, IBM does not provide any representations, assurances, guarantees, or warranties regarding performance. Any information about non-ibm ("vendor") products in this document has been supplied by the vendor and IBM assumes no responsibility for its accuracy or completeness. IBM, IBM FileNet Content Manager, DB2, WebSphere, AIX, Rational, and Tivoli are trademarks or registered trademarks of IBM Corporation in the United States, other countries, or both. Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. UNIX is a trademark of The Open Group. Windows is a registered trademark of Microsoft Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. Copyright IBM Corporation 2014 Produced in the United States of America All Rights Reserved The e-business logo, the eserver logo, IBM, the IBM logo, IBM Directory Server, DB2, FileNet, FileNet Content Manager and WebSphere are trademarks of International Business Machines Corporation in the United States, other countries or both. The following are trademarks of other companies: Solaris, Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries or both. Windows and Windows 2008 Enterprise Edition are trademarks of Microsoft Corporation in the United States and/or other countries Oracle 9i and all Oracle-based trademarks and logos are trademarks of the Oracle Corporation in the United States, other countries or both. Other company, product and service names may be trademarks or service marks of others. INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. Information in this paper as to the availability of products was believed accurate as of the time of publication. IBM cannot guarantee that identified products will continue to be made available by their suppliers. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-ibm web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY, USA

20 Copyright IBM Corporation 2014 IBM 3565 Harbor Boulevard Costa Mesa, CA USA Printed in the USA All Rights Reserved. IBM and the IBM logo are trademarks of IBM Corporation in the United States, other countries, or both. All other company or product names are registered trademarks or trademarks of their respective companies. The IBM home page on the Internet can be found at ibm.com