7/12/2010 Microsoft Corporation Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7 Benchmark Report Page 1

Transcription

1 Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Comparing Distributed Caching Performance on the Application Server Tier 7/12/2010 Microsoft Corporation 2010 Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7 Benchmark Report Page 1

2 This document supports the release of Windows Server 2008 R2 and the Microsoft.NET Framework 4.0. The information contained in this document represents the current view of Microsoft Corp. on the issues disclosed as of the date of publication. Because Microsoft must respond to changing market conditions, this document should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the companies, organizations, products, domain names, addresses, logos, people, places and events depicted in examples herein are fictitious. No association with any real company, organization, product, domain name, address, logo, person, place or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Microsoft grants you the right to reproduce this guide, in whole or in part. Microsoft may have patents, patent applications, trademarks, copyrights or other intellectual property rights covering subject matter in this document, except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights or other intellectual property Microsoft Corp. All rights reserved. Microsoft, Windows Server, the Windows logo, Windows, Active Directory, Windows Vista, Visual Studio, Internet Explorer, Windows Server System, Windows NT, Windows Mobile, Windows Media, Win32, WinFX, Windows PowerShell, Hyper-V, and MSDN are trademarks of the Microsoft group of companies. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

3 Contents Executive Summary... 6 Hardware Configurations Tested... 7 Summary of Key Findings... 8 Introduction Summary of Key Comparisons Base Comparisons Object Size Comparisons Scalability Comparisons Serialization Comparisons Cache Policy Implemented Full Disclosure Notice Middle-Tier Hardware Platforms Tested Distributed Cache Tier Hewlett-Packard C7000 Blade System with 4 HP ProLiant BL460c blades Application Server Tier Hewlett-Packard C7000 Blade System with 4 HP ProLiant BL460c blades Database Tier x HP ProLiant DL585 Server HP StorageWorks EVA 4400 Disk Array Test Methodology and Reporting Metrics Test Bed Specification Configuring Windows Server AppFabric and IBM extreme Scale Windows Server AppFabric IBM extreme Scale Running the Benchmarks and Capturing Reported Metrics Test Methodology for the Cache Benchmark Test Scripts Benchmark Results Pricing Table Small Object Scalability with Cache Failover Support Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7 Benchmark Report Page 3

4 Cache Statistics for Measurement Period Benchmark Discussion Small Object Scalability with No Cache Failover Cache Statistics for Measurement Period Benchmark Discussion Large Object Scalability with Cache Failover Support Cache Statistics for Measurement Period Benchmark Discussion Large Object Scalability with No Cache Failover Cache Statistics for Measurement Period Benchmark Discussion Conclusion Appendix A: Pricing Pricing for the Application Server Used in the Tests IBM extreme Scale 7.1/ Hewlett Packard BladeSystem C IBM WebSphere 7/ Hewlett Packard BladeSystem C Total IBM Software Licensing Costs by Configuration (MSRP) Total Microsoft Software Licensing Costs by Configuration (MSRP) Appendix B: The Application Workload Database Definition Object Definition Address Object Java Customer Object Java Order Object Java LineItemObject Java Address Object C# Customer Object C# Order Object C# LineItem Object C# App Servlet Java Test.aspx C# OrderBL Java Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 4

5 OrderBL C# Appendix C: Tuning Parameters WebSphere Tuning (Version bit) Windows Server 2008 R extreme Scale 7.1 Tuning Windows Server 2008 R Server Properties File ORB Tuning Deployment.xml ObjectGrid Deployment File ObjectGrid.XML ObjectGrid Configuration File NET Tuning AppFabric Client Configuration File Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 5

6 Executive Summary In June of 2010 Microsoft released Windows Server AppFabric, which is available for free download at Windows Server AppFabric is provided as extensions to the Application Server role of Windows Server, and any application is free to use its parts separately or together. AppFabric extends new features to server-side.net applications running on Windows Server The first release of Windows Server AppFabric has two parts: AppFabric Caching Services, which can speed up access to frequently accessed information such as data stored in backend databases as well as session data used by an ASP.NET application. AppFabric Hosting Services, making it easier to run and manage services created with Windows Communication Foundation, especially those built using Windows Workflow Foundation. This paper focuses on the AppFabric Caching Services and its relative performance to IBM WebSphere extreme Scale 7.1. AppFabric caching features provide a distributed cache platform for developing scalable, available, and highly-performant applications. Using AppFabric Caching Services, developers can deploy a distributed grid of servers that act as a coherent cache for middle-tier applications developed using the.net Framework 4.0. The AppFabric distributed caching nodes keep the cache partitioned across servers running in the cache cluster, and can optionally also provide failover support for the cache in case one or more nodes fail. The following are the key features of the AppFabric Caching Service: Caches any serializable CLR object and provides access through simple cache APIs Supports enterprise scale: tens to hundreds of computers Configurable to run as a service accessed over the network Supports common cache configurations Supports dynamic scaling by adding new nodes A high availability feature which supports continuous availability of your cached data by storing copies of that data on separate cache hosts Automatic load balancing Integration with administration and monitoring tools such as Event Tracing for Windows (ETW) and System Center. Provides tight integration with ASP.NET to be able to cache ASP.NET session data in the cache without having to write it to source databases. It can also be used as a cache for application data to be able to cache application data across the entire Web farm. IBM extreme Scale 7.1 similarly provides a distributed caching environment based on IBM s ObjectGrid technologies. Typical deployments of extreme Scale are in conjunction with Java-based application servers such as IBM WebSphere 7.0. IBM extreme Scale, like Microsoft AppFabric, partitions data across running nodes, and can optionally provide failover support for the cache in case one or more nodes fail. The goal of both technologies is similar: to enable developers to deploy applications which are extremely fast, reliable, and can scale out across potentially hundreds of computers in a distributed environment. By utilizing a distributed middle-tier cache, developers using Java and.net can dramatically reduce the load on central database servers, providing additional scalability. In this benchmark, a variety of configurations for each platform are tested and compared in terms of Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 6

7 throughput, response times, application server and cache server capacity consumption, and price. All configurations tested were run on the same hardware configurations for comparative purposes. Hardware Configurations Tested 1 Database Server: HP ProLiant DL585, 4 x AMD 64-bit 1.8 GHz. The server is configured with 32GB RAM. The database server runs 64-bit IBM DB2 v9.7 Enterprise Edition for Windows Server 2008 (in all extreme Scale tests), and 64-bit Microsoft SQL Server 2008 Enterprise Edition in all Windows Server AppFabric tests. Tuning details are provided in Appendix C. 1-4 Application Servers: Hewlett Packard BladeSystem C7000, two HP ProLiant BL460c blades each with one quad-core processor (4 cores per 3.0GHz. Each application server blade has 16GB RAM and is running Windows Server 2008 R2. The application server is IBM WebSphere bit (in all extreme Scale tests) and Microsoft.NET Framework bit (in all Windows Server AppFabric tests). Tuning details are provided in Appendix C. 2 Cache Servers: Hewlett Packard BladeSystem C7000, two HP ProLiant BL460c blades each with one quad-core processor (4 cores per 3.0GHz. Each cache server blade has 16GB RAM and is running Windows Server 2008 R2. The cache server is 64-bit IBM WebSphere extreme Scale 7.1 (internal version 4.0, V ) (in all extreme Scale tests) and Microsoft Windows Server AppFabric 64-bit (in all Windows Server App Fabric tests). For the IBM extreme Scale setups, IBM JVM version (IBM Windows 64-bit build pwa6460sr _01 SR2) is used as the container JVM. Tuning details are provided in Appendix C. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 7

8 Summary of Key Findings Key findings from this study include the following: Both IBM extreme Scale 7.1 and Microsoft Windows Server AppFabric are capable of dramatically improving the scalability and performance of business applications by providing a robust distributed caching grid. Both products can dramatically reduce the utilization/load on central database servers. Microsoft Windows Server AppFabric outperforms IBM extreme Scale 7.1 in key test cases with cache failover (replication) support turned on. Microsoft Windows Server AppFabric provides this better performance at a fraction of the cost of IBM extreme Scale when compared running on the same hardware configurations. In the small object tests with cache replication, the Microsoft Windows Server AppFabric solution costs just 22% of the IBM extreme Scale solution, yet provides 15% better performance as measured in transactions per second. In the large object tests with cache replication, the Microsoft Windows Server AppFabric solution costs just 22% of the IBM extreme Scale solution, yet provides 11% better performance as measured in transactions per second. The following table shows both the overall cost of the products as tested, as well as their relative performance as measured in peak sustained transactions per second. The metrics reported below are based on both the application server licensing costs (the cache clients); and the distributed cache server licensing costs. All pricing is detail in Appendix A, and is based on manufacturer s suggested retail price for the licenses required to support the hardware configurations tested. All configurations below are tested on the exact same hardware setups. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 8

9 Platform Tested IBM WebSphere with IBM extreme Scale 7.1 and cache failover configuration Microsoft.NET with Windows Server AppFabric and cache failover configuration IBM WebSphere with IBM extreme Scale 7.1 and cache failover configuration Microsoft.NET with Windows Server AppFabric and cache failover configuration Distributed Cache Servers Application Servers Total Middle Tier Software Licensing Costs 1 Performance as measured in peak TPS rates Small Object Tests with Cache Replication Price Performance Metric ($ cost/tps rate achieved) 2 4 $115, ,230 $35.62 per TPS achieved 2 4 $24, ,711 $6.69 per TPS achieved Large Object Tests with Cache Replication 2 4 $115, ,522 $45.62 per TPS achieved 2 4 $24, ,790 $8.90 per TPS achieved 1 See Appendix A for all pricing breakouts. Pricing includes all middle tier software licensing costs, including the appropriate MSRP pricing for IBM WebSphere 7; IBM extreme Scale 7.1; and Windows Server 2008 R2. Pricing does not include hardware costs of the middle tier servers, as these are the same for the IBM and Microsoft configurations compared (comparisons were run on the exact same hardware setups). Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 9

10 Introduction To perform the comparative analysis for these distributed caching platforms, the performance of several different scenarios is compared using a common application server workload. In all configurations there are two cache servers and one database server. The number of application server nodes is varied from one to four, with a variety of data captured as new application server nodes are added. The goal is to show the peak throughput as additional middle tier application server nodes are added. Non-caching tests are also conducted to show the relative performance and scalability with and without distributed caching services enabled. Another goal of the benchmark tests is to show the overhead required to use cache replication. With cache replication turned on, both MS Windows Server AppFabric and IBM extreme Scale keep multiple copies of cached data across running nodes. With two physical cache servers, each server stores both a set of primary cached data (as accessed from the application servers); and a replica of the primary data stored on other cache server(s). In such a configuration, should one cache server fail, another cache server then becomes the primary cache server for the failed server s primary cache, such that no cached data is lost. Both MS Windows Server AppFabric and IBM extreme Scale can support hundreds of cache servers, but in these tests, two physical cache servers are kept consistently running to measure the amount of application server load these servers can handle, as measured in peak throughput rates as application server nodes are added to the configuration with cache replication turned on. Summary of Key Comparisons In all cases listed below a variety of cache reads, cache removes and cache updates are performed, with a varying number of objects. The exact details for the workload (which is the same across all test runs) are provided in a following section. Additionally, for each test run the cache read percentage, cache remove percentage and cache insert/update percentage is reported. Also, an overall cache hit and cache miss rate is reported. Base Comparisons 1. No cache enabled. In this case all read operations hit the database directly. 2. Non-replicated cache. In this case the two cache servers each have the same amount of memory dedicated to caching objects as they are read from the database such that most subsequent reads come from the cache, and are not re-read from the database. In this configuration, each cache server has exactly 10 gigabytes dedicated to storing objects in the cache. Theoretically, up to 20 gigabytes of data can be cached. Both AppFabric and IBM extreme scale, however, manage their memory based on their CLR and Java-based garbage collection logic. In all cases in this configuration, the caches are utilizing an LRU (least recently used) algorithm for maintaining their caches within the applied memory limits. Both IBM extreme Scale and Microsoft AppFabric automatically partition the cached objects across servers such that equal load balancing is achieved across the two cache servers without any special application logic needed. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 10

11 3. Replicated cache. This case is similar to the non-replicated cache configuration above in that the application logic and objects cached are exactly the same. However, cache replication is turned on for failover support. Each server is still given 10 gigabytes to store objects; however, theoretically this means each server can cache 5 gigabytes of unique objects, with a total storage for 10 gigabytes of data since each object must be maintained twice (once as a primary on one server; and once replicated to the other server). Both IBM extreme Scale and Microsoft AppFabric automatically partition the primary and replica caches across servers such that equal load balancing is achieved. Object Size Comparisons For each of the three base comparisons listed above, individual benchmark runs were conducted across two object sizes: Small Orders Object (size in AppFabric cache is about 1K) Large Orders Object (size in AppFabric cache is about 7K) The amount of storage required for each object is dependent on the platform being tested; and for IBM extreme Scale also on the serialization type chosen (see serialization in the later section). Identical objects were defined in C# and Java for the business tier to act as model objects for the database. These objects were defined in C# for the.net/appfabric platform, and in Java for the WebSphere/eXtreme scale platform. The objects in all cases are marked as serializable, since the caching engines cache a serialized version of each object, with the application server tier automatically serializing and deserializing objects as they are placed into/taken from the distributed cache servers. The objects include Customers, Addresses, Orders and LineItems. The Order object includes a Customer object, and each Customer object contains two Address objects (shipping and billing). Additionally, each Order object contains 1 to n line items. For the small object test cases, each order has exactly one line item. For the large object test cases, each Order object contains exactly 50 distinct line items. See Appendix A for the object source listing in C# and Java, and for the corresponding database schema used for the database tier. Scalability Comparisons To see how two dedicated, distributed cache servers can add horizontal scalability to an application server tier, all benchmarks were run across 1, 2, 3 and 4 load-balanced application servers. The number of distributed cache servers is kept constant at 2 nodes, to measure how much load and additional scalability this fixed number of cache servers can achieve as the application server tier is scaled out. As application server nodes are added, the results show how transaction throughput scales for each scenario tested. By the time four application server nodes are running the bottleneck for both AppFabric and extreme Scale becomes the two cache servers. However, this bottleneck is hit at different times for each configuration run. At this point, additional cache servers could be added to further scale the solution for both environments. The tests are purposefully run with a fixed number of cache servers (2) to show the relative peak performance (in peak throughput) for the cache servers when running Windows Server AppFabric vs. IBM extreme Scale in each configuration tested. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 11

12 Serialization Comparisons Windows Server AppFabric always uses the built-in.net CLR serialization. However, for IBM extreme Scale, IBM suggests using a special byte array serialization (copymode = COPY_TO_BYTES ) that can help performance in some (but not all) cases. Hence, we tested IBM extreme Scale using both the default Java-based class serialization; and using the special COPY_TO_BYTES setting. We found for these tests the COPY_TO_BYTES mode much more performant. Cache Policy Implemented In all the scenarios tested, the cache policy is as follows: a) Objects are cached for entire test run duration until explicitly removed from the cache by the application or automatically evicted based on each product s memory-based eviction model. Each cache server is dedicated 10GB for cache storage, with memory-based eviction set at a 95% high-water mark. In the tested cases, however, the amount of RAM dedicated to holding cached objects is sufficient to hold the entire set of objects based on the measured percentage of operations (cache reads, cache removes, cache inserts/updates) during test runs. b) No near cache is used, because the business scenario demands that objects cannot be stale with respect to the cache servers. Note that both Windows Server AppFabric and IBM extreme Scale have a near cache that enables each application server to maintain a subset of cached data inmemory with the application server. However, for both products, use of the near cache also implies that the data at the application tier might be stale with respect to the data on the cache servers. For our scenarios, we wanted to ensure the application tier always pulled the data directly from the cache servers, so no cache staleness exists. c) An optimistic locking strategy is used. Cache.Put operations are used for both extreme Scale and AppFabric Caching Services are used, with a lock timeout of 10 seconds. No exclusive locks are taken on Get operations across both products. Additionally, no object updates are performed, but removes are performed. d) Tests are run using a 77.7% read, 11.15% remove and 11.15% insert mix. 2 This ensures that new cache puts and cache replication for the cache replication tests are continually exercised throughout the benchmark runs. With constant removals of objects during test runs, standard JVM and CLR garbage collection is also occurring, just as with real deployed applications. e) The database and the middle tier caches are fully loaded prior to test runs. Cache remove operations force a re-read from the database for the next access to that object; however cache removes do not remove the item from the database; rather just from the middle-tier cache such that the object is refreshed from the database on the next request for that object. 2 This is approximate given the cache is not under memory-based eviction (which is configured equivalently for both products). Even if the cache memory across servers can hold all objects, a percentage of inserts are being performed since removes are continually being called. If memory-based eviction is occurring (which is strictly based on the efficiency of each platform given the same fixed amount of memory and same high and low water marks), then the insert rate will be higher than the remove rate since objects are being removed by the infrastructure as well as explicitly by the benchmark workload. Based on actual statistics measured during test runs, memory-based eviction does not occur for any test run. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 12

13 Full Disclosure Notice The complete source code, test script flow and all testing methodology for this benchmark is available in this document. Any reader may use this code to perform the benchmark for themselves to verify the results, or perform the benchmark on different hardware configurations. Extensive time was taken to generate results that represent optimal tuning for the platforms tested, and we are quite confident in the results. We encourage customers to perform their own comparative testing and functional and technical reviews of the workload tested. Middle-Tier Hardware Platforms Tested A key goal of the testing was to document the performance and relative cost of an IBM WebSphere and extreme Scale 7.1 setup compared to the same hardware configuration running Microsoft.NET and Windows Server AppFabric. In all cases, the application server tier and distributed cache tier are running on the same Hewlett Packard C7000 BladeSystem (Intel-based) running Microsoft Windows Server 2008 R2; and the database is running on an HP DL585 server also running Microsoft Windows Server 2008 R2. In the extreme Scale/WebSphere tests, IBM DB bit Enterprise Edition is used as the database with access via IBM s latest JDBC type 4 driver. Data Access is via straight JDBC commands invoked via the main application Servlet, with data then displayed in a JSP page via in-line script. In the Microsoft AppFabric tests, SQL Server 2008 with ADO.NET 4.0 is used for the database. Data access is invoked by an ASP.NET Web Form and displayed via in-line script. The SQL statements and data access tier (JDBC and ADO.NET) are constructed to be identical across platforms. Complete pricing details, including hardware and software costs, are documented in Appendix A of this paper. For all tests, each platform was tested against the same database hardware and disk configuration, which was carefully monitored to ensure the databases were never a bottleneck for any middle-tier platform tested except in the no-cache configurations. In addition, network monitoring was carefully conducted to ensure the network was never an artificial bottleneck. Hence, the performance measured accurately captures the peak TPS rate achieved on the middle tier hardware and software platform tested. Distributed Cache Tier Hewlett-Packard C7000 Blade System with 4 HP ProLiant BL460c blades 2 Hewlett Packard ProLiant BL460c blades Each blade has one Quad-Core Intel Xeon E5450 (3.00GHz, 1333MHz FSB, 80W) Processor Each blade has 32 GB RAM Each blade is running Microsoft Windows Server 2008 R2/64-bit Each blade has 2 x 1 GB NICs Application Server Tier Hewlett-Packard C7000 Blade System with 4 HP ProLiant BL460c blades 1-4 Hewlett Packard ProLiant BL460c blades Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 13

14 Each blade has one Quad-Core Intel Xeon E5450 (3.00GHz, 1333MHz FSB, 80W) Processor Each blade has 32 GB RAM Each blade is running Microsoft Windows Server 2008 R2/64-bit Each blade has 2 x 1 GB NICs Database Tier 1 x HP ProLiant DL585 Server 4 x AMD Opteron Server is configured with 32 GB RAM. The server has 2 x 1GB NICs. The server is running IBM DB2 9.7 Enterprise Edition 64-bit or Microsoft SQL Server 2008 Enterprise 64-bit. The server is running Microsoft Windows Server 2008 R2 64-bit, Enterprise Edition. Database Storage Specification HP StorageWorks EVA 4400 Disk Array 96 15K drives total 2 logical volumes consisting of 48 drives each Logical Volume 1 for logging Logical Volume 2 for database Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 14

15 Test Methodology and Reporting Metrics For each result, we report a peak sustained transaction per second (TPS) throughput rate as averaged over a 30 minute measurement period as tracked by the testing tool. The cache is pre-loaded and then a 15 minute warm-up time is given to ensure measuring begins at steady state. For the tests, HP Mercury LoadRunner was used to drive load and capture the results. Note that each transaction (for the TPS rates) is defined as the return of a complete page to the user. It should be noted that for each page returned (each transaction), multiple cache operations are being performed. Each page exercises the following logic: 1. Get an Address object from the Address cache. If not in the cache, get from the database and place in the cache. The cache key is the customer id passed in from the browser. The key is a random number between 1 and 5000; hence the cache contains 5000 Address objects after loading. 2. Get a Customer object from the Customer cache. If not in the cache, get from the database and place in the cache. The cache key is the customer id passed in from the browser. The key is a random number between 1 and 5000; hence the cache contains 5000 Customer objects after loading. 3. Get 5 Order Objects from the Orders cache. If not in the cache, get from the database and place in the cache. The key is a random number between 1 and 1,000,000; hence the cache contains 1,000,000 Order objects after loading. 4. Remove the last Order object from the Orders cache as read from (or inserted into) the cache in step 3. For every page returned (each transaction as shown in the results) there are exactly 7 distinct cache reads and one cache remove. Because one Orders object is removed from the cache on every page exercised, roughly 11.15% of overall operations also involve a cache miss with subsequent database access and cache insert. Test Bed Specification In the test, 60 physical client machines (1.2 GHz, 512MB RAM) were used to run the test scripts, with each client driving several hundred distinct users with a one-second think time. Load balancing across the application servers is achieved via the test scripts, with multiple and identical scripts running for each application server. For each application server, two simultaneous scripts are run to achieve network load balancing across the two NICs in each server. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 15

16 Figure 1: Test bed used for the Cache benchmark using IBM extreme Scale 7.1 and Microsoft Windows Server AppFabric Caching Services. The number of application servers varies from 1 node to 4 nodes for the scalability test runs. There are always two distributed cache servers in every tested cache configuration. In the no-cache tests, the cache servers are not utilized. For.NET AppFabric, each cache server is running as an AppFabric Caching host within the two server cache cluster. Each cache host has exactly 10GB dedicated RAM storage. For IBM extreme Scale, each physical cache server is running three ObjectGrid containers. Each container has exactly 3.333GB heap space for a total of 10GB dedicated RAM storage per physical server, just as with the AppFabric tests. Performance gains are realized for extreme scale by running multiple containers/jvms per physical machine, as garbage collection cycles are shorter and more evenly distributed vs. running just one physical JVM instance (with 10GB heap) per server. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 16

17 Configuring Windows Server AppFabric and IBM extreme Scale Windows Server AppFabric Each physical cache server is running AppFabric caching services (a 2-server cluster setup). Each host is configured with 10GB dedicated storage with a 95% high-water mark before memory-based eviction will begin. The cluster has three distinct caches: Address Customers Orders For all tuning details of Microsoft Windows Server AppFabric and IIS/.NET, see Appendix C. IBM extreme Scale We found it advantageous with IBM extreme Scale to run multiple container nodes per physical server. This keeps the JVM garbage collection cycles shorter and more evenly distributed and produces higher overall throughput. We ran 3 ObjectGrid container/jvms per physical server (total ObjectGrid cluster has 6 nodes on two physical servers). There are three distinct backing maps/caches: Address Customers Orders Each backing map has 6 partitions. Each container gets a heap size of 3.333GB, so the total amount of memory available for the cache is identical to the AppFabric setup (10GB per physical server). We ran tests using three different garbage collection modes: -Xgcpolicy:optthruput -Xgcpolicy:optavgpause -Xgcpolicy:gencon We definitely found the default mode of optthruput offered the best throughput and most reliable configuration. All results are hence for this GC mode. For all tuning details of the JVM, IBM ORB, IBM WebSphere, IBM extreme scale, see Appendix C. Running the Benchmarks and Capturing Reported Metrics In all cases, users are added to the system with iterative test runs until peak throughput is obtained. Extensive benchmark runs were done prior to measurement to ensure proper tuning of the middle tier systems including the application servers and the cache servers. For each of the configurations tested we report the peak sustained TPS rate, and also a calculated dollar cost per TPS (price performance metric) so customers can better understand what they are paying for that performance using a standard normalized metric: cost of platform divided by peak TPS rate measured. This calculation mirrors how testing organizations such as the Transaction Processing Council (TPC) report price performance metrics. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 17

18 The cost calculations are based on the middle tier application server software licensing costs and cache server licensing costs since the hardware costs are the same for each configuration compared. Notes and details on the pricing of the middle tier software are included in Appendix A, and prices are based on published pricing from each vendor. Customers should understand that full.net capabilities are included in every edition of Windows Server 2008 R2, and upgraded versions of.net are made available for free download from MSDN (for example,.net 4.0 as tested here). Hence, there is no additional or separate application server cost associated with a.net application; while commercial JAVA EE 5 application servers such as WebSphere are separately licensed (and typically quite expensive) products. In addition, Windows Server AppFabric is a free download from MSDN; but IBM extreme Scale is a separately sold product. The total relative cost of the configurations tested and the dollar cost per TPS rates might, therefore, surprise some readers. It is also important to remember that this is a test of a typical but specific workload on the IBM WebSphere/eXtreme Scale and Microsoft.NET/Windows Server AppFabric platforms. While the workloads are exactly equivalent for each product as compared, different workloads and different configurations will yield different results for each platform. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 18

19 Test Methodology for the Cache Benchmark Test Scripts For the benchmark tests, HP Mercury LoadRunner was used to record test scripts browser interactions that exercise the Web application. These were run across 60 client machines (500 MHz Windows XP desktops with 512 MB RAM). User agents were configured to run with a one second think time between each request. Each benchmark run included a 15 minute warm up run to get to steady state, and a 30 minute measurement period. TPS rates were determined by LoadRunner by averaging across the 30 minutes. Error rates were monitored to ensure they remained at less than.01% during the measurement period. User loads were run for each application up to a number that represented peak throughput for that configuration, as determined in many iterative runs (literally hundreds) during the tuning stages. Extensive time was spent tuning IBM WebSphere and extreme Scale (see the appendix) to achieve peak throughput for the software/hardware configuration tested. Note that since the intent of this test is to test the performance of the cache servers, test scripts in this case were run directly against the IBM WebSphere Web Container (in-process HTTP listener) on port 9080, and IBM HTTP Server was not used as this introduces extra overhead in an IBM WebSphere environment. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 19

20 Benchmark Results For benchmark results detailed in this section, the following price comparison for middle tier software licensing costs can be applied. These are based directly on the manufacturer suggested pricing for application servers and distributed cache servers, as detailed in Appendix A. Pricing Table Platform IBM extreme Scale 7.1 with IBM WebSphere 7 Microsoft Windows Server AppFabric with Microsoft.NET Cost with 4 Application Server Nodes and Two Cache Server Nodes Cost to add (scale up) one Cache Server Node $115, $32, per cache server node $24, $5, per cache server node Costs breakouts detailed in Appendix A. Sample pricing is for 4 cores per application server node and 4 cores per cache server node. Costs include appropriate Windows Server 2008 R2 OS licensing costs. Small Object Scalability with Cache Failover Support Figure 2: Line chart view of Peak TPS Rates for the benchmark tests when scaling from one to four application server nodes. Cache nodes are fixed at two, with synchronous cache replication for AppFabric and IBM extreme Scale. In each case, one replica is maintained such that if one server fails, the cached data has a backup on the other server. Mercury LoadRunner is used to drive Http requests to the system under test, with the html pages returned to the virtual users. In this configuration, each Orders object has exactly one line item. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 20

21 Figure 3: Bar chart view of data as presented in Figure 2 above. Cache Statistics for Measurement Period The following table shows the actual cache statistics as measured using instrumented versions of the workload. The table shows that no memory-based eviction is occurring during the test runs. This is indicated by the fact the remove and insert rates are the same. Configuration Tested IBM extreme Scale w/ COPY_TO_BYTES IBM extreme Scale w/ COPY_READ_COMMIT Hit % Miss % Read % Remove Insert % % 85.60% 14.40% 77.80% 11.10% 11.10% 85.60% 14.40% 77.70% 11.10% 11.20% Microsoft Windows Server AppFabric 86.7% 13.3% 77.9% 11.00% 11.11% Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 21

22 Figure 4: Price performance metrics for IBM extreme Scale and Microsoft Windows Server AppFabric with 4 application server nodes and two cache server nodes in the small objects tests. Lower is better; as price performance ratios are the dollar cost of the platform tested divided by the total TPS rates for each platform. This provides a standardized metric that indicates how much the solution costs for the performance delivered. Benchmark Discussion The TPS throughput rates clearly show the advantage of a distributed cache for scaling the middle tier out. For both IBM extreme Scale 7.1 and Microsoft Windows Server AppFabric, throughput rates are throttled by the database (IBM DB2 or SQL Server 2008) even with just one application server node running. The distributed caches take this database bottleneck away, allowing throughput rates to scale well beyond what would be possible otherwise. In this benchmark, Windows Server AppFabric outperforms IBM extreme Scale 7.1 at four application server nodes acting as cache clients to the two distributed cache servers. This shows that Windows Server AppFabric is more efficient when compared to IBM extreme Scale, even when running in extreme scale COPY_TO_BYTES mode. Note that the default Copy_On_Read_Commit mode is significantly slower than extreme Scale COPY_TO_BYTES mode in this small object test. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 22

23 Small Object Scalability with No Cache Failover Figure 5: Peak TPS Rates for the benchmark tests when scaling from one to four application server nodes. Cache nodes are fixed at two, with no cache replicas maintained by AppFabric or IBM extreme Scale. Mercury LoadRunner is used to drive Http requests to the system under test, with the html pages returned to the virtual users. In this configuration, each Orders object has exactly one line item. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 23

24 Figure 6: Bar chart view of data as presented in Figure 4 above. Cache Statistics for Measurement Period The following table shows the actual cache statistics as measured using instrumented versions of the workload. The table shows that no memory-based eviction is occurring during the test runs. This is indicated by the fact the remove and insert rates are the same. Configuration Tested IBM extreme Scale w/ COPY_TO_BYTES IBM extreme Scale w/ COPY_READ_COMMIT Microsoft Windows Server AppFabric Hit % Miss % Read % Remove % Insert % 85.70% 14.30% 77.80% 11.10% 11.10% 85.70% 14.30% 77.60% 11.20% 11.20% 86.60% 13.40% 77.40% 11.20% 11.20% Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 24

25 Figure 7: Price performance metrics for IBM extreme Scale and Microsoft Windows Server AppFabric with 4 application server nodes and two cache server nodes in the small objects tests. Lower is better; as price performance ratios are the dollar cost of the platform tested divided by the total TPS rates for each platform. This provides a standardized metric that indicates how much the solution costs for the performance delivered. Benchmark Discussion This benchmark without cache replication shows that for both products, there is a throughput price to be paid for using cache replication, which is to be expected. However, in many cases the advantages will outweigh the performance drop-off given the reliability the cache replication modes provide. In this test, Microsoft Windows Server AppFabric continues to scale even through 4 application nodes (cache clients) connecting to the two distributed cache servers. It should be noted, however, that the cache servers do become the bottleneck at 4 client nodes, and very little additional scale would be achieved with 5 application servers. At this point, additional Windows Server AppFabric cache hosts would be added to the cache cluster to provide even more horizontal scalability for the application server tier beyond 4 application server cache clients. Even so, IBM extreme Scale, running on the same two cache servers, can only scale through 3 application server nodes. So this benchmark again shows Microsoft Windows Server AppFabric to be more efficient than IBM extreme Scale. The following chart shows the performance/throughput tradeoff of using cache replication vs. not using cache replication: Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 25

26 Figure 8: This chart shows the relative performance tradeoff for the small object tests when using cache replication vs. using no cache replication. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 26

27 Large Object Scalability with Cache Failover Support Figure 9: Peak TPS Rates for the benchmark tests when scaling from one to four application server nodes. Cache nodes are fixed at two, with synchronous cache replication for AppFabric and IBM extreme Scale. In each case, one replica is maintained such that if one server fails, the cached data has a backup on the other server. Mercury LoadRunner is used to drive Http requests to the system under test, with the html pages returned to the virtual users. In this configuration, each Orders object has exactly 50 line items. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 27

28 Figure 10: Bar chart view of data as presented in Figure 6 above. Cache Statistics for Measurement Period The following table shows the actual cache statistics as measured using instrumented versions of the workload. The table shows that no memory-based eviction is occurring during the test runs. This is indicated by the fact the remove and insert rates are the same. Configuration Tested IBM extreme Scale w/ COPY_TO_BYTES IBM extreme Scale w/ COPY_READ_COMMIT Microsoft Windows Server AppFabric Hit % Miss % Read % Remove % Insert % 85.70% 14.30% 77.80% 11.10% 11.10% 86.00% 14.00% 77.80% 11.10% 11.10% 86.50% 13.50% 77.90% 11.10% 11.00% Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 28

29 Figure 11: Price performance metrics for IBM extreme Scale and Microsoft Windows Server AppFabric with 4 application server nodes and two cache server nodes in the large objects tests. Lower is better; as price performance ratios are the dollar cost of the platform tested divided by the total TPS rates for each platform. This provides a standardized metric that indicates how much the solution costs for the performance delivered. Benchmark Discussion In this configuration IBM s default mode of Copy_On_Read_Commit never out produces the throughput rates when using IBM WebSphere 7/DB2 with no caching at all. The Copy_Bytes mode does provide significant scalability beyond the no-cache configuration with IBM WebSphere 7 and IBM DB2, however. But as with the small object benchmark, Windows Server AppFabric outperforms IBM extreme Scale. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 29

30 Large Object Scalability with No Cache Failover Figure 12: Peak TPS Rates for the benchmark tests when scaling from one to four application server nodes. Cache nodes are fixed at two, with no cache replicas maintained by AppFabric or IBM extreme Scale. Mercury LoadRunner is used to drive Http requests to the system under test, with the html pages returned to the virtual users. In this configuration, each Orders object has exactly 50 line items. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 30

31 Figure 13: Bar chart view of data as presented in Figure 8 above. Cache Statistics for Measurement Period The following table shows the actual cache statistics as measured using instrumented versions of the workload. The table shows that no memory-based eviction is occurring during the test runs. This is indicated by the fact the remove and insert rates are the same. Configuration Tested IBM extreme Scale w/ COPY_TO_BYTES IBM extreme Scale w/ COPY_READ_COMMIT Microsoft Windows Server AppFabric Hit % Miss % Read % Remove % Insert % 85.70% 14.30% 77.80% 11.10% 11.10% 85.70% 14.30% 77.80% 11.10% 11.10% 86.50% 13.50% 77.90% 11.10% 11.00% Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 31

32 Figure 14: Price performance metrics for IBM extreme Scale and Microsoft Windows Server AppFabric with 4 application server nodes and two cache server nodes in the large objects tests. Lower is better; as price performance ratios are the dollar cost of the platform tested divided by the total TPS rates for each platform. This provides a standardized metric that indicates how much the solution costs for the performance delivered. Benchmark Discussion The results of this benchmark without cache replication across cache hosts are similar to the results of the previous benchmark with cache replication. When compared to the previous benchmark, the results show for both products the performance tradeoff when using cache replication, which should be expected. IBM extreme Scale slightly outperforms Windows Server AppFabric, and only when running in the COPY_TO_BYTES mode with no cache replication in this large object test. The following chart shows the performance/throughput tradeoff of using cache replication vs. not using cache replication: Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 32

33 Figure 15: This chart shows the relative performance tradeoff for the large object tests when using cache replication vs. using no cache replication. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 33

34 Conclusion Both IBM extreme Scale 7.1 and Microsoft Windows Server AppFabric caching services provide the ability to reduce the database bottleneck while providing a reliable distributed cache for business objects. The exhaustive tests conducted in this benchmark show that Microsoft Windows Server AppFabric outperforms IBM extreme Scale in scenarios involving cache replication between distributed cache nodes. The pricing of the products tested also show a very significant price-performance advantage with Windows Server AppFabric. Specifically, the results show that: Microsoft Windows Server AppFabric provides better performance at a fraction of the cost of IBM extreme Scale 7.1 when compared running on the same hardware configurations. In the large object tests with cache replication, the Microsoft Windows Server AppFabric solution costs just 22% of the IBM extreme Scale solution, yet provides 15% better performance as measured in transactions per second. In the small object tests with cache replication, the Microsoft Windows Server AppFabric solution costs just 22% of the IBM extreme Scale solution, yet provides 11% better performance as measured in transactions per second. We encourage customers to review the source code included in Appendix B, examine the workload and perform their own comparative benchmarks of the two products. Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 34

35 Appendix A: Pricing The following pricing was used as the base pricing and price performance metric calculations. Pricing is based on published suggested retail pricing obtained directly from the vendor. Pricing does not include sales tax. Operating System Edition Windows Server 2008 R2 Standard Windows Server 2008, R2 External Connector License Windows Server 2008 R2 Enterprise Unit Price MSRP Platform Used $1, IBM WebSphere 7 and extreme Scale setups all servers and.net Application Server Nodes $1, IBM WebSphere 7, IBM extreme Scale, Microsoft.NET and AppFabric setups all servers (all configurations tested) $3, Microsoft.NET/ AppFabric setups for cache servers (Enterprise Edition is required for AppFabric distributed caching) Pricing for the Application Server Used in the Tests Pricing includes middle tier software licensing costs for the servers used in each configuration, as well as the appropriate Windows Server R2 prices. Database software costs and middle tier/database hardware costs were not included. We priced IBM WebSphere 7 Application Server Edition for the application server (50 Processor Value Units per core for Intel/AMD on the chipset tested). Note that for the.net 4.0/Microsoft Windows Server 2008 configuration, no separate application server is necessary:.net is integrated into Windows Server 2008 and new versions are made available as free downloads on MSDN. In addition, Windows Server 2008 AppFabric is a free download on MSDN, but requires Windows Server 2008 R2 Enterprise Edition to run the Cache Services. IBM extreme Scale 7.1/ Hewlett Packard BladeSystem C7000 Pricing Obtained from IBM Passport Advantage Published Pricing at 01.ibm.com/software/lotus/passportadvantage/pvu_licensing_for_customers.html. IBM WebSphere Application Server Processor Value Unit (PVU) License + SW Subscription & Support 12 Months 400 $ $57, (for both servers, 8 cores total) 3 50 PVUs per Intel XEON core; 8 Intel Xeon cores total on system as tested (4 cores each IBM extreme Scale server). Microsoft Windows Server AppFabric vs. IBM WebSphere extreme Scale 7.1 Benchmark Report Page 35