Benefit of Asynch I/O Support Provided in APAR PQ86769

Size: px

Start display at page:

Download "Benefit of Asynch I/O Support Provided in APAR PQ86769"

Andra Carroll
6 years ago
Views:

1 IBM HTTP Server for z/os Benefit of Asynch I/O Support Provided in APAR PQ86769 A review of the performance results realized in a benchmarking effort where the key was supporting large numbers of persistent sessions in a high-throughput environment. This document can be found on the web at: Search for document number WP100xxx under the category of "White Papers" Version Date: January 25, 2006 See "Document Change History" on page 8 for a description of the changes in this version of the document IBM Washington Systems Center Mike Cox Distinguished Engineer IBM Washington Systems Center coxm@us.ibm.com Don Bagwell IBM Washington Systems Center dbagwell@us.ibm.com

2 This page intentionally left blank

3 Table of Contents WP Benefits of Asynch I/O Support Provided in PQ86769 Executive Overview... 2 More on the Benchmark Queue Manager Configuration... 3 PQ APAR that supplied the enabling updates... 3 Background dilemma and solution via AsyncSockets... 3 The IBM HTTP Server for z/os -- configured in "scalable mode"... 4 Scalable web server -- without Queue Servers... 4 The WebSphere HTTP Server Plugin... 5 Key directives in the httpd.conf file... 6 Performance measurements achieved in benchmark... 7 Conclusion... 8 Document Change History

4 Executive Overview WP Benefits of Asynch I/O Support Provided in PQ86769 This paper is based on a benchmarking effort conducted by Mike Cox of the Washington Systems Center on behalf of a large banking customer. The benchmark involved many different test scenarios, but this paper focuses on one particular element of the test: The use of a a scalable IBM HTTP Server, but configured without queue servers. It used a queue manager only. And the use of a code enhancement which allowed the queue manager to host a large number persistent connections by employing asynchronous I/O. IBM HTTP Server WebSphere for z/os WebSphere HTTP Server Plugin Customer Application Configured as: Scalable Web Server Queue Manager only Code enhancement to support asynchronous I/O Scalable web server without queue servers, plus code enhancement to support asynchronous I/O This particular configuration scenario came about because of a need to overcome two limitations seen with the non-scalable HTTP Server and the more traditional scalable HTTP Server with queue servers: Traditional HTTP Server configuration Non-scalable HTTP Server Limitation Difficult to host large numbers of persistent connections. Persistent connections require a matching active thread. If all threads are used, requests are queued until a thread becomes available. Scalable HTTP Server with queue servers Capable of supporting large numbers of persistent connections, but at the "cost" performance from queuing requests to the queue servers. At the time of the benchmark the "code enhancement" was not part of a formal APAR. However, since then APAR PQ86769 was created, and APAR fix UQ88348 has been released to provide this enhancement as part of the maintenance stream for the IBM HTTP Server on z/os. In order to properly explain the benchmark configuration, some background on the HTTP Server scalable mode configuration will be offered. However, for a more detailed discussion of that, see "HTTP Server Planning, Installing, and Using" (SC ). The purpose of this paper is to provide the essential information about how this configuration was realized, and some information on the performance value realized in the benchmark. Bottom Line: A scalable HTTP Server configuration utilizing only a Queue Manager has approximately the same path-length as a non-scalable (or "standard") HTTP Server. With PQ86769 and the AsyncSockets directive, large numbers of persistent connections could be maintained without requiring the same number of dedicated threads, and the code path needed was comparable to a standard HTTP Server

5 More on the Benchmark Queue Manager Configuration PQ APAR that supplied the enabling updates This APAR is a collection of many things, some of which did not apply to the configuration being represented in this document. Specifically, the APAR listing cites the following enhancements:! AsyncSockets configuration directive! An environment variable for local memory pools! Environment variables for shared memory pools! New GWAPI plugin variables! Improvements for group files defined by %%SAF%%! Disabling of HTTP methods TRACE and OPTIONS by default! New options for request logging Of those, only the first was utilized for the benchmarking effort from which comes this paper. Background dilemma and solution via AsyncSockets Prior to the release of the fix for APAR PQ86769, a non-scalable HTTP Server required a socket and a worker thread for the duration of the persistent connection. There could never be more connections than the value of the MaxActiveThreads directive at any given time. When all threads were active, a request that arrived would be queued by TCP until both a socket and worker thread was freed at the end of some other persistent connection. In some circumstances, the HTTP Server could appear to be locked up (though in fact it was simply waiting for a socket/worker-thread to free up). One workaround to this was to turn off persistent connections. However, doing so implied considerable up-front processing overhead as connections (and in particular, SSL connections) were re-established each time. The value of persistent connections was to eliminate the recurring connection-setup overhead. However, when a large number of clients presented themselves, the value for MaxActiveThreads became the bottleneck. The number of threads could be increased, but at a cost of memory. In summary: Issue Shortage of worker threads (Received requests queued by TCP waiting on next thread available. Server could appear locked up.) WP Benefits of Asynch I/O Support Provided in PQ86769 Workaround Turn off persistence sessions Increase MaxActiveThreads Configure scalable HTTP Server with queue servers Downside to workaround Performance cost of establishing connections time and again. Increased memory usage Configuration complexity as well as performance cost of queuing requests to the queue servers. One of the code enhancement provided by APAR PQ86769 was designed to address this issue. This fix provided a new HTTP Server directive called AsyncSockets. This directive takes one parameter: Yes or No. When AsyncSockets yes is coded, the HTTP Server requires a socket for the duration of each persistent connection, but requires a thread only while processing the request. There can be more active connections than worker threads. The probability of a queued request awaiting a worker thread is greatly diminished. To use this new feature, several conditions must be met:! The APAR fix for PQ86769 must be applied to the system! The HTTP Server must be configured in scalable mode, with or without ApplEnv directives! The AsyncSockets directive must be coded with a value of yes in the httpd.conf file! Persistent connections enabled (with the MaxPersistRequest and PersistTimeout directives) The HTTP Server used for the benchmarking effort met all four conditions

6 WP Benefits of Asynch I/O Support Provided in PQ86769 The IBM HTTP Server for z/os -- configured in "scalable mode" The IBM HTTP Server for z/os is capable of taking advantage of some of the inherent function of the z/os operating system to "scale" to meet demand. Specifically, WLM may be used to monitor the HTTP Server's attainment of specified goals, and start additional copies of the web server to meet increased demand. To accomplish this, the HTTP Server can be configured to run as a queue manager or a queue server. A picture of the relationship between the two looks like this: Queue Servers Queue Manager IBM HTTP Server Request Queue IBM HTTP Server IBM HTTP Server Queue Servers then process the work request Work is received by Queue Manager. Work is queued, then dispatched to Queue Servers based on WLM monitoring of Queue Server capability. IBM HTTP Server Relationship between Queue Manager and Queue Server in a Scalable HTTP Server implementation Both the Queue Manager (QM) and the Queue Server (QS) are HTTP Servers. Parameters on the ICSPARM statement in the JCL start procedure indicate what type of server is being started. For example, a Queue Manager will carry the -SN parameter: //WSCQMGR PROC ICSPARM='-SN WSCSN1 -vv -r /etc/wscweb/wscqmgr.conf', // LEPARM='ENVAR("_CEE_ENVFILE=/etc/WSCWEB/wscqmgr.httpd.envvars")' : Note: For more on configuring the IBM HTTP Server for z/os in scalable mode, see "HTTP Server Planning, Installing, and Using" (SC ). As mentioned earlier, while the HTTP Server in scalable mode provided a way to increase the available pool of worker threads, it came at a performance cost. There is additional code associated with queuing the requests and managing the queue. In a high-throughput environment that can prove to be a bottleneck. Scalable web server -- without Queue Servers As mentioned, the typical configuration for a scalable HTTP Server is to have one Queue Manager and some number of Queue Servers. The Queue Manager queues and routes the work, and the Queue Servers do the work of serving out the traffic. But there's no reason a Queue Server must be used. The HTTP Server Planning, Installing, and Using guide indicates that a scalable mode configuration may consist of between "zero and n" Queue Servers

7 The solution utilized for the benchmark was to configure the HTTP Server in scalable mode, but not use the Queue Servers. This eliminated the code to queue and manage the requests, and allowed the use of PQ86769's new AsyncSockets directive to allow the connection count to exceed the worker thread count. So in picture format, the configuration used was this: Queue Manager WP Benefits of Asynch I/O Support Provided in PQ86769 IBM HTTP Server WebSphere AppServer Request Queue WebSphere HTTP Server Plugin Application Queue Manager with no Queue Servers Note: This does not imply a single point of failure. It is possible to configure multiple HTTP Servers in parallel, all configured as Queue Managers, all running the WebSphere Plugin. A front-end "IP sprayer" is then used to distribute traffic across the multiple HTTP Servers. HTTP Server redundancy is introduced, and session affinity is still maintained. Sysplex Distributor could be used for this purpose; many other solutions exist as well. The objective of the benchmark was to measure the throughput, not provide fault tolerance, so for the benchmark only a single Queue Manager was configured. In a more traditional scalable mode HTTP Server configuration, APPLENV directives in the Queue Manager's httpd.conf file tell the Queue Manager what to route to Queue Servers, and what WLM Application Environment to use: APPLENV /MyApp/* WLMAPPL Where:! /MyApp/* is a URL template used to identify received URLs for queuing to the Queue Servers! WLMAPPL is in this case an example of a WLM Application Environment that is to be used to either start a Queue Server or route a request to an existing Queue Server. When a single Queue Manager is used (as was the case for the benchmark), these APPLENV directives are removed. The Queue Manager then performs the work. In the case of the benchmark, that "work" was to pass requests off to the WebSphere HTTP Server Plugin running inside the HTTP Server's address space. The WebSphere HTTP Server Plugin WebSphere Application Server for z/os (as well as WebSphere for distributed platforms) ships with a piece of code called a "Plugin." That code is intended to run inside the HTTP Server, and provide routing and session affinity capabilities for backend WebSphere servers where J2EE applications are running. Note: It's called a "Plugin" because the HTTP Server has a set of APIs that permit programs to operate inside the HTTP Server itself. On the z/os platform, the HTTP Server runs as a single address space, and the Plugin runs inside that address space. The Plugin is configured into the HTTP Server with three directives in the httpd.conf file: - 5 -

8 WP Benefits of Asynch I/O Support Provided in PQ86769! ServerInit -- this defines the location of the Plugin executable, and provides a single parameter that points to the Plugin's configuration file, called by default plugin-cfg.xml. There will be one ServerInit directive for the Plugin.! Service -- provides a URL template used to identify requests received by the HTTP Server that are to be passed to the Plugin for processing. Between 1 and n Service directives will be used.! ServerTerm -- defines the method used to gracefully bring down the Plugin when a STOP is issued against the HTTP Server. There will be one ServerTerm directive. The schematic for the benchmark configuration therefore looked like this: HTTP Server JCL Start Procedure Defines server as Queue Manager -SN on ICSPARM Points to httpd.conf file -r on ICSPARM IBM HTTP Server WebSphere AppServer WebSphere HTTP Server Plugin Application Plugin plugin-cfg.xml file Defines backend WebSphere Appservers Provides routing rules Provides affinity rules HTTP Server httpd.conf file ServerInit defines Plugin executable code ServerInit points to plugin-cfg.xml Service statements direct requests to Plugin Queue Manager with no Queue Servers The key point of that picture is that no Queue Servers are part of the configuration. A single Queue Manager was used, in which the WebSphere Plugin ran. Requests received by the Queue Manager were passed to the Plugin, which then routed to the backend WebSphere Servers.??? Why bother with a scalable Queue Manager configuration? Why not just configure a vanilla HTTP Server? Because the AsyncSockets feature of PQ86769 required the HTTP Server be in scalable mode. So we configured the HTTP Server in scalable mode to use AsyncSockets, and limited it to a single Queue Manager to avoid the overhead of queuing to the Queue Servers. Key directives in the httpd.conf file There are many directives in the httpd.conf file that are important to allow the HTTP Server to function properly. However, for the purposes of this paper we'll focus on just four: - 6 -

9 WP Benefits of Asynch I/O Support Provided in PQ86769 AsyncSockets yes MaxActiveThreads 500 MaxPersistRequest PersistTimeout 90 minutes This is the new function provided by PQ With this, persistent connections require a thread only while processing a request, not locked in a one-thread-per-connection relationship as before. This is what permitted the maximum number of active threads to be limited (500, see next directive) and yet process thousands of persistent connections. Limited the number of active threads to 500. Prior to PQ86769 and AsyncSockets, this value would need to be set equal to the number of concurrent persistent sessions desired. Each thread consumes additional memory. The benchmark scenario called for 15,000 persistent connections. With PQ86769 and AsyncSockets, this value can be reduced (saving memory) and still support 15,000 persistent connections. That's the value of AsyncSockets. This directive is used to define how many requests may flow on a given persistent connection before the connection is taken down. The value used for the benchmark was very high (10 million), meaning that we desired a persistent connection to be a long-lived thing. Tearing down connections and rebuilding them consumes time and processing energy. Maintaining them is how high throughput is achieved. This directive defines how long the HTTP Server will wait on the client to send a request across a persistent connection before the connection is canceled. The purpose of this directive is to prevent a persistent connection from being held open permanently, even if the client has gone home for the day. For the benchmark the value was set to 90 minutes. The default value is 5 seconds. Our value of 90 minutes was arbitrary: by specifying 90 minutes we effectively told the HTTP Server to hold the persistent connection "forever." This was done so persistent connections weren't dropped between test runs of the benchmark workload. Performance measurements achieved in benchmark The benchmark effort in question consisted of 13 different test scenarios, involving different combinations of approximately 10 different variables. The purpose of this paper is not to cover the full measure of the benchmark results, but rather to focus on three of the tests: Test Description Scalable HTTP Server configured with Queue Servers Non-scalable HTTP Server with WebSphere Plugin Scalable HTTP Server configured with Queue Manager only Purpose Provided traditional scalable HTTP Server configuration performance value. Provided a "baseline" of standard HTTP Server so scalable mode test with no Queue Servers could be compared. Provided comparison of single Queue Manager with AsyncSockets (PQ86769) employed. Note: The test numbers 7, 8 and 13 correspond to the test numbers used for the actual benchmark report. They are offered here simply so those with access to the confidential customer report can relate the values offered in this paper with those offered in the final benchmark report. Test #8 is the "baseline" against which #7 and #13 are compared. Test #13 is the focus of this paper. It was this configuration that proved to be most suitable for the customer's needs

10 WP Benefits of Asynch I/O Support Provided in PQ86769 Here is a table with the results for key metrics captured. Notes follow. Metric Path length estimate (MI) Test 7 Scalable: QM + QS Test 8 Standard HTTP Server ("baseline test") Test 13 Scalable: QM only Path length delta from # MI +7.6% MI +1.7% Notes:! We assume the standard HTTP Server (with Plugin) is the baseline configuration. We see that the number of instructions required to process a request is somewhere in the neighborhood of million instructions.! The pre-pq86769 solution to provide 15K+ persistent connections would have been to configure a scalable HTTP Server with Queue Servers. That is represented by Test #7. We see that doing so implies an increase in approximately 100K instructions per request received, or an increase in code handled of about 7.6%.! With the AsyncSockets function provided with PQ86769, we see that the code path length is roughly comparable to our "baseline." At the "cost" of only 27K more instructions per request, we were able to achieve persistent connections for 15,000+ clients without requiring an excessively high number for MaxActiveThreads. This solution proved to be the most likely one for adoption by the customer for whom the benchmark was conducted. Conclusion The function provided with the fix for APAR PQ86769, in conjunction with the HTTP Server configured in scalable mode with only a Queue Manager, is a viable and attractive solution to the problem of a high number of persistent sessions in a large volume web site. The benchmark validated that with AsyncSockets employed and a Queue Manager-only scalable HTTP Server, results comparable to a standard HTTP Server could be realized, with relief from the worker thread constraints documented earlier. Document Change History Check the date in the footer of the document for the version of the document. January 23, 2006 January 25, 2006 Original document. Added WP number to the document. End of WP100xxx - 8 -

Comparing Some of the HTTP Front-End Options

Comparing Some of the HTTP Front-End Options Don Bagwell IBM Washington Systems Center 301-240-3016 dbagwell@us.ibm.com IBM Americas Advanced Technical Support -- Washington Systems Center Gaithersburg,