Known Features of WebSphere Application Server on z/os

Transcription

1 WebSphere Application Server for z/os V6.1 Hidden Gems: Little Known Features of WebSphere Application Server on z/os This document can be found on the web at: Search for document number WP under the category of "White Papers" Version Date: November 21, 2007 See "Document change history" on page 17 for a description of the changes in this version of the document IBM Software Group Application and Integration Middleware Software Written by David Follis IBM Poughkeepsie follis@us.ibm.com Edited by Wayne O'Brien IBM Poughkeepsie

2 Many thanks go to...rob Alderman, Don Bagwell, Mike Ginnick, Barry Gower, John Hutchinson, Mike Loos, Colette Manoni, and Tim Spewak. Introduction... 3 Sticking with the script... 3 To GMT or not to GMT using ras_time_local... 3 Keeping track when your applications don't... 4 Putting trace output in its place... 4 Managing your message output... 5 What s going on? The DISPLAY command... 7 DISPLAY,SERVERS... 7 DISPLAY,SERVANTS... 7 DISPLAY,LISTENERS and DISPLAY,CONNECTIONS... 8 DISPLAY,TRACE... 8 DISPLAY,JVMHEAP... 8 DISPLAY,WORK... 8 DISPLAY,WORK,CLINFO... 9 DISPLAY,WORK,MDBSTATS... 9 DISPLAY,ERRLOG DISPLAY,MODE Summary of DISPLAY sub-options Spreading the work around stateful session placement Please stand by pause/resume listeners Are your enclaves propagating? Should you care? Handling large IIOP messages in 64-bit mode Servant survivor staying up during a timeout flurry Throttle up scaling up to a lot of connected clients Document change history

3 Introduction WP Hidden Gems: Lesser Known Features of WebSphere Application Server on z/os During the development of WebSphere Application Server for z/os, we sometimes add small, useful functions to the product without much fanfare or publicity. In this IBM White Paper, we look more closely at some of these under-appreciated functions and try to give them the attention they deserve. We cover the basics of each function, why or when you might want to use it, and the details of how. Sticking with the script Before we get into all that, you probably want to download the JYTHON script that accompanies this document. Written by Mike Loos, the script does several things that you might find handy, and can serve as a base on which to build other scripts as needed. The script performs the following steps: 1. Reads your configuration and removes any settings of the variable, ras_log_logstreamname. (If you did not set-up the logstream for the error log, WebSphere Application Server generates errors trying to connect to a specified stream name; the script deletes any references you might have set up by accident.) 2. Creates several variables for the new options described in this document. 3. Saves your configuration and does a node-sync to have the changes take effect. Before running the script, read this document and then edit the script to remove any changes that you do not want. Also, set the correct time zone, which is located near the top of the script file. Thanks for providing the script, Mike. To GMT or not to GMT using ras_time_local Unless you happen to live in an area in which Greenwich Mean Time (GMT) is the local time, debugging a problem with a WebSphere server means having to cope with message time stamps that vary between GMT and local time. In the output from a WebSphere server, for example, the WTOs issued by the server show the local time. In the server error log, however, or if you have tracing turned on, those same messages are time-stamped in GMT. This headache isn't yours alone; if you have ever sent server output to IBM Customer Support for diagnosis, they probably had to ask you (or figure out) your GMT offset. Fortunately, this is all easily resolved. From the WebSphere administrative console, navigate to Environment > WebSphere Variables, and set the variable ras_time_local to 1. This setting causes the trace and error log messages to be written in local time. Now, those WTO time stamps will line up nicely with the trace and error log time stamps. To perform this change consistently across servers, set the variable at the cell level. To select local time for the daemon, prefix the variable name with DAEMON_, that is, specify the variable DAEMON_ras_time_local=1. For more information about setting the time zone, see the WebSphere Information Center, starting with this article: ries/ae/trun_svr_timezones.html

4 Time stamps in garbage collector output (verbosegc) remain in GMT, unless you set the time zone for Java. You can do so either at a system level or for just the WebSphere Application Server processes, as described in the WebSphere Information Center (the previous link). For the valid values for time zone, see the following link: m.ibm.websphere.zseries.doc/info/zseries/ae/rrun_svr_timezones.html. Keeping track when your applications don't Monitoring your applications is an important part of managing your application server. One type of situation to monitor is when an application returns an exception to its caller. That is, a client request enters the server and is processed, but something goes wrong either in the server itself or in the application processing resulting in an exception that is marshaled and returned to the client. Unless the application logs such failures (and failures detected by the application server itself might never be seen by the application), there isn t a good way to know if this is happening. That is, unless you use the little known variable protocol_bboc_log_return_exception. Set this variable to 1 to have the application server write message BBOO0169W to the error log whenever it returns an exception to a client. Another potential situation worth monitoring is when a client fails to wait for a request to complete. The application server detects this situation if it tries to respond to a client, but the client has closed the connection. To monitor for this type of event, set the variable protocol_bboc_log_response_failure to 1. Doing so causes the server to write message BBOO0168W to the error log. This message indicates whether a success or failure response was being sent, and includes information about the client that issued the request (such as host and port). To perform this change consistently across servers, define these variables at the cell level. Putting trace output in its place WebSphere Application Server provides many options to help you control tracing. You can, for example, trace events by severity or importance (such as "errors only" or "informational"), or specify a particular area of the product to trace ("communications," "security," and so on). To determine what tracing is active, use the DISPLAY,TRACE command (more on this useful command later). To adjust the active tracing, you can either use the WebSphere administrative console or enter the MVS MODIFY command with any of the following options: TRACEJAVA, TRACEBASIC, TRACEDETAIL, TRACESPECIFIC, TRACEALL, TRACEINIT, or TRACENONE. You also have these options for controlling where trace output should be kept: BUFFER, which you should probably always use, is an option that directs WebSphere Application Server to allocate internal buffers to hold the trace data. You can control the number and size of these buffers. When one buffer fills, WebSphere Application Server moves on to the next buffer, and so on, until it runs out of buffers. It then wraps back to the first buffer and overwrites it. (You can save the contents of the full buffers by having it sent to a CTRACE external writer.) Writing exception tracing to internal buffers is a good idea. Doing so probably will not affect performance because exceptions should not occur frequently and no I/O is used in writing the data. Further, if a serious problem occurs, IBM Customer Support can check the buffers in an SVCDUMP and possibly find debugging information in the traces

5 SYSPRINT is an option that causes tracing to be formatted and written to the job output SYSPRINT DD card (stdout), making the output easier to find and read. Depending on how much output is produced, however, this option can consume a lot of spool space and affect performance through I/O. TRCFILE is an option that allows you to keep the server trace output separate from the application output, for improved readability. You do this by setting trace output to TRCFILE, and defining a TRCFILE DD card in the JCL for the controller, servant, and adjunct. Otherwise, if your applications write data to standard out (using the Java System.out.println), and you have WebSphere tracing turned on and set to SYSPRINT, the WebSphere trace output is mixed with the application output, which can be difficult to read. Of course, TRCFILE has the same affect on spool usage and performance as SYSPRINT. You can change the SYSPRINT and TRCFILE settings dynamically, as follows: To turn SYSPRINT tracing on and off, enter the command: MODIFY <server>,tracetosysprint=yes or =NO. To turn TRCFILE tracing on and off, enter the command: MODIFY <server>, TRACETOTRCFILE=YES or =NO. Note that tracing to the internal buffers continues, regardless of whether you send trace output to SYSPRINT or TRCFILE. You can specify multiple options to select more than one destination for the output. During normal use, for example, you might have the application server send trace output to BUFFER. Whenever a problem occurs, you can switch tracing to SYSPRINT to make the trace output easier to find and read in those situations. Suppose, for example, that a situation arises in which IBM Customer Support needs to collect trace output to debug a problem. Here, you can enter the MVS MODIFY command to select the appropriate amount of tracing, such as TRACEDETAIL=(3,4) and specify TRACETOSYSPRINT=YES to direct tracing to SYSPRINT. Then, you can recreate the problem or do whatever you need to do to produce the trace output for IBM Customer Support. To stop tracing to SYSPRINT and resume your original tracing setup, enter these commands: MODIFY <server>, TRACETOSYSPRINT=NO MODIFY <server>, TRACEINIT Managing your message output Whenever a WebSphere Application Server process starts, it issues many write-to-operator (WTO) messages to indicate which configuration variables and Java properties are in effect. Because these settings control much of the behavior of WebSphere Application Server, the messages can be useful to you in verifying which options are in use, and to IBM Customer Support, for help in debugging a problem. But what, if anything, should you do with all of these messages? Usually, you need the messages to appear in the job output only for easy viewing or for capturing and sending to IBM Customer Support along with the other server output, such as trace data. However, because these messages are WTOs, the system also writes them to the hardcopy log (SYSLOG). Usually, this is not a problem, but if you ever need to find something in SYSLOG, you might have to look through hundreds of configuration messages, spewed out from every controller, servant, and adjunct. We heard enough complaints about this congestion that we did something about it in WebSphere Application Server, Version 5: We added an option to allow you to convert the WTOs to text written to a DD card. You supply the name of the DD card; we open it and write the messages there

6 To configure this capability, you set two variables: ras_default_msg_dd ras_hardcopy_msg_dd. Have these variables refer to separate DD cards with names you can define in the JCL for the controller, servant, and (if used) the adjunct. Why the need to define two DDs, you ask? Internally, WebSphere assigns routing and descriptor codes to each WTO it issues. WebSphere defines that some messages should go to the hardcopy log and other messages are to take the "default" for routing (which, as it happens, is also to go to the hardcopy log). Perhaps confusingly, this routing scheme yields two different sets of messages: BBOM0001I messages issued during initialization, which show environment variables and their settings, use the "default" routing Messages issued from Java that become WTOs use the hardcopy routing, such as the BBOJ0077 messages that report properties set at initialization. For details on message routing, see this link in the WebSphere Information Center: here.zseries.doc/info/zseries/ae/rtrb_msgrouting.html. In addition, your server can build up a lot of output over time if you have: Some tracing going to SYSPRINT Application code writing println( ) s to stdout Errors logged in the job output and messages turn up there regularly. Eventually, all of this output can consume enough spool space to be a problem. You might even have to recycle the server just to free up spool space. To keep your output under manageable limits, you might want to have JES2 spin the output (remove it from the server), so that you can eliminate it later through purging, printing, or some other means. Fortunately, JES2 provides an easy way to do this: Specify the SEGMENT keyword on the server's DD card. Here, JES2 spins the output whenever the output reaches the number of pages you specified as the SEGMENT size. The next question is, how big is a page? A page is defined by the presence of carriage control character 1 in column 1 of the output. You create this character by writing a form-feed (/f) to stdout. Why not just enter a 1 in column 1? Because z/os Language Environment assumes that if you write 1,' you mean '1,' not a form-feed, so it moves the '1' to column 2 for you. If you enter a formfeed (/f), however, Language Environment places a 1 in column 1. To have pages of a consistent size, you should write them at regular intervals (assuming that output is constant). WebSphere Application Server provides two options for setting intervals for writing form-feeds for output streams one for stdout and one for stderr. The options are called ras_stdout_ff_interval and ras_stderr_ff_interval; specify these values in minutes. (Because the SYSOUT DD card used by stderr is "special," you need to do extra setup to configure that option.) Suppose, for example, that you specify a SEGMENT size of '24' and specify that WebSphere Application Server is to write form-feeds every 60 minutes. Here, JES2 spins the output from the server once per day (regardless of how much output exists). For details, see the following article in the WebSphere Information Center: here.zseries.doc/info/zseries/ae/urun_rproperty_custproperties.html. For information about setting up message routing, see IBM Techdoc TD

7 What s going on? The DISPLAY command In its basic form, the MODIFY DISPLAY command tells you name of the application server and its code level: F <server>,display. To save keystrokes, you can enter the MODIFY command as F. Automation programs might use the MODIFY DISPLAY command simply to generate a known response to validate that the server is present and is at least a little responsive. The command does not drive the full request path, but it does make use some of the same resources used in the course of handling requests. But that's really just scratching the surface of what you can do with DISPLAY. This useful command includes quite a lot of interesting sub-options to help you monitor server processing at your installation. To see them all, enter: F <server>,display,help where <server> is the name of the controller procedure. (As a rule, you can see more information about any WebSphere Application Server MODIFY command or sub-command if you follow the command with,help.) To give DISPLAY the attention it deserves, let's look at some of the more interesting sub-options that are available for this command (a summary table appears at the end of this section). DISPLAY,SERVERS The SERVERS option, entered as F <server>,display,servers, reads the table used by the daemon for routing Internet Inter-ORB Protocol (IIOP) work requests among servers in this cell. For each server, this command reports: Cluster name Server name and code level Name of the MVS system on which the server is running. The DISPLAY,SERVERS command does not actually touch every server, but it does identify which servers WebSphere thinks are available and the routing information that WebSphere uses. In addition, even though this command examines information owned by the daemon, you can enter it for any controller or daemon in the cell to receive similar information. DISPLAY,SERVANTS The SERVANTS option, entered as F <server>,display,servants, simply lists the address space identifiers (ASIDs) of the servants owned by the specified controller. This command might make it easier for you to determine which servants are associated with a particular controller, as an alternative to displaying jobs under SDSF (which can be difficult, depending on your installation's job naming conventions). True, each servant reports its server name during start-up, but the DISPLAY,SERVANTS command gives you a way to see what's going on quickly. It can also be a quick way for automation to detect how many servants are associated with a particular controller, if you allow z/os Workload Management (WLM) to dynamically increase and decrease the number of servants

8 DISPLAY,LISTENERS and DISPLAY,CONNECTIONS These commands, which are new for Version 6.1, do the following: DISPLAY,LISTENERS reports on the listener ports in use, including the names of the channels with which the ports are associated. DISPLAY,CONNECTIONS reports on the clients connected to the server. The command includes sub-options to allow you to control how much information is displayed. For example, you can display just a count of the connections, or list them all. Or, you can restrict the display to a single connection type or list them all. These commands replace DISPLAY,SESSIONS and its sub-options from previous releases. Why did we make this change, you ask? After we restructured WebSphere in V6.1 to use the Channel Framework to manage connections, it became very complex to continue to present the information provided by DISPLAY,SESSIONS in the same way. Thus, in Version 6.1, the old DISPLAY,SESSIONS command now generates a response that it is obsolete, while the new commands provide the same data, although structured somewhat differently to reflect the change in the internal structure of the code. DISPLAY,TRACE The TRACE option, entered as F <server>,display,trace, provides information about what tracing options are turned on. You can display tracing options for the controller (the default) or for the servants by adding,srs to the command. To see tracing for both the controller and the servants, add,all. Displaying trace options is usually a good idea after you have made a change that IBM Customer Support has requested, so that you can verify that the settings in effect are the ones you want. DISPLAY,TRACE provides information about the tracing used in just the native (non-java) code. To see what tracing is used in the Java code, enter the command, DISPLAY,TRACE,JAVA. Again, you can expand this information by appending the suffix,srs or,all to the command. DISPLAY,JVMHEAP The Java Virtual Machine (JVM) provides an API, called QueryGCStatus, to let you check the size of the current Java Heap, how much of it is allocated, and how many times garbage collection has run. You can use the command DISPLAY,JVMHEAP to have WebSphere Application Server call the API for this information. While not even close to a substitute for VerboseGC data, an occasional look at the JVMHEAP information can provide you with some insight into how the Java Heap is managing to handle the demands created by the requests being run. As with DISPLAY,TRACE, you can append the,srs or,all suffix to the command to control which processes report this information. For information about the QueryGCStatus API, see the following link: DISPLAY,WORK This command, which is intended for automation, shows how much work the server is doing. The basic command and some of its sub-options, which act as filters (or provide extra information about how work is spread across the servants), reports the number of requests that have been handled by the server and how many requests are currently in the server

9 Understand that the server is not stopped while this data is collected. If you ask each servant to report how many requests it has processed, each servant will, in turn, report its information. By the time the last servant reports, however, the numbers reported for the first servant might be obsolete if the server is very busy. The DISPLAY,WORK command also reports the last time it was issued by anybody (automation, you, or another operator). It also tells you, for some fields, the change (or delta) in that field since DISPLAY,WORK was last issued. The sub-options for this command do not affect how the deltas are calculated. So, if you enter DISPLAY,WORK,ALL, followed by DISPLAY,WORK,SUMMARY, deltas are calculated on the second command, based on the numbers gathered to produce the results for the first command. Another oddity with the DISPLAY,WORK command is the somewhat unfortunate naming scheme used for the filters. You can select from EJB,SERVLET, or MDB, as follows: EJB reports on requests that arrive at the server through IIOP SERVLET reports on requests that arrive through HTTP, regardless of whether they are requests for servlets, jsps,.jpegs, or something else. MDB reports on the messages received by the MQ listeners in the controller. These counts reflect only how requests arrived at the server. If an HTTP request drives a servlet that calls an EJB, for example, only the count for SERVLET is incremented. DISPLAY,WORK,CLINFO This command, entered as F <server>,display,work,clinfo, relates to the XML file that is used to help classify requests with WLM. It displays one line of text for each node in the classification tree for IIOP and HTTP requests. The description tag from the XML file indicates which entry is being displayed. In the command output, each line indicates the number of times a node: Has been examined while trying to classify a request Has matched the request Has been used (it might match, but not be used if a sub-node is a better match). For each node that was used, this command also reports the cost or number of comparisons required to reach the node. This information can help you understand how your classification tree is being used and help you sort the more commonly used nodes towards the top of the tree to reduce the time spent searching the tree for a match. The last line of text for each section reports the total number of requests of this type that were classified and the total cost (in terms of nodes examined) to do so. For information about classifying workloads for WLM, see this link in the WebSphere Information Center: here.zseries.doc/info/zseries/ae/tjta_ztclass.html. DISPLAY,WORK,MDBSTATS The MDB throttle controls the amount of work that the server handles at a given time for a message-driven bean. If the controller is receiving messages and a throttle exists to prevent the server from absorbing too many messages from the queues, the server stops and begins listening to the queue to avoid being overrun with messages. If you have monitoring active (through the command MODIFY <server>,mdbstats=enable), you can see how the MDB throttle is behaving by entering the command MODIFY <server>,display,work,mdbstats

10 For information about monitoring MDB throttling, see this link in the WebSphere Information Center: here.zseries.doc/info/zseries/ae/rprf_tipsmdb.html DISPLAY,ERRLOG When you send a dump to IBM Customer Support, they often need to see your error log, too. If you neglect to include your error log with the dump, they will probably ask you to send it, which can delay debugging a problem. To avoid this delay in Version 6.1, we added a small cache that records the last 10 items written to the error log from any WebSphere Application Server process. We anchored the cache so that IBM Customer Support could find it easily in an SVCDUMP. And, because it seemed useful to be able to quickly and easily look at the cache and see what error messages had been logged recently without taking a dump or accessing the output of the server (or the logstream, if the error log is directed there), we also provided the DISPLAY,ERRLOG command. DISPLAY,MODE In Version 6.1 (PTF 4), we allow any server to run in either 31-bit or 64-bit mode. We added the command DISPLAY,MODE to give you a convenient way to check the server's bit mode (which is also indicated in the message issued by the server during initialization). Remember that the controller and servants for a server all execute in the same bit mode. Summary of DISPLAY sub-options Here's a summary of the DISPLAY sub-options that we covered in this section: Command DISPLAY DISPLAY,SERVERS DISPLAY,SERVANTS DISPLAY,LISTENERS Purpose Generates a response indicating that the server is alive. Displays a list of all servers in this cell in this sysplex. Displays the ASIDs of the servants owned by this controller. Displays information about each configured listener. DISPLAY,CONNECTIONS Displays information about each network connection to the server. DISPLAY,TRACE DISPLAY,TRACE,JAVA DISPLAY,JVMHEAP DISPLAY,WORK Displays currently active native tracing options. Displays current active Java tracing options. Displays basic information about the JVM heap. Displays counters of the number of requests processed by the server. DISPLAY,WORK,CLINFO Displays information about how requests are classified. DISPLAY,WORK,MDBSTATS Displays information about MDB throttling. DISPLAY,ERRLOG Displays the last 10 entries to the error log written by this server. DISPLAY,MODE Displays the bit mode the server is using (31 or 64). More specifics are provided in the WebSphere Information Center, beginning with this article:

11 Spreading the work around stateful session placement When an application server processes a request, an affinity is sometimes created for the servant region in which the request was processed. As a result, the server must continue to route the client's requests to the same servant region, to ensure that subsequent requests access the same session data. For HTTP requests, for example, an affinity often results for a servant region that contains the HttpSession data, such as the shopping cart. If the session data resides in memory, the server must route subsequent requests from the client to the same servant region, to ensure that the same shopping cart is accessed. Now, consider a different scenario that uses session data. Imagine an application for which session data is created every morning as part of a "logging on" process that your end-users perform. From 8 a.m. until 9 p.m., the "logging on" activity is very low, consisting of just a few requests at a time, as users begin to arrive at work. Although you defined the server (the target of this activity) to start three servant regions, z/os Workload Management (WLM) does not need more than a single servant to meet the goals for these requests, and so it only uses one. Every log-on request, therefore, generates an affinity for its client to this same servant region. Later in the day, people really get to work. Requests enter the server at a tremendous rate. Due to the server affinity created earlier in the day, WLM routes all of the requests to the same servant region; that's where their session data is located. See Figure 1. Figure 1. Server affinity causes requests to be routed to the same servant region Now, with the vastly increased workload, the servant region falls behind, leaving WLM unable to process the requests fast enough to meet response time goals. Here, server affinity gives WLM no choice; WLM cannot route the requests to the other two servers, which stand idle. How can we avoid this bottleneck situation? Earlier in the day, we should have spread out the initial requests among all three servants (even though we did not need to do so at the time) to balance the server affinities

12 WLM will do this for you just specify the option wlm_stateful_session_placement_on=1. With this setting enabled, WLM assumes that any request without an affinity will create one. Based on the current count of affinities for each servant region, WLM places each new request where it is most likely to even out the number of affinities among the servant regions. (This might look like a round-robin algorithm at times, but it's not.) Returning to our example, assume that we specify wlm_stateful_session_placement_on=1 to cause the few early morning requests to be divided evenly among the three servants. Later in the day, when the workload increases, WLM continues to spread the requests evenly among the three servants. Be careful, however, when using this option with multiple WLM service classes. When WLM has the option of placing a request wherever it wants (that is, without affinity), it places the request in a servant region that is bound to the service class associated with the request. Thus, when WLM is balancing affinities, it considers only the affinities of servant regions that are bound to the service class of the request. In our example, if one of the servant regions is bound to Service Class A, and a request arrives without affinity and associated with Service Class B, WLM places the request in one of the two servant regions belonging to Service Class B, ignoring the affinity count of the first servant region. See Figure 2. Figure 2. WLM ignores a servant region that is not bound to the service class of the requests This happens when the very first request placed by WLM is associated with a different service class than that of the other requests. That first request is a dummy request submitted by WebSphere Application Server to cause WLM to start the first servant region. If that request is associated with a different service class than the rest of the workload, this action binds one servant region to a different service class and messes up the affinity balancing. Eventually, when no more work arrives for that service class, WLM might associate it with the other service class, but that process can take a while. This situation can make the affinity balancing look as though it is not working correctly when it actually is. To see how WLM is managing the servant regions, you can use the WLMQUE tool provided by WLM development at this link:

13 Please stand by pause/resume listeners Suppose you want an application server to stop accepting work for a while, but you don t want to take the controller and its servants down, only to have to start them back up again later. Imagine, for example, that you need to apply maintenance to another subsystem, or you are having a temporary problem and you want work to stop coming to this server. Or, perhaps, you want to have a hot-standby server ready to pick up work whenever another server in the cluster gets overloaded or has problems. Instead of stopping the application server in these situations, you can use the MVS MODIFY command to pause the listeners for the server. With this command, which you enter as F <server>, PAUSELISTENERS, you make the following things happen: WebSphere Application Server shuts down the HTTP listener ports for the server. This action prevents new HTTP connections from entering the server; any upstream routers sensitive to the state of the port will stop routing work here. WebSphere also closes any persistent HTTP sessions that are waiting for new requests to arrive. The server contacts the WebSphere Application Server daemon and indicates that IIOP locate requests should no longer be forwarded to this server. The IIOP ports remain open, however, for the CORBA Object Transaction Service (OTS) to use to resolve any outstanding transactions. For MQ, WebSphere Application Server stops listening for messages. Any work already in-flight, however, continues, and ports used to communicate with the node agent remain open. Thus, any updates done through the WebSphere administrative console are still processed by the paused server. In short, the server remains up and available (to the administrator), and any work in-flight is allowed to continue. New clients, however, cannot reach the server or are, at least, encouraged to go elsewhere. To return the server to normal processing, enter the command F <server>,resumelisteners. Everything fires back up again: the HTTP listener re-opens; the daemon starts to route IIOP requests to this server again; the MQ listeners start looking for messages again. You can pause and resume a server as often as you like. Are your enclaves propagating? Should you care? According to the z/os Workload Management (WLM) documentation, an enclave is "a transaction that can span multiple dispatchable units (SRBs and tasks) in one or more address spaces and is reported on and managed as a unit." Based on your WLM policy, each enclave is associated with a service class and a report class, which allows WLM to manage and report on the work done by the request. For each new request that enters the application server, WebSphere does the following processing: 1. Works with WLM to classify the request and create an enclave associated with the appropriate service and report class. 2. Places the request on the WLM queue and a servant picks up the request. 3. Associates the enclave for the request with the dispatch thread and runs the request

14 But what happens if the request calls an EJB that resides in another application server? Here, WebSphere creates an IIOP connection for that server and sends the request there to be executed. Usually, the other application server performs the same processing and creates a WLM enclave in which the request can run. However, it s really part of the same request, so it shouldn't it run under the same enclave? Well, maybe. If the other server is located on the same z/os image, WebSphere propagates it to the other server (by means of an enclave token), so that the server does not create a second enclave for the same transaction. Thus, WLM can manage and report on the work done in the other server as part of the original request. This design fits very well with the official definition of an enclave; remember that part about "...can span multiple dispatchable units in one or more address spaces" -? This means that the SMF records for the enclave will report the CPU usage for both servers. SMF covers the entire transaction, even if more than one server was involved. In theory, this approach should make accounting and chargeback easier for you, because you do not need to correlate records from different servers. Everything is reported for the whole transaction. And that s true, most of the time. Sometimes, however, you might not want the enclave to be propagated to another server on the same system. This might be true when both servers are using different WLM service classes for their work and the second server has a limited number of servant regions. Remember that WLM groups requests having an enclave associated with the same service class into the same servant region (or regions). Suppose, in our example, that the second application server is defined to have only one servant region. Further, suppose that the service class of requests dispatched in that servant region is different from the service class associated with requests dispatched in the first server. If a request dispatched in the first server sends a request to the second server and the enclave is carried with it, WLM must dispatch the request in a servant region associated with that service class. However, the second server does not have one, and cannot create a new one because it is restricted to a single servant. In this situation, WLM cannot process the request. The request just sits in the queue until a timeout cleans things up. Here, you might want to tell WebSphere Application Server not to bother propagating the enclave. Or, you could increase the maximum number of servants for the second server to allow WLM to start a servant for this request. To stop enclave propagation, set the option protocol_iiop_local_propagate_wlm_enclave to zero (0). Handling large IIOP messages in 64-bit mode If your application server is running in 31-bit mode, we restrict the size of the Internet Inter-ORB Protocol (IIOP) messages that can be handled by the server. This control prevents the server from being overrun by data and running out of storage. In 64-bit mode, however, we remove this restriction because potentially huge amounts of storage are available to your application server. Nevertheless, you might still want to restrict the size of IIOP messages to limit the amount of virtual storage consumed by the server, or to prevent the messages from flooding the JVM heap. Of course, the origin of an IIOP message might influence your decision about limiting its size. Therefore, we provide you with two useful switches for handling these messages:

15 For IIOP messages that arrive over TCP/IP from clients on other systems, you can limit the size of IIOP messages through the configuration variable iiop_max_msg_megsize. For clients (or servers acting as clients) on the same system, you can manage the limit through the configuration variable local_comm_max_msg_megsize. You specify these values in megabytes. For any message that exceeds its allowable size, the server rejects the message and closes the connection. You might want to set local_comm_max_msg_megsize to a larger value to permit large messages to flow between servers on this system, but use iiop_max_msg_megsize to restrict the size of messages that arrive from elsewhere in the network. The default value for both of these variables is zero, which means that there is no limit---at least until you run out of 64-bit storage! Servant survivor staying up during a timeout flurry One advantage of running an application server on z/os is the option of ending a runaway or hanging request by ending the servant region in which the request is dispatched. If you are configured to run more than one servant, and a servant encounters a problem, WebSphere Application Server for z/os can abend the problem servant. This action allows the application server to continue processing work without interruption by routing requests to other servants. Eventually, WLM automatically starts a new servant to replace the one that ended. This capability is very useful unless the problem is one that affects all of your servants. For example, suppose that you have several applications installed in the server and one of them has a bug that causes it to loop. If all the servants receive a request for that application at the same time, then a dispatch thread in every servant will be stuck in a loop. This situation causes WebSphere Application Server for z/os to abend all of the servants at once, leaving you without servants to process requests for the other applications. Of course, you should fix the application bug, to prevent the problem from reoccurring. However, that might take awhile (some problems are complicated). What can you do in the meantime to minimize the damage? You have some choices. If you have another server in the same cluster, you can set variable control_region_dreg_on_no_srs to 1. This action causes the server to shut down its listeners when it has no other servant regions to which it must dispatch requests. The listeners re-open when the minimum number of servant regions is ready for work. But what if all of the servers in the cluster lose their servant regions at the same time? Or, you are not running in a clustered environment? Here, you should consider setting the variable control_region_timeout_save_last_servant to 1. This variable takes effect only when you have more than one servant active. When the controller is preparing to abend a servant for a timeout, it checks to see whether this is its last servant region. If so, the controller defers the abend. Thereafter, the controller periodically checks on whether any servants exist besides the one to be abended. When the controller eventually finds one, it abends the problem servant. This configuration allows requests for the other application to continue to be handled by a servant that contains a timed-out request until another servant is available to handle requests. But be careful---whatever problem caused the timeout (an endless loop in this case) is still occurring. While the controller waits for another servant to start, the request that is dispatched in the surviving servant is burning cycles in its endless loop. Of course, if your application hangs instead of looping, the situation is easier to live with

16 Throttle up scaling up to a lot of connected clients Towards the end of the development of WebSphere Application Server for z/os Version 6.1, we did some scalability testing, but the results of one test did not get much publicity. The goal of the test was to determine how many HTTP clients could connect to and use a single WebSphere Application Server on z/os server. In previous releases, running in 31-bit mode, the number of clients was constrained by limits in controller storage. Now, with a server running in 64-bit mode, we anticipated a much bigger number. We ran the test. In it, we defined a single controller with four servant regions. The clients ran with a small amount of think-time to prevent them from all sending requests to the server at the same time. Information about the details of the configuration (heap size, and so on) will have to wait for another IBM White Paper, but for now, we will just mention that we managed to have 120,000 HTTP clients concurrently connected and active, using a single WebSphere Application Server on z/os server

17 Document change history Check the date in the footer of the document for the version of the document. November 19, 2007 Initial Version November 21, 2007 Techdoc number WP assigned and updated in document. End of WP