B M C S O F T W A R E, I N C. Ross Cochran Principal SW Consultant

Transcription

1 B M C S O F T W A R E, I N C. PATROL FOR WEBSPHERE APPLICATION SERVER ADVANCED BEST PRACTICES Ross Cochran Principal SW Consultant

2 PAT R O L F O R W E B S P H E R E A P P L I C AT I O N S E R V E R ADVANCED BEST PRACTICES INTRO DU CTION This paper will cover the advanced best practices for PATROL for WebSphere Application Server. The paper picks up where the original basic best practices left off. So lets review what we covered in the basic best practices: Monitor your WebSphere Application Server for a few weeks to determine the normal peaks and valleys of resource utilization. Keep an eye on memory usage and verify Garbage Collection is happening, but not happening too much. Verify your servlets are not taking too long to timeout waiting on files. Threads are a finite resource; verify your JAVA programmers are releasing threads in a timely fashion. EJBs are becoming widely accepted and many companies are still learning how to use them. Remember an EJB can timeout much like a servlet waiting on a backend resource. By monitoring EJB and Servlet response times you can anticipate (and hopefully prevent) a frozen web site. Scrutinize your JAVA logs and only log meaningful messages. In theory a web site should be stateless (this is impractical) requiring minimum HTTP sessions in the WAS, however verify your JAVA programmers try to code as stateless as possible web sites. As you can see, the basic best practices approached J2EE management from a stovepipe approach meaning each object involved in a J2EE transaction is examined individually for abnormal behavior. Whereas, this paper will outline how to use PATROL for WebSphere Application Server to manage J2EE transactions from a transaction viewpoint instead of from a stovepipe viewpoint (Note This paper does not discuss how to enable the advanced monitoring required for these advanced best practices). There are three methods for managing a J2EE transaction. 1. You have a general idea of the WebSphere Application having a problem. For example, a customer has called the help desk complaining they have poor response time while trying to order plants. Having been around your WebSphere applications, you know the problem is in your Plants By WebSphere web application. 2. You have no idea where (or if) there is a problem. You need to query all (verses just only the Plants By WebSphere application) your J2EE applications and rank each application by response time. 3. Once you determine where the problem is located (in terms of a J2EE transaction), you then need to determine what is causing the problem. Many of these possible whats are covered in the basic best practices paper. However, this paper will dig further into other possible problem areas such as DB pools, objects, and threads.

3 YOU H AVE A GENE R AL I DEA W HIC H WEBSP HERE AP P LIC ATIO N I S H AVI N G A P ROBLEM The first thing you need is a view of all the J2EE components making up your Plants by WebSphere application. Go to the PATROL for WebSphere Application Server console and pull a Servlet Response Report. This report will show the average response time for all objects making up the application. By using the report shown below, we can determine the following: The Plants By WebSphere J2EE transaction is composed of a Shopping JSP, an Image servlet, and a Shopping Servlet. The Shopping JSP is having the best response time of all with six transactions and all six transactions have sub second response time. The Image servlet is next with an average response time of milliseconds for seventy-seven transactions. The Shopping Servlet is last with an average response time of milliseconds for seven transactions. It is clear now; the Shopping Servlet is the J2EE object taking the most response time. Access the report from the WebSphere icon Environment Name (Node name) Application Performance Plants By WebSphere Servlet Response Report

4 You have determined the Shopping Servlet is the J2EE object taking the most response time. Now you need to know when. Has it been all day, just recently, or maybe early in the morning? Go to the PATROL for WebSphere Application Server console and view the Servlet Response Time parameter for just the Shopping Servlet. For clarity, change the x-axis to a fifteen-minute interval and let the y-axis auto-scale. You notice the large spike (>15,000 milliseconds) before 06:40. Whereas all the other data points are near the trivial response time range. This definitely looks like the problem reported by the customers. Now we need to determine how many transactions were processed around the 06:40 time frame. Access the report from the WebSphere icon Environment Name (Node name) Application Performance Plants By WebSphere Servlet Response Time

5 Let s recap The customers called complaining of poor response time. The servlet response report confirmed the Shopping Servlet was the reason. The Shopping Servlet response time parameter confirmed the response time spike around 06:40. Now we need to determine the transaction rate through the Shopping Servlet around 06:40. Go to the PATROL for WebSphere Application Server console and pull the Servlet Request Rate for just the Shopping Servlet. For clarity, change the x-axis to a fifteen-minute interval and let the y-axis auto-scale. Notice the spike at 06:40. The request rate for the Shopping Servlet went from trivial to twenty per minute. Since there were no transactions prior to 06:40, my guess would be several customers woke up the day before mother s day and all wanted to order flowers at the same time. Since the request rate dropped off considerably after 06:40, this was probably a case of too many people trying at access the web at the same time. Access the report from the WebSphere icon Environment Name (Node name) Application Performance Plants By WebSphere Servlet Request Rate

6 . YOU H AVE NO I DEA W HE R E (O R I F ) T HE RE I S A P RO B LEM In the previous section of this paper, we pursed the response time issue on the premise the problem was in the Plants By WebSphere Application. But what if, you have no idea at all which WebSphere application is causing the response time problem? What do you do? Go to the PATROL for WebSphere Application Server console and pull the Web Application Performance Report. This report will show all the web applications currently loaded and their associated response times. Notice the Pants By WebSphere application has the highest average response time of milliseconds. OK, you have found the worst performing WebSphere application. Now we need to know, what component of the Pants By WebSphere application is causing the overall response time of the application to server? Access the report from the WebSphere icon Environment Name (Node name) App Server - Server Name Web Applications RM Knowledge Modules Commands Web App Performance Report Slowest Response Time

7 What component of the Pants By WebSphere application is causing the overall response time of the application to server? Access the PATROL for WebSphere Application Server console and pull the Servlet Performance Report. This report will show all the components (servlets) currently loaded and their associated response times. Remember this report shows the response times of currently loaded servlets. This means if you are trouble-shooting a response time problem from the past, you may not see the problem in this report. Notice in this report the Image Servlet shows the worst overall response time. However, the Shopping Servlet has six of the top ten slots for poor response time. In other words, the Image Servlet is the worst, but the Shopping Servlet has the majority of poor poorly performing URIs. It is worth pursing which parts of Shopping Servlet are having response time problems. Access the report from the WebSphere icon Environment Name (Node name) App Server - Server Name Web Applications RM Knowledge Modules Commands Servlet Performance Report Slowest Response Time

8 Now we need to know, what parts of the Shopping Servlet of the Pants By WebSphere application is causing the high response times. There are two method calls for the Shopping Servlet: 1. Do Get (1 st graph) 2. Do Post (2 nd graph) The simple question becomes Which method call is causing the problem? To determine this, open each method response time parameter and compare the two numbers. Below, the Do Get in on the left and the Do Post on the right. It should be clear the Shopping Servlet response time problem was because of the 17,000-millisecond response time in the Do Get method call. Access the report from the WebSphere icon Environment Name (Node name) App Server - Server Name Web Applications Plants By WebSphere Shopping Servlet Do Get or Do Post Method RT

9 WHAT IS C AU SING T H E P ROBLEM DB POOL S So far we know the Plants By WebSphere Shopping Servlet Do Get method call is having a response time problem. What is causing it? A classical source of response time problems is accessing backend databases. The JVM has pools of database connections for J2EE applications. There are two critical time parameters associated with accessing the database pools: 1. Time waiting to access the pool (1 st graph) 2. Once accessed, the duration of the database access (2 nd graph) From the charts below, we notice the Shopping Servlet spent zero time waiting for a connection to the database pool. However, it did spend two milliseconds accessing the Cloudscape database. Since (from the previous page) the overall servlet response time was 17,000 milliseconds, we can safely rule out database access as a source of the response time problem. Access the report from the WebSphere icon Environment Name (Node name) App Server - Server Name DB Pools Samples Cloudscape JDBC Provider (XA) Plants By WebSphere Data Source Average Wait Time or Average JDBC Time

10 We have determined the problem is not in the database?. What about the JVM? Is it allocating and freeing objects in memory correctly? As a JAVA application needs an object the JVM will allocate the memory needed to keep the object resident in memory. However, JAVA programmers have to be careful and be sure to release objects after use. If objects are not released after use by a JAVA application, then the corresponding JVM memory is also not released. This can cause the JVM to run out of memory. Let s review the number of objects allocated and the number of objects freed. If these numbers are close, then your JAVA application programmers are doing a good job of using (allocating and freeing) objects. Review the chart below for the objects allocated (blue line) and the objects freed (red line). Notice the parameters are close, so objects are being managed properly. Access the report from the WebSphere icon Environment Name (Node name) App Server - Server Name DB Pools Server Resources Objects Freed and Objects Allocated

11 We have determined the problem is not because of objects not being released. What about the JAVA applications waiting for a lock from the JVM? The JVM issues a lock (on the behave of a thread) for each object. When the thread is finished with the lock, it is released by the JVM. The problem that arises is a thread may have to wait for a lock (because another thread is not releasing it s locks). So we need to review the number of times a thread had to wait for a lock. From the graph below, you will notice the high (relative to other times) the 40,000 times the Plants By WebSphere thread had to wait for a lock from the JVM. This seems to correspond perfectly with our spike in the Shopping Servlet response time. This looks like the root cause of the customer s reported problem. The solution could be to review the Plants By WebSphere JAVA application and verify the JAVA programmers are not holding threads open too long. Access the report from the WebSphere icon Environment Name (Node name) App Server - Server Name DB Pools Server Resources Lock Waits

12 SUMM ARY We have used several features of the advanced functions of PATROL for WebSphere Application server to solve the reported problem. Let s recap our activities: For the I have an idea where the problem is approach: Pulled a servlet response report This showed us the Shopping Servlet was having a problem. Reviewed the Shopping Servlet response time over time This showed us the exact time of the response time spike. Examined the Shopping Servlet transaction rates We saw several simultaneous requests for the servlet during the response time spike. For the I have no idea where the problem is approach: Pulled a web application performance report This showed several URI request in the Shopping Servlet having a response time problem. Looked at the servlet performance report This revealed more Shopping Servlets requests had response time problems than any others servlets. Examined the method level response time The Do Get method call was causing the response time problem. Root cause analysis Checked the database connect and wait time The Shopping Servlet was not waiting or abusing database access Reviewed the JVM object allocations JVM objects were being allocated and freed properly Looked at the JVM thread locks This was the root cause of the problem The threads had to wait over 40,000 times for the JVM to allow a lock.