JBoss Enterprise Application Platform: Troubleshooting Guide

Transcription

1 JBoss Enterprise Application Platform: Troubleshooting Guide JBoss Enterprise Application Platform includes one of the leading Java EE certified application server which is used by many fortune companies to run their mission critical applications. This guide introduces some common tools and commands which can be used to obtain diagnosis information and covers some common troubleshooting tips related to applications running on JBoss EAP.This Troubleshooting Guide is intended for people who are responsible for maintaining,administering, or supporting JBoss Enterprise Application Platform (EAP) environments. Contents 1. Common Diagnostic Tools for Troubleshooting Shell Commands for Linux/Unix Users Tools in Java Standard Edition Tools available with JBoss EAP JMX-Console JBoss Operations Network Troubleshooting Systems bottleneck CPU Under utilization High utilization Disk IO OutOfMemory Errors Troubleshooting Application Server Troubleshooting Classloading JBoss AS Thread Pool Database Connection Pooling Clustering JBoss Messaging References Feedback...15 JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 1

2 1. Common Diagnostic Tools for Troubleshooting In this section of the document we will cover some commands and tools which are available as part of OS, JVM and Red Hat Subscription that can be very handy in troubleshooting different problems Shell Commands for Linux/Unix users There are many management and monitoring tools available for Unix/Linux users as executable shell commands. top and vmstat are the two most useful commands that can be used for troubleshooting some system level resource contention and can provide us information that can help us in pinning down the issue. top provides a dynamic real-time view of a running system. It can provide listing of most CPU intensive tasks and provides interactive interface for manipulating processes.for example below is the output of top command Figure 1: top Command output The upper section describes the statistics about the machine, including Tasks, CPU, Memory and Swaps, in real time. The lower section provides information on per-process basis. For example you can see that there is one java process with PID 6707 run by the user somil which is using 13.6 percent of the memory and 0.3 percent of the CPU cycles JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 2

3 Another useful command is vmstat which provides virtual memory stats. It can provide details about the paging activity, block I/O, memory etc which can be very useful in troubleshooting some performance related issues. It is recommended to use vmstat with delays of 5 to 10 seconds. For example below is the output of vmstat command with 10 second delay: Figure 2: vmstat Command output with 10 seconds delay We will see later in the document how we can use these shell commands in troubleshooting issues that we see with Java EE application deployed in JBoss Enterprise Application Platform Tools in Java Standard Edition The Java SE provides comprehensive monitoring and management support. It not only defines the management interfaces for the Java virtual machine, but also provides out-of-the-box remote monitoring and management on the Java platform and of applications that run on it. Much of this functionality is also available in JBoss Operations Network JConsole: The JConsole graphical user interface is a monitoring tool that complies to the Java Management Extensions (JMX) specification. JConsole uses the extensive instrumentation of the Java Virtual Machine (Java VM) to provide information about the performance and resource consumption of applications running on the Java platform JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 3

4 Figure 3: JConsole JConsole can be used to observe information about an application running on the Java platform.some of the core monitoring and management functionalities provided by the JConsole includes Detect low memory Enable or disable GC and class loading verbose tracing Deadlocks detection Control the log level in an application Java VisualVM: It is an intuitive graphical user interface that provides detailed information about Java technology-based applications (Java applications) while they are running on a given Java Virtual Machine. It combines several monitoring, troubleshooting, and profiling utilities into a single tool JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 4

5 Figure 4: Java VisualVM Java VisualVM is useful to Java application developers to troubleshoot applications and to monitor and improve the applications' performance. Java VisualVM can allow developers to generate and analyze heap dumps, track down memory leaks, perform and monitor garbage collection, and perform lightweight memory and CPU profiling Tools available with JBoss EAP JMX-Console JBoss EAP ships with jmx-console which can provide a raw view into the microkernel of the JBoss application server. It lists all registered services (MBeans) that are active in the application server and that can be accessed either through the JMX console itself or programmatically from Java code.it can be used to get a snapshot into how the app server is behaving in terms of services deployed, resource utilization etc JBoss Operations Network JBoss Operations Tool is a single integrated management and monitoring tool for all JBoss Middleware. From the JBoss Operations Network (JBoss ON) console you can inventory resources from the operating system to applications,control and audit your application configurations to standardize deployments and Manage, monitor and tune your applications for improved visibility, performance and availability. JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 5

6 2. Troubleshooting System Bottleneck In this section we will see how we can use some of the tools and commands from previous section to pin down the issues at the System level which may be causing issues with applications deployed in JBoss EAP. 2.1.CPU Under Utilization If your CPU is idle for too long it may not be a good thing. You should be suspicious if you have high CPU idle time along with No unusual input /output activity Independent of increased load Response time degrades too rapidly with increased load If you see any of these cases it may be possible that the application server is waiting for some resources to be released by other processes. You can obtain more information by getting Application Server ThreadDump by either using jmx-console that comes with JBoss EAP or by using jvisulavm that ships with oracle jdk High Utilization Generally its believed that if CPU utilization is high it may be due to some system bottleneck. While in some cases it may be true but its not always the case. For example below is the output from vmstat command bash-4.0$ vmstat procs memory swap io----system-----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st Figure 5: vmstat Command output showing high CPU Utilization In this case CPU is busy (90% ) but it does not necessarily mean that there is a system bottleneck. It may rather indicate that its running in optimal state. One should be concerned if the run queue (r value under procs column) is more that the number of CPUs on the machine. For example below is the out from vmstat command JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 6

7 bash-4.0$ vmstat procs memory swap io----system-----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st Figure 6: vmstat Command output showing CPU constraint In this case the CPU utilization is high (which may or may not be an issue), but at the same time the value of r under the procs column is 5 which exceeds the number of CPUs on the machine which confirms that there is CPU constraint. You can further examine the cpu column to determine where its spending most of its cycles. In this example CPU is spending 60% of its cycles in performing system calls as reflected by sy value under cpu column. This may be because the application may be executing lots of input/output, sockets or timestamp creation. For example there may be a class thats opening socket for each request, using a pooled approach may solve the problem, another example can be a class performing lots of input/output, using buffered approach can solve the problem. In multi CPU machine there might be a case where one of the CPU is experiencing high utilization. It may be caused by if single thread is used to manage some resources. You can check the garbage collection, make sure its configured properly and then you should verify if there is any contention for some resources Disk IO Generally reading and writing to a disk can be slow and may result in frequent bottleneck for enterprise application. In Linux (RHEL/Fedora) you can use iostat command to get input/output statistics. For example below is the output of iostat command JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 7

8 bash-4.0$ iostat -xd sda Linux fc11.i586 (localhost.sjain) 03/28/2011 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda Figure 7: iostat Command output From the Disk IO troubleshooting perspective we will be interested in last two column (for details on what other parameters mean in the output please refer to man page of iostat). svctm provides the average service time ( in millisecond) for I/O requests and %util provides the percentage of CPU time during which I/ O requests were issued to the device (bandwidth utilization for the device). You should pay attention if you find excessive values for these parameters in the output: High service time (svctm) High %util A high value for these two indicates consistently high reads/writes operations. Some possible causes for a Java Enterprise application are: Excessive logging Stateful Session Bean Passivation Poorly configured database cache. If the bottleneck is not caused by the application, then you should consider spreading the file system of the disk on to two or more disks. As an alternative, move the file system to another faster disk/controller or replace the existing disk/controller with a faster one OutOfMemory Errors This section attempts to cover some of the most common reasons why you may see an OutOfMemoryError from your JBoss application server. Interestingly, there are several cases where the JVM may report an OutOfMemoryError even if it is not really out of all of its available memory. Most modern Java virtual machines segments the memory heap into generations. Your virtual machine may complain about lack of memory when it has only exhausted one segment (a specific generation) from its total maximum heap size. Also under some conditions on Linux/Unix systems running out of some Operating System resources may yield an OutOfMemoryError for example the inability for the OS to create more new threads for the JVM. JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 8

9 It is possible to get an OutOfMemoryError when your Java virtual machine really does run out of its maximum heap. There could be few reasons for this, for example you may have a cache configuration that allows more instances to be kept in memory than the JVM really can fit into its heap. Or it may be simply that your JVM has been configured with a maximum heap size too small to run all your application server services. An OOME is also thrown when the permanent memory is exhausted and that is not part of the heap per se. That is a JVM specific area of memory where information on loaded classes is maintained. If you have a mountain of classes (e.g, a lot of EJBs and JSP pages) you can easily exhaust this area. Oftentimes an application will fail to deploy or fail to redeploy. Is that case increase your permanent memory space as follows to avoid OOME. The default with the -server switch is 64 megabytes: set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx512m -XX:MaxPermSize=128m You can set this in the run.bat (Windows) or run.sh (Linux) or run.conf You can use VisualVM or JConsole to get in depth information on classes being loaded and can narrow it to the classes causing memory leaks. 3. Troubleshooting JBoss Application Server 3.1. Troubleshooting AS Classloading By default JBoss Application Server uses a flat class loading model that avoids the need to redundantly include classes in different layers. While deploying your application if you encounter errors like ClassCastException, IllegalAccessErrors, VerifyErrors, it may be because you have applications that cannot share classes because of version conflicts. In that case you need to isolate the classes from other deployments. There are two levels of scoping, isolation from other deployments, and isolation that overrides the loading of JBoss server classes. For more details on how to configure scoped class loading refer to the Classloading and You whitepaper in the Red Hat customer portal 3.2. JBoss AS Thread Model The Java platform is designed to support concurrent programming. Application servers like JBoss uses a pool of worker threads to improve the performance ( as thread creation and destruction can be expensive) and also provides a control on the number of threads in the application. If you run into a situation where server has a poor response time or does not responds or just starts aborting the connection with RuntimeException it may be quite possible that JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 9

10 all your threads in the pool are busy. The error or exception you get is determined by Thread Pool configuration in jboss-service.xml. <mbean code="org.jboss.util.threadpool.basicthreadpool" name="jboss.system:service=threadpool"> <attribute name="name">jboss System Threads</attribute> <attribute name="threadgroupname">system Threads</attribute> <attribute name="keepalivetime">60000</attribute> <attribute name="maximumpoolsize">10</attribute> <attribute name="maximumqueuesize">1000</attribute> <attribute name="blockingmode">run</attribute> </mbean> Figure 8: Example of AS Thread Pool configuration When all the thread in the pool are busy, all the new request are added to the queue and queue size is determined by the QueueSize attribute. When the queue has reached its max, any subsequent request behavior would depend on the value of Blocking Mode attribute. Figure 9:JMX-Console with Thread Pool Screen Using JMX console you can check the size of the queue. If you have a steady (or worse, an increasing) QueueSize on your server, then you should consider raising the MaximumPoolSize pool size attribute.however, simply incrementing the queue size might not be enough to solve your problems analyze at first where your application is slowing down. For example, a very common scenario is that your threads are busy because they are handling slow or stuck JDBC JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 10

11 connections. Here, merely increasing the MaximumPoolSize would only move the bottleneck into another area. In this scenario, you should first tune your queries or increase the JDBC connection pool Database Connection Pooling Connection Pooling: Database connection are quite expensive to create and maintain because of the overhead of establishing a network connection and initializing a database connection session in the back end database. The database's ongoing management of all of its connection session can impose a major limiting factor on the scalability of the application. Valuable database resources such as locks, memory cursor, transaction logs, statement handles and temporary tables all tend to increase based on the number of concurrent connection sessions. Therefore it is very useful to use connection pool when connecting to database. Database trace file can be used for troubleshooting any issues related to database connection. It contains detailed information on the interaction with the database and can help understand what might be going wrong with your application. Refer to respective vendors drivers documentation for examples on trace file configuration. Most drivers use the tracefile property as follows: /config/db/myappdb={ tracefile : ${myapp}/logs/db.txt } Connection Verifiers: Depending on the nature of the application sometimes it may be useful to check validity of the connection before its checked out of connection pool by application ( if connection in the pool is invalid then any database operation performed using that connection will result in error). JBoss Connection Pool configuration allows few options that you can use to check the validity of the connection. For example you can invoke SQL on connection creation and / or pool access. Some database vendors provides exception sorters. Oracle provides an API to ping Database. For more information on various connection pooling parameters available in JBoss refer to DataSource Parameters Connection Leak: While Connection pooling has many advantages if not used properly can result in many issues which may be difficult to troubleshoot. If connections are not returned to the connection pool they will be kept open and the available connection will leak away. At some point threads will either block waiting for connection to be available or will generate exceptions perhaps after few retries. There can be number of reason. For example: Developer forgets to return the connection JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 11

12 Possible deadlocks- there is a resource contention Exception being thrown in the code and connection close code being skipped These conditions can all be hard to find. One solution is to trace who is acquiring and release locks. You can use JBoss Operations Network or log files to get the info. There is an open-source JDBC profiler called P6Spy. It acts as a proxy for your vendors JDBC driver, so it can log SQL execution times for analyzing connection issues. If you are using Hibernate, JPA or EJB3 you can enable the show_sql setting to view actual SQL generated Clustering Clustering allows you to run an application on several parallel servers (a.k.a cluster nodes) while providing a single view to application clients. Load is distributed across different servers, and even if one or more of the servers fails, the application is still accessible via the surviving cluster nodes. Clustering is crucial for scalable enterprise applications, as you can improve performance by adding more nodes to the cluster. Clustering is crucial for highly available enterprise applications, as it is the clustering infrastructure that supports the redundancy needed for high availability. JBoss Application Server uses JGroups and JBoss Cache for providing Clustering capabilities. For more details refer to the Clustering considerations whitepaper in the Red Hat customer portal. Nodes do not form a cluster: Make sure your machine is set up correctly for IP multicast. There are 2 test programs that can be used to detect this: McastReceiverTest and McastSenderTest. Go to the $JBOSS_HOME/server/all/lib directory and start McastReceiverTest, for example: $ java -cp jgroups.jar org.jgroups.tests.mcastreceivertest -mcast_addr port 5555 Then in another window start McastSenderTest: $ java -cp jgroups.jar org.jgroups.tests.mcastsendertest -mcast_addr port 5555 If you want to bind to a specific network interface card (NIC), use -bind_addr , where is the IP address of the NIC to which you want to bind. Use this parameter in both the sender and the receiver. JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 12

13 You should be able to type in the McastSenderTest window and see the output in the McastReceiverTest window. If not, try to use -ttl 32 in the sender. If this still fails, consult a system administrator to help you setup IP multicast correctly, and ask the admin to make sure that multicast will work on the interface you have chosen or, if the machines have multiple interfaces, ask to be told the correct interface. Once you know multicast is working properly on each machine in your cluster, you can repeat the above test to test the network, putting the sender on one machine and the receiver on another. Causes of missing heartbeats in Failure Detection Sometimes a member is suspected by Failure Detection Protocol because a heartbeat ack has not been received for some time T (defined by timeout and max_tries). This can have multiple reasons, e.g. in a cluster of A,B,C,D; C can be suspected if (note that A pings B, B pings C, C pings D and D pings A): B or C are running at 100% CPU for more than T seconds. So even if C sends a heartbeat ack to B, B may not be able to process it because it is at 100% B or C are garbage collecting, same as above. A combination of the 2 cases above. The network loses packets. This usually happens when there is a lot of traffic on the network, and the switch starts dropping packets (usually broadcasts first, then IP multicasts, TCP packets last). B or C are processing a callback. Let's say C received a remote method call over its channel and takes T+1 seconds to process it. During this time, C will not process any other messages, including heartbeats, and therefore B will not receive the heartbeat ack and will suspect C JBoss Messaging Message on logs: AIO wasn't located on this platform. Possible causes are: Linux is not your platform Just ignore the message as NIO will be selected automatically or change the journal type to NIO on jbm-configuration.xml The JBoss Messaging JNI wrapper is not on library.path Solution: Make sure you have libjbmlibaio32.so or libjbmlibaio64.so as provided on the download package. libaio is not installed Make sure you have libaio installed at your Linux distribution. This could be done as yum install libaio1 on Fedora or apt-get install libaio1 on Debian. (Refer to your manual to how to upgrade or install packages) libaio is very old All the testing has been done using libaio and Kernel 2.6. If you have older versions you might need to upgrade or change the journal type JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 13

14 to NIO on jbm-configuration.xml Low response time even though the CPU and IO seems ok Possible causes are: You are using short transactions in your system and you don't have TCPNoDelay set Make sure you have remoting-tcp-nodelay set to true on jbmconfiguration.xml The journal directory is a NFS You shouldn't use the journal over a NFS. Make sure you have direct access to the disk device. You are using Asynchronous IO on Linux and you don't have an ext3 or ext2 file system If using AIO, make sure the journal folder is on an ext2 or ext3 file system. JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 14

15 4. References top, vmstat, iostat : RHEL man pages JConsole: management/jconsole.html VisualVM: visualvm/index.html JMX-Console: JBoss Operations Network: JBoss_Operations_Network/2.4/html-single/Basic_Admin_Guide/index.html Out of Memory Exceptions: OutOfMemoryExceptions JBoss ClassLoading: JBoss_Enterprise_Application_Platform/5/html/Installation_Guide/ migration.apps.html#migration.app.loader 5/html-single/Administration_And_Configuration_Guide/ index.html#id troubleshooting.html 5. Questions/Comments/Issues If you have questions or comments about this whitepaper, please enter them in the Red Hat customer portal for this specific whitepaper: If you have a technical issue following this whitepaper please open a support case: JBoss Enterprise Application Platform : Troubleshooting Guide Somil Jain 15