J2EE DIAGNOSING J2EE PERFORMANCE PROBLEMS THROUGHOUT THE APPLICATION LIFECYCLE

Transcription

1 DIAGNOSING J2EE PERFORMANCE PROBLEMS THROUGHOUT THE APPLICATION LIFECYCLE

2 ABSTRACT Many large-scale, complex enterprise applications are now built and deployed using the J2EE architecture. However, many of these applications suffer from poor performance and scalability because the focus of the development process is on functionality, while performance and scalability are dealt with as an afterthought. This paper will present techniques for delivering high-performance applications to production, managing and measuring the performance of applications, and diagnosing the toughest J2EE problems throughout the application lifecycle. It will examine the various types of performance issues at each stage of the lifecycle and which diagnostic tools and techniques can best resolve them. TABLE OF CONTENTS Abstract 2 Diagnosing Performance Problems with Mercury Diagnostics for J2EE 7 Challenges for J2EE Applications and Performance 3 Performance Evaluation and Diagnostic Needs Throughout the Application Lifecycle 3 J2EE Diagnostics for J2EE for Mercury LoadRunner and Mercury Performance Center 8 Mercury Diagnostics for J2EE for Mercury Business Availability 8 Typical J2EE Application Performance Problems 6 J2EE Deep Diagnostics 9 Mercury Diagnostics Profiler for J2EE 10 Summary 11

3 Challenges for J2EE Applications and Performance Today s enterprises are choosing to build business applications that leverage the power, portability, and rapid development of Java technology. J2EE offers many advantages to developers, but introduces new challenges for the development, performance diagnosis, tuning, deployment, and management of applications. Successful delivery and management of typically complex J2EE-based applications require evaluating performance throughout the entire application lifecycle. An application may perform well in the development and QA environment, but fail to scale or exhibit transient performance problems in production. It is important to understand the impact of the infrastructure in which the application runs and the behavior of the many application components as they interact under load. From the diagnostic perspective, it is important to be able to isolate the problem by tier of the application architecture and by application component, and to have progressive drill-down visibility into J2EE performance problems, the J2EE environment, and into the actual code, with sufficient detail to determine the root cause of the problems. There are additional factors that can increase the difficulties of application delivery and management. The deployment lifecycle for many J2EE applications is compressed, due to increased pressures for quick time-to-market. Boundaries between development, QA, deployment, and production stages and IT groups are blurred. Centralized IT organizations may be managing hundreds of applications, without in-depth J2EE experience. Many applications have not been sufficiently architected for performance and scalability, with thorough consideration of design and usage patterns, and adequate attention to planning and testing performance against well-defined service objectives. J2EE scalability capabilities, while extensive, do not substitute for such efforts. Later in the lifecycle, applications may be pushed into production to meet deadlines, with insufficient validation of performance or scalability and inadequate tools to see into the application s internal behavior. This makes it difficult and expensive to fix problems. Getting to the root of performance problems in the complex, distributed, dynamic J2EE environment is truly a challenge. Performance Evaluation and Diagnostic Needs Throughout the Application Lifecycle During the application lifecycle, there are many IT stakeholders in application performance. They include application architects, developers, load testers and engineers in QA, application support staff, and site operations in production. They share common requirements for end-user application performance, but also have specific roles and needs unique to their environments. In the design and specification phases, a topic outside the scope of this diagnostic paper, performance and scalability issues need to be considered and incorporated, and specific business objectives need to be set. 3

4 In the development and unit-testing phase, application profiling can be useful to validate performance and functionality. Developers should test the performance of not just application components, but also performance of the end-to-end integrated application (as it would be tested by QA and/or used in production) against identified performance criteria. This step allows development teams to find problems earlier in the lifecycle by preparing the application to be load tested. During QA cycles, load testing typically follows integrated functional and regression testing. A complete application, including all interfaces with external systems, should be fully load tested prior to software release. Objectives include scalability and capacity estimation under load that realistically represent expected live usage, along with visibility into the internal performance behavior of the application and actionable data on bottlenecks. This should include transaction breakdown of latencies for each J2EE tier and method, along with additional specific root-cause diagnostic information. Most try using profiling tools for root-cause diagnostics and fail. Traditional profiling tools used at the developer desktop cannot be used in QA during load testing for performance diagnosis, due to the high overhead they impose. What is needed is a profiling technology that can work in a development environment to find and fix common problems in applications earlier in the lifecycle. Such a tool should allow development teams to release a solid application into QA that is ready to be load tested and where most obvious performance bugs have been found and addressed. J2EE diagnostic tools used in QA need to be designed for load and should be integrated with load testing tools to boost productivity, test effectively, and facilitate collaboration between application owners and QA teams to triage, diagnose, and solve problems under load conditions. During staging or pre-production, deployment teams should determine the performance and scalability of the application in the specific IT environment and configuration that will serve live users, verify projected peak load capability, and tune the application and infrastructure to meet its designed transaction throughput and response-time service levels, keeping the business objectives in mind. Load testing and performance diagnostic tools that can optimize both application and infrastructure and drill down to solve problems are essential during this stage. In live production, operations and application managers need to continuously monitor the health of the entire system, measure the performance of the application, and, if something starts to go wrong, quickly alert, triage, and isolate the problem area. Once the problem area is identified, the appropriate teams should be able to collaborate to pinpoint the root cause of the problem. It is important to not only proactively monitor end-user business processes, but also to diagnose issues that happen with real users using the systems. 4

5 MARKET REQUIREMENTS/ GOALS PRODUCTION- READY APPLICATION BUSINESS OWNER PRODUCTION OPERATIONS MERCURY DIAGNOSTICS FOR J2EE MERCURY BUSINESS AVAILABILITY CENTER DIAGNOSTICS DEVELOP MERCURY DIAGNOSTICS PROFILER MERCURY LOADRUNNER/MERCURY PERFORMANCE CENTER DIAGNOSTICS DEPLOY APPLICATION OWNER QA APPLICATION READY FUNCTIONAL APPLICATION (1 USER) LOAD-READY APPLICATION (5-20 USERS) BUSINESS- READY APPLICATION (50,000 USERS) QA/TESTING Figure 1: Mercury Diagnostics is a single solution that enables developers to find problems earlier in the lifecycle, QA teams to be empowered with the right data set to share with development to fix problems under load, and production operations teams to manage the application 24x7, and triage and diagnose issues as they occur. Let s see an example of how a typical collaborative effort on diagnostics might look using Mercury solutions for a new application being developed. The above diagram shows this process. The four key stakeholders are shown as the four circles each representing the business owner, the application owners, QA, and production operations. The application lifecycle is shown by dotted arrows on the outside, and the interactions of the various roles in the lifecycle are shown as solid arrows for each stage. The Mercury lifecycle solution facilitates collaboration between team members by providing a single technology across the lifecycle and role-based web dashboards that can drastically reduce the mean time to problem resolution. Throughout the entire application lifecycle, J2EE-specific visibility and diagnostic capabilities should be integrated with and complementary to multi-platform, multi-protocol tools for functional testing, load testing, and application performance management. Common tools and measurements help facilitate communication, while team members need specific features and capabilities suited to their roles. 5

6 Typical J2EE Application Performance Problems There are many problems that can surface during the application lifecycle. For J2EE applications, the user s experience of performance is affected by many external network infrastructure factors that are independent of application behavior. A broad set of monitors that can cover the complex environment is needed to triage problems quickly. From the J2EE application diagnostic perspective, it is essential to be able to capture and correlate specific external parameters, such as method arguments, that can drive performance problems in a J2EE method or sequence of transactions. Specific latencies and parameter captures are also frequently required to identify problems at J2EE interfaces to external systems, such as backend databases, legacy systems, and packaged software. Within the J2EE environment, some of the most common problems include: 1. Code problems: a. Slow methods Consistently slow methods Intermittently slow methods, related to specific user/data values driving problematic application behavior b. Memory problems, including memory thrashing and memory leaks c. Coding practices, such as using exceptions as a means to transfer control in the application, processing large result sets, etc. d. Synchronization problems, including both under-synchronization and over-synchronization for locks and threads 2. Application server configuration problems: a. JDBC connection pool size b. JVM heap size c. Thread pool sizes 3. Architecture and design problems, with a wide range of issues, such as: a. Data-marshalling problems resulting from filtering at the wrong tier b. Serial execution resulting from over synchronization in design of custom code An effective diagnostic solution must provide capabilities and techniques to triage and identify the root cause of each of these common problems, regardless of when they emerge, from development through production. Capturing application performance data requires the ability to tune up the data capture as you go from identification and triage to root-cause diagnostics. This is best achieved with lifecycle technology without having to install new tools to collect greater details for root-cause diagnostics. 6

7 MERCURY IT GOVERNANCE CENTER APPLICATION DEVELOPMENT APPLICATION DELIVERY APPLICATION MANAGEMENT BEA IBM SAP ORACLE... MERCURY QUALITY CENTER TEST MANAGEMENT FUNCTIONAL TESTING MERCURY PERFORMANCE CENTER LOAD TESTING MERCURY BUSINESS AVAILABILITY CENTER END USER, APPLICATION, SYSTEM LEVEL AND SERVICE LEVEL MANAGEMENT MERCURY DIAGNOSTICS PROFILER MERCURY DIAGNOSTICS MONITOR TRIAGE DIAGNOSE APPLICATION LIFECYCLE APPLICATION PLATFORMS - BEA, IBM, MICROSOFT, ORACLE, SAP, SIEBEL Figure 2: Mercury Diagnostics for J2EE combines J2EE-specific diagnostics with application delivery and application management solutions. Diagnosing Performance Problems with Mercury Diagnostics for J2EE Mercury Diagnostics for J2EE provides the industry s first top-down, end-to-end lifecycle approach to seamlessly manage, monitor, diagnose, and resolve critical problems in J2EE applications in both preand post-production environments. It can be implemented alone or integrated with Mercury LoadRunner, Mercury Performance Center, and Mercury Business Availability Center offerings. Mercury Diagnostics for J2EE provides a common, consistent foundation of shared assets, metrics, scripts, and monitors, as well as specific solutions for the differing requirements of application delivery and application management. Mercury Diagnostics for J2EE uses a combination of JMX, aggregation, and unique Total Trace technology, which can capture data to diagnose all of the problems previously described. It is the only solution that can capture, under load, necessary and sufficient information to get to the root cause for the full range of J2EE performance problems. Unlike other techniques, Total Trace allows capturing of every event (including arguments) at low overheads. Other transaction tracing techniques may use a threshold for collecting a fixed number of sampled events regardless of whether they are related to the business process in question or not. This could lead to capturing of sampled events that might be totally unrelated to the problem at hand. Another common technique used for tracing a transaction is to tag a server request or HTTP URL request with a specific header. This header then can be followed through the application to provide traceability across methods. Most common implementations of such techniques ignore the business process name that a specific method belongs to and are limited in their ability to trace across JVM boundaries. 7

8 For example, if a business transaction were to interact with several JVMs hosted on remote machines, it would be critical to trace the business transaction across the different JVM boundaries and provide diagnostics data without losing the business context of the transaction. The screen shot to the right shows how the remote calls crossing JVM boundaries are distinguished with a Remote Calls layer and critical data such as SQL queries and application exception information. Figure 3: Tracing remote method invocation across JVMs. Mercury Diagnostics for J2EE for Mercury LoadRunner and Mercury Performance Center Mercury Diagnostics for J2EE is fully integrated with Mercury LoadRunner and Mercury Performance Center to expose and diagnose the most common J2EE performance problems under load testing. Mercury Diagnostics for J2EE first correlates end-user problems to the web transaction tier for fast isolation and resolution, then pinpoints the infrastructure, application, or database problems. It analyzes the TCP/IP stack to break down a transaction s end-to-end response time from an end-user perspective to determine whether poor transaction time is caused by external network issues or by a specific server tier. It then guides users through an intuitive top-down analysis process across multiple application layers, tracing business performance problems from the end user all the way to the problematic component. In addition, Mercury Diagnostics for J2EE shows performance metrics for the typical J2EE tiers such as JSP/Servlets, EJBs, JMS, JDBC, and JNDI across a clustered environment in a single dashboard, allowing you to isolate and identify slow components, detect memory leaks, and more. Mercury Diagnostics for J2EE for Mercury Business Availability Mercury Diagnostics for J2EE is fully integrated with Mercury Business Availability Center to maximize the performance and availability of J2EE applications by providing a unified solution for your operations and application support teams to quickly monitor, diagnose, and optimize production applications. It provides teams with visibility across user, business process, application server, application platform suites such as portal servers, and system tiers down to the component and method level. Ultimately, high-performing, mission-critical applications protect service-level agreements (SLAs) and the bottom line. 8

9 The solution provides the industry s broadest monitoring capabilities that allow 24x7 monitoring of the entire application infrastructure, including the J2EE environment, from a single web-based console. Agentless monitoring for enterprise infrastructure components can optimize the total cost of systems management. In addition, it can facilitate capacity planning by ensuring that purchases of hardware and software are optimized. Figure 4: Transaction breakdown for real user for Portal Server environment showing the specific portlet performance. Mercury Business Availability Center provides correlation of end-user performance with root cause in the infrastructure and application layers. Its J2EE-specific diagnostics identify problems involving EJB components, methods, and JDBC calls with SQL statements. Mercury Diagnostics for J2EE integrated with Mercury Business Availability Center provides the same transaction breakdown capabilities as Mercury LoadRunner and Mercury Performance Center, using aggregated data capture, while optimized for low-overhead, real-time monitoring, triage, and problem isolation. Unlike other implementations that might need inclusion of client-side scripting code in web application pages to measure the response time from the browser, Mercury s diagnostics solution uses industryleading scripting technology to capture this information from an end-user perspective with synthetic scripts. A similar noninvasive technique is also employed to get diagnostics information for real users (actual people using the production systems as opposed to synthetic scripts) for production usage. J2EE Deep Diagnostics Complex problems, such as finding the cause of thread deadlocks or objects that aren t de-allocated and removed by Java Garbage Collection, are responsible for the majority of the time and cost spent in diagnosing the root cause of performance problems. J2EE Deep Diagnostics uses Mercury s Total Trace data capture, which captures every event in a selected area of the application with low overhead, by providing carefully tuned, predefined sets of byte-code instrumentation and a GUI for custom selection and automatic application of narrowly targeted instrumentation. In production, a Mercury Business Availability Center alert can trigger a deep 9

10 diagnostic data capture for offline analysis. This stops the common frustration of persistently recurring production performance problems that cannot be reproduced in a test bed. The same J2EE Deep Diagnostics works with Mercury LoadRunner when performance problems under load need better characterization for effective bug definition and rapid fixes. J2EE Deep Diagnostics provides multi-layer correlation of HTTP, servlets, JSPs, EJBs/objects, methods, and SQL calls. It correlates these internal component measurements with JMX and OS metrics to show problem details. It delivers graphical views of transaction traces, call chains, memory leaks and thrashing problems, synchronization details of locks held and threads blocked, latency charts, and source code views of problem methods. J2EE Deep Diagnostics captures details of specific instances of method invocations, including parameters passed and individual latencies, to pinpoint the source of intermittent problems caused by specific user data values or usage patterns. For further examples and details on how to solve synchronization problems, intermittent slow methods, and other J2EE performance problems using Mercury Diagnostics for J2EE, contact your Mercury representative. Mercury Diagnostics Profiler for J2EE Mercury Diagnostics Profiler for J2EE extends the application lifecycle solution into the development environment. Developers can use the same lifecycle probe technology used for load testing and production to find and fix common application problems earlier in the lifecycle. This also enables applications to be readied for load testing. Often, when QA groups start to load test the application, they find that the application is not ready for load testing. Specifically, the application breaks or doesn t scale beyond two to five users. Lack of application readiness for load testing and production wastes time and effort. Developers are often very interested in application performance data that is actionable. Mercury Diagnostics Profiler not only provides developers the ability to solve the most common problems discussed earlier, but also to find specific problems such as application exceptions being thrown, slowest SQL queries, memory leaks in primitive object types, JAVA objects, and JAVA collections. The following screen shots show how a developer can understand if and specifically what exceptions are being thrown in the application at run time. These could migrate to production as painful timebombs if left undetected earlier. For diagnosing problems in JAVA collections, it s not only important to understand which collections are growing fastest and which are the largest, but also to understand the contained class and where the collection is being allocated. For example, is the contained class in a collection a critical resource such 10

11 as DB connection or socket or a less important Integer object? Figure 5 shows how Mercury Diagnostics Profiler can provide this visibility to help find and solve more problems earlier in the lifecycle. Summary With Mercury Diagnostics for J2EE, enterprises can improve the readiness of their J2EE applications before they are placed into production, and dramatically accelerate problem identification, isolation, and resolution. Mercury Diagnostics for J2EE allows companies to reduce their costs of operations by providing a business process view into the application and by isolating and resolving problems before business users and the bottom line are impacted. The solution can also help companies optimize their existing infrastructure and avoid unnecessary hardware or software expenditures. Choosing the appropriate application profilers, application delivery tools, and application management tools is critical to guaranteeing application quality and ensuring that SLAs can be met. Mercury Diagnostics for J2EE is the industry s first, most complete application delivery and application management suite for the J2EE ecosystem that improves the quality, performance, and scalability of J2EE applications across the entire application lifecycle. Figure 5: Exception stack traces. Figure 6: JAVA collection memory diagnostics. 11

12 Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business outcome of IT Mercury Interactive Corporation. Patents pending. All rights reserved. Mercury Interactive, Mercury the Mercury logo, Mercury Diagnostics for J2EE, Mercury Diagnostics Profiler, Mercury LoadRunner, Mercury Performance Center, and Mercury Business Availability Center are trademarks or registered trademarks of Mercury Interactive Corporation in the United States and/or other foreign countries. All other company, brand, and product names are marks of their respective holders. WP