TUTORIAL: WHITE PAPER VERITAS Indepth for the J2EE Platform PERFORMANCE MANAGEMENT FOR J2EE APPLICATIONS 1
1. Introduction The Critical Mid-Tier... 3 2. Performance Challenges of J2EE Applications... 3 3. Requirements for Performance Management Solutions... 5 4. VERITAS Indepth for J2EE Performance Management for J2EE Applications... 5 5. Summary...10 2
1. INTRODUCTION THE CRITICAL MID-TIER Modern web applications employ mid-tier servers to meet stringent demands for scalability, high performance, and high availability. Web servers in the mid-tier handle user s requests and provide direct response when only static content is required. When dynamic content is required, user s requests are forwarded to application processing logic on the mid-tier, which also coordinates the handling of data requests from back end data servers or other remote services. With the potential for traffic surges and high numbers of concurrent user requests, the mid-tier has become a critical component of an application s performance and scalability strategy. Users/Client Mid-tier Server Database Server JDBC Browser HTTP HTML XML Presentation Layer Business-Logic Layer Database Layer Java has become the language of choice for development and deployment of mid-tier applications. Java running on the mid-tier is called server-side Java, to distinguish it from Java running on clients. Java 2 Enterprise Edition (J2EE) standardizes the operating environment for server-side Java applications. The J2EE standard specifies user page building services based on servlets and Java Server Pages (JSPs). It defines Enterprise Java Beans (EJBs) to encapsulate business logic. Both can connect to the back end database tier through standardized drivers based on Java Database Connectivity (JDBC) standards. J2EE application servers provide these services to Java applications transparently, allowing application developers to focus on business logic. A large number of independent vendors supply Java application servers running on a wide variety of server choices. The J2EE standard and associated application servers allow Java applications to be easily deployed in a range of configurations covering widely varying needs for performance and scalability. As a result, J2EE-based server-side Java has become the dominant platform for high performance web applications. The increasing popularity of J2EE increases its importance in ensuring performance and scalability of web applications. Performance management of J2EE applications and application servers is now a necessity to realize the gains in application scalability inherent in the mid-tier architecture. However, performance management of J2EE applications comes with its own set of challenges for IT staff. 2. PERFORMANCE CHALLENGES OF J2EE APPLICATIONS Multi-tiered web architectures are designed to respond to high traffic intensity and user variability, both in terms of volume and transaction mix. Just as the underlying architecture must scale to meet these demands so too must any performance management solution. Performance management of a web-based application must be at least as flexible as the underlying application architecture. Stringent application performance requirements mean that collection and analysis of performance management data must be accomplished with low impact on the operational system. Low overhead, real-time data collection must satisfy the needs for tactical monitoring while simultaneously providing comprehensive data needed for detailed 3
analysis and strategic planning. The ever-present dichotomy of low overhead vs. comprehensive collection of detailed data is exacerbated on the mid-tier of Web applications. Further complicating performance management is the rapid pace of application development and deployment. Application release cycles of one month are common. Rapid reconfiguration of performance management tools to address changing application characteristics is essential. Automation is critical both to meet time constraints, as well as to control costs. Application administrators look for automation in performance management solutions to allow them to deal with these frequent changes. One response to increasing time and cost constraints has been an increased use of third party application components. Performance management must respond to inclusion of these components as a substantial portion of the total application. These components will not have been engineered for specific operating environments, yet their performance must be managed in relation to a specific application environment. And often source code may not be available, or if it is, the expertise to use it is not readily available to the development staff. Measuring component performance in a specific application is an essential part of determining whether the benefits of component reuse are balanced by the possible introduction of operational scalability barriers. The role of quality assurance organizations has been expanded to emphasize pre-deployment capacity, scalability, and stress testing. QA is now a critical component of performance management. QA staff must design and run extensive testing to verify that applications have the ability to respond to high loads, and to provide scalability to meet increasing needs. Performance testing tools on the mid-tier must address QA s needs for accurate and detailed data. Rapid application evolution means load testing must be performed quickly and efficiently. Using the same performance management tools for both performance testing as well as operational performance monitoring facilitates performance analysis. Operational performance monitoring employs the same technology as load testing but must provide additional functions such as: Monitor service levels to ensure proper performance while minimizing investment and operating costs Set and trigger alarms based on J2EE performance Respond to tuning and configuration changes made to other systems (e.g., web server, networks, DB, and remote services) that shift and change the system bottlenecks Understand how application performance is affected by changes in operational databases, user behavior, workload variations and patterns such as time of day and day of week Establish accurate trends and forecasts to add the fewest resources at the latest date, while ensuring confidence in continued peak performance Prepare for future workload growth by understanding how your application responds to yesterday s spikes today s spike is tomorrow s workload Monitor remote services for service level agreement compliance The mid-tier s role in translating user requests into business logic processing and database requests creates another important challenge for performance management. Application performance managers need an accurate and timely correlation between and across these three keys elements to properly identify and diagnosis performance problems in the context of user experience. The mid-tier is not only dependent upon the host operating system and server configuration, but also the Java application server and Java Virtual Machine (JVM). Performance on the mid-tier is affected by the JVM runtime compiler, the indeterminacy caused by garbage collection, numerous choices for application server pool, queue, and cache management, to name just a few factors. The rapid advance of Java and application server technology resulting in myriad tuning parameters complicates the task of performance management on the mid-tier. IT staff, including development, quality assurance, system and application administrators, and database administrators, all have important roles in performance management of multi-tier Web architectures. New solutions for performance management of server side Java are needed to provide the data and analysis required for success in these complex and demanding roles. 4
3. REQUIREMENTS FOR PERFORMANCE MANAGEMENT SOLUTIONS Performance management solutions for J2EE applications must meet many demanding requirements. IT staff require a full range of performance management functions addressing the specific challenges posed by the J2EE-based midtier. Rapid application development and deployment mean that any solution must include extensive automation capabilities it must install and configure itself with minimal IT involvement. Additionally, instrumentation technology is required to function completely with third party components as well as custom developed software. Performance management solutions must support both monitoring and analysis. Solutions must present current application performance state in such a way that problems can be rapidly detected. Response time and throughput are essential performance metrics, as are indicators of concurrent user-induced application loading. To facilitate rapid problem diagnosis, performance management solutions must provide detailed correlation, drill down, and analysis functions. Correlation provides the power of associating two dissimilar measurements to yield insight that is not otherwise available. For example, CPU resource consumption measured from the underlying OS should be correlated with specific application activity. Application activity must be correlated with user requests. Each application component active in fulfilling a user request should be correlated to other components to provide a complete view of application behavior. Application behavior on the application server should be correlated with requests to the database, and the performance of these requests should be re-correlated to application performance, and ultimately to the performance of user request processing. Each correlation must be made quantitatively so that IT staff are quickly guided to problem diagnosis by following the path of greatest contributor first. All correlation and analysis should be performed automatically. Correlation of JVM activity with application activity provides additional critical insight into the causes of performance shortfalls. JVM sampling combined with application instrumentation is required to provide this information. Sampling can provide identification of CPU hot spots that can then be correlated with application components and user requests. Memory usage and garbage collection should be measured and correlated with application performance. CPU hot spots observed using sampling should be correlated with all levels of application components and user activity. Performance management solutions must provide a historical perspective of application performance, including the ability for detailed analysis of any particular time in the past. A user interface which naturally guides the user through the complex data and its analysis is not just convenient, but essential to handle the complexities of J2EE application performance management. 4. VERITAS INDEPTH FOR J2EE PERFORMANCE MANAGEMENT FOR J2EE APPLICATIONS VERITAS Indepth for J2EE is a performance management solution specifically designed and developed for J2EE applications. VERITAS Indepth for J2EE provides all of the critical functions required to manage performance on the J2EE-based mid-tier. VERITAS has designed VERITAS Indepth for J2EE to adhere to important principles that are directed at its effective use by IT staff: Scalable, multi-tier architecture Low overhead for collection of both operational monitoring and detailed analysis data Configurable to support a full range of application and architectures User interface is based on an intuitive data summarization and exploration paradigm Automatic installation and default configuration Intelligent, automated performance analysis and advice VERITAS Indepth for J2EE employs a multi-tier, fully distributable architecture to provide the scalability to meet the variable needs of Java applications and deployments. VERITAS Indepth for J2EE collectors run on the Java 5
application server mid-tier, using low overhead, accurate, and comprehensive technology. Data is correlated and summarized in real-time, then forwarded to a second tier, a VERITAS Indepth for J2EE 100% Java Archive Agent. The Archive Agent stores the data collected and correlated by the J2EE Collectors in a SQL database. IT staff can deploy the Archive Agent on the operational server, or a work group server to isolate archive overhead from the operational system. The VERITAS Indepth for J2EE Analysis Agent and User Interface tier (again 100% Java) is where detailed analysis is performed to support summary reporting, analysis, drill down, and historical evaluation of performance. This tier may be deployed with the Archive Agent tier, or own its own server. The Analysis Agent reads data from the database and analyzes it. The User Interface creates and localizes reports of current and recent activity, drill down and historical evaluation of performance. VERITAS Indepth for J2EE user interface uses a thin client HTML-based approach to allow IT staff to use all VERITAS Indepth for J2EE features from a standard browser either locally or from a remote location. All of VERITAS Indepth for J2EE components can be multi-deployed in many-to-one and many -to-many configurations to support multi-jvm, multi-server mid-tier application configurations. VERITAS Indepth for J2EE scalable architecture ensures that performance management overhead is negligible for even large-scale applications. VERITAS Indepth for J2EE's Multi -tiered Distributed Architecture The VERITAS Indepth for J2EE data collectors on the application server mid-tier use both application instrumentation and Java Virtual Machine sampling. Both are essential for comprehensive performance management. VERITAS Indepth for J2EE correlates the two types of metrics to increase the value of both to identify and remedy performance problem and scalability bottlenecks. In instrumentation, special recording hooks are inserted at critical points in the mid-tier application. During execution these hooks record events and time tags that are analyzed to provide an accurate and detailed history of performance. With its close ties to the application, instrumentation provides an application-centric view of performance correlated with both user page requests and backend relational database requests. JVM sampling provides a resource/bottleneck perspective of performance. Sampling involves periodic collection of the state of the JVM and the threads it is managing. Statistical analysis of these samples provides an aggregate synopsis of performance over an interval of time that provides valuable resource centric performance information, and is even more enlightening when correlated with instrumentation. VERITAS Indepth for J2EE instrumentation is automatically generated without requiring effort or training of development or application administration staff. The byte code instrumenter provides application specific events 6
during application execution. VERITAS Indepth for J2EE fault tolerant instrumentation provides the most accurate and detailed methods for monitoring and analysis of application performance. The VERITAS Indepth for J2EE approach provides the advantages of custom instrumentation without its time consuming effort because application location and structure are automatically discovered. Thus, collectors can rapidly respond to the changing application demands of the server-side Java environment. In addition, VERITAS Indepth for J2EE byte-code instrumentation technology can instrument third party Java components whose performance will otherwise appear as a large black box. VERITAS Indepth for J2EE provides unmatched installation speed with most out-of-the-box installations completed in less than 30 minutes. Instrumentation provides accurate summary level performance monitoring for HTTP service (Servlets and JSPs) and EJB service for EJB servers. Response time and throughputs are reported at user specified frequencies. Since a primary role of the mid-tier is to handle a high degree of concurrency, VERITAS Indepth for J2EE measures the number of concurrent service requests being processed by the application. Operations personnel use application performance summaries for efficient monitoring of service levels and initial problem isolation. Application event correlation is one of the strengths of the VERITAS Indepth for J2EE analysis component. To provide IT staff with necessary information to effectively evaluate performance and resolve problems, VERITAS Indepth for J2EE correlates data in numerous ways. Data extracted with collectors from the underlying operating system is correlated with application event data to measure CPU consumption by Java application component and specific end user requests. Both QA and production staff can use this data to identify application hot spots. As user load changes, or new software is released, QA and production staff can track corresponding performance changes. Java application components and database requests are correlated with end user requests to give IT staff a unique understanding of application performance and scalability in terms of end user demands. Multiple levels of calls among Java components are correlated to provide visibility into quantitative relationships among application components. VERITAS Indepth for J2EE correlates application components with database activity to isolate scalability bottlenecks between the mid-tier and the database tier. Using correlation technology leverages the unique vantage point of the mid-tier to associate user requests with database activity. VERITAS Indepth for J2EE can tell IT staffs how many trips to the database each user request requires, and how long each takes to complete. VERITAS Indepth for J2EE identifies and isolates the longest-running SQL statements to facilitate tuning of the database. 7
More extensive processing in the VERITAS Indepth for J2EE analysis component automatically ranks contributors to response time and other performance metrics. IT staff use these rankings to quickly drill down to the root cause of a problem. VERITAS Indepth for J2EE User Interface provides extensive use of drill-down hyperlinks making sharing and exchanging problem information easy. The VERITAS Indepth for J2EE JVM sampler feature provides another important source of data for performance analysis. Since garbage collection by the JVM can produce significant performance impact, VERITAS Indepth for J2EE uses a memory sampler to report memory usage statistics as well as details on garbage collection efficiency. Additional sampled data from the JVM identifies call sequences that consume significant amounts of CPU resources. These sequences are correlated with user requests and Java components in a drill down fashion to identify for IT staff possible hidden resources affecting user performance. VERITAS Indepth for J2EE provides a wide range of data presentations to enable quick and accurate performance monitoring and analysis. Current operational data is summarized for identification of abnormal conditions. Time series of recent data supplements current information to allow operations staff to detect changes that can lead to the source of abnormal conditions. VERITAS Indepth for J2EE also archives performance data (under full user control) and provides additional display and analysis capabilities for this historical data for use in detailed performance analysis and capacity planning. VERITAS Indepth for J2EE archive capabilities are also used by QA staff to record application behavior under controlled test load conditions and analyze the results. The wealth and depth of data measured by VERITAS Indepth for J2EE can be overwhelming, especially for already heavily burdened IT staff. Operational pressures compound the difficulties of rapid response to performance problems. VERITAS Indepth for J2EE addresses these challenges with its intelligent automation of problem identification and correction advice, called SmarTune. SmarTune is a fully configurable rules-based advisor that performs numerous checks though all instrumented response time data. The result is a multi-point check of J2EE application performance. SmarTune automatically searches through all the data for a data collection interval. It traverses all instrumented execution paths looking for both local and application-wide performance hot spots. Specialized rule-based search algorithms check for such common problems as long running SQL statements, excessive connection pool wait times, 8
and delays due to JNDI lookups. SmarTune hunts down general application hot spots based on standard or custom instrumented methods. SmarTune automatically searches application performance metrics to identify, categorize, and rank performance problems SmarTune doesn t stop with just finding hot spots. SmarTune ranks each problem area to allow rapid treatment of the most significant performance issues. SmarTune places each problem in a complete application context so that operations staff can precisely describe the problem conditions for resolution. For many problems, SmarTune offers advice on the nature of the problem and options for corrective action. All this information is based on industry best practices as defined by J2EE performance experts throughout the Java community. Finally, SmarTune is based on an extensible, tailorable rules engine. Operations or development staff can instruct SmarTune to look for specialized performance conditions and report them in a context that allows rapid diagnosis and correction. 9
SmarTune offers complete problem explanation and suggestions for corrective action within the J2EE application context 5. SUMMARY VERITAS Indepth for J2EE is a comprehensive performance management solution for J2EE applications, addressing the needs of system operators, managers, quality assurance specialists, and planners. The critical role of the J2EEbased mid-tier in application performance and scalability demands the specialized data collection and analysis capabilities provided by VERITAS Indepth for J2EE. 10
VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA 94043 650-527-8000 or 866-837-4827 For additional information about VERITAS Software, its products, or the location of an office near you, please call our corporate headquarters or visit our Web site at www.veritas.com. 11