APPLICATION PERFORMANCE MANAGEMENT VERITAS i 3 FOR SAP

Transcription

1 TUTORIAL: WHITE PAPER VERITAS i 3 for SAP APPLICATION PERFORMANCE MANAGEMENT VERITAS i 3 FOR SAP SAMPLE WHITE PAPER 1

2 1. Executive Summary Introduction The Typical Environment Challenges Managing the Environment The VERITAS i 3 Methodology VERITAS i The Client Web Browser Component The Network Component Web Server Component ITS (Agate) Component J2EE Servlets R/3 Application Server Component Database Component Storage Devices Component VERITAS i 3 Features Proactive Management Through Exceptions End-user Experience and SLAs Correlated End-to-End View Understanding and Correlating SAP WAS Activity Investigating the SAP R/3 Application Tier Investigating the SAP Database Tier Long-Term Analysis Putting All the Pieces Together A Typical Scenario Benefits of VERITAS i Summary

3 1. EXECUTIVE SUMMARY SAP R/3 (SAP) customers can now access their traditional SAP system using a web browser either through the current Internet Transaction Server (ITS) or the version 6.20 Web Application Server (WAS). Application Performance Management in these environments is more challenging due to the number of technologies involved, multiple points of failure and the increased number of components to manage. The major challenge facing SAP administrators today is defining a systematic process to understand application behavior, usage patterns, service levels and alerts. This challenge becomes even greater with the introduction of SAP technologies like ITS and application servers. These new technologies demand more knowledge, understanding, responsiveness, and around-the-clock high performance than their less complex, traditional SAP applications in order to adhere to service level commitments and maintain end-user productivity. Experienced SAP personnel recognize the need to include Application Performance Management best practices throughout the entire application life cycle. Another significant challenge is measuring the actual service being delivered to the end-user and then being able to determine the end-to-end service level contributors. These measurements are needed to make the appropriate tuning, design or architectural changes, as well as integrating the SAP applications with other legacy or `third party applications. The introduction of new technology from SAP and the ensuing complex architecture can impose new challenges on organizations who wish to upgrade to these newer versions. Should such an upgrade happen, the organization may assume an upgrade effort equivalent to up to 50% of the initial investment, both in cost and schedule. One of the main challenges in such upgrades is the ability to forecast and plan the infrastructure capacity necessary to track the usage patterns and performance of the old version and be able to compare loads, usage and performance of the new implementation to the old one, already adopted by the enterprise. Such an effort will result in faster time to deployment, better capacity planning and an overall savings to the upgrade process budget. VERITAS i 3 for SAP (VERITAS i 3 ) is the industry s first solution that provides the means to quickly and efficiently capture and correlate SAP application metrics across the whole SAP infrastructure. These metrics include: Real user activated SAP web pages and URLs, as well as SAP R/3 transactions Real end user service and response Time Network time and availability of remote geographical locations Response time and availability of your company s functional organizations. Invoked J2EE servlets running on the SAP WAS SAP R/3 transactions and screen numbers SAP Application modules SAP Users Database SQL statements Activities introduced by external RFC calls. VERITAS i 3 presents these important metrics in a manner that enables crisp communication along with rapid detection, correction, and verification throughout the application life cycle. This paper describes this unique approach. 2. INTRODUCTION The introduction of components like ITS and WAS enable users to access the traditional SAP system from anywhere, using a web browser. For the end-user, it means more functionality and convenience, but for SAP administrators and BASIS teams, it means more responsibility and complexity. In such environments, Application Performance Management is a vital necessity. 3

4 Many problems can occur in a complex environment either in the network, the web access components, the SAP application server, the database, or the storage devices. To make the situation worse, problems tend to be inconsistent (i.e. they can appear and disappear) or can occur only when a certain load is reached. Managing these environments is challenging. It is crucial to run 24x7 health checks for notification of potential problems before end-users are affected. It is important to understand the application behavior and the usage of the organization from a performance point of view to verify that business objectives are met, find where bottlenecks are, and the required next step to fix them. Often, companies rush to finish the SAP implementation phase which includes customizations and changes and pay less attention to performance issues, scalability and load. After the system goes live and performance problems begin, it is most expensive to fix them. VERITAS has developed an Application Performance Management solution for SAP (VERITAS i 3 ) that monitors the entire SAP infrastructure, including the new web application components introduced in version THE TYPICAL ENVIRONMENT There are three main typical SAP environments. The first, and most traditional one, comprises of a Microsoft Windows client running an SAP GUI connected directly to SAP application servers. SAP application servers connect to the database in order to fetch relevant data. The second one (Figure 1) introduces a component called Internet Transaction Server (ITS). Users are connected to the SAP system through a web browser. Requests come in to a web server, are routed to the ITS (Agate), and translated there into traditional SAP requests (DIAG or RFC protocol) before being submitted to the R/3 system. Responses are sent back the reverse way to the client in HTML format. All ITS translation between HTML and SAP protocol is transparent to the end-user. Figure 1. SAP R/3 system with ITS There can be multiple web servers and ITS gateways depending on the configuration. Since the ITS keeps the user state, load-balancing of incoming requests must make sure that once a user has connected through one of the ITS gateways, following requests should continue to go through the same ITS gateway. The third configuration (Figure 2), introduced in SAP version 6.20, replaces ITS with a J2EE-based Web Application Server (WAS). It introduces new technologies like Java servlets running in a J2EE server (In-Q-My). In this 4

5 configuration, a web browser connects to SAP WAS. The WAS anticipates the request and forwards it to the Java Server Pages layer (JSPs), which in turn connects to the SAP application server through SAP Java connector. The SAP Java connector makes use of RFC calls to interact with SAP application server. SAP application server connects to the database to fetch the required data. Responses are sent back the reverse way, from SAP back to the JSP and to the user. Figure 2. Information flow in SAP Version 6.20 The entire architecture with the different tiers, web browser, network, SAP web application server, SAP application server and the database is shown in the Figure 3. Figure 3. Entire multi-tier architecture in SAP Version 6.20 This new generation of SAP adds new technologies and functionality as well as new problems to manage. Managing traditional SAP application servers and database activity requires a lot of skill, tools, and experience. With the introduction of new web and Java components, the management tasks become more difficult. 4. CHALLENGES MANAGING THE ENVIRONMENT The management of SAP, spread across multiple tiers, makes it extremely difficult to pinpoint the source of a performance problem. There are significant challenges to managing these environments, including: Defining, measuring, and committing to organizational business objectives and SLAs, as actually be activated by the real organization users, from multiple geographic locations 5

6 Finding the SAP users, transactions, application modules, programs, functions, and screens that impact the performance and those that are most easily influenced by performance degradation Defining baselines of normal and abnormal application and usage behavior A comprehensive, yet straightforward, way to measure the user s experience Identifying problem patterns and characteristics before the user is affected Proactively, through alerts, assuring that satisfactory service levels are maintained Maintaining historical data for trending, exception and capacity planning analysis 24x7, real-time, low-overhead collection and correlation of performance metrics across the SAP infrastructure Correlating SAP application metrics like user activated URLs, invoked J2EE servlets, called JCO APIs to SAP application server, and database SQL statements across the whole SAP infrastructure The ability to break down response times by tier and application component to determine the amount of time spent in each component Determining the time spent in Java vs. ABAP vs. SQL statement processing Presenting one integrated correlated view of all SAP infrastructure components, from the web user, through the network, SAP WAS, traditional SAP application server, database, and storage devices Ability to focus on SAP application server bottlenecks like waiting for a work process, memory issues, inefficient database access, high CPU utilization, etc. Tracking the performance of local and remote geographical locations Coping with unpredictable load periods and handling massive transaction/data volumes Aiding SAP basis administrators in analyzing performance issues and suggest remedies These challenges are not covered by system management frameworks, or by traditional tools which only take uncorrelated snapshots from various servers along the SAP application infrastructure. System management frameworks focus on system parameters, availability, and utilization. Their primary focus is monitoring the system and the system elements health. Frameworks are well suited for detecting system level and network issues such as servers and applications with high utilization, high packet rates, and numerous retries. Unfortunately, frameworks don t focus on the end-user experience, transactions and usage. Therefore, if all system parameters look OK, how can we tell why location X suffers from poor response-time, or why certain SAP URLs are slow? The stovepipe approach, which takes isolated snapshots form various servers, does not resolve the problem either. Snapshots provide very coarse information consisting of generic statistics taken from each tier separately. For example, you can look at Java servlets, SAP application servers, or Oracle metrics and still not understand which component is causing a bottleneck. By following the real end-user transaction across the SAP tiers, snapshots can tell you that the average Java servlets, SAP transaction and SQL response times are OK, but they cannot tell which SQL statements belong to which transactions and, therefore, cannot help much in isolating the problem. Traditional SAP tools are limited in their success to meet the performance management challenges of today s business. Since these tools adopted a stovepipe approach they are limited in their ability to correlate performance between the different infrastructure components. The additional technologies (e.g. web, Java) being deployed in the latest release will only compound the problems for these traditional methods. Stovepipe solutions force technical experts to spend an excessive amount of time correlating multiple metrics to pinpoint the root cause of performance degradation. Even as traditional tools link various GUI s to create loosely integrated suites, it is still not enough to solve the problem. Compounding this issue, many of the data collectors deployed do not sample frequently enough (e.g. once every 5-10 minutes) and as a result, they rarely gather enough information to perform meaningful analysis. Performance problems are the toughest and most costly to pinpoint and resolve. This is particularly true if the tools used do not collect enough of the necessary information. Metrics for end-to-end application analysis must be captured with agents having application knowledge and visibility, and those that reflect the user experience. Real-time performance metrics should be collected 24x7, with extreme 6

7 efficiency and low over head and correlated across the SAP infrastructure path to enable problem isolation and root cause identification. Additionally, historical information must be kept for long-term analysis, exceptions, baselines, trends, and capacity planning. Using historical data, we can calculate typical SAP behavior for example, there may be peaks each Monday morning for some transactions. Knowledge of typical behavior facilitates generating exceptions for abnormal behavior and setting the right SLA thresholds. This information also helps calculate database table growth allowing us to allocate enough space in advance of a slowdown or create appropriate indexes to reduce the fetch times. In short, typical infrastructure frameworks provide important system and network management bottoms-up detail, but do not look at necessary end-user and transaction activity. Traditional tools look at each tier separately providing generic statistics for that tier but do not perform any correlation between tiers. These approaches lack the power to find, isolate, and focus on the root cause of the problem or help to identify the real user response-time experience. VERITAS i 3 fills this gap by providing the actual user experience and application end-to-end response-time contribution detail. This enables SAP administrators to see how their applications are performing from the perspective of their users and to correlate that experience to the underlying application components in terms of endto-end performance. 5. THE VERITAS i 3 METHODOLOGY It is vital for all organizations to understand the importance of a systematic approach to coping with performance problems, and to realize the inherent ineffectiveness of ad-hoc problem solving. The foundation of the VERITAS i 3 solution is a proven methodology designed to facilitate the rapid detection, isolation, analysis, correction and verification of application performance problems. All organizations can benefit from the discipline that VERITAS i 3 brings to the Application Performance Management process. The VERITAS Application Performance Management methodology comprises of five stages: Detect Proactively identify the symptoms that could indicate a performance problem Find Identify the problematic tier and application component Focus Discover the root cause of the problem Improve Take the steps required to improve performance Verify Make sure the steps taken have achieved the desired goal These stages combine to form a process that provides a systematic approach to finding and resolving all kinds of performance issues both predictable and unforeseen. For example, the methodology begins with an automated proactive detection (Detect) through baselines and SLAs, using the product s alert capabilities. Alerts can be directed to the appropriate infrastructure framework, organization, or person based on what was detected. Depending on the issue detected, the SAP BASIS Application Manager can view the application, end-to-end, and isolate (Find) the problematic tier and application component that is the source of the degradation. Problems could be caused by network delays, long-running servlets, RFC calls, long-running transactions on the R/3 application server or long database access times. The process of finding the problematic component is end-user focused, so that we actually follow the end-user transaction from the time it was activated by the user. After finding the problematic component, an in-depth analysis of that component is in order to focus on the root-cause of the problem (Focus). For example, after finding that SAP application server is the problematic component, we would like to understand who are the top-heavy users and transactions, whether it is because of a long queue wait time for a work process or application processing time, or a long database call. Another example, in the case of Java servlets, we would like to look at the different methods called including, R/3 JCO (Java connector) and understand whether it is because of a specific method or long connection time to the R/3 application server. 7

8 After focusing on the problem s root cause comes the actual fixing of the problem (Improve). For example, allocating additional R/3 work processes fixes a long queue time waiting for a work process. Adding an index to a database table fixes a long running SQL statement problem. Once the fix has been applied, we need to ensure that indeed the problem is fixed and will not reappear (Verify). Utilizing long-term information kept in the VERITAS Performance Warehouse, we can examine service times before and after the fix has been applied. Doing this shows whether or not we fixed the problem and if service times are as expected. 6. VERITAS i 3 VERITAS i 3 focuses on providing your organization with the ability to completely manage your application performance end-to-end including, the browser, the web server, R/3 web application server (which includes J2EE servlets and JCO), R/3 application server, database, and even into the physical storage. Conceptually, VERITAS i 3 is composed of Insight, Indepth and Inform functionality. Figure 4 below depicts the architectural components. INSIGHT.. Isolates the where/who? Application Response Time Segmentation INFORM.. When? Effective Communication Client Network Web Srvrs App Srvrs DB Srvrs Storage Provides the details associated with the underlying components INDEPTH.. Why? Figure 4. Architectural Components of VERITAS i 3 VERITAS Insight provides the agents and logic required to enable your IT organization to see the end-to-end response time contributions across the client, network, R/3 WAS, R/3 application server, and database server and to isolate the components which are responsible for slowdowns. The process starts by measuring the actual web-user response time at the application, transaction, or page level. Then, upon detection of a response time objective or baseline breach, the problematic tier and component are found, as described earlier in the methodology section. VERITAS Indepth provides in-depth drill-downs to identify the root cause of problems. It collects and correlates detailed component information (e.g. SAP application server utilization and workload, top-heavy transactions, J2EE servlet specific information, methods, Oracle resource consumption breakdown, slowest SQL statements, etc.). VERITAS Inform provides the real-time alerts that are driven by the baseline and thresholds defined. VERITAS Inform also includes the reporting components of VERITAS i 3. This rich report capability enables the presentation of ad hoc and scheduled reports. These reports are useful for service levels, trends, and exceptions. 8

9 VERITAS i 3 includes the VERITAS Performance Warehouse, which provides the common repository for long-term history, collected by the different VERITAS agents along the SAP application path (e.g. R/3 WAS, R/3 application server, database, and storage devices.) This facilitates trend analysis, long-term capacity planning, baseline calculations using historical data and correlated reports over time. VERITAS i 3 includes all necessary functionality to see the actual user s response-time and analyze the correlated contributors from end-to-end. These include: The actual service delivered to any SAP user over any given time period Discovery and monitoring of all SAP users and the performance of their transactions Web server URLs J2EE servlet invocations Java specific methods invocations, response-times and throughput SQL statements activated by specific SAP transactions Database resource consumption breakdown and linking of logical database access to physical storage SAP R/3 application server metrics: cpu usage, load average, page faults, connected users, active users The remainder of this section provides more detail on the functionality and data entities collected from each tier and component across SAP application infrastructure. Correlation Figure 5. Collection and correlation of performance metrics across SAP architecture As mentioned earlier, collection of data at each tier by itself is not sufficient. Collected data must be correlated across tiers to isolate performance problems by application components (e.g. web browser, network, R/3 WAS, R/3 AS, database, and storage.) Further, in many cases, there are multiple SAP versions with heterogeneous architectures interacting with each other. These systems may include the traditional SAP R/3 fat client, ITS, and the new WAS architecture. It is therefore important to correlate performance data not only between internal tiers, but also between mutltiple external SAP systems as well. The expandable scope of the VERITAS i3 solution allows it to adapt and monitor these SAP environments as they grow and connect with new technologies. 9

10 6.1 THE CLIENT WEB BROWSER COMPONENT The way to measure service level objectives is from the end-user point of view, at the browser level, measuring real SAP activity response-times. VERITAS has developed a unique technology that sends a tiny piece of code that is attached to each web page and monitors the end-user performance from the end-user browser collecting performance metrics 24X7 from all users. No installation is required at the end-user desktop. Main data entities collected by this component include: Response-time, which is composed of the time from first byte through rendering time Service Level Objectives both warning and critical SAP transactions as activated by the real user Business groups and functions Location origin the SAP transaction was activated from There are many additional collected data entities like connection type (LAN vs. dial-up), protocol type (HTTP vs. HTTPS), etc. 6.2 THE NETWORK COMPONENT VERITAS i 3 monitors the network response times for SAP clients at different geographical locations. This enables problem detection of remote users and sites as well as differentiation between front-end network time and back-end time. (Back-end time refers to the components from the web server to the database.) Main data entities collected by this component include: Client IP network addresses Request service times Front-end network time vs. R/3 back-end time differentiation Number of requests and throughput Number of bytes sent/received Response-time to remote geographical locations 6.3 WEB SERVER COMPONENT At the web server tier, VERITAS i 3 measures URL response times. It is important to understand which URLs are the heaviest ones from a response-time perspective, as well as the top URLs from a resource consumption perspective. In SAP 6.20 the web server component comes as part of the WAS. It is also important to understand that the above mentioned information is valuable only when one can effectively correlate the SAP Transaction activated from the Web browser with its underlying URLs and Network activities. This is only a piece of what VERITAS i3 delivers in the SAP environment Main data entities collected by this component include: URL service times (time to retrieve a specific URL by the web server and the back-end system) Web server machines and web server instances Domains URL file types Client IP address requesting the URL Response-time to remote geographical locations 6.4 ITS (AGATE) COMPONENT It is important to understand how much time is spent at the ITS (Agate) tier where translation between web requests are translated to traditional SAP requests (DIAG or RFC). This is done by measuring incoming and outgoing traffic to the ITS gateway. Along with information that is collected at the R/3 application server, we can understand if the ITS is the bottleneck 10

11 Main data entities collected by this component include: Request/Response service times Number of requests Throughput 6.5 J2EE SERVLETS SAP runs its JSPs on the WAS, which is Java enabled. It is important to analyze servlet activity to find potential slowdowns. Main data entities collected by this component include: JVM data HTTP service requests Servlet response time and throughput The activated SAP R/3 server methods through the RFC call, along with requested parameters Drill-down and correlation of called methods JDBC request response-time and SQL statements in case of a direct access to the database CPU time, JVM Memory garbage collectors, App server metrics 6.6 R/3 APPLICATION SERVER COMPONENT The R/3 application server, where SAP transactions are executed, plays a key role in the SAP architecture. A transaction being executed can have several potential performance problems at the application server, starting from waiting for the dispatcher and work process, continuing with internal SAP waits and ending with long database access times. It is important to continuously monitor the system for performance degradations. Main data entities collected by this component include: SAP users and transactions Work types: dialogs and batch Transaction processing times, wait times, and DB access times R/3 application modules R/3 program and functions Workload information and availability metrics It is possible to view the information and performance effects by different categories like R/3 systems, organizational business units, locations, and application servers. 6.7 DATABASE COMPONENT The R/3 application server uses the database to access required business information. An un-tuned database could cause a lot of performance problems. It is important to continuously monitor the database and collect the right performance metrics. It is equally important to analyze the R/3 database access and see if we can improve its performance. Main data entities collected by this component include: Database resource consumption breakdown (CPU, I/O, Buffer waits, etc) Database users, programs, and objects Executed SQL statements Data Base objects, internal locks and events as accessed by ABAP/SQLstatements as well as their performance SAP users and transactions issuing the SQL requests 11

12 It is important to emphasize the link between problematic SQL statements and their initiating SAP users and transactions. This enables us to find the users that execute resource-consuming transactions at the database level. 6.8 STORAGE DEVICES COMPONENT One of the primary reasons of performance problems has always been caused by inefficiently accessing physical devices. Problems could be of several types, (e.g. non-well balanced files on devices, contention between files on the same device, etc.). We would like to map logical database objects access to physical devices in order to understand I/O device access problems. Mapping logical Oracle I/O requests to physical storage devices becomes more crucial in RAID-based architectures where a file can spread over multiple physical devices, using striping techniques. VERITAS has developed mapping to EMC Symmetrix devices, HP XP and Hitachi LightPoint and IBM ESS Shark devices. 7. VERITAS i 3 FEATURES The StartPoint screen enables full end-to-end view of the SAP environment including, the web client, the web application server which includes the web server component and the J2EE servlets, R/3 application server, and the database. StartPoint provides an alert overview of the different components by category, in a simplified way to facilitate the communication of team members, e.g. performance or load alerts at the database. The next step is to launch one of the VERITAS i 3 components (Insight, Indepth, Inform). A typical scenario could start with a user launching VERITAS Indepth for SAP to get an overall view of his SAP environment or proactively receiving an alert indicating that a service level threshold has been exceeded. Long-term trend analysis and top resource-consumers could be easily obtained through reports using long-term metrics stored in the Performance Warehouse. Figure 6. Startpoint Screen with two SAP WAS enabled environments, one standard R/3 environment and an Enterprise Portal environment 12

13 7.1 PROACTIVE MANAGEMENT THROUGH EXCEPTIONS Managing a large SAP application infrastructure with hundreds/thousands of users is difficult, to say the least. The only way to do so is to work proactively rather than reactively. VERITAS i 3 provides the way to do so through the use of exceptions, based on automatically computed SLAs and baselines. Once an exception is detected by Veritas i3, the user is alerted about this normative deviation. In addition Veritas i3 will provide capacity trending that will alert the user should future unforeseen loads and performance issues impact the overall SLA and performance. The graphics below show an alert generated on a SAP transaction code breaching the SLA. Figure 7. Performance Alert generated on a CRM system Figure 8. Alert detail screen shows SAP transaction code VA01breaching the SLA 13

14 7.2 END-USER EXPERIENCE AND SLAS End-user satisfaction is extremely important. If end-users are satisfied, application managers are satisfied. VERITAS i 3 provides a way to measure response-times as experienced by end-users and to define SLA thresholds on those metrics. Figure 9. Example of Dashboard view with an availability compliance breach 7.3 CORRELATED END-TO-END VIEW Launching VERITAS Insight provides an end-to-end overview of the different tiers and components. This view is correlated across components, which means you would be able to view for example the J2EE servlets invoked by a specific URL or SQL statements executed by a specific user or transaction. The graphic below shows an example of an Insight overview screen for a multi-tier SAP environment. Figure 10. Example overview screen 14

15 On the upper left hand side, Service Time Comparison, we see the different tiers for Web, J2EE, R/3, and Oracle. In this example we can identify the Web tier as the main contributor to response time (bars in this chart are cumulative). Just below we can see the SAP application App Tiers Data Flow, from the web server to the database. The center part of the screen displays the main data entities for the selected component. For the web server it would be URLs, for J2EE servlets, for SAP application server transactions running in the application server, for Oracle database SQL statements. In this case, the main screen is showing the different transactions, users, work types dialogs, updates and application servers. The information is broken by network time, queue time, work time, and database time. This provides us an overview of what is going on in the SAP application servers. 7.4 UNDERSTANDING AND CORRELATING SAP WAS ACTIVITY VERITAS Insight provides us with an end-to-end view as well as segmentation of activity across tiers. Moreover, we can get high-level details on a tier we would like to focus on. To explore more on information on J2EE activity within the SAP WAS let s choose J2EE on the top menu. Performance metrics for the SAP web application server and JVM are displayed. Figure 11. Example screen of drilling down in J2EE We can view the different method invocations. In the screen above we see one method called com.sapmarket. We can also see on the left side of the screen details regarding the JVM and its scalability. Scalability means how the web application server is able to cope with increased load. More detailed information on the web application tier could be obtained by drilling down into its J2EE component. Drilling down shows the different URLs invoked, their correlation with executed servlets and methods invoked. By instrumenting the right J2EE classes we can continue to drill down and view the different methods invoked. One can instrument the right J2EE classes to the level where the J2EE component calls the SAP application server, the SAP R/3 JCO- Java Connector, and measure the service times serviced by the R/3 application server. 15

16 7.5 INVESTIGATING THE SAP R/3 APPLICATION TIER To get more detailed information on the SAP R/3 application tier, we can launch the VERITAS Indepth for SAP component. This component provides details on the SAP application server load, availability and response times, broken by systems, locations, organization units, users, transactions, SAP application modules, e.g. SD, FI, etc, by program names, by screen numbers and more. The next screens present such examples. Figure 12. In this example we can see SAP R/3 activity broken by Application server. We can also view the number of transactions and check whether they meet their SLA goal that was defined. Figure 13. The screen above provides details on the worst performing transactions executed for a specific production system. 16

17 For each transaction we can view its network, application and database times as well as its response time vs. a predefined SLA. Figure 14. This screen presents the top worst performing batch jobs sorted by total duration. Figure 15. Here is the detail view for a selected batch job. We can see queue time, application time, database time, memory resources consumed and buffer hit ratios for any given batch job. Information on additional application server resources could be obtained in a similar way as well as recommended values for important metrics. 7.6 INVESTIGATING THE SAP DATABASE TIER At the database level, detailed resource consumption breakdowns, SQL statements, users, and programs are provided. Moreover, a unique integration between Indepth for SAP and Indepth for Oracle allows for direct correlation of the SAP workload from a high-level SLA view all the way down to a sql statement that was executed in the database and even into the underlying storage where the datafile exists. This gives us an understanding of which 17

18 organization or location is responsible for generating poor response time. The graphics below shows a detailed resource consumption view of an SQL statement generated by a SAP application server. Figure 16. We can see the unique SQL statement generated by a transaction. In this case, the statement shows a long Oracle CPU time. Figure 17. We can see the analyze dates and the history of the execution plan. In this case we see that the time spent in Oracle for this SAP transaction increased after the execution plan changed. VERITAS i 3 also provides a complete database analysis technology that includes What-If simulation, SmarTune recommendation, and an index-tuning advisor which helps in creating indexes as well as simulating the impact of indexes before indexes are actually created. 7.7 LONG-TERM ANALYSIS Performance metrics collected from the client tier, web server, J2EE servlets, SAP application server and the database are kept in the i3 Performance Warehouse. These performance metrics are used for trend analysis and capacity planning. VERITAS i 3 provides a reporting feature called Foresight, which utilizes the Performance Warehouse to generate hundreds of out-of-the-box Adobe Acrobat reports. The reports range from Exception reports which show specific service level breaches, Alert reports which show deviation from nominal behavior, Profile reports 18

19 which enable you to analyze correlated performance data between tiers, Availability reports for both application servers and remote locations, and many types of Capacity planning reports that will enable you to plan for the future, based on past and present trends. Figure 18. Using VERITAS i 3 functionality you can generate hundreds of useful reports right out of the box. For example, the screen in Figure 18 is a report showing total appserver availability over a period of 7 days. We can see that for the selected time period, the CRM system has mostly remained above the Service level threshold for system availability. However, the bottom portion of the report details exactly when downtime occurred and for how long. 8. PUTTING ALL THE PIECES TOGETHER A TYPICAL SCENARIO Now that we are familiar with VERITAS i 3, let s look at a typical scenario. SAP components involved: R/3 and Oracle. Our scenario starts with a call into the help desk. An SAP user from the Sales Department is complaining about slow response time while entering sales orders into the R/3 system. He mentions that he normally receives excellent response time and feels that it has gotten worse in the last few days. Unfortunately, the user has no quantitative definition of what worse is, so normally our help desk staff would have no idea whether this user s subjective judgment about response time is better or worse than the companies established Service Level standards. It should be noted that without a solution like i3 in place, the helpdesk individuals only recourse is to open a ticket and assign it to the SAP Basis team. In this case, our helpdesk personnel can first refer to a Foresight Exception report (Figure 19) which shows that this user has an established baseline dialog response time of.45 seconds, however, in the last few days this has jumped to over 9 seconds which is an increase of over 2000%. 19

20 Figure 19. Foresight Exception report for a top SAP user.. At this point it is most likely that our Helpdesk individual would still open a ticket and forward this to the Basis team. It is important to note, however, that this particular issue has already been through one level of triage. Had the users response time been within the Service Level threshold, the Helpdesk could have taken care of the issue. Upon receiving the trouble ticket, the Basis team views the user s exception report and utilizes i3 s in-context launch capability (the link in the upper right hand corner of the exception report) to automatically start Indepth for SAP taking us directly to the time period and transaction in question. Figure 20. High dialog response time for a specific transaction 20

21 Looking more closely at the previous screen, we can see that this user has a high transaction load and recently has been spending a larger amount of time waiting on the database to fulfill his requests. Figure 21. Graph shows recent increase in database response time. Now that we know that the users response time increase is due to a poor performing transaction in the database, we can begin exploring the problem in the database tier. Once again, with the integration available in i3 for SAP, we can launch Indepth for Oracle carrying the context of our current investigation into the database tier. Figure 22. Use VERITAS Indepth for Oracle to examine the SQL statements of the SAP transaction and its history. 21

22 The previous screen reveals that the sql statement that is associated with our poor performing transaction has very high I/O Wait Time. We can drill down further to the spindle level to investigate why exactly the i/o contention is taking place or how other storage related problems might be involved in the problem. Figure 23. Using VERITAS i 3 s Indepth for Oracle to Investigate a sql statement. To summarize, i3 began from a static high-level user report which indicated extremely high end-user response time. We launched our investigation in-context through the R/3 architecture including both the application server tier and then into Oracle. We found that the users primary workload over the last 10 days included a very poorly performing transaction in the last 2 days. Further, this transactions poor response time was directly attributed to poor database response time. The database problem was narrowed down to high database i/o wait times. In order to investigate the above problems we had to have the knowledge of correlating SAP R/3 all the way from the SAPGUI client through to the database and even into the storage array. We also had to correlate SAP transactions with invoked SQL statements. Having this information is fundamental for segmenting a response time problem across the various SAP tiers. 9. BENEFITS OF VERITAS i 3 From a business perspective, using VERITAS i 3 can significantly reduce performance slowdowns, achieving rapid Return On Investment (ROI) while maintaining low Total Cost of Ownership (TCO). VERITAS i 3 significantly helps organizations to: Understand who is using SAP, which applications and transactions are being used, when, and where Proactively detect potential problems before they negatively impact end-users Improve end-user satisfaction with the application Increase ROI - detect, find and solve problems quicker Reduce TCO significantly reduce the amount of staff, time, and effort involved in the problem management process. It also enables less experienced people to isolate and solve problems quickly Verify adherence to business level objectives Conduct organizational health checks through baselines and service level exceptions Effectively communicate the service level achievements and improvements to the organization Eliminate blamestorming between IT organizations 22

23 Manage and tune the application to effectively utilize system resources Use just-in-time capacity, deferring equipment and associated software licenses and support costs until they are actually needed Better understand future capacity requirements, based on current SAP usage and growth patterns In summary, VERITAS i 3 is an excellent vehicle to measure and improve performance objectives, providing all the necessary mechanisms to realize significant savings in cost and effort while effectively managing application performance. 10. SUMMARY Managing SAP performance is a challenge that requires a unique solution. SAP has become an essential part of many organizations this trend continues to grow with the introduction of Netweaver, which involves new technologies like non-its web access and Java. While this is a great advance for end-users improving business efficiency, productivity, and satisfaction it poses great challenges for SAP administrators and others that are responsible for the smooth management of these complex applications. Experience has shown a direct correlation between provided service levels, productivity (internal) and revenue generation (external). There is also a direct correlation of expenses incurred to achieve required service levels. The penalty for not meeting the users performance expectations may mean loss of business, loss of opportunity and failure to realize a ROI for the effort and operational expenses. Delivering the right level of service has become very challenging. Managing the new SAP architecture is fundamentally more difficult due the inherent complexity introduced by multiple tiers, a mixture of new and old technologies in the path (e.g. J2EE servlets and RFC function calls to traditional SAP basis technology), as well as database calls and storage access. VERITAS i 3 addresses these challenges and actually simplifies the deployment and ongoing maintenance of SAP systems. Using VERITAS i 3 helps companies manage service levels, contain costs, and maximize the efficiency of operational investments. VERITAS i 3 is the industry s first solution that provides a way to quickly and efficiently capture and correlate SAP application metrics like URLs, J2EE servlets, JCO, RFC calls, application server transaction activity, and SQL statements across the SAP infrastructure. It presents these important metrics in a manner that enables crisp communication, rapid proactive or reactive detection, correction and verification throughout the application life cycle. VERITAS i 3 sets a new milestone in SAP performance management by delivering a comprehensive integrated software solution that addresses the major challenges associated with the management of SAP multi-tier application components. 23

24 VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA or For additional information about VERITAS Software, its products, or the location of an office near you, please call our corporate headquarters or visit our Web site at 24