Two-Tier Oracle Application

Transcription

1 Two-Tier Oracle Application This tutorial shows how to use ACE to analyze application behavior and to determine the root causes of poor application performance. Overview Employees in a satellite location of a large company access the Oracle server located in the company s headquarters. The connection is relatively fast they use a high-bandwidth, low-latency Metropolitan Area Network (MAN) with less than 4 ms latency. However, they complain that the access is slow and it takes more than 10 seconds to query the database. Your task is to determine the cause of the large application delay and suggest methods to improve the application performance. Typical reasons for a slow application performance include: A slow database server A slow network connection Bad application design 1

2 Network Model The following figure shows the network infrastructure: Oracle Client 100 Mbps LAN Metropolitan Area Network (MAN) 100 Mbps LAN Client Probe 10 Mbps 3.2 ms latency Oracle Server Data Center Probe Approach The following list summarizes a typical workflow for troubleshooting an application s performance. You will also follow this workflow in this tutorial. 1) Capture traffic 2) Filter irrelevant traffic and import the capture files into ACE 3) Visualize the application and network traffic 4) Analyze the application with AppDoctor 5) Analyze the data exchange 6) Suggest steps for improving the response time 2

3 The process starts with capturing the traffic and importing it into ACE. The traffic is then analyzed using AppDoctor, visualizing the data flow and looking into packet exchange. 3

4 Step 1: Capture Traffic Application capture files (also known as packet trace files ) form the raw data for an ACE file. The Application Capture utility enables you to capture traffic at multiple network locations simultaneously. You can install an unlimited number of capture agents throughout your network; then you can manage these agents and capture traffic directly from your local computer. To get a correct estimate of the network latency, you must capture traffic on both sides of the transaction in this case at both the company headquarters (data server location) and at the satellite office (database client location). ACE can then identify identical packets in the two captures and give you a correct breakdown of individual delay components such as network and server delay. Note In some cases, it may not be feasible to capture traffic at both endpoints of the network. Instead, you can capture on one side of the connection and provide network characteristics, such as bandwidth and latency. 4

5 The application traffic has already been captured for you. The traffic is stored in two files, each for one side of the connection. 5

6 Step 2: Import Capture Files into ACE After you have captured the traffic, you can import it into ACE. After you specify the capture files and related information, ACE detects that they contain the same traffic, merge the traces, visualize and analyze the application. To import the traffic into ACE, perform the following steps. Creating a New ACE File 1 Select File > New. Select ACE Files (*.atc.m) from the pull-down menu and click OK. The ACE Import: Choose Capture File(s) dialog box appears. This dialog box prompts you to specify the capture files, capture agent 6

7 locations, and any filters and preprocessing to be applied to the data before it is further processed. Specifying Capture Files Two capture files have been created. To import them into ACE, perform the following steps. 1 Click Add Capture File, navigate to the <reldir>\models\std\tutorial_req\module\ directory, and add the file Oracle_client_unfiltered.enc. <reldir> is the release directory where ACE Plus is installed. In the Windows environment this is often C:\Program Files\OPNET\<release number>. 7

8 2 Click Add Capture File, navigate to the <reldir>\models\std\tutorial_req\module\ directory, and add the file Oracle_server.enc. Specifying the Locations Where Captures were Taken It is often helpful to enter friendly names for capture agent locations, especially if you have multiple capture files. 1 Enter Client Probe in the Capture Agent Location field for the Oracle_client_unfiltered.enc trace. 2 Enter Data Center in the Capture Agent Location field for the Oracle_server.enc trace. Specifying Filters and Preprocessors The Filter column enables you to specify a filter to prevent irrelevant traffic in the capture file from being imported. The filter can be based on address, port, or protocol. The Default filter imports all TCP, UDP, and ICMP traffic and filters all broadcast traffic. To import other types of traffic, such as SNA, you must change the filter. Because the imported traffic was running over TCP, you do not need to set up a filter. 8

9 The Preprocessor column enables you to call an external application to translate the capture file type into one of the supported ACE formats. Again, this is not necessary now because ACE can read these capture files directly. Filtering Irrelevant Traffic You have specified the traces to be imported. The next step is to filter irrelevant traffic from the capture files. ACE is designed to analyze one user-level transaction. You must provide valid data; this means that the capture files should include only traffic that is relevant to the transaction of interest, with all other traffic excluded. You must filter out any extraneous packets during the import; otherwise, ACE considers these extraneous packets to be part of the application. Ideally, you should create one or more capture files that represent one and only one user-level transaction. In reality, however, you probably have other applications and traffic running in the same network. To obtain valid results, you must now filter irrelevant traffic from the ACE file. 9

10 You can identify and remove irrelevant traffic using the ACE Summary window. This window visualizes and sorts traffic based on different groupings such as protocol, tier pairs, and connections. You are interested in analyzing Oracle database traffic between the Oracle Server and Oracle client tiers; therefore, you can filter out all other traffic. 1 Click Next > in the Choose Capture Files dialog box. The ACE Summary window appears. The Tier Pair Circle (bottom) pane shows many tier pairs exchanging traffic. The Filter Irrelevant Traffic dialog box appears in front of the ACE Summary window. This dialog box enables you to filter out certain types of traffic if you know that they are irrelevant. 10

11 We know that the types of traffic shown are irrelevant to the application, so we will accept the default settings to filter out this traffic. 2 Click Filter. The ACE Summary window shows that there are now two tier pairs remaining after the filter operation. The Select Next Step dialog box appears. This dialog box enables you to bypass the ACE Summary window if you know that your capture data contains no irrelevant traffic. In this case, we know that the application of interest involves one tier pair only. However, the ACE Summary window shows two tier pairs. We conclude that additional filtering is necessary. 3 In the Select Next Step dialog box, select the ACE Summary: option. 11

12 4 Click Next >. The Tier Pair Circle (bottom center) pane shows that the tiers and are exchanging POP (Post Office Protocol) traffic a protocol used to transfer . This is not relevant to the Oracle application we want to examine in ACE. We can also see that the tiers and are exchanging SQLNET traffic, which is the protocol used by Oracle. We therefore decide to include the traffic for this tier pair and, by extension, exclude all other traffic 12

13 from the import. (Note: If you do not have a license for ACE Decode Module, the SQLNET traffic appears as TCP traffic.) 5 In the Traffic drilldown: (top left) pane, click on the < > tier pair so that it is the only tier pair that is selected. The Tier Pair Circle (bottom) pane updates to show information about the selected tier pair only. 13

14 6 In the Traffic drilldown: pane, click on the checkbox next to < > to include this tier pair. A checkbox appears next to the tier pair to show that all traffic for this tier pair is included. 7 Click the Analyze Included in ACE button (bottom right). The Rename Hosts window appears. Renaming Hosts At this point, ACE has imported the capture data, identified all tiers by their IP addresses, and filtered out extraneous traffic. The next step is to give each tier a friendly name. When a tier is renamed, subsequent references use the friendly name rather than the IP address. 1 Rename to Oracle Server. 14

15 2 Rename to Oracle Client. The table should look like this: 3 Click Next >. The Specify Tier Locations dialog box appears. Specifying Tier Locations This dialog box is used to enter information about the location of tiers with respect to a capture location. This is important to ensure that the network topology is represented correctly in ACE. In many cases especially when capture files are collected on both sides of a connection ACE detects the tier location. In this import, ACE detected that the Oracle Client tier is local to the Client Probe (as seen in the Network Model) and that the Oracle Server tier is local to the Data Center. These are the only two tiers of interest in this study, so you do not need to change anything in this dialog box. 15

16 1 Click Next>. The Specify Bandwidth Between Locations dialog box appears. ACE requires an accurate representation of the network where the application was captured. This means that you need to provide ACE with the relevant bandwidth and latency information in each segment and between each location in the transaction. Depending on your network configuration, you might need to specify both local bandwidths and remote bandwidths. In this import, you need to specify only the remote bandwidth between the Data Center and the Client Probe. Specifying Bandwidth between Locations This dialog box is used to characterize the network between capture locations that do not belong to the same LAN. 16

17 The bandwidth should represent the smallest link between the two locations. In most cases, this is the last mile bandwidth (T1, DSL, Frame Circuit etc.). ACE can use PathProbe to measure the bandwidth and latency refer to the documentation for more information about this utility. 1 Set the Bandwidth to 10,000 (10,000 kbps equals to 10Mbps). This is the bandwidth of the slowest link in the network (refer to the Example Topology picture in Network Model). 2 Click Next >. The Specify Latency between Locations dialog box appears. Specifying Latency between Locations When you import multiple capture files, ACE can calculate the latency between tiers automatically based on the bandwidth you specified. ACE can use PathProbe to measure the bandwidth and latency; refer to the documentation for more information about this utility. 1 Click Finish. 17

18 If the Getting Started Analyzing an Application Transaction window appears, click Close. The capture files have been imported and a new ACE window appears. 18

19 Step 3: Visualize the Application and Network Traffic You have imported the capture files into ACE and filtered out the irrelevant traffic. You are now ready to view the traffic and analyze the application in more detail. The first step is to view the Data Exchange Chart, which shows the application and network packets between tiers on a time line. 1 If the Data Exchange Chart is not already showing, click the Data Exchange Chart toolbar button. Data Exchange Chart 19

20 Each individual application message is represented by an arrow that starts at the source tier and ends at the destination tier. The arrow color represents the number of application bytes in the message. Each colored bar represents a group of messages; the amount of color represents the percentage of messages that fall into each application-payload size category. You can observe that about one-third of the messages are yellow ( application bytes per message) and about two-thirds are orange (1-100 application bytes per message). You can immediately conclude that the application is sending many small messages. 2 To differentiate the messages flowing in different directions, select View > Split Groups. Now the messages are split into two groups and you can detect more details. The upper group represents messages from the client to the server and the lower group represents messages from the server to the client. In the server-to-client message group, approximately 80 percent of the messages are orange. 20

21 In the client-to-server message group, approximately 70 percent of the messages are orange. 3 View message-group information by resting the mouse over the first group of messages. The tooltip shows that the message group represents 183 messages in each direction. 4 Notice that the data flow is about 12 seconds long: it starts at about 0 seconds and ends at about 12 seconds. This matches the response time reported by employees (more than 10 seconds). 21

22 After this initial investigation, you can perform more quantitative analysis using AppDoctor. 22

23 Step 4: Analyze with AppDoctor We will first look at AppDoctor s Summary of Delay to see how individual delays contribute to the total application response time. AppDoctor s Diagnosis will then identify the bottlenecks. Finally, more quantitative analysis will be performed looking at AppDoctor s Statistics. Examining the Summary of Delays AppDoctor s Summary of Delays provides insight into the root cause of the overall application delay. It divides the total application response time into five components: Tier Processing Delay This is the total time it takes to process the application at each tier. This time includes CPU processing time and user think time. Latency Delay due to latency in the network (latency is the time required for 1 bit to be transmitted across the network). Bandwidth delay Delay caused by the limited bandwidth of the network. Congestion delay This delay is a metric of network restriction to packet flow. It might be caused by packet queuing in the network. 23

24 Protocol delay Delay might be caused by flow-control mechanisms imposed by network protocols. TCP, for example, has several built-in flow-control mechanisms. Now you will examine individual delay components to determine which type of delay contributes most to the overall response time. 1 To see how individual delays contribute to the total application response time, select AppDoctor > Summary of Delays. Notice that the largest contributing factor to the application delay is the latency delay. It accounts for almost 60 percent of the total response time. 24

25 This is somewhat surprising because the measured latency between the client and the server is less than 4 ms. Diagnosing Bottlenecks AppDoctor Diagnosis provides a more detailed view of the potential bottlenecks affecting this transaction. Diagnosis tests the current transaction against issues that often cause performance problems in network-based applications, grouped by category. Values that exceed specific (user-configurable) thresholds are marked as bottlenecks or potential bottlenecks. 1 Click the Diagnosis tab to display the diagnosis. 25

26 The following categories are identified as bottlenecks: Protocol Overhead Chattiness Network Effect of Chattiness Effect of Latency 2 Click on one of the Bottleneck fields for Chattiness to see the definition of chattiness and bottleneck threshold in the lower window. 3 Repeat the previous step to learn more about Protocol Overhead, Network Effect of Chattiness, and Effect of Latency, which were also identified as bottlenecks. Note that the Chattiness, Network Effect of Chattiness, and Effect of Latency bottlenecks are related to sending many small application messages. 26

27 Performing Quantitative Analysis AppDoctor also provides summary statistics for the application transaction. 1 Click the Statistics tab to see the application statistics. Notice that the Response Time is seconds, which corresponds to the value that you saw previously in the Data Exchange Chart (step 4). Several statistics are relevant to this study. You will examine two in particular: the number of application turns and the maximum data exchanged in one turn. 27

28 Each change in direction is called an application turn because the application changes the direction of data flow. Applications with many application turns are generally considered chatty and are sensitive to network delay. The sensitivity occurs because each message must be received at a tier before the corresponding response is sent; as a result, each message is affected by network latency. Notice that the application used 2,157 turns to exchange 182,056 Bytes of data (see the Application Turns statistics). Notice also that the maximum amount of data sent in one application turn is 258 bytes in one direction (Oracle Client to Database Server) and 455 bytes in the other direction (see the Max Application Bytes per Turn statistics). This confirms the previous analysis in which chattiness and the network effect of chattiness was identified as the primary cause of the poor response time. Also note that the Effect of Latency is over 6 seconds. Here you see that the effect of latency alone, aggravated by the 2,157 turns, accounts for over 6 seconds of the total transaction response time. Because latency is largely a product of geographic distance and network hops, adding bandwidth will have a minimal effect on the response time. To minimize this component, 28

29 you can reduce the latency on the circuit or reduce the number of application turns. Latency is a physical constraint; therefore, changing the application behavior is more practical. The previous analysis suggested that the chattiness is the largest contributing factor to the application delay. Now you will go back to the Data Exchange Chart to examine packet exchange in more detail. 2 Click Close to close AppDoctor. 29

30 Step 5: Analyze the Data Exchange AppDoctor revealed issues with both the network and the data exchange between tiers. Now that you know more about the possible issues, the Data Exchange Chart might provide additional insight. You will now examine the capture data to see how packets are exchanged. First you will examine the beginning of the transaction (that is, the traffic between 0 and 0.15 seconds) more carefully. 1 Uncheck View > Split Groups to see the Data Exchange Chart for both directions. 2 Zoom to the area of chart that represents the second time period: Choose View > Set Visible Time Range, set Start time to 0 and End time to 0.15, and click OK. As you zoom into the Data Exchange Chart, the groups of messages resolve into the individual application messages. 30

31 3 Study the individual messages. Each arrowhead indicates the message direction. The data exchange exhibits a simple request-response pattern. Database Client sends a request, then Database Server receives the request and sends a response immediately. You conclude that the Data Exchange Chart confirms the analysis predicted by Summary of Delays and Diagnosis: The application is chatty, and the chattiness means that the network delay slows the application significantly. 31

32 At this point you suspect that the application was not written efficiently and that it should be rewritten. To improve the application response time, you might want to talk to the application developers to see whether there is a way to optimize the application design to send fewer requests. Examining ACE Decodes ACE Decode Module (ADM) adds information about the content of the packet. By understanding packet contents, you can better understand the particulars of the application itself. For example, you can examine the database query statements to determine whether records are being accessed efficiently. Skip this section if you do not have this module. In the following steps, you will view the contents of individual messages. 1 Right-click in the workspace (not on messages) in the Data Exchange Chart and select Full Zoom. The entire Data Exchange Chart reappears. 32

33 2 Right-click on the first (left-most) application message group and select Show Protocol Decodes for Selected Items from the pop-up menu. The Protocol Decodes Viewer appears. 33

34 3 Click on Frame 59 in the top pane. The decode for this frame appears in the bottom pane. The Protocol Decode Viewer shows decodes for all selected messages. The top pane shows Source, Destination, Size, Send Time, Receive Time, Decode and Decode Summary. The bottom pane shows the detailed decode for the selected frame. You can drill down further into the individual packets to see more details. The most detailed decodes appear by drilling into the individual protocol (TCP, IP, SQL, etc.). 34

35 4 Expand the SQLNET decode in the lower pane by clicking on the plus sign next to it. The decode shows the SQL Parse statement that is carried by this packet. You can also drill into the other protocols (TCP, IP, Ethernet) to see the content of the packet. You can use the decoder to examine the application design and to pinpoint the cause of the problem. Examining several request/response patterns typically does this. 35

36 Step 6: Suggest Steps for Improving the Response Time This tutorial showed how you can use ACE to visualize and diagnose application performance problems. Initially, you knew only that the application response time was more than 10 seconds. Now you know that the slow response was primarily due to application chattiness, aggravated by network latency. You could see that, although the latency between client and server was only 3.11 ms, that delay became significant because it was experienced for each of the many application turns. In this case, the best solution is to rewrite the application so that it transfers fewer, larger messages. Using the graphs, statistics, and protocol decodes provided in ACE, you can persuade the database developers to change the database transaction as required. 36