Ch. 13: Measuring Performance

Transcription

1 Ch. 13: Measuring Performance Kenneth Mitchell School of Computing & Engineering, University of Missouri-Kansas City, Kansas City, MO Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 1/3

2 Introduction Measuring performance is key in maintaining SLAs and preventing problems Input parameters describe system configuration, software environment, and workload of the system under study Representativeness of a model depends directly on the quality of the input data Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 2/3

3 Introduction Before one begins collecting data, three questions naturally arise: What are the information sources of performance data? What are the monitoring tools available for measuring response time and resource usage? What techniques are used to calculate model input parameters from typical performance data collected by standard measurement tools? Data are collected from different test points, carefully chosen Data will contain errors Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 3/3

4 Data Measurement data has 3 main uses: 1. Detection of operational problems such as high processor or link utilization, excessive paging activity, memory leaks, high error rates, high peak-to-average ration of number of concurrent users accessing the site etc. 2. Performance tuning such as optimizing OS performance or memory 3. Capacity planning to predict the performance of future systems Rely on runtime monitors Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 4/3

5 Measurement framework Framework depicted in Fig Two approaches to measuring a system Active measurement. Measurement agent uses the system and analyzes the performance Passive measurement. A device or an agent watches what the system does but does not add any requests to its workload. Measurements can be made at a single point or at many points Location of measurement agents should be carefully chosen and spread across the globe Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 5/3

6 Infrastructure measurement Measure resource utilization of servers and components of the site In Fig. 13.1, points E and F indicate data is being collected from the Web server and application layer Point G collects utilization data of the LAN segments Network sniffers can be located at point D (LAN to WAN) Operating system can be monitored at point E At point F a monitoring API can be embedded User perceived measurements can be collected at points A and B Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 6/3

7 Application Performance Measurement Quality of a web service is measured by the value of high-level application-oriented metrics Metrics are end-to-end response time, availability, and cost per application Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 7/3

8 Meaning of measurements Suppose you write an application that submits n = 100 requests to a search engine and records the response times The sample mean is defined as: R = 1 n i = 1 n R i The sample variance and standard deviation are: SR 2 = 1 n (R i n 1 R) 2 i=1 S R = 1 n (R i n 1 R) 2 i=1 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 8/3

9 Meaning of measurements Suppose we repeat the experiment m times and obtain m means. What is the true mean? We can make a probabilistic statement in the form of Pr[c 1 E( R) c 2 ] = 1 α E( R) is the population mean (c 1,c 2 ) is the confidence interval α is the significance level 100(1 α) is the confidence level (1 α) is the confidence coefficient Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 9/3

10 Meaning of measurements For small n we use the Students distribution For n > 30, a 100(1 α) confidence interval for E( R) can be computed as: ( R z 1 α/2 S R / n, R + z 1 α/2 S R / n) where R is the sample mean, S R the sample standard deviation, n the sample size, and z 1 α/2, the (1 α/2)-quantile of a unit normal variate z 1 α/2 = NORMSINV(1 α/2) Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 10/3

11 Meaning of measurements The minimum number of observations n to obtain an accuracy of r% on the estimate of the mean at a 100(1 α) confidence level is: ( ) 100 z1 α/2 S 2 R n = r R Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 11/3

12 Measurement Process What are the most used resources of a given system? Network bandwidth? Server I/O? Client desktop processor time? What is the average end-to-end response time of stock trading transactions? What does measuring really mean? Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 12/3

13 Measurement process measurement process involves 4 major steps: 1. Specify measurements: Decide on performance variables 2. Specify test points: Points from which performance data will be collected 3. Instrument and collect data: After selecting variables to be observed, one should deploy measurement tools to monitor the system 4. Analyze and transform data: Measurement tools gather large amounts of raw data. This must be distilled into meaningful information Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 13/3

14 Collection techniques and tools Event-driven: An interrupt that signals the arrival of a network packet can be viewed as an event that changes system state. At the OS level, the number of processes in the ready-queue, blocked-queue, and running can define the system. In event-mode, information is collected at the occurrence of certain events Event-handling overhead can be significant and create problems. Solution is to use dedicated hardware Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 14/3

15 Sampling Sampling is done at predefined instants Act of measuring can perturb the system based on number of variables sampled and the size of the sampling interval tradeoff between overhead and accuracy Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 15/3

16 Tools Monitors measure the level of activity in a system monitor must be an observer and not a participant Hardware monitor sensed predefined signals. Registers memory locations, I/O channels Software monitor consists of a set of routines embedded in the software with the aim of recording states and events in the system item Combination of hardware and software monitors is a hybrid monitor Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 16/3

17 Data Measurement data can be grouped into two categories: 1. System-level measurements: System wide resource statistics such as global CPU and disk utilization, total number of physical I/O operations, page fault rate, and total traffic through a router 2. Program-level measurements: Show program related information such as program identification, elapsed time, CPU time, number of I/O operations per execution, physical memory usage, and application traffic. Most information is from accounting systems Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 17/3

18 Accounting systems Collects information to be used as a means of apportioning charges to users of a system. Usually an integral part of multiuser operating systems Includes three groups of information: Identification (process, user, application), resource usage (CPU time, I/O operations, memory usage), and execution times DBMS and transaction processors may not be visible to accounting systems. So a special program may be needed to access Oracle Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 18/3

19 Program Analyzers Software tools especially designed to collect information about the execution of individual programs Can also be used as an optimization tool by pinpointing parts of a program that consume significant system resources For a transaction monitor can provide information on transaction count, mean number of I/O operations per transaction, and transaction mix Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 19/3

20 Logs Log files contain information about the services requested from a system, response provided and the origin of the requests. Web servers keep a log of their activities Numerous tools for analyzing Web server access logs Do not contain all information needed (i.e. transfer time) Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 20/3

21 Performance model parameters Parameters fall into 4 groups: Queues or devices, workload classes, workload intensity, service demands Step 1 is to define the queues that make up the model. Open or closed, component and system models Workloads are partitioned into classes. A class may be open or closed Workload intensity parameters give n indication of the number of transactions, requests, processes, or customers that contend for the resources of a network-based computing environment. λ for an open class and M and Z for a closed class Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 21/3

22 Model parameters Queueing models must take into account service demands that are not only hardware related Service demand includes time being serviced, but not waiting time D i = U i T /C 0 We have flexibility in interpreting U i Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 22/3

23 Example Suppose a system is monitored for a period of T = 10 minutes. The CPU is 90% busy. 80% by the OS kernel and 10% executing HTTP code. The number of HTTP requests during the measurement period was 30,000. What is the Demand on the CPU? D CPU = ( ) (10 60)/30, 000 = 0.018s Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 23/3

24 Parameter estimation Determining input parameters from measurement data There are 4 basic steps 1. Identify the type of execution environment. What are the software layers? 2. Specify the measurement process. Monitoring tools. Data to be obtained. Measurement interval 3. Monitor the system and collect performance data 4. Estimate input parameters. Calculate parameters Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 24/3

25 Parameter estimation cont. Service demands computed from measurement data Recall service demand law of a request of class r at resourcei D i,r = U i,r /X 0,r Need to determine utilization of each type of request Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 25/3

26 Parameter estimation cont. Ucpu,server os : utilization of the server process at the CPU measured by the operating system TotalCPUTime r : total CPU time for requests of class r measured by the operating system x 0,R : throughput of class r requests measured from a log generated by the server process or from a program analyzer at the server level Utilization of class r requests at the CPU Ucpu,r = U os cpu,server The service demand is: TotalCPUTime r r s=1 TotalCPUTime s Dcpu,r = Ucpu,r/X 0,r Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 26/3

27 Example Consider that an HTTP server received 21,600 requests during 1 hour as measured for the HTTP log. An analysis of this log places the requests in three categories according to file sizes: small (up to 20KB), medium (from 20 KB to 1 MB), and large (larger than 1 MB). Table 13.1 shows the total CPU time for each class of request as obtained form n=an instrumented HTTP server. It also shows the total number of requests processed in each category. If the overall utilization of the CPU measured by the OS was 86% and the machine was dedicated to being a Web server, what is the service demand at the CPU for each class of request? Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 27/3

28 Example Cont. Throughput per class are X small = 12, 960/3, 600 = 3.6 X medium = 6, 480/3, 600 = 1.8 X large = 2, 160/3, 600 = 0.6 The utilizations are U cpu,small = 0.86 U cpu,medium = 0.86 U cpu,large = , 555 1, = , = , = 0.11 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 28/3

29 Example Cont. The service demands per class of HTTP request are: D cpu,small = U cpu,small /X small = 0.46/3.60 = 0.13 D cpu,medium = U cpu,medium /X medium = 0.29/1.80 = 0 D cpu,large = U cpu,large /X large = 0.11/0.60 = 0.18 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 29/3

30 Obtaining model parameters Network management system SNMP SNMP consists of a manager and an agent Information is organized into a tree structure known as a management information based MIB Performance variables are logically grouped into branches. Interface, IP and TCP passive data collection: snoop, netramet, tcpdump, tcptrace, and libcoral active data collection: ping (RTT), timit (Web server) Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 30/3

31 Measurement tools LAN analyzers can only collect a few minutes of traffic remote monitoring MIB (RMON) Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 31/3

32 Servers Measurements can be collected at several layers (Fig. 13.5) Windows NT: NT performance monitor measures behavior of computer objects Counters organized for object type. Table 13.3 Unix: System activity reporting (SAR) records cumulative and average activity table ps, netstat, etc Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 32/3

33 Conclusion Performance models are useful for understanding and predicting the quantitative behavior of Web services Depends directly on quality of input parameters Performance measurement and parameter estimation Set of guidelines for converting performance measures into input parameters Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 33/3