Ch. 7: Benchmarks and Performance Tests

Transcription

1 Ch. 7: Benchmarks and Performance Tests Kenneth Mitchell School of Computing & Engineering, University of Missouri-Kansas City, Kansas City, MO Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 1/3

2 Introduction Best way to study performance is to run actual workload on hardware platform This is not feasible in many cases Alternative approach in two cases 1. Benchmarking: Running representative programs on different hardware and measuring the results 2. Performance testing: Predict performance for specific scenarios and applications Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 2/3

3 Nature of Benchmarks Time and rate are basic measures of system performance Also concerned with cost Processor speed in MIPS, but real performance may vary greatly due to architecture First step is to answer the following: 1. What is a particular benchmark actually testing? 2. How does the benchmark resemble the user environment workload? 3. What is the benchmark really measuring? Benchmarks are good tools for comparing systems, but not accurate tools for sizing or capacity planning for a given Web service Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 3/3

4 Benchmark Hierarchy Coarse grained and fine grained E-commerce system: Coarse-grained Different granularities is hierarchy 1. Basic operations: Addition or Multiplication. Dhrystone 2. Toy benchmarks: Towers of Hanoi etc. (not helpful in predicting real performance) 3. Kernels: Bits of real code to measure processor performance. Livermore Loops and Linpack 4. Real programs: Used to solve reap problems. SPEC, TPC Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 4/3

5 Avoiding Pitfalls First step is to understand the environment. Where and how the benchmark tests were carried out Processor specification (model cache and number of processors), memory, I/O subsystem (e.g., disks, models and speed), network (e.g. LAN WAN, and operational conditions), and software (e.g., operating system, compilers, transaction processing monitor, and database management system) 1. Is the configuration similar to the actual system? 2. How representative of the actual workload are the benchmark tests? Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 5/3

6 Common Benchmarks Can be found through users groups and Web searches 1. Relevance: Provide meaningful measures within problem domain 2. Understandable: The benchmark results should be easy to understand 3. Scalable: Must be applicable to a wide range of systems, in terms of cost, performance, and configuration 4. Acceptable: The benchmarks should present unbiased results recognized by users and vendors Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 6/3

7 Common Benchmarks Two consortia are System Performance Evaluation Corporation (SPEC) and Transaction Processing Performance Council (TPC) SPECxxxx where xxxx is the generation. Two suites: int and fp. Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 7/3

8 Workload SPEC are selected from academic and scientific programs fp are written in FORTRAN and C int are written in C and C++. See Table 7.1 used to calculate results in next session Performance is relative to a 300-MHz Sun Ultra5 1 0, which gets a score of 100 (year 2000) Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 8/3

9 Results For measuring speed, each benchmark test has its own ratio reference time ratio = run time SPECint is the mean of 12 normalized ratios, one for each program in Table 7.1 In like manner are calculated rates and values for floating point Summarized in Table 7.2 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 9/3

10 Example Suppose you are considering upgrading to a new webserver with faster CPU and larger cache Ratio of new vs old is 489/363 = 1.35 New CPU service demand is D old /1.35 New CPU time is smaller and can be used in performance models (Chapter 9 and 10) Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 10/3

11 Web Benchmarks Set of tests to measure Web servers Webstone, SPECweb, and Scalable URL Reference Generator (SURGE) Measurements carried out on small isolated LANs Real analysis must take network congestion, errors and latency into account Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 11/3

12 SPECweb Measures max simultaneous connections that a Web server is able to support while meeting specific throughout and error requirements Both static and dynamic generated HTML support for HTTP 1.1 POSIX Unix of Windows NT Prime client coordinates and executes requests on client machines and gathers data Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 12/3

13 SPECweb Workload is drawn from large Web servers at popular websites Different types of requests in Table 7.3 Different file sizes in Table 7.4 Measures the number of active load generating threads Table 7.5 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 13/3

14 Example A media company is planning to revamp its portal. The systems administrator considers that the SPECweb workload could be used as an approximation for the initial workload of the company web site. Estimates that the number of concurrent customers, N, will be 10,000 IT management defines a limit of 4 seconds for the average user perceived response time The business analyst estimates that the average think time, Z for a customer is 3 seconds, therefore: X 0 N R + Z = 10, = 1429requests/sec Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 14/3

15 Example What is the average number of simultaneous requests generated by the 10,000 customers? Consider network time and site time Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 15/3

16 Example cont. Avg. network time is around 1.2 sec Therefore web site time, R site, should not exceed 2.8 seconds. Using Little s law, the average number of simultaneous connections Nconn is Nconn = X 0 R site = = 4, 001 By examining the SPECweb benchmark results the analyst was able to find a system to support 4,001 connections with a throughput of 1429 requests/sec. Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 16/3

17 Webstone Configurable C/S benchmark for HTTP servers Max. throughput and avg. response time Master process spawns client processes generating HTTP requests to server 4 synthetic page mixes to model workload, file sizes and access frequencies Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 17/3

18 Webstone Specification parameters: 1. Number of clients that request pages 2. Type of page defined by file access and frequency 3. Number of pages available on the server under test 4. The number of client machines, where client processes execute on. Also tests dynamic pages. HTML, CGI, and API. Both NSAPI and ISAPI Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 18/3

19 Results Results are discussed in Table 7.6 Little s load factor (LLF) should be the same as the average number of connections Lower value means that the server is overloaded From table: 12099/(10 60) ( ) = Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 19/3

20 Example Capacity planner wants to size the outgoing links for the above server During 10 minute test the server had 4,576 requests for at an average HTTP request size of 100 bytes. Incoming traffic was: (4, )/(10 60) = bps Outgoing traffic is given by server throughput: (215, 181 8) = 1.72Mbps Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 20/3

21 Analytic-Based generators Used to generate workloads (SURGE) Web distributions frequently follow a power law: P[X > x] x α 0 < α 2 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 21/3

22 Analytic-Based generators Parameter values: 1. File sizes: Lognormal body and Pareto tail p(x) = 1 xσ 2π x µ) e (ln 2 /2σ 2 p(x) = αk α x (α+1) 2. Request sizes: Different by popularity follow a Pareto dist. 3. Popularity: Zipf s law N(P) = kr(p) 1 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 22/3

23 Analytic-Based generators Parameter values: 1. Embedded references: Pareto 2. Temporal Locality: Stack distance - Lognormal 3. Off times: Idle times of the processes. Weibull distribution p(x) = bxb 1 a b e (x/a)b where b is the shape and a is the scale parameter Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 23/3

24 System Benchmarks Measure entire system. Processor, I/O, network, database, compilers, and OS. TPC: Processor, I/O, network, OS, database management system, and transaction monitor. 4 types of benchmarks: C, H, R, and W TPC-H and TPC-R evaluate the price/performance ratio of a given system executing decision support applications Long and complex queries against databases Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 24/3

25 TPC-C Transaction for orders to wholesale supplier Workload: New-order, Payment, Delivery, Order-Status, Stock-level Results are shown in table 7.7 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 25/3

26 TPC-C Example Manager is considering new software The performance of current and new software is 30,000 and 36,000 respectively for a ratio of 1.2 Will be analyzed in chapters 8 and 9 Cost is also an issue, First is $10.03 and the second is $12.29, so new software is 22.5% higher than the current software. Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 26/3

27 TPC-W Evaluates sites that supports E-business activities Retail store that sells products and services over the Internet Workload 1. Browse: Home, Browse, Select, Product detail, and Search 2. Order: Shopping Cart, Login, Buy Request, Buy Confirm, Order Inquiry, and Order Display Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 27/3

28 TCP-W Three different types of session profiles 1. Browsing mix: 95% browse 5% order. 0.69% buy/visit ratio 2. Shopping mix: 80% browse 20% order. 1.2% buy/visit ratio 3. Ordering mix: 50% browse 50% order % buy/visit ratio Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 28/3

29 Results 3 throughput metrics. WIPS Web Interactions per Second See Table 7.8 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 29/3

30 Performance Testing Steady state as well as Peak load activity Three types based on load intensity: 1. Load testing: Simulated load mimics regular regimen of operation 2. Stress testing: Worst-case scenarios, load heavier than expected 3. Spike testing: Short periods of time where load is many times normal Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 30/3

31 Performance Testing various modes Component monitoring mode: Locally generated load Simulated network mode: load is generated at several locations on backbone Distributed peer-to-peer mode: load is generated ant many lightweight clients. End user perception of performance. Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 31/3

32 Methodology Main steps in Fig Defining testing objectives Determine Web server capacity Max. number of concurrent users within limits of SLA Capacity of application layer Bottlenecks in Web service infrastructure Network impact on end-user perceived response time Capacity of DB server Identify most expensive Web functions Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 32/3

33 Environment 2. Understanding the environment Infrastructure, software, network connectivity, network protocols, etc. 3. Specifying the test plan Should provide a detailed audit trail How are input variables controlled or changed? What is desired degree of confidence in the measurements? 4. Specifying the workload Create scripts that mimic user behavior and generate the load Customer profiles Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 33/3

34 Testing 5. Setting up the test environment Installing measurement and testing tools Manual vs. automatic testing 6. Running the tests Execution should follow performance plan Document discrepancy between expected and actual results Prepare detailed description Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 34/3

35 Testing 7. Analyzing the results Most important step Should be able to determine bottleneck, source of performance problem Diagnose and specify new tests Remove bottlenecks See example 7.6 Kenneth Mitchell, CS & EE dept., SCE, UMKC p. 35/3