1.6 Computer Performance

Transcription

1 1.6 Computer Performance Performance How do we measure performance? Define Metrics Benchmarking Choose programs to evaluate performance Performance summary Fallacies and Pitfalls How to avoid getting fooled by performance calculations and demos! 1 Defining Performance Which airplane has the best performance? Airplane Passengers Range (mi) Speed (mph) Boeing Boeing BAC/Sud Concorde Douglas DC How much faster is the Concorde compared to the 747? How much bigger is the 747 than the Douglas DC-8? Concord is the fastest; DC-8 has the longest range, Performance means different things to different people! 2

2 Performance of computers Which computer is the fastest? For which application? Performance cannot be separated from a particular application or application class Scientific FP performance Commercial memory and I/O Program development Integer performance 3 Performance What s important to who? Individual computer user minimize elapsed time for program = end_time start_time Also called response time, execution time Computer center manager maximize completion rate = #jobs completed/second Called throughput 4

3 Definitions Performance is in units of things per sec bigger is better If we are primarily concerned with response time performance(x) = 1 execution_time(x) " X is n times faster than Y " means n = performance(x) performance(y) Speed-up = time_old / time_new 5 Relative Performance If an Pentium III runs a program in 8 seconds and a PowerPC runs the same program in 10 seconds, how many times faster is the Pentium III? n = 10 / 8 = 1.25 times faster (or 25% faster) Why might someone chose to buy the PowerPC in this case? 6

4 Elapsed Time / response time / real time counts everything (disk and memory accesses, I/O, OS, etc.) a useful number, but often not good for comparison purposes Alternative: just time processor (CPU) is working only on your program (since multiple processes running at same time) CPU time Measuring Performance doesn't count I/O or time spent running other programs can be broken up into system time (in OS), and user time (in user program) Our focus: user CPU time time spent executing the lines of code that are "in" our program 7 Different aspects cascade% time myprogram 90.7u 12.9s 2:39 65% User CPU time 90.7 seconds System CPU time 12.9 seconds Elapsed time is 2 minutes and 39 seconds Percentage of elapsed time that is CPU time is 65% 8

5 How to Measure Time? User Time seconds CPU Time: Computers constructed using a clock that runs at a constant rate and determines when events take place in the hardware These discrete time intervals called clock cycles (or informally clocks or cycles) Length of clock period: clock cycle time and clock rate which is the inverse of the clock period; Cycle rate vs. period (cycle time) 500MHz P-III runs 500M cycles/sec 1 clock cycle = 2ns 2GHz P-IV runs 2G cycles/sec 1 clock cycle = 0.5ns 9 CPU performance and its factors The time to execute a given program can be computed as CPU time = CPU clock cycles for a program x clock cycle time = CPU clock cycles for a program / clock rate The number of CPU clock cycles can be determined by CPU clock cycles = (instructions/program) which gives x (clock cycles/instruction) = Instruction count x CPI = IC x CPI CPU time = IC x CPI x CC = IC x CPI / CR The units for this are instructions ciock cycles seconds seconds = x x program instruction clock cycle 10

6 Example CPU time = CPU clock cycles for a program / clock rate Our program runs in 10 seconds on computer A, which has a 400MHz clock. We are trying to help a computer designer build a machine B that will run the same program in 6 seconds. The designer has determined that a substantial increase in the clock rate is possible, but this increase will affect the rest of the CPU design, causing machine B to require 1.2 times as many clock cycles as machine A for this program, what clock rate should we tell the designer to target? CPU time (A) = clock cycles(a) / clock rate (A) CPU time (B) = clock cycles(b) / clock rate (B) Known: CPU time (A) = 10 s, CPU time (B) = 6 s, clock cycles (B) = 1.2 * clock cycles(a), clock rate (A) = 400MHz, Find: clock rate (B) =? 11 How to Calculate the 3 Components? Clock Cycle Time: in specification of computer (Clock Rate in ad) Instruction Count: Count instructions in loop of small program Use simulator to count instructions Hardware counter in spec. register (Pentium II) CPI: Calculate: cpu Clock cycles Instruction Count Hardware counter in special register (PII) 12

7 Examples If a computer has a CR= 50 MHz, how long does it take to execute a program with 1,000 instructions, if the CPI for the program is 3.5? CPU time = IC x CPI / CR = 1000 x 3.5 / (50 x 10-6 ) If a computer s clock rate increases from 200 MHz to 250 MHz and the other factors 6 remain the same, how many times faster will the computer be? CPU time old clock rate new 250 MHz = = = 1.25 CPU time new clock rate old 200 MHZ 13 Example 2 Two implementation of the same ISA: Cycletime CPI Machine A 1 ns 2.0 Machine B 2 ns 1.2 Which is faster? Assuming I instructions Performance(A) Performance(B) = ExecutionTime(B) ExecutionTime(A) = I I = =1.2 14

8 Computing CPI The CPI is the average number of cycles per instruction. If for each instruction type, we know its frequency and number of cycles need to execute it, Example CPI = n CPI j F j j=1 Op Freq i CPI i Prod (% Time) ALU 50% 1.5 (23%) Load 20% (45%) Store 10% 3.3 (14%) Branch 20% 2.4 (18%) Instruction Mix CPI= 2.2 (Where time spent) 15 Do not confuse with CPU clock cycles! CPI = n CPI j F j j=1 A compiler designer is trying to decide between two code sequences for a particular machine. The hardware designer has supplied the following facts: for instruction class A, B, and C, their CPI are 1, 2 and 3, respectively. For a particular high-level language statement, the compiler writer is considering two code sequences that require the following instruction counts: Code sequence instruction counts for instruction class A B C Which code sequence executes the most instructions? Which will be faster? What is the CPI for each sequence? # of instructions: Seq1: Seq2: faster? CPU clock cycles: 2*1+1*2+2*3 4*1+1*2+1*3 CPI: clock cycles / IC n CPU clock cycles = CPI j IC j=1 j 16

9 How to improve performance? Iron Law Instructions Cycles Time = X X Program Instruction Cycle (IC) (CPI) (cycle time) Compiler Designer Processor Designer Chip Designer Instructions/Program Instructions executed, not static code size Determined by algorithm, compiler, ISA Cycles/Instruction Determined by ISA and CPU organization Overlap among instructions reduces this term Time/cycle Determined by technology, organization, clever circuit design 17 Factors affecting CPU Performance Which factors are affected by each of the following? CPU time = Seconds = Instructions x Cycles x Seconds Program Program Instruction Cycle IC CPI clock rate Program x Compiler x x Instr. Set Arch. X x Organization x x Technology x 18

10 1.9 Evaluating Performance: Benchmarking Benchmarks Programs chosen to measure performance Predict performance of actual workload saves effort and money, Representative? Honest? SPEC System Performance Evaluation Cooperative Dozen benchmarks Which programs? Best case - you run the same set of programs everyday Or, typical of expected class of applications e.g., compilers/editors, scientific applications, graphics, etc. Small benchmarks nice for architects and designers easy to standardize, can be abused 19 Benchmarks: SPEC CPU2006 System Performance Evaluation Cooperative Formed in 80s to combat benchmarketing SPEC89, SPEC92, SPEC95, SPEC2000, SPEC2006 For CPU time performance only 12 integer and 17 floating-point programs Sun Ultra-5 300MHz reference machine has score of 100 Report Geometric mean of ratios to reference machine 20

11 Benchmarks: SPEC CINT2000 Benchmark 164.gzip 175.vpr 176.gcc 181.mcf 186.crafty 197.parser 252.eon 253.perlbmk 254.gap 255.vortex 256.bzip2 300.twolf Description Compression FPGA place and route C compiler Combinatorial optimization Chess Word processing, grammatical analysis Visualization (ray tracing) PERL script execution Group theory interpreter Object-oriented database Compression Place and route simulator 21 Benchmarks: SPEC CFP2000 Benchmark 168.wupwise 171.swim 172.mgrid 173.applu 177.mesa 178.galgel 179.art 183.equake 187.facerec 188.ammp 189.lucas 191.fma3d 200.sixtrack 301.apsi Description Physics/Quantum Chromodynamics Shallow water modeling Multi-grid solver: 3D potential field Parabolic/elliptic PDE 3-D graphics library Computational Fluid Dynamics Image Recognition/Neural Networks Seismic Wave Propagation Simulation Image processing: face recognition Computational chemistry Number theory/primality testing Finite-element Crash Simulation High energy nuclear physics accelerator design Meteorology: Pollutant distribution 22

12 SPEC CINT2000 and CFP200 ratings for the Pentium III and P 4 23 CINT2006 for Opteron X Name Description IC 10 9 CPI Tc (ns) Exec time Ref time SPECratio perl Interpreted string processing 2, , bzip2 Block-sorting compression 2, , gcc GNU C Compiler 1, , mcf Combinatorial optimization ,345 9, go Go game (AI) 1, , hmmer Search gene sequence 2, , sjeng Chess game (AI) 2, , libquantum Quantum computer simulation 1, ,047 20, h264avc Video compression 3, , omnetpp Discrete event simulation , astar Games/path finding 1, , xalancbmk XML parsing 1, ,143 6, Geometric mean 11.7 High cache miss rates 24

13 SPEC INT 2006 Example FIGURE 1.18 SPECINTC2006 benchmarks running on a 2.66 GHz Intel Core i As the equation on page 35 explains, execution time is the product of the three factors in this table: instruction count in billions, clocks per instruction (CPI), and clock cycle time in nanoseconds. SPECratio is simply the reference time, which is supplied by SPEC, divided by the measured execution time. The single number quoted as SPECINTC2006 is the geometric mean of the SPECratios. Copyright 2014 Elsevier Inc. All rights reserved SPECpower of Server FIGURE 1.19 SPECpower_ssj2008 running on a dual socket 2.66 GHz Intel Xeon X5650 with 16 GB of DRAM and one 100 GB SSD disk. Power consumption of server at different workload levels Performance: ssj_ops/sec, Power: Watts (Joules/sec) Overall ssj_ops per Watt = ssj_ops i poweri i= 0 i=

14 Benchmark Pitfalls Benchmark not representative Your workload is I/O bound, SPEC is useless Benchmark is too old Benchmarks age poorly; benchmarketing pressure causes vendors to optimize compiler/hardware/software to benchmarks Need to be periodically refreshed 27 Summarizing Performance Machine A Machine B Program Program Total Simplest: total execution time Performance B Performance A Execution time A = Execution time B = = 9.1 B is 9.1x faster than A if program 1 and program 2 run equal number of times 28

15 Summarizing Performance 2. Arithmetic mean (AM) n 1 time ( i) i = 1 n AM(A) = 1001/2 = AM(B) = 110/2 = /55 = 9.1x For n programs, each run an equal number of times 3. Weighed arithmetic mean program is weighted by its frequency in the workload n ( weight( i) time( i) ) i= 1 29 Geometric Mean 4. Geometric mean of ratios = Independent of reference machine n Machine A (s) Machine B (s) Program Program If we take ratios with respect to machine A Machine A (ratio) Machine B (ratio) Program Program n ratio( i) i=1 Average for machine A is 1 average for machine B is 5.05 If we take ratios with respect to machine B Machine A Program Program Machine B Can t both be true!!!, Don t use arithmetic mean on ratios! Average for machine A is 5.05 Average for machine B is 1 30

16 Benchmarking & performance summary Benchmarks Programs chosen to measure performance SPEC System Performance Evaluation Cooperative Consists of many small programs Summarizing performance 1. Total execution time 2. Arithmetic mean (AM) 3. Weighed arithmetic mean 4. Geometric mean of ratios Fallacies and Pitfalls Pitfall#1: expecting the improvement of one aspect of a computer to increase performance by an amount proportional to the size of the improvement Example: a program runs in 100 seconds on a computer, with multiply operations responsible for 80 seconds of the time. How much do I have to improve the speed of multiplication if I want my program to run 5 times faster? 32

17 Amdahl s Law Execution time after improvement = ( execution time affected by improvement amount of improvement + execution time unaffected ) From the data of last example: Execution time after improvement = 80/n + (100-80) Since we want the performance to be 5 times faster, the new execution time should be 20 seconds, giving 20 = 80/n + (100-80) => n =? 33 Amdahl s Law Example Execution time after improvement = ( execution time affected by improvement amount of improvement + execution time unaffected ) Your boss asks you to improve performance Two options Improve the ALU used 95% of time by 10% improve the squareroot unit used 5%, by a factor 10 Execution time affected Amount of improve ment 95% % 10 5% Execution time after improveme nt 34

18 From Amdahl s Law, how do we improve performance Execution time after improvement = ( execution time affected by improvement amount of improvement + execution time unaffected ) Make the common case fast Favor the frequent case over the infrequent case Example: The new CPU is 10 times faster. The CPU is busy with computation 40% time and I/O waiting for 60%. execution time affected = 0.4 execution time unaffected = 0.6 amount of improvement =10, new execution time =? 35 Fallacies and Pitfalls Pitfall#2: Using a subset of the performance equation (IC x CPI x CCT) as a performance metric MIPS and MFLOPS Millions of instructions per second Millions of floating point instructions per second MIPS = instruction count / (execution time x 10 6 ) = clock rate / (CPI x 10 6 ) Higher MIPS means faster machine But MIPS has serious shortcomings 36

19 Problems with MIPS Ignore program (IC): Cannot compare two computers with different instruction sets Different instruction sets different instruction counts Varies for different programs on same machine Different instruction mixes different MIPS rate Can vary inversely with performance When is MIPS ok? Same compiler, same ISA 37 Summary of Performance Evaluation Different performance metrics, (response time, throughput, CPU time, MIPS and MFLOPS Performance =1/time = IC x CPI x CC (Iron Law) Good benchmarks, such as the SPEC benchmarks, can provide an accurate method for evaluating and comparing computer performance. Summarize performance: total execute time, AM, WAM, GM Amdahl s law provides an efficient method for determining speedup due to an enhancement. Make the common case fast! Why MIPS measurement is not used? 38