Performance Evaluation of Computer Systems Course Overview

Transcription

1 Performance Evaluation of Computer Systems Course Overview Vlastimil Babka, Lubomír Bulej Tomáš Kalibera, Peter Libič Petr Tůma FACULTY OF MATHEMATICS AND PHYSICS CHARLES UNIVERSITY IN PRAGUE

2 Performance Evaluation... Performance Response time, number of operations per second, processor utilization, memory occupation... Evaluation Measuring either real systems or simulation models Theoretical analysis of mathematical models of systems

3 of Computer Systems Ultimately, Ultimately,we wewant wantto toknow know how howfast fastis isour oursoftware software Algorithmic complexity is only part of the picture In reality, multiple layers affect performance and the effects are hard to observe in isolation Class libraries, virtual machine (C#, Java) Native libraries (C/C++ libraries, math libraries...) Operating system, sometimes hypervisor Hardware (CPU, memory, disks...)

4 ? How hard can it be

5 Java Hash Set Lookup Performance HashSet.contains HashSet.contains offers offersconstant constanttime timeperformance performance

6 Java Hash Set Lookup Performance HashSet.contains HashSet.contains offers offersconstant constanttime timeperformance performance

7 Applications... When purchasing a new hardware or software Which one is the best fit for my purpose? When designing a new software Will it meet the performance requirements? Which design alternative will have better performance? When designing a new CPU (or compiler or VM ) Will adding optimization X improve the performance? When a system is running (too) slow Where is the bottleneck and how can I fix it? Which set of parameters yields the best performance?

8 Techniques... Design time implementation not ready Analytical Modeling Simulation Smaller part of this lecture Implementation or testing time implementation (at least partially) ready Measurement Benchmarking Monitoring Profiling Biggest part in this lecture Differences in accuracy, flexibility, time, cost...

9 Measurement Basics

10 Performance Measurement What to measure about the system? Performance metrics How to collect the measurements? Time measurement, performance event counting How to summarize the results? Disruptions lead to noise in the collected data How to present the results? Data visualization in plots How to compare (two) alternatives? Which differences are really significant?

11 Performance Metrics Properties that somehow describe performance of a computer system derived from what we can measure Typically, what we can measure is Counts of some events of interest Duration of some time interval Size of a parameter

12 Types of Performance Metrics Responsiveness: related to time duration Response time time elapsed from start of service request to end of service response Operation time time of the service itself Network delay, cache miss penalty Rate metrics: # of events per time interval Throughput number of requests finished per second Network bandwidth (bits/s), messages/s Utilization: % of time the system is busy Resource utilization = % of time resource is busy Often resource with high utilization = bottleneck

13 What is a good metric? Depends on the goal (can include power, cost...) To assess a single complete system (HW + SW), response times, throughput, (utilization) is enough It is based on what the users perceive To assess e.g. CPU alone, any metric based on a single number is often misleading Clock rate of the CPU? No way... MIPS? But, which instructions? Benchmark suites? (Linpack, SPEC ) Better but... Are the benchmarks representative? Danger of vendors tuning optimizations!

14 Time Measurements

15 Measuring Time Accuracy: how much (systematically) the samples differ from the real time If we measure code that runs exactly one second, it says how the average time of this code, as measured using the timer, differs from one second Precision: (lack of) scatter in samples The variability around mean of the measurements Resolution (granularity): the smallest difference that can be measured Duration of one tick of the timer

16 Measuring Time Accuracy Frequency Precision Mean Resolution True Value Time

17 Available Timers Most systems have multiple timers Low resolution real time clock (~1 s) Date keeping Medium resolution timers (~ ms) Scheduler alarms Multimedia events High resolution timers (~1 ns) Processor clock Event counters Multiple interfaces to access those timers, do not use the first gettime() call available!

18

19 Some Java Timers System.currentTimeMillis() Time since , ms units, undefined resolution (depending on the JVM and underlying OS) System.nanoTime() Current value of the most precise available system timer Arbitrary start value, can overflow This method provides nanosecond precision, but not necessarily nanosecond accuracy. No guarantees are made about how frequently values change. In our terms: ns units, but undefined resolution Can see: from padded microseconds to real nanoseconds

20 Processor Timers Most current processors have an internal register that counts clock ticks Typical characteristics Pentium (and newer) TSC register Itanium ITC register Often directly readable from application code Resolution similar to processor clock Might be subject to frequency scaling, idle states Might be separate per core (and drift) Read your CPU documentation!

21 Reading Pentium TSC Intel processors #include <stdint.h> Forcing serialization manually inline uint64_t rdtsc() { uint32_t lo, hi; } asm volatile ( "xorl %%eax, %%eax\n" "cpuid\n" "rdtsc\n" : "=a" (lo), "=d" (hi) : : "%ebx", "%ecx"); return (uint64_t) hi << 32 lo; AMD and some Intel processors asm volatile ("rdtscp" : "=a" (time_lo), "=d" (time_hi) :: "%ecx"); Guarantees serialization, also returns core ID

22 Performance Monitoring

23 Performance Monitoring Counters With the increasing complexity of processors it is difficult to tell what is going on inside Performance can be influenced for example by improper allocation of variables to registers improper mixture of various types of instructions conditional or indirect branching that is hard to predict memory access pattern with poor locality properties Very difficult to identify for programmer

24 Performance Monitoring Counters Hardware registers in the CPU that count certain events inside the processor architecture Performance event Event of interest that the processor can count Performance counter Register that counts selected performance event Typically, there are many types of events but only a few counters Limits how many events can be counted together Multiplexing needed for more events Loss of precision Interference effects

25 Performance Events Examples of performance events tick of the processor clock, useful for providing a baseline to which other events can be related retirement of a processor instruction, useful for judging the efficiency of instruction execution (IPC) occurrence of stall when executing an instruction, useful for discovering reasons for slow execution miss in the memory cache, useful for identifying code that accesses memory in a pattern that the memory hardware finds difficult to predict Some event types can be further classified by context (mask) or location

26 Caveats of Performance Counters Privileges for working with the counters Reading usually user-space, setting from kernel only Need for special kernel interfaces and libraries Overhead, especially in the kernel calls Read documentation Create experiments carefully to verify assumptions! Limited number of!counters and their combinations Pipelining and speculative execution makes the events hard to attribute to specific instructions Availability and precise meaning of events varies from processor to processor Sometimes even processor revision (errata)

27 PAPI Library Standard API layer over multiple counter interfaces Per-thread virtual performance counters Set of platform-independent preset events Translated to (combinations of) native events cache hits = cache requests - cache misses Many events available papi_avail, papi_native_avail Support for counter multiplexing

28 Designing Experiments

29 What Can Disrupt Results Precision Overhead Direct access to counters in nanoseconds ~ clocks for serialized time read ~ clocks for performance counter read Warmup! Querying time through API in microseconds Interference Random interrupts, concurrentrandomization processes, user! activity Systematic triggered or scheduled activities, resource allocation

30 Illustration: Systematic Interference Remote server invocation time in Java Why the slight sloping down here? Why are the outliers here? And what about the sudden speed up? Sequential invocation number

31 Illustration: Systematic Interference Time to calculate FFT in C++ Why very stable inside one run but very different between restarts? Many sequential measurements Vertical bars indicate restart

32 What to do for meaningful results? Stabilize configuration of the experiment This includes avoiding hardware and software updates Not everything can be controlled and thus stabilized Memory allocation, code layout... Randomizing might be the only option Randomize workload, parameters The intent is to make workload into random influence (instead of systematic) that can be averaged over Minimize disruptions Because they do not contribute to understanding, only increase variance and therefore required number of observations

33 Examining Results

34 Summarizing Results Typically a lot of numbers with a lot of (possible) explanations Infeasible to evaluate separately Single number always loses some information A good compromise is needed Preparing to analyze results Correct aggregation Correct statistical processing Good graphs

35 Statistics or Not? Many statistical methods have too strict assumptions Observations are IID (independent identically distributed) Behavior is memoryless Distribution is normal These are actually rarely met in practice! Often those methods are all we have

36 Aggregation: Arithmetic Mean n 1 x A = xi n i=1 Used for variables that are additive in nature their sum should make sense (for example durations) Very sensitive to outliers, single outlier can move the mean anywhere Trimmed mean Mean calculated after X% of extreme values are discarded Useful for situations where the presence of outliers needs to be reflected but the outlier values themselves make the standard mean useless

37 Aggregation: Geometric Mean xg = n n xi i=1 Used for multiplicative variables for example ratios or overheads Example: Optimizations speed up 10, 20 and 30 percent Geometric mean is 19.7 percent Sequence of 3 optimizations will speed up = times or 71.5 percent

38 Aggregation: Median Value in the middle of sorted series of samples Useful for samples with asymmetric distribution Robust for up to 50% outliers included in the observations Other quantiles Often also helpful But not a single number anymore

39 Plotting Simple plots There is almost always something better At least good to indicate standard deviation Box and whisker plots Scatter plots Box from low to high quartile Identify dependencies Zooming out to detect patterns Line for median Whiskers for min and max

40 Illustration: Box And Whiskers Time of remote invocation It is more the outliers that move the average! Number of threads used

41 Modeling Performance

42 Performance Models If program is a deterministic sequence of steps, and we know the timing of each step, we know the timing of the program, right? Example from Xu et al.: Performance Modeling and Prediction of EJB with LQN Templates Determinism of execution Timing of steps Where to get the models How to solve the models

43 Performance Modeling Formalisms Markov Chains Not really for humans Queueing Networks Network of servers that process queued requests Process Algebras Rewriting rules describing program behavior Petri Nets Network of places visited by tokens

44 Queueing Networks System of servers and queues Parametrized Arrival rate λ Service time μ Other details The 1 means there is one server M/M/1 μ First M means interarrival time λis exponentially distributed Second M means service time is exponentially distributed

45 Petri Nets Places holding tokens Tokens traverse transitions Timed Colored Conditional Consumer starts here Consuming transition Producer starts here Place for holding buffer

46 Creating Performance Models Manually Only for small systems Automatically Derived from software still only in early stages Derived from models sophisticated model transformations Plus annotations Estimates Measurement

47 Introduction to Dependable Systems

48 Introduction to Dependable Systems

49 What Kind of Results to Expect Models can be made to fit observations nicely Careful calibration on real system Used within reasonable parameter space But that is really not the target application... Models reveal scalability behavior Necessary failures Anomalous behavior Overfitting the model

50 Related Courses NSWI131 Vyhodnocování výkonnosti... More details about presented topics Also profiling, simulation, benchmarks comparing alternatives, making conclusions NSWI126 Nástroje pro vývoj a monitorování... Practical experience with profiling... NMAI060 Pravděpodobnostní metody NMAI061 Metody matematické statistiky For statistical background