Servers: Take Care of your Wattmeters!

Transcription

1 Solving some Mysteries in Power Monitoring of Servers: Take Care of your Wattmeters! M. Diouri 1, M. Dolz 2, O. Gluck 1, L. Lefevre 1, P. Alonso 3, S. Catalan 2, R. Mayo 2, and E. Quintana-Orti 2 1- INRIA Avalon Team, LIP Laboratory (CNRS, ENS, INRIA, University of Lyon) 2- Depto. de Ingenieria y Ciencia de Computadores, Universitat Jaume I 3- Depto. de Sistemas Informaticos, Universitat Politecnica de Valencia laurent.lefevre@inria.fr Vienna (Austria), April 22 th, st Conference on Energy-Efficiency on Large Scale Distributed Systems 1/29

2 Context Introduction LSDS consume enormous amounts of energy. Green! : A strong research effort towards energy-efficient approaches What is the power consumption of the server running my application? Need of accurate monitoring of power and energy consumption of these systems - but monitoring, for what? evaluate the energy consumption of an HPC execution? detect peaks of consumption of a large scale HPC system? analyze the power profile of an application? compare green programming // applications? 2/29

3 Context and Motivations Evaluate/compare external and internal power monitoring devices 2 different target platforms: a server and a desktop machine Experimental results for a variety of benchmarks: stress the main components (CPU, RAM, HDDs, and NICs) Goal: analyze and compare the measurements of: energy consumption; power consumption; power profiles. 3/29

4 Introduction Power measurement framework External devices OmegaWatt Power tracing server WattsUp? Application node Computer Power supply unit Mainboard Ethernet Power tracing daemon Internal devices PowerMon 2 National Instruments DAS Microcontroller based wattmeter 4/29 2 M. Diouri1, M. Dolz2, O. Gluck1, L. Lefevre1, P. Alonso3, S. Solving Catalansome, R. Mysteries Mayo2, and in E. Power Quintana-Orti Monitoring2 of Servers: Ta

5 Power Measurement Devices Table 1. Specifications of the wattmeters. External AC Internal DC Power Meter OmegaWatt WattsUp PowerMon2 NI DCM Manufactured by OmegaWatt a WattsUp? b RENCI ilab c National Instruments d Universitat Jaume I #Channels Channel type Standard power PC cord Standard power PC cord All ATX-related lines (3.3 V, 5Vand12V) e 12 V ATX-related lines 12 V ATX-related lines Power nature Average Instantaneous Instantaneous Instantaneous Instantaneous Microcontroller - - Atmel ATmega16 NI9205 NIcDAQ-9178 Microchip PIC 18 Power sensors - - Analog Devices ADM1191 resistors Sampling Rate (S/s) per channel LEM HXS 20-NP transducers LEM HXS 20-NP transducers f Accuracy < ±1% < ±1.5% ±5% ±1% ±1% Interface RS232 USB USB USB RS232 Price 600 e 200 e 125 e 2700 e Not commercialized a OmegaWatt: 5/29 b M. Diouri 1, WattsUp: M. Dolz 2, O. Gluck 1, L. Lefevre 1, P. Alonso 3, S. Solving Catalansome 2, R. Mysteries Mayo 2, and in E. Power Quintana-Orti Monitoring 2 of Servers: Ta

6 Target platforms a desktop computer: Intel Desktop 1 Intel Ivy Bridge Core i7-3770k 1 CPU = 4 cores running at 3.50 GHz 16 GB of RAM a server machine: AMD Server 4 AMD Opteron CPU = 12 cores (total of 48 cores) running at 2.10 GHz 256 GB of RAM 6/29

7 Benchmarks idle: machine switched on and running only the OS! iperf: network throughput measurements (TCP or UDP). hdparm: intensive I/O with the hard disk drive. cpuburn: heats up any CPU to the maximum temperature. burnmmx: stresses the cache and memory interfaces. 7/29

8 Compare variability and accuracy in terms of energy consumption. on Intel Desktop and AMD Server. execution of the benchmarks, during 60 seconds each. idle: 0 core hdparm and iperf: 1 core cpuburn and burnmmx: all the cores using the different wattmeters: external: 1 sample per seconde internal: PowerMon (100 Hz), NI (1000 Hz), DCM (28 Hz) measure the aggregated energy = addition of energy measurements in all the 12 V lines 8/29

9 7000 Intel_Desktop AMD_Server Energy (J) Energy (J) idle hdparm iperf burnmmx cpuburn Benchmarks 0 idle hdparm iperf burnmmx cpuburn Benchmarks OmegaWatt WattsUp PowerMon NI DCM OmegaWatt WattsUp PowerMon NI DCM Figure: Energy consumption of the benchmarks external (similar) vs internal (differences) measurements - different components used in the internal wattmeters - different amount of samples per second internal lower than external measurements - do not account PSU and other components (HDDs, GPUs) - a less significant gap for AMD Server (relatively) 9/29

10 Analyze and compare the power measured by the different wattmeters. For 3 different workloads (cpuburn, idle, hdparm) Running on the 2 different target platforms. Boxplots to show the distribution of the power measurements min, max, median, lower and upper quartiles outliers measurements 10/29

11 - IDLE Intel_Desktop External AMD_Server Internal 40 External Internal 280 Power (W) OmegaWatt WattsUp PowerMon NI DCM Wattmeters Power (W) OmegaWatt WattsUp PowerMon NI DCM Wattmeters External: similar power variability for both machines Internal: visible differences especially for AMD_Server. Internal: more dispersed values and many outliers due to the high sampling frequency! real power fluctuations? or noise? 11/29

12 - HDPARM Power (W) External Intel_Desktop Internal OmegaWatt WattsUp PowerMon NI DCM Wattmeters Power (W) External AMD_Server Internal OmegaWatt WattsUp PowerMon NI DCM Wattmeters External on AMD_Server: higher power measurements with OmegaWatt loosing accuracy with high power (more than 300W)? OmegaWatt registers some strange outliers Internal: same conclusions as for idle 12/29

13 - CPUBURN Intel_Desktop AMD_Server External Internal 100 External Internal 750 Power (W) Power (W) OmegaWatt WattsUp PowerMon NI DCM Wattmeters OmegaWatt WattsUp PowerMon NI DCM Wattmeters AMD_Server: a more visible gap for external and internal With AMD_Server, PowerMon2 higher than WattsUp while PowerMon2 measures internally! less accurate for too high power (more than 500W)? On AMD_Server: highly dispersed measurements power fluctuations for AMD_Server (not for Intel_Desktop)? need to analyze the power profile for cpuburn! 13/29

14 - NI Analyze and compare power profiles on both machines 20 seconds of idle, hdparm, and cpuburn from both external and internal wattmeters. simultaneously OmegaWatt and PowerMon2 simultaneously WattsUp and NI. Analyzing the internal behavior of the benchmarks for example, to detect some special power increases that could be filtered by external wattmeters. 14/29

15 - Intel_Desktop Intel_Desktop 120 idle hdparm cpuburn 120 idle Intel_Desktop hdparm cpuburn Power (W) Power (W) Time (s) PowerMon OmegaWatt Time (s) NI WattsUp External (nearly the same) vs Internal (some variations) much more noise with NI: high sampling rate : allows to observe the peaks! 15/29

16 - AMD_Server AMD_Server 900 idle hdparm cpuburn 900 idle AMD_Server hdparm cpuburn Power (W) Power (W) Time (s) PowerMon OmegaWatt Time (s) External: different power profiles for hdparm and cpuburn different specifics of the devices. potential environment changes (e.g., room temperature) external measurements not simultaneously! Internal: PowerMon2 highly displaced from NI. AMD_Server: spikes and drops when running cpuburn BIOS-mainboard settings and fans: constantly on and off to maintain the platform s temperature at a constant level. 16/29 NI WattsUp

17 Configurable Sample Rate - NI on Intel_Desktop evaluate the impact of the sample rate. measure the power consumption with NI a frequency of 1,000 S/s during 30 seconds for hdparm and cpuburn on both machines. Then, we reduce the sample rate from 1000 S/s to 1S/s. X values => replaced by the average value 17/29

18 Configurable Sample Rate - NI on Intel_Desktop hdparm (left side) High sample rate (more than 200 S/s) masks spikes and drops Low sample rate (less than 50 S/s) hides interesting power fluctuations: like the high spikes just before a drop (at 50 S/s) cpuburn (right side) power fluctuates between 57 W and 63 W (at 1,000 S/s). thinner when reducing the sample rate. Below 50 S/s: a constant line devoid of noise. 18/29

19 Configurable Sample Rate - NI on Intel_Desktop What we need? a medium sample rate (between 50 S/s and 200 S/s) for the power profile of hdparm A low sample rate (1 S/s) is enough to observe the power profile of cpuburn. 19/29

20 Configurable Sample Rate - NI on AMD_Server hdparm similar to what we observed on Intel_Desktop cpuburn a different behavior with AMD_Server a medium sample rate (between 50 S/s and 200 S/s) is required to see clearly the spikes and drops spikes and drops even with 1 S/s real power fluctuations and not simply a noise 20/29

21 Configurable Sample Rate - NI on AMD_Server A very high sample rate (500 S/s) is not always necessary May cause some noise that masks the general shape The best sample rate is not always the highest one but the one that best enables to understand fluctuations. 21/29

22 Internal Channel Analysis: PowerMon2 specific analysis of the power profiles using PowerMon2 with idle, hdparm, and cpuburn - varying number of Cores for Intel_Desktop - varying number of Sockets for AMD_Server running one both machines measure independently, the 3.3 V, 5 V and 12 V lines sample rate = 100 S/s 22/29

23 Internal Channel Analysis - PowerMon2 on Intel_Desktop Intel_Desktop 100 hdparm cpuburn idle 4 cores 3 cores 2 cores 1 core 90 Power (W) Sum of all lines MotherBoard 3.3V MotherBoard 5V MotherBoard 12V Socket Part1 12V Socket Part2 12V Time (s) 12 V socket lines are split in 2 idle: 3.3 V and 5 V lines: more power than 12 V lines - power remains constant in all the lines hdparm: 3.3 V and 12 V mainboard lines show a plain profile - 5 V mainboard fluctuate in conjunction with 12 V socket cpuburn: 5 V and 12 V socket lines increase their power V and 12 V mainboard: a very plain profile 23/29

24 Internal Channel Analysis - PowerMon2 on AMD_Server AMD_Server hdparm cpuburn idle sockets 3 sockets 2 sockets 1 socket Power (W) Sum of all lines MotherBoard 3.3V MotherBoard 5V MotherBoard 12V Socket Part1 12V Socket Part2 12V Socket Part3 12V Socket Part4 12V Time (s) The 4 sockets do not match with the 4 12 V socket lines. power measured by each of the 4 12 V socket lines changes. idle: a flat profile for all the lines - 12 V lines more power than 3.3 V and 5 V lines. hdparm: 12 V lines draw the natural spikes of hdparm cpuburn: 12 V mainboard higher power than12 V sockets - starts dropping down when less sockets are working 24/29

25 Introduction analyze and evaluate different external and internal wattmeters using an AMD server and an Intel desktop machine running different benchmarks a complete comparison in terms of power and energy internal: differences in energy and in power variability : different components to measure the power different sampling rates internal lower than external except for a special scenario: cpuburn power measurements on AMD_Server. PowerMon2 higher than WattsUp- lost of accuracy for more than 500W? 25/29

26 Introduction internal power: more dispersed and generate many outliers. high frequency of internal wattmeters! real power fluctuations or noise? internal power profiles: some differences, more visible with NI very high sampling rate of the NI. displaced specifically for hdparm and for cpuburn. differences in the devices potential environment changes (room temperature) 26/29

27 Introduction cpuburn on AMD_Server: spikes and drops fans alternatively on and off in order to cool the server. Not for Intel_Desktop. measuring at a very high sample rate is not always necessary - may provoke some noise that masks the power profile the appropriate sample rate is not always the highest possible but the one that best enables to understand the power fluctuations. measuring (3.3 V, 5 V and 12 V) lines in a separate way detect where power fluctuations come from lines are not related to specific components 27/29

28 Introduction To achieve accurate and reliable measurements : calibration? (oscilloscope) Internal wattmeters not practical for monitoring very LSDS : - not easy to plug (servers opened) // price per node What about internal embedded measurements (IPMI and RAPL counters)? Promising techniques: do not require an extra device Adaptation needed : bad acuracy, low sample rate, intrusiveness 28/29

29 Introduction 29/29