MULTITHERMAN: Out-of-band High-Resolution HPC Power and Performance Monitoring Support for Big-Data Analysis

Transcription

1 MULTITHERMAN: Out-of-band High-Resolution HPC Power and Performance Monitoring Support for Big-Data Analysis EU H2020 FETHPC project ANTAREX (g.a ) EU FP7 ERC Project MULTITHERMAN (g.a ) EETHPC, Frankfurt Antonio Libri A. Bartolini, F. Beneventi, A. Borghesi and L. Benini

2 Outline Motivation & D.A.V.I.D.E. Overview DiG: High Res Out-of-Band Pow & Perf Mon ExaMon: Scalable Data Collection and Analytics Monitored Metrics and Data Visualization 2

3 Motivation Innovative Scenarios Coarse grain DIMM Node ACC CPU CPU ACC req Job Scheduler req util Fine grain Fine Grain Power and Performance Measurements: Verify and classify node performance (e.g. In spec / out of spec behaviour, aging and wear out) Predictive maintenance Per user - Energy / Performance accounting System Power Capping (bound from grid): Job scheduler decides the job that enters in the system Ensures operating power below a maximum power consumption level Reduce Power budget during new Installations and Power Shortage 3

4 D.A.V.I.D.E. (#18 Green500 Nov 17) D.A.V.I.D.E. SUPERCOMPUTER (Development of an Added Value Infrastructure Designed in Europe) 4

5 D.A.V.I.D.E. SUPERCOMPUTER (Development of an Added Value Infrastructure Designed in Europe) OCP form factor compute node based on IBM Minsky 4x HSMX2 Tesla P100 2 x POWER8 with NVLink BusBar 2xIB EDR LIQUID COOLING DiG ETH Zurich / University of Bologna OUT-OF-BAND HIGH RESOLUTION POWER AND PERFORMANCE MONITORING 5

6 Outline Motivation & D.A.V.I.D.E. Overview DiG: High Res Out-of-Band Pow & Perf Mon ExaMon: Scalable Data Collection and Analytics Monitored Metrics and Data Visualization 6

7 DiG: High Resolution Out-of-band Power Monitoring Beaglebone Black Out-of-band Zero overhead Collect more than 1.5 ks/s, 7/7d, 24/24h, for all users Architecture independent (i.e. tested on Intel, ARM and IBM) Fine grain down to ms scale ks/s + avg) IoT communication technology (MQTT) scalable Time synchronous (NTP, PTP) 7

8 DiG: Example of fine grain monitoring Coarse Grain View 1 Node - 15 min BB View How do we correlate these activities with applications phases? 45 Nodes -4s 15 min 45 Nodes -1s 8

9 DiG: Fine grain correlation with jobs activities μs resolved time stamps Clock CPU1 CPUn Cold air/water Clock BBB1 BBBn CRAC RPM FAN Power C node1 GPU Perf counters Rack HPC cluster Hot air/water 9

10 DiG: Fine grain correlation with jobs activities Several Metrics Cache Miss Power BBB1 BBBn P0 Pn Time APP MPI Synch Node n CPU1 Node 1 Temp Time Node Ok nup to 1ms CPUn but what about real-time Node 1analysis at higher Clock frequencies for a footballfield size cluster of computing nodes? Clock Parallel Application Cold air/water CRAC RPM FAN Power GPU Perf counters C node1 Rack HPC cluster Hot air/water 10

11 DiG: live FFT on the power traces Real-time Frequency analysis on power supply and more a live oscilloscope Application 1 Application 2 User -> to discriminate application phases Sys Admin -> to detect malicious users Designers -> to debug and optimize power delivery network 11 11

12 Outline Motivation & D.A.V.I.D.E. Overview DiG: High Res Out-of-Band Pow & Perf Mon ExaMon: Scalable Data Collection and Analytics Monitored Metrics and Data Visualization 12

13 ExaMon: Scalable Data Collection and Analytics Aggregates job s power and performance in real-time and at a fine granularity for Big Data analysis Based on open-src SW Store, Process, Visualize and Analyze Monitored Data Historical Buffer (i.e. 2 Weeks) Perpetual per Job Web frontend Open-src beta release: 13

14 ExaMon: Scalable Data Collection and Analytics Applications Grafana Apache Spark Python Matlab ADMIN Cassandra node 1 NoSQL Kairosdb Cassandra node M Front-end Host: davidefe01 Docker containers ~45KS/s MQTT2Kairos MQTT Brokers MQTT2kairos Broker 1 Broker M MQTT Target Facility Back-end MQTT enabled monitoring agents (e.g. Dig) 14

15 ExaMon: Scalable Data Collection and Analytics Applications Grafana Apache Spark Python Matlab NoSQL Cassandra node 1 Kairosdb Cassandra node M ADMIN MQTT2Kairos MQTT Brokers MQTT2kairos MQTT Broker 1 Target Facility Broker M Monitoring Agents: Send Power and Performance measurements to the FE Key-Value pairs (TS, Data) Topic (Metric) 15

16 ExaMon: Scalable Data Collection and Analytics Grafana Apache Spark Applications Python NoSQL Matlab Broker: Forward data to the listeners (e.g. kairosdb) ADMIN Cassandra node 1 MQTT2Kairos Broker 1 Kairosdb MQTT Brokers Cassandra node M MQTT2kairos Broker M Mqtt2kairosdb: Interface between MQTT and KairosDB KairosDB is a front-end to handle time series in Cassandra MQTT Target Facility Cassandra: NoSQL database Highly scalable Optimized to balance the load on multiple nodes 16

17 ExaMon: Scalable Data Collection and Analytics Applications ADMIN Grafana Cassandra node 1 MQTT2Kairos Apache Spark Python Matlab NoSQL Kairosdb MQTT Brokers Cassandra node M MQTT2kairos Application layer: Grafana, Apache Spark, etc Aggregate metrics for Data Visualization, ML Analysis, Post Processing, etc Broker 1 Broker M MQTT Target Facility 17

18 Outline Motivation & D.A.V.I.D.E. Overview DiG: High Res Out-of-Band Pow & Perf Mon ExaMon: Scalable Data Collection and Analytics Monitored Metrics and Data Visualization 19

19 D.A.V.I.D.E. Monitoring Agents DiG SW Daemons Pow_pub MQTT Broker DiG Power (1s,1ms): ~45 ks/s 19

20 D.A.V.I.D.E. Monitoring Agents DiG SW Daemons MQTT Broker Pow_pub IPMI_pub IPMI: 89 metrics per node every 5s DiG 20

21 D.A.V.I.D.E. Monitoring Agents DiG SW Daemons MQTT Broker Pow_pub IPMI_pub OCC_pub OCC: 242 metrics per node every 10s DiG 21

22 D.A.V.I.D.E. Monitoring Agents DiG SW Daemons MQTT Broker MQTT D.A.V.I.D.E. Front-End Pow_pub IPMI_pub OCC_pub PSU_pub IPMI DiG Liteon Overall Rack info (e.g., Total Power) 22

23 D.A.V.I.D.E. Monitoring Agents DiG SW Daemons MQTT Broker MQTT D.A.V.I.D.E. Front-End Pow_pub IPMI_pub OCC_pub PSU_pub Cooling_pub IPMI Info Liquid Cooling Asetek DiG Liteon 23

24 Real-Time Monitoring, Processing, Visualization we provide for all the users a debug portal that can be accessed to gather and visualize its job data DiG 24

25 Real-Time Monitoring, Processing, Visualization DiG Users can now see power breakdown for each node, live 25

26 Conclusion With D.A.V.I.D.E. and its monitoring infrastructure we provide real-time analysis on computing nodes high resolution power and performance measurements up to ms scale upcoming FFT live analysis Data Storage, Processing and Visualization, along with Big Data Analysis support 27

27 Thanks for your interest. Co-funded by EU FP7 ERC Project Multitherman (No ) This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No (ANTAREX) Contact: Antonio Libri, a.libri@iis.ee.ethz.ch Unibo/ETH: A. Bartolini, F. Beneventi, A. Borghesi, and L. Benini

28 Backup Slides 28

29 Team & Research Activity A. Bartolini, and L. Benini Topics: 1. Examon: holistic and scalable real-time monitoring for HPC centers open source, beta release Q2/Q (F. Beneventi, 2. Large-scale multiscale Power and Thermal modeling, and control (F. Pittino, C. Conficoni) a. Development of power and thermal models at core level for monitoring and control of large HPC clusters in production b. Datacentre cooling modeling and optimization 29

30 Team & Research Activity 3. Fine grain monitoring and management (A. Libri, D. Cesarini) a. DiG: multi-platform and production ready solution, based on low-cost open-source HW, already integrated with E4 technology b. Application aware power and thermal management: open source, beta release Q Power and performance aware Job scheduler (A. Borghesi) a. System level power capping and optimization 5. Datacentre automation (A. Borghesi, A. Libri) a. Big-data and deep learning based automated power optimization b. Big-data and deep learning based automated anomaly detection DiG 30

31 D.A.V.I.D.E. Fine Grain Power Metric Broker Pow_pub MQTT Metric Name power Ts Description Unit 1s,1ms Power consumption at node power plug W DiG Power: ~45 ks/s 31

32 D.A.V.I.D.E. IPMI Metrics Broker IPMI_pub MQTT IPMI DiG IPMI: 89 metrics per node every 5s 32

33 D.A.V.I.D.E. OCC Metrics Broker OCC_pub MQTT Per-component power IPMI DiG OCC: 242 metrics per node every 10s 33

34 D.A.V.I.D.E. PSU Metrics Broker PSU_pub MQTT IPMI Running in the front-end Full Rack info 34

35 D.A.V.I.D.E. Liquid Cooling Metrics Broker MQTT Cooling_pub IPMI Running in the front-end Per Rack Cooling info 35

36 ExaMon: Batch & Streaming Pandas dataframe (Batch) examon-client (REST) Bahir-mqtt (Spark connector) (Streaming) Open-src beta release: 36