Oak Ridge National Laboratory Computing and Computational Sciences

Transcription

1 Oak Ridge National Laboratory Computing and Computational Sciences Computer Science and Mathematics Power Measurement for High Performance Computing: State of the Art PMP 2011 Chung-Hsing Hsu Steve Poole July 28, 2011

2 Observations Power utilization is a major concern. You cannot improve if you cannot measure. You cannot measure if you cannot monitor. More power-monitoring devices are available. The measurement methods are standardized. Facility & IT people will work together.

3 Why Work Together The gap between common idling assumptions and real world conditions [Brill 2010]: Assume: power draw tracks IT work (network traffic). Reality: significant variation in IT work but flat power draws at UPS. The limits of power modeling based on extrapolation [Hackenberg 2010, Davis 2011]: Reality: inter-node variability in power draw. Reality: method valid for load-balanced IT work.

4 This Survey Power measurement methods from the power system to the IT system: Classified based on measurement domains. Hierarchical domain structure. Simultaneous measurements. Hardware-based methods: Stand-alone meters and Integrated sensors. Drivers and Interfaces. The real challenge is in the real-time analysis of massive sensor data.

5 Hierarchical Measurement Domains

6 Simultaneous Measurements A method for PUE calculation:

7 Seven Measurement Domains A site. An HPC facility. An HPC machine. A cabinet (or a server rack). A compute node (or a server). A multi-core processor. A processor core.

8 Domain 1: A Site ORNL Primary 161-kV substation and associated lines: Source: TVA, Environmental Assessment 2005.

9 Domain 2: An HPC Facility 100 of 567 Eaton PowerNet meters for CSB: Source: J. Rogers, CUG CSB

10 Domain 2: An HPC Facility Other options: wireless monitoring. Source: 42U.com.

11 Domain 3: An HPC Machine The Jaguar Cray XT5: Source: J. Rogers, CUG 2009.

12 Domain 3: An HPC Machine 3 meters for monitoring Jaguar XT5. Source: Wenning et al., FEMP 2010.

13 Domain 3: An HPC Machine Monitor a 26-hour HPL run: Source: J. Rogers, CUG 2009.

14 Domain 4: A Cabinet Aggregate over all cabinet measurements: Source: Server Technology website.

15 Domain 5: A Compute Node Inline meters are the most popular, but IPMIcompliant servers are gaining attention: Watts Up? PRO Yokogawa WT210

16 Domain 6: A Processor PowerPack [Ge 2005]: Insert resistors for the PSU wires Cray XT3/4/5 [Laros 2009]: Interrogate the VRMs from L0 PowerMon2 [Bedard 2010]: Plug into the PSU L0 VRM

17 Domain 7: A Processor Core The issue of accounting: Shared and distributed. Intel Core i7 CPU Source: M. Schuette, LostCircuits 2008

18 To Sum Up Today we have more options to use integrated sensors for each domain. The measurement process is also slowly being standardized. The real challenge is in the real-time analysis of massive sensor data: The challenge of scale. The challenge of accounting: For end users For power loss

19 The Challenge of Accounting Accounting for power loss: Source: P. Scheihing, DOE EERE

20 Thank You! Acknowledgments: This work was supported by the United States Department of Defense and used resources of the Extreme Scale Systems Center at Oak Ridge National Laboratory. The work was performed at the Oak Ridge National Laboratory, which is managed by UT-Battelle, LLC under Contract No. De-AC05-00OR22725.