Energy Efficiency and WCT Innovations

Transcription

1 Energy Efficiency and WCT Innovations Zeeshan Kamal Siddiqi HPC Leader Middle East, Turkey and Africa (META) Lenovo 2017 Lenovo. All rights reserved.

2 Why do we have a problem? Higher TDP Processors Data Center Power/Space limits High Electricy Cost Performance is Power/Thermal capped Waste Heat Reuse Energy aware solutions 2

3 01/03/ 01/10/ 01/05/ 01/12/ 01/07/ 01/02/ 01/09/ 01/04/ 01/11/ 01/06/ 01/01/ 01/08/ 01/03/ 01/10/ 01/05/ Spec_fp Rate TDP (W) Intel Xeon Server processor history Release date Code Processor core/chip TDP(W) Spec FP Spec_fp Rate 2006/6/26 Woodcrest Intel Xeon /11/12 Harpertown Intel Xeon x /3/30 Nehalem Intel Xeon x /3/16 Westmere-EP Intel Xeon x /5/1 SandyBridge Intel Xeon E /1/9 IvyBridge Intel Xeon E5-2697v /9/9 Haswell Intel Xeon E5-2699v /3/9 Bradwell Intel Xeon E5-2699v /7/11 Skylake Intel Xeon Platinum Intel processor TDP & Spec_fp Rate Spec_fp Rate TDP(W) Processor performance trend Spec_fp rate with 2 processors/node has increased 40 times the past 11 years ( ). The number of cores on the chip increase 14 times. After being flat, since 2014 TDP increases linearly with Spec_fp rate. Current maximum TDP is 205W. To sustain increased performance servers will have to be less dense or use new cooling technology 2017 Lenovo. All Rights Reserved 3

4 Industry Thermal Challenges NVIDIA /AMD GPU Xeon 75W Phi AMD Nervana 500W Maintaining Moore s Law with increased competition is resulting in higher component power Increased memory count, NVMe adoption, and I/O requirements are driving packaging and feature tradeoffs (superset of features doesn t fit in 1U) Shared cooling fan power savings no longer exist for dense 2S nodes architectures due to non spreadcore CPU layout high airflow requirements 2017 Lenovo. All Rights Reserved 4

5 Server Power Trends ASHRAE HPC 2017 Lenovo Confidential. All rights reserved. *ASHRAE = American Society of Heating, Refrigerating, and Air-Conditioning Engineers. The group provides operating environment standards for datacenter operations. 5

6 Data Center Level Cooling and Power Limits Node power density trends cannot be cooled at data center level Partial rack population or rack level power capping may be required for Dense and 1U 2017 Lenovo. All Rights Reserved 6

7 Direct water cooled systems Direct Water cooling CPU/DIMMS/VRs upto 90% of heat goes to water Inlet water temperature Up to C => Free cooling all year long in most geo Water is hot enough to be efficiently reused like with Adsorption chiller => ERE << Lenovo Internal. All rights reserved Lenovo. All Rights Reserved NextScale Chassis Scalable Manifold 7

8 Typical Data Center Power and Cooling Topology Fuel Oil 48 Hrs. Typical Generators N+1 Uninterruptible Power Supply Batteries min UPS PDU A Data Center 25-30C deg air Cooling Towers ~29C deg water Static Switch A PDU B Servers ~13C deg water ~7C deg water ~35C deg water Chillers N+1 Makeup Water Storage Utility Provider 2 Sources Static Switch B Raised Floor ~13C deg air CRAH Units Computer Room Air Handler 2017 LENOVO All Rights Reserved 8

9 WCT Cooling Concept A: >83% free-cooling year round, no RDHX on WCT racks Cooling towers Existing chiller load WCT Cooling w/dry-coolers, output to ~38C, 100% freecooling for 90% of WCT load. 18C - 45C secondary supply Chiller External Dry-coolers 38C HX HX CDU/HX ~55C Chiller back-up for RDHX and WCT loop HX 17C 45C 23C Cooling towers RDHX RDHX RDHX RDHX RDHX 2017 Lenovo Confidential. All rights reserved. RDHX RDHX RDHX CDU/HX 9

10 Lenovo cooling technologies 2017 Lenovo. All rights reserved. 10

11 ThinkSystem SD650 Two Servers and Four Highest Power CPUs in 1U Water Delivery & Return Power BoardS CPUs 12 DIMMs Outlet x16 PCIe Slot Disk Drive Slot Inlet 12 DIMMs M.2 Slot 2017 Lenovo- All rights reserved. 12

12 ThinkSystem SD650 Direct Water Cooling Inlet Outlet 2015 Lenovo Internal. All rights reserved. 13

13 SD650 Improved Node Water Cooling Architecture Focus on maximizing efficiency for high (50 C) inlet water temperatures Direct water cooling of processors, memory, voltage regulation devices and IO devices (Network and Disk) Water circuit traverses all critical components to optimize cooling Memory Water chanels DISK 2017 Lenovo. All Rights Reserved Conductive plate 14

14 Heat Extraction/Performance Expectations Compute Tray Configuration (dual compute nodes) Single Rack Steady State Power (w/linpack, Turbo ON/OFF) 90% Heat Removal 85% Heat Removal Heat-to-water Heat-to-air* Heat-to-water Heat-to-air* 2 server / tray with TDP=165W CPU, 16x16GB DDR4 Memory, 2x NIC 33.5kW 29.3kW 4.2kW 27.6kW 5.9kW 2 server / tray with TDP=145W CPU, 16x 16GB DDR4 Memory, 2x NIC 30.7kW 26.7kW 4.0kW 25.3kW 5.4kW 2 server / tray with TDP=135W CPU, 16x 16GB DDR4 Memory, 2x NIC 29.4kW 25.6kW 3.8kW 24.1kW 5.3kW 2 server / tray with TDP=120W CPU, 16x 16GB DDR4 Memory, 2x NIC 27.3kW 23.7kW 3.6kW 22.4kW 4.9kW Assumptions: 36x compute trays (dual node), 72 servers + 6x switches / Rack Performance is dependent on many facility variables such as ambient temp, water temp, flow rate, etc Based on testing, it is reasonable to assume 85-90% heat-to-water given the availability of typical environmental input parameters Lenovo. All Rights Reserved 15

15 Value of Direct Water Cooling with Lenovo Higher TDP processors Reduced server power consumption Lower processor power consumption (~ 5%) No fan per node (~ 4%) Reduce cooling power consumption With DWC at 45 C, we assume free cooling all year long ( ~ 25%) Free cooling all year long => Less chillers => CAPEX savings 50+% 2017 Lenovo. All Rights Reserved Total savings = ~35-40% 16

16 Liquid cooling enables lower power consumption and/or higher performance 2017 Lenovo. All rights reserved. 17

17 Reusing heat to produce chilled water 2017 Lenovo. All rights reserved. 18

18 PUE and ERE PUE PUE = Total Facility Power IT Equipment Power Power usage effectiveness (PUE) is a measure of how efficiently a computer data center uses its power; PUE is the ratio of total power used by a computer facility ] to the power delivered to computing equipment. Ideal value is 1.0 It does not take into account how IT power can be optimised ERE ERE = Total Facility Power Treuse IT Equipment Power Energy Reuse Effectiveness measures how efficient a data center reuses the power dissipated by the computer ERE is the ratio of total amount of power used by a computer facility ] to the power delivered to computing equipment. An ideal ERE is 0.0. If no reuse, ERE = PUE 2017 Lenovo. All rights reserved. 19

19 Energy cost and savings of various cooling soutions 2017 Lenovo. All rights reserved. 20

20 Cooling comparison Air Cooled Air Cooled with Rear Door Heat Exchangers Direct Water Cooled Standard air flow with internal fans Fits in any datacenter Maximum flexibility Broadest choice of configurable options supported Supports Native Expansion nodes (Storage NeX, PCI NeX) PUE ~2 1.5 ERE ~2 1.5 Choose for broadest choice of customizable options 2017 Lenovo. All Rights Reserved Air cool, supplemented with RDHX door on rack Uses chilled water with economizer (18 C water) Enables extremely tight rack placement PUE ~ ERE ~ Choose for balance between configuration flexibility and energy efficiency Direct water cooling with no internal fans Higher performance per watt Free cooling (45 C water) Energy re-use Densest footprint Ideal for geos with high electricity costs and new data centers Supports highest wattage processors PUE ~ 1.1 ERE < < 1 with hot water Choose for highest performance and energy efficiency 21

21 How to report and control power and energy 2017 Lenovo. All rights reserved. 22

22 - simplifies Power Optimization Establish visibility Platform power consumption Inlet & outlet temperatures Airflow Power supplies Resource utilization Purley server support Further IBM PDU support Evaluation Licensing Security hardening UI enhancements Gain insights Power consumption and cooling analyses Low server utilization identification Platform power characteristics DC physical layout NEW Now Packaged with XClarity Controller Enterprise 2017 Lenovo All rights reserved. Automate for Continuous Efficiencies Enable power-aware VM migration Enable power-aware job scheduling Continued operation in the presence of power outages Optimize power usage with user-defined policies Control power consumption during boot & runtime Control responses to power supply events 23

23 Comprehensive Visibility and Control into Power Usage Dashboard Cooling analysis Workload evaluation 2017 Lenovo All rights reserved. 24

24 42U Rack Power - kw System Power & Cooling Technology Trends ASHRAE projects maximum rack power ~50kW by 2020 Power tracking below max: HPC ~15% below; Enterprise ~50% below target Legacy air cooling hit cooling limits at ~12 kw per rack, limiting density of nodes & racks Dense rack footprints require non-conventional cooling U 1S 1U 2S 1U 4S 2U 2S 2U 4S Maximum Server Power ASHRAE Update 2012 Intel ½ U, dual Xeon/ dual KNL * Source ASHRAE Datacom Series 2 handbook - Datacom Equipment Power Trends and Cooling Applications Liquid or alternative technology or combination cooling Year RDHx or In-Row Containment Conventional Air Cooling 2017 Lenovo. All rights reserved. 25

25 Value of Direct Water Cooling Any temperature: Lower processor power consumption (~ 6%), No fan per node ( ~4%) Higher TDP processor Higher performance ( ~5 to 7%) In this case, processor power consumption is not reduced Hot water temperature Provides free cooling all year long ~ 20% power savings on compression chillers to generate chilled water potentially less chillers At 50 C water temperature Heat reuse through adsorption chillers 2017 Lenovo. All rights reserved. 26

26 Conclusion 2017 Lenovo All rights reserved. 27

27 Conclusion Lenovo delivers all aspects of power and energy HW: - Monitoring power and energy of servers - Mimizing power or Improving performance of servers - Improving Energy Efficiency of Data Center witl all cooling solutions including heat reuse SW: - xcat/confluent/xcc Energy Manager: monitor power / temparature of nodes and racks - Energy Aware Run with SLURM plugin: controling power/energy of running applications Service: - Defining and implementating the best cooling solution for the Data Center 2017 Lenovo. All rights reserved. 28

28 2017 Lenovo Internal. All rights reserved.