Maximise Data Center Value with Higher Density Infrastructure Dramatic cost savings enabled; Requires solving unique electrical + cooling challenges; 08 December 2015
Speaker gokhan.gulal@4g.net.tr Genel Müdür 4G İletişim Teknolojileri A.Ş.
Data Center Investments Keep Increasing Continued Infrastructure Growth Number of data centers increasing Average size of data centers also increasing Even though footprint of equipment is decreasing Driven by Underlying Storage and Compute Demand business intelligence / data mining e-commerce at every scale private cloud better & cheaper bandwidth RESULTS OpEx requirement (or CapEx for co-location) increasing; Budgets not keeping pace; Massive consolidation efforts and shift to co-location;
Even When Server Power Draw Decreasing Year Watts / U 2004 (DL360G3) 360 W 2008 (DL360G5) 238 W 2011 (DL360G7) 157 W 2013 (DL360G8) 170 W 2015 (DL360G9) 105W Source: Data Center Pulse. Wiersma, Jan. Where is the Rack Density Trend Going? ; HP QuickSpecs Combining major CPU improvements in power/mips better underlying h/w architecture e.g. High-efficiency PSUs e.g. Move to SSDs e.g. Improved front-side bus architectures (RAM) standard 1U / 2U servers are actually less power-hungry per U.
Customers Face Immense Upfront Costs: Infrastructure + Real Estate Air Handlers Switchgear Transformers Chillers / Economizers Generators CRAC / CRAH UPS RPP / Panel Board / Distribution Board Owner operated = CapEx Co-location facility = OpEx Before a single CPU is powered on, incredible investments must be made; Land and Building
Real Estate Prices Increasing Prohibitively In certain regions, no viable candidate sites available for data center development; e.g. Hong Kong (Google, other multinationals) Some companies convert commercial buildings into data center (limits space and usability) Real Estate Costs + Infrastructure Costs # of Cabinets Per rack investments are massive and fixed ; = fixed investment per rack Therefore, massive returns from increasing cabinet power density.
Increased Power Density Enables Consolidation Increase to 11KVA Add 7.5 KVA Avg load /rack = 3.5 KVA Rack 1 Rack 2 Rack 3 Rack 4 Rack 5 Rack 6 Rack 7 Rack 8 3.5 KVA Reduce Racks Free Space
but Typically Requires Shift to Blades @ 3ph Phase 16A 32A 63A 16A 32A 63A Single phase Single phase Single phase Three phase Three phase Three phase Input Voltage 230V 230V 230V 400V 400V 400V Output Voltage 230V 230V 230V 230V 230V 230V Active power 3.7 kw 7.4 kw 14.5 kw 11 kw 22.1kW 43.6 kw # of servers 19 38 75 52* 57 52* # of blades n/a 32 (2 chassis) 64 (4 chassis) 48 (3 chassis) 80* (5 chassis) * Could power more, but cannot fit in standard cabinet HP Proliant DL360pG8 1U server, 192 watts at 70% load HP Proliant C7000 w/ 16x Proliant BL460c Gen8, 3710 watts at 70% load
415v / 3ph : Dramatically Increases Density
But High Density Racks Face Unique Challenges Weight. Can cabinet handle increased static and dynamic load? Cooling. Can air handlers, chillers, etc. be scaled to exhaust increased heat? Power. Adequate UPS and other infrastructure? Understand complexities of branch circuit in cabinet? Known, proven solutions exist for all these challenges. Cost more per cabinet vs. lower density; But far outweighed by cost savings of increased utilization.
Proper Feed Sizing for High Density Blade Chassis Prevalence of blade servers increasing every year; Increased confusion regarding power interconnects required to maintain true 2N; Issue compounded when clients solely consider power capacity guidelines of RPP / distribution panel feed; e.g. Cisco UCS 5108 6U height; ~1800W typical; ~2300W peak; 4x Power Supplies, up to 2500W each; e.g. 415V, 3phase WYE; 32a Supply 23,000VA / ~22,400 Watts; In theory, should support up to (22400 2300) = 9 chassis; Let s try seven (48U rack)
Proper Feed Sizing for High Density Blade Chassis First Connect Six Chassis (14.2kVA) Peak = 4.9A per connection; Peak (Failure Mode) = on one plug
Proper Feed Sizing for High Density Blade Chassis Even with B-side down, each circuit breaker at 61% load. Very safe.
Proper Feed Sizing for High Density Blade Chassis Failure Mode: B-Side power down and some A-side power supplies fail. Still safe.
Proper Feed Sizing for High Density Blade Chassis Add 7 th Chassis. Need to share breakers.
Proper Feed Sizing for High Density Blade Chassis 14.7A Still safe so far. Two breakers load to 14.7A if B-side power down. 14.7A
Proper Feed Sizing for High Density Blade Chassis 14.7A 14.7A Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis 14.7A 14.7A Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis 14.7A Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis 14.7A Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis 14.7A Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis 19.6A Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis
Proper Feed Sizing for High Density Blade Chassis Not truly redundant! One bad power supply (during B-side maintenance) can shut down 3 chassis Violates best practice concept of isolated failure domains.
Proper Feed Sizing for High Density Blade Chassis Example Result: (One Possible Solution) This cabinet provides two redundant 23kVA power feeds. Does not necessarily translate into, Can implement 23kVA of load. 7 chassis @ ~16.5kVA did not work! Apparent maximum = 6 chassis @ 36U (14.1kVA); Wastes 40% of power capacity; Wastes 25% of rack space (assuming 48U cabinet);
Thank You!