Internet-Scale Datacenter Economics: Costs & Opportunities High Performance Transaction Systems 2011 James Hamilton, 2011/10/24 VP & Distinguished Engineer, Amazon Web Services email: James@amazon.com web: mvdirona.com/jrh/work blog: perspectives.mvdirona.com
Agenda Quickening Pace Infrastructure Innovation Influence of Cloud computing Power Distribution Cooling & Building Designs Networking & Server Innovation Talk does not necessarily represent positions of current or past employers 2011/10/24 http://perspectives.mvdirona.com 2
Pace of Innovation Datacenter pace of innovation increasing More innovation in last 5 years than previous 15 Driven by cloud services & extraordinary-scale internet applications like search Cost of infrastructure dominates service cost Not just a cost center High focus on infrastructure innovation Driving down cost Increasing aggregate reliability Reducing resource consumption footprint 2011/10/24 http://perspectives.mvdirona.com 3
Perspective on Scaling Each day Amazon Web Services adds enough new capacity to support all of Amazon.com s global infrastructure through the company s first 5 years, when it was a $2.76B annual revenue enterprise 2011/10/24 http://perspectives.mvdirona.com 4
Where Does the Money Go? Assumptions: Facility: ~$88M for 8MW critical power Servers: 46,000 @ $1.45k each Commercial Power: ~$0.07/kWhr Power Usage Effectiveness: 1.45 18% 13% Monthly Costs 4% Servers 57% Networking Equipment Power Distribution & Cooling Power 8% Other Infrastructure 3yr server & 10 yr infrastructure amortization Observations: 31% costs functionally related to power (trending up while server costs down) Networking high at 8% of overall costs & 19% of total server cost (often more) From: http://perspectives.mvdirona.com/2010/09/18/overalldatacentercosts.aspx 2011/10/27 http://perspectives.mvdirona.com 5
Agenda Quickening Pace Infrastructure Innovation Influence of Cloud computing Power Distribution Cooling & Building Designs Networking & Server Innovation 2011/10/24 http://perspectives.mvdirona.com 6
208V Power Distribution High Voltage Utility Distribution ~11% lost in distribution.997*.94*.98*.98*.99 = 89% IT Load (servers, storage, Net, ) Generators UPS & Gen often on 480V 115kv Sub-station 13.2kv UPS: Rotary or Battery Transformers Transformers ~1% loss in switch gear & conductors 13.2kv 13.2kv 480V 0.3% loss 99.7% efficient 6% loss 94% efficient, ~97% available 2% loss 98% efficient 2% loss 98% efficient Note: Two more levels of power conversion at server 2011/10/27 http://perspectives.mvdirona.com 7
Removing Power Conversions Remove final conversion prior to server 480VAC line-to-neutral yields 277VAC 400VAC line-to-neutral yields 230VAC Both good but later supports standard PSUs Another option is HVDC distribution 400DC an interesting option Improved efficiency but higher capital cost 2011/10/27 http://perspectives.mvdirona.com 8
Power Distribution Efficiency Summary 2 more power conversions at servers 5. Power Supply: often under 80% at typical load 6. On board voltage regulators (VRMs or VRDs) Rules to minimize power distribution losses: Oversell power (more load than provisioned power) Avoid conversions (fewer & better) Increase efficiency of conversions High voltage as close to load as possible Size voltage regulators to load & use efficient parts High voltage direct current a small potential gain 2011/10/27 http://perspectives.mvdirona.com 9
Agenda Quickening Pace Infrastructure Innovation Influence of Cloud computing Power Distribution Cooling & Building Designs Networking & Server Innovation 2011/10/24 http://perspectives.mvdirona.com 10
Cold Hot Blow down & Evaporative Loss at 8MW facility: ~200,000 gal/day Cooling Tower CWS Pump Mechanical Systems Heat Exchanger (Water-Side Economizer) A/C Condenser A/C Compressor Primary Pump A/C Evaporator 20% of total power Server fans 6 to 9W each Diluted Hot/Cold Mix leakage cold fans Overall Mechanical Losses ~22% Computer Room Air Handler Air Impeller 2011/10/24 http://perspectives.mvdirona.com 11
Hot Aisle Containment Intel Facebook Open Compute Intel WriteLine 2011/10/24 http://perspectives.mvdirona.com 12
ASHRAE 2008 Recommendations ASHRAE 2008 Allowable Class 1 ASHRAE 2008 Recommended Class 1 64-81F 59F-90F 2011/10/24 http://perspectives.mvdirona.com 13
ASHRAE 2011 Recommendations ASHRAE 2011 Allowable Class 1 89F-95F ASHRAE 2011 Recommended Class 1 81F 64-81F 59F-90F 2011/10/24 http://perspectives.mvdirona.com 14
Network Equipment Building Standard NEBS Telco Standard (~1970) ASHRAE 2011 Allowable Class 1 89F-95F ASHRAE 2011 Recommended Class 1 81F ASHRAE 2008 Allowable Class 1 ASHRAE 2008 Recommended Class 1 64-81F 59F-90F 2011/10/24 http://perspectives.mvdirona.com 15
Most Datacenters Still Run Cold Most datacenters run down in this range 89F-95F 81F ASHRAE 2008 Recommended Class 1 ~77F 64-81F 59F-90F 2011/10/24 http://perspectives.mvdirona.com 16
Air Cooling Component temps specs higher than historically hottest place on earth Al Aziziyah, Libya: 136F/58C (1922) Just a mechanical engineering problem More air or better mechanical designs Tradeoff: semi-conductor leakage & power to move more air vs cooling savings Currently available equipment temp limits: 40C/104F: CloudRack C2 & most net gear 35C/95F: Most of the server industry Memory: 3W - 20W Temp Spec: 85C-105C Hard Drives: 7W- 25W Temp Spec: 50C-60C Processors/Chipset: 40W - 200W I/O: 5W - 25W Thanks to Ty Schmitt& Giovanni Coglitore Temp Spec: 60C-70C Temp Spec: 50C-60C 2011/10/27 http://perspectives.mvdirona.com 17
Innovative Shell Designs Evaporative cooling only High pressure misting on right Damp media design below Full building ductless cooling Facebook Prineville above & below EcoCooling 2011/10/27 http://perspectives.mvdirona.com 18
Modular and Pre-fab DC Designs Microsoft ITPAC Fast & economic deployments Sub-1.2 PUE designs Air-side economized In some cases no mechanical cooling ISO standard shipping containers offered by Dell, HP, SGI, IBM, Amazon Perdix 2011/10/27 http://perspectives.mvdirona.com 19
Cooling Looking Forward River water or lake water cooling Google Belgium & Finland, Deepgreen Switzerland Not new: Toronto metro area cooling Water direct to the rack IBM idataplex Water direct to components Done before 80 s IBM S/370 308x & 3090 series Again when heat densities climb back to that level Direct on component spray cooling 2011/10/24 http://perspectives.mvdirona.com 20
Agenda Quickening Pace Infrastructure Innovation Influence of Cloud computing Power Distribution Cooling & Building Designs Networking & Server Innovation 2011/10/24 http://perspectives.mvdirona.com 21
Sea Change in Networking Current networks over-subscribed Forces workload placement restrictions Goal: all points in datacenter equidistant Mainframe model goes commodity Competition at each layer over vertical integ. Get onto networking on Moore s Law path ASIC port count growth at near constant cost Competition: Broadcom, Marvell, Fulcrum, 2011/10/24 http://perspectives.mvdirona.com 22
Software Defined Networking Metering Routing QOS Monitoring Network Operating System (Onix) Openflow Simple Packet Forwarding Hardware Nick McKeown Openflow Simple Packet Forwarding Hardware Openflow Simple Packet Forwarding Hardware Distributed control plane with central control Research examples: VL2, PortLand, & others Openflow Simple Packet Forwarding Hardware Onix/OpenFlow gaining industry support & traction quickly Openflow Simple Packet Forwarding Hardware 2011/10/24 http://perspectives.mvdirona.com 23
Server Innovation Removing all unnecessary cost & power Omit lid, Depop board, efficient components, 12V-only PSUs Form factor: fractional RU & multi-server modules Shared power supplies N supplies for M servers Run supplies at most efficient load Shared large back-of-rack fans Cell phone technology predicts future server generations Super high-density storage platforms Increasing server to disk ratio for cold storage Soon: ARM architecture, low power servers, multiple servers on board, proprietary net fabric 2011/10/24 http://perspectives.mvdirona.com 24