A New Era of Hardware Microservices in the Cloud. Doug Burger Distinguished Engineer, Microsoft UW Cloud Workshop March 31, 2017

Transcription

1 A New Era of Hardware Microservices in the Cloud Doug Burger Distinguished Engineer, Microsoft UW Cloud Workshop March 31, 2017

2

3 Moore s Law Dennard Scaling has been dead for a decade Moore s La is o er Re e er, it s a rate Silicon scaling continues But lots of weirdnesses 3

4

5 Specialization (?)

6 Hardware Microservices HuS A instance 1 HuS A instance 2 Cloud (or client) service HuS B instance 1 HuS B instance 2 IP + us HuS C instance 1

7 Catapult v2 Mezzanine card Catapult V2 Architecture WCS 2.0 Server Blade DRAM Catapult V2 DRAM DRAM 40Gb/s QPI CPU Gen3 2x8 Gen3 x8 NIC FPGA QSFP Switch WCS Gen4.1 Blade with Mellanox NIC and Catapult FPGA QSFP CPU QSFP 40Gb/s The architecture justifies the economics Pikes Peak 1. Can act as a local compute accelerator 2. Can act as a network/storage accelerator 3. Can act as a remote distributed computing fabric Option Card Mezzanine Connectors WCS Tray Backplane 7

8 Configurable Cloud CPU compute layer Reconfigurable compute layer Converged network

9 Top-of-Rack Switch Server NIC HuS shell Support 4 8 FIFO 40G MAC x8 PCIe Core (HIP 0) Host CPU x8 PCIe Core (HIP 1) 40G 256 MAC FIFO G Bypass Ctrl GB DDR3 72 DDR3 Controller 512 Network Switch Lightweight Transport Layer Slot DMA Engine 256 Mb QSPI Config Flash 4 Config Flash (RSU) Clock 128 Elastic Router User Logic User Logic I2C SEU JTAG Temp. Role ASLs Shell 9

10 LTL Network reach and latencies 25 LTL L2 LTL Average Latency LTL 99.9th Percentile Round-Trip Latency (us) Example L1 latency histogram LTL L1 10 Example L0 latency histogram LTL L0 (same TOR) 5 Examples of L2 latency histograms for different pairs of FPGAs Number of Reachable Hosts/FPGAs 10K K K

11 Scaling the HuS fabric L2 L1 TOR TOR L1 TOR TOR Deep neural networks Web search ranking Web search ranking SQL GFT Offload TOR

12 Use case: shared service Most accelerators have more throughput than a single host requires Share excess capacity, use fewer instances Frees up FPGAs for other use services Sustains 3.6x clients / FPGA 73% remaining FPGAs available th Bed-Wide Remote-Local 95th Bed-Wide Remote-Local

13 Hardware as a Service architecture

14 Scaling HuS FPGAs To Ultra-Large DNNs Thanks to Eric Chung and team Distribute NN models across as many FPGAs as needed (up to thousands) Recent Imagenet competition: 152layer model Use HaaS and LTL to manage multifpga execution Only vectors travel over network Low FPGA-FPGA latency at ~1.8us per L0 hop

15 Huge infrastructure: Scale is the enabler Mi rosoft s Cloud usiness growing in the triple digits annually Dublin Chicago Quincy Cheyenne Finland Amsterdam Boydton Shanghai Des Moines Hong Kong San Antonio Singapore Brazil FPGAs Deployed across 15 countries and 5 continents Australia Catapult included in every new server for all major services In large scale production in both Bing and Azure (and others) Hardware Microservices (+ DNNs) in a subset and scaling Japan

16 Looking forward

17 Gen2 Shell Abstractions 4 8 Host CPU FIFO FIFO x8 PCIe Core (HIP 1) Mb QSPI Config Flash 4 DDR3 Controller 40G 256 MAC Network Switch Lightweight Transport Layer Slot DMA Engine G Bypass Ctrl 40G MAC x8 PCIe Core (HIP 0) 4 GB DDR3 Top-of-Rack Switch Server NIC 128 SDN Elastic Router Config Flash (RSU) Clock User Logic I2C SEU JTAG Temp. Role ASLs Shell 17

18 Democratizing Hardware Microservices Network Host FPGA Third-party ASIC

19

20 FPGAs Deployed across 15 countries and 5 continents Catapult included in every new server for all major services In large scale production in both Bing and Azure (and others) Hardware Microservices (+ DNNs) in a subset and scaling

21 Accelerated Networking Generic Flow Table (GFT) rule based packet rewriting 10x latency reduction vs software, CPU load now <1 core 25Gb/s throughput at < 25µs latency the fastest cloud network VMs On Haswell, AES GCM-128 costs 1.26 cycles/byte[1] (5+ 2.4Ghz cores to sustain 40Gb/s) CBC and other algorithms are more expensive AES CBC-128-SHA1 is 11µs in FPGA vs 4µs on CPU (1500B packet) Higher latency, but significant CPU savings GFT FPGA 40G [1] S. Gulley, et al. Has ell Cryptographi Perfor a e Flow Action > , >80 Decap, DNAT, Rewrite, Meter Crypto 40G NIC 40GGFT Table

22 Training Inference Client Cloud Humans GPUs ASICs FPGAs