Flexplane: An Experimenta0on Pla3orm for Resource Management in Datacenters

Transcription

1 Flexplane: An Experimenta0on Pla3orm for Resource Management in Datacenters Amy Ousterhout 1, Jonathan Perry 1, Hari Balakrishnan 1, Petr Lapukhov 2 1 MIT, 2 Facebook

2 Datacenter Networks Applica0ons have diverse requirements Dozens of new resource management schemes Low latency: DCTCP Min FCT: PDQ, RCP, pfabric, PERC Deadlines: D 3, D 2 TCP Difficult to experiment with schemes in real networks Requires changes to hardware routers

3 Experimenta0on with Resource Management Experimenta0on in real networks Programmable hardware - limited flexibility SoVware routers - limited throughput Emula0on limited throughput By Altera Corpora0on - Altera Corpora0on, CC BY 3.0

4 Experimenta0on with Resource Management Experimenta0on in simula0on (e.g., ns, OMNeT++) Does not accurately model real network stacks, NICs, and distributed applica0ons Does not run in real 0me No exis0ng approach to experimenta0on provides accuracy, flexibility, and high throughput

5 Our Contribu0ons Key idea: whole- network emula0on Flexplane: a pla3orm for faithful experimenta0on with resource management schemes Accurate predicts behavior of hardware Flexible express schemes in C++ High throughput 761 Gbits/s

6 Approach: Whole- Network Emula0on Real Network Emulated Network Emulator class MyScheduler { } class MyAQM { } packet

7 Abstract Packets Resource management schemes are data- independent Concise representa0on of one MTU Source, des0na0on, flow, ID Custom per- scheme fields Transmiged over the real network Consume 1-2% of network bandwidth

8 Emulator Real- 0me network simulator Faster than standard network simulators Time divided into abstract- packet- sized 0meslots Omits endpoint sovware Emulated Network Real Network Emulator

9 Sending Real Packets Drop packet if dropped in emula0on Otherwise, modify and send ECN marks, rates (e.g., in RCP, PDQ, PERC)

10 Accuracy Goal: predict behavior of a hardware network Hardware latency: unloaded delay + queuing delay Added latency of Flexplane: RTT to emulator Unloaded delay Queuing delay in real network Real Network Emulator Emulated Network

11 Flexplane API Decouples schemes from framework int route(abstractpkt *pkt) Emulator int classify(abstractpkt *pkt, int port) Emulator (pagern) enqueue(abstractpkt *pkt, int port, int queue) AbstractPkt *schedule(int output_port) Emulator (generic) Endpoints input(abstractpkt **pkts, int n) output(abstractpkt **pkts, int n) prepare_request(sk_buff *skb, char *request_data) prepare_to_send(sk_buff *skb, char *alloca0on_data) incoming packets outgoing packets route classify enqueue schedule

12 Mul0core Emulator Architecture Common approaches are ill- suited for Flexplane Parallel (RouteBricks, IX) Pipeline (Fastpass) CPU core abstract packet 4 - > 3 aggrega0on 1 - > 3 ToR endpoints 8 - > > 8 endpoints 1-6 ToR, aggrega0on endpoints 7-12

13 Mul0core Emulator Architecture Pin network components to cores, connect in a graph Router state not shared across cores Loose synchroniza0on, batching, prefetching aggrega0on ToR endpoints CPU core FIFO queues

14 Implementa0on Emulator uses Intel DPDK for low- latency NIC access Endpoints run a Linux qdisc

15 Accuracy U0lity Emulator throughput Evalua0on

16 Flexplane is Accurate Bulk TCP: 5 senders, 1 receiver Throughput Gbits/s vs. 9.4 Gbits/s in hardware Similar queue occupancies Median Queue Occupancies (MTUs) Hardware Flexplane DropTail RED DCTCP : Flexplane achieves similar queue occupancie

17 Flexplane is Accurate Average Normalized FCT RPC web search workload DCTCP Average Normalized FCT (0, 100KB] Load Average Normalized FCT Load Load 250 Average Normalized FCT Load (b) DCTCP, (0, 100KB] 0 (b) DCTCP, (0, (c) 100KB] DropTail, (10MB, ) (d) DCTCP, (10MB, ) Accurate 0.2 to within % 0.8 for loads up to 60% Load es the average normalized flow completion times of hardware for DropTail and DCTCP. The left ow sizes; the right two graphs show results for large flow sizes. ic Scheduler { Average Normalized FCT Flexplane 5 Hardware Observe behavior not visible in simula0ons 250 Priority Queueing DCTCP (10MB, )

18 Flexplane is Easy to Use Implemented several schemes in dozens of lines of code scheme LOC drop tail queue manager 39 RED queue manager 125 DCTCP queue manager 43 priority queueing scheduler 29 round robin scheduler 40 HULL scheduler 60 pfabric QM, queues, scheduler 251

19 Example: DCTCP in Flexplane void enqueue(abstractpkt *pkt, uint32_t port, uint32_t queue) { uint32_t qlen = m_bank- >occupancy(port, queue); if (qlen >= m_dctcp_params.q_capacity) { /* no space to enqueue, drop this packet */ drop(pkt, port); return; } /* mark if queue occupancy is greater than K */ if (qlen > m_dctcp_params.k_threshold) { /* Set ECN mark on packet */ mark_ecn(pkt, port); } } m_bank- >enqueue(port, queue, pkt);

20 Flexplane Enables Experimenta0on Evalua0ng trade- offs between resource management schemes High priority FCT (us) HULL DCTCP DropTail HULL better Priority Queueing RED Round Robin Priority Queueing DropTail pfabric Low priority FCT (ms) 35 30

21 Flexplane Enables Experimenta0on Experiment with real distributed applica0ons such as Spark % Change in Completion Time Relative to DropTail Coordinate descent Sort DCTCP +4.4% -4.8% HULL +29.4% -2.6% e 4: Percent change in completion time of two Spark Performance depends on network and CPU

22 Emulator Throughput Emulator provides 761 Gbits/s of aggregate throughput with 10 total cores Router throughput (Gbits/s) racks connected with Agg isolated racks Emulation cores

23 Emulator Throughput Throughput varies by 29% across schemes Throughput (Gbits/s) DCTCP DropTail Priority Queueing Round Robin Resource management scheme HULL RED pfabric 81x as much throughput per clock cycle as RouteBricks

24 Flexplane: an Experimenta0on Pla3orm Whole- network emula0on Flexplane: a pla3orm for faithful experimenta0on with resource management schemes Accuracy, flexibility, and high throughput hgps://github.com/aousterh/flexplane