Priority-aware Coflow Placement and scheduling in Datacenters

Size: px

Start display at page:

Download "Priority-aware Coflow Placement and scheduling in Datacenters"

Jewel Johnston
5 years ago
Views:

1 Pririty-aware Cflw Placement and scheduling in Datacenters Speaker: Lin Wang Research Advisr: Biswanath Mukherjee

2 Intrductin Cflw Represents a cllectin f independent flws that share a cmmn perfrmance gal. Cflw s perfrmance depends n its slwest flw. Cflw aware scheduling benefits distributed data prcessing applicatins. State-f-arts Mst f current wrk fcus n ptimizing the netwrk scheduling algrithm t imprve cflws perfrmance. They assume predetermined cflw placement, i.e. the endpint lcatins f a Cflw are preset. But Cflw s placement can be mre flexible in practice.

3 Cflw placement challenge Challenge fr inter-flw relatinship in a Cflw E.g., in a ne-t-many Cflw, all cnstituent flws share the same sender lcatin. In many-t-many Cflw, the relatinship is even mre cmplex. Because any member flw shares its tw ends pints with tw different grups f flws. Thus, we need t take care f such inter-flw relatinship fr placement decisins.

4 2D Placement fr Cflw Netwrk Mdel Tplgy designs such as Fat-tree r Cls enable full bisectin bandwidth in datacenters. Assume nn-blcking N-prt switch with link bandwidth B. Switch prts are cnnected t ndes, which can be hst machines r TR switches. Only edge links are cngested and cre is cngestin free. Scheduling bjective Minimize Cflw cmpletin time (CCT). It is the duratin t finish all flws in a Cflw t speed up applicatin level perfrmance.

5 2D Placement fr Cflw Prblem Statement K Cflws arrive at varius time. We want t decide the placement fr each new-arrived Cflw. The placement f a Cflw can be represented by mapping functins We assume when a Cflw arrives, its traffic demand D is available. Thus, we need t decide the placement f a new Cflw given the existing previus Cflws, s that the sum f all Cflws CCTs is minimized.

6 2D Placement fr Cflw Prblem Analysis The sum f CCTs is jintly determined by Cflw s placement and the netwrk scheduling during runtime. First, Cflws placement decides the ptimal sum f CCTs achievable by any netwrk scheduling plicy. Secnd, after Cflws are placed, the sum f CCts will be further determined by the netwrk scheduling plicy, which arbitrates bandwidth allcatin fr each Cflw. 2D Placement fcus n finding Cflw placement that minimizes the sum f CCTs under ptimal netwrk scheduling. Given specific placement, finding the ptimal scheduling plicy t minimize the ttal CCTs is NP-hard.

7 Mtivatin Example 1

8 2D Placement fr Cflw Observatins

9 Mtivatin Example 2

10 2D Placement fr Cflw First step Calculate the traffic demand requested n each endpints fr Cflw t place. Secnd step 2D-Placement cnsiders each sender (r receiver) in the descending rder f their requested demand, and place the sender (r receiver) nt the input (r utput) prt with the minimum traffic lad. Cmplexity O(n^2).

11 Simulatin Apply tw netwrk schedulers Varys assume accurate Cflw traffic request. Aal tries t apprximate Varys with unknwn sizes s as t tlerate errr in the requested demand.

12 Explit benefits f Pririty-aware Cflw Placement and scheduling in Datacenters Weakness f previus wrk In previus wrk, Cflw placement and scheduling are cnsidered independently. 2D-placement nly cnsiders current wrklad when placing Cflw.

13 Explit benefits f Pririty-aware Cflw Placement and scheduling in Datacenters Pririty-aware Cflw placement and scheduling When placing a new Cflw, we als take scheduling methd int cnsideratin. In this way, Cflw placement and scheduling can benefit each ther as they are mre cnsistent. Fllwing the bservatin in 2D-placement methd, we still place the new cflw nt light weighted ndes. But we will nt nly cnsider the size f current flw, but als ther measurements f current flw. Fr example, we can either place the new flw nt the minimum distant nde r the minimum lad nde. In ur PSON architecture, lngest queue fist (LQF), largest number f packets first (LNPF), ldest packet first (OPF), and less space switch tuning time first (LSSTTF) can all be taken int accunt.

14 Explit benefits f Pririty-aware Cflw Placement and scheduling in Datacenters Weakness f previus wrk In previus wrk, Cflw placement and scheduling are cnsidered independently. 2D-placement nly cnsiders current wrklad when placing Cflw.

15 Explit benefits f Pririty-aware Cflw Placement and scheduling in Datacenters Simulatin Settings We use benchmark Hadp (map-reduce) and web-search wrklads. These wrklads cntain a diverse mix f shrt and lng flws with a heavy-tailed flw size distributin. In the websearch wrklad, mre than 75% f all bytes are frm 50% f the flws with sizes in the range 1 t 20 MB. The Hadp wrklad is less skewed: ~50% f the flws are less than 100MB in size and 4% flws are larger than 80GB. We cmpare ur mixed weight pririty methd with mindist and minlad methds.

16 Explit benefits f Pririty-aware Cflw Placement and scheduling in Datacenters Simulatin Results Hadp data flws mix

17 Explit benefits f Pririty-aware Cflw Placement and scheduling in Datacenters Simulatin Results Web search data flws mix

Datacenter Traffic Measurement and Classification

Datacenter Traffic Measurement and Classificatin Speaker: Lin Wang Research Advisr: Biswanath Mukherjee Grup meeting 6/15/2017 Datacenter Traffic Measurement and Analysis Data Cllectin Cllect netwrk events