Coded Distributed Computing

Transcription

1 Coded Distibuted Computing Salman Avestimeh USC joint wok with Songze Li (USC), Qian Yu (USC), and Mohammad Maddah-Ali (Bell-Labs) Asiloma Confeence Nov. 2016

2 Infastuctues fo (Big) Data Analytics How to optimally utilize esouces? stoage communication esouces computation enegy time

3 Fundamental Tadeoffs between Resouces stoage communication how to optimally tade netwok esouces? computation enegy time

4 Computation-Communication Tadeoff L() Communication Load What is the optimal Tadeoff? Computation Load

5 Computation-Communication Tadeoff MapReduce-Type Famewoks L() Map Reduce Map Reduce Map Reduce Communication Load # of exchanged intemediate values What is the optimal Tadeoff? Computation Load aveage # times each file is mapped

6 Today s Design L() Fully Distibuted Communication Load 1 K 1 K 1 L =1 Fully Centalized Computation Load K K: numbe of paticipating seves K Fo K=100, doubling computation only educes the communication by 1%!

7 Today s Design L() Fully Distibuted Communication Load K 1 K 1 L =1 K 1 Fully Centalized Computation Load K Is this the best tadeoff?

8 Coded Distibuted Computing L() Communication Load K 1 K 1 Computation Load K Comm. Load (Coded) L coded =(1 K )1 = L uncoded Fo K=100, doubling computation educes the communication by 50%! communication load 1 computation load [*] A fundamental tadeoff between computation and communication in distibuted computing, axiv

9 Key Idea Caeful assignment of tasks to seves, such that multicast coding oppotunity of size aises in the data shuffling phase L() Communication Load Computation Load K

10 Key Idea Caeful assignment of tasks to seves, such that multicast coding oppotunity of size aises in the data shuffling phase Example: 6 inputs, 3 seves, 3 functions, computation load of =

11 Key Idea Caeful assignment of tasks to seves, such that multicast coding oppotunity of size aises in the data shuffling phase Example: 6 inputs, 3 seves, 3 functions, computation load of =2 CDC Mapping

12 Key Idea Caeful assignment of tasks to seves, such that multicast coding oppotunity of size aises in the data shuffling phase Example: 6 inputs, 3 seves, 3 functions, computation load of =2 CDC Mapping L uncoded = L coded = Each coded packet is useful fo two seves

13 Theoem: Coded Distibuted Computing scheme achieves L coded = L uncoded =(1 K )1 L() Communica(on Load K 1 K 1 Computa(on Load K 13

14 Can we do bette? Theoem: The poposed Coded Computing famewok is optimal. L () =L coded () = L uncoded() L() Communica(on Load K Computa(on Load Communication Load x Computation Load ~ constant

15 L() Communication Load What is the Impact? Computation Load K

16 (1) Speeding Up Distibuted Computing We can educe the total computation time by tading Map time with Shuffle time T total = E[T Map + T Shu e + T Reduce ] T total, CDC =min E[T Map + T Shu e Fo example, conside distibuted soting using Teasot algoithm Ø by choosing =23, CDC pomises to povide 90% speed up! + T Reduce ] Map Pack Shuffle Unpack Reduce Total (sec.) (sec.) (sec.) (sec.) (sec.) (sec.) Table0.85 I PERFORMANCE OF TERASORT SORTING 12GB DATA WITH K =16 NODES AND 100 MBPS NETWORK SPEED

17 Pactical Challenges of CDC Multicast capability Ø Coded Distibuted Computing takes advantage of coded multicast to educe comm. load Ø Netwok-laye multicast is disabled in most data cente netwoks like Amazon EC2 Ø We can instead use application-laye multicast by using Massage Passing Inteface (MPI) Encoding/Decoding Complexity Ø The delays caused by Encoding/Decoding cannot be ignoed Ø We limit ouselves to small coding gains Stoage Optimization Ø Coded Distibuted Computing optimizes input file placement to enable coded multicast Ø Data is often stoed without pio knowledge of the computation tasks (e.g., HDFS, GFS) Ø We can instead use andom placement!

18 Expeiments ove Amazon EC2 We have implemented Coded Teasot ove Amazon EC2 Map Pack Shuffle Unpack Reduce Total (sec.) (sec.) (sec.) (sec.) (sec.) (sec.) Table0.85 I PERFORMANCE OF TERASORT SORTING 12GB DATA WITH K =16 NODES AND 100 MBPS NETWORK SPEED Results: Table II SORTING 12 GB DATA WITH K =16NODES AND 100 MBPS NETWORK SPEED CodeGen Map Pack/Encode Shuffle Unpack/Decode Reduce Total Time Speedup (sec.) (sec.) (sec.) (sec.) (sec.) (sec.) (sec.) TeaSot: CodedTeaSot: = CodedTeaSot: = CDC povides 50% - 70% speed up

19 (2) Beaking the Paallelization Limit Cuent view: map execution time data shuffling load numbe of seves numbe of seves Ø Spead just enough to make Map execution time data shuffle time e.g., ishuffle [Guo, et al. 13] Shuffle Reduce Reduce Tasks Map Tasks Incease # seves by 4 times 500 Time (s) 125 Coded Computing with Computation Load =2 250

20 (3) Scalable Wieless Distibuted Computing A Dataset of N Files: w 1,w 2,...,w N 2 GF (2 F ) K Mobile Uses: Use k has an input, and wants to compute output function ( d {z} k ; w 1,...,w N ) {z } input dataset d k Uses can communicate via an access point at the edge of a wieless netwok Uplink Downlink Access Point Dataset Befoe computation, Use k can stoe a subset of µn files in local memoy U k Output Input Use 1 files Use 2 Output files Use K Output Input files Input

21 Gain of Coded Distibuted Computing Without coding, the equied communication load is L u = L d = L uncoded = K(1 µ) scales linealy with netwok size How much coded distibuted computing can help? each node pocesses µn files thee ae N files and K nodes each file is pocessed at µk nodes Coded distibuted Computing povides coding gain of linealy with netwok size Uplink µk, which scales Downlink Access Point Dataset files files Output Input Use 1 Use 2 Output files Use K Output Input Input

22 Pomising a Scalable Design L u = L d = L coded = K(1 µ) Kµ = 1 µ 1 can accommodate any numbe of uses without inceasing the communication load ibuted computing famewok. Uplink Downlink Access Point Dataset files files Output Input Use 1 Use 2 Output files Use K Output Input Input 1: System model of the edge-facilitated wieless distib [*] A Scalable Famewok fo Wieless Distibuted Computing, axiv:

23 Conclusions and Futue Diections Coding plays a fundamental ole in distibuted computing by enabling optimal tadeoffs between esouces Many futue diections Multistage computation models Heteogeneous netwoks Multi-laye and stuctued netwoks Othe tadeoffs stoage communica-on computa-on enegy -me