Communication vs Distributed Computation: an alternative trade-off curve

Communication vs Distibuted Computation: an altenative tade-off cuve Yahya H. Ezzeldin, Mohammed amoose, Chistina Fagouli Univesity of Califonia, Los Angeles, CA 90095, USA, Email: {yahya.ezzeldin, mkamoose, chistina.fagouli}@ucla.edu axiv:1705.08966v1 [cs.it] 24 May 2017 Abstact In this pape, we evisit the communication vs. distibuted computing tade-off, studied within the famewok of MapReduce in [1]. An implicit assumption in the afoementioned wok is that each seve pefoms all possible computations on all the files stoed in its memoy. Ou stating obsevation is that, if seves can compute only the intemediate values they need, then stoage constaints do not diectly imply computation constaints. We examine how this affects the communicationcomputation tade-off and suggest that the tade-off be studied with a pedetemined stoage constaint. We then poceed to examine the case whee seves need to pefom computationally intensive tasks, and may not have sufficient time to pefom all computations equied by the scheme in [1]. Given a theshold that limits the computational load, we deive a lowe bound on the associated communication load, and popose a heuistic scheme that achieves in some cases the lowe bound. I. INTRODUCTION Distibuted computation acoss a set of wieless netwoked seves is well motivated fo seveal pactical constaints: we may want to speed up computation time so as to finish a computation faste; we may have patial view of the files needed fo computation acoss seves; we may have limited memoy in each seve; o we may be motivated by enegy constaints. In this pape we conside the distibuted computing famewok studied in [1], that follows the achitectue of MapReduce [2]. Ou stating obsevation is that, the system in [1] does not explicitly sepaate computation fom stoage. The system uses a cluste of seves to compute Q output functions fom N input files. Each file is stoed in diffeent seves, balancing the amount of stoage acoss seves. The wok in [1] calculates the tade-off between the amount of computation and communication that seves need to do fo such file placement. Howeve, an undelying assumption of the deived tade-off, is that each seve pefoms all possible computations on all the files stoed in its memoy. It is natual to ask: is it indeed useful to pefom all possible computations? The following simple example illustates that this is not always the case. Conside a cluste with =3 seves, N=3 files and Q=3 output functions. All 3 files ae available at each seve and each seve is equied to compute only one of the output functions. In this case, instead of pefoming 9 computations pe seve (as assumed in [1]), each seve only needs to pefom computations elated to its dedicated output function, i.e., only 3 computations ae needed pe seve. Ou fist contibution is to genealize this obsevation and deive an altenative tade-off cuve to the scheme in [1]. We explicitly use thee paametes: C total the total amount of computation equied; that captues the memoy equiements; and the communication load L. We conside the placement and communication scheme in [1], and calculate the minimum numbe of computations each seve needs to pefom. We take into account the amount computed by the seve fo its assigned output functions, the amount that need to be communicated to othe seves, and the amount needed to use as side infomation to decode tansmissions fom othe seves. We then poceed to examine the case whee seves need to pefom computationally intensive tasks, and in paticula, do not have sufficient time to pefom all computations the cuve in [1] equies. Such a scenaio may occu in wieless, whee we may have cheap mobile devices with low computational powe that need to coopeatively pefom time-citical opeations, fo scientific computing o vitual eality applications. We ask, if the cluste is limited to pefom an amount of computation below a theshold, what is the esulting minimum communication equied to achieve the function computation. Ou second contibution is to deive a lowe bound fo the communication-computation tade-off when a cluste has a limited computation budget. Fo this lowe bound, we assume that the files ae distibuted acoss the cluste with a pedetemined level of edundancy that does not gow with the available computation budget. We show that a scheme diectly infeed fom [1] pefoms pooly when compaed against the deived lowe bound. Finally, we develop a distibuted computing scheme inspied by [1] and show though numeical evaluation that the communication-computation tade-off it povides is compaable to the afoementioned lowe bound. Related Wok. Minimizing communication load fo distibuted computation tasks has eceived consideable attention in the liteatue: stating fom distibuted boolean function computation between two paties [3], [4] to the moe genealized theoy of communication complexity [5], [6]. A key concept in educing the needed amount of communication is though netwok coding. A pominent example of this concept is in the context of distibuted cache netwoks [7], [8], [9], whee coding is used in eithe the data placement o data delivey phases to educe the amount of communication in the delivey phase. Recently, coding was also consideed in the context of distibuted computing systems that ae based on the MapReduce famewok [1], [10], [11]. In fact, the authos in [1] povided a Coded Distibuted Computing (CDC) scheme which educes the amount of communication needed in the

data shuffling phase by using coded multicast tansmissions. Ou wok diffes in that we sepaate computation and stoage, and thus deive altenative tade-off cuves depending on the elative values of these paametes. II. SYSTEM MODEL Notation. Calligaphic lettes denote sets though out the pape. A denotes the cadinality of the set A. The expession [a : b] denotes the set of integes fom a to b. MapReduce famewok. We conside a cluste of seves that computes Q output functions φ q, q [1 : Q], fom N input files w n, n [1 : N]. In this pape, we assume that the seves shae a lossless boadcast domain: a tansmission fom a seve can be losslesly eceived by all othe seves. We assume the cluste uses a MapReduce famewok to compute the set of Q functions in a distibuted manne. MapReduce is based on the assumption that each output function can be calculated as a function of some intemediate pocessing of the files. In othe wods, φ q (w 1,..., w N ) = h q (v q,1,..., v q,n ), whee v q,n = g q,n (w n ) is the intemediate value computed fom file w n elevant to the output function φ q, and has length T bits. In MapReduce teminology, the intemediate value is computed (o mapped ) using a map function g q,n and h q educes the intemediate values {v q,n } N n=1 to output φ q. Based on this decomposition, the computation model in [1] consists of thee phases: Map, Shuffle and Reduce. Additionally, a Placement phase distibutes files and tasks among the seves in the cluste. We next descibe each of the phases: 1) Placement Phase: Each seve k is loaded with a subset M k of the N files, such that k M k = [1 : N]. Each seve k is also assigned to compute a patition W k of the output functions, whee k W k = [1 : Q]. 2) Map Phase: Each seve k computes a subset C k of the intemediate values elated to M k, i.e., C k {v q,n q [1 : Q], n M k }. At the end of the Map phase, the assigned computation subsets satisfy that k C k = {v q,n q [1 : Q], n [1 : N]}. Remak 1. In MapReduce, files ae mapped by pesenting them as (key, value) pais to a map( ) function that outputs a set of intemediate (key, value) pais based on the input pai. Although, the same map( ) build is used acoss the seves, the function can output diffeent sets intemediate values based on the seve ID by including this infomation in the key. 3) Shuffle Phase: Fo a seve k to compute a function φ q whee q W k, it needs all the intemediate messages V q = {v q,n q W k, n [1:N]}. Thus in the Shuffle phase, the seves exchange intemediate values, such that each seve has access to all its needed sets V q. The shuffling scheme can be descibed as follows: each seve k ceates a message X k that is a function of its locally computed intemediate values and boadcasts this message X k to the emaining 1 nodes. 4) Reduce Phase: In the Reduce phase, seve k uses its locally computed intemediate values and the eceived tansmissions X 1,..., X to decode the set of the needed intemediate values V q, q W k. Using V q, the nodes can now compute the desied functions φ q = h q (V q ), q W k. Pefomance metics. We measue the pefomance of this computation cluste acoss thee paametes: the load edundancy (), the computation load (C total ) and the communication load (L), defined as follows: Load Redundancy. We define the load edundancy as the aveage numbe of times a file is assigned acoss the seves. k=1 M k We denote this by, i.e., N. Load edundancy captues memoy constaints. Computation Load. We define the computation load C total k C k as the total numbe of computations pefomed acoss seves in the cluste. Communication Load. We define the communication load L b(x i) k=1 QNT, as the numbe of bits tansmitted in the Shuffle phase nomalized by QNT, whee b(x i ) is the numbe of bits used to epesent X i and QNT is the total numbe of bits in all intemediate values v q,n, fo q = [1 : Q] and n = [1 : N]. Fom the definition, we have 0 L 1. The definitions of L and follow [1]; howeve in this pape, we explicitly sepaate the edundancy fom the computation load, and use diffeent paametes fo each. III. ON THE RELATION BETWEEN REDUNDANCY AND COMPUTATION An undelying assumption in [1] is that each seve k must compute all the intemediate value fo its stoed files M k. In othe wods, C k = {v q,n q [1 : Q], n M k }. In this case, the load edundancy is linealy popotional to the total numbe of computations in the system as C k = M k Q and can be theefoe egaded as the computation load. Howeve, if the seve can selectively choose which intemediate values of M k to compute in the Map phase (as long as the communication load is the same), then the total numbe of computations is not necessaily linealy coelated with. Consequently, an incease in does not necessaily esult in an incease in the numbe of computations pefomed by the cluste. Fo example, assume that Q = and each seve is equied to compute 1 output function (without loss of geneality, W k = {k}). Then, we have C total = fo both = 1 and =. Fo = 1, each file is available at only one seve, thus each seve needs to compute all intemediate values fo all files stoed in its memoy. Fo =, all files ae available at each seve. Thus, each seve needs only to compute N intemediate values elated to its output function. In both cases, the optimal communication load L() = is achieved [1, Theoem 1]. Note that fo =, if the seves computed all intemediate values fo thei files, thee would be computations instead of. Late in this section, we chaacteize the minimum computation load needed by the Coded Distibuted Computing (CDC) scheme in [1] in ode to achieve the optimal communication load L () in [1, Theoem 1] fo = [1 : ]. As we see late, taking this minimum computation load into account changes the tade-off in [1] fo CDC. As a peliminay to that discussion, we next biefly descibe the CDC scheme in [1].

An oveview of the CDC scheme. Assume that N and Q ae sufficiently lage so that N = ( ) η1 and Q = η 2 fo some η 1, η 2 N. The CDC scheme opeates as follows (see [1] fo a complete desciption): 1) Placement Phase: A disjoint subset M T of the files is assigned to each subset T of seves whee M T = η 1. Evey seve is thus assigned a set of N/ files and evey η 1 patition of these files is shaed with a unique set of 1 othe seves. Evey seve k is also assigned a unique subset W k of the output functions to calculate such that W k = η 2. 2) Map Phase: Evey seve computes all possible intemediate function values fo the files it has. 3) Shuffling Phase: The shuffling phase epeats the following pocedue fo evey set S [1 : ] of size + 1: (i) Fo evey i S, define S i =S\{i} and identify VS i i as V i S i {v q,n n j Si M j, q W i }. (1) The set VS i i epesents the intemediate values that ae needed by seve i to compute functions in W i, which can be computed exclusively by all seves in S i (ecall that a file is eplicated at exactly seves). Note that VS i i =η 1 η 2. (ii) Split evey intemediate value in VS i i into disjoint pats of T/ bits and associate each pat with a seve in S i. Thus we split the set VS i i into patitions denoted by VS i, j S i,j i, each T of size η 1 η 2. Each seve j will be esponsible to convey its pat to seve i with coded boadcast tansmissions. (iii) Afte splitting all sets VS i i fo all i S (we have + 1 T such sets), seve k sends the bit-wise XOR of all the η 1 η 2 - bit pats in Uk S i S Vi S, i.e., it makes η i,k 1η 2 boadcast tansmissions each of size T bits. Each tansmission is useful to all othe nodes in S; moeove, each seve in S has the equied side infomation to decode the pat it needs. 4) Reduce Phase: In the educe phase, evey seve uses its locally computed intemediate values and the decoded intemediate values in the shuffling phase to compute the η 2 output functions assigned to it in the initialization phase. Next we discuss the minimum computation load needed fo the CDC scheme. Minimum Computations. The next poposition chaacteizes the minimum computation equied by the CDC scheme. Poposition 1. Fo the placement scheme in [1] with = [1 : ], the communication load L () = can be achieved with computation load ( + 1) C total =. (2) Poof. We fist note that evey seve i locally computes all intemediate values equied by the functions in W i and coesponding to the files in M i ; we denote these intemediate values as C Mi,W i. Thus, we have C Mi,W i = {v q,n q W i, n N M i } C i. Note that C Mi,W i = W i M i =η 2. In addition to C Mi,W i, seve i also pefoms a set of computations equied to cay out shuffling in the CDC scheme. We denote this set by C T Xi. To calculate the numbe of computations in C T Xi, we distinguish between computations equied by seve i to decode its needed intemediate values (fom tansmissions in the shuffling phase) and the computations needed to ceate its tansmissions X i in the shuffling phase. Obseve (fom the desciption of the CDC scheme ealie and in [1]) that in any S [1 : ] of size + 1 whee i S, seve i uses the sets {VS k k,i, k S\{i}} to constuct its tansmission. In addition, since the emaining pats {VS k k,j k S\{i}, j S\{i, k}} will be XOR-ed (at the othe seves) with pats needed by seve i, then seve i should compute the intemediate values k S\{i} VS k k in ode to decode its equested intemediate values as well as constuct its tansmissions in the shuffling phase. This amounts to k S,k i Vk S k = η 1 η 2 computations fo evey set S. Thus, the total numbe of computations by seve i, C i, is C i = C T Xi + C Mi,W i ( ) (( ) (i) 1 N (ii) 1 = η 1 η 2 + η 2 = η 2 η 1 + N ) ( ( 1 (iii) = Q ) ) N ( )+ N = ( +1) 2, (3) whee: (i) follows since seve i appeas in only ( ) 1 subsets of size + 1; (ii) and (iii) follow fom the assumptions that N = ( ) η1 and Q = η 2. Fom symmety, the total numbe of computations in the Map phase equals C total = C i. Note fom (2) that C total is quadatic in. Thus, we cannot view as a diect measue of computation load since both the communication load L as well as the numbe of computations C total educe fo ( + 1)/2. Fig. 1 shows the elation in (2) fo N = 2520 and = Q = 10 vesus the numbe of computations if a seve compute all map functions fo each of its stoed files. If we use [1, Theoem 1] and Poposition 1 to couple C total and L, then we get the tade-off shown in Fig. 2 fo the CDC scheme, whee the ed line is a scaled vesion of the tade-off in [1]. Fom Fig. 2, it can be seen that if we ae fee to choose fo a given C total, then the optimal tade-off happens at C total = = 25200; by picking = = 10. This gives a communication load equal to zeo while achieving the minimum computation load. This obsevation suggests that we can bette undestand the communication-computation tade-off, if we conside it with a pedefined edundancy load () that does not change with the computation load C total. Thus, in the emainde of the pape, we conside as a paamete of the cluste (with, Q and N), and show how we can exploit this edundancy to pefom coded distibuted computing when at most C total computations ae allowed. IV. AN ACHIEVABLE COMMUNICATION-COMPUTATION TRADE-OFF Conside a distibuted computing cluste with paametes N, Q, and load edundancy, whee epesents the numbe of times each file is stoed acoss the seves in the cluste. Fo ou discussion in this section, we assume that [1 : ] and that the file placement (fo a given ) follows the stategy in [1]. We ae inteested in answeing the question: If the

Computation Load (Ctotal) 10 5 N = 2520, Q = 10, = 10 3 Minimum computations equied fo CDC Computations pefomed fo CDC in [1] 2.5 2 1.5 1 0.5 0 2 4 6 8 10 Load Redundancy () Fig. 1: Computation Load vs. Load Redundancy. Communication Load (L) 0.8 0.6 0.4 0.2 N = 2520, Q = 10, = 10 Minimum computations equied fo CDC Computations pefomed fo CDC in [1] 0 0 0.5 1 1.5 2 2.5 Computation Load (C total ) 10 5 Fig. 2: Communication Load vs. Computation Load. Communication Load (L) 0.8 0.6 0.4 0.2 N = 2520, Q = 10, = 10, = 5 Split-CDC (Patial Computations) Split-CDC (Intege Computations) CDC-fit Lowe Bound 3 4 5 6 7 Computation Load (C total ) 10 4 Fig. 3: Communication Load vs. Computation Load fo S-CDC. cluste is allowed to pefom at most C total computations, what is the minimum communication load L(, C total ) needed in ode to compute Q output functions using the cluste? If C total ( +1)/, then fom Poposition 1, we can diectly use the CDC scheme descibed in [1], to achieve the( optimal ) communication load L(, C total ) = L () = 1 1. Howeve, when Ctotal < ( + 1)/, then the available computation budget is not enough to pefom the shuffling and decoding equied by the CDC scheme. In this case, can the CDC scheme be adapted to wok with a estictive computation budget? Fom [1], we can infe a simple modification to the CDC scheme, which we efe to as CDCfit. In this scheme, we use CDC on the cluste while opeating it with a lowe load edundancy that fits the computation constaints. In othe wods, we pick = max{ C total ( + 1)/, } and opeate the cluste as if the files ae only epeated times. This ensues that thee ae enough computations to satisfy CDC ( fo) and achieve the communication load L( ) = 1 1. A natual question to ask hee is whethe this is the best possible appoach? To chaacteize this, we next develop a lowe bound on the communication load when the cluste has a computation load C total and load edundancy. Lowe Bound on Communication load. We povide hee a lowe bound on the communication load fo only a paticula class of shuffling schemes. In this class, given a boadcast tansmission sent duing the shuffling phase, seve i can decode its equied intemediate value fom that tansmission using only side infomation that it has locally computed. i.e., it does not ely on futue tansmissions to povide it with enough linea combinations to decode its equied intemediate values. In what follows, an l-type tansmission denotes a boadcast tansmission made by a seve duing shuffling, which consists of the XOR of equally-sized pats of l intemediate values. The weight of an l-type tansmission is the size of the intemediate value pats used in the tansmission. In ode to elax ou lowe bound, we assume that a seve can pefom patial computations on the files, i.e., if a seve wants to tansmit a faction of ft bits (with 0 f 1) of v q,n (ecall v q,n is made of T bits), then it only expends f of a computation. With this assumption, we can obseve the following popeties of ou cluste: Obs. 1. Each seve has N/ files stoed locally, and needs to eceive ( )N Q intemediate values though shuffling. Obs. 2. Fo a cluste with load edundancy, all feasible tansmission have l. This follows by noting that an l-type tansmission is assumed to satisfy l seves 1. Theefoe, each intemediate value involved in this tansmission is computed once at the tansmitte, and computed once at each of the othe l 1 seves which would utilize this intemediate value as side infomation to decode the tansmission. Since each file is epeated acoss seves, then l. Obs. 3. In the shuffling phase, each l-type tansmission and weight f T incus an added computation cost to the cluste equal to l 2 ft. To see this, note that the seve sending this tansmission makes lf T computations. Moeove, an l-type tansmission seves l seves, each of which would have to do (l 1)f T computations to acquie the needed side infomation. Theefoe we get lft + l(l 1)fT = l 2 ft. Let z l be the numbe of l-type tansmissions. Then, the communication load fo a shuffling scheme is lowe bounded by the solution of the following Linea Pogam (LP) L lb (C total ) = s.t. z l l z i 0, min {z 1,...,z } l z l ( ), i [1 : ], z l l 2 + C total whee: (i) the fist constaint is a necessay condition fo the shuffling phase to delive ( )QN intemediate values to 2 each seve in the cluste; (ii) the second condition is a necessay condition fo the total computation (local computations and shuffling computations) to not exceed C total. Note that the esult of the LP is a lowe bound to the communication load since the fist constaint is not sufficient to ensue that each seve eceives its needed intemediate values. Fig 3 compaes the communication-computation tade-off fo the afoementioned CDC-fit scheme with the lowe bound in (4). The two tade-offs ae close only towads high computation loads which allows the system to opeate with an close to 1 If it is only useful fo less than l seves then the tansmitte could have XOR-ed less intemediate values to geneate the tansmission. (4)

the natual of the cluste. Next, we popose a modification to the CDC scheme denoted as Split-CDC (S-CDC) that povide a communication-computation tade-off close to the tade-off suggested by the lowe bound in (4). Split-CDC (S-CDC). In ode to intoduce S-CDC, we make the following obsevations on the shuffling stategy in CDC. Obs. 1. The set VS i i descibed in (1) is of size VS i i = η 1 η 2. Obs. 2. Fo evey subset S of + 1 seves, the computations needed to satisfy all seves in S is (+1)η 1 η 2 and the numbe of packets communicated among them is +1 η 1η 2. Obs. 3. Fom (1), it is not had to see that fo any subset S S such that S > 1, VS i i VS i i S i,. The pevious obsevations suggest the following modification to the CDC scheme. Each subset S of size + 1 can be split into disjoint subsets of smalle size. Each smalle subset S can still be used to satisfy its membes with the set VS i i as pe Obsevation 3. Theefoe, by using subsets {S } of size diffeent than + 1, this would allow the scheme to exhibit diffeent levels of communications and computations pe S (based on the size of the splits), as evident fom Obsevation 2. The possible sizes of S ae [1 : ], which we efe to as the split size. Fo, define j = +1 +1 and = ( + 1) j( + 1) 1. Thus we can split set S into j disjoint sets S ( ) of size ( + 1) and one set S ( ) of size ( + 1). Fo each set in S ( ), the needed numbe of computations is ( + 1)η 1 η 2 and the needed numbe of communicated packets is +1 η 1 η 2. If S ( ) is not empty, then simila expession follow (except when S ( ) = 1, whee we need η 1 η 2 computations and η 1 η 2 packets exchanges to send the intemediate values though unicast tansmissions fom any seve in S ( ) ). Finally, since VS i i = η 1 η 2, fo evey subset S of size + 1, CDC would natually incu η 1 η 2 tansmission ounds, each deliveing exactly one intemediate value in VS i i fo all seves in S. Thus, ou obsevations suggest that CDC can opeate each of these tansmission ounds with a diffeent splitting size of S; thus the name Split-CDC (S-CDC). Fo a tansmission ound using split size, the total computations and communications pe subset S of size +1 is { j ( + 1) + ( + 1), 0, Comp( ) = j ( + 1) + 1, = 0, { Comm( j +1 ) = + +1, 0, j +1 + 1, = 0. z η 1η 2 S-CDC can now be fomally descibed. Let be the faction of the intemediate values in VS i i pe subset S that is deliveed using split size. Then, S-CDC woks as follows: z 1) Detemine the optimal values of η 1η 2 fo [1 : ] - this is done via solving the LP in (5). 2) Fo each S [1:] of size +1 and split size [1:]: Split set S into j disjoint sets S ( ) of size ( + 1) and one set S ( ) of size ( + 1). Use enough computations and communications pe each of the subsets S ( ) and S ( ) as pe the CDC scheme, to delive z intemediate values to all seves in S. The computations and communications needed to do so is equal to z Comp( ) and z Comm( ) espectively. What emains is to find the optimal values of z. We do so via solving the following LP, which minimizes the total communication load subject to a total computation constaint. L P (C total ) = min {z 1,...,z } s.t. ( + 1 ( ) + 1 z l Comm(l) z l = η 1 η 2, z i 0, i [1 : ], ) z l Comp(l) + C total. (5) Note that in (5), the vaiables z l ae allowed to take nonintege values which means that we ae allowing the seves to do patial computations of the intemediate values if that is what they will need to tansmit o decode. To estict patial computations, we can appoximate the solution of (5) to get a suboptimal intege-valued solution ẑl. Note that if an optimal solution of (5) is non-intege, then thee exists only two nonzeo elements of {zl }; we denote these two elements as z l 1 and z l2 whee l 1 < l 2. Then fo ou appoximate solution, we define ẑl 2 = zl 2 and ẑl 1 = η 1 η 2 zl 2. This gives us ẑ l Comm(l). a communication load ˆL P (C total ) = ( +1) Fig. 3 compaes the pefomance of S-CDC with the lowe bound in (4) fo N = 2520, Q = = 10 and = 5 when patial computations ae allowed. In this paticula setup, Fig. 3 shows that by peventing patial computations, we only incu a small faction of the communication load as an expense. REFERENCES [1] S. Li, M. A. Maddah-Ali, and A. S. Avestimeh, Fundamental tadeoff between computation and communication in distibuted computing, in IEEE Intenational Symposium on Infomation Theoy (ISIT), 2016, pp. 1814 1818. [2] J. Dean and S. Ghemawat, Mapeduce: simplified data pocessing on lage clustes, Communications of the ACM, vol. 51, no. 1, pp. 107 113, 2008. [3] A. C.-C. Yao, Some complexity questions elated to distibutive computing (peliminay epot), in Poceedings of the eleventh annual ACM symposium on Theoy of computing, 1979, pp. 209 213. [4] A. Olitsky and J. Roche, Coding fo computing, IEEE Tans. on Infomation Theoy, vol. 47, no. 3, pp. 903 917, 2001. [5] E. ushilevitz and N. Nisan, Communication complexity, 2006. [6]. Becke and U. Wille, Communication complexity of goup key distibution, in Poceedings of the 5th ACM confeence on Compute and communications secuity, 1998, pp. 1 6. [7] M. A. Maddah-Ali and U. Niesen, Fundamental limits of caching, IEEE Tans. on Infomation Theoy, vol. 60, no. 5, pp. 2856 2867, 2014. [8] N. aamchandani, U. Niesen, M. A. Maddah-Ali, and S. N. Diggavi, Hieachical coded caching, IEEE Tans. on Infomation Theoy, vol. 62, no. 6, pp. 3212 3229, 2016. [9] J. Hachem, N. aamchandani, and S. Diggavi, Content caching and delivey ove heteogeneous wieless netwoks, in IEEE Confeence on Compute Communications (INFOCOM), 2015, pp. 756 764. [10] S. Li, M. A. Maddah-Ali, and A. S. Avestimeh, Coded distibuted computing: Staggling seves and multistage dataflows, in 54th Annual Alleton Confeence on Communication, Contol, and Computing. IEEE, 2016, pp. 164 171. [11] Q. Yu, S. Li, M. A. Maddah-Ali, and A. S. Avestimeh, How to optimally allocate esouces fo coded distibuted computing? axiv pepint axiv:1702.07297, 2017.