A Stochastic Process on the Hypercube with Applications to Peer to Peer Networks

A Stochastic Process on the Hypercube with Applications to Peer to Peer Networs [Extene Abstract] Micah Aler Department of Computer Science, University of Massachusetts, Amherst, MA 0003 460, USA micah@cs.umass.eu Eran Halperin International Computer Science Institute an Computer Science Division University of California Bereley, CA 9470. eran@cs.bereley.eu Vijay V. Vazirani College of Computing, Georgia Institute of Technology, Atlanta, GA 3033 080. vazirani@cc.gatech.eu. Richar M. Karp International Computer Science Institute Bereley, CA 94704, USA, an Computer Science Division University of California Bereley, CA 9470,USA arp@cs.bereley.eu ABSTRACT Consier the following stochastic process execute on a graph G = (V, E) whose noes are initially uncovere. In each step, pic a noe at ranom an if it is uncovere, cover it. Otherwise, if it has an uncovere neighbor, cover a ranom uncovere neighbor. Else, o nothing. This can be viewe as a structure coupon collector process. We show that for a large family of graphs, O(n) steps suffice to cover all noes of the graph with high probability, where n is the number of vertices. Among these graphs are -regular graphs with = Ω(log n log log n), ranom -regular graphs with = Ω(log n) an the -imensional hypercube where n =. This process arises naturally in answering a question on loa balancing in peer-to-peer networs. We consier a istribute hash table in which eys are partitione across a set of processors, an we assume that the number of processors Supporte by NSF Research Infrastructure Awar EIA- 00809 an by NSF Faculty Early Career Development Awar CCR-033664 Supporte in part by NSF grants CCR-98095 an CCR- 0555 an DARPA cooperative agreement F3060-00-- 060. Permission to mae igital or har copies of all or part of this wor for personal or classroom use is grante without fee provie that copies are not mae or istribute for profit or commercial avantage an that copies bear this notice an the full citation on the first page. To copy otherwise, to republish, to post on servers or to reistribute to lists, requires prior specific permission an/or a fee. STOC 03, June 9, 003, San Diego, California, USA. Copyright 003 ACM 583 674 9/03/0006...$5.00. grows ynamically, starting with a single processor. If at some stage there are n processors, the number of queries require to fin a ey is log n+o(), the number of pointers maintaine by each processor is log n + O(), an moreover the worst ratio between the loas of processors is O(), with high probability. To the best of our nowlege, this is the first analysis of a istribute hash table that achieves asymptotically optimal loa balance, while still requiring only O(log n) pointers per processor an O(log n) queries for locating a ey; previous methos require Ω(log n) pointers per processor an Ω(log n) queries for locating a ey. Categories an Subject Descriptors F.3 [Mathematics an Computing]: Probability an Statistics Stochastic processes; D. [Data]: Data Structures Distribute ata structures General Terms Algorithms Keywors Peer to peer, coupon collector, hypercube, hash table, loa balancing. INTRODUCTION In the past few years, there has been a consierable amount of research on peer-to-peer applications. A central problem for such applications is eveloping a istribute protocol that fins a ata item store on one of a potentially large number of processors. We can thin of this as a istribute hash table: given a ey, we want to hash that ey to a value

that ientifies the processor that hols that ey. We also require an efficient metho for fining that processor. This hash table shoul function without centralize control an shoul also provie an efficient technique for ealing with the frequent an unpreictable arrival an eparture of the processors that store the table. Research that has aresse this problem inclues [3,,, 0, 4, 6, 8]. In this paper, we consier istribute hash tables of the following type. Each ey is hashe to a real number in the interval [0, ]. The tas of storing this interval is partitione across the processors. In particular, the processors are organize using a (not necessarily complete) binary tree, where each leaf of the tree correspons to a processor. Each leaf is also assigne a binary string escribing the path from the root to that leaf, with left branches represente by a 0 an right branches represente by a. Let S(i) be the string assigne to the ith leaf in the left to right orering of leaves. Let n be the number of processors, an let N(i) be the real number (in [0, )) obtaine by interpreting the string S(i) as a binary ecimal. The processor corresponing to leaf i < n is responsible for storing items that hash to the real interval [N(i), N(i + )]. In aition, the processor corresponing to leaf is also responsible for the real interval [0, N()]. The processor corresponing to leaf n is responsible for the real interval [N(n), ]. In orer to allow navigation through this ata structure, processors store pointers to each other. In particular, the processor at leaf j maintains a pointer to the processor at leaf i if S(i) is a prefix of a string obtaine from S(j) by flipping one bit in S(j) an appening it with zeros. In Figure we show an example of such a tree, illustrating the pointer set of one of the vertices. Such pointers can be thought of as a generalization of the hypercube. Using these pointers, if a search for an item starts at any processor, the hashe item can be foun using at most h queries to processors, where h is the height of the tree (our analysis shows that h = log n+o()). Also note that the number of pointers neee for each processor is at most h. These pointers are also use to hanle processor arrivals an epartures. An arrival is accomplishe by first fining an appropriately chosen leaf of the tree, an then performing a split operation on that leaf. A split of leaf i replaces i with an internal noe, as well as two chilren of that noe. The processor that previously resie at leaf i is assigne to one chil, an the new arrival is assigne to the other chil. Storage of the content hashe to the original region as well as the pointers hel by processors are ajuste appropriately. The process for choosing which leaf to split is central to this paper, an will be iscusse below. We further suggest a eparture strategy which seems to wor well in practice, although it is left open to obtain a rigorous analysis for that process. We shall refer to this istribute hash table as the hypercubic hash table. The hypercubic hash table is motivate by the hash table introuce in []. Instea of the hypercubic structure we consier, [] consiers a -imensional torus, where each processor is responsible for a subregion of the torus, an maintains pointers to processors that are responsible for neighboring regions. Arrivals are hanle by splitting an existing region in half along one imension. The hypercubic hash table we consier here can be viewe as the structure from [], where the imension is chosen to be so large that no imension is ever split more than once. There are a number of measures of the performance of a Figure : An example of a tree containing the strings 000, 00, 000, 00, 0, 00, 0,. Each of the hosts is locate in one of the leaves. The tree is virtual in the sense that its eges o not exist in the networ. The otte lines are the actual connections in the networ. In this case, the host corresponing to the leaf has pointers to the hosts locate in 00 an 000. These neighbors are etermine by flipping one of the bits in the string, an paing it with zeros at the en. istribute hash table. Of primary interest are the number of queries require to locate a requeste item, the number of pointers each processor must store, as well as how well the storage loa is balance between the processors. The istribute hash tables of [4], [], [0], an [3] all require O(log n) queries for location an O(log n) pointers per processor for a ata structure with n processors. The hash table of [] requires O(n / ) queries for location, an O() pointers per processor. The hash table of [8] requires O(log n) queries, O() pointers in expectation, an O(log n) pointers per processor with high probability. In this paper, we concentrate on the thir inication of performance: loa balancing. The analysis of this measure is typically more involve than that of the other two escribe above, an it has seen much less attention in the existing wor. Let V max(d) be the maximum fraction of the range of the hash function store on any processor with hash table D, an let V min(d) be the corresponing minimum fraction. We say that a istribute hash table D achieves loa balance α if V max(d)/v min(d) = α. For example, the hypercubic hash table we consier achieves loa balance if the unerlying binary tree is complete. Note that the fraction of the hash function range is an appropriate measure of loa balance, since we expect the number of hashe items to be much larger than the number of processors. The actual loa balance achieve by the hypercubic hash table will epen on the rule for choosing which noe of the tree to split on a processor arrival. The vanilla version of the [] scheme suggests a strategy for choosing a region to split that, in our framewor, is equivalent to choosing a

value in the interval [0, ] uniformly at ranom, an then splitting the leaf of the binary tree that is responsible for the region containing that value. Such a rule can be easily implemente in a istribute fashion, but it is not ifficult to show that the resulting istribute hash table on n noes achieves loa balance Ω(log n). The istribute hash tables of [4], [], an [0] use similar splitting rules. The scheme from [3] also achieves loa balance Ω(log n). They escribe how to improve this to loa balance O(), but this requires the number of pointers store at each processor to be increase to Θ(log n). In this paper, we stuy a secon technique suggeste in [] for choosing which noe to split. A leaf noe is chosen ranomly as before. However, instea of splitting that leaf noe, we consier the set consisting of that noe as well as all the leaves that the processor assigne to the chosen leaf has pointers to. The leaf noe in this set with minimum istance from the root is split. If there is more than one such noe, the originally chosen noe is given preference, an, if that noe oes not have the minimum istance, one of the noes that o is chosen uniformly at ranom. This splitting rule is shown in [] to perform quite well through simulations, but no analytical evience for this has existe prior to the wor escribe here. We emonstrate that the resulting hypercubic hash table achieves constant loa balance. In particular, we emonstrate that there are constants c an c such that after n split operations have been performe, with high probability the maximum height of any noe in the tree is log n+c, an the minimum height is log n c. To the best of our nowlege, this is the first analysis of a istribute hash table that achieves asymptotically optimal loa balance, while still only requiring O(log n) queries for location, an O(log n) pointers per processor. Furthermore, the rule we consier for performing a split oes not affect the number of queries require for the split by more than a constant factor. In particular, the number of queries require to fin a ranomly chosen initial noe is O(log n); the improve rule for choosing which noe to split only requires querying the O(log n) neighbors of this noe. In orer to prove the above bouns, we introuce the following ranom process on the -imensional hypercube. In this process, each noe coul be either covere or uncovere. The initial state is when all noes are uncovere. At each step of the process we pic a ranom noe v of the hypercube. If v is uncovere, we cover it. If v is covere an all its neighbors are also covere, we o nothing. Otherwise, we ranomly pic one of its uncovere neighbors an cover it. We call this process Process. We show that with high probability, Process covers all vertices of the hypercube after O(n) steps, an each of the first Ω(n) steps covers an uncovere noe, where n =. We then use these bouns to show that the tree representing the hypercubic hash table is balance, which provies the bouns claime above. Process can be efine on any graph G = (V, E). If the graph is an inepenent set, this is exactly the coupon collector process an therefore, with high probability, the process covers all noes after O(n log n) steps. We show that our proof techniques can be extene to other graphs: we prove that for every -regular graph, Process covers all We point out that while [8] oes not aress the issue of loa balance at all, their istribute hash table coul also be use to achieve a similar result by using a technique similar to that use to improve the loa balance of [3]. log n log vertices after O(n + n ) steps. We also show that for ranom -regular graphs the stronger boun of O(n + n log n ) hols. These extensions may be useful in the esign of ifferent algorithms an ata structures. We note that inepenently, Alon [] prove that for any (n,, λ)-graph G, O(n+n( λ ) log n) steps are sufficient. Alon further shows that for any -regular graph on n vertices, the expecte number of steps that Process uses until it covers the whole graph is at least n n + n log e(n/). Alon s proof also gives a boun of n+(+o())n log e n/ steps for ranom regular graphs. This paper is organize as follows: in the remainer of this section, we provie more etails concerning other propose istribute hash tables, as well as other relate wor. In Section, we reuce the tas of analyzing the performance of the hypercubic hash table to the problem of analyzing a simple stochastic process on the hypercube. In Section 3, we analyze this hypercubic process. In Section 4, we escribe extensions of our techniques to other graphs. In Section 5 we provie an alternative proof of the boun on the cover time of the ranom process on the hypercube. In Section 6 we escribe the results of some simulations.. Previous Wor In the istribute hash table of [0], each participating processor i is assigne a ranom -bit string S(i), where is large enough that it is unliely for two processors to have the same string. Each hashe ey is store on the processor i such that S(i) is closest to its hash value in lexicographic orering. Note that choosing a ranom -bit string for an arrival is similar to the version of the hypercubic hash table where a new arrival is hanle by splitting a ranomly chosen noe (without regar to the neighbors of that noe). The main ifference is that the entire string S(i) stays with the processor in the technique of [0]; with the technique we consier here only the portion of the ranomly chosen real number that escribes the first currently unsplit noe of the binary tree is ever use. The rule use in [0] for choosing which pointers between processors to maintain can also be thought of as a generalization of the hypercube. This rule taes into account locality in the unerlying networ, an thus has the avantage that the real networ length of a path in the ata structure will typically be consierably shorter than if this measure is not taen into account. The istribute hash table of [0] is static (i.e., it oes not allow for the arrival an eparture of processors), but techniques for maing it ynamic have been escribe in [], [4], an [6]. In the istribute hash table of [3], ata items an processors are hashe to integers chosen uniformly at ranom from the range [0, ]. Each ata item is store on the processor with the next largest hash value (mo ) to the ata item s hashe value. In orer to locate ata items, each processor stores log n pointers: for each integer j, 0 j log n, the processor with the ith largest hash value stores a pointer to the (i+ j mo )th largest hash value. It is not ifficult to show that this ata structure achieves loa balance Θ(log n) with high probability. This can be improve to O() by simulating log n virtual processors on each actual processor. However, this increases the G is an (n,, λ)-graph if G is a regular graph on n vertices, such that the absolute value of every eigenvalue of the ajacency matrix of G, besies the largest one, is at most λ.

number of pointers that each real processor must store to Θ(log n). The iea of using the best out of multiple possibilities to improve loa balance has been stuie extensively in the analysis of balls into bins games (see [7] an [3] for early wor, an [9] for a recent survey). The scenario consiere there, as well as the notion of loa balance, is significantly ifferent from this paper. However, the observation that more choices is helpful is similar.. THE STOCHASTIC PROCESS In orer to analyze the hypercubic hash table escribe in the previous section, we consier Process which was efine in the introuction. Let n = be the number of noes in the hypercube. We show the following: Theorem.. Process covers all the noes of the hypercube in O(n) steps with high probability; the probability of error is at most inverse polynomial in n. Theorem.. There exists a constant c, such that with high probability, each of the first cn steps of Process cover an uncovere vertex; the probability of error is at most inverse polynomial in n. Observe that a boun of O(n log n) in Theorem. follows trivially from the coupon collector argument, since the above process ominates the coupon collector process. A boun of O(n) steps is easy to show for the following moification of the coupon collector process: pic log n items at ranom an if any of them is not covere, cover a ranom uncovere item. In our case, the ranomization is in the choice of the initial noe the = log n noes that are examine subsequently are completely etermine by this choice. On the other han, each noe has a funnel of size, its neighbors, that helps it get covere we will use this critically in the proof. The subsequent sections of this paper concern the analysis of this stochastic process. In the remainer of this section, we emonstrate the implications of Theorem. to the hypercubic hash table. We state these implications as bouns on the minimum an maximum height of any leaf in the unerlying binary tree. From these bouns, it follows easily that after n split operations, the hypercubic hash table achieves asymptotically optimal loa balance, while still only requiring O(log n) queries for location, an O(log n) pointers per processor. Corollary.3. There is a constant c such that after n split operations have been performe on the hypercubic hash table, the probability that the minimum height of any noe in the tree is less than log n c is at most inverse polynomial in n. Proof. At any point uring the n split operations, consier some level l of the tree such that every noe at level l has been split, but there is some noe at level l that has not been split. It is easy to see that the process of inserting a new processor to the hash table is equivalent to the hypercube process in the following sense. The vertices of the hypercube are the noes in level l. A noe is covere if it has alreay been split, an is uncovere otherwise. From Theorem., we now that there is a constant α such that after α l more split operations have been performe, the probability that there are any unsplit noes on level l is inverse polynomial in l. Let c = log(4α). For each level i of the tree, i log n log log n c, we allocate α i log n split operations to level i. These splits are further subivie into log n phases of α i split operations each. Given that all noes at level i have been split before a given phase for level i, with probability at least / there are no unsplit noes at level i after that phase. Thus, if all noes at level i are split before the log n phases allocate to level i, the probability that i has any unsplit noes after the split operations allocate to it is at most inverse polynomial in n. For each level i of the tree, log n log log n c < i log n c, we allocate α i split operations to level i. This is sufficient that the probability that any such i has any unsplit noes is also at most inverse polynomial in n. Note that we have allocate a total of at most n split operations. Since we are consiering a total of only log n c levels of the tree, the probability that any of the first log n c levels of the tree fails to have every noe split is still inverse polynomial in n. Corollary.4. There is a constant c such that after n split operations have been performe on the hypercubic hash table, the probability that the maximum height of any noe in the unerlying binary tree is larger than log n + c is at most inverse polynomial in n. Proof. Consier the level l = log n + c for c large enough. We can view the process of splitting vertices of level l as ominate by the hypercube process on the hypercube with l vertices. By Theorem., the first c l insertions o not split any vertex in level l + with high probability, where the probability for error is inverse polynomial in l, that is, inverse polynomial in n. By setting c = log (/c), we get that with high probability we o not split any vertex in level l + in the first n insertions. Thus, the corollary follows. 3. ANALYSIS OF THE STOCHASTIC PROCESS In this section we prove Theorem.. 3. An easier boun of O(n log log n) A boun of O(nlog log n) is easier to establish an is quite instructive, so we will present it first. We first nee some notations an efinitions. In all the statements below, high probability will mean one minus inverse polynomial in n probability; the exact polynomial epens on the multiplier of n chosen in the number of steps execute. An uncovere noe will also be calle free. For a vertex v, let N F(v) an N C(u) be the set of free an covere neighbors of v respectively. For a vertex v an a set A, let v(a) be the number of neighbors of v in A. We will use the following Chernoff-lie upper boun for large eviations (the proof can be foun, e.g., in []). Lemma 3.. Let Z be a ranom variable with the binomial istribution, Z B(n, p). For every a > 0, prob(z < np a) < e a /(np). It is easy to see that O(n) steps suffice to ensure that with high probability each noe has at most / uncovere

neighbors; in fact, this is true for the simpler coupon collector process as well. We will call this phase 0, an will execute log more phases of O(n) steps each, numbere,..., log. We say that phase i is successful if at the en of phase i, N F (v) for every vertex v. The claime i+ boun follows from: Lemma 3.. With high probability, for i =,..., log, phase i is successful. Proof. Assume that at the beginning of phase i, each noe has at most uncovere neighbors. Consier a noe i v such that N F(v). We will show that at the en of i+ phase i, with high probability, v has at most uncovere i+ neighbors, hence proving the lemma. Let L enote the free neighbors of v, an L the covere neighbors of L, though not incluing v. We now consier a single step, an we show that as long as L, the probability to cover a vertex in L i+ is large. Clearly, in orer to cover such a vertex, the process has to choose a vertex u of L, an then to choose one of its u(l ) neighbors in L. Recall that the number of free neighbors of u is boune by. We thus get that this i probability is at least u L n u(l)i = i n v L N C(v) i n L 4n. Let c be a large constant. Consier a sequence of cn steps of the algorithm, an let X be a ranom variable, where X B(cn, ). Clearly, as long as L, the number of vertices of L covere by the process is ominating 4n i+ X, that is, the probability that after cn steps we still have L is at most the probability that X <. By i+ i+ Lemma 3. we get prob(x < i+ ) prob(x < c 4 4 (c /i )) e (c /i ) /8c < e, where the last inequality hols for c large enough. We now use the union boun, an since the total number of vertices is, an the total number of phases is log, the probability that phase i is not successful for some i is at most log e which is inverse polynomial in n. 3. Exploiting ifferential progress via convexity At any step uring Process, ifferent noes will be at ifferent stages of progress, where by progress of a noe we mean the number of its covere neighbors. A shortcoming of the argument given above is that it has no way of taing avantage of noes that are further along it only utilizes the worst case progress mae in previous phases to boun the number of steps neee in future phases. From the proof of Lemma 3., it is easy to see that if each noe of L ha < / i+j uncovere neighbors, then O(n/ j ) steps woul guarantee that with high probability the number of uncovere neighbors of v are halve. Lemma 3.3 below shows that this woul be the case even if on average noes in L ha fewer uncovere neighbors, thereby enabling us to argue about the progress mae by aggregates of noes. Conuct a breath first search starting with noe v, an let B i(v) be the noes at level i, i =,...,. W.l.o.g. assume that v is labele with the -bit string (0,..., 0). Then, B i(v) contains all noes having exactly i s in their labels. Therefore, the number of neighbors of a noe u B i(v) in the proceeing set B i (v) is exactly i those labels obtaine by flipping one of the s in u s label to a zero. Define L i(v) B i(v), for i 5 as follows: L (v) is the set of free neighbors of v, L (v) the set of covere neighbors of L (v), L 3(v) the set of free neighbors of L (v), L 4(v) is the set of covere neighbors of L 3(v), an L 5(v) the set of free neighbors of L 4(v). Furthermore, efine N i(v) B i(v), for i 5 as follows: N (v) is the set of neighbors of L (v), N 3(v) is the set of free neighbors of N (v), N 4(v) is the set of covere neighbors of N 3(v) an N 5(v) is the set of free neighbors of N 4(v). Note that N i(v) L i(v) for i =,3, 4. Lemma 3.3. Assume that at least half the neighbors of each noe are covere. Let v be a vertex an let i, j be such that i log an i + j log. Suppose an i+ NF(v) < i L 3(v) 3 i+j. Then, after O( n ) steps, the number of free neighbors of v j must be smaller than with high probability. i+ Proof. Recall that for every l, a noe in L l+ (v) can have at most È l + neighbors in L l (v). By this property, we have that u L (v) NF(u) 3 L3(v) + L(v), since we count each vertex of L 3(v) at most three times. Since i+j log an L (v), we get that L (v) i 3, i+j È an thus, u L (v) NF(u) 3. Furthermore, since i+j each vertex has at least / covere neighbors, an each vertex of L (v) has at most two neighbors in L (v) we have that L (v) L (v). 4 We now lower boun the i+3 probability of covering a vertex of L (v) as long as L (v). This probability is at least i+ u L (v) n N F (u) È n L(v) u L (v) NF(u) i+j L (v) 3 n j 8 n, where the first inequality follows from the convexity of the function f(x) = /x. The proof now follows by the same arguments given in Lemma 3..

3.3 Establishing a linear boun As before, we will have log + phases numbere 0,,..., log, an will prove that at the en of the i th phase, the number of free neighbors of each noe is at most, with i+ high probability. Phase 0 is as before, an taes O(n) steps. In each of the remaining phases, we will execute a number of iterations whose purpose is to thin own L 3(v), for each noe v. This time aroun, L 4(v) acts as a funnel, an since we are ealing with an aggregate of noes, the funnel is much larger an so the number of steps require is fewer than in Lemma 3.. Next, we will thin own L (v). By Lemma 3.3, this also taes fewer steps than before. The number of iterations execute in phase i is t i = min{i,log i}. Each iteration within phase i consists of O(n/ i ) steps. Lemma 3.4. By the en of iteration j within phase i, with high probability, for each noe v with N F (v) / i+, L 3(v) 3 i+j. Proof. The proof is similar to that of Lemma 3., with L 4(v) acting as a funnel to cover noes in L 3(v). We will actually prover a stronger claim: By the en of iteration j within phase i, with high probability, for each noe v with N F (v) / i+, N 3(v) 3 i+j. Clearly, since L 3(v) N 3(v), the lemma follows. Assume that N 3(v) > 3 in the beginning of iteration j (otherwise we are one with i+j N 3(v) for iteration j). Recall that for every u N 4(v), u(n 3) 4 an that for every u N 3(v), N C(u) /, an on the other han u(n ) 3. Thus, N 4(v) N 3(v) (/8 3) N 3(v) /6 for sufficiently large. As long as N 3(v) >, the probability of covering a vertex in N 3(v) is at i+j least u N 4 (v) n N i i N4(v) F(u) n 8n N3(v) > 3 i+j+4 n. We note that the quantity i + j is not increasing with i since i + j log. Assume that iteration j consists of cn/ i steps for c large enough. If N 3(v) > 3 3 i+j, the number of vertices of N 3(v) covere in iteration j is a ranom variable ominating the binomial ranom variable X B( cn, 3 ). By Lemma 3., i i+j+4 n prob(x < N3(v) ) prob(x < 3 ) i+j = prob(x < c3 3 i+j+4 ( c i+j 6 )) < e Ω(3 / i+j) e Ω(), where the last two inequalities follow from the fact that c is large enough, an that i + j log. We now use the union boun over all vertices, an all O(log ) iterations, an we get that for every v, by the en of iteration j, N 3(v) 3 i+j with high probability. After the t i iterations in phase i, we execute O( n ) more t i steps. By Lemma 3.3, these suffice to ensure with high probability that N F (v) for all noes v. Finally, since i+ log i= log t i + i i= log t i i= log i + i i= i = O(), the total number of steps is O(n), hence proving Theorem.. 3.4 Establishing the Lower Boun In this section we prove Theorem.. We shall show that there is a constant c such that with high probability each of the first cn steps covers an uncovere vertex. We will use the following lemma (its proof can be foun, e.g., in []): Lemma 3.5. Let Z be a ranom variable with a binomial istribution Z B(n, p). For every λ >, prob(z > λnp) < (e λ λ λ ) np. In orer to prove Theorem., we introuce the following efinitions. As before, Ë for every v V we efine L (v) = N F(v) an L (v) = u L (v) NC(u) {v}. We say that step i is successful if after step i, for every vertex v, L (v) /. The theorem follows from the following claim: Claim 3.6. There exists a universal constant c, such that for t c 3, the first tn steps are successful with high probability. 3 Proof. The proof is by inuction on t. For t a positive integer, let A t be the event that the first tn steps are successful. For the base case, we show that A 3 occurs with high probability. In orer to cover a vertex of L (v), we have to either choose a vertex u of L (v), or choose vertex v. Since L (v) {v} L (v) we get that the probability of covering a vertex in L (v) is at most. Let X be the ranom variable that enotes the number L (v) after the n first n/ 3 steps. Let Y be a ranom variable with the binomial istribution, Y B( n, ). We get that Y ominates 3 n X, an by Lemma 3.5, prob(x > /4) prob(y > /4) = e Ω( ln ). () Therefore, by a union boun, with high probability, for every vertex v, L (v) > /. We now prove the inuctive step: we assume that A t occurs with high probability, an show that if t 3, then A t+ also occurs with high probability. Let A t(v) be the event that after the first tn steps, L (v) 3/4. We first show that if A 3 t occurs, then we can actually mae the stronger claim that for every vertex v, A t(v) occurs with high probability. We then use a similar argument as above to show that, with high probability, in the last n/ 3 steps, for any v, at most /4 aitional vertices in L (v) get covere. To show that A t(v) occurs with high probability, note that Pr[A t(v)] Pr[A t(v) A t]+pr[a t], an thus, by the inuctive hypothesis, we only nee to show that the probability of A t(v) A t is inverse polynomial in n. Thus, we focus on

the first tn/ 3 steps, an only consier the case where for every u L (v), L (u) /. Since each vertex in L (v) has at most two neighbors in L (u), in each of the first tn/ 3 steps, the probability of covering a vertex of L (v) is at most 4 L(v) 4. Let X n n be the ranom variable that enotes the number L (v) after the first tn/ 3 steps. Let Y be a ranom variable with the binomial istribution, Y B( tn, 4 ). We get that 3 Y n ominates X, an by Lemma 3.5, prob(x > /4) prob(y > /4) = e Ω( ln(3 /t)), as long as t < c 3 for sufficiently small (but constant) c. If c is small enough, then we get by a union boun that for every vertex v, X (v) /4. We are now left with the last n/ 3 steps. We efine X = L (v) X to be the number of vertices remove from L (v) in the last n/ 3 steps, an efine Y as before, i.e. Y B( n, ). Then the situation is exactly as in the 3 n base case, an therefore Equation () hols, an by a union boun we can assume that for every vertex v, X < /4, an thus, L (v) > /. This completes the proof. 4. EXTENSIONS OF THE PROCESS In this section we introuce an analyze a variant of Process, an give some upper bouns on the covering time of the processes in ifferent graphs. We first show that our proof hols for every -regular graph. We have the following theorem: Theorem 4.. Let G = (V, E) be a -regular graph with log n log n noes. With high probability, after O(n( + )) steps of Process, all noes are covere. Proof. We first note, that in the proof of Lemma 3. we i not use the fact that graph is the hypercube, but simply the fact that the graph is -regular for = log n. Thus, we only have to show that the same hols for -regular graphs for log n. The proof has the same flavor as Lemma 3.. We have log phases. After phase i, we will have at most uncovere neighbors for every vertex v. Since the coupon i collector process is ominate by Process, it is easy to see that by using Θ(n+nlog n log ) steps for phase 0, we have that each vertex has at least / covere neighbors with high probability. As long as L / i+, the probability of covering a vertex in L is at least u L n u(l)i = i n v L N C(v) i n L 4n. Consier a sequence of cn steps of the algorithm where c will be etermine later, an let X be a ranom variable, where X B(cn, ). As long as L 4n /i+, the number of vertices of L covere by the process stochastically ominates X, that is, the probability that after cn steps we still have L / i+ is at most the probability that X < / i+. By Lemma 3. we get prob(x < for c > i. i+ ) prob(x < c 4 4 (c /i )) e Θ(c), If / i > 0log n, we tae c = Θ(/ i ), an otherwise we tae c = Θ( log n ). We thus get that prob(x < ) is inverse polynomial in n, an by applying a union boun i+ over all log phases an all n vertices, we get that with high probability after phase i each vertex has at most / i+ uncovere neighbors. The total number of steps we use for all the phases is at most O(n + n log n log ). As a corollary of Theorem 4. we get that for every - regular graph with > log n log log n, O(n) steps of Process are sufficient to cover all noes with high probability. Moify Process so that after picing a ranom noe, v, it only covers a ranom uncovere neighbor of v. If all neighbors of v are covere, it oes nothing. Call this Process. In this process, a noe has a funnel of size all the way until it is covere. Hence, phase 0 is unnecessary, an the rest of the proof remains essentially unchange to show the following theorem: Theorem 4.. Using Process, O(n) steps suffice to cover all noes of the hypercube with high probability. Furthermore, O(n+n ) steps are sufficient to cover all noes log n log of any -regular graph. Next, consier a family of graphs having maximum egree, minimum egree Ω() an the following aitional property: For every v V, if we conuct Breath first search from v, an let B i for i =,,... enote the levels. Assume that for i =,3, 4 an for every u B i, the egree of u into B i is boune by a constant. Then, it is easy to see that the proof of Theorem. goes through, an we get that O n( log n + ) steps suffice to cover all noes of the graph with high probability using either Process or Process. Since ranom -regular graphs satisfy these conitions, we get Theorem 4.3. For a ranom -regular graph, the number of steps suffice to cover all noes of the graph with high probability using either Process or Process is at most O n( log n + ). Consier now the infinite process inferre by the hypercubic hash table, where there is an infinite sequence of arrivals. The following theorem shows that the probability that there never is a ifference of more than between the shallowest an eepest levels of the tree is at least e Ω(). This emonstrates that this process is self correcting. Theorem 4.4. The probability that there is never a ifference of more than between the shallowest an eepest levels of the tree in the hypercubic hash table over an infinite sequence of arrivals is at least e Ω() m Proof. Let A m enote the event that until step m the ifference between the shallowest an eepest levels is at least. We first note that one can moify the constants c, c from Corollaries.3 an.4 such that the events escribe in the lemmas happen with probability at most where m is the number of steps. Assume that is much larger than the constants c, c. Consier the first N = c steps. By Corollary.4, with probability at least /N the eepest level until that point is at most. Therefore, prob(a... A N). On N

the other han, by Corollaries.3 an.4, prob(a m). m Therefore, prob( m N. m>n A m) m>n prob(a m) m>n Since N = Θ( ) the theorem follows. 5. AN ALTERNATIVE PROOF TO THEOREM. We now introuce a ifferent proof of Theorem. using coupling. This proof provies a tighter analysis than the one given in Section 3, but it is not clear how to exten this proof to other cases such as -regular graphs. We will prove the theorem for Process, but we note that the proof is almost ientical for Process. Let T n be the cover time of Process, that is, the number of steps until every vertex is covere. We shall investigate the probability istribution of T n. We prove the Theorem by comparing T n with the cover times of two other stochastic processes: a ranom probing process with cover time P n an a rejection process with cover time R n. By comparing the ranom probing process with the stanar coupon collector process we show that P n = O(R n) with high probability. We then show that the rejection process can be viewe as the ranom probing process slowe own by a constant factor. Finally, we complete the proof with a coupling argument showing that T n is stochastically smaller than R n. The proof is esigne for clarity rather than for optimizing the constant factor in the time boun for T n. The ranom probing process is as follows. Initially all vertices are uncovere. At each step a multiset S of vertices is chosen uniformly at ranom with replacement. If S contains an uncovere vertex then an occurrence of an uncovere vertex in S is chosen uniformly at ranom an the vertex becomes covere; otherwise no vertex becomes covere at this step. Define the cover time P n of this process as the number of steps until every vertex is covere. We shall relate P n to the stanar coupon collector process, in which elements are rawn with replacement inepenently an uniformly at ranom from an n-set until all elements have been rawn. Let C n be the number of raws in the coupon collector process. Lemma 5.. P n is stochastically smaller than Cn+n( ) Proof. Consier the following metho of implementing the ranom probing process: at each step the vertices in S are inspecte sequentially in a ranom orer until an uncovere vertex is foun or the set S has been exhauste. Once an uncovere vertex has been foun the remaining vertices in S are iscare without being inspecte. The number of inspecte vertices is istribute as C n, an the number of iscare vertices is always less than or equal to n( ). But P n is equal to the number of inspecte vertices plus the number of iscare vertices. Lemma 5.. P(n) < n( log e + ) whp Proof. The probability that C(n) > n(ln n+) is boune above by n( /n) n(ln n+) which is exponentially small in. Combining this fact with the above lemma an the fact that = ln n the result follows. log e For any vertex v, let N(v) be the set of neighbors of v in the hypercube. Consier a state S (i.e., a esignation of each vertex of the hypercube as covere or uncovere). Let a(u, S) enote the number of uncovere neighbors of u an let a (u, S) enote the number of uncovere vertices at istance from u. Define Cov(v, S), the covering probability of an uncovere vertex v, as the probability that v will get covere at the next step of the covering process, given that the È u N(v) present state is S. Then Cov(v, S) = /a(u, S). n We now give the efinitions an lemmas require for efining an analyzing the rejection process. Let p be a rational number i/n between 0 an. A ranom p-state is a state rawn uniformly at ranom from the set of states in which there are exactly pn uncovere vertices. Lemma 5.3. In a ranom p-state S, the probability is less than n that there exists an uncovere vertex u such that Cov(u, S) < ( p) 6.4pn. Proof. In a ranom p-state S the ranom variable a (u, S) has a hypergeometric istribution with mean p. Using Chvatal s boun on the tail of the hypergeometric istribution [5] the probability that a (u, S) > s p, where s >, is less than (e/s) s( )p. Let r be the value of s where this boun is equal to 6. Then, by a union boun, with probability 4, a (u, S) r p for every vertex u. Let v be an uncovere vertex in state S. If a (v, S) rp then È u N(v) a(u, S) = + a(u, S) + rp an, by the convexity of the function /x it follows that Cov(v, S) n(+rp( )). Therefore, with probability ( 4 ), this inequality hols for every uncovere vertex in S. We now examine the ratio +rp( ). We shall show that +p( ) this ratio is less than 3.. Since the ratio is less than r, we may assume that r > 3.. Since (e/r)) r( )p = 6, (r/e) r ( )p = 6. Let a > 0 be such that r/e = 6 a. Then ae6 a ( )p =. It follows that p( ) =, rp( ) = ae6 a /a an +rp( ) = +/a. It is easy to show that this +p( ) + ae6 a function is boune over the positive reals, an numerical computation shows that it is less than equal to 3. for all positive a. Next we observe that ( ( p) )(+p( )). Putting p the inequalities together, the lemma follows. The rejection process is esigne so that, at any stage where the number of uncovere vertices is pn, all p-states are equally liely. We shall show by inuction over the steps of the process that this property hols (with high probability) an that, when the process is in a p-state S, every uncovere vertex u satisfies Cov(u, S) < ( p). The process operates as follows. Initially all vertices of the -imensional unit 6.4pn hypercube are uncovere. At each step, let S be the state, let i be the number of uncovere vertices, an let p = i/n. A vertex v is chosen at ranom. If v has at least one uncovere neighbor then an uncovere neighbor u of v is chosen uniformly at ranom. By inuction hypothesis, assume that Cov(u, S) ( p). With probability ( p) the 6.4pn 6.4pnCov(u,S) step is accepte an u becomes covere. With the complementary probability the step is rejecte. Each uncovere vertex has probability ( p) of becoming covere at this 6.4pn step.

Since, at each step, all uncovere vertices are equally liely to become covere, it follows that, any stage where pn vertices are uncovere, the state is a ranom p-state, as was require for the inuction in the above analysis. Next we compare the rejection process with the ranom probing process an with the original covering process. In a p-state the ranom probing process has probability ( p) of covering a vertex, whereas the rejection process has probability ( ( 6.4 p) ) of covering a vertex. Thus the progress of the rejection process is stochastically the same as that of a process which, at each step, tosses a coin with probability of heas /6.4 an, if heas comes up, executes a step of the ranom probing process. It follows that the number of steps of the rejection process is less than 6.5n( + ) log e with high probability. Finally, we observe that the number of steps in the original covering process is stochastically smaller than the number of steps in the rejection process. To see this we use a coupling argument. Assume that, at each step, the two processes select the same ranom vertex, an inspect its neighbors in the same orer to select an uncovere vertex if one exists. Then, step by step, the set of vertices covere by the rejection process is a subset of the set of vertices covere by the covering process. Summing up, we have: Theorem 5.4. T n < 6.5n( log e +) with high probability. 6. EXPERIMENTAL RESULTS AND A DELETION PROCESS We ran some simulations to stuy the performance of Processes an in practice. As seen in Figure, in practice the number of steps require to cover all noes of the hypercube is boune by n. Furthermore, it is not clear cut, but it seems that the constant in Process is slightly better than the constant in Process. Next, we efine a leaf eletion process that is in the spirit of our leaf aition process: suppose leaf i nees to be elete. Consier the leaves that i points to an among the ones at the eepest level pic a ranom one, say j. It is easy to see that the sibling of j will also be a leaf, say. Now, the processor at j taes over i s job (the processor at i is relieve), an the processor at taes over j an s job an sits at the parent of j an. We o not have a goo analysis of this eletion process at present, but report the following experiments which suggest that it shoul also perform well. The first set of experiments start with complete binary trees of epth 0, an at each step performing a elete at a leaf which is chosen uniformly from the set of leaves. It was observe that the ifference in epth of the eepest an shallowest leaves was boune by 4 throughout the process. The next set of experiments involve starting with a complete binary tree of epth 0, an aing 0,000 more noes as follows: pic a leaf, an with probability 0.5 eclare it a permanent leaf, an with probability 0.5 a two new chilren to it. In the resulting tree, the ifference between epths of eepest an shallowest leaves was 0. Next, we performe ranom eletes on this tree. After 8,000 steps, the ifference went own to 0. We observe that the ifference was almost monotonically ecreasing, in the sense that it increase by only a few times (thus, it woul go from Figure : The istribution of the constants, such that Processes an respectively cover all noes after n an n steps respectively. We preforme 00 experiments on the 5-imensional hypercube. The x-axis is the constants ( or ), an the y-axis refers to how many experiment each constant appeare in. Clearly, both constants are always smaller than, an Process is more concentrate than Process, an is usually faster. 4 to 5, before going to 3, but not from 4 to 5 an then to 6, before going to 3). 7. REFERENCES [] N. Alon. personal communication, 00. [] N. Alon an J. H. Spencer. The Probabilistic Metho. John Wiley an Sons, Inc., 000. [3] Y. Azar, A. Broer, A. Karlin, an E. Upfal. Balance allocations. In Proceeings of the 6th ACM Symposium on Theory of Computing, pages 593 60, 994. [4] A. D. Joseph B. Y. Zhao, J. Kubiatowicz. Tapestry: An infrastructure for fault-tolerant wie-area location an routing. Technical Report UCB/CSD-0-4, UC Bereley, 00. [5] V. Chvatal. The tail of the hypergeometric istribution. Discrete Mathematics, 5:85 87, 979. [6] K. Hilrum, J. Kubiatowicz, S. Rao, an B. Zhao. Distribute object location in a ynamic networ. In Proceeings of ACM Symposium on Parallel Algorithms an Architectures, 00. [7] R. Karp, M. Luby, an F. Meyer auf er Heie. Efficient pram simulation on a istribute memory machine. In Proceeings of the 4th ACM Symposium on Theory of Computing, 99. [8] D. Malhi, M. Naor, an D. Ratajcza. Viceroy: A scalable an ynamic emulation of the butterfly. In Proceeings of the st ACM Symposium on Principles of Distribute Computing, 00. [9] M. Mitzenmacher, A. Richa, an R. Sitaraman. The power of two ranom choices: A survey of the

techniques an results. In P. Paralos, S. Rajasearan, an J. Rolim, eitors, Hanboo of Ranomize Computing. Kluwer Press, to appear. [0] C. Greg Plaxton, R. Rajaraman, an A. W. Richa. Accessing nearby copies of replicate objects in a istribute environment. In Proceeings of ACM Symposium on Parallel Algorithms an Architectures, June 997. [] S. Ratnasamy, P. Francis, M. Hanley, R. Karp, an S. Shener. A scalable content-aressable networ. In Proceeings of the 00 ACM SIGCOMM Conference, 00. [] A. Rowstron an P. Druschel. Pastry: Scalable, istribute object location an routing for large-scale peer-to-peer systems. In Proceeings of IFIP/ACM International Conference on Distribute Systems Platforms (Mileware), pages 39 350, November 00. [3] I. Stoica, R. Morris, D. Karger, M. F. Kaashoe, an H. Balarishnan. Chor: A scalable peer-to-peer looup service for internet applications. In Proceeings of the 00 ACM SIGCOMM Conference, 00.