Effect of Replica Placement on the Reliability of Large-Scale Data Storage Systems

Transcription

1 Effect of Replica Placement on the Reliaility of Large-Scale Data Storage Systems Vinodh Venkatesan, Ilias Iliadis, Xiao-Yu Hu, Roert Haas IBM Research - Zurich {ven, ili, xhu, rha}@zurich.im.com Christina Fragouli École Polytechnique Fédérale de Lausanne christina.fragouli@epfl.ch Astract Replication is a idely used method to protect largescale data storage systems from data loss hen storage nodes fail. It is ell knon that the placement of replicas of the different data locks across the nodes affects the time to reuild. Several systems descried in the literature are designed ased on the premise that minimizing the reuild times maximizes the system reliaility. Our results hoever indicate that the reliaility is essentially unaffected y the replica placement scheme. We sho that, for a replication factor of to, all possile placement schemes have mean times to data loss MTTDLs ithin a factor of to for practical values of the failure rate, storage capacity, and reuild andidth of a storage node. The theoretical results are confirmed y means of event-driven simulation. For higher replication factors, an analytical derivation of MTTDL ecomes intractale for a general placement scheme. We therefore use one of the alternate measures of reliaility that have een proposed in the literature, namely, the proaility of data loss during reuild in the critical mode of the system. Whereas for a replication factor of to this measure can e directly translated into MTTDL, it is only speculative of the MTTDL ehavior for higher replication factors. This measure of reliaility is shon to lie ithin a factor of to for all possile placement schemes and any replication factor. We also sho that for any replication factor, the clustered placement scheme has the loest proaility of data loss during reuild in critical mode among all possile placement schemes, hereas the declustered placement scheme has the highest proaility. Simulation results reveal hoever that these properties do not hold for the corresponding MTTDLs for a replication factor greater than to. This indicates that some alternate measures of reliaility may not e appropriate for comparing the MTTDL of different placement schemes. I. INTRODUCTION In today s large-scale distriuted storage systems, vast amounts of user data are stored among a large numer of nodes and disks. Distriuted peer-to-peer storage systems, such as Farsite, OceanStore, CFS, PAST, Glacier, and Shark, aim at providing inexpensive, highly-availale storage ithout centralized servers see [] and the references therein. In the presence of component failures, such as node and disk failures, reliaility, long-term duraility, and high availaility are ensured y storing user data in a redundant manner. Redundancy is achieved y employing the estalished, idely used replication and erasure coding schemes. Large-scale data storage systems use various redundancy schemes to prevent data loss that can occur ecause of multiple node failures. Replication is one of the idely used schemes here each data lock is replicated and the replicas are stored in different nodes to improve the chances that at least one replica survives hen multiple storage nodes fail. To maintain redundancy in the system, henever a node fails, a reuild process is initiated to create copies of the locks that ere lost. Wide-scale replication increases the reliaility, availaility, and duraility, ut it also increases the andidth and storage requirements of the system. Ho the replicas are placed plays an important role in ho much time the reuild process takes, and this in turn affects the reliaility of the system. In this paper, upper and loer ounds are derived on one particular measure of reliaility of the storage system for all possile replica placement schemes. To keep the prolem analytically tractale, the measure of reliaility that is used is different from the usual measure of reliaility, the mean time to data loss MTTDL. For a replication factor of to, this measure of reliaility allos an explicit calculation of MTTDL, hereas for higher replication factors, it is only speculative of the nature of the MTTDL. The theoretical results otained strongly indicate that this measure of reliaility is affected only negligily y the choice of the replica placement scheme for a ide range of node failure rates and node reuild rates. This is further supported y event-driven simulations hich agree ith the theoretical MTTDL predictions for a replication factor of to. Hoever, for a replication factor of three, simulation results sho that this is no longer true, and that the reliaility ill e strongly affected y the choice of replica placement scheme. This implies that the measure of reliaility used, hile suitale for predicting the MTTDL for a replication factor of to, is no longer suitale for predicting the MTTDL for higher replication factors. From our simulation results, e conjecture that another measure of reliaility may e more appropriate for predicting the MTTDL ehavior. As a result, e have also considered an alternate measure of reliaility and verified its appropriateness. For a replication factor of to, e demonstrate that reducing reuild times, and consequently the indo of vulneraility, does not necessarily lead to improved reliaility. This is ecause the reliaility depends not only on the indo of vulneraility, ut also on the numer of nodes that have the replicas of the data in the node lost. Distriuting replicas across many nodes increases the proaility that a second failure affects some of these replicas, therey causing data loss. The remainder of the paper is organized as follos: Section II discusses some related ork; Section III descries

2 the storage system model and the parameters considered; Section IV descries to measures of reliaility of a storage system; Section V contains the main contriution of this paper; Section VI gives the derivation of the main result; Section VII shos event-driven simulation results on MTTDL to compare ith the theoretical predictions; and Section VIII concludes the paper. II. RELATED WORK Data placement issue has een considered in [2]. The emphasis of that ork as on redundancy placement, namely, the placement of erasure coded data, rather than replica placement. The reliaility of a system ith the numer of nodes equal to the replication factor is addressed in [3]. The paper provides an explicit expression of MTTDL for such a system. Decentralized storage systems, such as CFS, OceanStore, Ivy, and Glacier, use replication to provide reliaility, ut employ a variety of different strategies for placement and maintenance. In architectures that employ distriuted hash tales DHTs, the choice of algorithm for data replication and maintenance can have a significant impact on oth performance and reliaility [4]. The paper proposes five different placement schemes. The scheme that minimizes the proaility of data loss is the lock placement scheme, in hich replicated data is stored in the same set of nodes. Similar results are also presented in [5], [6]. The findings of these orks match ith our theoretical results, hich also sho that less distriuted schemes have higher reliaility. System reliaility depends oth on the recovery mechanism and on the replica placement scheme. Fast recovery schemes reduce the indo of vulneraility and therefore improve the system reliaility [7], [8], [9]. Reuild times are reduced y appropriate replica placement strategies. In particular, distriuting replicas over many storage nodes in the system aids in quick reuild upon failure. Hoever, their analysis is ased on an idealistic assumption that replica-sets referred to as redundancy sets and groups in [7], [8], and as ojects in [9] fail independently. In contrast, in our analysis e assume that nodes fail independently and take into account the correlations among different replica-sets that this induces. As e sho in this paper, this leads to different results. Largely to approaches have een taken in comparing reliailities of systems ith different placement schemes: i approximate methods to compute MTTDL [9], and ii use of measures of reliaility other than MTTDL, such as the proaility that a storage system survives ithout data loss until time t as a function of t [5], [6], and proaility of data loss ithin a fixed period of time [8], [4]. In this paper, e take the latter approach and use simulations to compare the MTTDL ehavior to the ehavior of the measure used. III. SYSTEM MODEL The model and assumptions of the storage system considered and the failure and reuild model used are descried in this section. Tale I lists the different parameters used. TABLE I PARAMETERS OF A STORAGE SYSTEM c storage capacity of each node ytes n numer of storage nodes r replication factor s size of each data lock ytes reuild andidth availale at each node ytes/s λ Failure rate of a storage node s A. Storage System The storage system considered is a lock-ased storage system comprising n storage nodes ith total data storage capacity of nc ytes, here c is the capacity of each storage node. Every user data lock is of size s ytes, and is replicated r times. These r replicas are stored in the system such that no to replicas of a data lock are in the same node. The exact ay in hich the r replicas of each data lock are stored depends on the placement scheme used. For our theoretical calculations, e impose no restriction on the set of placement schemes that can e used. Hoever, for simulations, e use the folloing three schemes: a declustered, clustered,and c k-clustered placement. a Declustered Placement: Ther replicas of each data lock are stored in some r nodes out of the n nodes in the system. There are n r ays of choosing r nodes from the n nodes. In this placement scheme, all n r choices are equally used for storing replicas. Therefore, hen a node fails, the replicas of the locks in the failed node ill e spread over all remaining nodes. As the total capacity of the system is nc and the total size of a lock and its replicas is rs, thisplacementispossile only if s nc/ r n r.wetypicallyconsideranodeith capacity c =2TB consisting of 2 disks, each having TB capacity [0] and numer of nodes from n =4to n =96 in our simulations. For declustered placement to e possile for 00 nodes, the size of a data lock s must e less than aout 2.2 GB and 2.5 GB for a replication factor of to and three, respectively. Furthermore, for all n r choices to e equally used, the numer of data locks must e a multiple of n r.ifitisnot,thereilledifferencesinthenumerof locks that are shared among different sets of nodes. Hoever, for realistic values of node numers, e.g. n 000, andsmall locks sizes, e.g. s 0 MB, this difference is negligile. Clustered Placement: In this placement scheme, the n nodes are divided into disjoint sets of r nodes. All r nodes in agivensetaremirrorsofeachother,thatis,theystorereplicas of the same set of data locks. c k-clustered Placement: Thisisageneralizationofaove to schemes. In this placement scheme, the n nodes are divided into disjoint sets of k nodes called clusters. Each of these clusters is an independent storage system ith k nodes ith a declustered placement scheme. No data lock in one cluster is replicated in another cluster. It is easy to see that n-clustered placement is the same as declustered placement and r-clustered placement is the same as clustered placement. So for different values of k eteen r and n, ehavearoad range of different placements schemes.

3 B. Failure Model Storage nodes are comprised of one or more disks, a memory, processor, netork interface, and poer supply. Typically, these components are less reliale than the disks, and the failure of any of these components leads to a node failure []. The disks inside a node are assumed to e protected y a RAID scheme hich also corrects unrecoverale or latent sector errors y using either scruing or intra-disk redundancy [2]. Disk failures are assumed to cause a node failure only if the RAID system is unale to recover from these failures. Furthermore, in systems that use SMART Self-Monitoring Analysis and Reporting Technology for disks, the disks can e aggressively replaced to ensure that disk failures are not the main cause of node failures. Therefore, in our model, node failures are assumed to e caused primarily y the failure of components other than disks, such as the RAID-controller, memory, processor, netork interface, and poer supply. The failure of these components, and therefore the failure of nodes, is assumed to e independent ith exponentially distriuted times to failure. In particular, the time to failure of a node, T F,isassumedtoeexponentiallydistriuteithrateλ, that is, T F expλ. Notethatthisassumptionisincontrastto disk failures, hich are neither independent nor exponentially distriuted [3], [4]. Hoever, the aove model may not apply to node failures that are caused y softare ugs, DDoS attacks, virus/orm infections, node overloads and human error, as these factors may result in correlated node failures [5]. C. Reuild Model When a node fails, a reuild process is initiated to restore the lost replicas and ring the system ack to its original state, in hich each lock has r replicas. Spare space is assumed to e reserved on each node for reuild, and the ne replicas of the lost locks are created in the spare space of the surviving nodes. Once the ne replicas have een created in the spare space, a ne node is rought in and these nely created replicas are transferred to the ne node. The main advantage of creating replicas in the spare space first as opposed to creating replicas directly in a ne node is that reuild can e done in parallel distriuted reuild, see Fig. using the reuild andidth availale at many surviving nodes, therey reducing the reuild time. Also, once the replicas have een created, the system can survive another node failure ithout data loss. If a ne node as rought in first and the ne replicas ere created directly in the ne node, the reuild speed ould e restricted to the rite andidth availale at the ne node. This leads to a higher proaility that any of the surviving nodes fail efore reuild completes, therey causing data loss. During the reuild process, an average read-rite andidth of ytes/s is assumed to e availale at each node for reuild. This is usually only a fraction of the total andidth availale at each node, the remainder eing used to serve system-user requests. During the reuild process, let there e i data locks that are to e read from node i and i locks that are distriuted reuild 2 3 distriution of critical locks =, 2, = move ne replicas to ne node after successful reuild 2 3 distriution of ne replicas =, 2, = criticaldatalocks non-criticaldatalocks sparespace nelycreatedreplicasofcriticaldatalocks Fig.. Example of reuild model for a replication factor r =2system. When one node fails, the critical data locks are present in the surviving nodes. The reuild process creates replicas of these critical locks y copyingthemfrom one surviving node to another in parallel. to e ritten to node i. Theeffective andidth i availale for reading locks from node i is proportional to the amount of data read, that is, i i = i + i. One may argue that the fastest ay to read out all i locks efore node i fails ould e to first read all these locks using the full reuild andidth availale, and then to rite the i locks. Hoever, in a large system, if all i i = locks from all the nodes are read in parallel at full reuild andidth ithout copying them to other nodes simultaneously, then a large reliale uffer capale of storing data at rates of up to n is required to store these locks until they have een fully read out, and then copy them from the uffer to the nodes. We assume that in general such a large uffer capale of storing data at such high speeds is not availale. This means that e need to interleave the reads and rites at each node, that is, a fe locks are read out from each node, then the copies of these locks are ritten to other nodes, and so on until all locks have een read and copied. This results in an effective read andidth i as given aove. We assume that a typical disk has a read-rite andidth of 40 MB/s. Therefore a node ith 2 disks ill have aout 480 MB/s of read-rite andidth. During the reuild process, if afifthofthisisusedforreuildonanaverage,then = 96 MB/s. For a system ith n = 00 nodes, the netork

4 andidth must support up to aout n 9.6 GB/s for exchange of data among all nodes during reuild. This numer is, hoever, the orst-case estimate, hich holds for the case of declustered placement ecause the replicas of the lost data locks are present in all surviving nodes. On the other hand, clustered placement only requires aout r = 0.2 and 0.3 GB/s of netork andidth for exchange of data during reuild for a replication factor of 2 and 3, respectively.this is ecause in clustered placement the replicas of the lost data lock are present only in r nodes, and ne replicas are ritten effectively to one node. In this paper, the netork andidth is considered to e sufficiently high > n to allo exchange of data among all nodes during reuild. The mean time to complete reading all i locks from node i is given y E [ T i R i, i ] = is i = { s i + i if i 0, 0 if i =0. The nodes are expected to serve user requests hile performing reuild and there is also some randomness in the location of the data to e reuilt. As the andidth availale for reuild is only on average, thetimetocompletereadingall i locks at node i is assumed to e exponentially distriuted, that is, 2 T i R i, i exp /E [ T i R i, i ]. 3 It is further assumed that TR i i, i is independent of T j R j, j for i j and independent of the times to failure TF i.thisisecause,once i and i are fixed for all nodes, the only sources of randomness are the location of the locks to e read and the serving of user requests. The location of these locks and the user requests are not assumed to have any specific patterns that might induce correlations in the access times across different nodes. IV. MEASURES OF RELIABILITY To measures of reliaility that ill e used in this paper are descried in this section. A. Mean Time to Data Loss MTTDL Adatalossissaidtohaveoccurredinthesystemifthe replicas of at least one data lock have een lost y the system and cannot e restored. The average time it takes for the system until a data loss event occurs, also referred to as the mean time to data loss MTTDL, is a ell-knon measure of reliaility of the system that is idely used y the storage systems community. Analytically computing this measure for a replication-ased system ith a given replica placement scheme under certain failure and reuild models of the nodes is intractale exceptfor afeselectcases,suchasfortheasicmirroringschemefor replications factors r 2 [3]. To circumvent this prolem, some authors have proposed approximate continuous-time Markov chain models that enale analytical tractaility of the MTTDL computation [9], hereas others have proposed different measures of reliaility, such as the proaility that astoragesystemsurvivesithoutdatalossuntiltimet as a function of t [5], [6]. We take the latter approach and use a different measure of reliaility, namely, the proaility of data loss during reuild in the critical mode of the system. This measure has also een used in [6, Eq. 38]. The difference to earlier ork [5], [6] is that in this measure e also take the reuild time into account. Reuild time, and hence the time indo of vulneraility, is knon to e greatly affected y the placement scheme used and hence it is an important factor to e considered in measuring reliaility. For a replication factor of to, e directly relate the nely introduced measure to the MTTDL. B. Proaility of Data Loss during Reuild in Critical Mode To keep the prolem analytically tractale, a simple measure of reliaility is used. Assume that at a given point in time, r nodes of the system, chosen uniformly at random, have failed. This ill result in the loss of one or more replicas of some user data locks. Data locks that have lost r copies and have only one other copy surviving in the system are called critical locks. Data locks that have 2 or more copies in the system are called non-critical locks. The nodes containing these critical locks are called critical nodes and the system is said to e in a critical mode hen there is at least one critical lock in the system. The reuild process attempts to first create replicas of these critical locks to prevent data loss that can occur if any of the critical nodes fail. The measure of reliaility P is defined as P =Pr{DL r nodes failed}, 4 here DL is the event that data loss occurs ecause of a critical node failure efore the critical locks in that node have een copied to another node. If this event does not occur, the system goes to a ne state upon exiting the critical mode. As the reuild of non-critical data locks is still pending, this state is different from the initial one ith all data locks having r replicas. Modeling of the exact operation of the system using a Markov chain requires an enormous numer of such intermediate states. This is hy the evaluation of MTTDL is intractale. The proaility of loss of non-critical data locks caused y to or more node failures efore the reuild of critical locks completes is typically of higher order than P and hence ignored. This can e seen in to placement examples: i In declustered placement, if a node fails after the critical locks in it have een copied to another node, it does not result in data loss. Hoever, if to or more nodes fail after the corresponding critical data in them have een copied to other nodes, it could result in the loss of non-critical data locks. The proaility of such an event happening efore the reuild completes is hoever negligile compared to P. ii In clustered placement, each mirrored set is an independent storage system hich is unaffected y node failures and reuilds in other mirrored sets. Given that r nodes failed, there are to cases: a all these r nodes elonged to

5 the same mirrored set, or all these r nodes did not elong to the same mirrored set. In case a, the system is in critical mode, ut all the non-critical data have r replicas. This means that the proaility of losing a non-critical data lock y losing r nodes in the same mirrored set efore the reuild in the critical mirrored set completes is negligile compared to P. Incase,therearenocriticallockstoreuildand so the time to reuild critical locks is zero. Therefore, the proaility of losing non-critical data locks efore reuild completes is zero. For a replication factor of to, the measure of reliaility P can e directly translated to MTTDL hen the mean time to reuild is much smaller than the mean time to failure, that is, hen E[T R ] E[T F ]. MTTDL = nλp, r =2,E[T R] E[T F ]. 5 This can e shon for a k-clustered placement and therey for declustered and clustered placements as ell as follos. Let MTTDL clus. e the MTTDL of a cluster. As the clusters are independent of each other and there are n/k clusters in total, the MTTDL of the system is MTTDL clus. /n/k. Inagiven cluster, the loss of a node leads to critical mode. The mean time taken to lose a node is E[T F ]=E[min{TF,,Tk F }]= kλ. As the proaility of data loss in critical mode is P, the cluster enters critical mode P times on average efore data loss occurs. Assuming that the time to reuild E[T R ] E[T F ],MTTDL clus. = kλ P.Therefore,MTTDL = MTTDL clus. /n/k = nλp. V. EFFECT OF REPLICA PLACEMENT ON RELIABILITY The folloing proposition shos that the measure of reliaility P is not affected much ithin a factor of to y the choice of replica placement scheme. Proposition V.. For the system model descried in Section III, for all possile replica placement schemes and for any replication factor r 2, themeasureofreliailityp as defined in Section IV-B is ounded as follos: λnc n r + λc P < 2λnc n r. 6 Proof: The proof is given in Section VI. These ounds lie approximately ithin a factor of to for all practical values of the failure rate λ, nodereuildandidth, andnodecapacityc. Forλ =0 5 h, =480MB/s, and c =2TB, the factor + λc is equal to.00 hich implies that the reliaility of all placement schemes for a given replication factor is of aout the same order. Even for lopoer systems such as FAWN [7] and Pergamum [8] ith comparale failure rate λ and an order of magnitude higher reuild time c,thefactor + λc is equal to.0 and the ounds are still close to each other. Furthermore, as ill e shon in Section VI-B, Lemma VI., the upper ound corresponds to the case of uniformly distriuted placement of replicas, also referred to as random placement [9]; and the loer ound corresponds to the case of mirrored placement of replicas, also referred to as asic mirroring [5]. Similar results have also een otained y others [5], [6], aleit only for a select numer of schemes and a replication factor of to. The aove result differs from the already knon results in to main ays: it holds for all possile placement schemes and for any replication factor, and 2 it takes the effect of reuild process into account as it is knon that the reuild times can differ vastly for different placement schemes. The intuition ehind the result is that, in critical mode, hen all n critical nodes take part in reuild in parallel, the reuild time can e reduced n times; hoever, as the failure of any of these n nodes during reuild can result in data loss, the proaility that data loss occurs during reuild in critical mode stays the same. The factor to stems from the fact that, in declustered placement, each node does an equal numer of reads and rites of critical data, therey reducing the effective reuild andidth per node to /2, hereasin clustered placement, the nodes having critical data only do reads and the reuild andidth is. Therefore,underthis measure of reliaility, clustered placement is aout a factor to etter than declustered placement. We formalize this intuition in the aove Proposition. For a replication factor of to, e have the folloing ound on MTTDL: 2λ 2 nc < MTTDL λ 2 + λc, r =2. 7 nc The aove ound follos from 5 and Proposition V.. Once again, e oserve that, for practical values of λ, c/n, and, themttdlliesithinafactoroftoforallplacement schemes. The clustered placement scheme has the highest MTTDL, hich is aout a factor of to etter than that of declustered placement scheme. This is validated y eventdriven simulation results in Section VII. Remark. Note that the ounds on P and correspondingly on MTTDL for a replication factor of to do not depend on the size of data locks s. Althoughlarge-sizeddatalocks may not permit certain placement schemes, the ounds still hold true for all s, 0 <s c. VI. PROOF OF PROPOSITION V. The definition of P in 4 can e expanded as follos: hen r nodes fail, let there e critical locks. Denote the distriution of critical locks in the surviving nodes y =,, n r+,here i is the numer of critical locks in the ith surviving node, and n r+ i= i =. As the r failed nodes ere chosen uniformly at random from the n nodes in the system, the replica placement scheme chosen induces a proaility distriution on, Pr{ r nodes failed}. Inthecriticalmode,thereuild process attempts to make replicas of these critical locks efore the failure of a critical node results in data loss. Let =,,n r+ denote the distriution of the first replicas of all the critical locks in the surviving nodes. Once these replicas are created, the system is no longer in critical mode. As this distriution is chosen such that replica of

6 acriticallockfromanodeisnotcreatedinthesamenode,the distriution depends on itself, and therefore is expressed as a function of. Notethattheremayemorethanone choice of for a given, thatis, is not unique. When computing the ounds on P, efindthechoicesof that maximize and minimize P. Dependingonthefailurerate λ, thereuildandidth, thedistriutionofcriticallocksto e read, thedistriutionofreplicasofcriticallockstoe ritten, thereisacertainproaility,pr{dl, }, that there is data loss ecause of a failure that occurs efore successful reuild of these critical locks. The proaility P is expressed in terms of these to conditional proailities as follos: P = Pr{DL, } Pr{ r nodes failed}, 8 here the summation is over all possile distriutions of critical locks under the replica placement scheme chosen. An upper ound on P is otained as follos: P = Pr{DL, } : P i= Pr{ r nodes failed} 9 max : P max Pr{DL, i= } Pr{ r nodes failed} 0 = : P i= max : P max Pr{DL, i= } Pr{ critical locks r nodes failed} }{{} =: q = q max : P max Pr{DL, i= }, 2 here 9 follos y splitting the sum in 8 into to parts y introducing the numer of critical locks ; 0follos y pulling the maximum value of Pr{DL, } out of the inner summation; and follos y noting that the second summation is equivalent to Pr{ critical locks r nodes failed} ecause it counts all possile distriutions of critical locks. Similarly a loer ound on P is otained as follos: P q min : P min Pr{DL, i= }. 3 We no compute the terms inside the summation in inequalities 2 and 3 in the next three susections. A. Reuild at Any One Node Oing to our assumptions on failure and reuild models, and y making use of 2, the proaility that all the critical locks in node i are successfully read efore the node fails is given y Pr{TR i i, i <TF i } = +λe [ TR i i, i 4 ] = + λs i + i if i 0, 5 if i =0. B. Reuild at All Nodes In a critical mode ith critical locks, the proaility of data loss is equal to one minus the proaility that each of the n r + nodes successfully completes reading its critical locks, that is, Pr{DL, } n r+ = i= Sustituting 5 in 6, e get Pr{DL, } = Pr{T i R i, i <T i F }. 6 i I + λs 7 i + i, here I ={i : i 0, i n r +} is the set of critical nodes. The folloing lemma gives the maximum and the minimum of the aove proaility: Lemma VI.. For any distriution of critical locks =,, n r+ such that the total numer of critical locks is, andanydistriution =,,n r+ of the first replicas of these critical locks such that no to replicas of the same lock lie on the same node, the proaility of data loss efore successful completion of reuild is ounded as follos: λs + λs Pr{DL, } < 2λs. The loer ound is achieved for {, j = for some j and i =0 i j}; theset{, i + i = achieves the highest proaility of data loss. 2 n r+ i} Proof: See Appendix A. The aove lemma is a key result in the proof of Proposition V.. It shos the main points of this paper: i the proaility of data loss during reuild in critical mode for agivennumerofcriticallocks lies ithin a tight range of values; ii the loest proaility of data loss occurs hen all critical locks are in one node, hich is the case in clustered placement; and iii the highest proaility of data loss occurs hen the reuild is uniformly distriuted across all nodes, hich is the case in declustered placement. C. Expected Numer of Critical Blocks As q, definedin,istheproailityofhaving critical locks hen r nodes fail, the expected numer

7 of critical locks hen r nodes fail is given y E[] = q. Lemma VI.2. For any placement scheme, the expected numer of critical locks given r node failures is E[] = nc q = s r. n distinct data Proof: Let x,x 2,,x d e the d := nc sr locks hich have een replicated r times and stored in the system. Any given x i is replicated and stored in r different nodes. So the data lock x i can ecome critical hen any r out of these r nodes fail, hich can occur in r r = r ays. Given the failure of r nodes chosen uniformly at random from n nodes, the proaility that x i ecomes critical is independent of the replica placement scheme and is alays equal to Pr{x i is critical r nodes fail} = r n, i. r Given that there are a total of d distinct locks, the expected numer of critical locks hen r nodes fail is given y E[] =d Pr{x i is critical r nodes fail} = nc sr r n = nc r s n. r D. Upper Bound The upper ound on P in 6 is otained as follos: P < 2λs q = 2λnc n r here the inequality in the first step follos y applying Lemma VI. in 2, and the second step follos from Lemma VI.2. E. Loer Bound The loer ound on P in 6 is otained as follos: P λs = λnc λs, q + λs 8 q 9 + λc + λc n r, 20 here 8 follos y applying Lemma VI. in 3; 9 follos y noting that the numer of critical locks cannot e greater than the numer of locks on one node, that is, c s ;and20follosfromlemmavi.2. A. Placement Schemes VII. SIMULATION RESULTS Three different placements schemes ere used in the simulations - a declustered, clustered, andc k-clustered placement. FromourtheoreticalresultonMTTDLforareplication factor of to 7, e expect clustered placement to have the highest MTTDL, folloed y k-clustered placement and then y declustered placement. We also expect that clustered placement is etter than declustered placement y aout a factor of to. This is attriuted to the fact that, in declustered placement, each node performs equal numer of reads and rites of critical data, therey reducing the effective reuild andidth per node to /2, hereas in clustered placement, the nodes having the critical data perform reads only and therefore the reuild andidth is. B. Simulation Method Event-driven simulations ere used to calculate the MTTDL for the three placements schemes. Three types of events drive the simulation time forard: a failure events, reuildcomplete events, andc node-restore events. Thestateofthe system is maintained y three variales - time, thesimulated time, activenodes,thenumerofactivenodesinthesystem, and a vector of length r+ dataexposure =d 0,,d r, here d i is the numer of distinct data locks that have lost i replicas. Data loss occurs hen d r > 0. Ateacheventthese variales are updated. a Failure Event: Afailureeventtriggersthefolloing:i decreasing activenodes y one, ii scheduling the next failure event after time T F activenodes λ, iiiupdating dataexposure y taking into account the fact that a partial reuild of the most exposed data has occurred, and iv scheduling the reuild-complete event ased on the most exposed data in dataexposure and the placement scheme used. By nature of the reuild process, data placement is preserved, that is, declustered remains declustered and clustered remains clustered. This is ecause, hen the placement is declustered, critical locks are read from and ritten to all nodes at the same time and the ne replicas are placed such that declustering is preserved. When the placement is clustered, the replicas are created in a ne node directly instead of creating them in the spare space of existing nodes first and then copying them to a ne node. This preserves clustered placement. We have another tunale parameter, namely, the time taken to detect the failure of a node and start the reuild process, T delay. This is added to T R hile scheduling the reuild-complete events. This parameter is seen to have an influence only hen T delay is comparale to /nλ. Forpracticalsystems,/λ is on the order of 00, 000 h. If the system has n =00nodes, /nλ =000h. Typically T delay is much smaller than 000 handsoedonotpresenttheeffectofthisparameterinthe simulation results presented in this paper. Reuild-Complete Event:Areuild-completeeventtriggers the folloing i updating dataexposure y setting the amount of most exposed data to zero and adding this amount to a loer exposure level this means that the reuild process alays creates replicas of the most exposed data first, and ii scheduling the node-restore event hen all data have r copies completion of reuild process. The node-restore event is the time hen all the replicas that ere nely created have een successfully transferred to ne nodes and the numer of nodes is rought ack to n. Thenumerofnodestorestoreisstored in nodestorestore.

8 TABLE II RANGE OF VALUES OF DIFFERENT PARAMETERS FOR SIMULATION. Parameter Meaning Range c storage capacity of each node 2 TB n numer of storage nodes 4 to 00 r replication factor 2, 3 reuild andidth availale at 96 MB/s each node λ failure rate of a storage node h c Node-Restore Event: ThiseventincreasesactiveNodes y nodestorestore. For each set of parameters, the simulation is run 00 times, and the MTTDL and its ootstrap 95% confidence intervals are computed. Whereas for declustered placement, the simulation is run for n nodes, for clustered and k-clustered placement, the simulations are run only for one cluster, that is, r and k nodes respectively, and the otained MTTDL of the cluster is divided y n/r and n/k, respectively,tootainthemttdl of the system. This is ecause clusters are independent of the each other, and the numer of clusters is n/r and n/k for clustered and k-clustered placement, respectively. C. Simulation Results Tale II shos the range of different parameters that ere used for the simulations. Typical values for practical systems are used for all parameters, except for the mean time to failure of a node for a replication factor of three. For simulating a system ith a replication factor of three, the mean time to failure has een chosen artificially lo 000 h instead of 00, 000 h to run the simulations fast ecause the running times of simulations ith λ = 0 5 h are prohiitively high. Although this approach scales don the MTTDL y making failure events more frequent, it has een used as in [9] ecause it preserves the ratios of MTTDLs of the various schemes. Replication Factor To: Fig.2shosthecomparisonof theoretically predicted MTTDLs from the ounds in 7 and simulated values of MTTDL as a function of the numer of nodes for a system ith a replication factor of to ith declustered and clustered schemes. It is seen that the theoretical predictions are quite accurate. In addition, a third theoretical curve for the mirrored placement scheme ased on the formula from [6, Eq. 46] is plotted. It is oserved that this curve coincides ith the upper ound of 7, hich corresponds to clustered placement. The simulated MTTDL values for 4-clustered placement scheme are found to lie eteen the corresponding MTTDL values for clustered and declustered placement schemes. This is in agreement ith our theoretical prediction. Replication Factor Three: Fig.3shosthesimulatedvalues of MTTDLs for a replication factor of three for clustered and declustered placements. Declustered placement appears to e generally etter than clustered placement. The theoretical values for clustered placement ased on [3, Eq. 2] agree ith the simulation values. To investigate hy the ehavior of MTTDL is different from that of P, eplotp otained from the simulations MTTDL days r = 2 /λ = 00,000 hours declustered theoretical 0 5 clustered theoretical clustered theoretical [3] declustered simulated clustered simulated 4 clustered simulated Numer of nodes Fig. 2. Comparison of theoretically predicted and simulated values of MTTDL for a replication factor of to ith mean time to failure of a node equal to 00, 000 h; For the simulated results, 95% ootstrap confidence intervals are shon. MTTDL days r = 3 /λ = 000 hours 0 3 clustered theoretical [2] declustered simulated clustered simulated 6 clustered simulated Numer of nodes Fig. 3. Comparison of theoretically predicted and simulated values of MTTDL for replication factor three ith mean time to failure of a node equal to 000 h; For the simulated results, 95% ootstrap confidence intervals are shon. in Fig. 4. We oserve that the simulation values are half of the theoretical ones. This is ecause the theoretical results are otained assuming there is no reuild eteen node failures, hereas in simulation, hen the second node fails, approximately half of the data in the first lost node has already een reuilt. The simulation results, hoever, support the theoretical results of Proposition V. and Lemma VI. in that the declustered placement has a higher proaility of data loss y factor of to than the clustered placement. This shos that the measure of reliaility used, hile eing suitale for predicting the MTTDL for a replication factor of to, is no longer suitale for predicting the MTTDL for higher replication factors. In Fig. 5 e plotted the proaility of data loss given one node failure otained from our simulations. This measure, the

9 Pr of data loss given 2 node failures declustered theoretical clustered theoretical declustered simulated clustered simulated 6 clustered simulated Numer of nodes 0 3 r = 3 /λ = 000 hours Fig. 4. Comparison of theoretically predicted and simulated values of proaility of data loss given to node failures for a system ith a replication factor of three and mean time to failure of a node equal to 000 h; For the simulated results, 95% ootstrap confidence intervals are shon. Pr of data loss given node failure declustered theoretical clustered theoretical declustered simulated clustered simulated 6 clustered simulated r = 3 /λ = 000 hours Numer of nodes Fig. 5. Comparison of theoretically predicted and simulated values of proaility of data loss given one node failure for a system ith a replication factor of three and mean time to failure of a node equal to 000 h; For the simulated results, 95% ootstrap confidence intervals are shon. theoretical calculation of hich is ork in progress, appears to etter predict the MTTDL for a replication factor of three. Note that, for a replication factor of to, this measure is the same P. VIII. CONCLUSION AND FUTURE WORK In this paper, e shoed that all placement schemes have MTTDL values that differ y at most a factor of to for apracticalstoragesystemusingareplicationfactorofto. We used a measure of reliaility that is not only simple enough to enale sufficient analytical tractaility ut also comprehensive enough to take into account the effect of the placement schemes on the reuild process. The measure used, namely, the proaility of data loss during reuild in critical mode, is shon to e affected y at most a factor of to y the choice of placement scheme for any replication factor. Hoever, simulation results reveal that this property also holds for the corresponding MTTDLs, ut only for a replication factor of to. For higher replication factors, the measure used is not a suitale indicator of the MTTDL ehavior. This suggests that alternate measures of reliaility are not alays appropriate for comparing the MTTDL of different placement schemes. We also sho that the clustered placement scheme has the loest proaility of data loss during reuild in critical mode, hereas the declustered placement scheme has the highest proaility. This particular result is consistent ith results of [4], [5], [6]. Hoever, it differs from the results of [7], [8], [9]. We elieve that the inconsistency ith the latter set of pulications is mainly ecause their analysis is ased on an idealistic assumption that the replica sets referred to as redundancy sets or groups in [7], [8], and as ojects in [9] fail independently. In contrast, in our analysis e take into account the correlations among the failures of different replica sets that are induced y node failures. It is likely that the proaility of data loss given the first node failure is a suitale measure that can e used to compare MTTDLs of different placement schemes. Hoever, it is still to e seen hether this proaility is as intractale as MTTDL or not. On the other hand, e conjecture that the core results of this paper extend eyond replication-ased systems. It is likely that such results also exist for general erasure codes. APPENDIX A PROOF OF LEMMA VI. Let the total numer of data locks read from and ritten to each node e given y the distriution v := +. Let the set of all v for a given numer of critical locks e denoted y V, that is,v :={v critical locks}. Lemma A.. The set V has the folloing properties: i 0 v i, i {,,n r +}, v V, ii i v i =2, v V, iii The corner points of the convex hull of V are {v v j = v k = for some j, k and v i =0 i j, k}. Proof: i For any node i, v i 0 as it is the sum of the numer of locks, hich is alays non-negative. The total numer of critical locks is and the ne replicas of critical locks are created such that no to replicas of the same lock lie on the same node. This means that, the numer v i = i + i for any node i cannot e greater than, ecauseifthesum is greater than, ythepigeonholeprinciple,therehastoe at least one lock ith to of its replicas on the same node. ii i v i = i i + i i = + =2. iii The corner points of the convex hull of V are exactly the points here v i s are alloed to take the extremal values of 0 and. Asthesumofallv i should e 2 according to Property ii, the corner points are given the y set of all v, here v j = v k = for some j, k {,,n r +} and v i =0 i j, k.

10 A. Upper Bound To otain the upper ound in Lemma VI., it suffices to find see 7 max : P max + λs i= v V v i. i I Then, as all terms are inside the product aove are nonnegative y Lemma A. Property i, e use the Arithmetic- Geometric Mean Inequality to get i I + λs v i = + λs i v i I I + 2λs I, I here the second step follos from Lemma A. Property ii. Equality holds aove hen all v i s are equal. As the sum of all v i s is 2, equalityholdshenv i = i + i = 2 n r+ i. Plugging the aove inequality into 7, e get Pr{DL, } min : P i= + 2λs I. I By Taylor s theorem, it can e shon that + 2λs I > 2λs. 2 I Therefore, Pr{DL, } < 2λs. B. Loer Bound To otain the loer ound in Lemma VI., it suffices to find see 7 min : P min + λs i= v V v i. i I }{{} =: fv The function fv is a concave function defined on the convex hull of V. Therefore, the minimum of the function is attained at the corner points of the convex hull, hich y Lemma A. Property iii, are given y {v V v j = v k = for some j, k {,,n r +} and v i =0 i j, k}. The aove minimization prolem thus reduces to finding min : P i= i {j,k} I + λs. The minimum is attained hen {j, k} I =.Without loss of generality, let {j, k} I ={j}. Thisimplies j =. Therefore,pluggingtheaoveminimuminto7,eget Pr{DL, } λs, 22 + λs here equality holds for all {, j = for some j and i =0 i j}. ACKNOWLEDGMENT The authors ould like to thank Rüdiger Uranke of EPFL for his participation and support in the discussion of this ork, and the revieers for comments, hich helped improve the presentation of this paper. REFERENCES [] C. Miller, A. R. Butt, and P. Butler, On utilization of contriutory storage in desktop grids, in Proc. IEEE International Parallel and Distriuted Processing Symposium IPDPS 08, April2008,pp. 2. [2] K. Greenan, E. L. Miller, and J. Wylie, Reliaility of XORased erasure codes on heterogeneous devices, in Proc. 38th Annual IEEE/IFIP International Conference on Dependale Systems and Netorks DSN 08, June2008,pp [3] S. Ramahadran and J. Pasquale, Analysis of long-running replicated systems, in Proc. 25th IEEE International Conference on Computer Communications INFOCOM 06, 2006,pp. 9. [4] M. Leslie, J. Davies, and T. Huffman, A comparison of replication strategies for reliale decentralised storage, Journal of Netorks, vol., no. 6, pp , Decemer [5] A. Thomasian and M. Blaum, Mirrored disk organization reliaility analysis, IEEE Transactions on Computers, vol. 55, pp , Decemer [6] M. Jiang, J. Zhou, M. Hu, and Y. X. Ding, Fuzzy reliaility of mirrored disk organizations, in Proc International Conference on Convergence Information Technology ICCIT 07, 2007,pp [7] Q. Xin, E. L. Miller, T. Scharz, D. D. E. Long, S. A. Brandt, and W. Litin, Reliaility mechanisms for very large storage systems, in Proc. 20th IEEE / th NASA Goddard Conference on Mass Storage Systems and Technologies MSS 03, 2003,pp [8] Q. Xin, E. L. Miller, and T. J. E. Scharz, Evaluation of distriuted recovery in large-scale storage systems, in Proc. 3th IEEE International Symposium on High Performance Distriuted Computing HPDC 04, 2004, pp [9] Q. Lian, W. Chen, and Z. Zhang, On the impact of replica placement to the reliaility of distriuted rick storage systems, in Proc. 25th IEEE International Conference on Distriuted Computing Systems ICDCS 05, 2005,pp [0] IBM, XiV Storage System Specifications. [Online]. Availale:.xivstorage.com [] W. Jiang, C. Hu, Y. Zhou, and A. Kanevsky, Are disks the dominant contriutor for storage failures?: A comprehensive study of storage susystem failure characteristics, ACM Transactions on Storage, vol. 4, no. 3, pp. 25, Novemer [2] I. Iliadis, R. Haas, X.-Y. Hu, and E. Eleftheriou, Disk scruing versus intra-disk redundancy for high-reliaility raid storage systems, in Proc ACM International Conference on Measurement and Modeling of Computer Systems SIGMETRICS 08, 2008,pp [3] B. Schroeder and G. A. Gison, Understanding disk failure rates: What does an MTTF of,000,000 hours mean to you? ACM Transactions on Storage, vol.3,no.3,pp. 3,Octoer2007. [4] E. Pinheiro, W.-D. Weer, and L. A. Barroso, Failure trends in a large disk drive population, in Proc. 5th USENIX conference on File and Storage Technologies FAST 07, 2007,pp [5] S. Nath, H. Yu, P. B. Gions, and S. Seshan, Sutleties in tolerating correlated failures in ide-area storage systems, in Proc. 3rd conference on Netorked Systems Design & Implementation NSDI 06, 2006,pp [6] A. Dholakia, E. Eleftheriou, X.-Y. Hu, I. Iliadis, J. Menon, and K. Rao, A ne intra-disk redundancy scheme for high-reliaility RAID storage systems in the presence of unrecoverale errors, ACM Transactions on Storage, vol.4,no.,pp. 42,May2008. [7] D. G. Andersen, J. Franklin, M. Kaminsky, A. Phanishayee, L. Tan, and V. Vasudevan, FAWN: A fast array of impy nodes, in Proc. 22nd ACM Symposium on Operating Systems Principles SOSP 09, Octoer [8] M. W. Storer, K. Greenan, E. L. Miller, and K. Voruganti, Pergamum: Replacing tape ith energy efficient, reliale, disk-ased archival storage, in Proc. 6th USENIX Conference on File and Storage Technologies FAST 08, Feruary2008,pp. 6.