Regenerating Codes for Errors and Erasures in Distributed Storage

Transcription

1 Regenerating Codes or Errors and Erasures in Distributed Storage K V Rashmi, Nihar B Shah, Kannan Ramchandran, Fellow, IEEE, and P Vijay Kumar, Fellow, IEEE arxiv:0050v csit] 3 May 0 Abstract Regenerating codes are a class o codes proposed or providing reliability o data and eicient repair o ailed nodes in distributed storage systems In this paper, we address the undamental problem o handling errors and erasures during the data-reconstruction and node-repair operations We provide explicit regenerating codes that are resilient to errors and erasures, and show that these codes are optimal with respect to storage and bandwidth requirements As a special case, we also establish the capacity o a class o distributed storage systems in the presence o malicious adversaries While our code constructions are based on previously constructed Product- Matrix codes, we also provide necessary and suicient conditions or introducing resilience in any regenerating code I INTRODUCTION Distributed storage systems play a vital role in today s age o big data For cost considerations, these storage systems oten employ commodity hardware, which makes ailures a norm rather than an exception In order to saeguard the precious data against such ailures, the data is typically stored in a redundant manner In this paper, we consider a distributed storage system consisting o n storage nodes in a network, each having a capacity to store symbols over a inite ield F q Data comprising B symbols (the message) is to be stored across these n nodes An end-user (called a data collector) must be able to reconstruct the entire message by downloading the data stored in any k o these n nodes It ollows that such a system can tolerate ailure o any (n k) nodes, and under solely this requirement, can be realised using any n, k] maximum distance separable (MDS) code Frequent node ailures also call or eicient handling o the ailure events When a storage node ails, it is replaced by a new, empty node This replacement node is required to obtain the data that was stored previously in the ailed node, by downloading data rom the remaining nodes in the network We will term this process as repair or regeneration o a node A typical means o accomplishing this is to download the entire message rom the network, and extract the desired data rom it However, downloading the entire message, when it eventually stores only a raction k o it, is clearly wasteul o the network resources Regenerating codes ] are a class o codes that aim to reduce the amount o download during repair, while retaining the storage eiciency o traditional MDS codes Under the K V Rashmi, Nihar B Shah and Kannan Ramchandran are with the Dept o EECS, University o Caliornia, Berkeley, CA 94703, USA {rashmikv, nihar, kannanr}@eecsberkeleyedu P Vijay Kumar is with the Dept o ECE, Indian Institute O Science, Bangalore, India vijay@eceiiscernetin P Vijay Kumar is also an adjunct aculty member o the Electrical Engineering Systems Department at the University o Southern Caliornia, Los Angeles, CA This project is supported in part by an AFOSR grant (FA ) k= k= d=3 d=3 DC DC (a) (b) new new node node Fig : An example o the system parameters under a regenerating code (in the absence o errors/erasures) The system comprises o n = 5 storage nodes: (a) reconstruction is accomplished rom any k = nodes, (b) repair rom any d = 3 nodes operation o a regenerating code, a replacement node connects to any d ( k) existing nodes (termed helper nodes), and downloads symbols rom each This setting is illustrated in Fig With regenerating codes, the total amount o data d downloaded or repair is much smaller than the total size o the message B It is shown in ] that the parameters associated with a regenerating code must necessarily satisy k B min (, (d i)) () i=0 A regenerating code is said to be optimal i it satisies this bound with equality Since both storage and bandwidth come at a cost, it is naturally desirable to minimize both as well as However, it can be deduced (see ]) that achieving equality in (), or ixed values o B and n, k, d], leads to a tradeo between the storage space and the amount o download or repair d The two extreme points in this tradeo are termed the minimum storage regenerating (MSR) and minimum bandwidth regenerating (MBR) points These points have been well studied in the literature, and several explicit constructions o codes operating at these points are available ] 8] It has also been shown in 8] that essentially all other points on the tradeo curve are not achievable In this paper, we address the problem o handling errors and erasures in distributed storage networks using regenerating codes In particular, we are interested in codes that can perorm reconstruction and eicient repair in the presence o errors and erasures at the nodes or in the links Such codes are clearly useul in handling errors and packet losses occurring in the network In addition, such codes can also be used to provide security in distributed storage systems, where malicious adversaries may corrupt the data stored in some nodes in the system

2 IEEE International Symposium on Inormation Theory (ISIT), 0 n n n 3 n 4 n 5 n 6 a c a + 3c a + 8c 4a + c 4a + 3c b d 7a + b + 3d 3a + 6b + 8d a + 9b + d 9a + b + 3d (c + d) 8a + b + 3(c + d) a MDS in a, b, (c+d) Decode a, b and (c+d) even under one error Retain a and b b Fig : An example o a universally resilient MSR code with parameters n = 6, k =, d = 3], (B = 4, =, = ) Also depicted is an instance o one error correction during repair o node, by connecting to (d + ) = 5 nodes The aspect o security in distributed storage systems employing regenerating codes is studied in 9], where an outer bound is provided or the total amount o data that can be stored securely in the presence o malicious adversaries The model presented in 9] considers correction o a ixed number o errors, by designing encoding and storage algorithms speciically or this purpose It is also shown that the code in 8] achieves this bound with an appropriate choice o the underlying MDS code, or the case d = n at the MBR point However, apart rom this case, no other constructions o secure regenerating codes are known in the literature In the present paper, we provide a new approach or handling errors and erasures in regenerating codes Under our system model, the data is encoded and stored assuming no error/erasure-resiliency requirements The task o correcting the errors or erasures is perormed in the decoding stage by downloading a larger amount o data In contrast to 9], our approach allows or choosing a dierent level o resiliency during each event o repair or reconstruction, depending on the prevalent network state A second advantage o our approach is that it allows or introducing resilience in regenerating codes that were not designed or handling errors and erasures We present explicit code constructions or the parameters (i) MSR, all n, k, d k ] and (ii) MBR, all n, k, d] In addition, we show the optimality o these codes through tight outer bounds on the storage and bandwidth requirements This establishes the capacity o such systems or these parameters Moreover, this also establishes the capacity o regenerating codes in the presence o malicious adversaries or these parameters, which had remained open The decoding algorithms have a (polynomial) complexity, identical to that o Reed-Solomon codes The codes presented here are based on a Product- Matrix construction introduced in ], that also possess other appealing properties such as linearity, scalability, and ease o implementation An example o an MSR error/erasure resilient code is depicted in Fig A natural question that ollows is whether any regenerating code can be made resilient to errors and erasures in this ashion In this paper, we also answer this question by providing necessary and suicient conditions or a regenerating code to be resilient to errors and erasures It turns out that, to date, the product-matrix codes are the only codes that satisy these properties While we were writing this paper, we came across a contemporaneous independent work 0] that is related to the present paper, and deals with byzantine ault tolerance using product-matrix codes o ] The authors use a CRC to check the integrity o data during repair and reconstruction, and a eedback scheme to iteratively correct them However, CRC based schemes are not applicable in settings such as protection against malicious adversaries, since the CRC can also be corrupted by the adversary The present paper takes a more undamental look at the problem o handling errors and erasures in regenerating codes The rest o the paper is organized as ollows The system model is described in Section II, and outer bounds or this model are also provided in this section Explicit constructions o error-resilient regenerating codes are provided in Section III Necessary and suicient conditions or providing error and erasure resiliency in any regenerating code are presented in Section IV II SYSTEM MODEL We consider a block-based model where the message is divided into blocks, and there is no coding across the blocks All operations o encoding, decoding and repair are perormed independently across the blocks Thus the regenerating code parameters (or the error-ree case) described in Section I can be considered as pertaining to a single block o data More concretely, we consider a block to consist o B message symbols, and the storage capacity in each o the n nodes to be symbols per block One can reconstruct the B-message symbols by downloading the data pertaining to this block rom any subset o k nodes, and regenerate the data stored in any node by downloading symbols each (pertaining to this block) rom any d nodes We assume that the granularity o any error or erasure is one block In other words, we assume that during repair, all the symbols passed by a helper node suer the same ate: either all these symbols are erased, or all are in error, or all are perectly received Similarly, during reconstruction, all the symbols passed by a node are assumed to suer the same ate The assumption o block errors/erasures can accurately model several scenarios o interest, two o which are described here Consider handling o packet drops during transmission across a network The size o a packet during reconstruction

3 IEEE International Symposium on Inormation Theory (ISIT), 0 (a) DC DC κ=3 κ=3 (b) Δ=4 Δ=4 new new node node Fig 3: The system setting or an (s =, t = 0)-resilient regenerating code with k = and d = 3 Here, connectivity during (a) reconstruction is κ = k + = 3, and (b) repair is = d + = 3 and repair can be assumed to be a multiple o and (since the packet size will usually be much larger than these parameters) Thus, when a packet is delayed or dropped, all the or symbols corresponding to a block are erased In the security scenario, to account or compromise o any node or link to a malicious adversary, one needs to protect against corruption o the entire data on that node or link Thus, we can model the security scenario in our ramework by simply considering the entire data as a single block We note that the absence o this assumption will allow arbitrarily scattered errors and erasures in the model It can be shown that codes attempting to guard against such scattered errors/erasures require signiicantly larger overheads As discussed above, many applications can be modelled as having block errors, thus avoiding these overheads We now deine ormally, the error/erasure handling capability o a regenerating code Deinition ((s, t)-resilient code): A regenerating code is (s, t)-resilient i it can correct upto s erasures and t errors during repair as well as reconstruction As discussed previously, under our system model, errors and erasures are corrected by downloading additional data during reconstruction or repair One way to obtain additional data is to connect to a larger number o nodes, and we choose this approach More precisely, we allow a connectivity o ( d) nodes during repair and κ ( k) nodes during reconstruction The parameters and κ depend on the error/erasure correcting capability expected out o the system Fig 3 depicts the (, 0)- resilient version o the system in Fig We now provide an outer bound on the capacity o resilient regenerating codes The bound is obtained by adapting the outer bound o 9] (or the omniscient adversary case) to our system model, and extending it to handle erasures as well Theorem : A (s, t)-resilient regenerating code, connecting to and κ nodes or repair and reconstruction respectively, must satisy k B min (, (d i)) () i=0 where d = s t and k = κ s t Proo (sketch): The bound can be derived either using cut-set arguments in an inormation low graph as in ], 9] or using inormation theoretic arguments as in 8] The complete proo is available in ] We will call an (s, t)-resilient regenerating code as optimal i it meets the bound in Theorem Clearly, or (s, t)- resilience, we need d + s + t n (3a) k + s + t n (3b) where (3a) represents the connectivity required during repair, and (3b) during reconstruction In many applications o interest, it may be desired to provide dierent levels o reliability during dierent instances o reconstruction and repair For instance, under changing network states in the packet erasure setting, or varying threat levels under the security setting We deine codes that possess such a property as universally resilient codes Deinition (Universally resilient code): A regenerating code is universally resilient i it is simultaneously (s, t)- resilient or all s and t satisying (3) Thus a universally resilient code can correct upto s erasures and t errors by downloading (or ) symbols each rom s+t additional nodes during repair (or reconstruction), as long as d + s + t n (or k + s + t n) In the next section, we present constructions o productmatrix codes that are universally resilient and optimal Beore moving on to the constructions, we briely digress to explore some connections with network coding Relation to Network Coding: The regenerating codes problem described above, i relaxed to the requirement o repair o only the systematic nodes, turns out to be a non-multicast network coding problem (see, Section I-C]) While the occurrence o errors and erasures in multicast network coding are well studied in the literature 3] 5], the results o the present paper lead to a class o non-multicast networks or which the error/erasure capacity o the network can be achieved by codes that are linear, deterministic and explicit III ERROR/ERASURE-RESILIENT REGENERATING CODES We provide explicit constructions o universally resilient MSR and MBR codes or ) MSR, all parameters n, k, d k ], and ) MBR, all parameters n, k, d], which meet the outer bound provided in Theorem Thus, this also establishes the capacity o such a system or these parameter values These codes are based on product-matrix (PM) codes that were introduced in ] As discussed in Section II, in our approach, the encoding algorithm is identical to the error/erasure ree case Hence, we irst briely describe the product-matrix code construction or the error/erasure ree case ], which meets the bound in () We then present the decoding algorithms (or both repair and reconstruction) that can handle s erasures and t errors or all values o s and t satisying (3), with = (d + s + t) and κ = (k + s + t) These parameters satisy the bound in Theorem with equality, thereby establishing the optimality o these codes We begin with the minimum storage case, and subsequently present the minimum bandwidth case The example depicted in Fig is an optimal, universally resilient MSR code 3

4 IEEE International Symposium on Inormation Theory (ISIT), 0 A Universally resilient MSR Codes MSR codes use the minimum possible storage at each node Since the data rom any k nodes should suice to reconstruct all the B message symbols, each node must necessarily store at-least a raction k o the entire data Hence or an MSR code we have = B k To meet the bound () with equality (in absence o errors/erasures), an MSR code must satisy B = k, d = + (k ) (4) In this section we present explicit constructions o optimal, universally resilient MSR codes or all parameter values n, k, d k ] The code is designed or the case d = k, which can be extended to any d > k via the shortening technique or MSR codes provided in ], 4] When d = k, rom (4), we get = (k ), B = k(k ) (5) Since both and B are multiples o, we obtain the optimal code or the desired parameters (B,, ) by irst constructing an optimal code or = (k ), B = k(k ) = ( + ), =, (6) and then concatenating this code times in parallel The PM-MSR code in ] can be described in terms o an (n ) code matrix C = ΨM, with the i th row o C containing the symbols stored in node i The (n d) encoding matrix Ψ is o the orm Ψ = Φ ΛΦ], where Φ is an (n ) matrix and Λ is an (n n) diagonal matrix satisying: (a) any rows o Φ are linearly independent, (b) any d rows o Ψ are linearly independent, and (c) the diagonal elements o Λ are all distinct The choice o the matrix Ψ governs the choice o the inite ield F q, eg, choosing Ψ as Vandermonde (careully chosen to satisy condition (c)) permits any q 4n The ((d = ) ) message matrix M is o the orm M = S S ] t, where S and S are ( ) symmetric matrices The superscript t is used to denote the transpose o a vector or matrix The two symmetric matrices S and S together contain ( +) distinct elements, which are populated by the B = ( + ) message symbols This completes the description o the encoding algorithm The ollowing theorems show that this code is optimally universally resilient during repair and reconstruction Theorem (MSR Repair): In the MSR code presented, the symbols stored in any node can be recovered by downloading symbols each rom any = d + s + t nodes, in the presence o upto s (block) erasures and t (block) errors Proo: Since we consider only block errors and erasures, it suices to describe the repair algorithm or the code with =, and the same algorithm is applied in parallel to obtain the repair algorithm or the desired code Consider ] ailure o node in the system, and let φ t λ φ t be the row o Ψ corresponding to the ailed node Thus the symbols stored in node are ] φ t λ φ t M = φ t S + λ φ t S (7) The code can be converted to a systematic orm (such as in Fig ) by a simple symbol remapping technique as shown in, Section V-B] The replacement or the ailed node connects to an arbitrary set {h j j =,, } o nodes To acilitate repair o node, node h j computes the inner product ψ t Mφ h j and passes on this value to the replacement node Letting m = Mφ, we can write the symbol passed by node h j as ψ t m h j Thus the symbols obtained at the destination are Ψ rep m, where ] t Ψ rep = ψ h ψ h ψ h Since any d rows o Ψ are linearly independent by construction, and since Ψ rep comprises a subset o the rows o Ψ, Ψ rep m is simply an MDS encoding o the d symbols in the vector m It ollows that this code has a minimum distance o ( d + ) = (s + t + ) which allows us to recover m using standard decoding algorithms 6] in the presence o upto s erasures and t errors Thus the replacement node now has access to ] m = Mφ = S φ S φ Since S and S are symmetric matrices, the replacement node has access to φ t S and φ t S Using this it can obtain φ t S + λ φ t S, which is precisely the data previously stored in node Theorem (MSR Reconstruction): In the MSR code presented, a data-collector can reconstruct all the B message symbols by downloading data stored in any κ = k + s + t nodes in the presence o upto s (block) erasures and t (block) errors Proo (sketch): The data reconstruction property o the code in the error-ree case, as shown in ], implies that the data passed by the κ nodes are MDS over the inite ield F q Over this inite ield, the message is o size k, and the minimum distance o this MDS code is (κ k + ) = (s + t + ) This guarantees reconstruction o the k source symbols over F q, and equivalently the k = B source symbols over F q, in the presence o upto s erasures and t errors Explicit data-reconstruction algorithms are provided in ] B Universally resilient MBR Codes MBR codes achieve minimum possible download during repair: a replacement node downloads only what it stores, resulting in d = To meet the bound () with equality (in absence o errors/erasures) an MBR code must satisy B = ( kd ( k )), = d (8) In this section we present explicit constructions o optimal, universally resilient MBR codes or all parameter values n, k, d] As in the MSR case, B and are multiples o, and we irst construct codes or B = ( kd ( k )), = d, = (9) The desired code can be obtained by concatenating copies o this code 4

5 IEEE International Symposium on Inormation Theory (ISIT), 0 The PM-MBR code in ] has a similar orm, C = Ψ M, as the PM-MSR code The MBR code has the (n d) encoding matrix Ψ o the orm Ψ = Φ Σ], where Φ is an (n k) matrix satisying: (a) any k rows o Φ are linearly independent, (b) any d rows o Ψ are linearly independent For instance, one can choose Ψ to be a Vandermonde matrix The (d d) message matrix M is symmetric and consists o the B message symbols arranged as M = S T T t 0 Here, the ((d k) k) matrix T and the (k k) symmetric matrix S contain the B = kd ( ) k = k(d k) + k(k+) message symbols as their elements The ollowing theorems show that this code is optimally universally resilient during repair and reconstruction Theorem 3 (MBR Repair): In the MBR code presented, the symbols stored in any node can be recovered by downloading symbols each rom any = (d + s + t) nodes, in the presence o upto s (block) erasures and t (block) errors Proo: As in the case o MSR, it is suicient to describe the repair algorithm or the code with = Consider ailure o node in the system, and let ψ t be the row o Ψ corresponding to the ailed node Thus the symbols stored in node are ψ t M We will ollow the notation as in Theorem The helper node h j passes the symbol ψ t Mψ h j Denoting m = Mψ, the symbols obtained at the destination can be written as Ψ rep m where ] t Ψ rep = ψ h ψ h ψ h By construction, Ψ rep m corresponds to an MDS encoding o the vector m As in the case o MSR, this code has minimum distance o ( d+) = (s+t+) which allows us to recover m in the presence o upto s erasures and t errors Since the message matrix M is symmetric, m t = ψt M t = ψ t M is precisely the set o symbols required Theorem 4 (MBR Reconstruction): In the MBR code presented, a data-collector can reconstruct all the B message symbols by downloading data stored in any κ = (k + s + t) nodes in the presence o upto s (block) erasures and t (block) errors Proo (sketch): As in the MSR case, the proo exploits the reconstruction property o PM-MBR codes in the error-ree case ] The reconstruction property implies that, over F q, the minimum distance o the code is (κ k + ) = (s + t + ) This guarantees reconstruction o the k symbols over F q, and equivalently the B source symbols over F q, in the presence o upto s erasures and t errors Explicit data-reconstruction algorithms are provided in ] ] IV NECESSARY AND SUFFICIENT CONDITIONS The conversion o the product-matrix codes into universally resilient codes, as described in Section III, raises a natural question as to whether any regenerating code can be made universally resilient in a similar manner We answer this question by providing a necessary and suicient condition or the same To the best o our knowledge, the only codes today that satisy this condition are the product-matrix codes Theorem 5: An n, k, d] regenerating code can be made universally resilient i and only i the ollowing condition holds: during any instance o repair, the data passed by a node h helping in the repair, to the ailed node, depends only on h and, and not on the identities o the other nodes helping in this repair The proo o the theorem is provided in ] Remark : Clearly, since the number o nodes contacted during repair must satisy (d + s + t) (n ), the requirement o having either s > 0 or t > 0 requires that n > (d + ) Thus the code should not restrict the number o nodes n to be (d ) The only explicit regenerating codes that support n > (d+) are the high-rate approximately-exact MSR codes o 3] and the product-matrix codes ] However, the MSR codes o 3] do not satisy the condition provided in Theorem 5 As shown in Section III, the product-matrix codes satisy this condition ACKNOWLEDGEMENT The authors would like to thank Salim El Rouayheb and Sameer Pawar or ruitul discussions REFERENCES ] A G Dimakis, P B Godrey, Y Wu, M Wainwright, and K Ramchandran, Network coding or distributed storage systems, IEEE Trans In Theory, vol 56, no 9, pp , 00 ] K V Rashmi, N B Shah, and P V Kumar, Optimal exact-regenerating codes or the MSR and MBR points via a product-matrix construction, IEEE Trans In Theory, vol 57, no 8, pp , Aug 0 3] K V Rashmi, N B Shah, P V Kumar, and K Ramchandran, Explicit construction o optimal exact regenerating codes or distributed storage, in Proc Allerton Con, Urbana-Champaign, Sep 009 4] N B Shah, K V Rashmi, P V Kumar, and K Ramchandran, Explicit codes minimizing repair bandwidth or distributed storage, in Proc IEEE ITW, Cairo, Jan 00 5] C Suh and K Ramchandran, Exact regeneration codes or distributed storage repair using intererence alignment, in Proc ISIT, Jun 00 6] I Tamo, Z Wang, and J Bruck, MDS array codes with optimal rebuilding, in Proc IEEE ISIT, St Petersburg, Jul 0 7] D Papailiopoulos, A Dimakis, and V Cadambe, Repair optimal erasure codes through hadamard designs, in Proc Allerton Con, 0 8] N B Shah, K V Rashmi, P V Kumar, and K Ramchandran, Distributed storage codes with repair-by-transer and non-achievability o interior points on the storage-bandwidth tradeo, IEEE Trans In Theory, vol 58, no 3, 0 9] S Pawar, S El Rouayheb, and K Ramchandran, Securing dynamic distributed storage systems against eavesdropping and adversarial attacks, IEEE Trans In Theory, vol 57, no 0, pp , 0 0] Y Han, R Zheng, and W Mow, Exact regenerating codes or byzantine ault tolerance in distributed storage, in Proc INFOCOM, 0 ] K V Rashmi, N B Shah, K Ramchandran, and P V Kumar, Unconditionally secure storage-bandwidth optimal distributed storage codes, in preparation ] N B Shah, K V Rashmi, P V Kumar, and K Ramchandran, Intererence alignment in regenerating codes or distributed storage: Necessity and code constructions, IEEE Trans In Theory, vol 58, no 4, 0 3] N Cai and R Yeung, Network error correction, ii: Lower bounds, Comm in Inormation & Systems, vol 6, no, pp 37 54, 006 4] S Jaggi, M Langberg, S Katti, T Ho, D Katabi, and M Médard, Resilient network coding in the presence o byzantine adversaries, in INFOCOM, 007 5] R Koetter and F Kschischang, Coding or errors and erasures in random network coding, IEEE Trans In Theory, vol 54, no 8, pp , 008 6] F MacWilliams and N Sloane, The Theory o Error-Correcting Codes, Part I North-Holland Publishing Company, 977 5