Network Codes for Next-Generation Distributed Storage:! Opportunities & Challenges. Kannan Ramchandran

Transcription

1 Network Codes for Next-Generation Distributed Storage:! Opportunities & Challenges Kannan Ramchandran University of California, Berkeley Joint work with: Salim El Rouayheb (UCB), Hao Zhang (UCB), Sameer Pawar (UCB) & Minghua Chen (CUHK)

2 Acknowledgments Many students, former students, colleagues and collaborators have shaped and contributed to this talk. An incomplete list (in alphabetical order) includes: Alex Dimakis (USC) Nima Noorshams (UCB), P. Vijay Kumar & his students N. Shah & K. Rashmi (IISc.), Abhay Parekh (UCB) Changho Suh (UCB) Special thanks to Raymond Yeung & Alfred Ho (CUHK).

3 Data Everywhere Airport Coffee shop Office Home Increasing popularity of data-enabled devices and Hot spots, 4G, Wimax, Wi-fi etc, => always online infrastructure. Truly mobile users Want to access seamlessly and to share data anytime anywhere

4 Danger signs

5 Off To The Cloud Want our files to follow us wherever we go Put them in the cloud Get them from the cloud Data Centers P2P Systems Cloud Applications

6 Server Cloud: is the honeymoon Main Challenge: Server cloud formed of unreliable and untrusted components over? The first and fundamental principle in building robust architecture is to design for failure. Component failures are the norm rather than the exception. Google File System

7 This talk: VoD System with Distributed Caches Distributed Cache Network can be: Set-Top Boxes, home gateways ISP P2P infrastructure nodes: Comcast Idle Internet PC users: Thunder, PPStream, PPLive, Wuala etc Demand distribution Heavy tail Massive video library Cache Network Server Cache Network Goal: Build in reliability and scalability upfront in the architecture

8 Building in reliability and efficiency

9 Three-Tier Architecture: Separation principle Commodity disks Tier I: Server Cloud Tier II: Distributed Cache Network (DCN) Data reliability Serve as life-line Hard guarantees Scalability Adaptivity Soft guarantees Tier III: Users Availability QoS

10 Distributed Cache Network Problem Statement: Minimize server load Subject to: Connectivity constraints Storage constraints Bandwidth constraints Who connects to whom? Choices to be made: What packets to store? Of what movies? Which packets to serve? To which user? Combinatorial explosion: seems to be a hopeless task! Scalability requirement distributed solution

11 Talk Outline I Backdrop & Motivation Scalable & Reliable VoD system II Server Tier Reliability Regenerating codes Uncoded Exact Repair: DRESS codes Min. server load: dist. opt. alg. Adaptivity and growth III Distributed Cache Network

12 A Multi-dimensional Problem Reliability Codes Decoding complexity C rate Traditional Coding Theory Coding for cloud storage

13 Different Codes for Different Folks 2MB File a b n 1 n 2 n 3 Replication a 1MB a b a New node a Download 1 MB (4,2) MDS Erasure Code n 1 n 2 n 3 a b a+b b New node a+b a Download 2MB n 4 b n 4 a+2b tolerates 1 failure tolerates 2 failures Reliability Bandwidth Complexity/ Disk Read

14 Regeneration Codes: Overview Classical approach: download 2 MB 1MB 1MB a But new node (e) is downloading 2 MB to store only 1MB! Q: Possible to download less? a b A: If (e) connects to 2 peers, no But it s possible to download 1.5MB (MSR: Min. Storage Regen. code) b a+b Q: Is this the best possible? a+2b e A: No, by storing a bit more, BW can reduce to 1.2 MB (MBR: Min. BW Regen. Code) When coding is used, creating new packets is not a trivial task. The problem is that to create a new packet we must have access to the entire data object

15 Proof sketch: Information flow graph 1MB a a data collector S a b b c d b c d β β β e data collector α =1MB Dimakis, G., W., W., R β 2 β 1/2MB Total download 1.5MB

16 Proof sketch: reduction to multicasting data collector a a data collector S b c d b c d β β β e data collector data collector data collector data collector Storage Code regeneration is reduced to multicasting on the information flow graph. sufficient iff minimum of the min cuts is larger than file size M. (using Network coding theorem by Yeung et al.) Per-link repair BW Total Repair BW= (M/k)/(1-R) where R= (k-1)/(n-1) Dimakis, G., W., W., R. 07

17 Fundamental Storage/Bandwidth Tradeoff File = 20 MB (40,20) code Storage cost Internet, Wireless MBR Min. Bandwidth Regime Code Storage Bandwidth Savings MDS 1 MB 20 MB MSR 1 MB 2 MB 10x MBR 1.33 MB x Per Failure Savings Optimal Tradeoff can be attained with Random Linear Network Coding MSR Archival applications Min. Storage Regime MDS Repair BW

18 Yearly Savings: Data Centers Google data center in Oregon 800,000 servers Total data= 1.6 million TB Failure rate=4% per year Repair every 2 days 3-replication reliability ( ) Reliability achieved by a [9,6] MBR code Comparison to 3x Replication Yearly Savings Percentage Number Storage 20% 6400 hard disks Bandwidth 20% 6400 TB Why? Same amount of data Replication needs more hard disks More failures higher repair bandwidth

19 Random Linear Network Codes 1mb 1mb a1 a2 b1 b2 1 1 (n,k,d) # nodes contacted for repair b1+b2 # nodes user contacts Total # of nodes a1+b1 a2+b2 a1+2b1 a2+2b a1+b1+2a2+2b2 a1+2b1+3a2+6b2 e1 e2 Is that the end of the story?

20 Exact Regeneration WHY? a b 1. Alphabet Size 2. Systematic form of the data 3. Security (talk by Salim) M (file size) c d e = a We want exact regeneration!

21 Exact repair problem: Non-Multicast a " user File b "... replacement node a, b wants a only a+b user... a+2b user Want everything k choose n users that want all the data n repair nodes that want distinct data A very hard problem in general

22 Tradeoff with Exactness Constraint Storage cost Product-Matrix codes (Rashmi et al.), MBR Focus of this talk (DRESS codes: El Rouayheb & R) Intermediate points not achievable (Shah et al. 10) MSR Asymptotically achievable (Suh & R 10, Cadambe et al. 10): Interference Alignment techniques MDS Repair BW

23 Multiple System Considerations Bandwidth-efficiency comes with a price-tag. 1) Excessive disk I/O delay in repair 2) Complexity: Linear mixing is not free 3) Personal conversation with Jay Wyley (HPL) and Arif Merchant (Google) Can we have codes with min overhead of: 1) Repair bandwidth 2) Disk reads 3) Computational complexity?

24 DRESS codes No data processing for repair: uncoded repair Minimum disk reads for repair Linear encoding/decoding complexity Maximum Distance Separable file MDS Code Fractional Repetition Distributed REplication-based Simple Storage Storage Cloud Surprise: no loss of bandwidth efficiency Bonus: well-suited for security applications System price: repair not as flexible w.r.t. who can help

25 Codes For d=n-1 Min Bandwidth Regime (MBR) (n,k,d)=(4,2,3) 3MB Max file size= 5 MB: cut-set bound file 5MB parity-check MDS Code user Uncoded Repair Minimum reads No processing How about d<n-1? Replacement node Rashmi, Shah, Kumar & R '09

26 Formal DRESS Codes Lines in projective planes intersect in exactly one point Fano Plane Projective plane of order 2 Points Packets Lines Storage nodes

27 Higher Order Projective Planes order m=2 m=3 m=4 Projective planes are guaranteed to exist for any prime power order m.

28 More Codes From Steiner Systems A Steiner system is a collection of points & lines such that: 1. There are points in total 2. Each line contains exactly points We want t=2 3. There is exactly one line that contains any given points Fano plane =S(2, 3, 7) # pts on a line Total # points S(2,3,9)

29 Codes from Steiner Systems Theorem : ( ElRouayheb & R Allerton 10) A Steiner system gives a capacity-achieving DRESS code with parameters using correspondence lines: nodes, points: packets. DRESS Codes? Steiner Systems Projective planes Dual codes? Open problem: DRESS codes beyond Steiner systems????

30 System Design : Formal DRESS Codes in the Server Cloud Stripe 20 MB MDS Code Stripe=20 MB (delay constraint) Edges=packets Outer MDS code rate 0.5 (cache network constraint) DRESS code based on regular graphs Vertices= storage nodes (n,k,d)=(13,5,6) Regular Graph on n=16 Every packet repeated twice vertices and degree d=6 Each node has 6 packets Any 5 nodes have 20 distinct packets (independent linear equations)

31 Talk Roadmap I Background & Motivation II Servers Min. server load: dist. opt. alg. Adaptivity and growth: III Distributed Cache Network

32 Recap: Three-Tier Architecture Tier I: Server Cloud The bottom line: Server Bandwidth Tier II: Cache Nodes Demand distrib. Heavy tail movies Tier III: Users

33 Recap: Distributed Cache Network Goal: Minimize total number of packets served by the server Subject to: Connectivity constraints Storage constraints Bandwidth constraints Who connects to whom? Choices: What packets to store? of which movies? Which packets to serve? to which user? Challenge: Combinatorially explosive optimization problem that needs to be solved in a fully distributed manner

34 Step 1: From Movies to Flavors Use coding to obviate packet scheduling nightmare Manage quantity rather than identity Random Linear Network Codes x 1 x 2 x 1 No Coding x 2 x 2 x 1 x 2 x 2 Coding x 1 +x 2 x 1 +x 2 Who connects to whom? How many What packets to store? How many Which packets to serve and to whom? Combinatorial problem: Hard Convex Optimization Problem: Easy

35 Distributed Cache Network: Architectural Goals Scalability: Match arbitrary demand distribution Robust to node churn/failures Dynamics (fluctuations) Easy to deploy ( plug and play ) Easy to maintain ( disposable ) Server Cloud Self-sustaining : Decentralized: don t bug the server Min. overhead of BW, storage, disk reads, complexity (field-size) Security: avoid creating new packets Desirable to avoid coding at cache layer.

36 What codes do we want at the cache layer? Desired abstraction: Like rateless fountain codes Easy to grow and shrink as needed Maintain quasi-mds property: user gets his data by contacting k (1+ε) nodes But: Distributed Not dependent on large blocklengths Uncoded Caveat: Need to be compatible with the code structure of the server layer Proposed solution: Informal DRESS Codes

37 Informal DRESS Codes M packets Reed- Solomon θ Randomly throw αpackets Outer code of server layer User needs to collect M distinct packets

38 Informal DRESS Code: User Guarantees # nodes user needs to contact: k_min=4, k_avg=5.6, k_max~9 (39,20) outer code 5 packets per node (20,12) outer code 3 packets per node Theorem :(Pawar et al. ISIT`10) Let F be the number of colors observed by a user contacting k nodes then with

39 Decentralized Growing M packets Reed- Solomon θ Minimize the number of knocks on door. Growth simulates rateless fountain code But fully distributed + the cache layer Ex.: α= New cache node

40 Tradeoff Between User and Growth : Intuition M packets Reed- Solomon θ R 0 Cache nodes Code rate R=M/θ Almost no replicas R M Many replicas User User Growth Growth

41 User vs. Growth Tradeoff M packets R=M/θ Reed- Solomon θ Average Analysis: K: # nodes contacted by user E(G) G: # nodes contacted for growth E(K)

42 CACHE nodes USERS

43

44

45

46 Idea: Use Markov Approximation with bounded gap Fully distributed algorithm: soft worst-neighbor choking

47 stay in the max- u.lity configura.on for most of the.me Idea: Use Markov Approximation with bounded gap Fully distributed algorithm: soft worst-neighbor choking sum = 650kbps 100kbps 300kbps Each user- cache node connec8on graph is called a configura8on f System u8lity under f is U f

48

49

50

51 Distributed Load-Balancing

52 Conclusion Massively scalable & reliable distributed content- delivery system Architectural principle: separation of reliability and scalability Role of codes critical: what codes to use? BW, storage, disk I/O, complexity, latency, energy, security DRESS codes: uncoded repair Server Layer: Formal DRESS Codes Deterministic constructions from Proj. Spaces & Steiner system Distributed Cache Layer: Informal DRESS Codes Randomized constructions to enable distributed digital fountain Many open problems in theory, code construction, architecture, system design,

53 Open problems Theory & code construction: Exact- Repair MDS codes for rates above ½? Info- theoretic opt. storage/bw tradeoffs of uncoded repair? DRESS codes: beyond projective planes & Steiner Systems? Fundamental multi- dimensional tradeoffs including system parameters like latency, disk reads, energy, security,? Formal connections to interference alignment in wireless systems? System design and architecture: Incentive mechanisms/economics/game- theory for DCN? Role of caching in wireless systems? Practical systems challenges and testbed deployment?

54 Vision of tomorrow s cloud server? Distributed Cache Network

55 QUESTIONS?