Modern Erasure Codes for Distributed Storage Systems
|
|
- Joel Maxwell
- 6 years ago
- Views:
Transcription
1 Modern Erasure Codes for Distributed Storage Systems Srinivasan Narayanamurthy (Srini) NetApp
2 Everything around us is changing! r The Data Deluge r Disk capacities and densities are increasing faster than the disk transfer rates r Increased delay to recover using classical techniques lead to availability exposure r Changing Storage Technologies r Architectures: Scale-out, Distributed Storage, Cloud, Converged r Media: Flash, NVM, SMR, Tape, et al. r Features: Geo-distribution, Security, Commodity hardware (Failure is a norm!) r Newer Dimensions of Erasure Codes r Optimality tradeoffs redefined r More about this inside 2
3 Organization r Background r Erasure Codes Timeline r Classical Codes - (n, k) code r Modern Codes r Locally Repairable Codes (Codes on Codes) r Regenerating Codes (Network Codes) r Technical Analysis r Optimality Tradeoff and Reliability Analysis r System Requirements and Codes r Literature & Key Players 3
4 Background Timeline Classical (n, k) codes 4
5 Timeline Overview Classical Codes Fountain Codes Locally Repairable Codes Regenerating Codes Tradeoff against Storage Overhead Reliability Performance Repair Degree Repair Bandwidth 5
6 Timeline 1950s Hamming Fountain Luby, LT Network Codes for Storage On the Locality Local Regeneration MDS Array Reed- Solomon Tornado Repairing RS THEORY RAID RAID-6 RapTor Hierarchical Optimal I/O SYSTEMS Pyramid Azure (mlrc) XORBAS (flrc) Double Replica Butterfly
7 Classical (n, k) Codes (6,4) Encode Decode Repair Chunk Encode Retrieve Decode Re-encode Replace Object / File k data blocks n encoded blocks Surviving k blocks Reconstructed data Lost blocks n encoded blocks Think distributed systems; repairs are expensive! 7
8 Modern Erasure Codes Locally Repairable Codes Regenerating codes 8
9 Locally Repairable codes Microsoft Azure (mlrc) Hierarchical (Bottom-Up) & Pyramid (Top-Down) Codes Repair Degree Locality/ Max. Recoverability Locality/Min. Dist. (k, l, r) codes : (6, 2, 2) Facebook XORBAS (flrc) RS (14,10) & optimizing single block failures 9
10 Regenerating Codes Functional Repair A B P1 = A+2B A B C D P2 = 2C+D An Information Flow Graph & Min-Cut Bound C D A+B+C+D A+2B+C+2D A+2B+3C+D P3 = 4A+5B+4C+5D 5A+7B+8C+7D 3A+2B+2C+3D 6A+9B+6C+6D 10
11 Repair By Transfer (MBR) Codes Local Regenerating Codes P P MBR No Codes Exist! P Storage Storage / Repair BW Pentagon Code Repair Bandwidth MSR 11
12 Technical Analysis Optimality Tradeoffs Reliability Analysis System Requirements 12
13 Summary of Codes and their Tradeoffs Codes/Family Tradeoff MDS Storage overhead Reliability Replication & Parity Storage overhead Reliability Reed-Solomon Storage overhead Reliability Near-Optimal Correction capability Computational Complexity Fountain Rate Probability of Correction Codes on Codes Storage overhead Repair Degree (Fan-in) Azure (mlrc) MDS Maximum Recoverability XORBAS (flrc) Locality Minimum Distance Regenerating Storage overhead Repair Bandwidth Local Regenerating Storage overhead Reconstruction Cost 13
14 Reliability Analysis replica Storage Overhead Locally Repairable Codes RS (14,10) flrc (10,6,5) MTTDL mlrc (6,2,2) Repair Traffic Regenerating Codes Storage Overhead MTTDL Code Length Locally Repairable Codes Regenerating Codes 14
15 System Requirements and Example Codes Architecture System Properties of the System Requirements for a Code Generalpurpose storage array Geo-distributed storage Secure Storage Distributed Systems Most Important Reliability & Performance Repair over WAN is expensive Security Parallelism & Availability Least Important Most Important Least Important Cost Reliability Complexity Storage overhead across DR sites Storage overhead Storage overhead Local repair Faster degraded reads Systematic Storage overhead Repair time Storage overhead Example family/code MSR, SD/STAIR Codes LRC Nonsystematic codes; MBR Replication Workload Big Data (say, Hadoop) Large volumes of data Write latency Storage overhead Repair bandwidth Regenerating (MSR/MBR), systematic 15
16 Literature & Key Players Theory & Systems 16
17 Researchers, Big Players & Startups UT Austin Alex Dimakis Network Codes for Storage (2010) UC Berkeley Kannan R PM (MSR) RBT ( ) MSR/MBR Points (2013) IISc Vijay Kumar Pyramid (2007) Parikshit Gopalan Locality (2011) MIT Muriel Medard NTU Oggier XORBAS (2013) Greenan Piggyback (2013) Hitchhiker (2014) Flat XOR (2010) U Tennessee James Plank PM (RBT) (2015) SD-Codes (2013) PMDS (2013) Double Rep (2014) Chinese U HongKongP atrick PC Lee GF Intel SIMD (2013) GF-Complete STAIR Jerasure (2013) (2014) (2014) mlrc (2012) Network Flow & Linear Coding RLNC RS, Fountain Fountain Self-repairing (2011) THEORY Non-systematic RS SYSTEMS RS 17
18 Related Areas r Cross-object Coding r Sector & Disk failures PMDS, SD, STAIR Codes r Other media r Flash: LDPC, WOM, Multi-write codes; NVM r Security r Dispersal, AONT-RS r Cloud r NC-Cloud r Transformational Codes: r Transform encoded data to different parameters as they become hot/cold without decoding and re-encoding 18
19 Schrodinger s Code The condition of any system is unknown until a repair is complete. 19
Modern Erasure Codes for Distributed Storage Systems
Modern Erasure Codes for Distributed Storage Systems Storage Developer Conference, SNIA, Bangalore Srinivasan Narayanamurthy Advanced Technology Group, NetApp May 27 th 2016 1 Everything around us is changing!
More informationYour Data is in the Cloud: Who Exactly is Looking After It?
Your Data is in the Cloud: Who Exactly is Looking After It? P Vijay Kumar Dept of Electrical Communication Engineering Indian Institute of Science IISc Open Day March 4, 2017 1/33 Your Data is in the Cloud:
More informationFast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration Techniques. Tianli Zhou & Chao Tian Texas A&M University
Fast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration Techniques Tianli Zhou & Chao Tian Texas A&M University 2 Contents Motivation Background and Review Evaluating Individual
More informationScreaming Fast Galois Field Arithmetic Using Intel SIMD Instructions. James S. Plank USENIX FAST. University of Tennessee
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions James S. Plank University of Tennessee USENIX FAST San Jose, CA February 15, 2013. Authors Jim Plank Tennessee Kevin Greenan EMC/Data
More informationNetwork Coding for Distributed Storage Systems* Presented by Jayant Apte ASPITRG 7/9/13 & 7/11/13
Network Coding for Distributed Storage Systems* Presented by Jayant Apte ASPITRG 7/9/13 & 7/11/13 *Dimakis, A.G.; Godfrey, P.B.; Wu, Y.; Wainwright, M.J.; Ramchandran, K. "Network Coding for Distributed
More informationAll About Erasure Codes: - Reed-Solomon Coding - LDPC Coding. James S. Plank. ICL - August 20, 2004
All About Erasure Codes: - Reed-Solomon Coding - LDPC Coding James S. Plank Logistical Computing and Internetworking Laboratory Department of Computer Science University of Tennessee ICL - August 2, 24
More informationPerformance Models of Access Latency in Cloud Storage Systems
Performance Models of Access Latency in Cloud Storage Systems Qiqi Shuai Email: qqshuai@eee.hku.hk Victor O.K. Li, Fellow, IEEE Email: vli@eee.hku.hk Yixuan Zhu Email: yxzhu@eee.hku.hk Abstract Access
More informationA survey on regenerating codes
International Journal of Scientific and Research Publications, Volume 4, Issue 11, November 2014 1 A survey on regenerating codes V. Anto Vins *, S.Umamageswari **, P.Saranya ** * P.G Scholar, Department
More informationLocally repairable codes
Locally repairable codes for large-scale storage Presented by Anwitaman Datta Nanyang Technological University, Singapore Joint work with Frédérique Oggier & Lluís Pàmies i Juárez @ IISc 2012 & NetApp
More informationNetwork Codes for Next-Generation Distributed Storage:! Opportunities & Challenges. Kannan Ramchandran
Network Codes for Next-Generation Distributed Storage:! Opportunities & Challenges Kannan Ramchandran University of California, Berkeley Joint work with: Salim El Rouayheb (UCB), Hao Zhang (UCB), Sameer
More informationCloud Storage Reliability for Big Data Applications: A State of the Art Survey
Cloud Storage Reliability for Big Data Applications: A State of the Art Survey Rekha Nachiappan a, Bahman Javadi a, Rodrigo Calherios a, Kenan Matawie a a School of Computing, Engineering and Mathematics,
More informationEnabling Node Repair in Any Erasure Code for Distributed Storage
Enabling Node Repair in Any Erasure Code for Distributed Storage K. V. Rashmi, Nihar B. Shah, and P. Vijay Kumar, Fellow, IEEE Abstract Erasure codes are an efficient means of storing data across a network
More informationCodes for Modern Applications
Codes for Modern Applications Current Research Topics in the Code and Signal Design Group P. Vijay Kumar Indian Institute of Science, Bangalore, India Nov. 16, 2018 1/28 Codes for Modern Applications:
More informationHADOOP 3.0 is here! Dr. Sandeep Deshmukh Sadepach Labs Pvt. Ltd. - Let us grow together!
HADOOP 3.0 is here! Dr. Sandeep Deshmukh sandeep@sadepach.com Sadepach Labs Pvt. Ltd. - Let us grow together! About me BE from VNIT Nagpur, MTech+PhD from IIT Bombay Worked with Persistent Systems - Life
More informationScreaming Fast Galois Field Arithmetic Using Intel SIMD Instructions
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions Ethan L. Miller Center for Research in Storage Systems University of California, Santa Cruz (and Pure Storage) Authors Jim Plank Univ.
More informationEC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures
EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures Haiyang Shi, Xiaoyi Lu, and Dhabaleswar K. (DK) Panda {shi.876, lu.932, panda.2}@osu.edu The Ohio State University
More informationPrivacy, Security, and Repair in Distributed Storage Systems. Siddhartha Kumar
Privacy, Security, and Repair in Distributed Storage Systems Siddhartha Kumar Thesis for the Degree of Philosophiae Doctor (PhD) University of Bergen, Norway 2018 Privacy, Security, and Repair in Distributed
More informationEECS 121: Coding for Digital Communication & Beyond Fall Lecture 22 December 3. Introduction
EECS 121: Coding for Digital Communication & Beyond Fall 2013 Lecture 22 December 3 Lecturer: K. V. Rashmi Scribe: Ajay Shanker Tripathi 22.1 Context Introduction Distributed storage is a deeply relevant
More informationErasure Codes for Heterogeneous Networked Storage Systems
Erasure Codes for Heterogeneous Networked Storage Systems Lluís Pàmies i Juárez Lluís Pàmies i Juárez lpjuarez@ntu.edu.sg . Introduction Outline 2. Distributed Storage Allocation Problem 3. Homogeneous
More informationFinding the most fault-tolerant flat XOR-based erasure codes for storage systems Jay J. Wylie
Finding the most fault-tolerant flat XOR-based erasure codes for storage systems Jay J. Wylie HP Laboratories HPL-2011-218 Abstract: We describe the techniques we developed to efficiently find the most
More informationOn Coding Techniques for Networked Distributed Storage Systems
On Coding Techniques for Networked Distributed Storage Systems Frédérique Oggier frederique@ntu.edu.sg Nanyang Technological University, Singapore First European Training School on Network Coding, Barcelona,
More informationRepair Pipelining for Erasure-Coded Storage
Repair Pipelining for Erasure-Coded Storage Runhui Li, Xiaolu Li, Patrick P. C. Lee, Qun Huang The Chinese University of Hong Kong USENIX ATC 2017 1 Introduction Fault tolerance for distributed storage
More informationCapacity Assurance in Hostile Networks
PhD Dissertation Defense Wednesday, October 7, 2015 3:30 pm - 5:30 pm 3112 Engineering Building Capacity Assurance in Hostile Networks By: Jian Li Advisor: Jian Ren ABSTRACT Linear network coding provides
More informationCoding Techniques for Distributed Storage Systems
Coding Techniques for Distributed Storage Systems Jung-Hyun Kim, {jh.kim06}@yonsei.ac.kr, Hong-Yeop Song Yonsei Univ. Seoul, KORE 3 rd CITW, Oct. 25. 1, 2013 Contents Introduction Codes for Distributed
More informationA Comparative Study on Distributed Storage and Erasure Coding Techniques Using Apache Hadoop Over NorNet Core
Master Thesis University of Bergen Department of Informatics A Comparative Study on Distributed Storage and Erasure Coding Techniques Using Apache Hadoop Over NorNet Core Author: Maximiliano Vela Supervisor:
More informationOptimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 57, NO 8, AUGUST 2011 5227 Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction K V Rashmi,
More informationErasure coding and AONT algorithm selection for Secure Distributed Storage. Alem Abreha Sowmya Shetty
Erasure coding and AONT algorithm selection for Secure Distributed Storage Alem Abreha Sowmya Shetty Secure Distributed Storage AONT(All-Or-Nothing Transform) unkeyed transformation φ mapping a sequence
More informationClay Codes: Moulding MDS Codes to Yield an MSR Code
Clay Codes: Moulding MDS Codes to Yield an MSR Code Myna Vajha, Vinayak Ramkumar, Bhagyashree Puranik, Ganesh Kini, Elita Lobo, Birenjith Sasidharan, and P. Vijay Kumar, Indian Institute of Science, Bangalore;
More informationSCALING UP OF E-MSR CODES BASED DISTRIBUTED STORAGE SYSTEMS WITH FIXED NUMBER OF REDUNDANCY NODES
SCALING UP OF E-MSR CODES BASED DISTRIBUTED STORAGE SYSTEMS WITH FIXED NUMBER OF REDUNDANCY NODES Haotian Zhao, Yinlong Xu and Liping Xiang School of Computer Science and Technology, University of Science
More informationImproving Cloud Storage Cost and Data Resiliency with Erasure Codes Michael Penick
Improving Cloud Storage Cost and Data Resiliency with Erasure Codes Michael Penick Commodity Storage Hosting storage FTP backup Goals Inexpensive (use commodity hardware) Resilient to failures Highly available
More informationCSC310 Information Theory. Lecture 21: Erasure (Deletion) Channels & Digital Fountain Codes. November 22, 2006 see
CSC310 Information Theory Lecture 21: Erasure (Deletion) Channels & Digital Fountain Codes Sam Roweis Recovering From Erasures 2 How can we recover from erasures? For the BEC, we saw that low-density parity
More informationData Integrity Protection scheme for Minimum Storage Regenerating Codes in Multiple Cloud Environment
International Journal of Advanced Research in Computer Engineering & Technology (IJARCET) Data Integrity Protection scheme for Minimum Storage Regenerating Codes in Multiple Cloud Environment Saravana
More informationCSC310 Information Theory. Lecture 22: Erasure (Deletion) Channels & Digital Fountain Codes. November 30, 2005 see
CSC310 Information Theory Lecture 22: Erasure (Deletion) Channels & Digital Fountain Codes Sam Roweis Recovering From Erasures 2 How can we recover from erasures? For the BEC, we saw that low-density parity
More informationRethinking Erasure Codes for Cloud File Systems: Minimizing I/O for Recovery and Degraded Reads
Rethinking Erasure Codes for Cloud File Systems: Minimizing I/O for Recovery and Degraded Reads Osama Khan, Randal Burns, James S. Plank, William Pierce and Cheng Huang FAST 2012: 10 th USENIX Conference
More informationA Survey on Erasure Coding Techniques for Cloud Storage System
A Survey on Erasure Coding Techniques for Cloud Storage System Kiran P. Pawar #1, R. M. Jogdand *2 #1 M.Tech, Department of Computer Science, Gogte Institute of technology, Udyambag Belagavi, Karnataka,
More informationA RANDOMLY EXPANDABLE METHOD FOR DATA LAYOUT OF RAID STORAGE SYSTEMS. Received October 2017; revised February 2018
International Journal of Innovative Computing, Information and Control ICIC International c 2018 ISSN 1349-4198 Volume 14, Number 3, June 2018 pp. 1079 1094 A RANDOMLY EXPANDABLE METHOD FOR DATA LAYOUT
More informationDiskReduce: Making Room for More Data on DISCs. Wittawat Tantisiriroj
DiskReduce: Making Room for More Data on DISCs Wittawat Tantisiriroj Lin Xiao, Bin Fan, and Garth Gibson PARALLEL DATA LABORATORY Carnegie Mellon University GFS/HDFS Triplication GFS & HDFS triplicate
More informationEverything You Wanted To Know About Storage (But Were Too Proud To Ask) The Basics
Everything You Wanted To Know About Storage (But Were Too Proud To Ask) The Basics Today s Presenters Bob Plumridge HDS Chief Technology Officer - EMEA Alex McDonald NetApp CTO Office 2 SNIA Legal Notice
More informationCoding theory for scalable media delivery
1 Coding theory for scalable media delivery Michael Luby RaptorQ is a product of Qualcomm Technologies, Inc. Application layer erasure coding complements traditional error coding Forward Error Correction
More informationOn the Speedup of Recovery in Large-Scale Erasure-Coded Storage Systems (Supplementary File)
1 On the Speedup of Recovery in Large-Scale Erasure-Coded Storage Systems (Supplementary File) Yunfeng Zhu, Patrick P. C. Lee, Yinlong Xu, Yuchong Hu, and Liping Xiang 1 ADDITIONAL RELATED WORK Our work
More informationPANASAS TIERED PARITY ARCHITECTURE
PANASAS TIERED PARITY ARCHITECTURE Larry Jones, Matt Reid, Marc Unangst, Garth Gibson, and Brent Welch White Paper May 2010 Abstract Disk drives are approximately 250 times denser today than a decade ago.
More informationSoftware-defined Storage: Fast, Safe and Efficient
Software-defined Storage: Fast, Safe and Efficient TRY NOW Thanks to Blockchain and Intel Intelligent Storage Acceleration Library Every piece of data is required to be stored somewhere. We all know about
More informationCooperative Pipelined Regeneration in Distributed Storage Systems
Cooperative ipelined Regeneration in Distributed Storage Systems Jun Li, in Wang School of Computer Science Fudan University, China Baochun Li Department of Electrical and Computer Engineering University
More informationFountain Codes Based on Zigzag Decodable Coding
Fountain Codes Based on Zigzag Decodable Coding Takayuki Nozaki Kanagawa University, JAPAN Email: nozaki@kanagawa-u.ac.jp Abstract Fountain codes based on non-binary low-density parity-check (LDPC) codes
More informationStorage systems have grown to the point where failures are inevitable,
A Brief Primer JAMES S. P LANK James S. Plank received his BS from Yale University in 1988 and his PhD from Princeton University in 1993. He is a professor in the Electrical Engineering and Computer Science
More informationOn Data Parallelism of Erasure Coding in Distributed Storage Systems
On Data Parallelism of Erasure Coding in Distributed Storage Systems Jun Li, Baochun Li Department of Electrical and Computer Engineering, University of Toronto, Canada {junli, bli}@ece.toronto.edu Abstract
More informationLecture 4 September 17
CS 559: Algorithmic Aspects of Computer Networks Fall 2007 Lecture 4 September 17 Lecturer: John Byers BOSTON UNIVERSITY Scribe: Sowmya Manjanatha In today s lecture, we discussed and characterized the
More informationCORE: Augmenting Regenerating-Coding-Based Recovery for Single and Concurrent Failures in Distributed Storage Systems
CORE: Augmenting Regenerating-Coding-Based Recovery for Single and Concurrent Failures in Distributed Storage Systems Runhui Li, Jian Lin, Patrick P. C. Lee Department of Computer Science and Engineering,
More informationA Performance Evaluation of Open Source Erasure Codes for Storage Applications
A Performance Evaluation of Open Source Erasure Codes for Storage Applications James S. Plank Catherine D. Schuman (Tennessee) Jianqiang Luo Lihao Xu (Wayne State) Zooko Wilcox-O'Hearn Usenix FAST February
More informationInformation-theoretically Secure Regenerating Codes for Distributed Storage
Information-theoretically Secure Regenerating Codes for Distributed Storage Nihar B. Shah, K. V. Rashmi and P. Vijay Kumar Abstract Regenerating codes are a class of codes for distributed storage networks
More informationOptimize Storage Efficiency & Performance with Erasure Coding Hardware Offload. Dror Goldenberg VP Software Architecture Mellanox Technologies
Optimize Storage Efficiency & Performance with Erasure Coding Hardware Offload Dror Goldenberg VP Software Architecture Mellanox Technologies SNIA Legal Notice The material contained in this tutorial is
More informationRaptor Codes for P2P Streaming
Raptor Codes for P2P Streaming Philipp Eittenberger 1, Todor Mladenov 2, Udo Krieger 1 1 Faculty of Information Systems and Applied Computer Science Otto-Friedrich University Bamberg, Germany 2 Department
More informationECS High Availability Design
ECS High Availability Design March 2018 A Dell EMC white paper Revisions Date Mar 2018 Aug 2017 July 2017 Description Version 1.2 - Updated to include ECS version 3.2 content Version 1.1 - Updated to include
More informationActiveScale Erasure Coding and Self Protecting Technologies
WHITE PAPER AUGUST 2018 ActiveScale Erasure Coding and Self Protecting Technologies BitSpread Erasure Coding and BitDynamics Data Integrity and Repair Technologies within The ActiveScale Object Storage
More informationRegenerating Codes for Errors and Erasures in Distributed Storage
Regenerating Codes or Errors and Erasures in Distributed Storage K V Rashmi, Nihar B Shah, Kannan Ramchandran, Fellow, IEEE, and P Vijay Kumar, Fellow, IEEE arxiv:0050v csit] 3 May 0 Abstract Regenerating
More informationEncoding-Aware Data Placement for Efficient Degraded Reads in XOR-Coded Storage Systems
Encoding-Aware Data Placement for Efficient Degraded Reads in XOR-Coded Storage Systems Zhirong Shen, Patrick P. C. Lee, Jiwu Shu, Wenzhong Guo College of Mathematics and Computer Science, Fuzhou University
More informationSo you think you can FEC?
So you think you can FEC? Mikhail Misha Fludkov (fludkov) Pexip R&D Erlend Earl Graff (egraff) Pexip R&D 8 th GStreamer Conference 21 October 2017 Prague, Czech Republic Outline Introduction to FEC Designing
More informationScreaming Fast Galois Field Arithmetic Using Intel SIMD Instructions
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions James S. Plank EECS Department University of Tennessee plank@cs.utk.edu Kevin M. Greenan EMC Backup Recovery Systems Division Kevin.Greenan@emc.com
More informationDistributed File System Based on Erasure Coding for I/O-Intensive Applications
Distributed File System Based on Erasure Coding for I/O-Intensive Applications Dimitri Pertin 1,, Sylvain David, Pierre Évenou, Benoît Parrein 1 and Nicolas Normand 1 1 LUNAM Université, Université de
More informationCSEP 561 Error detection & correction. David Wetherall
CSEP 561 Error detection & correction David Wetherall djw@cs.washington.edu Codes for Error Detection/Correction ti ti Error detection and correction How do we detect and correct messages that are garbled
More informationDisk Arrays Nov. 7, 2011!
15-410...Failure is not an option...! Disk Arrays Nov. 7, 2011! Garth Gibson! Dave Eckhardt!! Contributions by! Michael Ashley-Rollman! 1 L25_RAID Overview! Historical practices! Striping! Mirroring! The
More informationBCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding. Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University
BCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University Outline Introduction and Motivation Our Design System and Implementation
More informationDecentralized Distributed Storage System for Big Data
Decentralized Distributed Storage System for Big Presenter: Wei Xie -Intensive Scalable Computing Laboratory(DISCL) Computer Science Department Texas Tech University Outline Trends in Big and Cloud Storage
More informationOn the Speedup of Single-Disk Failure Recovery in XOR-Coded Storage Systems: Theory and Practice
On the Speedup of Single-Disk Failure Recovery in XOR-Coded Storage Systems: Theory and Practice Yunfeng Zhu, Patrick P. C. Lee, Yuchong Hu, Liping Xiang, and Yinlong Xu University of Science and Technology
More informationShort Code: An Efficient RAID-6 MDS Code for Optimizing Degraded Reads and Partial Stripe Writes
: An Efficient RAID-6 MDS Code for Optimizing Degraded Reads and Partial Stripe Writes Yingxun Fu, Jiwu Shu, Xianghong Luo, Zhirong Shen, and Qingda Hu Abstract As reliability requirements are increasingly
More informationActiveScale Erasure Coding and Self Protecting Technologies
NOVEMBER 2017 ActiveScale Erasure Coding and Self Protecting Technologies BitSpread Erasure Coding and BitDynamics Data Integrity and Repair Technologies within The ActiveScale Object Storage System Software
More informationRAID (Redundant Array of Inexpensive Disks)
Magnetic Disk Characteristics I/O Connection Structure Types of Buses Cache & I/O I/O Performance Metrics I/O System Modeling Using Queuing Theory Designing an I/O System RAID (Redundant Array of Inexpensive
More informationInterference Alignment in Regenerating Codes for Distributed Storage: Necessity and Code Constructions
2134 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 58, NO 4, APRIL 2012 Interference Alignment in Regenerating Codes for Distributed Storage: Necessity and Code Constructions Nihar B Shah, K V Rashmi, P
More informationFPGA Implementation of Erasure Codes in NVMe based JBOFs
FPGA Implementation of Erasure Codes in NVMe based JBOFs Manoj Roge Director, Data Center Xilinx Inc. Santa Clara, CA 1 Acknowledgement Shre Shah Data Center Architect, Xilinx Santa Clara, CA 2 Agenda
More informationNetworked Systems and Services, Fall 2018 Chapter 2. Jussi Kangasharju Markku Kojo Lea Kutvonen
Networked Systems and Services, Fall 2018 Chapter 2 Jussi Kangasharju Markku Kojo Lea Kutvonen Outline Physical layer reliability Low level reliability Parities and checksums Cyclic Redundancy Check (CRC)
More informationHP AutoRAID (Lecture 5, cs262a)
HP AutoRAID (Lecture 5, cs262a) Ion Stoica, UC Berkeley September 13, 2016 (based on presentation from John Kubiatowicz, UC Berkeley) Array Reliability Reliability of N disks = Reliability of 1 Disk N
More informationDiskReduce: Making Room for More Data on DISCs. Wittawat Tantisiriroj
DiskReduce: Making Room for More Data on DISCs Wittawat Tantisiriroj Lin Xiao, in Fan, and Garth Gibson PARALLEL DATA LAORATORY Carnegie Mellon University GFS/HDFS Triplication GFS & HDFS triplicate every
More informationOnline data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland
Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Abstract. The Data and Storage Services group at CERN is conducting
More informationApache Hadoop 3. Balazs Gaspar Sales Engineer CEE & CIS Cloudera, Inc. All rights reserved.
Apache Hadoop 3 Balazs Gaspar Sales Engineer CEE & CIS balazs@cloudera.com 1 We believe data can make what is impossible today, possible tomorrow 2 We empower people to transform complex data into clear
More informationPacket-Level Forward Error Correction in Video Transmission
Packet-Level Forward Error Correction in Video Transmission Matteo Mazzotti, Enrico Paolini, Marco Chiani, Davide Dardari, and Andrea Giorgetti University of Bologna Wireless Communications Laboratory
More informationBuilding High Speed Erasure Coding Libraries for ARM and x86 Processors. Per Simonsen, CEO, MemoScale May 2017
Building High Speed Erasure Coding Libraries for ARM and x86 Processors Per Simonsen, CEO, MemoScale May 2017 Agenda MemoScale company and team Erasure coding - brief intro MemoScale erasure codes Performance
More informationExploration of Erasure-Coded Storage Systems for High Performance, Reliability, and Inter-operability
Virginia Commonwealth University VCU Scholars Compass Theses and Dissertations Graduate School 2016 Exploration of Erasure-Coded Storage Systems for High Performance, Reliability, and Inter-operability
More informationEC-Cache: Load-balanced, Low-latency Cluster Caching with Online Erasure Coding
EC-Cache: Load-balanced, Low-latency Cluster Caching with Online Erasure Coding Rashmi Vinayak UC Berkeley Joint work with Mosharaf Chowdhury, Jack Kosaian (U Michigan) Ion Stoica, Kannan Ramchandran (UC
More informationPCM: A Parity-check Matrix Based Approach to Improve Decoding Performance of XOR-based Erasure Codes
15 IEEE th Symposium on Reliable Distributed Systems : A Parity-check Matrix Based Approach to Improve Decoding Performance of XOR-based Erasure Codes Yongzhe Zhang, Chentao Wu, Jie Li, Minyi Guo Shanghai
More informationResume Maintaining System for Referral Using Cloud Computing
Resume Maintaining System for Referral Using Cloud Computing Pranjali A. Pali 1, N.D. Kale 2 P.G. Student, Department of Computer Engineering, TSSM S PVPIT, Bavdhan, Pune, Maharashtra India 1 Associate
More informationParity Logging with Reserved Space: Towards Efficient Updates and Recovery in Erasure-coded Clustered Storage
Parity Logging with Reserved Space: Towards Efficient Updates and Recovery in Erasure-coded Clustered Storage Jeremy C. W. Chan, Qian Ding, Patrick P. C. Lee, and Helen H. W. Chan, The Chinese University
More informationEE 6900: FAULT-TOLERANT COMPUTING SYSTEMS
EE 6900: FAULT-TOLERANT COMPUTING SYSTEMS LECTURE 6: CODING THEORY - 2 Fall 2014 Avinash Kodi kodi@ohio.edu Acknowledgement: Daniel Sorin, Behrooz Parhami, Srinivasan Ramasubramanian Agenda Hamming Codes
More informationA Simulation Analysis of Reliability in Erasure-Coded Data Centers
A Simulation Analysis of Reliability in Erasure-Coded Data Centers Mi Zhang, Shujie Han, and Patrick P. C. Lee Department of Computer Science and Engineering, The Chinese University of Hong Kong {mzhang,sjhan,pclee}@cse.cuhk.edu.hk
More informationError Control Coding for MLC Flash Memories
Error Control Coding for MLC Flash Memories Ying Y. Tai, Ph.D. Cadence Design Systems, Inc. ytai@cadence.com August 19, 2010 Santa Clara, CA 1 Outline The Challenges on Error Control Coding (ECC) for MLC
More informationApplied Erasure Coding in Networks and Distributed Storage
arxiv:1803.01358v1 [cs.it] 4 Mar 2018 Katina Kralevska Applied Erasure Coding in Networks and Distributed Storage Thesis for the degree of Philosophiae Doctor Trondheim, March 2018 Department of Telematics,
More informationMODERN storage systems, such as GFS [1], Windows
IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 1, JANUARY 2017 127 Short Code: An Efficient RAID-6 MDS Code for Optimizing Degraded Reads and Partial Stripe Writes Yingxun Fu, Jiwu Shu, Senior Member, IEEE,
More informationLec 19 - Error and Loss Control
ECE 5578 Multimedia Communication Lec 19 - Error and Loss Control Zhu Li Dept of CSEE, UMKC Office: FH560E, Email: lizhu@umkc.edu, Ph: x 2346. http://l.web.umkc.edu/lizhu slides created with WPS Office
More informationCOS 318: Operating Systems. Storage Devices. Vivek Pai Computer Science Department Princeton University
COS 318: Operating Systems Storage Devices Vivek Pai Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall11/cos318/ Today s Topics Magnetic disks Magnetic disk
More informationAn Improvement of Quasi-cyclic Minimum Storage Regenerating Codes for Distributed Storage
An Improvement of Quasi-cyclic Minimum Storage Regenerating Codes for Distributed Storage Chenhui LI*, Songtao LIANG* * Shanghai Key Laboratory of Intelligent Information Processing, Fudan University,
More informationApplied Erasure Coding in Networks and Distributed Storage
Doctoral thesis Katina Kralevska Doctoral theses at NTNU, 2016:341 NTNU Norwegian University of Science and Technology Thesis for the Degree of Philosophiae Doctor Faculty of Information Technology, Mathematics
More informationCSCI-1680 Link Layer Reliability Rodrigo Fonseca
CSCI-1680 Link Layer Reliability Rodrigo Fonseca Based partly on lecture notes by David Mazières, Phil Levis, John Janno< Last time Physical layer: encoding, modulation Link layer framing Today Getting
More informationUnderstanding System Characteristics of Online Erasure Coding on Scalable, Distributed and Large-Scale SSD Array Systems
Understanding System Characteristics of Online Erasure Coding on Scalable, Distributed and Large-Scale SSD Array Systems arxiv:179.5365v2 [cs.dc] 19 Sep 217 Sungjoon Koh, Jie Zhang, Miryeong Kwon, Jungyeon
More informationEfficient Load Balancing and Disk Failure Avoidance Approach Using Restful Web Services
Efficient Load Balancing and Disk Failure Avoidance Approach Using Restful Web Services Neha Shiraz, Dr. Parikshit N. Mahalle Persuing M.E, Department of Computer Engineering, Smt. Kashibai Navale College
More informationDistributed File System based on Erasure Coding for I/O Intensive Applications
Distributed File System based on Erasure Coding for I/O Intensive Applications Dimitri Pertin, Sylvain David, Pierre Evenou, Benoît Parrein, Nicolas Normand To cite this version: Dimitri Pertin, Sylvain
More informationDell Technologies IoT Solution Surveillance with Genetec Security Center
Dell Technologies IoT Solution Surveillance with Genetec Security Center Surveillance December 2018 H17436 Sizing Guide Abstract The purpose of this guide is to help you understand the benefits of using
More informationDiskReduce: RAID for Data-Intensive Scalable Computing (CMU-PDL )
Research Showcase @ CMU Parallel Data Laboratory Research Centers and Institutes 11-2009 DiskReduce: RAID for Data-Intensive Scalable Computing (CMU-PDL-09-112) Bin Fan Wittawat Tantisiriroj Lin Xiao Garth
More informationRAID. Redundant Array of Inexpensive Disks. Industry tends to use Independent Disks
RAID Chapter 5 1 RAID Redundant Array of Inexpensive Disks Industry tends to use Independent Disks Idea: Use multiple disks to parallelise Disk I/O for better performance Use multiple redundant disks for
More informationData Protection for Cisco HyperFlex with Veeam Availability Suite. Solution Overview Cisco Public
Data Protection for Cisco HyperFlex with Veeam Availability Suite 1 2017 2017 Cisco Cisco and/or and/or its affiliates. its affiliates. All rights All rights reserved. reserved. Highlights Is Cisco compatible
More informationGraph based codes for distributed storage systems
/23 Graph based codes for distributed storage systems July 2, 25 Christine Kelley University of Nebraska-Lincoln Joint work with Allison Beemer and Carolyn Mayer Combinatorics and Computer Algebra, COCOA
More informationELEC 691X/498X Broadcast Signal Transmission Winter 2018
ELEC 691X/498X Broadcast Signal Transmission Winter 2018 Instructor: DR. Reza Soleymani, Office: EV 5.125, Telephone: 848 2424 ext.: 4103. Office Hours: Wednesday, Thursday, 14:00 15:00 Slide 1 In this
More information