Modern Erasure Codes for Distributed Storage Systems
|
|
|
- Joleen Hodge
- 6 years ago
- Views:
Transcription
1 Modern Erasure Codes for Distributed Storage Systems Storage Developer Conference, SNIA, Bangalore Srinivasan Narayanamurthy Advanced Technology Group, NetApp May 27 th
2 Everything around us is changing! The Data Deluge Disk capacities and densities are increasing faster than the disk transfer rates Increased delay to recover using classical techniques lead to availability exposure Changing Storage Technologies Architectures: Scale-out, Distributed Storage, Cloud, Converged Media: Flash, NVM, SMR, Tape, et al. Features: Geo-distribution, Security, Use of commodity hardware (Failure is a norm!) Newer Dimensions of Erasure Codes Optimality tradeoffs redefined More about this inside 2 Erasure coding usage is growing, and is now available in an increasing number of newer object, file and block storage arrays, but not in traditional general purpose disk arrays. Gartner
3 Organization Background Erasure Codes Timeline Classical Codes - (n, k) code Modern Codes Codes on Codes Network Codes Technical Analysis Optimality Tradeoff and Reliability Analysis System Requirements and Codes Literature & Key Players 3
4 Background Timeline Classical (n, k) codes 4
5 Timeline Overview Classical Codes Fountain Codes Codes on Codes Network Codes Tradeoff against Storage Overhead Reliability Performance Repair Degree Repair Bandwidth 5
6 Erasure Codes Timeline 1950s Hamming Reed- Solomon Fountain Luby, LT Tornado Network Codes for Storage On the Locality Local Regeneration THEORY SYSTEMS RAID RAID-6 RapTor Pyramid Hierarchical Azure (mlrc) XORBAS (flrc) Double Replica I/O, BW Storage (FAST 15)
7 Classical Codes (n, k) code; Illustration (6, 4) code Chunk Encode Retrieve Decode Re-encode Replace Object / File k data blocks n encoded blocks Surviving k blocks Reconstructed data Lost blocks n encoded blocks Encode Decode Repair Think distributed systems; repairs are expensive! 7
8 Modern Codes Codes on Codes Network Codes 8
9 Codes On Codes (n 1, k 1 ) + (n 2, k 2 ) & (k, l, r) codes Hierarchical & Pyramid Codes Azure (mlrc) Locality/ Max. Recoverability Repair Degree k=6 data fragments, l=2 local parities and r=2 global parities Decoding 3 and 4 failures in mlrc RS (14,10) Hierarchical Bottom Up Pyramid Top Down Facebook (XORBAS) Single block failures Locality/ Min. Distance 9
10 Regenerating Codes Inspired by Network Codes A B P1 = A + 2B A B C D C D A + B + C + D A+ 2B + C + 2D P2 = 2C + D P3 = 4A + 5B + 4C + 5D An Information Flow Graph & Min-Cut Bound A + 2B + 3C + D 3A + 2B + 2C + 3D Functional Repair 5A + 7B + 8C + 7D 6A + 9B + 6C + 6D 10
11 Regenerating Codes Repair By Transfer (RBT), MBR Code Local Regenerating Code P MBR P P Storage No Codes Exist! Storage / Repair BW 11 Pentagon Code Repair Bandwidth MSR
12 Technical Analysis Optimality Tradeoffs Reliability Analysis System Requirements 12
13 Summary of Codes and their Tradeoff Code/Family Tradeoff MDS Storage overhead Reliability Replication & Parity (RAID) Storage overhead Reliability Reed-Solomon Storage overhead Reliability Near-Optimal Correction capabilities Computational Complexity Fountain Rate Probability of Correction Codes on Codes Storage overhead Repair Degree (Fan-in) Azure (mlrc) MDS Maximum Recoverability XORBAS (flrc) Locality Minimum Distance Regenerating Storage overhead Repair Bandwidth Local Regenerating Storage overhead Reconstruction Cost 13
14 Reliability Analysis MTTDL is in the order of E+12 (for LRC) and E+09 (for Regenerating) Locally Repairable Codes Regenerating Codes replica RS (14,10) flrc mlrc 3-replica RS (14,10) flrc (10,6,5) mlrc (6,2,2) (6,2,2) 0 Storage Overhead MTTDL Repair Traffic Storage Overhead MTTDL Code Length 14
15 System Requirements & Example Codes Architecture System Properties of the System Requirements for a Code General-purpose storage array Geo-distributed storage Secure Storage Distributed Systems Most Important Least Important Most Important Least Important Reliability & Performance Repair over WAN is expensive Security Parallelism & Availability Cost Reliability Complexity Storage overhead across DR sites Storage overhead Storage overhead Local repair Faster degraded reads Systematic Storage overhead Repair time Storage overhead Example family/ code MSR, SD/STAIR Codes LRC Non-systematic codes; MBR Replication Workload Big Data (say, Hadoop) Large volumes of data Write latency Storage overhead Repair bandwidth Regenerating (MSR/MBR), systematic 15
16 Literature & Key Players Theory & Systems 16
17 Literature & Key Players The Researchers, The Big Companies & The Startups! MSR/MBR Points (2013) UT Austin Alex Dimakis XORBAS (2013) Greenan Network Codes for Storage (2010) UC Berkeley Kannan R Piggyback (2013) Hitchhiker (2014) GF Intel SIMD (2013) Flat XOR (2010) Jerasure (2014) U Tennessee James Plank PM (MSR) RBT ( ) PM (RBT) (2015) SD-Codes (2013) GF-Complete (2013) PMDS (2013) IISc Vijay Kumar STAIR (2014) Pyramid (2007) Double Rep (2014) Chinese U HongKong Patrick PC Lee Parikshit Gopalan mlrc (2012) Locality (2011) MIT Muriel Medard Network Flow & Linear Coding RLNC RS, Fountain Fountain NTU Oggier Self-repairing (2011) Non-systematic RS THEORY SYSTEMS RS 17
18 Other Relevant Areas Cross-object Coding Sector & Disk failures PMDS, SD, STAIR Codes Other media: Flash: LDPC, WOM, Multi-write codes; NVM Security Dispersal, AONT-RS Cloud NC-Cloud Transformational Codes: Transform encoded data to different parameters as they become hot/cold without decoding and re-encoding 18
19 Thank you. 19
Modern Erasure Codes for Distributed Storage Systems
Modern Erasure Codes for Distributed Storage Systems Srinivasan Narayanamurthy (Srini) NetApp Everything around us is changing! r The Data Deluge r Disk capacities and densities are increasing faster than
Your Data is in the Cloud: Who Exactly is Looking After It?
Your Data is in the Cloud: Who Exactly is Looking After It? P Vijay Kumar Dept of Electrical Communication Engineering Indian Institute of Science IISc Open Day March 4, 2017 1/33 Your Data is in the Cloud:
Locally repairable codes
Locally repairable codes for large-scale storage Presented by Anwitaman Datta Nanyang Technological University, Singapore Joint work with Frédérique Oggier & Lluís Pàmies i Juárez @ IISc 2012 & NetApp
Fast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration Techniques. Tianli Zhou & Chao Tian Texas A&M University
Fast Erasure Coding for Data Storage: A Comprehensive Study of the Acceleration Techniques Tianli Zhou & Chao Tian Texas A&M University 2 Contents Motivation Background and Review Evaluating Individual
Network Coding for Distributed Storage Systems* Presented by Jayant Apte ASPITRG 7/9/13 & 7/11/13
Network Coding for Distributed Storage Systems* Presented by Jayant Apte ASPITRG 7/9/13 & 7/11/13 *Dimakis, A.G.; Godfrey, P.B.; Wu, Y.; Wainwright, M.J.; Ramchandran, K. "Network Coding for Distributed
All About Erasure Codes: - Reed-Solomon Coding - LDPC Coding. James S. Plank. ICL - August 20, 2004
All About Erasure Codes: - Reed-Solomon Coding - LDPC Coding James S. Plank Logistical Computing and Internetworking Laboratory Department of Computer Science University of Tennessee ICL - August 2, 24
Performance Models of Access Latency in Cloud Storage Systems
Performance Models of Access Latency in Cloud Storage Systems Qiqi Shuai Email: qqshuai@eee.hku.hk Victor O.K. Li, Fellow, IEEE Email: vli@eee.hku.hk Yixuan Zhu Email: yxzhu@eee.hku.hk Abstract Access
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions. James S. Plank USENIX FAST. University of Tennessee
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions James S. Plank University of Tennessee USENIX FAST San Jose, CA February 15, 2013. Authors Jim Plank Tennessee Kevin Greenan EMC/Data
Cloud Storage Reliability for Big Data Applications: A State of the Art Survey
Cloud Storage Reliability for Big Data Applications: A State of the Art Survey Rekha Nachiappan a, Bahman Javadi a, Rodrigo Calherios a, Kenan Matawie a a School of Computing, Engineering and Mathematics,
Enabling Node Repair in Any Erasure Code for Distributed Storage
Enabling Node Repair in Any Erasure Code for Distributed Storage K. V. Rashmi, Nihar B. Shah, and P. Vijay Kumar, Fellow, IEEE Abstract Erasure codes are an efficient means of storing data across a network
Codes for Modern Applications
Codes for Modern Applications Current Research Topics in the Code and Signal Design Group P. Vijay Kumar Indian Institute of Science, Bangalore, India Nov. 16, 2018 1/28 Codes for Modern Applications:
A survey on regenerating codes
International Journal of Scientific and Research Publications, Volume 4, Issue 11, November 2014 1 A survey on regenerating codes V. Anto Vins *, S.Umamageswari **, P.Saranya ** * P.G Scholar, Department
Network Codes for Next-Generation Distributed Storage:! Opportunities & Challenges. Kannan Ramchandran
Network Codes for Next-Generation Distributed Storage:! Opportunities & Challenges Kannan Ramchandran University of California, Berkeley Joint work with: Salim El Rouayheb (UCB), Hao Zhang (UCB), Sameer
HADOOP 3.0 is here! Dr. Sandeep Deshmukh Sadepach Labs Pvt. Ltd. - Let us grow together!
HADOOP 3.0 is here! Dr. Sandeep Deshmukh sandeep@sadepach.com Sadepach Labs Pvt. Ltd. - Let us grow together! About me BE from VNIT Nagpur, MTech+PhD from IIT Bombay Worked with Persistent Systems - Life
Erasure Codes for Heterogeneous Networked Storage Systems
Erasure Codes for Heterogeneous Networked Storage Systems Lluís Pàmies i Juárez Lluís Pàmies i Juárez lpjuarez@ntu.edu.sg . Introduction Outline 2. Distributed Storage Allocation Problem 3. Homogeneous
Repair Pipelining for Erasure-Coded Storage
Repair Pipelining for Erasure-Coded Storage Runhui Li, Xiaolu Li, Patrick P. C. Lee, Qun Huang The Chinese University of Hong Kong USENIX ATC 2017 1 Introduction Fault tolerance for distributed storage
EECS 121: Coding for Digital Communication & Beyond Fall Lecture 22 December 3. Introduction
EECS 121: Coding for Digital Communication & Beyond Fall 2013 Lecture 22 December 3 Lecturer: K. V. Rashmi Scribe: Ajay Shanker Tripathi 22.1 Context Introduction Distributed storage is a deeply relevant
Data Integrity Protection scheme for Minimum Storage Regenerating Codes in Multiple Cloud Environment
International Journal of Advanced Research in Computer Engineering & Technology (IJARCET) Data Integrity Protection scheme for Minimum Storage Regenerating Codes in Multiple Cloud Environment Saravana
EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures
EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures Haiyang Shi, Xiaoyi Lu, and Dhabaleswar K. (DK) Panda {shi.876, lu.932, panda.2}@osu.edu The Ohio State University
On Coding Techniques for Networked Distributed Storage Systems
On Coding Techniques for Networked Distributed Storage Systems Frédérique Oggier frederique@ntu.edu.sg Nanyang Technological University, Singapore First European Training School on Network Coding, Barcelona,
Capacity Assurance in Hostile Networks
PhD Dissertation Defense Wednesday, October 7, 2015 3:30 pm - 5:30 pm 3112 Engineering Building Capacity Assurance in Hostile Networks By: Jian Li Advisor: Jian Ren ABSTRACT Linear network coding provides
Privacy, Security, and Repair in Distributed Storage Systems. Siddhartha Kumar
Privacy, Security, and Repair in Distributed Storage Systems Siddhartha Kumar Thesis for the Degree of Philosophiae Doctor (PhD) University of Bergen, Norway 2018 Privacy, Security, and Repair in Distributed
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions Ethan L. Miller Center for Research in Storage Systems University of California, Santa Cruz (and Pure Storage) Authors Jim Plank Univ.
Finding the most fault-tolerant flat XOR-based erasure codes for storage systems Jay J. Wylie
Finding the most fault-tolerant flat XOR-based erasure codes for storage systems Jay J. Wylie HP Laboratories HPL-2011-218 Abstract: We describe the techniques we developed to efficiently find the most
A Comparative Study on Distributed Storage and Erasure Coding Techniques Using Apache Hadoop Over NorNet Core
Master Thesis University of Bergen Department of Informatics A Comparative Study on Distributed Storage and Erasure Coding Techniques Using Apache Hadoop Over NorNet Core Author: Maximiliano Vela Supervisor:
Clay Codes: Moulding MDS Codes to Yield an MSR Code
Clay Codes: Moulding MDS Codes to Yield an MSR Code Myna Vajha, Vinayak Ramkumar, Bhagyashree Puranik, Ganesh Kini, Elita Lobo, Birenjith Sasidharan, and P. Vijay Kumar, Indian Institute of Science, Bangalore;
Software-defined Storage: Fast, Safe and Efficient
Software-defined Storage: Fast, Safe and Efficient TRY NOW Thanks to Blockchain and Intel Intelligent Storage Acceleration Library Every piece of data is required to be stored somewhere. We all know about
Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 57, NO 8, AUGUST 2011 5227 Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction K V Rashmi,
Erasure coding and AONT algorithm selection for Secure Distributed Storage. Alem Abreha Sowmya Shetty
Erasure coding and AONT algorithm selection for Secure Distributed Storage Alem Abreha Sowmya Shetty Secure Distributed Storage AONT(All-Or-Nothing Transform) unkeyed transformation φ mapping a sequence
SCALING UP OF E-MSR CODES BASED DISTRIBUTED STORAGE SYSTEMS WITH FIXED NUMBER OF REDUNDANCY NODES
SCALING UP OF E-MSR CODES BASED DISTRIBUTED STORAGE SYSTEMS WITH FIXED NUMBER OF REDUNDANCY NODES Haotian Zhao, Yinlong Xu and Liping Xiang School of Computer Science and Technology, University of Science
A Survey on Erasure Coding Techniques for Cloud Storage System
A Survey on Erasure Coding Techniques for Cloud Storage System Kiran P. Pawar #1, R. M. Jogdand *2 #1 M.Tech, Department of Computer Science, Gogte Institute of technology, Udyambag Belagavi, Karnataka,
Improving Cloud Storage Cost and Data Resiliency with Erasure Codes Michael Penick
Improving Cloud Storage Cost and Data Resiliency with Erasure Codes Michael Penick Commodity Storage Hosting storage FTP backup Goals Inexpensive (use commodity hardware) Resilient to failures Highly available
Coding Techniques for Distributed Storage Systems
Coding Techniques for Distributed Storage Systems Jung-Hyun Kim, {jh.kim06}@yonsei.ac.kr, Hong-Yeop Song Yonsei Univ. Seoul, KORE 3 rd CITW, Oct. 25. 1, 2013 Contents Introduction Codes for Distributed
A RANDOMLY EXPANDABLE METHOD FOR DATA LAYOUT OF RAID STORAGE SYSTEMS. Received October 2017; revised February 2018
International Journal of Innovative Computing, Information and Control ICIC International c 2018 ISSN 1349-4198 Volume 14, Number 3, June 2018 pp. 1079 1094 A RANDOMLY EXPANDABLE METHOD FOR DATA LAYOUT
Everything You Wanted To Know About Storage (But Were Too Proud To Ask) The Basics
Everything You Wanted To Know About Storage (But Were Too Proud To Ask) The Basics Today s Presenters Bob Plumridge HDS Chief Technology Officer - EMEA Alex McDonald NetApp CTO Office 2 SNIA Legal Notice
CSC310 Information Theory. Lecture 21: Erasure (Deletion) Channels & Digital Fountain Codes. November 22, 2006 see
CSC310 Information Theory Lecture 21: Erasure (Deletion) Channels & Digital Fountain Codes Sam Roweis Recovering From Erasures 2 How can we recover from erasures? For the BEC, we saw that low-density parity
On the Speedup of Recovery in Large-Scale Erasure-Coded Storage Systems (Supplementary File)
1 On the Speedup of Recovery in Large-Scale Erasure-Coded Storage Systems (Supplementary File) Yunfeng Zhu, Patrick P. C. Lee, Yinlong Xu, Yuchong Hu, and Liping Xiang 1 ADDITIONAL RELATED WORK Our work
PANASAS TIERED PARITY ARCHITECTURE
PANASAS TIERED PARITY ARCHITECTURE Larry Jones, Matt Reid, Marc Unangst, Garth Gibson, and Brent Welch White Paper May 2010 Abstract Disk drives are approximately 250 times denser today than a decade ago.
CSC310 Information Theory. Lecture 22: Erasure (Deletion) Channels & Digital Fountain Codes. November 30, 2005 see
CSC310 Information Theory Lecture 22: Erasure (Deletion) Channels & Digital Fountain Codes Sam Roweis Recovering From Erasures 2 How can we recover from erasures? For the BEC, we saw that low-density parity
Rethinking Erasure Codes for Cloud File Systems: Minimizing I/O for Recovery and Degraded Reads
Rethinking Erasure Codes for Cloud File Systems: Minimizing I/O for Recovery and Degraded Reads Osama Khan, Randal Burns, James S. Plank, William Pierce and Cheng Huang FAST 2012: 10 th USENIX Conference
Optimize Storage Efficiency & Performance with Erasure Coding Hardware Offload. Dror Goldenberg VP Software Architecture Mellanox Technologies
Optimize Storage Efficiency & Performance with Erasure Coding Hardware Offload Dror Goldenberg VP Software Architecture Mellanox Technologies SNIA Legal Notice The material contained in this tutorial is
A Performance Evaluation of Open Source Erasure Codes for Storage Applications
A Performance Evaluation of Open Source Erasure Codes for Storage Applications James S. Plank Catherine D. Schuman (Tennessee) Jianqiang Luo Lihao Xu (Wayne State) Zooko Wilcox-O'Hearn Usenix FAST February
ECS High Availability Design
ECS High Availability Design March 2018 A Dell EMC white paper Revisions Date Mar 2018 Aug 2017 July 2017 Description Version 1.2 - Updated to include ECS version 3.2 content Version 1.1 - Updated to include
Fountain Codes Based on Zigzag Decodable Coding
Fountain Codes Based on Zigzag Decodable Coding Takayuki Nozaki Kanagawa University, JAPAN Email: nozaki@kanagawa-u.ac.jp Abstract Fountain codes based on non-binary low-density parity-check (LDPC) codes
ActiveScale Erasure Coding and Self Protecting Technologies
WHITE PAPER AUGUST 2018 ActiveScale Erasure Coding and Self Protecting Technologies BitSpread Erasure Coding and BitDynamics Data Integrity and Repair Technologies within The ActiveScale Object Storage
DiskReduce: Making Room for More Data on DISCs. Wittawat Tantisiriroj
DiskReduce: Making Room for More Data on DISCs Wittawat Tantisiriroj Lin Xiao, Bin Fan, and Garth Gibson PARALLEL DATA LABORATORY Carnegie Mellon University GFS/HDFS Triplication GFS & HDFS triplicate
Regenerating Codes for Errors and Erasures in Distributed Storage
Regenerating Codes or Errors and Erasures in Distributed Storage K V Rashmi, Nihar B Shah, Kannan Ramchandran, Fellow, IEEE, and P Vijay Kumar, Fellow, IEEE arxiv:0050v csit] 3 May 0 Abstract Regenerating
Storage systems have grown to the point where failures are inevitable,
A Brief Primer JAMES S. P LANK James S. Plank received his BS from Yale University in 1988 and his PhD from Princeton University in 1993. He is a professor in the Electrical Engineering and Computer Science
On the Speedup of Single-Disk Failure Recovery in XOR-Coded Storage Systems: Theory and Practice
On the Speedup of Single-Disk Failure Recovery in XOR-Coded Storage Systems: Theory and Practice Yunfeng Zhu, Patrick P. C. Lee, Yuchong Hu, Liping Xiang, and Yinlong Xu University of Science and Technology
On Data Parallelism of Erasure Coding in Distributed Storage Systems
On Data Parallelism of Erasure Coding in Distributed Storage Systems Jun Li, Baochun Li Department of Electrical and Computer Engineering, University of Toronto, Canada {junli, bli}@ece.toronto.edu Abstract
Coding theory for scalable media delivery
1 Coding theory for scalable media delivery Michael Luby RaptorQ is a product of Qualcomm Technologies, Inc. Application layer erasure coding complements traditional error coding Forward Error Correction
Cooperative Pipelined Regeneration in Distributed Storage Systems
Cooperative ipelined Regeneration in Distributed Storage Systems Jun Li, in Wang School of Computer Science Fudan University, China Baochun Li Department of Electrical and Computer Engineering University
CORE: Augmenting Regenerating-Coding-Based Recovery for Single and Concurrent Failures in Distributed Storage Systems
CORE: Augmenting Regenerating-Coding-Based Recovery for Single and Concurrent Failures in Distributed Storage Systems Runhui Li, Jian Lin, Patrick P. C. Lee Department of Computer Science and Engineering,
Information-theoretically Secure Regenerating Codes for Distributed Storage
Information-theoretically Secure Regenerating Codes for Distributed Storage Nihar B. Shah, K. V. Rashmi and P. Vijay Kumar Abstract Regenerating codes are a class of codes for distributed storage networks
Raptor Codes for P2P Streaming
Raptor Codes for P2P Streaming Philipp Eittenberger 1, Todor Mladenov 2, Udo Krieger 1 1 Faculty of Information Systems and Applied Computer Science Otto-Friedrich University Bamberg, Germany 2 Department
RAID (Redundant Array of Inexpensive Disks)
Magnetic Disk Characteristics I/O Connection Structure Types of Buses Cache & I/O I/O Performance Metrics I/O System Modeling Using Queuing Theory Designing an I/O System RAID (Redundant Array of Inexpensive
Distributed File System Based on Erasure Coding for I/O-Intensive Applications
Distributed File System Based on Erasure Coding for I/O-Intensive Applications Dimitri Pertin 1,, Sylvain David, Pierre Évenou, Benoît Parrein 1 and Nicolas Normand 1 1 LUNAM Université, Université de
Decentralized Distributed Storage System for Big Data
Decentralized Distributed Storage System for Big Presenter: Wei Xie -Intensive Scalable Computing Laboratory(DISCL) Computer Science Department Texas Tech University Outline Trends in Big and Cloud Storage
ActiveScale Erasure Coding and Self Protecting Technologies
NOVEMBER 2017 ActiveScale Erasure Coding and Self Protecting Technologies BitSpread Erasure Coding and BitDynamics Data Integrity and Repair Technologies within The ActiveScale Object Storage System Software
Building High Speed Erasure Coding Libraries for ARM and x86 Processors. Per Simonsen, CEO, MemoScale May 2017
Building High Speed Erasure Coding Libraries for ARM and x86 Processors Per Simonsen, CEO, MemoScale May 2017 Agenda MemoScale company and team Erasure coding - brief intro MemoScale erasure codes Performance
Encoding-Aware Data Placement for Efficient Degraded Reads in XOR-Coded Storage Systems
Encoding-Aware Data Placement for Efficient Degraded Reads in XOR-Coded Storage Systems Zhirong Shen, Patrick P. C. Lee, Jiwu Shu, Wenzhong Guo College of Mathematics and Computer Science, Fuzhou University
Interference Alignment in Regenerating Codes for Distributed Storage: Necessity and Code Constructions
2134 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 58, NO 4, APRIL 2012 Interference Alignment in Regenerating Codes for Distributed Storage: Necessity and Code Constructions Nihar B Shah, K V Rashmi, P
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions
Screaming Fast Galois Field Arithmetic Using Intel SIMD Instructions James S. Plank EECS Department University of Tennessee plank@cs.utk.edu Kevin M. Greenan EMC Backup Recovery Systems Division Kevin.Greenan@emc.com
Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland
Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Abstract. The Data and Storage Services group at CERN is conducting
BCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding. Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University
BCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University Outline Introduction and Motivation Our Design System and Implementation
Resume Maintaining System for Referral Using Cloud Computing
Resume Maintaining System for Referral Using Cloud Computing Pranjali A. Pali 1, N.D. Kale 2 P.G. Student, Department of Computer Engineering, TSSM S PVPIT, Bavdhan, Pune, Maharashtra India 1 Associate
Short Code: An Efficient RAID-6 MDS Code for Optimizing Degraded Reads and Partial Stripe Writes
: An Efficient RAID-6 MDS Code for Optimizing Degraded Reads and Partial Stripe Writes Yingxun Fu, Jiwu Shu, Xianghong Luo, Zhirong Shen, and Qingda Hu Abstract As reliability requirements are increasingly
Parity Logging with Reserved Space: Towards Efficient Updates and Recovery in Erasure-coded Clustered Storage
Parity Logging with Reserved Space: Towards Efficient Updates and Recovery in Erasure-coded Clustered Storage Jeremy C. W. Chan, Qian Ding, Patrick P. C. Lee, and Helen H. W. Chan, The Chinese University
Lecture 4 September 17
CS 559: Algorithmic Aspects of Computer Networks Fall 2007 Lecture 4 September 17 Lecturer: John Byers BOSTON UNIVERSITY Scribe: Sowmya Manjanatha In today s lecture, we discussed and characterized the
FPGA Implementation of Erasure Codes in NVMe based JBOFs
FPGA Implementation of Erasure Codes in NVMe based JBOFs Manoj Roge Director, Data Center Xilinx Inc. Santa Clara, CA 1 Acknowledgement Shre Shah Data Center Architect, Xilinx Santa Clara, CA 2 Agenda
DiskReduce: Making Room for More Data on DISCs. Wittawat Tantisiriroj
DiskReduce: Making Room for More Data on DISCs Wittawat Tantisiriroj Lin Xiao, in Fan, and Garth Gibson PARALLEL DATA LAORATORY Carnegie Mellon University GFS/HDFS Triplication GFS & HDFS triplicate every
HP AutoRAID (Lecture 5, cs262a)
HP AutoRAID (Lecture 5, cs262a) Ion Stoica, UC Berkeley September 13, 2016 (based on presentation from John Kubiatowicz, UC Berkeley) Array Reliability Reliability of N disks = Reliability of 1 Disk N
Apache Hadoop 3. Balazs Gaspar Sales Engineer CEE & CIS Cloudera, Inc. All rights reserved.
Apache Hadoop 3 Balazs Gaspar Sales Engineer CEE & CIS balazs@cloudera.com 1 We believe data can make what is impossible today, possible tomorrow 2 We empower people to transform complex data into clear
Always-On Erasure Coding
Always-On Erasure Coding 385 Moffett Park Dr. Sunnyvale, CA 94089 844-478-8349 www.datrium.com Technical Report Introduction Erasure Coding & RAID Goals 1 ] Always-on Erasure Coding 2 ] Two drive failure
An Improvement of Quasi-cyclic Minimum Storage Regenerating Codes for Distributed Storage
An Improvement of Quasi-cyclic Minimum Storage Regenerating Codes for Distributed Storage Chenhui LI*, Songtao LIANG* * Shanghai Key Laboratory of Intelligent Information Processing, Fudan University,
So you think you can FEC?
So you think you can FEC? Mikhail Misha Fludkov (fludkov) Pexip R&D Erlend Earl Graff (egraff) Pexip R&D 8 th GStreamer Conference 21 October 2017 Prague, Czech Republic Outline Introduction to FEC Designing
Disk Arrays Nov. 7, 2011!
15-410...Failure is not an option...! Disk Arrays Nov. 7, 2011! Garth Gibson! Dave Eckhardt!! Contributions by! Michael Ashley-Rollman! 1 L25_RAID Overview! Historical practices! Striping! Mirroring! The
Exploration of Erasure-Coded Storage Systems for High Performance, Reliability, and Inter-operability
Virginia Commonwealth University VCU Scholars Compass Theses and Dissertations Graduate School 2016 Exploration of Erasure-Coded Storage Systems for High Performance, Reliability, and Inter-operability
EC-Cache: Load-balanced, Low-latency Cluster Caching with Online Erasure Coding
EC-Cache: Load-balanced, Low-latency Cluster Caching with Online Erasure Coding Rashmi Vinayak UC Berkeley Joint work with Mosharaf Chowdhury, Jack Kosaian (U Michigan) Ion Stoica, Kannan Ramchandran (UC
Dell Technologies IoT Solution Surveillance with Genetec Security Center
Dell Technologies IoT Solution Surveillance with Genetec Security Center Surveillance December 2018 H17436 Sizing Guide Abstract The purpose of this guide is to help you understand the benefits of using
WHITE PAPER SINGLE & MULTI CORE PERFORMANCE OF AN ERASURE CODING WORKLOAD ON AMD EPYC
WHITE PAPER SINGLE & MULTI CORE PERFORMANCE OF AN ERASURE CODING WORKLOAD ON AMD EPYC INTRODUCTION With the EPYC processor line, AMD is expected to take a strong position in the server market including
CSEP 561 Error detection & correction. David Wetherall
CSEP 561 Error detection & correction David Wetherall djw@cs.washington.edu Codes for Error Detection/Correction ti ti Error detection and correction How do we detect and correct messages that are garbled
Efficient Load Balancing and Disk Failure Avoidance Approach Using Restful Web Services
Efficient Load Balancing and Disk Failure Avoidance Approach Using Restful Web Services Neha Shiraz, Dr. Parikshit N. Mahalle Persuing M.E, Department of Computer Engineering, Smt. Kashibai Navale College
Data Protection for Cisco HyperFlex with Veeam Availability Suite. Solution Overview Cisco Public
Data Protection for Cisco HyperFlex with Veeam Availability Suite 1 2017 2017 Cisco Cisco and/or and/or its affiliates. its affiliates. All rights All rights reserved. reserved. Highlights Is Cisco compatible
A Simulation Analysis of Reliability in Erasure-Coded Data Centers
A Simulation Analysis of Reliability in Erasure-Coded Data Centers Mi Zhang, Shujie Han, and Patrick P. C. Lee Department of Computer Science and Engineering, The Chinese University of Hong Kong {mzhang,sjhan,pclee}@cse.cuhk.edu.hk
Graph based codes for distributed storage systems
/23 Graph based codes for distributed storage systems July 2, 25 Christine Kelley University of Nebraska-Lincoln Joint work with Allison Beemer and Carolyn Mayer Combinatorics and Computer Algebra, COCOA
Error Control Coding for MLC Flash Memories
Error Control Coding for MLC Flash Memories Ying Y. Tai, Ph.D. Cadence Design Systems, Inc. ytai@cadence.com August 19, 2010 Santa Clara, CA 1 Outline The Challenges on Error Control Coding (ECC) for MLC
MODERN storage systems, such as GFS [1], Windows
![MODERN storage systems, such as GFS [1], Windows](/thumbs/92/107866960.jpg "MODERN storage systems, such as GFS [1], Windows")
IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 1, JANUARY 2017 127 Short Code: An Efficient RAID-6 MDS Code for Optimizing Degraded Reads and Partial Stripe Writes Yingxun Fu, Jiwu Shu, Senior Member, IEEE,
ECFS: A decentralized, distributed and faulttolerant FUSE filesystem for the LHCb online farm
Journal of Physics: Conference Series OPEN ACCESS ECFS: A decentralized, distributed and faulttolerant FUSE filesystem for the LHCb online farm To cite this article: Tomasz Rybczynski et al 2014 J. Phys.:
COS 318: Operating Systems. Storage Devices. Vivek Pai Computer Science Department Princeton University
COS 318: Operating Systems Storage Devices Vivek Pai Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall11/cos318/ Today s Topics Magnetic disks Magnetic disk
CSCI-1680 Link Layer Reliability Rodrigo Fonseca
CSCI-1680 Link Layer Reliability Rodrigo Fonseca Based partly on lecture notes by David Mazières, Phil Levis, John Janno< Last time Physical layer: encoding, modulation Link layer framing Today Getting
Networked Systems and Services, Fall 2018 Chapter 2. Jussi Kangasharju Markku Kojo Lea Kutvonen
Networked Systems and Services, Fall 2018 Chapter 2 Jussi Kangasharju Markku Kojo Lea Kutvonen Outline Physical layer reliability Low level reliability Parities and checksums Cyclic Redundancy Check (CRC)
GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures
GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,
Packet-Level Forward Error Correction in Video Transmission
Packet-Level Forward Error Correction in Video Transmission Matteo Mazzotti, Enrico Paolini, Marco Chiani, Davide Dardari, and Andrea Giorgetti University of Bologna Wireless Communications Laboratory
PCM: A Parity-check Matrix Based Approach to Improve Decoding Performance of XOR-based Erasure Codes
15 IEEE th Symposium on Reliable Distributed Systems : A Parity-check Matrix Based Approach to Improve Decoding Performance of XOR-based Erasure Codes Yongzhe Zhang, Chentao Wu, Jie Li, Minyi Guo Shanghai
Understanding System Characteristics of Online Erasure Coding on Scalable, Distributed and Large-Scale SSD Array Systems
Understanding System Characteristics of Online Erasure Coding on Scalable, Distributed and Large-Scale SSD Array Systems arxiv:179.5365v2 [cs.dc] 19 Sep 217 Sungjoon Koh, Jie Zhang, Miryeong Kwon, Jungyeon
Scale-out Data Deduplication Architecture
Scale-out Data Deduplication Architecture Gideon Senderov Product Management & Technical Marketing NEC Corporation of America Outline Data Growth and Retention Deduplication Methods Legacy Architecture
Distributed File System based on Erasure Coding for I/O Intensive Applications
Distributed File System based on Erasure Coding for I/O Intensive Applications Dimitri Pertin, Sylvain David, Pierre Evenou, Benoît Parrein, Nicolas Normand To cite this version: Dimitri Pertin, Sylvain
GridBlocks DISK - Distributed Inexpensive Storage with K-availability
GridBlocks DISK - Distributed Inexpensive Storage with K-availability Mikko Pitkanen Helsinki Institute of Physics Technology Programme mikko.pitkanen@hip.fi Juho Karppinen Helsinki Institute of Physics
Applied Erasure Coding in Networks and Distributed Storage
arxiv:1803.01358v1 [cs.it] 4 Mar 2018 Katina Kralevska Applied Erasure Coding in Networks and Distributed Storage Thesis for the degree of Philosophiae Doctor Trondheim, March 2018 Department of Telematics,