BCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding. Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University
|
|
- Cory Summers
- 5 years ago
- Views:
Transcription
1 BCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University
2 Outline Introduction and Motivation Our Design System and Implementation Evaluation
3 Outline Introduction and Motivation Our Design System and Implementation Evaluation
4 In-memory KV-Store A crucial building block for many systems Data cache (e.g. Memcached and Redis in Facebook, Twitter) In-memory database Availability is important for in-memory KV-Stores Facebook reports that it takes hours to recover 120GB data of an in-memory database from disk to memory Data redundancy in distributed memory is essential for fast failover
5 Two redundancy schemes Replication is a classical way to provide data availability E.g., Repcached, Redis Client Write request High bandwidth cost Update Data node High memory cost Backup node Backup node
6 Two redundancy schemes Erasure coding is a space-efficient redundancy scheme The increase of CPU speed enables fast data recovery Encoding/Decoding rates can reach 40Gb/s on single core [1] Client Write request High bandwidth cost Data Node Update Parity Node Data Node Parity Node Data Node Low memory cost [1] Efficient and Available In-memory KV-Store with Hybrid Erasure Coding and Replication, FAST 16
7 In-place Update A traditional mechanism for encoding small objects Update(obj4->obj4 ) Delta(obj4, obj4 ) Data node 1 Data node 2 Data node 3 Parity node 1 Update (obj3->obj3 ) obj1 obj4 obj7 obj2 obj5 obj8 obj3 obj6 obj9 Pp Pp Pp Update(obj8->obj8 ) Bandwidth cost is the same as 3-replication Parity node 2 pp pp Pp Our goal: both memory efficiency and bandwidth efficiency
8 Outline Introduction and Motivation Our Design System and Implementation Evaluation
9 Our Design Aggregate write requests and encode objects in a new coding stripe invalid Batch coding Append Data node 1 obj1 obj4 obj7 obj4 Data node 2 obj2 obj5 obj8 obj8 Data node 3 obj3 obj6 obj9 obj3 Batch node Parity node 1 P P P P Parity node 2 P P P P
10 Latency Analysis Batch coding induces extra request waiting time Formalize the waiting time W W = f(t, k) Request throughput number of data nodes Latency bound Ɛ K = 3
11 Garbage Collection Recycle updated or deleted blocks and release extra parity blocks Move-based garbage collection Original stripes Batched stripes Data nodes Move Parity nodes Much bandwidth cost for updating parity blocks GC GC
12 Garbage Collection How to reduce the GC bandwidth cost? Intuition: GC the stripes with the most invalid blocks Greedy block moving Original stripes Batched stripes Data nodes Parity nodes GC GC Two block moves to release two coding stripes
13 Garbage Collection How to further reduce block move? Intuition: make the updates focus on few stripes Popularity-based data arrangement Data nodes Hot Original stripes Cold Hot Batched stripes Cold Parity nodes GC GC Only one block move to release two coding stripes
14 Bandwidth Analysis Theorem GC bandwidth + Coding bandwidth <= In-place update bandwidth Detailed proof can be found in our paper
15 Outline Introduction and Motivation Our Design System and Implementation Evaluation
16 System Architecture Client Client Preprocessing Batch process Batch coding Data process Data process Data process Client Garbage collection Metadata management Parity process Parity process Storage group
17 Handle Write Requests Client Set(k1, v1) Client set(k2, v2) Client set(k3, v3) Hash table Batch process Batch coding v2 v1 v3 P1 Stripe index Update v2 Data process 1 v1 Data process 2 v3 Data process 3 P1 Parity process 1 P2 Parity process 2 P2 b1
18 Handle Read Requests v2 Data process 1 Client get(k1) Hash table Batch process Stripe index get(b1) v1 Data process 2 v3 Data process 3 Key Stripe id P1 Parity process 1 k1 k2 k3 b1 b1 b1 P2 Parity process 2
19 Recovery Recover the request data first Client get(k1) 1. Get values according to stripe id from any k storage processes Batch process Decoder v2 Data process 1 v1 Data process 2 v3 Data process 3 P1 Parity process 1 2. Recover the lost blocks v2 P1 P2 v1 P2 Parity process 2
20 Outline Introduction and Motivation Our Design System and Implementation Evaluation
21 Evaluation Cluster configuration 10 machines running SUSE Linux 11 containing 12 * AMD Opteron Processor 4180 CPUs 1Gb/s Ethernet Targets of comparison In-place update EC (Cocytus[1]) Replication (Rep) Workload YCSB with different key distributions 50%:50% read/write ratio [1] Efficient and Available In-memory KV-Store with Hybrid Erasure Coding and Replication, FAST 16
22 Bandwidth Cost Save up to 51% bandwidth cost Bandwidth cost for different coding schemes.
23 Throughput Up to 2.4x improvement Throughput performance for different coding schemes.
24 Memory Save up to 41% memory cost Memory consumption for different redundancy schemes
25 Latency Read latency Write latency
26 Conclusion Efficiency and availability are two crucial features for inmemory KV-Stores We build BCStore, an in-memory KV-Store which applies erasure coding for data availability We design batch coding mechanism to achieve high bandwidth efficiency for write workload We propose a heuristic garbage collection algorithm to improve memory efficiency
27 Thanks! Q&A
28 Severity of Bandwidth Cost Prevalence of write requests in large-scale web services Peak load can easily run out of network bandwidth and degrade service performance Monetary cost of bandwidth becomes several times higher Especially under the commonly used peak-load pricing model Bandwidth amplification would be more serious with the increase of m (number of parity servers) Budget of bandwidth resource is usually limited in workload-sharing cluster Our goal: High memory efficiency and bandwidth efficiency
29 Our Design Batch write requests and append a new coding stripe Batch coding Append Data node 1 obj1 obj4 obj7 obj4 Data node 2 obj2 obj5 obj8 obj8 Data node 3 obj3 obj6 obj9 obj3 Parity node 1 Parity node 2 P P P P
30 Challenges Recycle the memory space of data blocks which are deleted or updated Data blocks and parity blocks are appended to the storage Updated blocks can not be delete directly Encode variable-sized data efficiently Variable-sized data can not be appended to previous storage space directly
31 Garbage Collection Popularity-based data arrangement Hot Data node 1 Sort Data node 2 Data node 3 Batched objects cold Parity node1 Parity node2
32 Encoding Variable-size Data Virtual coding stripes (vcs) Each virtual coding stripe has a large fixed-length space and is aligned in virtual address Data node 1 Physical space Data node 1 Data node 2 Data node 3 Parity node 1 Parity node 2 vcs1 vcs2 vcs3 Virtual space
33 Bandwidth Cost Bandwidth cost for moderate-skewed Zipfian workload (RS(3,2))
34 Throughput Throughput performance for moderate-skewed Zipfian workload
35 Throughput Throughput for recovery
36 In-place Update A traditional mechanism for coding small objects Data node 1 obj1 obj4 obj7 Data node 2 obj2 obj5 obj8 Data node 3 obj3 obj6 obj9 Parity node 1 P P P Parity node 2 P P P
37 Garbage Collection How to further reduce block move? Intuition: make the updates focus on few stripes Popularity-based data arrangement Data nodes Hot Original stripes Cold Hot Batched stripes Cold Parity nodes GC GC
38 Bandwidth Analysis Theorem GC bandwidth + Coding bandwidth <= In-place update bandwidth Original stripes Batched stripes Data nodes Parity nodes GC GC Worst case of GC bandwidth
39 Bandwidth Cost Bandwidth cost for different throughput. (RS(5,4))
40 Recovery 1. Get latest batch id Data process 1 M Batch process Data process 2 Client Replication Data process 3 3. Serve requests M Batch process 2. Update the latest stable batch id and reconstruct metadata Parity process 1 Parity process 2
BCStore: Bandwidth-Efficient In-memory KV-Store with Batch Coding
: Bandwidth-Efficient In-memory KV-Store with Batch Coding Shenglong Li, Quanlu Zhang, Zhi Yang and Yafei Dai Peking University Institute of Big Data Technologies Shenzhen Key Lab for Cloud Computing Technology
More informationAtlas: Baidu s Key-value Storage System for Cloud Data
Atlas: Baidu s Key-value Storage System for Cloud Data Song Jiang Chunbo Lai Shiding Lin Liqiong Yang Guangyu Sun Jason Cong Wayne State University Zhenyu Hou Can Cui Peking University University of California
More informationCascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching
Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching Kefei Wang and Feng Chen Louisiana State University SoCC '18 Carlsbad, CA Key-value Systems in Internet Services Key-value
More informationSMORE: A Cold Data Object Store for SMR Drives
SMORE: A Cold Data Object Store for SMR Drives Peter Macko, Xiongzi Ge, John Haskins Jr.*, James Kelley, David Slik, Keith A. Smith, and Maxim G. Smith Advanced Technology Group NetApp, Inc. * Qualcomm
More informationarxiv: v1 [cs.db] 25 Nov 2018
Enabling Efficient Updates in KV Storage via Hashing: Design and Performance Evaluation Yongkun Li, Helen H. W. Chan, Patrick P. C. Lee, and Yinlong Xu University of Science and Technology of China The
More informationSDPaxos: Building Efficient Semi-Decentralized Geo-replicated State Machines
SDPaxos: Building Efficient Semi-Decentralized Geo-replicated State Machines Hanyu Zhao *, Quanlu Zhang, Zhi Yang *, Ming Wu, Yafei Dai * * Peking University Microsoft Research Replication for Fault Tolerance
More informationAmbry: LinkedIn s Scalable Geo- Distributed Object Store
Ambry: LinkedIn s Scalable Geo- Distributed Object Store Shadi A. Noghabi *, Sriram Subramanian +, Priyesh Narayanan +, Sivabalan Narayanan +, Gopalakrishna Holla +, Mammad Zadeh +, Tianwei Li +, Indranil
More informationCGAR: Strong Consistency without Synchronous Replication. Seo Jin Park Advised by: John Ousterhout
CGAR: Strong Consistency without Synchronous Replication Seo Jin Park Advised by: John Ousterhout Improved update performance of storage systems with master-back replication Fast: updates complete before
More informationParaFS: A Log-Structured File System to Exploit the Internal Parallelism of Flash Devices
ParaFS: A Log-Structured File System to Exploit the Internal Parallelism of Devices Jiacheng Zhang, Jiwu Shu, Youyou Lu Tsinghua University 1 Outline Background and Motivation ParaFS Design Evaluation
More informationEfficient and Available In-memory KV-Store with Hybrid Erasure Coding and Replication
Efficient and Available In-memory KV-Store with Hybrid Erasure Coding and Replication Heng Zhang, Mingkai Dong, and Haibo Chen, Shanghai Jiao Tong University https://wwwusenixorg/conference/fast16/technical-sessions/presentation/zhang-heng
More informationAmazon ElastiCache 8/1/17. Why Amazon ElastiCache is important? Introduction:
Amazon ElastiCache Introduction: How to improve application performance using caching. What are the ElastiCache engines, and the difference between them. How to scale your cluster vertically. How to scale
More informationEfficient and Available In-Memory KV-Store with Hybrid Erasure Coding and Replication
Efficient and Available In-Memory KV-Store with Hybrid Erasure Coding and Replication HAIBO CHEN, HENG ZHANG, MINGKAI DONG, ZHAOGUO WANG, YUBIN XIA, HAIBING GUAN, and BINYU ZANG, Shanghai Key Laboratory
More informationFacilitating Magnetic Recording Technology Scaling for Data Center Hard Disk Drives through Filesystem-level Transparent Local Erasure Coding
Facilitating Magnetic Recording Technology Scaling for Data Center Hard Disk Drives through Filesystem-level Transparent Local Erasure Coding Yin Li, Hao Wang, Xuebin Zhang, Ning Zheng, Shafa Dahandeh,
More informationUCS Invicta: A New Generation of Storage Performance. Mazen Abou Najm DC Consulting Systems Engineer
UCS Invicta: A New Generation of Storage Performance Mazen Abou Najm DC Consulting Systems Engineer HDDs Aren t Designed For High Performance Disk 101 Can t spin faster (200 IOPS/Drive) Can t seek faster
More informationCSE 124: Networked Services Lecture-17
Fall 2010 CSE 124: Networked Services Lecture-17 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa10/cse124 11/30/2010 CSE 124 Networked Services Fall 2010 1 Updates PlanetLab experiments
More informationHashKV: Enabling Efficient Updates in KV Storage via Hashing
HashKV: Enabling Efficient Updates in KV Storage via Hashing Helen H. W. Chan, Yongkun Li, Patrick P. C. Lee, Yinlong Xu The Chinese University of Hong Kong University of Science and Technology of China
More informationNV-Tree Reducing Consistency Cost for NVM-based Single Level Systems
NV-Tree Reducing Consistency Cost for NVM-based Single Level Systems Jun Yang 1, Qingsong Wei 1, Cheng Chen 1, Chundong Wang 1, Khai Leong Yong 1 and Bingsheng He 2 1 Data Storage Institute, A-STAR, Singapore
More informationFederated Array of Bricks Y Saito et al HP Labs. CS 6464 Presented by Avinash Kulkarni
Federated Array of Bricks Y Saito et al HP Labs CS 6464 Presented by Avinash Kulkarni Agenda Motivation Current Approaches FAB Design Protocols, Implementation, Optimizations Evaluation SSDs in enterprise
More informationCold Storage: The Road to Enterprise Ilya Kuznetsov YADRO
Cold Storage: The Road to Enterprise Ilya Kuznetsov YADRO Agenda Technical challenge Custom product Growth of aspirations Enterprise requirements Making an enterprise cold storage product 2 Technical Challenge
More informationSFS: Random Write Considered Harmful in Solid State Drives
SFS: Random Write Considered Harmful in Solid State Drives Changwoo Min 1, 2, Kangnyeon Kim 1, Hyunjin Cho 2, Sang-Won Lee 1, Young Ik Eom 1 1 Sungkyunkwan University, Korea 2 Samsung Electronics, Korea
More informationHigh-Performance Key-Value Store on OpenSHMEM
High-Performance Key-Value Store on OpenSHMEM Huansong Fu*, Manjunath Gorentla Venkata, Ahana Roy Choudhury*, Neena Imam, Weikuan Yu* *Florida State University Oak Ridge National Laboratory Outline Background
More informationParallel File Systems for HPC
Introduction to Scuola Internazionale Superiore di Studi Avanzati Trieste November 2008 Advanced School in High Performance and Grid Computing Outline 1 The Need for 2 The File System 3 Cluster & A typical
More informationDiskReduce: Making Room for More Data on DISCs. Wittawat Tantisiriroj
DiskReduce: Making Room for More Data on DISCs Wittawat Tantisiriroj Lin Xiao, Bin Fan, and Garth Gibson PARALLEL DATA LABORATORY Carnegie Mellon University GFS/HDFS Triplication GFS & HDFS triplicate
More informationBe Fast, Cheap and in Control with SwitchKV Xiaozhou Li
Be Fast, Cheap and in Control with SwitchKV Xiaozhou Li Raghav Sethi Michael Kaminsky David G. Andersen Michael J. Freedman Goal: fast and cost-effective key-value store Target: cluster-level storage for
More informationGearDB: A GC-free Key-Value Store on HM-SMR Drives with Gear Compaction
GearDB: A GC-free Key-Value Store on HM-SMR Drives with Gear Compaction Ting Yao 1,2, Jiguang Wan 1, Ping Huang 2, Yiwen Zhang 1, Zhiwen Liu 1 Changsheng Xie 1, and Xubin He 2 1 Huazhong University of
More informationA Gentle Introduction to Ceph
A Gentle Introduction to Ceph Narrated by Tim Serong tserong@suse.com Adapted from a longer work by Lars Marowsky-Brée lmb@suse.com Once upon a time there was a Free and Open Source distributed storage
More informationA Light-weight Compaction Tree to Reduce I/O Amplification toward Efficient Key-Value Stores
A Light-weight Compaction Tree to Reduce I/O Amplification toward Efficient Key-Value Stores T i n g Y a o 1, J i g u a n g W a n 1, P i n g H u a n g 2, X u b i n He 2, Q i n g x i n G u i 1, F e i W
More informationOptimizing Flash-based Key-value Cache Systems
Optimizing Flash-based Key-value Cache Systems Zhaoyan Shen, Feng Chen, Yichen Jia, Zili Shao Department of Computing, Hong Kong Polytechnic University Computer Science & Engineering, Louisiana State University
More informationThe term "physical drive" refers to a single hard disk module. Figure 1. Physical Drive
HP NetRAID Tutorial RAID Overview HP NetRAID Series adapters let you link multiple hard disk drives together and write data across them as if they were one large drive. With the HP NetRAID Series adapter,
More informationCA485 Ray Walshe Google File System
Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage
More informationpblk the OCSSD FTL Linux FAST Summit 18 Javier González Copyright 2018 CNEX Labs
pblk the OCSSD FTL Linux FAST Summit 18 Javier González Read Latency Read Latency with 0% Writes Random Read 4K Percentiles 2 Read Latency Read Latency with 20% Writes Random Read 4K + Random Write 4K
More informationNPTEL Course Jan K. Gopinath Indian Institute of Science
Storage Systems NPTEL Course Jan 2012 (Lecture 39) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,
More informationWhite Paper. EonStor GS Family Best Practices Guide. Version: 1.1 Updated: Apr., 2018
EonStor GS Family Best Practices Guide White Paper Version: 1.1 Updated: Apr., 2018 Abstract: This guide provides recommendations of best practices for installation and configuration to meet customer performance
More informationToward Energy-efficient and Fault-tolerant Consistent Hashing based Data Store. Wei Xie TTU CS Department Seminar, 3/7/2017
Toward Energy-efficient and Fault-tolerant Consistent Hashing based Data Store Wei Xie TTU CS Department Seminar, 3/7/2017 1 Outline General introduction Study 1: Elastic Consistent Hashing based Store
More informationErasure Coding in Object Stores: Challenges and Opportunities
Erasure Coding in Object Stores: Challenges and Opportunities Lewis Tseng Boston College July 2018, PODC Acknowledgements Nancy Lynch Muriel Medard Kishori Konwar Prakash Narayana Moorthy Viveck R. Cadambe
More informationvsan Remote Office Deployment January 09, 2018
January 09, 2018 1 1. vsan Remote Office Deployment 1.1.Solution Overview Table of Contents 2 1. vsan Remote Office Deployment 3 1.1 Solution Overview Native vsphere Storage for Remote and Branch Offices
More informationReducing Solid-State Storage Device Write Stress Through Opportunistic In-Place Delta Compression
Reducing Solid-State Storage Device Write Stress Through Opportunistic In-Place Delta Compression Xuebin Zhang, Jiangpeng Li, Hao Wang, Kai Zhao and Tong Zhang xuebinzhang.rpi@gmail.com ECSE Department,
More informationBig and Fast. Anti-Caching in OLTP Systems. Justin DeBrabant
Big and Fast Anti-Caching in OLTP Systems Justin DeBrabant Online Transaction Processing transaction-oriented small footprint write-intensive 2 A bit of history 3 OLTP Through the Years relational model
More informationIBM V7000 Unified R1.4.2 Asynchronous Replication Performance Reference Guide
V7 Unified Asynchronous Replication Performance Reference Guide IBM V7 Unified R1.4.2 Asynchronous Replication Performance Reference Guide Document Version 1. SONAS / V7 Unified Asynchronous Replication
More informationPaxos Replicated State Machines as the Basis of a High- Performance Data Store
Paxos Replicated State Machines as the Basis of a High- Performance Data Store William J. Bolosky, Dexter Bradshaw, Randolph B. Haagens, Norbert P. Kusters and Peng Li March 30, 2011 Q: How to build a
More informationStrata: A Cross Media File System. Youngjin Kwon, Henrique Fingler, Tyler Hunt, Simon Peter, Emmett Witchel, Thomas Anderson
A Cross Media File System Youngjin Kwon, Henrique Fingler, Tyler Hunt, Simon Peter, Emmett Witchel, Thomas Anderson 1 Let s build a fast server NoSQL store, Database, File server, Mail server Requirements
More informationNative vsphere Storage for Remote and Branch Offices
SOLUTION OVERVIEW VMware vsan Remote Office Deployment Native vsphere Storage for Remote and Branch Offices VMware vsan is the industry-leading software powering Hyper-Converged Infrastructure (HCI) solutions.
More informationSRCMap: Energy Proportional Storage using Dynamic Consolidation
SRCMap: Energy Proportional Storage using Dynamic Consolidation By: Akshat Verma, Ricardo Koller, Luis Useche, Raju Rangaswami Presented by: James Larkby-Lahet Motivation storage consumes 10-25% of datacenter
More informationBig data, little time. Scale-out data serving. Scale-out data serving. Highly skewed key popularity
/7/6 Big data, little time Goal is to keep (hot) data in memory Requires scale-out approach Each server responsible for one chunk Fast access to local data The Case for RackOut Scalable Data Serving Using
More informationAdvanced Architectures for Oracle Database on Amazon EC2
Advanced Architectures for Oracle Database on Amazon EC2 Abdul Sathar Sait Jinyoung Jung Amazon Web Services November 2014 Last update: April 2016 Contents Abstract 2 Introduction 3 Oracle Database Editions
More informationPresented by Nanditha Thinderu
Presented by Nanditha Thinderu Enterprise systems are highly distributed and heterogeneous which makes administration a complex task Application Performance Management tools developed to retrieve information
More informationA New Key-value Data Store For Heterogeneous Storage Architecture Intel APAC R&D Ltd.
A New Key-value Data Store For Heterogeneous Storage Architecture Intel APAC R&D Ltd. 1 Agenda Introduction Background and Motivation Hybrid Key-Value Data Store Architecture Overview Design details Performance
More informationRepair Pipelining for Erasure-Coded Storage
Repair Pipelining for Erasure-Coded Storage Runhui Li, Xiaolu Li, Patrick P. C. Lee, Qun Huang The Chinese University of Hong Kong USENIX ATC 2017 1 Introduction Fault tolerance for distributed storage
More informationSHRD: Improving Spatial Locality in Flash Storage Accesses by Sequentializing in Host and Randomizing in Device
SHRD: Improving Spatial Locality in Flash Storage Accesses by Sequentializing in Host and Randomizing in Device Hyukjoong Kim 1, Dongkun Shin 1, Yun Ho Jeong 2 and Kyung Ho Kim 2 1 Samsung Electronics
More informationA New Key-Value Data Store For Heterogeneous Storage Architecture
A New Key-Value Data Store For Heterogeneous Storage Architecture brien.porter@intel.com wanyuan.yang@intel.com yuan.zhou@intel.com jian.zhang@intel.com Intel APAC R&D Ltd. 1 Agenda Introduction Background
More informationBe Fast, Cheap and in Control with SwitchKV. Xiaozhou Li
Be Fast, Cheap and in Control with SwitchKV Xiaozhou Li Goal: fast and cost-efficient key-value store Store, retrieve, manage key-value objects Get(key)/Put(key,value)/Delete(key) Target: cluster-level
More informationVMware vsphere Clusters in Security Zones
SOLUTION OVERVIEW VMware vsan VMware vsphere Clusters in Security Zones A security zone, also referred to as a DMZ," is a sub-network that is designed to provide tightly controlled connectivity to an organization
More informationvsan Management Cluster First Published On: Last Updated On:
First Published On: 07-07-2016 Last Updated On: 03-05-2018 1 1. vsan Management Cluster 1.1.Solution Overview Table Of Contents 2 1. vsan Management Cluster 3 1.1 Solution Overview HyperConverged Infrastructure
More informationDistributed Video Systems Chapter 5 Issues in Video Storage and Retrieval Part 2 - Disk Array and RAID
Distributed Video ystems Chapter 5 Issues in Video torage and Retrieval art 2 - Disk Array and RAID Jack Yiu-bun Lee Department of Information Engineering The Chinese University of Hong Kong Contents 5.1
More information- SLED: single large expensive disk - RAID: redundant array of (independent, inexpensive) disks
RAID and AutoRAID RAID background Problem: technology trends - computers getting larger, need more disk bandwidth - disk bandwidth not riding moore s law - faster CPU enables more computation to support
More informationvsan 6.6 Performance Improvements First Published On: Last Updated On:
vsan 6.6 Performance Improvements First Published On: 07-24-2017 Last Updated On: 07-28-2017 1 Table of Contents 1. Overview 1.1.Executive Summary 1.2.Introduction 2. vsan Testing Configuration and Conditions
More informationCorrelation based File Prefetching Approach for Hadoop
IEEE 2nd International Conference on Cloud Computing Technology and Science Correlation based File Prefetching Approach for Hadoop Bo Dong 1, Xiao Zhong 2, Qinghua Zheng 1, Lirong Jian 2, Jian Liu 1, Jie
More informationI/O CANNOT BE IGNORED
LECTURE 13 I/O I/O CANNOT BE IGNORED Assume a program requires 100 seconds, 90 seconds for main memory, 10 seconds for I/O. Assume main memory access improves by ~10% per year and I/O remains the same.
More informationRAID SEMINAR REPORT /09/2004 Asha.P.M NO: 612 S7 ECE
RAID SEMINAR REPORT 2004 Submitted on: Submitted by: 24/09/2004 Asha.P.M NO: 612 S7 ECE CONTENTS 1. Introduction 1 2. The array and RAID controller concept 2 2.1. Mirroring 3 2.2. Parity 5 2.3. Error correcting
More informationvsan Security Zone Deployment First Published On: Last Updated On:
First Published On: 06-14-2017 Last Updated On: 11-20-2017 1 1. vsan Security Zone Deployment 1.1.Solution Overview Table of Contents 2 1. vsan Security Zone Deployment 3 1.1 Solution Overview VMware vsphere
More informationGUIDE. Optimal Network Designs with Cohesity
Optimal Network Designs with Cohesity TABLE OF CONTENTS Introduction...3 Key Concepts...4 Five Common Configurations...5 3.1 Simple Topology...5 3.2 Standard Topology...6 3.3 Layered Topology...7 3.4 Cisco
More informationWhat's new in Jewel for RADOS? SAMUEL JUST 2015 VAULT
What's new in Jewel for RADOS? SAMUEL JUST 2015 VAULT QUICK PRIMER ON CEPH AND RADOS CEPH MOTIVATING PRINCIPLES All components must scale horizontally There can be no single point of failure The solution
More informationI/O CANNOT BE IGNORED
LECTURE 13 I/O I/O CANNOT BE IGNORED Assume a program requires 100 seconds, 90 seconds for main memory, 10 seconds for I/O. Assume main memory access improves by ~10% per year and I/O remains the same.
More informationCOSC 6385 Computer Architecture Storage Systems
COSC 6385 Computer Architecture Storage Systems Edgar Gabriel Spring 2016 I/O problem Current processor performance: e.g. Pentium 4 3 GHz ~ 6GFLOPS Memory Bandwidth: 133 MHz * 4 * 64Bit ~ 4.26 GB/s Current
More informationIsilon Performance. Name
1 Isilon Performance Name 2 Agenda Architecture Overview Next Generation Hardware Performance Caching Performance Streaming Reads Performance Tuning OneFS Architecture Overview Copyright 2014 EMC Corporation.
More informationJinho Hwang (IBM Research) Wei Zhang, Timothy Wood, H. Howie Huang (George Washington Univ.) K.K. Ramakrishnan (Rutgers University)
Jinho Hwang (IBM Research) Wei Zhang, Timothy Wood, H. Howie Huang (George Washington Univ.) K.K. Ramakrishnan (Rutgers University) Background: Memory Caching Two orders of magnitude more reads than writes
More informationTailwind: Fast and Atomic RDMA-based Replication. Yacine Taleb, Ryan Stutsman, Gabriel Antoniu, Toni Cortes
Tailwind: Fast and Atomic RDMA-based Replication Yacine Taleb, Ryan Stutsman, Gabriel Antoniu, Toni Cortes In-Memory Key-Value Stores General purpose in-memory key-value stores are widely used nowadays
More informationPebblesDB: Building Key-Value Stores using Fragmented Log Structured Merge Trees
PebblesDB: Building Key-Value Stores using Fragmented Log Structured Merge Trees Pandian Raju 1, Rohan Kadekodi 1, Vijay Chidambaram 1,2, Ittai Abraham 2 1 The University of Texas at Austin 2 VMware Research
More informationWorkload-Aware Elastic Striping With Hot Data Identification for SSD RAID Arrays
IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, VOL. 36, NO. 5, MAY 2017 815 Workload-Aware Elastic Striping With Hot Data Identification for SSD RAID Arrays Yongkun Li,
More informationPurity: building fast, highly-available enterprise flash storage from commodity components
Purity: building fast, highly-available enterprise flash storage from commodity components J. Colgrove, J. Davis, J. Hayes, E. Miller, C. Sandvig, R. Sears, A. Tamches, N. Vachharajani, and F. Wang 0 Gala
More informationSYSTEM UPGRADE, INC Making Good Computers Better. System Upgrade Teaches RAID
System Upgrade Teaches RAID In the growing computer industry we often find it difficult to keep track of the everyday changes in technology. At System Upgrade, Inc it is our goal and mission to provide
More informationThe Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 presented by Kun Suo Outline GFS Background, Concepts and Key words Example of GFS Operations Some optimizations in
More informationPhysical Representation of Files
Physical Representation of Files A disk drive consists of a disk pack containing one or more platters stacked like phonograph records. Information is stored on both sides of the platter. Each platter is
More informationGFS: The Google File System
GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one
More informationIntroduction to I/O and Disk Management
1 Secondary Storage Management Disks just like memory, only different Introduction to I/O and Disk Management Why have disks? Ø Memory is small. Disks are large. Short term storage for memory contents
More informationCloudian Sizing and Architecture Guidelines
Cloudian Sizing and Architecture Guidelines The purpose of this document is to detail the key design parameters that should be considered when designing a Cloudian HyperStore architecture. The primary
More informationIntroduction to I/O and Disk Management
Introduction to I/O and Disk Management 1 Secondary Storage Management Disks just like memory, only different Why have disks? Ø Memory is small. Disks are large. Short term storage for memory contents
More informationStorage Devices for Database Systems
Storage Devices for Database Systems 5DV120 Database System Principles Umeå University Department of Computing Science Stephen J. Hegner hegner@cs.umu.se http://www.cs.umu.se/~hegner Storage Devices for
More informationOn the Speedup of Single-Disk Failure Recovery in XOR-Coded Storage Systems: Theory and Practice
On the Speedup of Single-Disk Failure Recovery in XOR-Coded Storage Systems: Theory and Practice Yunfeng Zhu, Patrick P. C. Lee, Yuchong Hu, Liping Xiang, and Yinlong Xu University of Science and Technology
More informationIBM Spectrum NAS. Easy-to-manage software-defined file storage for the enterprise. Overview. Highlights
IBM Spectrum NAS Easy-to-manage software-defined file storage for the enterprise Highlights Reduce capital expenditures with storage software on commodity servers Improve efficiency by consolidating all
More informationConcepts Introduced in Chapter 6. Warehouse-Scale Computers. Programming Models for WSCs. Important Design Factors for WSCs
Concepts Introduced in Chapter 6 Warehouse-Scale Computers A cluster is a collection of desktop computers or servers connected together by a local area network to act as a single larger computer. introduction
More informationNear Memory Key/Value Lookup Acceleration MemSys 2017
Near Key/Value Lookup Acceleration MemSys 2017 October 3, 2017 Scott Lloyd, Maya Gokhale Center for Applied Scientific Computing This work was performed under the auspices of the U.S. Department of Energy
More informationPYTHIA: Improving Datacenter Utilization via Precise Contention Prediction for Multiple Co-located Workloads
PYTHIA: Improving Datacenter Utilization via Precise Contention Prediction for Multiple Co-located Workloads Ran Xu (Purdue), Subrata Mitra (Adobe Research), Jason Rahman (Facebook), Peter Bai (Purdue),
More informationCS5460: Operating Systems Lecture 20: File System Reliability
CS5460: Operating Systems Lecture 20: File System Reliability File System Optimizations Modern Historic Technique Disk buffer cache Aggregated disk I/O Prefetching Disk head scheduling Disk interleaving
More informationStaggeringly Large Filesystems
Staggeringly Large Filesystems Evan Danaher CS 6410 - October 27, 2009 Outline 1 Large Filesystems 2 GFS 3 Pond Outline 1 Large Filesystems 2 GFS 3 Pond Internet Scale Web 2.0 GFS Thousands of machines
More informationC 1. Recap. CSE 486/586 Distributed Systems Distributed File Systems. Traditional Distributed File Systems. Local File Systems.
Recap CSE 486/586 Distributed Systems Distributed File Systems Optimistic quorum Distributed transactions with replication One copy serializability Primary copy replication Read-one/write-all replication
More informationHigh Performance Computing Course Notes High Performance Storage
High Performance Computing Course Notes 2008-2009 2009 High Performance Storage Storage devices Primary storage: register (1 CPU cycle, a few ns) Cache (10-200 cycles, 0.02-0.5us) Main memory Local main
More informationCurrent Topics in OS Research. So, what s hot?
Current Topics in OS Research COMP7840 OSDI Current OS Research 0 So, what s hot? Operating systems have been around for a long time in many forms for different types of devices It is normally general
More informationPrincipled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models
Principled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models Suli Yang*, Jing Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau * work done while at UW-Madison
More informationEngineering Fault-Tolerant TCP/IP servers using FT-TCP. Dmitrii Zagorodnov University of California San Diego
Engineering Fault-Tolerant TCP/IP servers using FT-TCP Dmitrii Zagorodnov University of California San Diego Motivation Reliable network services are desirable but costly! Extra and/or specialized hardware
More informationAnti-Caching: A New Approach to Database Management System Architecture. Guide: Helly Patel ( ) Dr. Sunnie Chung Kush Patel ( )
Anti-Caching: A New Approach to Database Management System Architecture Guide: Helly Patel (2655077) Dr. Sunnie Chung Kush Patel (2641883) Abstract Earlier DBMS blocks stored on disk, with a main memory
More informationA2E: Adaptively Aggressive Energy Efficient DVFS Scheduling for Data Intensive Applications
A2E: Adaptively Aggressive Energy Efficient DVFS Scheduling for Data Intensive Applications Li Tan 1, Zizhong Chen 1, Ziliang Zong 2, Rong Ge 3, and Dong Li 4 1 University of California, Riverside 2 Texas
More information1. Introduction. Traditionally, a high bandwidth file system comprises a supercomputer with disks connected
1. Introduction Traditionally, a high bandwidth file system comprises a supercomputer with disks connected by a high speed backplane bus such as SCSI [3][4] or Fibre Channel [2][67][71]. These systems
More informationHP Dynamic Deduplication achieving a 50:1 ratio
HP Dynamic Deduplication achieving a 50:1 ratio Table of contents Introduction... 2 Data deduplication the hottest topic in data protection... 2 The benefits of data deduplication... 2 How does data deduplication
More informationEC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures
EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures Haiyang Shi, Xiaoyi Lu, and Dhabaleswar K. (DK) Panda {shi.876, lu.932, panda.2}@osu.edu The Ohio State University
More informationRAMCube: Exploiting Network Proximity for RAM-Based Key-Value Store
RAMCube: Exploiting Network Proximity for RAM-Based Key-Value Store Yiming Zhang, Rui Chu @ NUDT Chuanxiong Guo, Guohan Lu, Yongqiang Xiong, Haitao Wu @ MSRA June, 2012 1 Background Disk-based storage
More informationFuxiSort. Jiamang Wang, Yongjun Wu, Hua Cai, Zhipeng Tang, Zhiqiang Lv, Bin Lu, Yangyu Tao, Chao Li, Jingren Zhou, Hong Tang Alibaba Group Inc
Fuxi Jiamang Wang, Yongjun Wu, Hua Cai, Zhipeng Tang, Zhiqiang Lv, Bin Lu, Yangyu Tao, Chao Li, Jingren Zhou, Hong Tang Alibaba Group Inc {jiamang.wang, yongjun.wyj, hua.caihua, zhipeng.tzp, zhiqiang.lv,
More informationDiskReduce: Making Room for More Data on DISCs. Wittawat Tantisiriroj
DiskReduce: Making Room for More Data on DISCs Wittawat Tantisiriroj Lin Xiao, in Fan, and Garth Gibson PARALLEL DATA LAORATORY Carnegie Mellon University GFS/HDFS Triplication GFS & HDFS triplicate every
More informationE-Store: Fine-Grained Elastic Partitioning for Distributed Transaction Processing Systems
E-Store: Fine-Grained Elastic Partitioning for Distributed Transaction Processing Systems Rebecca Taft, Essam Mansour, Marco Serafini, Jennie Duggan, Aaron J. Elmore, Ashraf Aboulnaga, Andrew Pavlo, Michael
More informationAdvanced Database Systems
Lecture II Storage Layer Kyumars Sheykh Esmaili Course s Syllabus Core Topics Storage Layer Query Processing and Optimization Transaction Management and Recovery Advanced Topics Cloud Computing and Web
More information