A Self-Designing Key-Value Store
|
|
|
- Augustine Stokes
- 5 years ago
- Views:
Transcription
1 A Self-Designing Key-Value Store Niv Dayan, Wilson Qin, Manos Athanassoulis, Stratos Idreos
2 storage is cheaper inserts & updates price per GB workload time
3 storage is cheaper inserts & updates price per GB workload time need for write-optimized database structures
4 need for write-optimized database structures LSM-tree invented 1996 now time
5 need for write-optimized database structures Key-Value Stores LSM-tree invented 1996 now time
6 Key-Value Stores
7 workload hardware Key-Value Stores? performance
8 workload hardware Key-Value Stores? performance
9 workload hardware Suboptimal Design Key-Value Stores? performance
10 Map Trade-Offs Navigate What-if?
11 LSM-tree Key-Value Stores What are they really?
12 updates memory storage level buffer 0
13 level updates memory storage buffer sort & flush runs 0 1
14 updates memory storage 0 buffer sort & flush 1 runs 2 sort-merge
15 level memory storage 0 buffer exponentially increasing capacities 1 O(log(N)) levels 2 3
16 lookup level key X memory storage fence 0 buffer pointers 1 one I/O per run 2 3 X
17 lookup level key X memory storage fence 0 buffer pointers 1 one I/O per run 2 3 X
18 lookup level key X memory storage Bloom fence 0 buffer filters pointers X
19 lookup level key X memory storage Bloom fence 0 buffer filters pointers 1 true negative 2 3 X
20 lookup level key X memory storage Bloom fence 0 buffer filters pointers 1 2 true negative false positive 3 X
21 lookup level key X memory storage Bloom fence 0 buffer filters pointers true negative false positive true positive X
22 Performance & Cost Tradeoffs lookup level key X memory storage Bloom fence 0 buffer filters pointers true negative false positive true positive X
23 Performance & Cost Tradeoffs lookup level key X memory storage Bloom fence 0 buffer filters pointers true negative false positive true positive X bigger filters fewer false positives
24 Performance & Cost Tradeoffs lookup level key X memory storage Bloom fence 0 buffer filters pointers true negative false positive true positive X bigger filters fewer false positives memory vs. lookups
25 Performance & Cost Tradeoffs lookup level key X memory storage Bloom fence 0 buffer filters pointers more merging fewer runs true negative false positive true positive X bigger filters fewer false positives memory vs. lookups
26 Performance & Cost Tradeoffs lookup level key X memory storage Bloom fence 0 buffer filters pointers more merging fewer runs true negative false positive true positive lookups vs. updates X bigger filters fewer false positives memory vs. lookups
27 main memory lookup cost update cost
28 main fixed memory memory lookup cost existing systems lookup cost update cost update cost
29 main fixed memory memory lookup cost merge less existing systems merge lookup cost update cost more update cost
30 main memory lookup cost less memory lookup cost update cost more memory update cost
31 Problem 1: Problem 2: lookup cost existing systems update cost
32 Problem 1: Problem 2: suboptimal design lookup cost existing systems update cost
33 fixed memory x Problem 1: suboptimal design lookup cost existing systems Problem 2: Pareto frontier update cost x
34 x Problem 1: Problem 2: suboptimal design hard to tune lookup cost x update cost
35 x Bloom filters size lookups vs. memory Problem 1: Problem 2: suboptimal design hard to tune lookup cost x update cost
36 x Problem 1: suboptimal design lookup merge policy greed cost max throughput lookups vs. updates Problem 2: hard to tune x update cost
37 Monkey: Optimal Navigable Key-Value Store Niv Dayan, Manos Athanasollis, Stratos Idreos SIGMOD 2017 Best of SIGMOD
38 Monkey: Optimal Navigable Key-Value Store observations: c insights: steps:
39 Monkey: Optimal Navigable Key-Value Store filters observations: fixed false positive rates c insights: lookup cost = pi suboptimal optimize allocation steps: asymptotically better memory vs. lookups
40 Monkey: Optimal Navigable Key-Value Store filters LSM-tree merge observations: fixed false positive rates policy? performance c insights: lookup cost = pi suboptimal log sorted array steps: optimize allocation asymptotically better memory vs. lookups updates vs. lookups navigate
41 Monkey: Optimal Navigable Key-Value Store filters LSM-tree merge ad-hoc memory policy trade-offs observations: fixed false positive rates? performance lookups updates c lookup cost = pi log lookup existing insights: cost sorted Monkey suboptimal array update cost optimize allocation steps: updates vs. lookups answer what-if asymptotically better design questions navigate memory vs. lookups
42 for fixed memory WiredTiger lookup cost Cassandra, HBase RocksDB, LevelDB Monkey Pareto frontier update cost
43 for fixed memory lookup cost max throughput WiredTiger Cassandra, HBase RocksDB, LevelDB Monkey Pareto frontier update cost
44 Monkey: Optimal Navigable Key-Value Store filters LSM-tree merge ad-hoc memory policy trade-offs observations: fixed false positive rates? performance lookups updates c lookup cost = pi log lookup existing insights: cost sorted Monkey suboptimal array update cost optimize allocation steps: updates vs. lookups answer what-if asymptotically better design questions navigate memory vs. lookups
45 Monkey: Optimal Navigable Key-Value Store filters LSM-tree merge ad-hoc memory policy trade-offs observations: fixed false positive rates? performance lookups updates c lookup cost = pi log lookup existing insights: cost sorted Monkey suboptimal array update cost optimize allocation steps: updates vs. lookups answer what-if asymptotically better design questions navigate memory vs. lookups
46 memory storage buffer Bloom filters fence pointers data f
47 memory storage buffer Bloom filters fence pointers data f
48 memory storage buffer Bloom filters fence pointers data f
49 memory storage Bloom fence buffer < > data filters pointers f
50 memory storage Bloom filters data f
51 memory storage X bits per entry Bloom filters data f
52 memory storage X bits per entry Bloom filters data f
53 memory X bits per entry Bloom filters false positive rate p = e bits M - ln(2) 2 entries N
54 memory X bits per entry Bloom filters p p p false positive rate p = e bits M - ln(2) 2 entries N
55 worst-case I/O overhead: memory X bits per entry Bloom filters p p p false positive rate p = e bits M - ln(2) 2 entries N
56 worst-case I/O overhead: O( p ) memory X bits per entry Bloom filters p p p false positive rate p = e bits M - ln(2) 2 entries N
57 worst-case I/O overhead: O( p ) memory X bits per entry Bloom filters p - bits M ln(2) 2 entries N p false = e positive rate p p
58 worst-case I/O overhead: O( e -M/N ) memory X bits per entry Bloom filters p p p false positive rate p = e bits M - ln(2) 2 entries N
59 worst-case I/O overhead: O( e -M/N ) memory X bits per entry Bloom filters p O(log(N)) p p
60 worst-case I/O overhead: O( log(n) e -M/N ) memory X bits per entry Bloom filters p O(log(N)) p p
61 worst-case I/O overhead: O( log(n) e -M/N ) memory X bits per entry Can we do better? Bloom filters p p p
62 lookup Bloom fence data key X filters pointers runs
63 lookup Bloom fence data key X filters pointers runs false positive I/O false positive I/O false positive false positive I/O I/O false positive I/O
64 lookup Bloom fence data key X filters pointers runs false positive I/O false positive I/O false positive I/O false positive I/O most memory false positive I/O
65 lookup Bloom fence data key X filters pointers runs false positive false positive I/O I/O most memory saves false at most 1 I/O I/O positive false positive I/O most memory false positive I/O
66 Bloom filters false positive rates reallocate some most memory
67 same Bloom memory, fewer lookup I/Os filters false positive rates reallocate some most memory
68 relax false positive rates 0 < p2 < 1 0 < p1 < 1 xx xx 0 < p0 < 1
69 relax false positive rates 0 < p2 < 1 0 < p1 < 1 0 < p0 < 1
70 relax model false positive rates lookup cost = f(p0, p1 ) 0 < p2 < 1 0 < p1 < 1 0 < p0 < 1 memory footprint = f(p0, p1 )
71 relax model optimize false positive rates lookup cost = f(p0, p1 ) 0 < p2 < 1 0 < p1 < 1 0 < p0 < 1 memory footprint = f(p0, p1 ) in terms of p0, p1
72 Bloom filters false p2 positive p1 rates p0 lookup cost = pi
73 memory Bloom filters footprint p2 false positive p1 bits - ln(2) 2 rates p0 false positive rate = e entries lookup cost = pi
74 memory Bloom filters footprint p2 false positive rates p1 p0 bits = - ln( false ) positive rate entries ln(2) 2 lookup cost = pi
75 memory Bloom filters footprint p2 bits(p2, N/T 2 ) false positive rates p1 p0 bits(p1, N/T) bits(p0, N) lookup cost = pi
76 memory Bloom filters footprint p2 bits(p2, N/T 2 ) false positive rates p1 p0 bits(p1, N/T) bits(p0, N) lookup cost = pi memory = - c N ln(p i) T i false positive rates size ratio constant entries
77 Bloom filters p2 optimize false positive p1 rates p0 lookup cost = pi memory = c N - ln(p i) T i
78 Monkey Bloom filters p0/t 2 exponential decrease false positive p0/t rates p0
79 Monkey Bloom filters State-of-the-Art Bloom filters false p0/t 2 exponential decrease p same positive rates p0/t p p0 p
80 Monkey Bloom filters State-of-the-Art Bloom filters < p0/t 2 < p false positive p0/t < p rates p0 > p lookup cost = pi < = p
81 Monkey Bloom filters State-of-the-Art Bloom filters < p0/t 2 < p false positive p0/t < p rates p0 > p lookup cost = pi < = p = O( e -M/N ) = O( log(n) e -M/N ) N number of entries M overall memory for Bloom filters
82 Monkey Bloom filters State-of-the-Art Bloom filters < p0/t 2 < p false positive p0/t < p rates p0 > p lookup cost = pi < = p = O( e -M/N ) = O( log(n) e -M/N ) asymptotic win N number of entries M overall memory for Bloom filters lookup cost increases at slower rate as data grows
83 Monkey Bloom filters false positive rates p0/t 2 p0/t p0 convergent geometric series
84 Monkey Bloom filters false p0/t 2 positive p0/t rates p0 - ln(pi) memory = c entries T i
85 Monkey Bloom filters false p0/t 2 positive p0/t rates p0 memory = c entries -ln(lookup cost)
86 Monkey Bloom filters false p0/t 2 positive p0/t rates p0 memory = c entries -ln(lookup cost) model lookups vs. memory trade-off
87 fixed memory Problem 1: Problem 2: suboptimal filters allocation hard to tune lookup cost existing systems update cost
88 fixed memory x Problem 1: Problem 2: suboptimal filters allocation hard to tune lookup cost existing systems x update cost
89 x Bloom filters size lookups vs. memory Problem 1: Problem 2: suboptimal filters allocation hard to tune lookup cost x update cost
90 x merge policy greed Problem 1: suboptimal filters allocation lookup max throughput lookups vs. updates cost Problem 2: hard to tune x update cost
91 Monkey: Optimal Navigable Key-Value Store filters LSM-tree merge ad-hoc memory policy trade-offs observations: fixed false positive rates? performance lookups updates c lookup cost = pi log lookup existing insights: cost sorted Monkey suboptimal array update cost optimize allocation steps: updates vs. lookups answer what-if asymptotically better design questions navigate memory vs. lookups
92 Monkey: Optimal Navigable Key-Value Store filters LSM-tree merge ad-hoc memory policy trade-offs observations: fixed false positive rates? performance lookups updates lookup cost = pi log c lookup existing insights: cost sorted Monkey suboptimal array update cost optimize allocation steps: updates vs. lookups answer what-if asymptotically better design questions navigate memory vs. lookups
93 Identify merge policy size ratio
94 Identify Map merge policy size ratio lookups updates
95 Identify Map merge policy sorted log size ratio lookups LSM-tree array updates
96 Identify Map merge policy log size ratio lookups sorted array updates
97 Identify Map Navigate merge policy log workload hardware size ratio lookups sorted array updates maximum optimal throughout
98 Merge Policies Tiering write-optimized Leveling read-optimized
99 Tiering write-optimized Leveling read-optimized
100 Tiering write-optimized Leveling read-optimized T runs per level
101 Tiering write-optimized Leveling read-optimized T runs per level merge & flush
102 Tiering write-optimized Leveling read-optimized T runs per level
103 Tiering write-optimized Leveling read-optimized T runs per level merge
104 Tiering write-optimized Leveling read-optimized T runs per level T times bigger flush
105 Tiering write-optimized Leveling read-optimized T runs per level T times bigger
106 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level
107 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level lookup cost: O(T logt(n) e -M/N ) O(logT(N) e -M/N ) runs per level levels false positive rate levels false positive rate
108 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level lookup cost: O(T logt(n) e -M/N ) O(logT(N) e -M/N ) runs per level levels false positive rate levels false positive rate
109 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level lookup cost: O(T e -M/N ) O(e -M/N ) runs per level false positive rate false positive rate
110 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level lookup cost: O(T e -M/N ) O(e -M/N ) update cost: O(logT(N)) O(T logt(n)) levels merges per level levels
111 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level lookup cost: O(T e -M/N ) O(e -M/N ) update cost: O(logT(N)) O(T logt(n)) size ratio T
112 Tiering write-optimized Leveling read-optimized 1 run per level 1 run per level lookup cost: O(e -M/N ) = O(e -M/N ) update cost: O(log(N)) = O(log(N)) size ratio T
113 Tiering write-optimized Leveling read-optimized T runs per level 1 run per level lookup cost: O(T e -M/N ) O(e -M/N ) update cost: O(logT(N)) O(T logt(n)) size ratio T
114 Tiering write-optimized Leveling read-optimized O(lNl) runs per level 1 run per level lookup cost: O(Na e -M/N ) O(e -M/N ) update cost: O(1) O(N) size ratio T
115 Tiering write-optimized Leveling read-optimized O(lNl) runs per level 1 run per level lookup cost: log O(Na e -M/N ) sorted array O(e -M/N ) update cost: O(1) O(N) size ratio T
116 log lookup Tiering cost Leveling sorted array update cost
117 log lookup Tiering cost T=2 Leveling update cost sorted array T size ratio
118 log lookup Tiering cost log T=2 LSM-tree Leveling sorted array update cost sorted array T size ratio
119 log workload hardware lookup Tiering cost T=2 Leveling sorted maximum optimal throughout array update cost T size ratio
120 x merge policy greed Problem 1: suboptimal filters allocation lookup lookups vs. updates cost Problem 2: hard to tune max throughput x update cost
121
122 better asymptotic scalability lookup latency (ms) LevelDB Monkey number of entries (log scale)
123 better asymptotic scalability workload adaptability lookup latency (ms) LevelDB Monkey throughput T4 T2L L4 L4 navigable Monkey L6 L6 L8 L8 L16 fixed Monkey LevelDB (F) number of entries (log scale) % lookups in workload
124 Map Trade-Offs Navigate What-if?
125 Map Trade-Offs Navigate What-if? Thanks!
Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging
Dostoevsky: Better Space-ime rade-offs for LSM-ree Based Key-Value Stores via Adaptive Removal of Superfluous Merging Niv Dayan, Stratos Idreos Harvard University ABSRAC We show that all mainstream LSM-tree
PebblesDB: Building Key-Value Stores using Fragmented Log Structured Merge Trees
PebblesDB: Building Key-Value Stores using Fragmented Log Structured Merge Trees Pandian Raju 1, Rohan Kadekodi 1, Vijay Chidambaram 1,2, Ittai Abraham 2 1 The University of Texas at Austin 2 VMware Research
The Right Read Optimization is Actually Write Optimization. Leif Walsh
The Right Read Optimization is Actually Write Optimization Leif Walsh leif@tokutek.com The Right Read Optimization is Write Optimization Situation: I have some data. I want to learn things about the world,
A Light-weight Compaction Tree to Reduce I/O Amplification toward Efficient Key-Value Stores
A Light-weight Compaction Tree to Reduce I/O Amplification toward Efficient Key-Value Stores T i n g Y a o 1, J i g u a n g W a n 1, P i n g H u a n g 2, X u b i n He 2, Q i n g x i n G u i 1, F e i W
How To Rock with MyRocks. Vadim Tkachenko CTO, Percona Webinar, Jan
How To Rock with MyRocks Vadim Tkachenko CTO, Percona Webinar, Jan-16 2019 Agenda MyRocks intro and internals MyRocks limitations Benchmarks: When to choose MyRocks over InnoDB Tuning for the best results
RocksDB Key-Value Store Optimized For Flash
RocksDB Key-Value Store Optimized For Flash Siying Dong Software Engineer, Database Engineering Team @ Facebook April 20, 2016 Agenda 1 What is RocksDB? 2 RocksDB Design 3 Other Features What is RocksDB?
YCSB++ Benchmarking Tool Performance Debugging Advanced Features of Scalable Table Stores
YCSB++ Benchmarking Tool Performance Debugging Advanced Features of Scalable Table Stores Swapnil Patil Milo Polte, Wittawat Tantisiriroj, Kai Ren, Lin Xiao, Julio Lopez, Garth Gibson, Adam Fuchs *, Billie
systems & research project
class 4 systems & research project prof. HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS265/ index index knows order about the data data filtering data: point/range queries index data A B C sorted A B C initial
CSE-E5430 Scalable Cloud Computing Lecture 9
CSE-E5430 Scalable Cloud Computing Lecture 9 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 15.11-2015 1/24 BigTable Described in the paper: Fay
CACHE-OBLIVIOUS MAPS. Edward Kmett McGraw Hill Financial. Saturday, October 26, 13
CACHE-OBLIVIOUS MAPS Edward Kmett McGraw Hill Financial CACHE-OBLIVIOUS MAPS Edward Kmett McGraw Hill Financial CACHE-OBLIVIOUS MAPS Indexing and Machine Models Cache-Oblivious Lookahead Arrays Amortization
SLM-DB: Single-Level Key-Value Store with Persistent Memory
SLM-DB: Single-Level Key-Value Store with Persistent Memory Olzhas Kaiyrakhmet and Songyi Lee, UNIST; Beomseok Nam, Sungkyunkwan University; Sam H. Noh and Young-ri Choi, UNIST https://www.usenix.org/conference/fast19/presentation/kaiyrakhmet
The Skiplist-Based LSM Tree
The Skiplist-Based LSM Tree ARON SZANTO, Harvard University Log-Structured Merge (LSM) Trees provide a tiered data storage and retrieval paradigm that is aractive for write-optimized data systems. Maintaining
LSM-trie: An LSM-tree-based Ultra-Large Key-Value Store for Small Data
LSM-trie: An LSM-tree-based Ultra-Large Key-Value Store for Small Data Xingbo Wu Yuehai Xu Song Jiang Zili Shao The Hong Kong Polytechnic University The Challenge on Today s Key-Value Store Trends on workloads
SILT: A MEMORY-EFFICIENT, HIGH-PERFORMANCE KEY- VALUE STORE PRESENTED BY PRIYA SRIDHAR
SILT: A MEMORY-EFFICIENT, HIGH-PERFORMANCE KEY- VALUE STORE PRESENTED BY PRIYA SRIDHAR AGENDA INTRODUCTION Why SILT? MOTIVATION SILT KV STORAGE SYSTEM LOW READ AMPLIFICATION CONTROLLABLE WRITE AMPLIFICATION
HashKV: Enabling Efficient Updates in KV Storage via Hashing
HashKV: Enabling Efficient Updates in KV Storage via Hashing Helen H. W. Chan, Yongkun Li, Patrick P. C. Lee, Yinlong Xu The Chinese University of Hong Kong University of Science and Technology of China
Atlas: Baidu s Key-value Storage System for Cloud Data
Atlas: Baidu s Key-value Storage System for Cloud Data Song Jiang Chunbo Lai Shiding Lin Liqiong Yang Guangyu Sun Jason Cong Wayne State University Zhenyu Hou Can Cui Peking University University of California
TokuDB vs RocksDB. What to choose between two write-optimized DB engines supported by Percona. George O. Lorch III Vlad Lesin
TokuDB vs RocksDB What to choose between two write-optimized DB engines supported by Percona George O. Lorch III Vlad Lesin What to compare? Amplification Write amplification Read amplification Space amplification
A Comparative Study of Secondary Indexing Techniques in LSM-based NoSQL Databases
A Comparative Study of Secondary Indexing Techniques in LSM-based NoSQL Databases Mohiuddin Abdul Qader, Shiwen Cheng, Vagelis Hristidis {mabdu002,schen064,vagelis}@cs.ucr.edu Department of Computer Science
FUNCTIONALLY OBLIVIOUS (AND SUCCINCT) Edward Kmett
FUNCTIONALLY OBLIVIOUS (AND SUCCINCT) Edward Kmett BUILDING BETTER TOOLS Cache-Oblivious Algorithms Succinct Data Structures RAM MODEL Almost everything you do in Haskell assumes this model Good for ADTs,
Scalable and Secure Internet Services and Architecture PRESENTATION REPORT Semester: Winter 2015 Course: ECE 7650
Scalable and Secure Internet Services and Architecture PRESENTATION REPORT Semester: Winter 2015 Course: ECE 7650 SUBMITTED BY: Yashwanth Boddu fq9316@wayne.edu (1) Our base design uses less than 1 byte
arxiv: v1 [cs.db] 25 Nov 2018
![arxiv: v1 [cs.db] 25 Nov 2018](/thumbs/91/105669237.jpg "arxiv: v1 [cs.db] 25 Nov 2018")
Enabling Efficient Updates in KV Storage via Hashing: Design and Performance Evaluation Yongkun Li, Helen H. W. Chan, Patrick P. C. Lee, and Yinlong Xu University of Science and Technology of China The
Flash Storage Complementing a Data Lake for Real-Time Insight
Flash Storage Complementing a Data Lake for Real-Time Insight Dr. Sanhita Sarkar Global Director, Analytics Software Development August 7, 2018 Agenda 1 2 3 4 5 Delivering insight along the entire spectrum
WiscKey: Separating Keys from Values in SSD-Conscious Storage
WiscKey: Separating Keys from Values in SSD-Conscious Storage LANYUE LU, THANUMALAYAN SANKARANARAYANA PILLAI, HARIHARAN GOPALAKRISHNAN, ANDREA C. ARPACI-DUSSEAU, and REMZI H. ARPACI-DUSSEAU, University
class 13 scans vs indexes prof. Stratos Idreos
class 13 scans vs indexes prof. Stratos Idreos HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/ b-tree - dynamic tree - always balanced 35,50 35, 12,20 50, 1,2,3 12,15,17 20, Stratos Idreos 2 /24 select from
An Adaptive Stabilization Framework for DHT
An Adaptive Stabilization Framework for DHT Gabriel Ghinita, Yong Meng Teo Department of Computer Science National University of Singapore {ghinitag,teoym}@comp.nus.edu.sg Overview Background Related Work
Algorithm Performance Factors. Memory Performance of Algorithms. Processor-Memory Performance Gap. Moore s Law. Program Model of Memory II
Memory Performance of Algorithms CSE 32 Data Structures Lecture Algorithm Performance Factors Algorithm choices (asymptotic running time) O(n 2 ) or O(n log n) Data structure choices List or Arrays Language
FaRM: Fast Remote Memory
FaRM: Fast Remote Memory Problem Context DRAM prices have decreased significantly Cost effective to build commodity servers w/hundreds of GBs E.g. - cluster with 100 machines can hold tens of TBs of main
Suppose you are accessing elements of an array: ... or suppose you are dereferencing pointers: temp->next->next = elem->prev->prev;
CSE 100: B-TREE Memory accesses Suppose you are accessing elements of an array: if ( a[i] < a[j] ) {... or suppose you are dereferencing pointers: temp->next->next = elem->prev->prev;... or in general
On Log-Structured Merge for Solid-State Drives
On Log-Structured Merge for Solid-State Drives Risi Thonangi VMware Inc. rthonangi@vmware.com Jun Yang Duke University junyang@cs.duke.edu Abstract Log-structure merge (LSM) is an increasingly prevalent
LSbM-tree: Re-enabling Buffer Caching in Data Management for Mixed Reads and Writes
27 IEEE 37th International Conference on Distributed Computing Systems LSbM-tree: Re-enabling Buffer Caching in Data Management for Mixed Reads and Writes Dejun Teng, Lei Guo, Rubao Lee, Feng Chen, Siyuan
YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores
YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores Swapnil Patil M. Polte, W. Tantisiriroj, K. Ren, L.Xiao, J. Lopez, G.Gibson, A. Fuchs *, B. Rinaldi * Carnegie
Best Practices for Validating the Performance of Data Center Infrastructure. Henry He Ixia
Best Practices for Validating the Performance of Data Center Infrastructure Henry He Ixia Game Changers Big data - the world is getting hungrier and hungrier for data 2.5B pieces of content 500+ TB ingested
Hiroki Yasuga, Elisabeth Kolp, Andreas Lang. 25th September 2014, Scientific Programming
Hiroki Yasuga, Elisabeth Kolp, Andreas Lang 25th September 2014, Scientific Programming What is sorting and complexity? Big O notation Sorting algorithms: Merge sort Quick sort Comparison: Merge sort &
Write-Optimization in B-Trees. Bradley C. Kuszmaul MIT & Tokutek
Write-Optimization in B-Trees Bradley C. Kuszmaul MIT & Tokutek The Setup What and Why B-trees? B-trees are a family of data structures. B-trees implement an ordered map. Implement GET, PUT, NEXT, PREV.
Indexing: Overview & Hashing. CS 377: Database Systems
Indexing: Overview & Hashing CS 377: Database Systems Recap: Data Storage Data items Records Memory DBMS Blocks blocks Files Different ways to organize files for better performance Disk Motivation for
Join Processing for Flash SSDs: Remembering Past Lessons
Join Processing for Flash SSDs: Remembering Past Lessons Jaeyoung Do, Jignesh M. Patel Department of Computer Sciences University of Wisconsin-Madison $/MB GB Flash Solid State Drives (SSDs) Benefits of
CS 110 Computer Architecture
CS 110 Computer Architecture Performance and Floating Point Arithmetic Instructor: Sören Schwertfeger http://shtech.org/courses/ca/ School of Information Science and Technology SIST ShanghaiTech University
UpBit: Scalable In-Memory Updatable Bitmap Indexing. Guanting Chen, Yuan Zhang 2/21/2019
UpBit: Scalable In-Memory Updatable Bitmap Indexing Guanting Chen, Yuan Zhang 2/21/2019 BACKGROUND Some words to know Bitmap: a bitmap is a mapping from some domain to bits Bitmap Index: A bitmap index
PERFORMANCE ANALYSIS AND OPTIMIZATION OF SKIP LISTS FOR MODERN MULTI-CORE ARCHITECTURES
PERFORMANCE ANALYSIS AND OPTIMIZATION OF SKIP LISTS FOR MODERN MULTI-CORE ARCHITECTURES Anish Athalye and Patrick Long Mentors: Austin Clements and Stephen Tu 3 rd annual MIT PRIMES Conference Sequential
COS 318: Operating Systems. NSF, Snapshot, Dedup and Review
COS 318: Operating Systems NSF, Snapshot, Dedup and Review Topics! NFS! Case Study: NetApp File System! Deduplication storage system! Course review 2 Network File System! Sun introduced NFS v2 in early
MongoDB Revs You Up: What Storage Engine is Right for You?
MongoDB Revs You Up: What Storage Engine is Right for You? Jon Tobin, Director of Solution Eng. --------------------- Jon.Tobin@percona.com @jontobs Linkedin.com/in/jonathanetobin Agenda How did we get
Chapter 9 Memory Management
Contents 1. Introduction 2. Computer-System Structures 3. Operating-System Structures 4. Processes 5. Threads 6. CPU Scheduling 7. Process Synchronization 8. Deadlocks 9. Memory Management 10. Virtual
Scalable Enterprise Networks with Inexpensive Switches
Scalable Enterprise Networks with Inexpensive Switches Minlan Yu minlanyu@cs.princeton.edu Princeton University Joint work with Alex Fabrikant, Mike Freedman, Jennifer Rexford and Jia Wang 1 Enterprises
UNIT 1 ANALYSIS OF ALGORITHMS
UNIT 1 ANALYSIS OF ALGORITHMS Analysis of Algorithms Structure Page Nos. 1.0 Introduction 7 1.1 Objectives 7 1.2 Mathematical Background 8 1.3 Process of Analysis 12 1.4 Calculation of Storage Complexity
ElasticBF: Fine-grained and Elastic Bloom Filter Towards Efficient Read for LSM-tree-based KV Stores
: Fine-grained and Elastic Bloom Filter Towards Efficient Read for LSM-tree-based KV Stores Yueming Zhang 1,YongkunLi 1,2,FanGuo 1,ChengLi 1,2,YinlongXu 1,2 1 University of Science and Technology of China
Data Structures for IPv6 Network Traffic Analysis Using Sets and Bags. John McHugh, Ulfar Erlingsson
Data Structures for IPv6 Network Traffic Analysis Using Sets and Bags John McHugh, Ulfar Erlingsson The nature of the problem IPv4 has 2 32 possible addresses, IPv6 has 2 128. IPv4 sets can be realized
Dynamic Memory Allocation
1 Dynamic Memory Allocation Anne Bracy CS 3410 Computer Science Cornell University Note: these slides derive from those by Markus Püschel at CMU 2 Recommended Approach while (TRUE) { code a little; test
GearDB: A GC-free Key-Value Store on HM-SMR Drives with Gear Compaction
GearDB: A GC-free Key-Value Store on HM-SMR Drives with Gear Compaction Ting Yao, Huazhong University of Science and Technology and Temple University; Jiguang Wan, Huazhong University of Science and Technology;
Performance Benefits of Running RocksDB on Samsung NVMe SSDs
Performance Benefits of Running RocksDB on Samsung NVMe SSDs A Detailed Analysis 25 Samsung Semiconductor Inc. Executive Summary The industry has been experiencing an exponential data explosion over the
Redesigning LSMs for Nonvolatile Memory with NoveLSM
Redesigning LSMs for Nonvolatile Memory with NoveLSM Sudarsun Kannan, University of Wisconsin-Madison; Nitish Bhat and Ada Gavrilovska, Georgia Tech; Andrea Arpaci-Dusseau and Remzi Arpaci-Dusseau, University
class 12 b-trees 2.0 prof. Stratos Idreos
class 12 b-trees 2.0 prof. Stratos Idreos HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/ A B C A B C clustered/primary index on A Stratos Idreos /26 2 A B C A B C clustered/primary index on A pos C pos
Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching
Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching Kefei Wang and Feng Chen Louisiana State University SoCC '18 Carlsbad, CA Key-value Systems in Internet Services Key-value
An Efficient Memory-Mapped Key-Value Store for Flash Storage
An Efficient Memory-Mapped Key-Value Store for Flash Storage Anastasios Papagiannis, Giorgos Saloustros, Pilar González-Férez, and Angelos Bilas Institute of Computer Science (ICS) Foundation for Research
Intro to DB CHAPTER 12 INDEXING & HASHING
Intro to DB CHAPTER 12 INDEXING & HASHING Chapter 12: Indexing and Hashing Basic Concepts Ordered Indices B+-Tree Index Files B-Tree Index Files Static Hashing Dynamic Hashing Comparison of Ordered Indexing
The Role of Database Aware Flash Technologies in Accelerating Mission- Critical Databases
The Role of Database Aware Flash Technologies in Accelerating Mission- Critical Databases Gurmeet Goindi Principal Product Manager Oracle Flash Memory Summit 2013 Santa Clara, CA 1 Agenda Relational Database
SQL, NoSQL, MongoDB. CSE-291 (Cloud Computing) Fall 2016 Gregory Kesden
SQL, NoSQL, MongoDB CSE-291 (Cloud Computing) Fall 2016 Gregory Kesden SQL Databases Really better called Relational Databases Key construct is the Relation, a.k.a. the table Rows represent records Columns
TSAR : A Two Tier Sensor Storage Architecture using Interval Skip Graphs
TSAR : A Two Tier Sensor Storage Architecture using Interval Skip Graphs Authors: Peter Desnoyers, Deepak Ganesan, Prashant Shenoy ( University of Massachusetts, Amherst, MA) Presented by: Nikhil Raghavan
Making the Most of Hadoop with Optimized Data Compression (and Boost Performance) Mark Cusack. Chief Architect RainStor
Making the Most of Hadoop with Optimized Data Compression (and Boost Performance) Mark Cusack Chief Architect RainStor Agenda Importance of Hadoop + data compression Data compression techniques Compression,
IBM System Storage DS8870 Release R7.3 Performance Update
IBM System Storage DS8870 Release R7.3 Performance Update Enterprise Storage Performance Yan Xu Agenda Summary of DS8870 Hardware Changes I/O Performance of High Performance Flash Enclosure (HPFE) Easy
Codes, Bloom Filters, and Overlay Networks. Michael Mitzenmacher
Codes, Bloom Filters, and Overlay Networks Michael Mitzenmacher 1 Today... Erasure codes Digital Fountain Bloom Filters Summary Cache, Compressed Bloom Filters Informed Content Delivery Combining the two
Benchmarking Cloud Serving Systems with YCSB 詹剑锋 2012 年 6 月 27 日
Benchmarking Cloud Serving Systems with YCSB 詹剑锋 2012 年 6 月 27 日 Motivation There are many cloud DB and nosql systems out there PNUTS BigTable HBase, Hypertable, HTable Megastore Azure Cassandra Amazon
Presented by Nanditha Thinderu
Presented by Nanditha Thinderu Enterprise systems are highly distributed and heterogeneous which makes administration a complex task Application Performance Management tools developed to retrieve information
Lesson n.11 Data Structures for P2P Systems: Bloom Filters, Merkle Trees
Lesson n.11 : Bloom Filters, Merkle Trees Didactic Material Tutorial on Moodle 15/11/2013 1 SET MEMBERSHIP PROBLEM Let us consider the set S={s 1,s 2,...,s n } of n elements chosen from a very large universe
[ 11.2, 11.3, 11.4] Analysis of Algorithms. Complexity of Algorithms. 400 lecture note # Overview
![[ 11.2, 11.3, 11.4] Analysis of Algorithms. Complexity of Algorithms. 400 lecture note # Overview](/thumbs/93/113368535.jpg "[ 11.2, 11.3, 11.4] Analysis of Algorithms. Complexity of Algorithms. 400 lecture note # Overview")
400 lecture note #0 [.2,.3,.4] Analysis of Algorithms Complexity of Algorithms 0. Overview The complexity of an algorithm refers to the amount of time and/or space it requires to execute. The analysis
Dynamic Memory Allocation. Gerson Robboy Portland State University. class20.ppt
Dynamic Memory Allocation Gerson Robboy Portland State University class20.ppt Harsh Reality Memory is not unbounded It must be allocated and managed Many applications are memory dominated Especially those
Algorithm Performance Factors. Memory Performance of Algorithms. Processor-Memory Performance Gap. Moore s Law. Program Model of Memory I
Memory Performance of Algorithms CSE 32 Data Structures Lecture Algorithm Performance Factors Algorithm choices (asymptotic running time) O(n 2 ) or O(n log n) Data structure choices List or Arrays Language
Single-pass restore after a media failure. Caetano Sauer, Goetz Graefe, Theo Härder
Single-pass restore after a media failure Caetano Sauer, Goetz Graefe, Theo Härder 20% of drives fail after 4 years High failure rate on first year (factory defects) Expectation of 50% for 6 years https://www.backblaze.com/blog/how-long-do-disk-drives-last/
Write-Optimized and High-Performance Hashing Index Scheme for Persistent Memory
Write-Optimized and High-Performance Hashing Index Scheme for Persistent Memory Pengfei Zuo, Yu Hua, Jie Wu Huazhong University of Science and Technology, China 3th USENIX Symposium on Operating Systems
10 Million Smart Meter Data with Apache HBase
10 Million Smart Meter Data with Apache HBase 5/31/2017 OSS Solution Center Hitachi, Ltd. Masahiro Ito OSS Summit Japan 2017 Who am I? Masahiro Ito ( 伊藤雅博 ) Software Engineer at Hitachi, Ltd. Focus on
1. For the sieve technique we solve the problem, recursively mathematically precisely accurately 2. We do sorting to, keep elements in random positions keep the algorithm run in linear order keep the algorithm
Operating Systems Design Exam 2 Review: Spring 2011
Operating Systems Design Exam 2 Review: Spring 2011 Paul Krzyzanowski pxk@cs.rutgers.edu 1 Question 1 CPU utilization tends to be lower when: a. There are more processes in memory. b. There are fewer processes
- SLED: single large expensive disk - RAID: redundant array of (independent, inexpensive) disks
RAID and AutoRAID RAID background Problem: technology trends - computers getting larger, need more disk bandwidth - disk bandwidth not riding moore s law - faster CPU enables more computation to support
How TokuDB Fractal TreeTM. Indexes Work. Bradley C. Kuszmaul. MySQL UC 2010 How Fractal Trees Work 1
MySQL UC 2010 How Fractal Trees Work 1 How TokuDB Fractal TreeTM Indexes Work Bradley C. Kuszmaul MySQL UC 2010 How Fractal Trees Work 2 More Information You can download this talk and others at http://tokutek.com/technology
Data Processing at the Speed of 100 Gbps using Apache Crail. Patrick Stuedi IBM Research
Data Processing at the Speed of 100 Gbps using Apache Crail Patrick Stuedi IBM Research The CRAIL Project: Overview Data Processing Framework (e.g., Spark, TensorFlow, λ Compute) Spark-IO FS Albis Streaming
CS:3330 (22c:31) Algorithms
What s an Algorithm? CS:3330 (22c:31) Algorithms Introduction Computer Science is about problem solving using computers. Software is a solution to some problems. Algorithm is a design inside a software.
SED 762. Transcript EPISODE 762 [INTRODUCTION]
![SED 762. Transcript EPISODE 762 [INTRODUCTION]](/thumbs/94/119799702.jpg "SED 762. Transcript EPISODE 762 [INTRODUCTION]")
EPISODE 762 [INTRODUCTION] [00:00:00] JM: RocksDB is a storage engine based on the log-structured merge-tree data structure. RocksDB was developed at Facebook to provide a tool for embedded databases.
class 10 b-trees 2.0 prof. Stratos Idreos
class 10 b-trees 2.0 prof. Stratos Idreos HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/ CS Colloquium HV Jagadish Prof University of Michigan 10/6 Stratos Idreos /29 2 CS Colloquium Magdalena Balazinska
CS 416: Opera-ng Systems Design March 23, 2012
Question 1 Operating Systems Design Exam 2 Review: Spring 2011 Paul Krzyzanowski pxk@cs.rutgers.edu CPU utilization tends to be lower when: a. There are more processes in memory. b. There are fewer processes
CSC630/CSC730 Parallel & Distributed Computing
CSC630/CSC730 Parallel & Distributed Computing Analytical Modeling of Parallel Programs Chapter 5 1 Contents Sources of Parallel Overhead Performance Metrics Granularity and Data Mapping Scalability 2
C has been and will always remain on top for performancecritical
Check out this link: http://spectrum.ieee.org/static/interactive-the-top-programminglanguages-2016 C has been and will always remain on top for performancecritical applications: Implementing: Databases
OPS-23: OpenEdge Performance Basics
OPS-23: OpenEdge Performance Basics White Star Software adam@wss.com Agenda Goals of performance tuning Operating system setup OpenEdge setup Setting OpenEdge parameters Tuning APWs OpenEdge utilities
Java Without the Jitter
TECHNOLOGY WHITE PAPER Achieving Ultra-Low Latency Table of Contents Executive Summary... 3 Introduction... 4 Why Java Pauses Can t Be Tuned Away.... 5 Modern Servers Have Huge Capacities Why Hasn t Latency
High-Performance Packet Classification on GPU
High-Performance Packet Classification on GPU Shijie Zhou, Shreyas G. Singapura, and Viktor K. Prasanna Ming Hsieh Department of Electrical Engineering University of Southern California 1 Outline Introduction
External Sorting. Chapter 13. Comp 521 Files and Databases Fall
External Sorting Chapter 13 Comp 521 Files and Databases Fall 2012 1 Why Sort? A classic problem in computer science! Advantages of requesting data in sorted order gathers duplicates allows for efficient
Smooth Scan: Statistics-Oblivious Access Paths. Renata Borovica-Gajic Stratos Idreos Anastasia Ailamaki Marcin Zukowski Campbell Fraser
Smooth Scan: Statistics-Oblivious Access Paths Renata Borovica-Gajic Stratos Idreos Anastasia Ailamaki Marcin Zukowski Campbell Fraser Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q16 Q18 Q19 Q21 Q22
Tools for Social Networking Infrastructures
Tools for Social Networking Infrastructures 1 Cassandra - a decentralised structured storage system Problem : Facebook Inbox Search hundreds of millions of users distributed infrastructure inbox changes
Big and Fast. Anti-Caching in OLTP Systems. Justin DeBrabant
Big and Fast Anti-Caching in OLTP Systems Justin DeBrabant Online Transaction Processing transaction-oriented small footprint write-intensive 2 A bit of history 3 OLTP Through the Years relational model
Outline. runtime of programs algorithm efficiency Big-O notation List interface Array lists
Outline runtime of programs algorithm efficiency Big-O notation List interface Array lists Runtime of Programs compare the following two program fragments: int result = 1; int result = 1; for (int i=2;
Optimizing Space Amplification in RocksDB
Optimizing Space Amplification in RocksDB Siying Dong, Mark Callaghan, Leonidas Galanis, Dhruba Borthakur, Tony Savor and Michael Stumm 2 Facebook, Hacker Way, Menlo Park, CA USA 942 {siying.d, mcallaghan,
Question And Answer.
Q.1 What is the number of swaps required to sort n elements using selection sort, in the worst case? A. Θ(n) B. Θ(n log n) C. Θ(n2) D. Θ(n2 log n) ANSWER : Option A Θ(n) Note that we
Part II: Software Infrastructure in Data Centers: Key-Value Data Management Systems
CSE 6350 File and Storage System Infrastructure in Data centers Supporting Internet-wide Services Part II: Software Infrastructure in Data Centers: Key-Value Data Management Systems 1 Key-Value Store Clients
MUHAMMAD FAISAL MIT 4 th Semester Al-Barq Campus (VGJW01) Gujranwala
MUHAMMAD FAISAL MIT 4 th Semester Al-Barq Campus (VGJW01) Gujranwala faisalgrw123@gmail.com Reference MCQ s For MIDTERM EXAMS CS502- Design and Analysis of Algorithms 1. For the sieve technique we solve
CHAPTER 4 BLOOM FILTER
54 CHAPTER 4 BLOOM FILTER 4.1 INTRODUCTION Bloom filter was formulated by Bloom (1970) and is used widely today for different purposes including web caching, intrusion detection, content based routing,
Curriculum 2013 Knowledge Units Pertaining to PDC
Curriculum 2013 Knowledge Units Pertaining to C KA KU Tier Level NumC Learning Outcome Assembly level machine Describe how an instruction is executed in a classical von Neumann machine, with organization
Computer Sciences Department
Computer Sciences Department Building Cheap and Large CAMs Using BufferHash Ashok Anand Steven Kappes Aditya Akella Suman Nath Technical Report #65 February 29 Building Cheap and Large CAMs Using BufferHash
MILC: Inverted List Compression in Memory
MILC: Inverted List Compression in Memory Yorrick Müller Garching, 3rd December 2018 Yorrick Müller MILC: Inverted List Compression In Memory 1 Introduction Inverted Lists Inverted list := Series of sorted
CS/ENGRD 2110 Object-Oriented Programming and Data Structures Spring 2012 Thorsten Joachims. Lecture 10: Asymptotic Complexity and
CS/ENGRD 2110 Object-Oriented Programming and Data Structures Spring 2012 Thorsten Joachims Lecture 10: Asymptotic Complexity and What Makes a Good Algorithm? Suppose you have two possible algorithms or
Federated Array of Bricks Y Saito et al HP Labs. CS 6464 Presented by Avinash Kulkarni
Federated Array of Bricks Y Saito et al HP Labs CS 6464 Presented by Avinash Kulkarni Agenda Motivation Current Approaches FAB Design Protocols, Implementation, Optimizations Evaluation SSDs in enterprise
Main Memory (Part II)
Main Memory (Part II) Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) Main Memory 1393/8/17 1 / 50 Reminder Amir H. Payberah
vsan 6.6 Performance Improvements First Published On: Last Updated On:
vsan 6.6 Performance Improvements First Published On: 07-24-2017 Last Updated On: 07-28-2017 1 Table of Contents 1. Overview 1.1.Executive Summary 1.2.Introduction 2. vsan Testing Configuration and Conditions