Part II: Data Center Software Architecture: Topic 2: Key-value Data Management Systems. SkimpyStash: Key Value Store on Flash-based Storage

Transcription

1 ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 2: Key-value Data Management Systems SkimpyStash: Key Value Store on Flash-based Storage Presented by Aravind Mohan

2 Distinct Features Designed for High Throughput and Low Latency server applications. Low RAM at about 1 byte per key value pair, scale key-value stores on Flash on a lean RAM size budget. Implemented by using the Hash table directory in RAM to index (K,V) pairs stored in a log structured manner in Flash. Move most of the pointers that locate each key-value pair from RAM to Flash itself. (Break barrier of Flash pointer 4 bytes worth of RAM)

3 Background Server side applications require persistent and high throughput key-value stores to function. Flash Memory using Solid state devices getting popular in Desktop and Server applications.

4 Background MySpace.com switched from hard disk to flash storage as primary storage for data center operations. Facebook release Flash Cache by caching data in SSDs for faster read and writes. Challenges of Flash memory can be addressed by two approaches: 1)Device driver software level inside the device. 2)Operating system level. (SkimpyStash uses app 2) To get maximum benefit out of SSDs it is required to use Flash aware data structures and algorithms. Example Avoid random writes high latency and reduce Flash device life time through increased page wearing.

5 Contribution Design Innovation: Move pointers to Flash to break flash pointer barrier. It is implemented by 1) Resolving hash table collisions by linear chaining where multiple keys that collide to the same hash table buckets are chained in a linked list. 2) Storing linked list on Flash with a pointer in each hash table bucket in RAM pointing to the beginning record of the chain on flash. (Multiple flash reads per lookup) 3) Two-choice based load balancing. At most one bucket chain on flash needs to be searched during a lookup. 4) Compaction procedures to pack bucket chain records contiguously onto the flash pages.

6 Contribution Evaluation on data center server applications: SkimpyStash can be used as a high throughput persistent key-value storage layer for many server side applications. Real world use cases: Xbox LIVE Primitive online multi player games, Inline storage deduplication. SkimpyStash provides throughputs from few 10,000s to upwards of 100,000 get-set operations/sec on the evaluated applications.

7 Flash Memory Advantages: Fast read operations for both random and sequential read operations. (No mechanical head movement). Disadvantage: Does not allow in-place update(overwrites). Write performed after Erase In-place update problem: Write performed in pages granularity, erase performed in Block granularity. Read 25 us Write 200 us Erase 1500 us Block Pages

8 FTL Flash transition layer (FTL) is an intermediate software inside SSD makes linear flash memory device act like virtual disk. Logical read Flash memory commands Logical write Mapping physical pages and logical pages, flag current physical page for garbage collection. Flash block can be erased for limited no of times. Effective for sequential access and show poor Performance for random access.

9 Key Value Store Use Cases Online multi player gaming: Requirement: To scale multiple users per game To scale multiple game instances run thru online service To persist the server side game state. (Resume from interrupted state, offline analysis, verification of player action). Set and Get key operation impacts the response time and Throughput. Deduplication: Eliminate redundancy of data across full and incremental backup.

10 Key Value Store Use Cases Requirement: To scale tens of terabytes to petabytes of data volume. To provide high throughput (Disk storage fails) and inline storage deduplication system is desired. Chunk hash index is too big to fit in RAM. Existing solutions do not offer much performance than lookup hitting in RAM. SkimpyStash can be used as chunk hash index for inline deduplication systems. Serving lookups from flash memory.

11 SkimpyStash Design. Flash Constraints: Random Write Random writes need to update the data portions within pages. (New physical pages need to be created and data movement). Write less than flash page size Writing an amount less than the page size, leads to unwanted page wasting. SkimpyStash Components: RAM Write Buffer Buffer (K,V) writes before writing to flash. (Avoid wasting pages 2 or 4 KB in size) RAM Hash Table(HT) Directory (K,V) contains a pointer to the next record and hence chaining.

12 SkimpyStash Design. Two-choice based load balancing to reduce wide variation in bucket sizes (hence, chain lengths and associated lookup times). Bloom filter in each hash table directory slot in RAM for summarizing the records in that bucket so that at most one bucket chain on flash needs to be searched during a lookup. Flash Store -- Persistent storage for the key,value pairs. Circular append log. Writes are inserted to the tail of the log. Garbage recycled from the head of the log. Valid entries moved back to the tail of the log.

13 SkimpyStash Design.

14 SkimpyStash Operations Key Lookup(Get) First lookup the RAM write buffer. If there is a Miss then lookup the HT directory and search the linked list in Flash in respective bucket. Key Insert(Set) First write to the RAM write buffer. When write buffer threshold occurs, at configurable time interval, entries in Write buffer are written to Flash and inserted into RAM HT directory. Delete Insertion of null value for the key. Garbage collector comes into action eventually.

15 HT Directory Design RAM Overhead in FAWN and ChunkStash is 6 bytes. SkimpyStash reduces the overhead to 4 bytes by moving most of the pointers to (K,V) to Flash itself. Multiple Flash reads upon lookup of a key. (Because of chain of key,value pairs in a bucket) This tradeoff is needed to avoid RAM Overhead. Average no of keys in a bucket (k) is used as a parameter to control the tradeoff factor.

16 Base Design Map Function(h) maps keys to slots in HT. Avg no of records in a bucket = k. Accommodate n key value pairs we need n/k slots in Hash Table. For example let us assume k = 4 and n =16 then we have: (4 slots) HT Buckets `

17 HT Directory Design Lookup operation Hash function h is used to obtain the bucket and the pointer in the slot is followed to chain of records to find the key. Avg Case flash read (k/2) Worst case (size of bucket) Insert operation Hash function h is used to obtain the bucket and the new record is inserted into the head of the linked list. (New records always in the beginning of the list, old records are in the end of the list) Delete Operation Same as Lookup and Insert with null values for the key

18 HT Directory Design RAM Overhead in FAWN and ChunkStash is 6 bytes. SkimpyStash reduces the overhead to 4 bytes by moving most of the pointers to (K,V) to Flash itself. Multiple Flash reads upon lookup of a key. (Because of chain of key,value pairs in a bucket) This tradeoff is needed to avoid RAM Overhead. Average no of keys in a bucket (k) is used as a parameter to control the tradeoff factor.

19 Load Balancing across Buckets Maintain bucket chain of same size (Avoid Skewing). Power of two choice idea. Each key would be hashed to two candidate buckets. Two hash functions h1 and h2 used but actual insert is done into the bucket that has fewer elements. Load balancing is implemented by adding 1 byte of storage to each slot in HT.

20 Bloom Filter Mechanism Lookup in both candidate buckets is a bottleneck. (As number of flash reads doubles.) Bloom filter added per each HT slot to summarize keys inserted into each bucket. (1 byte/key) Lookup Search buckets only if the bloom filter associated with the bucket indicates key exist. Insert Records inserted into the same bucket as early one. (Do not follow the load balancing strategy as it goes against the design principles.) In short it reduce variations in the the number of keys assigned to each bucket (hence, chain lengths and associated lookup times).

21 Compaction Mechanism Bucket size is inversely proportional to that of the Lookup time. Compacting the chain on a flash in a bucket by placing valid keys contiguously in more than on page. If m (K,V) pairs can be Packed in one page, no of flash reads in a bucket of k records is (k/2m) average case and (k/m) -- worst case

22 Garbage Collection Mechanism Garbage records (holes) accumulate in the log as a result of compaction and key update/delete operations. When a certain threshold of garbage accumulates, cleaning operation is performed to clean and compact the logs. Scan Lookup HT match match Valid Invalid

23 Summary SkimpyStash is used as high throughput persistent key-value storage layer for a broad range of server class applications. Low RAM footprint at about 1 byte per key-value pair, which is more aggressive than earlier designs. Evaluation done in real world datasets from Xbox LIVE Primetime online multi-player game and inline storage dedu- plication

24 Questions How is SkimpyStash design useful? Refer section 3.1 and 3.2 What are the different operations permitted in SkimpyStash and how are they implemented? Refer section 4.4 What is the Base deign of Hash Table? How does the based deign improved in this paper to address the challenges in Base Design? Refer section 4.5

25 Questions (1) What s advantage and disadvantage of the design that only keeps a hash table directory in RAM and moving the hash table buckets to the flash? (Section 1 Our Contribution ) (2) What s the issue that the Power of two choice idea is adopted to address? What s the issue Bloom Filter is adopted to address? Provide a few specifics on how the two issues are addressed respectively. (Section 4.5 Load Balancing across Buckets and Bloom Filter per Bucket )