DeclStore: Layering is for the Faint of Heart

Transcription

1 DeclStore: Layering is for the Faint of Heart [HotStorage, July 2017] Noah Watkins, Michael Sevilla, Ivo Jimenez, Kathryn Dahlgren, Peter Alvaro, Shel Finkelstein, Carlos Maltzahn

2 Layers on layers on layers Application Special App Object Block File Logging Key-value Special App Middleware Storage Interfaces POSIX Fancy Storage Unified Storage 2

3 Application-specific storage systems Application Special App Object Block File Logging Key-value Special App Middleware Storage Interfaces POSIX Fancy Storage Unified Storage 3

4 Unified storage systems Application Special App Object Block File Logging Key-value Special App Middleware POSIX Examples OnTap Fancy Storage Swift Ceph GPFS EMC Storage Interfaces Unified Storage 4

5 Eliminating layers: big challenges, big rewards Big Question If unified storage becomes the new normal, what new challenges will need to be addressed in system design? Talk Outline Object Block File Logging Key-value Special App Storage Interfaces Unified Storage Motivating unified storage Storage interface design space Declarative storage interfaces 5

6 Motivation: systems trend towards generality Technology Trend Example Specialized unstructured data management systems JSON data type in relational database management systems Specialized real-time and embedded systems Embedded Linux and RT_PREEMPT Specialized storage systems Embedded Ceph and unified system 6

7 Motivation: breaking the narrow waist model User Defined Classes Examples Locking, Logging, Concurrency control, Metadata management Remote compute Developers willing to break layers and use non-standard APIs 7

8 Why in practice do people break down layers? Application I in AP e Cod and h t pamiddleware POSIX ef cies n e fici Special App es, rvic risks e s f t, n o, cos o i t lica ioning p u D rovis r-p ove Fancy Storage Object Block File Logging Key-value Special App Storage Interfaces Unified Storage 8

9 Unified storage has its own set of challenges Special App es, rvic risks e s f t, n o, cos o i t lica ioning p u D rovis r-p ove Logging Key-value Special App ty afe ies) S, os ficienc Q, f ility th ine b a t Por ode pa c s Storage u l Interfaces p ( Fancy Storage Unified Storage Application I in AP e Cod and h t pamiddleware POSIX ef cies n e fici Object Block File 9

10 Exploring design challenges in unified storage Application ies e Cod PI i A and h t a pmiddleware POSIX c cien i f f ne, Special App ices sks v r e ri of s cost, n atio ning, c i l Dup rovisio r-p ove Fancy Storage Object Block File Logging Key-value Special App Storage Interfaces Unified Storage 10

11 Malacology: programmable interface research platform Target storage interface (Goal) Composed, generic service glue layer Internal sys services (Building blocks) Malacology: A Programmable Storage System, M. Sevilla, N. Watkins, I. Jimenez, P. Alvaro, S. Finkelstein, J. LeFevre, C. Maltzahn, EuroSys 17 Mantle: A Programmable Metadata Load Balancer for the Ceph File System, M. Sevilla, N. Watkins, C. Maltzahn, I. Nassi, S. Brandt, et. al, SC 15 CORFU: A Shared Log Design for Flash Clusters, Mahesh Balakrishnan, Dahlia Malkhi, Vijayan Prabhakaran, and Ted Wobber, Michael Wei, et. al, NSDI 12 11

12 Keeping pace with evolving storage systems Hard-wired prototype implementations How to take advantage of... Software upgrades New hardware Additional services Performance features 12

13 Example: building the CORFU shared-log interface CORFU1 is a dist. shared-log High-performance design ?? Append and random reads I/O parallelism (striping) Soft-state network counter Challenges Find a good mapping Performance optimizations Minimal maintenance CORFU: A Shared Log Design for Flash Clusters, Mahesh Balakrishnan, Dahlia Malkhi, Vijayan Prabhakaran, and Ted Wobber, Michael Wei, et. al, NSDI 12 13

14 Example: building CORFU distributed shared-log Implementation options Partitioning Log partitioning?: 1-1, striping, etc... 14

15 Example: building CORFU distributed shared-log Implementation options Partitioning Metadata Log partitioning?: 1-1, striping, etc... 15

16 Example: building CORFU distributed shared-log Implementation options 0 Partitioning Metadata I/O interfaces Log partitioning?: 1-1, striping, etc... ByteStream, K/V FS (xfs, btrfs) BlueStore K/V (Rocks, LMDB) 16

17 Example: building CORFU distributed shared-log Implementation options 0 Partitioning Metadata I/O interfaces Log partitioning?: 1-1, striping, etc... 2x Constructed 4 versions 2 partitioning * 2 I/O interfaces Partitioning (1-1, striping) I/O interface (bytestream, K/V) ByteStream, K/V FS (xfs, btrfs) 2x BlueStore K/V (Rocks, LMDB) 17

18 Example: building CORFU distributed shared-log Implementation options Partitioning Metadata I/O interfaces Log partitioning?: 1-1, striping, etc... 2x Constructed 4 versions 0 2 partitioning * 2 I/O interfaces Partitioning (1-1, striping) I/O interface (bytestream, K/V) Hard-wired implementations Few 1000 s LOC Manually optimize and switch between approaches ByteStream, K/V FS (xfs, btrfs) 2x BlueStore K/V (Rocks, LMDB) 18

19 Example: shared-log performance toss-up in 2014 Four implementations Ceph version circa 2014 Graph takeaways Similar top performers Our claim Clear performance losers Performance Comparison of 4 Designs Appends / Sec Ceph 2014 Select simpler implementation Added complexity for no benefit What is complexity? Lines of code Conceptual 19

20 Example: clear design choice in 2016 Takeaway: A reasonable choice in 2014 would be a poor choice in 2016 after a simple upgrade Performance Comparison of 4 Designs Appends / Sec Same implementations Same hardware / benchmark Newer version of Ceph Clear performance winner in 2016 Ceph 2014 Appends / Sec Ceph

21 Problem: navigation of large design space interfaces / data models you are here heterogeneous hardware / memories features & tunables (>1000 in Ceph) time / versions 21

22 DeclStore: express storage interfaces declaratively Freedom from storage system and domain expertise Abstractions over storage services and interfaces The point is we need a high-level language Many possible mappings Many degrees of freedom Generate storage interface implementations Exploit relational database optimization research 22

23 DeclStore: building on a strong foundation Based on Bloom programming language High-level declarative language (Alvaro, CIDR `11) Many degrees of freedom on reordering Abstract relational data model Programs are queries Subject to optimization Evidence this is possible... LogicBlox (Aref, SIGMOD 2015) Dedalus (Alvaro, et. al, Datalog 2010) Declarative Networking (Loo, SIGMOD 2006) 23

24 Example: a declarative specification for CORFU System state modeled as a set of relations Interfaces are queries over a request stream Persistent state Input/Output Operate on persistent state Collection operations Full specification Just another slide s worth Compare 1K s LOC C++ # persistent relations table :epoch, [:epoch] table :log, [:pos] => [:state, :data] # interfaces are also like tables interface input, :op, [:type, :pos, :epoch] => [:data] interface output, :ret, [:type, :pos, :epoch] => [:retval] bloom :write do temp :valid_write <= write_op.notin(found_op) log <+ valid_write { o [o.pos, 'valid', o.data] } ret <= valid_write { o [o.type, o.pos, o.epoch, 'ok'] } ret <= write_op.notin(valid_write) { o [o.type, o.pos, o.epoch, 'read-only'] } end Brados: Declarative, Programmable Object Storage, Noah Watkins, Michael Sevilla, Ivo Jimenez, Neha Ojha, Peter Alvaro, Carlos Maltzahn, UCSC-SOE

25 What s the catch? This is a storage system, not a database... Generality introduces overhead Additional complexity and layers We don t want layers! The cost of online optimization 25

26 Taking advantage of time scales: offline optimization We are generating storage service implementations Automate integration on new features and hardware Planned upgrade points v.s. per-request optimization DeclStore Interface Definition System Upgrade Point Opt. Timescale (Month / Year) System Upgrade Point Opt. System Upgrade Point Request (< 1ms) 26

27 Example: a declarative specification for CORFU No hard-wired choice of storage interface! Bytestream vs K/V Partitioning strategy Take advantage of existing analysis tools and techniques! Convincingly correct! # persistent relations table :epoch, [:epoch] table :log, [:pos] => [:state, :data] # interfaces are also like tables interface input, :op, [:type, :pos, :epoch] => [:data] interface output, :ret, [:type, :pos, :epoch] => [:retval] bloom :write do temp :valid_write <= write_op.notin(found_op) log <+ valid_write { o [o.pos, 'valid', o.data] } ret <= valid_write { o [o.type, o.pos, o.epoch, 'ok'] } ret <= write_op.notin(valid_write) { o [o.type, o.pos, o.epoch, 'read-only'] } end 27

28 Example: performance benefit of group commit Ceph I/O handling is conservative Log operations are independent Lots of internal code traversal Using efficient interfaces Determined through analysis Amortize across I/O and code path Basic Operation Batching Queuing, locking, ordering Vector or range-based interfaces See paper for more details Cost models and outliers 28

29 Conclusion: expanding the set of storage interfaces... is becoming increasingly common will create an entirely new set of challenges; we showed new systems being built old systems being adapted hard-wired solutions are difficult to maintain even a basic software upgrade can have a big impact through declarative specifications can address many concerns lower maintenance costs automate system evolution 29

30 Thank you Questions? DeclStore: Layering is for the Faint of Heart Noah Watkins, Michael Sevilla, Ivo Jimenez, Kathryn Dahlgren, Peter Alvaro, Shel Finkelstein, Carlos Maltzahn

31 What else we are shooting for... Volatile views of persistent data Physical design migration Example: soft-state network counter in CORFU Example: indexes, row vs. column. vs. hybrid Program analysis and optimization Example: exploit non-interference 31

32 Cost models are an important component Batching with range operations Sensitive to outliers Wasted I/O Simple heuristics for identifying outliers outperform Basic-Batch Order of magnitude > IOPS Minimal reduction over Opt-Batch Cost models needed for automatic strategy selection orthogonal to program analysis 32