YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores

Transcription

1 YCSB++ benchmarking tool Performance debugging advanced features of scalable table stores Swapnil Patil M. Polte, W. Tantisiriroj, K. Ren, L.Xiao, J. Lopez, G.Gibson, A. Fuchs *, B. Rinaldi * Carnegie Mellon University * National Security Agency

2 Importance of scalable table stores For data processing and analysis For systems services (e.g., metadata in Colossus) 2

3 Growing complexity of table stores Growing set of HBase features HBASE release RangeRowFilters Batch updates Bulk load tools RegEx filtering Scan optimizations Co-processors Access Control Simple, lightweight complex, feature-rich stores Supports a broader range of applications Hard to debug performance issue and complex component interactions 3

4 State of table store benchmarking YCSB: Yahoo Cloud Serving Benchmark [Cooper2010] Modular design to test different table stores Great for CRUD (create-read-update-delete) benchmarking, but not for sophisticated features Need richer tools for understanding advanced features in table stores 4

5 This talk: YCSB++ tool NEW EXTENSIONS IN YCSB++ Distributed, coordinated and multi-phase testing Fine-grained, correlated monitoring using OTUS [Ren2011] TABLE STORE FEATURES TESTED BY YCSB++ Batch writing Table pre-splitting Bulk loading Weak consistency Server-side filtering Fine-grained security Tool released at 5

6 Talk Outline Motivation YCSB++ architecture Illustrative examples of using YCSB++ Summary and ongoing work 6

7 Original YCSB framework HBASE Workload Parameters Workload Executor Threads API Adaptor OTHER DBS Stats Storage Servers Configurable workload generation to test stores API adaptor converts read(k) to hbase_get(k) 7

8 YCSB++ supports new table store HBASE Workload Parameters EXTENSIONS Workload Executor EXTENSIONS Threads Stats API Adaptor ACCUMULO Storage Servers New DB adaptor for Apache Accumulo table store New parameters and workload executor extensions 8

9 Coordinated & multi-phase tests MULTI-PHASE HBASE Workload Parameters EXTENSIONS Workload Executor EXTENSIONS Threads Stats API Adaptor ACCUMULO COORDINATION YCSB clients Storage Servers ZooKeeper-based coordination & synchronization Enables heavy workloads and asymmetric testing 9

10 Coordinated & multi-phase tests Distributed, multi-client tests using YCSB++ Allows clients to co-ordinate their test actions Rely on shared data structures in ZooKeeper Useful for testing weak data consistency Multi-phase tests in YCSB++ Can construct tests comprising of different phases Built on ZooKeeper-based barrier-synchronization Used for understanding high-speed ingest features 10

11 Collective monitoring in YCSB++ MULTI-PHASE HBASE Workload Parameters EXTENSIONS Workload Executor EXTENSIONS Threads Stats API Adaptor ACCUMULO COORDINATION OTUS MONITORING YCSB clients Storage Servers Fine-grained resource monitoring using Otus [Ren2011] Collects from YCSB, table stores, HDFS and /proc 11

12 Talk Outline Motivation YCSB++ architecture Illustrative examples of using YCSB++ Case study: HBase and Accumulo Both are Bigtable-like table stores Summary and ongoing work 12

13 Primer on Bigtable-like stores (1) Incoming mutation logged in memory (unsorted order) (2) MINOR COMPACTION Memtables written to sorted, indexed store files in HDFS Data Insertion 1 Memtable 2 Tablet T N Tablet Servers Write Ahead Log Sorted Indexed Store Files 3 (Fewer) Store Files HDFS nodes (3) MAJOR COMPACTION LSM-tree based file merging (in background) 13

14 Accumulo table store Started at NSA; now an Apache project Built for high-speed ingest and scan workloads New features in Accumulo Iterator framework for user-specified programs placed in different stages of DB pipeline E.g., Supports joins and stream processing Also provides fine-grained cell-level access control 14

15 Before I talk about examples YCSB++ provides Abstractions to construct distributed, parallel tests Has in-built tests that use these abstractions Monitoring that collects and correlates system (store/fs/os) state with observed performance YCSB++ does not provide Root cause diagnosis of performance problems Merely points you to where you should look 15

16 FEATURES TESTED BY YCSB++ ILLUSTRATIVE EXAMPLE Table bulk loading Batch writing Weak consistency Table pre-splitting Server-side filtering Access control Table bulk loading High-speed ingestion through minimal data migration Need careful tuning and configuration [Sasha2002] 16

17 Table bulk loading in action (2) IMPORT store files into table stores to make data available for users Tablet servers Hbase Hbase Hbase Hbase Data files Hadoop tool HFile HFile HFile HFile HDFS cluster (1) FORMAT existing data files to store-file specific format using Hadoop 17

18 8-phase bulk load test in YCSB++ Measurement phase Light mix of Read/Update operations Interleaved to study performance over time P f P i M M S M L f L i Pre-load data Insert 6M rows in empty table Load data Load 48M rows in existing table Sleep Let servers finish balancing work 18

19 Multi-phase tests show variation P f P i M L f L i M S M ACCUMULO Read Latency (ms) Measurement phase time (s) 10x latency variation; lasts for a long time! Uniformly low latency after store is steady (no inserts) 19

20 Monitoring rebalancing at servers P f P i M L f L i M S M 1000 StoreFiles 100 Tablets Compactions Running time of the 8-phase test (sec) Let s take a closer look at correlating performance with server-side state 20

21 Effect of server-side work on latency StoreFiles ACCUMULO Read Latency (ms) StoreFiles and Tablets increase with splitting Tablets Compactions Experiment Running Time (sec) 21 Background compactions reduce number of store files

22 YCSB++ helps study different policies ACCUMULO Read Latency (ms) R/U 1 (Phase 3) R/U 2 (Phase 6) R/U 3 (Phase 8) Measurement Phase Running Time (Seconds) HBASE Read Latency (ms) Measurement Phase Running Time (Seconds) 100 StoreFiles Tablets 10 Compactions ACCUMULO Experiment Running Time (sec) HBASE Experiment Running Time (sec) 22

23 FEATURES TESTED BY YCSB++ ILLUSTRATIVE EXAMPLE Table bulk loading Batch writing Weak consistency Table pre-splitting Server-side filtering Access control Batching writes at clients Improves insert throughput and latency Newly inserted data may not be immediately visible to others 23

24 Batching improves throughput Inserts per second (1000s) Hbase Accumulo 10 KB 100 KB 1 MB 10 MB Batch size 6 clients create 9 million 1-KB records on 6 servers Small batches: high client CPU utilization limits work Large batches: servers are saturated, limits benefit 24

25 Weak consistency test in YCSB++ Table store servers CLIENT #1 CLIENT #2 Insert {K:V} (10 6 records) Store client Batch YCSB++ ZooKeeper Enqueue K (sample 1% records) Poll and dequeue K Store client 4 Read {K} YCSB++ ZooKeeper-based multi-client coordination Clients use a shared producer-consumer queue to communicate keys to be tested 25

26 Test setup details YCSB++ tests on 1% of keys inserted by C1 C1 inserts 1 million keys, C2 reads 10K keys Sampling avoids overloading ZooKeeper R-W lag for key K = time required by C2 to read K successfully If C2 can t read K in the first attempt, tries again Report the time lag for fraction of keys that need multiple read()s 26

27 Batch writing causes time lag Fraction of requests HBASE lag for different buffer sizes (a) HBase: Time lag for different buffer sizes 10 KB ( <1%) 100 KB (7.4%) 1 MB ( 17%) 10 MB ( 23%) read-after-write time lag (ms) Read-after-Write time lag (sec) Fraction of requests ACCUUMULO lag for different buffer sizes (b) Accumulo: Time lag for different buffer sizes 10 KB ( <1%) 100 KB (1.2%) 1 MB ( 14%) 10 MB ( 33%) read-after-write time lag (ms) Read-after-Write time lag (sec) Delayed writes may not be seen for ~100 seconds Batching implementations affect latency; YCSB++ helps understand differences 27

28 FEATURES TESTED BY YCSB++ OTHER DETAILS Batch writing Weak consistency Table bulk loading Table pre-splitting Server-side filtering Access control ACM SOCC 2011 paper available Poster session(s) 28

29 Evolving YCSB++ Future work Study additional table stores (Cassandra, MongoDB and CouchDB) Test more features: Iterators and co-processors Understanding table store features Cost-benefit tradeoff of different heuristics for compactions on tablet servers Dynamo-style eventual consistency 29

30 Summary: YCSB++ tool For performance debugging & benchmarking advanced features using extensions to YCSB Weak consistency semantics Fast insertion (pre-splits, bulk loads) Server-side filtering Fine-grained access control Distributed clients using ZooKeeper Multi-phase testing (with Hadoop) New workload generators and database client API extensions Two case-studies: HBase & Accumulo Download at