HyPer on Cloud 9. Thomas Neumann. February 10, Technische Universität München

Transcription

1 HyPer on Cloud 9 Thomas Neumann Technische Universität München February 10, 2016

2 HyPer HyPer is the main-memory database system developed in our group a very fast database system with ACID transactions quite comprehensive SQL support (SQL92 plus many SQL99 features) queries are compiled into machine code using LLVM very little overhead, excellent performance You can try it out (or download a demo) online: We use it for teaching, too. (Easy to use web interface for students). Thomas Neumann HyPer on Cloud 9 2 / 13

3 Hyper in the Cloud Currently HyPer is primarily run on a single dedicated node but of course we looked at running HyPer in the cloud Wolf Rödiger will talk about distributed processing next I will concentrate on issues we found during experiments some of them quite surprising; could be relevant for your project, too not all of them have perfect solutions yet Thomas Neumann HyPer on Cloud 9 3 / 13

4 Costs of Virtualization Cloud applications usually run on virtualized machined shields the application from the actual hardware simplified management, allows for easy migration, etc. several different techniques to do that (containers, hypervisors, etc.) what does that actually cost? in-memory processing is CPU bound in an ideal world virtualization would be (nearly) free Thomas Neumann HyPer on Cloud 9 4 / 13

5 Thomas Neumann HyPer on Cloud 9 5 / 13

6 Thomas Neumann HyPer on Cloud 9 6 / 13

7 Thomas Neumann HyPer on Cloud 9 7 / 13

8 Thomas Neumann HyPer on Cloud 9 8 / 13

9 Thomas Neumann HyPer on Cloud 9 9 / 13

10 Cooperative Behavior Cloud infrastructure relies upon cooperative behavior virtualized applications utilize ideal resources combined system looks more powerful than it really is hogging resources is bad for cloud processing Main resource for HyPer (and probably most other DBMSs): main memory not only for the real data, but also for intermediate results some queries have large intermediate results allocate a lot of memory Thomas Neumann HyPer on Cloud 9 10 / 13

11 Allocating and Releasing Memory Memory allocation is a multi-step process: 1. pages are created in the page table, but (usually) not physically assigned 2. upon access, pages are assigned and zeroed out 3. upon release, page table is modified again Problem: this does not scale a lot of locking and overhead within the kernel costs more than a factor 2 performance with large queries on large hosts even more pronounced on Windows, there freeing pages is surprisingly expensive Solutions? never release memory? (i.e., do your own memory management) best performance, but hogs resources non-portable techniques? (madvise etc.) Unsatisfying. Thomas Neumann HyPer on Cloud 9 11 / 13

12 Durability For ACID, we must 1) not report commit until the WAL record hits durable media, and 2) avoid data corruption during writes. difficult to guarantee even on bare metal often requires battery-backed RAID controllers to avoid all issues (including tearing) How do we guarantee that on virtualized hardware? nearly impossible the (virtualized) hardware lies to us an inconveniently timed power failure can lead to a disaster Microsoft installed special hooks for their cloud DBMS, but that is not publicly usable other vendors probably, too, but not generic solutions exists Thomas Neumann HyPer on Cloud 9 12 / 13

13 Conclusion Databases in the cloud are an interesting topic many effects that we see are implementation artifacts choice of virtualization product has a large impact many open issues concerning both performance and correctness but apparently many people are happy with living dangerously Still many open research questions. Thomas Neumann HyPer on Cloud 9 13 / 13

14 HyPer on Cloud 9 Wolf Rödiger Technische Universität München

15 Traditional Data Warehouse OLTP Extract Transform Load OLAP Database Data Warehouse Wolf Rödiger HyPer on Cloud 9 2 / 28

16 HyPer: Hybrid OLTP & OLAP HyPer OLTP & OLAP on the same data at the same time Efficient snapshotting Data-centric code generation OLTP OLAP > 100,000 TPC-C TX/s Best-of-breed response times Real-time analytics fork() OLTP OLAP snapshot a* OLTP process a copy on update OLAP Wolf Rödiger HyPer on Cloud 9 3 / 28

17 Wishlist for HyPer in the Cloud Scale query performance with the cluster size Sustain transaction performance Scale capacity to process larger workloads Elastically add servers to the cluster Provide high availability Wolf Rödiger HyPer on Cloud 9 4 / 28

18 1st Design: Full Replication via Redo Log Multicasting primary HyPer persistent storage OLTP multicast redo log redo log snapshot secondaries OLAP load balancing Tobias Mühlbauer, Wolf Rödiger, Angelika Reiser, Alfons Kemper, Thomas Neumann, ScyPer: Elastic OLAP Throughput on Transactional Data, DanaC Wolf Rödiger HyPer on Cloud 9 5 / 28

19 Scale query throughput, sustain transaction performance 150k OLTP throughput OLAP throughput 60k transactions/s 100k 50k 40k 20k queries/h number of nodes 0 Wolf Rödiger HyPer on Cloud 9 6 / 28

20 Elasticity: Materialized snapshots and multicasting primary HyPer persistent storage OLTP 3. subscribe to multicast 2. apply redo log 1. load snapshot secondaries new server OLAP 4. receive queries Wolf Rödiger HyPer on Cloud 9 7 / 28

21 High Availability: Secondaries can replace the primary primary HyPer persistent storage 3. start multicast 1. replay redo log secondaries 2. accept OLTP OLTP OLAP new primary Wolf Rödiger HyPer on Cloud 9 8 / 28

22 Multi-Tenancy: Flexible database-as-a-service deployment secondary primary primary Tenant 1 Tenant 1 Tenant 2 Tenant 4 Tenant 3 Tenant 1 Tenant 4 Tenant 4 Tenant 2 Tenant 3 secondary secondary secondary Wolf Rödiger HyPer on Cloud 9 9 / 28

23 1st Design: Full Replication via Redo Log Multicasting Scales query throughput with the cluster size Sustains transaction performance Allows to elastically add servers to the cluster Provides high availability Limited to workloads that fit into a single server Same query response times as a single server Wolf Rödiger HyPer on Cloud 9 10 / 28

24 2nd Design: Query Processing on Fragmented Relations Lineitem Lineitem 11 Customer Customer 11 Orders Orders 11 Lineitem Lineitem 21 Customer Customer 21 Orders Orders 21 Lineitem Lineitem 31 Customer Customer 31 Orders Orders 31 Lineitem Lineitem 41 Customer Customer 41 Orders Orders 41 Lineitem Lineitem 51 Customer Customer 51 Orders Orders 51 Lineitem Lineitem 61 Customer Customer 61 Orders Orders 61 Fragment relations across servers to use the combined capacity of the cluster, making room for larger workloads Wolf Rödiger, Tobias Mühlbauer, Alfons Kemper, Thomas Neumann, High-Speed Query Processing over High-Speed Networks, VLDB Wolf Rödiger HyPer on Cloud 9 11 / 28

25 Main Challenges 128 GB 59.7 GB/s CPU CPU 0 CPU CPU 1 10 cores 16 GB/s QPI QPI 10 cores 59.7 GB/s 128 GB 16 GB/s PCIe GB/s host 0 host 1 4 GB/s Infiniband 4 QDR HCA host 2 host 3 host 4 host 5 Challenge 1: Network is a bottleneck to query processing Challenge 2: Utilize all cores and all servers Wolf Rödiger HyPer on Cloud 9 12 / 28

26 Challenge 1: Network is a Bottleneck Network is bottleneck for distributed processing Manual schema-tuning is time-consuming and workload-dependent TPC-H speed-up (SF 100) Faster network hardware is not enough, software 0 has to change TCP/IP (40 Gb/s InfiniBand) TCP/IP (1 Gb/s Ethernet) number of servers Wolf Rödiger HyPer on Cloud 9 13 / 28

27 Remote Direct Memory Access Zero-copy network communication: High network throughput Almost no CPU cost Less memory bus traffic Less CPU cost, higher throughput than TCP RDMA achieves full speed at only 4 % CPU load Recently used to implement a distributed radix join* Application Buffer Operating System sender Application Buffer Operating System receiver HCA HCA *Barthels et al., Rack-Scale In-Memory Join Processing using RDMA, SIGMOD Wolf Rödiger HyPer on Cloud 9 14 / 28

28 Network Scheduling Uncoordinated network communication causes switch contention Communicate in a strict round-robin fashion Synchronize via low-latency RDMA operations Network scheduling improves throughput by 40 % for an 8-server InfiniBand cluster throughput in GB/s all-to-all scheduling number of servers Wolf Rödiger HyPer on Cloud 9 15 / 28

29 Challenge 2: Utilize all cores and all servers Exchange partitions tuples for joins and aggregations Traditionally used to parallelize for multiple cores and servers But: Exchange does not scale well due to inflexible design TPC-H speed-up (SF 300) HyPer ( operators) Vectorwise ( operators) 6 (1) 30 (5) 60 (10) 90 (15) 120 (20) number of cores (per server) Wolf Rödiger HyPer on Cloud 9 16 / 28

30 Classic Exchange Fixed degree of parallelism An needs a buffer for every other : #buffers per server = servers cores 2 2,400 buffers 1 GB of main memory per server Also: Each join key value is processed by a specific operator Heavy hitters are assigned to a single Wolf Rödiger HyPer on Cloud 9 17 / 28

31 Decoupled Exchange Communicate indirectly via communication multiplexers Address servers not threads: Decreases memory consumption Reduces negative impact of heavy hitters Improves applicability of broadcast optimization Local load balancing via work stealing* important for good scalability CM CM CM *Leis et al., Morsel-driven parallelism: a NUMA-aware query evaluation framework for the many-core age, SIGMOD Wolf Rödiger HyPer on Cloud 9 18 / 28

32 Hybrid Parallelism = RDMA-based Communication + Decoupled Exchange Operators server 0 NUMA socket 1 parallelized query pipeline parallelized query pipeline 7. produce tuples 6. deserialize tuples NUMA socket 0 per thread next operator per thread 1. consume tuples prev. operator RDMA Key NUMA node retain count ID last message bytes used content incoming message 2. partition + serialize tuples 5a. receive local or 5b. steal work communication multiplexer receive queue NUMA socket 1 receive queue NUMA socket 0 per server network scheduler per server RDMA RDMA Key RDMA Key Key NUMA NUMA node NUMA node node retain retain count retain count count ID 3. send message ID ID when full last message last message bytes used used bytes used content content 4. reuse message outgoing messages message pool send queues per socket Wolf Rödiger HyPer on Cloud 9 19 / 28

33 Hybrid parallelism scales in the number of cores... TPC-H speed-up (SF 300) HyPer (hybrid parallelism) HyPer () Vectorwise () 6 (1) 30 (5) 60 (10) 90 (15) 120 (20) number of cores (per server) Wolf Rödiger HyPer on Cloud 9 20 / 28

34 ... and in the number of servers in the cluster TPC-H speed-up (SF 100) RDMA (40 Gb/s InfiniBand) + scheduling TCP/IP (40 Gb/s InfiniBand) TCP/IP (1 Gb/s Ethernet) number of servers Wolf Rödiger HyPer on Cloud 9 21 / 28

35 How do we compare? HyPer (partitioned) 20,739 HyPer (chunked) 16,090 Vectorwise (partitioned) 3,856 MemSQL (partitioned) Impala (chunked) Spark SQL (chunked) k 10k 15k 20k 25k TPC-H queries per hour (SF 100) Note: MemSQL takes several seconds for query compilation, but these compilation times are not included in this experiment. Wolf Rödiger HyPer on Cloud 9 22 / 28

36 2nd Design: Query Processing on Fragmented Relations Utilizes the combined capacity of the cluster Reduces query response times Scales query throughput with the cluster size? How can we add servers with minimal disruption?? How can we survive server failures?? How can we sustain the transaction performance? Wolf Rödiger HyPer on Cloud 9 23 / 28

37 Outlook: Elasticity via Highest Random Weight Hashing e5fa4 4f2b3 1c1fb 7448d 8798a ff5e c13ff 7d5d9 a3db5 24e7a f6f9e Orders 3 Orders 5 Orders 2 Orders 6 new server 553b6 021e7 360d0 Orders 1 Orders 4 452e2 56f9b 62d4b Redistribute data when servers are added/removed Avoids reshuffling the whole database, balances load evenly Mukherjee et al., Distributed Architecture of Oracle Database In-memory, VLDB Wolf Rödiger HyPer on Cloud 9 24 / 28

38 Outlook: High availability via HDFS-style replication Orders PART 5 Orders PART 1 Orders PART 4 8 Orders PART 2 5 Orders PART 1 2 Orders PART 3 Orders PART 5 Orders PART 2 Orders PART 3 Orders PART 2 Orders PART 3 Orders PART 5 Orders PART 4 Orders PART 4 Orders PART 1 Replication factor of x allows for x 1 server failures Wolf Rödiger HyPer on Cloud 9 25 / 28

39 Outlook: Sustain transaction performance OLAP Distributed transactions are overly expensive: Two-phase commit Global locking Deadlock detection H-Store model: partitioned execution of transactions Works well for TPC-C Drawbacks: Requires schema tuning Not for all workloads OLTP OLTP OLTP Wolf Rödiger HyPer on Cloud 9 26 / 28

40 Outlook: Hot/cold approach for distributed transactions primary HyPer HOT OLTP multicast redo log secondaries COLD COLD 7 7 HOT HOT COLD COLD 4 4 COLD COLD 1 1 COLD COLD 8 8 HOT HOT COLD COLD 5 5 COLD COLD 2 2 HOT HOT COLD COLD 9 9 COLD COLD 6 6 COLD COLD 3 3 OLAP distributed queries on global TX-consistent snapshots Wolf Rödiger HyPer on Cloud 9 27 / 28

41 HyPer on Cloud 9 Scale capacity to process larger workloads Reduce query response times Scale query throughput with the cluster size Elastically add servers to the cluster Provide high availability Sustain transaction performance Wolf Rödiger HyPer on Cloud 9 28 / 28

42 Backup Slides Wolf Rödiger HyPer on Cloud 9 1 / 5

43 Keeping secondaries up-to-date Logical log: transaction identifier and parameters Physical log: insert, update and delete operations Replay log on secondaries Logical log excludes: Readers Aborts Redo logging Undo logging Physical log also excludes: Read operations Costly transaction logic execution time 800 ms 600 ms 400 ms 200 ms 0 ms writers readers aborts normal execution 23% faster logical log replaying 56% faster physical log replaying Wolf Rödiger HyPer on Cloud 9 2 / 5

44 Redo Log Multicasting Independent of cluster size UDP unreliable, PGM reliable TPC-C transactions 60,000 log entries/s 1,500 B physical log entry/tx 250 B logical log entry/tx 1 GbE vs. InfiniBand 1 GbE needs: Group commits LZ4 compression ( 50 %) InfiniBand (IPoIB) is sufficient; PGM is CPU-bound 1 GbE PGM Bandwidth [Mbit/s] 675 Packets [1,000/s] 43 Latency [µs] InfiniBand 4 QDR PGM Bandwidth [Mbit/s] 1,832 Packets [1,000/s] 112 Latency [µs] 13.5 Wolf Rödiger HyPer on Cloud 9 3 / 5

45 Guaranteeing correct results 3. read a 1. read a 2. read a 1. fork() LSN 101 LSN 100 LSN 103 LSN write a=a* a* a a* a Order-preserving serializability violation Diverging distributed reads Logical time defined by log sequence number (LSN) Global TX-consistent snapshots avoid consistency problems Wolf Rödiger HyPer on Cloud 9 4 / 5

46 NUMA one socket interleaved NUMA-aware 0 1k 2k 3k 4k 5k TPC-H queries per hour (SF 100) NUMA-aware allocation of message buffers improves TPC-H performance by a factor of 2 for a 4-socket server Our communication multiplexer provides NUMA-local message buffers to the decoupled operators Wolf Rödiger HyPer on Cloud 9 5 / 5