S-Store: Streaming Meets Transaction Processing

Transcription

1 S-Store: Streaming Meets Transaction Processing H-Store is an experimental database management system (DBMS) designed for online transaction processing applications Manasa Vallamkondu

2 Motivation Reducing the latency of results with the applications like monitoring systems by typically running the operations/queries in main memory, by avoiding the extreme latency caused by disk access. S-store can simultaneously accommodate OLTP and streaming applications

3 S-store S-store is a combination of stream processing and transaction processing. S-Store can simultaneously accommodate OLTP and streaming applications. S-Store is an extension of H-Store - an opensource, in-memory, distributed OLTP database system.

4 Glossary H- store- H-Store is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications. It is a highly distributed, row-store-based relational database that runs on a cluster on shared-nothing, main memory executor nodes. Stream processing is a computer programming paradigm, equivalent to dataflow programming and reactive programming, that allows some applications to more easily exploit a limited form of parallel processing. Such applications can use multiple computational units, such as the FPUs on a GPU or field programmable gate arrays (FPGAs), without explicitly managing allocation, synchronization, or communication among those units. Ad hoc analysis is a business intelligence process designed to answer a single, specific business question. The product of ad hoc analysis is typically a statistical model, analytic report, or other type of data summary.

5 Application Domain Real time Data Ingestion - An analytics warehouse must be updated periodically with recent data. Transaction mechanism is needed for adding new data into warehouse. S-Store is well-positioned to satisfy the need of ETL tools for working on streaming data. Shared Mutable State - S-Store is useful beyond real-time ETL

6 Financial Information Exchange Data

7 Computational Model Hybrid workloads(combination of independent OLTP transactions and streaming transactions) are supported with well-defined correctness guarantees- 1. ACID guarantees for individual transactions (both OLTP and streaming) 2. Ordered Execution guarantees for dataflow graphs of streaming transactions 3. Exactly-Once Processing guarantees for streams (i.e., no loss or duplication)

8 Transaction Execution A Transaction Execution essentially corresponds to an atomic batch and its subsequent processing by a stored procedure.

9 H-Store system H-Store is an open-source, main-memory OLTP engine, transactions are predefined as stored procedures. Transaction executions (TEs) are instantiated by binding input parameters of a stored procedure to real values and running it. H- store initiates the transaction in a layer called Partition engine (PE) which is responsible for managing transaction distribution, scheduling, coordination, and recovery.

10 H-Store PE manages another layer with Execution engine (EE)) that is responsible for the local execution of SQL queries. A client program connects to the PE via a stored procedure execution request. If the stored procedure requires SQL processing, then the EE is invoked with these sub-requests

11 S-Store Architecture Includes stream processing which enables the management of Inputs from streaming clients and dataflow graphs of stored procedures at the PE layer Triggers at both the PE and the EE layers stream- and window-based queries at the EE layer In-memory stream and window state.

12 S-Store Architecture

13 Experiments A number of micro-experiments were performed to evaluate the optimizations achieved by S-Store over H- Store with transactional stream processing workloads. Execution Engine Triggers - In S-Store, the SQL statements of stored procedure can be activated using EE triggers and the execution takes place inside the EE layer. where the submission of the set of SQL statements (an insert and a delete) for each query as a separate execution batch from PE to EE.

14 Experiments Partition Engine Triggers -S-Store s PE triggers to an equivalent implementation in H-Store, which has no such trigger support in its PE. Serializing transaction requests severely limits H-Store s performance where as S-Store uses a streaming scheduler which can activate the next transaction directly within the PE and can prioritize these triggered transactions ahead of the current scheduling queue

15 Results Execution Engine triggers Partition Engine Triggers

16 Results S-Store processes 2,200 batches per second. S-Store is able to handle multiple asynchronous transaction requests from the client and still preserve the tuple processing order. In PE weak recovery not only achieves better throughput during normal operation, but it also provides lower recovery time.

17 Conclusion S-Store is a new model of transactions for stream processing, that seamlessly combines OLTP transaction processing with our transactional stream processing model. S-Store shows how the symbiosis can be implemented in the context of a main-memory, OLTP DBMS in a straight-forward way. S-Store is shown to outperform H-Store, Esper, and Storm on a streaming workload that requires transactional state access, while at the same time providing stronger correctness guarantees.

18 Future Work Extending S-Store to operate on multiple nodes by addressing a number of research issues including data and workload partitioning, distributed recovery, and distributed transaction scheduling and handling of dynamic and hybrid (OLTP+streaming) workloads.