Introduction to Column Stores with MemSQL. Seminar Database Systems Final presentation, 11. January 2016 by Christian Bisig

Transcription

1 Final presentation, 11. January 2016 by Christian Bisig

2 Topics Scope and goals Approaching Column-Stores Introducing MemSQL Benchmark setup & execution Benchmark result & interpretation Conclusion Questions and feedback 2

3 Scope and goals

4 Scope and goals Understandable preparation of the topic Column Stores Why and for what is the columnar data storage used? Introduction to MemSQL Columnar table usage in MemSQL Benchmark (MemSQL vs. PostgreSQL) (and in-memory tables vs. columnar tables) Deliverables: Article Presentation Tutorial modul 4

5 Approaching Column-Stores

6 Approaching Column-Stores A decomposition storage model SIGMOD conference Vertical partitioned data C-Store: A Column-oriented DBMS One of the first column-store DBMS The Design and Implementation of Modern Column-Oriented Database Systems

7 Approaching Column-Stores Business process automation Mostly transactional based (OLTP) e.g. register new client data, execute money transaction, etc. Additionally, business process improvement through gaining business intelligence Analytical processing (OLAP) e.g. evaluate client purchases, budget forecasts, etc. 7

8 Approaching Column-Stores Row based: Column based: 8

9 Approaching Column-Stores! Dos Large table scans and aggregations Range queries, BETWEEN, IN, <, > Data compression (sparse and repeated data) Large data load " Don ts Random / specific searches Large transaction volume (inserts and updates) Small inserts and updates (single-record insert performance) 9

10 Introducing MemSQL

11 Intro MemSQL Developed as in-memory database Added columnar tables with version 3.0 Provides a solution for both OLTP (row tables, in-memory tables) and OLAP (columnar tables on the harddisk) Wire compatible to MySQL Compiled queries 11

12 Intro MemSQL Two-tier architecture Distributed Systems (commodity hardware) Reference tables Shard tables Lock-free data structures Skip-Lists, Hash-Tables, Stacks, Queues MVCC 12

13 Intro MemSQL Sharding (Shard tables) Data partitioning distributed on leafs Reference tables 13

14 Intro MemSQL Row Table Columnar Table CREATE TABLE gnis ( x double precision not null, y double precision not null, fid integer primary key, name text, class text, state text, county text, elevation integer, map text ); CREATE COLUMNAR TABLE gnis_col ( x double precision not null, y double precision not null, fid integer, name text, class text, state text, county text, elevation integer, map text, KEY (`fid`) USING CLUSTERED COLUMNSTORE, SHARD KEY() ); 14

15 Intro MemSQL MemSQL column-store segmentation To consider: Every Insert or update creates a new row-segment-group The more row-segment-groups the worse the performance 15

16 Intro MemSQL Compression in MemSQL compression algorithms Dictionary (tokenization), Run-length-encoding example with osm_poi_tag_ch table Table-statistics compression rate of 3.6:1 which results in around 72% space savings 16

17 Benchmark setup & execution

18 Benchmark setup MemSQL (v ) running with Creating the GNIS tables as columnar and row tables Comparing the performance of columnar and row tables PostgreSQL (9.4) row tables Benchmark on: imac (late 2009), 2.8 GHz Intel Core i7, OSX El Capitan Ram: 16GB 1067 MHz DDR3 SSD 500GB, Read: ~260MB/s, Write: ~270MB/s 18

19 Benchmark setup SQL Load script major changes to original scripts: Instead of PostgreSQL \copy command to load CSV > LOAD DATA LOCAL INFILE INTO TABLE Instead of CREATE TABLE AS SELECT > CREATE TABLE and INSERT INTO SELECT for the creation of the 1mio, 2mio, 3mio record tables Slightly different naming (e.g. column name keyz instead of key ) 19

20 Benchmark execution Python scripts for benchmark execution Both for PostgreSQL and MemSQL no reasonable timing mechanism in MemSQL Using psycopg2 (PostgreSQL) and Mysqldb python drivers. Ran every query 3 times on row (PostgreSQL) and column / row (MemSQL) and took the best run of each to compare. Second benchmark part: A script for bulk insert/update/delete 20

21 Benchmark execution Python Script excerpt: 21

22 Benchmark execution Python Script excerpt: 22

23 Benchmark result & interpretation

24 Benchmark result 24

25 Benchmark result Single tuple data manipulation Inserts Updates Deletes 25

26 Benchmark interpretation Four points to mention: 1. Specific search on non index column and multiple tuples in result set, performs well 2. Both types have their field of events (e.g. specific search or range search) 3. Bad joined select performance on column-store 4. Column-stores are well suitable for a large amount of single data manipulation operations 26

27 Conclusion

28 Conclusion Not an option to compromise one store type over the other. Each one has its field of event SSD can not compensate column-store I/O disadvantages Impressed by the compression and performance Had a fight with measuring execution times in MemSQL Interested to test MemSQL in a larger setup 28

29 Any questions? 29

30 References: Image Slide 1: Image Slide 11: Slide 18, Docker: Author: Christian Bisig, Student for Master of Science Engineering at Hochschule für Technik Rapperswil Master Research Unit, Software and Systems Hardware/Software used for tests: imac (late 2009) CPU: 2.8 GHz Intel Core i7 Ram: 16GB 1067 MHz DDR3 SSD 500GB Read: ~260MB/s Write: ~270MB/s OS: OSX El Capitan