Data Systems that are Easy to Design, Tune and Use. Stratos Idreos

Transcription

1 Data Systems that are Easy to Design, Tune and Use

2 data systems that are easy to: (years) (months) design & build set-up & tune (hours/days) use e.g., adapt to new applications, new hardware, spin off alternative designs fast, create truly tailored solutions, enable more applications, bring development/set-up cost down (start-ups/scientific labs), etc

3 declarative interface ask what you want db system the system decides how to best store and access data

4 too many preparation options lead to complex installation schema storage load indexing query timeline expert users - idle time - workload knowledge!=exploration, easy

5 users/applications declarative interface ask what you want DBA db system

6 DBA1 users/applications DBA2 need to choose the proper system & workloads/ applications change rapidly data system 1 data system 2

7 be able to query the data immediately & with good performance schema X X X storage load indexing query

8 be able to query the data immediately & with good performance schema X X X storage load indexing query raw data explore data and gain knowledge immediately

9 design space it all starts with how we store data every bit matters from static to dynamic designs

10

11 idle time workload knowledge external tools human control

12 initialization querying indexing idle time workload knowledge external tools human control

13 initialization querying indexing idle time workload knowledge external tools human control

14 initialization querying indexing idle time workload every query is treated as an advice knowledge external tools on how data should be stored human control

15 initialization querying indexing continuous, lightweight actions to co-locate relevant data idle time workload every query is treated as an advice knowledge external tools on how data should be stored human control

16 MonetDB Presorted Sel. Crack Sid. Crack MySQL Presorted Response time (milli secs) TPC-H Query 15 normal MonetDB selection cracking Query sequence

17 MonetDB Presorted Sel. Crack Sid. Crack MySQL Presorted Response time (milli secs) TPC-H Query 15 normal MonetDB selection cracking presorted MonetDB preparation cost 3-14 minutes Query sequence

18 MonetDB Presorted Sel. Crack Sid. Crack MySQL Presorted Response time (milli secs) presorted MonetDB preparation cost 3-14 minutes TPC-H Query Query sequence normal MonetDB selection cracking MonetDB with sideways cracking

19 MonetDB Presorted Sel. Crack Sid. Crack MySQL Presorted Response time (milli secs) presorted MonetDB preparation cost 3-14 minutes TPC-H Query Query sequence normal MonetDB selection cracking MonetDB with sideways cracking

20 MonetDB Presorted Sel. Crack Sid. Crack MySQL Presorted Response time (milli secs) presorted MonetDB preparation cost 3-14 minutes TPC-H Query Query sequence normal MonetDB selection cracking MonetDB with sideways cracking

21 MonetDB Presorted Sel. Crack Sid. Crack MySQL Presorted Response time (milli secs) presorted MonetDB preparation cost 3-14 minutes TPC-H Query Query sequence normal MonetDB selection cracking MonetDB with sideways cracking

22 query plan scan

23 query plan scan db

24 query plan scan loading

25 query plan scan files loading access raw data adaptively on-the-fly

26 query plan scan files access raw data adaptively on-the-fly cache loading

27 selective parsing file indexing file splitting online statistics query plan scan files access raw data adaptively on-the-fly cache loading

28 basics (CIDR07) databases with adaptive storage updates (SIGMOD07) >1 columns >1 columns (SIGMOD09) storagerestrictions (SIGMOD09) robustness robustne (PVLDB12) algorithms (PVLDB11) benchmarking (TPCTC10) concurrency control (PVLDB12) adaptive storage (SIGMOD14) time-series (SIGMOD14) multi-cores (SIGMOD15) encryption (SIGMOD16) 12

29 design space

30 design space data system X

31 design space data system X adaptive data system Y

32 dai data 2.5 grows today y 2 [IB data systems are nearly everywhere

33 dai data 2.5 grows today y 2 [IB data systems are nearly everywhere continuous need for new and tailored data systems

34 dai data 2.5 grows today y 2 [IB data systems are nearly everywhere continuous need for new and tailored data systems tomorrow

35 daily data data* daily skills data years [IBMbigdata] years [StratosGuess] data system use, set-up, tune, design

36 e.g., modern main-memory optimized systems: first ideas in 80s, first advanced architectures in 90s, first rather complete designs in early 2000s, mainstream industry adoption still limited indexing, limited cost based optimizations,

37 disk memory flash easily utilize past concepts

38 self-designing data systems data+queries+hardware data system

39 self-designing data systems data+queries+hardware data system easy to design

40 1. write/extend modules in a high level language (optimizations) 2. modules= storage/execution/data flow 3. try out >>1 designs (sets of modules) ACM SIGMOD Blog, June 15

41 H20, ACM SIGMOD14, + evolutionary prototype row-store hybrid store column-store key-value store automatically choose the right layout using genetic algorithms + generate tailored code

42 H20, ACM SIGMOD14, + evolutionary prototype row-store hybrid store column-store key-value store automatically choose the right layout using genetic algorithms + generate tailored code construct access methods out of basic components (and their tradeoffs) e.g., scan*, tree*, bloom filters, bitmaps, hash tables, etc. RUM conjecture, EDBT16

43 SQL/correctness=expertise+knowledge!=everyone can explore data systems that are easy to use dbtouch cidr2013/icde2014 show me something interesting DATA Queriosity bigdata2015

44 data systems today allow us to answer queries fast db data systems tomorrow should allow us to find fast which queries to ask db explore

45 thank you! Martin Kersten Stefan Manegold Goetz Graefe Harumi Kuno Anastasia Ailamaki Themis Palpanas Eleni Petraki Ioannis Alagiannis Miguel Branco Renata Borovica Erietta Liarou Felix Halim Ronald Yap Panos Karras Holger Pirk Kostas Zoumpatianos Manos Athanassoulis Lukas Maas Abdul Wasay Mike Kester Dhruv Gupta