AN ELASTIC MULTI-CORE ALLOCATION MECHANISM FOR DATABASE SYSTEMS

Transcription

1 1 AN ELASTIC MULTI-CORE ALLOCATION MECHANISM FOR DATABASE SYSTEMS SIMONE DOMINICO 1, JORGE A. MEIRA 2, MARCO A. Z. ALVES 1, EDUARDO C. DE ALMEIDA 1 FEDERAL UNIVERSITY OF PARANÁ, BRAZIL 1, UNIVERSITY OF LUXEMBOURG 2

2 2 OUTLINE Introduction NUMA effect in OLAP PetriNet Multi-Core Allocation Mechanism Evaluation Conclusion

3 INTRODUCTION 3

4 NON-UNIFORM MEMORY ACCESS (NUMA) Avoid memory Interconnect link contention in multicore machines C C1 C C5 Node: Portion of C2 C3 C6 C7 memory and LLC LLC associated processor DRAM DRAM Shared L3 Cache Node Node 1

5 NON-UNIFORM MEMORY ACCESS (NUMA) Avoid memory Interconnect link contention in multicore machines Node: Portion of memory and Local C C1 C2 C3 LLC Remote C C5 C6 C7 LLC associated processor DRAM DRAM Shared L3 Cache Fast local RAM and slow remote RAM Node Node 1

6 5 NON-UNIFORM MEMORY ACCESS (NUMA) Threads may run in any Interconnect link core over time T1 T9 T2 LLC LLC DRAM DRAM Shared L3 Cache Node Node 1

7 5 NON-UNIFORM MEMORY ACCESS (NUMA) Threads may run in any Interconnect link core over time Local Data is mapped at thread startup T1 T9 T2 LLC LLC DRAM DRAM Shared L3 Cache Node Node 1

8 5 NON-UNIFORM MEMORY ACCESS (NUMA) Threads may run in any Interconnect link core over time Local Remote Data is mapped at T1 T2 thread startup T9 Remote access through LLC LLC the interconnection DRAM DRAM Shared L3 Cache Node Node 1

9 5 NON-UNIFORM MEMORY ACCESS (NUMA) Threads may run in any Interconnect link core over time Local Data is mapped at T1 thread startup T2 T9 Remote access through LLC LLC the interconnection DRAM DRAM Shared L3 Cache Migration: load balance, Node Node 1 reduce intercon. usage

10 NUMA EFFECT IN OLAP 6

11 7 PARALLEL QUERY PROCESSING IN NUMA SELECT * FROM A, B WHERE A.ID = B.ID AND B.DATE >= A.ID = B.ID σ B.DATE >= A B

12 7 PARALLEL QUERY PROCESSING IN NUMA SELECT * FROM A, B WHERE A.ID = B.ID AND B.DATE >= for t1 in left.getnext(): buildhashtable(t1) for t2 in right.getnext(): if probe(t2): emit(t1 t2) Parallelism is baked in the query plan at planning time σ for t in child.getnext(): if evalpred(t):emit(t) Batch work is assigned to threads statically (tuples or columns) Ex. VOLCANO and MONETDB A for t in A: emit(t) B for t in B: emit(t) Encapsulation of parallelism in the volcano query processing system G. Graefe Sigmod, 199 NUMA obliviousness through memory mapping M. Gawadi, M. Kersten Damon@Sigmod, 215

13 8 NOT NUMA-FRIENDLY PARALLELISM TPC-H Query 6 on 1GB database -node Quad-Core AMD Opteron 8 Series (6Gb RAM/Node) MonetDB (v )

14 8 NOT NUMA-FRIENDLY PARALLELISM TPC-H Query 6 on 1GB database -node Quad-Core AMD Opteron 8 Series (6Gb RAM/Node) MonetDB (v )

15 9 IMPACT OF REMOTE ACCESS TPC-H Q6: SQL vs. C language (raw performance) OS/C MonetDB Interconnect Traffic MB/s (1^3) Users

16 9 IMPACT OF REMOTE ACCESS TPC-H Q6: SQL vs. C language (raw performance) Dense/C Sparse/C OS/C MonetDB Interconnect Traffic MB/s (1^3) Not NUMA-friendly processing The OS isn t, either Users

17 9 IMPACT OF REMOTE ACCESS TPC-H Q6: SQL vs. C language (raw performance) Dense/C Sparse/C OS/C MonetDB Interconnect Traffic MB/s (1^3) Specific cores are allocated threads pinned to cores Users

18 9 IMPACT OF REMOTE ACCESS SQL query vs. C language (raw performance) GOAL: Dense/C Sparse/C OS/C MonetDB MITIGATE THE DATA MOVEMENT HANDING 8 OUT TO THE OS THE LOCAL OPTIMUM 6 NUMBER OF CORES IN SPECIFIC NODES Interconnect Traffic MB/s (1^3) Users DB is not NUMA-friendly OS isn t, too threads pinned to cores

19 9 IMPACT OF REMOTE ACCESS SQL query vs. C language (raw performance) CHALLENGES: LOCAL 8 OPTIMUM NUMBER OF CORES CPU-CORE ALLOCATION 6 Interconnect Traffic MB/s (1^3) 2 Dense/C Sparse/C OS/C MonetDB Users DB is not NUMA-friendly OS isn t, too threads pinned to cores

20 1 PETRINET MULTI-CORE ALLOCATION MECHANISM

21 11 OVERVIEW OF OUR MULTI-CORE ALLOCATION MECHANISM 2 Monitoring usage 1 Monitoring threads

22 11 OVERVIEW OF OUR MULTI-CORE ALLOCATION MECHANISM 2 Monitoring usage 1 Monitoring threads 3 Priority allocation

23 11 OVERVIEW OF OUR MULTI-CORE ALLOCATION MECHANISM Allocate cores

24 12 LOCAL OPTIMUM NUMBER OF CORES 1 CPU (%) # of Cores 8 CPU (%) # of Cores State Transition

25 12 LOCAL OPTIMUM NUMBER OF CORES 1 CPU (%) # of Cores 8 CPU (%) # of Cores State Transition

26 12 LOCAL OPTIMUM NUMBER OF CORES 1 CPU (%) # of Cores 8 CPU (%) # of Cores State Transition

27 12 LOCAL OPTIMUM NUMBER OF CORES CPU (%) Overloaded state CPU (%) # of Cores # of Cores State Transition Idle state

28 13 PETRINET MULTI-CORE ALLOCATION MECHANISM (DEFINITIONS) Places={Stable, Idle, Overload, Provision, Checks} Transitions={t,..., t7 }, (i.e., conditions) Arcs=<pi, tj > or <tj, pi>, (i.e., I/O functions)

29 1 OVERLOADED STATE Monitoring CPU (%) # of Cores State Transition

30 1 OVERLOADED STATE Overloaded state CPU (%) # of Cores State Transition

31 1 OVERLOADED STATE Overloaded state Adding cores CPU (%) # of Cores Action: allocate cores State Transition

32 15 ALLOCATION MODES Sparse Dense Node Node 1 Node Node Node 2 Node 3 Node 2 Node 3 Time Time allocated next to allocate

33 15 ALLOCATION MODES Sparse Dense Node Node 1 Node Node Node 2 Node 3 Node 2 Node 3 Time Time allocated next to allocate

34 16 ADAPTIVE MODE Node Node 1 Release Priority queue Allocate # of DB Thread Pages + Node 2 Node 3 /proc/[pid]/numa_maps allocated next

35 16 ADAPTIVE MODE Node Node 1 Release Priority queue Allocate # of DB Thread Pages + Node 2 Node 3 /proc/[pid]/numa_maps allocated next

36 16 ADAPTIVE MODE Node Node 1 Release Priority queue Allocate # of DB Thread Pages + Node 2 Node 3 /proc/[pid]/numa_maps allocated next

37 EXPERIMENTS 17

38 18 SETUP TPC-H 1GB and 1GB database up to 256 users -node Quad-Core AMD Opteron 8 Series (6GB RAM/Node), Agr. Bandwidth 1.6 Gb/s, L3 (6MB) Debian Linux 8 Jessie Prototype in C language Database systems: MonetDB (v ) NUMA-Aware Microsoft SQL Server (v217 Dev. RC2)

39 T5 T5 T6 19 IMPACT OF THE MIGRATION OF THREADS (TPC-H QUERY 6, 1 USER, 1 GB DATABASE, MONETDB) T6 T7 T8 T9 T1 T11 T12 T13 T1 T15 T16 T17 T18 T19 T2 Threads Core Number Timestamp (ms) OS/MonetDB Sparse (c) (a) T7 T8 T9 T1 T11 T12 T13 T1 T15 T16 T17 T18 T19 T2 Dense (b) Adaptive (d)

40 2 CONCURRENT EXECUTION OF TPC-H (256 USERS, 1 GB DATABASE, MONETDB/SQLSERVER) Mem TP (GB/s) Mem TP (GB/s) Mem TP (GB/s) Mem TP (GB/s) (a) (b) OS/MonetDB Adaptive/MonetDB OS/SQL Server (c) 3 Adaptive/SQL Server Time (s) (d) S S1 S2 S3 MonetDB SQL Server

41 2 CONCURRENT EXECUTION OF TPC-H Mem TP (GB/s) Mem TP (GB/s) Mem TP (GB/s) Mem TP (GB/s) (256 USERS, 1 GB DATABASE, MONETDB/SQLSERVER) (a) (b) OS/MonetDB Adaptive/MonetDB OS/SQL Server (c) 3 Adaptive/SQL Server Time (s) (d) S S1 S2 S3 Less cores

42 21 NUMA-FRIENDLINESS: INTERCONNECTION RATIO,9 (256 USERS, 1 GB DATABASE, MONETDB, TPC-H) OS/MonetDB Dense Sparse Adaptive HT/IMC Ratio,6,3 Better Q9 Q13 Q19 Q22

43 22 NUMA-FRIENDLINESS: INTERCONNECTION RATIO,3 (256 USERS, 1 GB DATABASE, SQL SERVER, TPC-H) OS/MonetDB Dense Sparse Adaptive HT/IMC Ratio,2,1 Better Q9 Q13 Q19 Q22

44 23 AN ELASTIC MULTI-CORE ALLOCATION MECHANISM FOR DATABASE SYSTEMS CONCLUSIONS

45 2 CONCLUSIONS Abstract model for the allocation of cores Thread scheduling works better with less cores (local optimum number of cores) Less thread migrations with less data movement: up to 3.87x of reduction on traffic ratio Up to 26% on energy savings

46 THANKS! ACKNOWLEDGMENTS: FUNDAÇAO ARAUCARIA, CAPES, CNPQ AND FUNDO NACIONAL DE DESENVOLVIMENTO DA EDUCAÇÃO (FNDE).

47 23 NUMA-FRIENDLINESS: INTERCONNECTION RATIO (MONETDB, 256 USERS, 1 GB DATABASE, TPC-H) Execution Speedup (Adaptive Mode) HT/IMC ratio Q1 Q9 Q8 Q7 Q6 Q5 Q Q3 Q2 Q1 Q11 Q12 Q13 Queries Q22 Q21 Q2 Q19 Q18 Q17 Q16 Q15 Q1 OS/MonetDB Dense Sparse Adaptive

48 23 NUMA-FRIENDLINESS: INTERCONNECTION RATIO (SQL SERVER, 256 USERS, 1 GB DATABASE, TPC-H) Execution Speedup (Adaptive Mode) HT/IMC ratio Q1 Q9 Q8 Q7 Q6 Q5 Q Q3 Q2 Q1 Q11 Q12 Q13 Queries Q22 Q21 Q2 Q19 Q18 Q17 Q16 Q15 Q1 OS/SQL server Dense Sparse Adaptive

49 23 ENERGY SAVINGS (MONETDB, 256 USERS, 1 GB DATABASE, TPC-H) 18 26% 16 Energy (J) Q22 Q21 Q2 Q19 Q18 Q17 Q16 Q15 Q1 Q13 Q12 Q11 Q1 Q9 Q8 Q7 Q6 Q5 Q Q3 Q2 Q1 Q22 Q21 Q2 Q19 Q18 Q17 Q16 Q15 Q1 Q13 Q12 Q11 Q1 Q9 Q8 Q7 Q6 Q5 Q Q3 Q2 Q1 OS scheduler/monetdb Adaptive (a) CPU HT (b)

50 23 ENERGY SAVINGS (SQL SERVER, 256 USERS, 1 GB DATABASE, TPC-H) 7 11% 6 Energy (J) Q22 Q21 Q2 Q19 Q18 Q17 Q16 Q15 Q1 Q13 Q12 Q11 Q1 Q9 Q8 Q7 Q6 Q5 Q Q3 Q2 Q1 Q22 Q21 Q2 Q19 Q18 Q17 Q16 Q15 Q1 Q13 Q12 Q11 Q1 Q9 Q8 Q7 Q6 Q5 Q Q3 Q2 Q1 OS scheduler/sql Server Adaptive CPU HT

51 12 PETRINET MULTI-CORE ALLOCATION MECHANISM (DEFINITION: LOCAL OPTIMUM NUMBER OF CORES) w is the current workload of the database threads nalloc is the number of allocated CPU cores (nalloc ntotal )

52 12 PETRINET MULTI-CORE ALLOCATION MECHANISM (DEFINITION: LOCAL OPTIMUM NUMBER OF CORES) u is the average resource usage of database threads between minimum and maximum thresholds ntotal is the number of available CPU cores p(x) is the performance function, where x assumes the number of CPU cores nalloc or ntotal