The Internals of the Monet Database

Transcription

1 Bogdan Dumitriu Lee Provoost Department of Computer Science University of Utrecht June 6, 2005

2 Outline 1 Classical databases Monet database 2 3

3 Classical databases Classical databases Monet database Most existing databases: OLTP oriented high performance on large # of small updates table data clustered by row on disk unsuited for query-intensive due to a lot of unnecessary I/O

4 Classical databases Classical databases Monet database Most existing databases: OLTP oriented high performance on large # of small updates table data clustered by row on disk unsuited for query-intensive due to a lot of unnecessary I/O

5 Classical databases Classical databases Monet database Most existing databases: OLTP oriented high performance on large # of small updates table data clustered by row on disk unsuited for query-intensive due to a lot of unnecessary I/O

6 Classical databases Classical databases Monet database Most existing databases: OLTP oriented high performance on large # of small updates table data clustered by row on disk unsuited for query-intensive due to a lot of unnecessary I/O

7 Vertical fragmentation Classical databases Monet database Keyword: vertical fragmentation Id Name Postal Code Date of Birth 1 John 2345 BP Jane 6146 TY Bob 8127 PR Id Name 1 John 2 Jane 3 Bob Id Postal Code BP TY PR Id Date of Birth

8 Monet goals Classical databases Monet database The goals of the Monet database: 1 primary: achieve high performance on query-intensive applications 2 support multiple logical data models 3 providing parallelism 4 extensibility to specific application domains

9 Monet goals Classical databases Monet database The goals of the Monet database: 1 primary: achieve high performance on query-intensive applications 2 support multiple logical data models 3 providing parallelism 4 extensibility to specific application domains

10 Monet model Classical databases Monet database The model employed by Monet: Decomposed Storage Model kernel of primitives on binary tables vertical fragmentation is explicit a simple, elegant and very flexible model downside: a lot of joining (partially solved)

11 Monet model Classical databases Monet database The model employed by Monet: Decomposed Storage Model kernel of primitives on binary tables vertical fragmentation is explicit a simple, elegant and very flexible model downside: a lot of joining (partially solved)

12 Monet model Classical databases Monet database The model employed by Monet: Decomposed Storage Model kernel of primitives on binary tables vertical fragmentation is explicit a simple, elegant and very flexible model downside: a lot of joining (partially solved)

13 Monet model Classical databases Monet database The model employed by Monet: Decomposed Storage Model kernel of primitives on binary tables vertical fragmentation is explicit a simple, elegant and very flexible model downside: a lot of joining (partially solved)

14 Monet and main memory Classical databases Monet database Monet is above all a main memory DBMS: shifts cost of processing from I/O to CPU cycles both its algorithms & its data structures are optimized for main memory access not a main memory-only DBMS, though: uses OS-controlled virtual memory during operation uses disk for long-term storage (naturally)

15 Monet architecture

16 in a nutshell

17 MIL types 1. t A f A v t T atomic data types fixed: A f = {bit, chr, sht, int, lng, flt, dbl, oid} variable A v = {str} 2. T 1, T 2 T bat[t 1, T 2 ] T the BAT (Binary Table) type each tuple in a BAT called a Binary Unit (BUN) left column - head right column - tail can be nested

18 MIL types 1. t A f A v t T atomic data types fixed: A f = {bit, chr, sht, int, lng, flt, dbl, oid} variable A v = {str} 2. T 1, T 2 T bat[t 1, T 2 ] T the BAT (Binary Table) type each tuple in a BAT called a Binary Unit (BUN) left column - head right column - tail can be nested

19 Ingmar s question Ingmar: On page 6 it is mentioned that MIL supports nested BATs. That sounds really interesting, but what are they (BATs within BATs, because that doesn t sound like a BAT anymore)?

20 MIL features Main features of the language: basic unit of execution: the operator operators can be overloaded, most are also polymorphic MIL is a dynamically typed language a procedural block-structured language (if-then-else, allows extension modules provides the usual operators (=,, <, >,,, etc.)

21 MIL features Main features of the language: basic unit of execution: the operator operators can be overloaded, most are also polymorphic MIL is a dynamically typed language a procedural block-structured language (if-then-else, allows extension modules provides the usual operators (=,, <, >,,, etc.)

22 MIL features Main features of the language: basic unit of execution: the operator operators can be overloaded, most are also polymorphic MIL is a dynamically typed language a procedural block-structured language (if-then-else, allows extension modules provides the usual operators (=,, <, >,,, etc.)

23 MIL features Main features of the language: basic unit of execution: the operator operators can be overloaded, most are also polymorphic MIL is a dynamically typed language a procedural block-structured language (if-then-else, allows extension modules provides the usual operators (=,, <, >,,, etc.)

24 MIL features Main features of the language: basic unit of execution: the operator operators can be overloaded, most are also polymorphic MIL is a dynamically typed language a procedural block-structured language (if-then-else, allows extension modules provides the usual operators (=,, <, >,,, etc.)

25 MIL features Main features of the language: basic unit of execution: the operator operators can be overloaded, most are also polymorphic MIL is a dynamically typed language a procedural block-structured language (if-then-else, allows extension modules provides the usual operators (=,, <, >,,, etc.)

26 Core functionality of MIL offered by a BAT algebra of operators which: have an algebraic definition are free of side-effects take BATs and return BATs closed algebra on BATs

27 The mirror operator Ingmar: What s the use of the mirror operator? Why would you want a table with identical columns? Possible uses: perform an set operation on a BAT (or on 2 BATs) perform a join operation... certainly many others

28 The mirror operator Id Name 4 Jack 2 Bill 1 Bob 6 Clare Id Name 1 Daniels 2 Gates 3 Hope 4 Jones 5 Heart 6 James mirror

29 The mirror operator Id Name 4 Jack 2 Bill 1 Bob 6 Clare Id Name 1 Daniels 2 Gates 3 Hope 4 Jones 5 Heart 6 James mirror

30 The mirror operator Id Id Id Name 1 Daniels 2 Gates 3 Hope 4 Jones 5 Heart 6 James join Id Id 4 Jones 2 Gates 1 Daniels 6 James

31 Grouping operators Id Name 1 Bob 2 Amy 3 Bob 4 Clare 5 Clare 6 Bob Id Year-of-birth unary group binary group Id GID Id GID

32 Grouping operators Id Name 1 Bob 2 Amy 3 Bob 4 Clare 5 Clare 6 Bob Id Year-of-birth unary group binary group Id GID Id GID

33 Grouping operators Id Name 1 Bob 2 Amy 3 Bob 4 Clare 5 Clare 6 Bob Id Year-of-birth unary group binary group Id GID Id GID

34 Split operator Id Name 1 Bob 2 Amy 3 Joe 4 Clare 5 Susan 6 Jeff split (n = 2) Id Id

35 Fragment operator Id Name 1 Bob 2 Amy 3 Joe 4 Clare 5 Susan 6 Jeff Id Id fragment Id 3 6 BAT 1 nil 2 nil 3 nil 4 nil 5 nil 6 nil

36 Multi-join map Let vol(x, y, z) = x y z be the function for computing the volume of a parallelepiped. Id W Id H Id L [vol] Id Vol

37 Pump GID Name 2 July 1 Ethan 2 Jill 1 Clare 3 Bob 4 Tim GID Name 2 July 1 Ethan 2 Jill 1 Clare 3 Bob 4 Tim GID count {count} 6 GID Count

38 Pump GID Name 2 July 1 Ethan 2 Jill 1 Clare 3 Bob 4 Tim GID Name 2 July 1 Ethan 2 Jill 1 Clare 3 Bob 4 Tim GID count {count} 6 GID Count

39 Peter s question... which I hope answers Peter s request: Peter: Can you show an example of how the pump operator is used, and the results it creates? Peter: What is the str s in the save, load and remove operators?

40 Peter s question... which I hope answers Peter s request: Peter: Can you show an example of how the pump operator is used, and the results it creates? Peter: What is the str s in the save, load and remove operators?

41 Peter s question A BAT Buffer Pool manages all known BATs It administers logical & physical names bbpname (BAT[any,any], str s) : bit can be used to name a BAT This name is global The str s refers to this logical name

42 Laurence s question Laurence: Can you show, in a step-by-step fashion how the OQL query on page 109 is translated to MIL?

43 Order class class Order { attribute date day; attribute float discount; relation Set<Item> items; } Item class class Item { attribute float price; attribute float tax; relation Order order inverse Order.items; }

44 Order class class Order { attribute date day; attribute float discount; relation Set<Item> items; } Item class class Item { attribute float price; attribute float tax; relation Order order inverse Order.items; }

45 The OQL query SELECT year, sum(total) FROM ( SELECT price * tax AS total, year(item.order.day) AS year FROM item WHERE order.discount BETWEEN 0.00 AND 0.06) GROUP BY year ORDER BY year

46 The Order class: order day oid date 100 4/4/ /4/ /2/ /4/ /2/ /2/98 order discount oid float order item oid oid

47 The Item class: item price oid float item tax oid float item order oid oid

48 ORD NIL := select(order discount, "between", 0.0, 0.6) order discount oid float ORD NIL oid oid 100 nil 102 nil 103 nil 104 nil

49 ORD SEL := ORD NIL.mark(oid(0)) ORD NIL oid oid 100 nil 102 nil 103 nil 104 nil ORD SEL oid oid

50 SEL DAY := join(ord SEL.reverse, order day, "=") ORD SEL.reverse oid oid order day oid date 100 4/4/ /4/ /2/ /4/ /2/ /2/98 SEL DAY oid date 0 4/4/98 1 1/2/98 2 9/4/98 3 7/2/98

51 SEL YEA := [year](sel DAY) SEL DAY oid date 0 4/4/98 1 1/2/98 2 9/4/98 3 7/2/98 SEL YEA oid int

52 GRP SEL := group(sel YEA).reverse SEL YEA oid int GRP SEL oid oid

53 GRP GRP := unique(grp SEL.mirror) GRP SEL.mirror oid int GRP GRP oid oid 0 0

54 GRP YEA := join(grp GRP, SEL YEA, "=") GRP GRP oid oid 0 0 SEL YEA oid int GRP YEA oid int 0 98

55 ITM SEL := join(item order, ORD SEL, "=") item order oid oid ORD SEL oid oid ITM SEL oid oid

56 UNQ ITM := ITM SEL.mark(oid(0)).reverse ITM SEL oid oid UNQ ITM oid oid

57 SEL UNQ := ITM SEL.reverse.mark(oid(0)) ITM SEL oid oid SEL UNQ oid oid

58 UNQ PRI := join(unq ITM, item price, "=") UNQ ITM oid oid item price oid float UNQ PRI oid float

59 UNQ TAX := join(unq ITM, item tax, "=") UNQ ITM oid oid item tax oid float UNQ TAX oid float

60 UNQ TOT := [*](UNQ PRI, UNQ TAX) UNQ PRI oid float UNQ TAX oid float UNQ TOT oid float

61 GRP UNQ := join(grp SEL, SEL UNQ, "=") GRP SEL oid oid SEL UNQ oid oid GRP UNQ oid oid

62 GRP TOT := join(grp UNQ, UNQ TOT, "=") GRP UNQ oid oid UNQ TOT oid float GRP TOT oid float

63 GRP SUM := {sum}(grp TOT, GRP GRP) GRP TOT oid float GRP GRP oid oid 0 0 GRP SUM oid float

64 table("1", GRP YEA, GRP SUM) GRP YEA oid int 0 98 GRP SUM oid float Result: int float

65 Running the server Start the Monet server Mserver # Monet Database Server V4.6.2 # Copyright (c) , CWI. All rights reserved. # Compiled for <arch>; dynamically linked. # Visit for further info. MonetDB> Allowing MIL clients MonetDB>module(mapi); MonetDB>listen(50000).fork(); Shutting down MonetDB>quit();

66 Running the server Start the Monet server Mserver # Monet Database Server V4.6.2 # Copyright (c) , CWI. All rights reserved. # Compiled for <arch>; dynamically linked. # Visit for further info. MonetDB> Allowing MIL clients MonetDB>module(mapi); MonetDB>listen(50000).fork(); Shutting down MonetDB>quit();

67 Running the server Start the Monet server Mserver # Monet Database Server V4.6.2 # Copyright (c) , CWI. All rights reserved. # Compiled for <arch>; dynamically linked. # Visit for further info. MonetDB> Allowing MIL clients MonetDB>module(mapi); MonetDB>listen(50000).fork(); Shutting down MonetDB>quit();

68 Running the clients Start the Mapi client MapiClient # Monet Database Server V4.6.2 # Copyright (c) , CWI. All rights reserved. # Compiled for <arch>; dynamically linked. # Visit for further info. mil> Start Mknife java -jar Mknife jar (choose MIL demo)

69 Running the clients Start the Mapi client MapiClient # Monet Database Server V4.6.2 # Copyright (c) , CWI. All rights reserved. # Compiled for <arch>; dynamically linked. # Visit for further info. mil> Start Mknife java -jar Mknife jar (choose MIL demo)

70 Using the SQL front-end Start the SQL front end MonetDB>module(sql server); MonetDB>sql server start(); Use Mapi client MapiClient -l sql sql> Use JDBC client java -jar share/monetdb/lib/monetdb JDBC.jar -umonetdb (use monetdb as password)

71 Using the SQL front-end Start the SQL front end MonetDB>module(sql server); MonetDB>sql server start(); Use Mapi client MapiClient -l sql sql> Use JDBC client java -jar share/monetdb/lib/monetdb JDBC.jar -umonetdb (use monetdb as password)

72 Using the SQL front-end Start the SQL front end MonetDB>module(sql server); MonetDB>sql server start(); Use Mapi client MapiClient -l sql sql> Use JDBC client java -jar share/monetdb/lib/monetdb JDBC.jar -umonetdb (use monetdb as password)