Semi-Joins and Bloom Join. Databases: The Complete Book Ch 20

Transcription

1 Semi-Joins and Bloom Join Databases: The Complete Book Ch 20 1

2 Practical Concerns UNION R1 S1 R1 S2 R2 S1 RN SM R1 R2 RN S1 S2 SM 2

3 Practical Concerns UNION R1 S1 R1 S2 R2 S1 RN SM R1 R2 RN S1 S2 SM Where does the computation happen? How does the data get there? 2

4 Distributing the Work Let s start simple what can we do with no partitions? B R S 3

5 Distributing the Work Let s start simple what can we do with no partitions? B R S R and S may be any RA expression 3

6 Distributing the Work B R S Node 1 4

7 Distributing the Work B R S Node 1 No Parallelism! 4

8 Distributing the Work B Node 3 R Node 1 S Node 2 5

9 Distributing the Work All of R and All of S get sent! B Node 3 R Node 1 S Node 2 Lots of Data Transfer! 5

10 Distributing the Work B R Node 1 S Node 2 6

11 Distributing the Work All of R get sent B R Node 1 S Node 2 Better! We can guess whether R or S is smaller. 6

12 Distributing the Work What can we do if R is partitioned? U B B R1 S R2 7

13 Distributing the Work There are lots of partitioning strategies, but this one is interesting. U B B R1 Node 1 S R2 Node 2 Node 3 8

14 Distributing the Work it can be used as a model for partitioning S U B B R1 Node 1 S1 R2 Node 2 Node 3 9

15 Distributing the Work it can be used as a model for partitioning S U B B R1 Node 1 S2 R2 Node 2 Node 3 10

16 Distributing the Work and neatly captures the data transfer issue. U B B R1 Node 1 S R2 Node 2 Node 3 11

17 Distributing the Work So let s use it: Si joins with R1,R2,,RN locally. 12

18 Distributing the Work So let s use it: Si joins with R1,R2,,RN locally. Goal: Minimize amount of data sent from Rk to Si 12

19 Distributing the Work So let s use it: Si joins with R1,R2,,RN locally. Goal: Minimize amount of data sent from Rk to Si Solution 1: Use the partitioning strategy (like last lecture) 12

20 Distributing the Work So let s use it: Si joins with R1,R2,,RN locally. Goal: Minimize amount of data sent from Rk to Si Solution 1: Use the partitioning strategy (like last lecture) Solution 2: Hints to figure out what Rk should send 12

21 Sending Hints Rk B Si The naive approach Node 1 Node 2 Rk Si 13

22 Sending Hints Rk B Si The naive approach Node 1 Send me Node 2 Rk Rk Si 14

23 Sending Hints Rk B Si The naive approach Rk Node 1 Node 2 Rk Si 15

24 Sending Hints Rk B Si The smarter approach πb( ) Si Node 1 Node 2 Rk Si 16

25 Sending Hints Rk B Si The smarter approach πb( ) Si Node 1 Node 2 πb( ) Rk Rk Si Si 17

26 Sending Hints Rk B Si The smarter approach Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> 18 <2,X> <3,Y> <6,Y>

27 Sending Hints Rk B Si The smarter approach Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> 19 Send me rows with a B of 2,3, or 6 <2,X> <3,Y> <6,Y>

28 Sending Hints Rk B Si The smarter approach Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <3,D> 20 Send me rows with a B of 2,3, or 6 <2,X> <3,Y> <6,Y>

29 Sending Hints Rk B Si The smarter approach Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <3,D> 20 Send me rows with a B of 2,3, or 6 This is called a semi-join. <2,X> <3,Y> <6,Y>

30 Sending Hints Now Node 1 sends as little data as possible but Node 2 needs to send a lot of data. Can we do better? 21

31 Sending Hints Rk B Si Strategy 1: Parity Bits Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,X> 0 <6,Y>

32 Sending Hints Rk B Si Strategy 1: Parity Bits Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> Send me data with a parity bit of 0 0 <2,X> 0 <6,Y>

33 Sending Hints Rk B Si Strategy 1: Parity Bit Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <4,E> 24 Send me data with a parity bit of 0 0 <2,X> 0 <6,Y>

34 Sending Hints Rk B Si Strategy 1: Parity Bit Node 1 sending too much is ok! (Node 2 still needs to compute B) Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <4,E> 24 Send me data with a parity bit of 0 0 <2,X> 0 <6,Y>

35 Sending Hints Rk B Si Strategy 1: Parity Bit Node 1 sending too much is ok! (Node 2 still needs to compute B) Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <4,E> 24 Send me data with a parity bit of 0 Problem: One parity bit is too little 0 <2,X> 0 <6,Y>

36 Sending Hints Rk B Si Strategy 1: Parity Bit Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> Problem: One parity bit is too little 25 0 <2,X> 1 <3,Y> 0 <6,Y>

37 Sending Hints Rk B Si Strategy 2: Parity Bits Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <3,D> 26 Send me data with parity bits 10 or 11 10<2,X> 11<3,Y> 10<6,Y>

38 Sending Hints Rk B Si Strategy 2: Parity Bits Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,B> <2,C> <3,D> 26 Send me data with parity bits 10 or 11 Problem: Almost as much data as πb 10<2,X> 11<3,Y> 10<6,Y>

39 Sending Hints Can we summarize the parity bits? 27

40 Bloom Filters Alice Bob Carol Dave 28

41 Bloom Filters Alice Bob Carol Dave Bloom Filter 29

42 Bloom Filters Is Alice part of the set? Alice Bob Carol Dave Bloom Filter 30

43 Bloom Filters Yes Is Alice part of the set? Alice Bob Carol Dave Bloom Filter 30

44 Bloom Filters Yes Is Alice part of the set? Alice Bob Carol Dave Bloom Filter Is Eve part of the set? 30

45 Bloom Filters Yes Is Alice part of the set? Alice Bob Carol Bloom Filter No Is Eve part of the set? Dave 30

46 Bloom Filters Yes Is Alice part of the set? Alice Bob Carol Bloom Filter No Is Eve part of the set? Dave Is Fred part of the set? 30

47 Bloom Filters Yes Is Alice part of the set? Alice Bob Carol Bloom Filter No Is Eve part of the set? Dave Yes Is Fred part of the set? 30

48 Bloom Filters Yes Is Alice part of the set? Alice Bob Carol Bloom Filter No Is Eve part of the set? Dave Bloom Filter Guarantee Test definitely returns Yes if the element is in the set Test usually returns No if the element is not in the set Yes Is Fred part of the set? 30

49 Bloom Filters A Bloom Filter is a bit vector M - # of bits in the bit vector K - # of hash functions For ONE key (or record): For i between 0 and K: bitvector[ hashi (key) % M ] = 1 31

50 Bloom Filters A Bloom Filter is a bit vector M - # of bits in the bit vector K - # of hash functions For ONE key (or record): For i between 0 and K: bitvector[ hashi (key) % M ] = 1 Each bit vector has ~K bits set 31

51 Bloom Filters Key 1 Key Filters are combined by Bitwise-OR e.g. (Key 1 Key 2) = Key Key

52 Bloom Filters Key 1 Key 2 Key Filters are combined by Bitwise-OR e.g. (Key 1 Key 2) = How do we test for inclusion? Key

53 Bloom Filters Key 1 Key 2 Key 3 Key Filters are combined by Bitwise-OR e.g. (Key 1 Key 2) = How do we test for inclusion? (Key & Filter) == Key? (Key 1 & S) = (Key 3 & S) = (Key 4 & S) =

54 Bloom Filters Key 1 Key 2 Key Filters are combined by Bitwise-OR e.g. (Key 1 Key 2) = How do we test for inclusion? (Key & Filter) == Key? Key (Key 1 & S) = (Key 3 & S) = (Key 4 & S) =

55 Bloom Filters Key 1 Key 2 Key Filters are combined by Bitwise-OR e.g. (Key 1 Key 2) = How do we test for inclusion? (Key & Filter) == Key? Key (Key 1 & S) = (Key 3 & S) = X (Key 4 & S) =

56 Bloom Filters Key 1 Key 2 Key Filters are combined by Bitwise-OR e.g. (Key 1 Key 2) = How do we test for inclusion? (Key & Filter) == Key? Key (Key 1 & S) = (Key 3 & S) = X 32 (Key 4 & S) = False Positive

57 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> 33 <2,X> <3,Y> <6,Y>

58 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 with a B in the Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> 34 Send me rows bloom filter summarizing the set {2,3,6} <2,X> <3,Y> <6,Y>

59 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 <2,B> with a B in the Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,C> <3,D> <4,E> 35 Send me rows bloom filter summarizing the set {2,3,6} <2,X> <3,Y> <6,Y>

60 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 <2,B> with a B in the Node 2 <1,A> <2,B> <2,C> <3,D> <4,E> <2,C> <3,D> <4,E> This is called a bloom-join. 35 Send me rows bloom filter summarizing the set {2,3,6} <2,X> <3,Y> <6,Y>