Bloom Filters DOAG Webinar, 12 August 2016 Christian Antognini Senior Principal Consultant

Size: px

Start display at page:

Download "Bloom Filters DOAG Webinar, 12 August 2016 Christian Antognini Senior Principal Consultant"

David Daniels
5 years ago
Views:

1 DOAG Webinar, 12 August 2016 Christian Antognini Senior Principal Consultant BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

@ChrisAntognini Senior principal consultant,

antognini@trivadis.com http://antognini.

Logical and physical database design Query

Author of Troubleshooting Oracle Performance

2 @ChrisAntognini Senior principal consultant, trainer and partner at Trivadis Focus: get the most out of Oracle Database Logical and physical database design Query optimizer Application performance management Author of Troubleshooting Oracle Performance (Apress, 2008/14) OakTable Network, Oracle ACE Director 2 12/08/2016

3 Agenda 1. What Is a Bloom Filter? 2. A Simple Implementation in PL/SQL 3. and Oracle Database 4. Hash Joins Parallel Execution Exadata In-Memory Column Store 5. Partition Pruning 3 12/08/2016

4 What Is a Bloom Filter? 4 12/08/2016

5 What Is a Bloom Filter? A Bloom filter isn t something specific to Oracle Database. It was first developed in 1970 by Burton H. Bloom long before Oracle existed. A Bloom filter is a data structure used to support membership queries. 5 12/08/2016

6 What Are the Properties of a Bloom Filter? The amount of space needed to store the Bloom filter is small compared to the amount of data belonging to the set being tested. The time needed to check whether an element is a member of a given set is independent of the number of elements contained in the set. False negatives aren t possible. False positives are possible, but their frequency can be controlled. 6 12/08/2016

7 How Does a Bloom Filter Work? A Bloom filter is based on an array of m bits (b 1, b 2,, b m ) that are initially set to 0. To set the bits to 1, k independent hash functions (h 1, h 2,, h k ), each returning a value between 1 and m, are used. In order to store a given element into the bit array, each hash function must be applied to it and, based on the return value r of each function (r 1, r 2,, r k ), the bit with the offset r is set to 1. Since there are k hash functions, up to k bits in the bit array are set to /08/2016

8 How Does a Bloom Filter Work? Example (m=16, k=4) r 1 = h 1 (e) r 2 = h 2 (e) r 3 = h 3 (e) r 4 = h 4 (e) = 8 = 1 = 6 = 13 set b 8 to 1 set b 1 to 1 set b 6 to 1 set b 13 to 1 Bit Array b 1 b 2 b 3 b 4 b 5 b 6 b 7 b 8 b 9 b 10 b 11 b 12 b 13 b 14 b 15 b /08/2016

9 How Does a Bloom Filter Work? Interactive Demo Jason Davies wrote a Bloom filter implementation in JavaScript: /08/2016

10 When to Use a Bloom Filter? Use Case λ [msg/s] ϰ [msg/s] ϰ λ System Bloom Cache Data Filter Remote Service 10 12/08/2016

11 A Simple Implementation in PL/SQL 11 12/08/2016

12 A Simple Implementation in PL/SQL bloom_filter.sql To provide you with a concrete example, I ve written a very simple (i.e, neither space nor time efficient) Bloom filter in PL/SQL. The package implementing it provides three routines: init to initialize the bit array add_value to store a string in the bit array contain to check whether a string is stored in the bit array 12 12/08/2016

13 and Oracle Database 13 12/08/2016

14 and Oracle Database The Oracle Database uses Bloom filters in several situations: To reduce data transfer between processes in parallel hash joins (10.2+). Exadata offloading (11.1+) To implement a partition pruning technique (11.1+). To support result caches (11.1+). To reduce data transfer between Exadata storage cells and database servers in serial hash joins ( ) To optimize in-memory scans returning data to hash joins ( ) 14 12/08/2016

15 Hash Joins Parallel Execution 15 12/08/2016

16 Parallel Hash Joins Without Bloom Filter (1) When a hash join is executed in parallel, there are several sets of processes that exchange data. Part of the data sent by the processes may be discarded because it doesn't fulfill the join condition /08/2016

17 Parallel Hash Joins Without Bloom Filter (2) SELECT * FROM t1, t2 WHERE t1.id = t2.id AND t1.mod = Operation Name TQ IN-OUT PQ Distrib SELECT STATEMENT PX COORDINATOR PX SEND QC (RANDOM) HASH JOIN BUFFERED :TQ10002 Q1,02 P->S QC (RAND) Q1,02 PCWP PX RECEIVE Q1,02 PCWP PX SEND HASH :TQ10000 Q1,00 P->P HASH PX BLOCK ITERATOR Q1,00 PCWC TABLE ACCESS FULL T1 Q1,00 PCWP PX RECEIVE Q1,02 PCWP PX SEND HASH :TQ10001 Q1,01 P->P HASH PX BLOCK ITERATOR Q1,01 PCWC TABLE ACCESS FULL T2 Q1,01 PCWP /08/2016

18 Parallel Hash Joins With Bloom Filter (1) Bloom Filter ❶ Q1,02 creates the Bloom filter ❷ Q1,01 uses the Bloom filter 18 12/08/2016

19 Parallel Hash Joins With Bloom Filter (2) Operation Name TQ IN-OUT PQ Distrib SELECT STATEMENT PX COORDINATOR PX SEND QC (RANDOM) HASH JOIN BUFFERED :TQ10002 Q1,02 P->S QC (RAND) Q1,02 PCWP PX JOIN FILTER CREATE :BF0000 Q1,02 PCWP PX RECEIVE PX SEND HASH Q1,02 PCWP :TQ10000 Q1,00 P->P HASH PX BLOCK ITERATOR Q1,00 PCWC TABLE ACCESS FULL T1 PX RECEIVE Q1,00 PCWP Q1,02 PCWP PX SEND HASH :TQ10001 Q1,01 P->P HASH PX JOIN FILTER USE :BF0000 PX BLOCK ITERATOR Q1,01 PCWP Q1,01 PCWC TABLE ACCESS FULL T2 Q1,01 PCWP /08/2016

20 Parallel Hash Joins With Bloom Filter (3) The query optimizer decides whether a Bloom filter is to be used based on the estimated join selectivity and on the amount of data to be processed. Hints PX_JOIN_FILTER and NO_PX_JOIN_FILTER can override decision. To display runtime statistics about Bloom filters, V$SQL_JOIN_FILTER exists. SQL> SELECT filtered, probed, probed-filtered AS sent 2 FROM v$sql_join_filter 3 WHERE qc_session_id = sys_context('userenv','sid'); FILTERED PROBED SENT /08/2016

21 Hash Joins Exadata 21 12/08/2016

22 Exadata and Hash Joins With Bloom filters can be offloaded to Exadata storage cells. Parallel hash joins (11.1+) Serial hash joins ( ) Previous releases and non-exadata deployments support them in very limited scenarios 22 12/08/2016

23 Serial Hash Joins With Offloaded Bloom Filter SELECT * FROM t1, t2 WHERE t1.id = t2.id AND t1.mod = Id Operation Name SELECT STATEMENT * 1 HASH JOIN 2 JOIN FILTER CREATE :BF0000 * 3 TABLE ACCESS STORAGE FULL T1 4 JOIN FILTER USE :BF0000 * 5 TABLE ACCESS STORAGE FULL T access("t1"."id"="t2"."id") 3 - storage("t1"."mod"=42) filter("t1"."mod"=42) 5 - storage(sys_op_bloom_filter(:bf0000,"t2"."id")) filter(sys_op_bloom_filter(:bf0000,"t2"."id")) 23 12/08/2016

24 Hash Joins In-Memory Column Store (IMCS) 24 12/08/2016

25 IMCS and Hash Joins With In-memory scans returning data to hash joins can take advantage of Bloom filters. At least the probe input must be IM-enabled Supported for both serial and parallel executions 25 12/08/2016

26 In-memory Scan With Bloom Filter SELECT * FROM t1, t2 WHERE t1.id = t2.id AND t1.mod = Id Operation Name SELECT STATEMENT * 1 HASH JOIN 2 JOIN FILTER CREATE :BF0000 * 3 TABLE ACCESS INMEMORY FULL T1 4 JOIN FILTER USE :BF0000 * 5 TABLE ACCESS INMEMORY FULL T access("t1"."id"="t2"."id") 3 - inmemory("t1"."mod"=42) filter("t1"."mod"=42) 5 - inmemory(sys_op_bloom_filter(:bf0000,"t2"."id")) filter(sys_op_bloom_filter(:bf0000,"t2"."id")) 26 12/08/2016

27 Partition Pruning 27 12/08/2016

28 Subquery Pruning The query optimizer is able to perform partition pruning based on literal values, bind variables or join conditions. Up to 10.2, when partition pruning is based on join conditions, the query optimizer can use subquery pruning for hash and merge joins. Subquery pruning is of limited usage because part of the query is executed twice /08/2016

29 Join-filter Pruning To avoid the double execution related to subquery pruning, provides another type of partition pruning: join-filter pruning. SELECT * FROM t1, t2 WHERE t1.id = t2.id AND t1.mod = Operation Name Pstart Pstop SELECT STATEMENT HASH JOIN PART JOIN FILTER CREATE :BF0000 PARTITION HASH ALL 1 8 TABLE ACCESS FULL T1 1 8 PARTITION HASH JOIN-FILTER :BF0000 :BF0000 TABLE ACCESS FULL T2 :BF0000 :BF /08/2016

Core Messages Bloom filters aren t specific to Oracle Database, they were invented in 1970.

30 Core Messages Bloom filters aren t specific to Oracle Database, they were invented in In Oracle Database Bloom filters are used to optimize various operations. They are devoted to improving the performance of SQL statements that process large amounts of data /08/2016

31 References Bloom, Burton H., Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, Antognini, Christian, Davies, Jason, JavaScript implementation. Oracle Corporation, Readme Information for Oracle Database 11g Release 2 ( ). Oracle Database Documentation, Lewis, Jonathan, Bloom Filter. Oracle Scratchpad Blog, Koppelaars, Toon, /08/2016

Christian Antognini Senior Principal Consultant

32 Christian Antognini Senior Principal 32 12/08/2016

Oracle Database 18c New Performance Features

Oracle Database 18c New Performance Features Christian Antognini @ChrisAntognini antognini.ch/blog BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART