On-Disk Bitmap Index In Bizgres

Transcription

1 On-Disk Bitmap Index In Bizgres Ayush Parashar and Jie Zhang 1

2 Agenda Introduction to On-Disk Bitmap Index Bitmap index creation Bitmap index creation performance - index size and creation time Performance with varying cardinality Query performance Summary 2

3 Introduction Access method specially efficient for lowcardinality column of high-dimensional fact table Takes a fraction of space as compared to B-tree Very less index creation time Very effective on queries with multiple-conditions in where clause 3

4 Introduction: continued.. Consider the following data: 4

5 Introduction: B-Tree representation 5

6 Introduction: Bitmap representation 6

7 Bitmap index creation Steps in bitmap-index creation Build bin indexes Apply encoding schemes: Equality encoding scheme used in the present implementation Apply compression schemes Contributors: Jie Zhang, Mark Kirkwood, Gavin Sherry 7

8 Bitmap index creation: continued.. 8

9 On-disk Bitmap Index - Performance Table definition from OSDL - DBT 3 benchmark Size of table is corresponding to scale factor 10 for DBT 3 Table Name Indexed Column Data Type Cardinality LINEITEM L_SHIPMODE character(10) 7 LINEITEM L_QUANTITY numeric(15,2) 50 LINEITEM L_LINENUMBER integer 7 LINEITEM L_SHIPMODE, L_QUANTITY * multi-column index character(10), numeric(15,2) 350 ORDERS O_ORDERSTATUS character(1) 3 ORDERS O_ORDERPRIORITY character(15) 5 CUSTOMER C_MKTSEGMENT character(10) 5 CUSTOMER C_NATIONKEY integer 25 9

10 On-disk Bitmap Index - Index Size Performance Index size - Size of bitmap index is a fraction of B-tree index 10

11 On-disk Bitmap Index - Creation Time Performance Index creation time: Up to 7 times faster index creation 11

12 On-disk Bitmap Index - Performance with varying cardinality Index size with varying cardinality: Total rows 250 million 12

13 On-disk Bitmap Index - Performance with varying cardinality Index creation time with varying cardinality: Total rows 250 million 13

14 On-disk Bitmap Index - Query Performance Query 1 SELECT sum(lineitem.l_discount) FROM lineitem, orders, customer, nation WHERE nation.n_name='united STATES' AND customer.c_mktsegment='automobile' AND orders.o_orderstatus='p' AND orders.o_orderpriority='2-high' AND lineitem.l_quantity=5 AND lineitem.l_shipmode='air' AND lineitem.l_linenumber=5 AND customer.c_custkey=orders.o_custkey AND orders.o_orderkey=lineitem.l_orderkey AND nation.n_nationkey=customer.c_nationkey; Query 2 SELECT avg(lineitem.l_tax) FROM lineitem, orders WHERE orders.o_orderstatus='f' AND orders.o_orderpriority='4-not SPECIFIED' AND lineitem.l_linenumber=5 AND lineitem.l_shipmode='truck' AND lineitem.l_quantity=2 AND orders.o_orderkey=lineitem.l_orderkey; Query 3 SELECT count(*) FROM lineitem WHERE l_linenumber=1; Query 4 SELECT count(*) FROM lineitem WHERE l_linenumber in (1,2) AND l_shipmode IN ('RAIL','TRUCK'); Query 5 SELECT count(*) FROM lineitem WHERE l_linenumber=5 AND l_shipmode='rail' AND l_quantity=18; 14

15 On-disk Bitmap Index - Query Performance Query Performance: Run1, Run2 and Run3 indicate that the same query has been run consecutively three-times 15

16 On-disk Bitmap Index - Query Performance Example: Query I explain plan output QUERY PLAN Aggregate (cost= rows=1 width=9) -> Merge Join (cost= rows=1649 width=9) Merge Cond: ("outer".l_orderkey = "inner".o_orderkey) -> Sort (cost= rows=17412 width=13) Sort Key: lineitem.l_orderkey -> Bitmap Heap Scan on lineitem (cost= rows=17412 width=13) Recheck Cond: ((l_quantity = 2::numeric) AND (l_linenumber = 5) AND (l_shipmode = 'TRUCK'::bpchar)) -> BitmapAnd (cost= rows=17412 width=0) -> Bitmap Index Scan on l_quantity_bm_idx(on-disk bitmap index) (cost= rows= width=0) Index Cond: (l_quantity = 2::numeric) -> Bitmap Index Scan on l_linenumber_bm_idx(on-disk bitmap index) (cost= rows= width=0) Index Cond: (l_linenumber = 5) -> Bitmap Index Scan on l_shipmode_bm_idx(on-disk bitmap index) (cost= rows= width=0) Index Cond: (l_shipmode = 'TRUCK'::bpchar) -> Sort (cost= rows= width=4) (21 rows) Sort Key: orders.o_orderkey -> Bitmap Heap Scan on orders (cost= rows= width=4) Recheck Cond: (o_orderpriority = '4-NOT SPECIFIED'::bpchar) Filter: (o_orderstatus = 'F'::bpchar) -> Bitmap Index Scan on o_orderpriority_bm_idx(on-disk bitmap index) (cost= rows= width=0) Index Cond: (o_orderpriority = '4-NOT SPECIFIED'::bpchar) 16

17 Summary On-disk Bitmap Index: New feature in Bizgres 0.9 Provides dramatic improvements in index creation time and space used by the index Dramatically improves response time for large classes of ad hoc data-warehousing queries 17

18 Thanks Questions..? References Bitmap Index Design and Evaluation: Chee-Yong Chan and Yannis E. Ioannidis Compressed bitmap indices for efficient query processing: Kesheng Wu Ekow J. Otoo Arie Shoshani On-Disk Bitmap Index Performance in Bizgres A Greenplum Whitepaper: