THE HYPERDYADIC INDEX AND GENERALIZED INDEXING AND QUERY WITH PIQUE

Size: px

Start display at page:

Download "THE HYPERDYADIC INDEX AND GENERALIZED INDEXING AND QUERY WITH PIQUE"

Delilah Richard
6 years ago
Views:

1 THE HYPERDYADIC INDEX AND GENERALIZED INDEXING AND QUERY WITH PIQUE David A. Boyuka II, Houjun Tang, Kushal Bansal, Xiaocheng Zou, Scott Klasky, Nagiza F. Samatova 6/30/2015 1

2 OVERVIEW Motivation Formal Indexing Model Parallel Indexing and Query Unified Engine (PIQUE) Hyperdyadic Tree Index (HD-tree) Results 6/30/2015 2

3 OVERVIEW Motivation Formal Indexing Model Parallel Indexing and Query Unified Engine (PIQUE) Hyperdyadic Tree Index (HD-tree) Results 6/30/2015 3

INDEXING FOR SCIENTIFIC DATA Scientific datasets continue to grow to unprecedented scale Some simulations use 100K+ supercomputer cores for days/weeks Yield 100 TB multi-variable, read/append-only

4 INDEXING FOR SCIENTIFIC DATA Scientific datasets continue to grow to unprecedented scale Some simulations use 100K+ supercomputer cores for days/weeks Yield 100 TB multi-variable, read/append-only data Data production outstripping I/O, storage, and analysis capacity Index-accelerated querying enables exploratory analysis of extreme-scale data. Bitmap Indexing (FastBit) Inverted Indexing (ALACRITY) S3D temperature data render (courtesy Sriram Lakshminarasimhan) 6/30/2015 4

5 OBSERVATIONS Similarities between bitmap and ALACRITY indexing Bitmap (Bit vectors) 2, 7 0, 3, 5 1, 4, 6 ALACRITY (Inverted Lists) 6/30/2015 5

6 OBSERVATIONS Similarities between bitmap and ALACRITY indexing. Binning Bitmap (Bit vectors) 2, 7 0, 3, 5 1, 4, 6 ALACRITY (Inverted Lists) 6/30/2015 5

7 OBSERVATIONS Similarities between bitmap and ALACRITY indexing. Binning Data structure Bitmap (Bit vectors) 2, 7 0, 3, 5 1, 4, 6 ALACRITY (Inverted Lists) 6/30/2015 5

8 OBSERVATIONS Similarities between bitmap and ALACRITY indexing. Binning Data structure Set operations (Query evaluation) Bitmap (Bit vectors) 2, 7 0, 3, 5 1, 4, 6 ALACRITY (Inverted Lists) 6/30/2015 5

9 OBSERVATIONS Similarities between bitmap and ALACRITY indexing. Binning Data structure Set operations (Query evaluation) General indexing model to encompass both? Bitmap (Bit vectors) 2, 7 0, 3, 5 1, 4, 6 ALACRITY (Inverted Lists) Related? 6/30/2015 5

10 OBSERVATIONS Similarities between bitmap and ALACRITY indexing. Binning Data structure Set operations (Query evaluation) General indexing model to encompass both? New indexing methods from generalized model? Bitmap (Bit vectors) 2, 7 0, 3, 5 1, 4, 6 ALACRITY (Inverted Lists)???? Related? 6/30/2015 5

11 OVERVIEW Motivation Formal Indexing Model Parallel Indexing and Query Unified Engine (PIQUE) Hyperdyadic Tree Index (HD-tree) Results 6/30/2015 6

12 BACKGROUND: THE BITMAP INDEX Bit i in bitmap j is set iff the ith record has value j A = {X, Y, Z} Index Single-variable query: merge bitmaps for relevant values via OR Multi-variable query: merge single-variable bitmaps via AND, OR etc /30/2015 7

13 BACKGROUND: THE BITMAP INDEX Bit i in bitmap j is set iff the ith record has value j A = {X, Y, Z} Index Single-variable query: merge bitmaps for relevant values via OR Multi-variable query: merge single-variable bitmaps via AND, OR etc. Distinct Bitmaps Evaluated values constraints Query: tttt 22, OR /30/2015 7

BACKGROUND: THE BITMAP INDEX Bit i in bitmap j is set iff the ith record has value j A = {X, Y, Z} Index Single-variable query: merge bitmaps for relevant values via OR Multi-variable query: merge

14 BACKGROUND: THE BITMAP INDEX Bit i in bitmap j is set iff the ith record has value j A = {X, Y, Z} Index Single-variable query: merge bitmaps for relevant values via OR Multi-variable query: merge single-variable bitmaps via AND, OR etc. Distinct Bitmaps Evaluated values constraints Query: tttt 22, 22 AND pppp 999, /30/ OR OR AND Evaluated query

15 BACKGROUND: ALACRITY INVERTED INDEX RID i in RID list j iff the ith record has value j RID Value Sig-bits Binning RID B1 B2 B Index Unconventional use of document-search index on scientific data Single-variable query: take union of RID lists for relevant values Multi-variable query: union, intersection, etc. of single-variable RID lists Bin B1 1; 3 B2 2 B3 4 Inverted List 6/30/2015 8

16 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) List (Merge, Intersection) 6/30/2015 9

17 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) List (Merge, Intersection) Quantization 6/30/2015 9

18 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) List (Merge, Intersection) Quantization RID set 6/30/2015 9

19 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) List (Merge, Intersection) Quantization RID set Set operations 6/30/2015 9

20 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) Encoding List (Merge, Intersection) Quantization RID set Set operations 6/30/2015 9

21 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) Encoding List (Merge, Intersection) Quantization RID set Set operations A = {W, X, Y, Z} Equality Encoding (= W) (= X) (= Y) (= Z) Range Encoding ( W) ( X) ( Y) 6/30/2015 9

22 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) Encoding Equality/Range/Interval List (Merge, Intersection) Quantization RID set Set operations A = {W, X, Y, Z} Equality Encoding (= W) (= X) (= Y) (= Z) Range Encoding ( W) ( X) ( Y) 6/30/2015 9

23 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) Encoding Equality/Range/Interval List (Merge, Intersection) Equality Quantization RID set Set operations A = {W, X, Y, Z} Equality Encoding (= W) (= X) (= Y) (= Z) Range Encoding ( W) ( X) ( Y) 6/30/2015 9

Equality/Range/Interval List (Merge, Intersection) Equality Quantization RID set Set operations Encoding

24 BEGINNING OF A FORMAL INDEX MODEL Concept Bitmap ALACRITY Binning Fixed, Precision etc. Significant-bits Data Structure Bit vector Inverted list Query Evaluation Bitwise (AND, OR, NOT) Encoding Equality/Range/Interval List (Merge, Intersection) Equality Quantization RID set Set operations Encoding A = {W, X, Y, Z} Equality Encoding (= W) (= X) (= Y) (= Z) Range Encoding ( W) ( X) ( Y) 6/30/2015 9

25 FORMALIZING AN INDEXING MODEL 1. A Quantization is applied to data (values bins ) 2. Bins are composed via Encoding (bin RSets encoded RSets ) 3. An RSet Representation is the data structure used to store RSets 6/30/

26 INDEX MODEL COMPONENT REQUIREMENTS RSet Representation: Supports set operations (union, intersection, complement) Serializable (for disk storage) Invertible Encoding: how to compose bin RSets into encoded Rsets (encode) recover any bin RSet based on encoded RSets (decode) 6/30/

27 OVERVIEW Motivation Formal Indexing Model Parallel Indexing and Query Unified Engine (PIQUE) Hyperdyadic Tree Index (HD-tree) Results 6/30/

28 PIQUE IMPL.: INDEXING ALGORITHM Bitmap E 1 = E 2 = ALACRITY E 1 = 2,7,12 E 2 = 0,2,5,7,10,11,12,14,15 6/30/

29 PIQUE IMPL.: UNIFIED QUERY ALGORITHM Bitmap Result: Convert to RIDs: (0,5,10,11,14,15) ALACRITY Result: (0,5,10,11,14,15) Step 4: convert result RSet to RIDs Step 3: perform set operations on encoded RSets as dictated by encoding Compute B 2 = E 2 E 1 = Want B 2 Want bins CD, EF Query 1: [C, F] Compute B 2 = E 2 E 1 = (0,2,5,7,10,11,12,14,15) (2,7,12) Step 2: invert encoding to decide which encoded RSets to read Step 1: apply quantization to compute bin range 6/30/

30 IMPACT: UNIFIED INDEXING/QUERY (PIQUE) Unified, modular indexing/query framework Enables new combinations of Quantization, RSet Representation, and Encoding techniques Allows direct comparison of different indexing methods Shared baseline minimizes confounding factors 6/30/

31 OVERVIEW Motivation Formal Indexing Model Parallel Indexing and Query Unified Engine (PIQUE) Hyperdyadic Tree Index (HD-tree) Results 6/30/

32 BACKGROUND: QUADTREES/OCTREES Quadtree: a tree where every internal node has exactly 4 children Region Quadtree: recursive quadrant decomposition of space Internal nodes are gray; further resolved by children Leaves are black (filled)/white (empty). Octree: 3D analogue; 8-child internal nodes; octant-wise Storage A pointer-based tree format is hard to serialize compactly Constant-bit linear quadtree (CBLQ) 05/05/2015 David A. Boyuka II 17

THE k-hyperdyadic TREE (khd-tree) A tree with the same structure as a kd quadtree However, the interpretation differs from a kd region quadtree Recursively partitions finite, discrete, 1D space into

33 THE k-hyperdyadic TREE (khd-tree) A tree with the same structure as a kd quadtree However, the interpretation differs from a kd region quadtree Recursively partitions finite, discrete, 1D space into dyadic intervals Stored (serialize) using constant-bit linear quadtree (CBLQ) coding Adapted from image compression literature, generalized to kd A 1D discrete space, its conceptual 2HD-tree, and the CBLQ-coded 2HD-tree 6/30/

34 THE HD-TREE AS AN RSET REPRESENTATION Supports set operations? Yes, via CBLQ algorithms. Serializable? Yes, by virtue of CBLQ coding. Yet, using CBLQ coding/algorithms as-is gives poor performance CBLQ made for image compression, tuned for different operations We develop several improvements for this new usage context 1. Optimized bulk construction 2. Optimized bulk set operations 3. Better compression 4. Fast conversion to RIDs 6/30/

35 HD-TREE ALGORITHMS: BULK CONSTRUCTION Original CBLQ construction algorithm: convert bitmap to CBLQ Inefficient for our use: O n for each bin, O nn overall Instead: use bulk insertion of runs of RIDs into bin HD-trees Maintain next word for each level Bulk insert fills lowest word, recursively bubbles up on overflow Analysis: Each push is O log n, so O n lll n time overall Amortized O c log n for c pushes of n total RIDs c Pushes evenly-distrib. across bins: O b n log n b Note: b n n/b = O n lll b 6/30/

36 HD-TREE ALGORITHMS: BULK SET OPERATIONS Original algorithm: binary, combine two CBLQs by pairing words Binary expression tree to combine k > 2 CBLQs Large k inefficient: scans 2k 2 CBLQs, get progressively larger Cache-inefficient (jumps between left/right operands, whole CBLQs) Instead: develop single-pass k-ary set operations algorithm Analysis: Each operand examined exactly once; no intermediate HD-trees Each level of each HD-tree is scanned sequentially (cache efficient) 6/30/

37 HD-TREE ALGORITHMS: BETTER COMPRESSION The original CBLQ coding uses 2 bits per code (white, black, gray) However, the bottom level of words (i.e., lowest level of internal nodes) can have only white or black children, not gray Therefore: use 1 bit per code for the last level of words Bonus: doesn t qualitatively affect set operation algorithms All access the same tree level across operands at a given time 8 bytes 6 bytes 6/30/

38 OVERVIEW Motivation Formal Indexing Model Parallel Indexing and Query Unified Engine (PIQUE) Hyperdyadic Tree Index (HD-tree) Results 6/30/

39 HD-TREES AND PIQUE: INDEXING RESULTS Can build both HD-tree and WAH bitmap indexes using PIQUE Notice higher compression: x for 3HD, x for 4HD 6/30/

40 HD-TREES AND PIQUE: QUERY RESULTS Can also use the single PIQUE query algorithm over both indexes Also, compare FastQuery parallel bitmap index implementation In these tests, HD-trees achieve equal or better performance Query: eeeeee > x for x on the x-axis 6/30/

41 THANK YOU! 6/30/

j Single-variable query: take union of RID lists for relevant values

42 BACKGROUND: ALACRITY INVERTED INDEX Unconventional use of document-search index on scientific data RID i in RID list j iff the ith record has value j Single-variable query: take union of RID lists for relevant values Multi-variable query: intersection, etc. of single-variable RID lists 6/30/

BACKGROUND: ALACRITY INVERTED INDEX Unconventional use of document-search index on scientific data RID i in RID list j iff the ith record has value j

43 BACKGROUND: ALACRITY INVERTED INDEX Unconventional use of document-search index on scientific data RID i in RID list j iff the ith record has value j Single-variable query: take union of RID lists for relevant values Multi-variable query: intersection, etc. of single-variable RID lists 6/30/

44 PIQUE IMPL.: UNIFIED QUERY ALGORITHM Index Encoding RSet Representation Convert B 2 to RIDs (0,5,10,11,14,15) Compute B 2 = E 2 E 1 Want B 2 Convert B 3 to RIDs (1,3,4,6,8,9,13) Compute B 3 = -E 2 Want B 3 Step 4: convert result RSet to RIDs Step 3: perform set operations on encoded RSets as dictated by encoding Step 2: invert encoding to decide which encoded RSets to read Quantization Want bins CD, EF Query 1: [C, F] Want bin GJ Query 2: [G,J] Step 1: apply quantization to compute bin range 6/30/

45 MOTIVATION BITMAP AND INVERTED INDEXES Compressed bitmap indexes commonly used for scientific data Efficiently index many attributes and perform multi-variate queries Compression typically yields smaller index than B+-tree Recent ALACRITY inverted index developed for scientific data With compression, often smaller than compressed bitmap index However, multivariate queries are difficult to evaluate efficiently These indexes bear similarities despite differing data structures Can we formalize this similarity? Do other, related indexes exist? 6/30/ Bitmaps 2, 7 0, 3, 5 1, 4, 6 Inverted Lists???? Related? 30

46 BACKGROUND: 3 BITMAP INDEX EXTENSIONS Compression: compress bitmaps to reduce storage and I/O Ex: WAH, which allows bitwise operations on compressed form Binning (or quantization): maps values down to a smaller set Ex: round floating-point values to the nearest integer Helps cope with high-cardinality data, limits the number of bitmaps Trade-off: error in index results, or necessitates candidate checks Encoding: alternative to direct value-to-bitmap mapping Range encoding: ith bit in jth bitmap set iff ith record has value j Can reduce query time, query I/O, or index size (but not all) 6/30/

47 BEGINNING OF A FORMAL INDEX MODEL Similarities between bitmap and inverted indexes: A data structure (bitmap, RID list) exists for each distinct value #This data structure may be compressed (WAH, PFOR-Delta) Queries are answered via merging/intersecting the data structures Binning is possible, with similar effect #Difference: bitmap vs. inverted list as underlying data structure Response: abstract this piece as the RID set data type #Difference: bitwise operations vs. list union/intersection used Response: abstract this piece as set operations #Difference: encoding is not applicable to inverted indexes Response: consider inverted indexes to be equality encoded 6/30/

48 INDEX MODEL COMPONENT REQUIREMENTS Quantization: Function to map original values to smaller set of quantized keys Has preimage: can compute original value range for quantized key RSet Representation: Data structure implementing the RSet abstract data type Supports set operations (union, intersection, complement) Serializable (for disk storage) Encoding: Algorithm dictating how to compose bin RSets into encoded RSets Invertible : recover any bin RSet based on encoded RSets 6/30/

49 IMPACT: UNIFIED INDEXING/QUERY (PIQUE) This model enables a unified, modular indexing/query framework Core, fixed algorithms for index building and query processing Allows conceptually-minimal plugins defining new quantization, RSet representations, and index encodings Enables new combinations of Quantization, RSet Representation, and Encoding techniques Allows direct comparison of different indexing methods Shared baseline minimizes confounding factors 6/30/

50 PIQUE IMPL.: UNIFIED INDEXING ALGORITHM Input: data, and algorithms for the 3 method plugins Step 1: scan the data (one pass), quantize, identify runs, push runs into RSet representations Quantize each value Identify runs of equal quantized keys Push the run of RIDs (first push empty to advance to current position, then push filled of appropriate length) Step 2: top off, collect, sort all RSets by q.key Step 3: compose bin RSets into encoded RSets 6/30/

51 HD-TREE ALGORITHMS: BULK SET OPERATIONS = Level 1 Level 2 actions output actions output mappings[0] operands[0] UUUU UUUU UUUU UUUU X -1, 1, , 0110, 1100 mappings[1] operands[1] UUUU UUUU 0000 UDDU DDUU , 1, , 1100, 1001 mappings[2] operands[2] DUUU DUDU 1010 DDDU DDUD DUUU 1222 DDDD 1111 DDDU 1110 DDUD /30/

52 HD-TREE ALGORITHMS: CONVERSION TO RIDS Original algorithm: does not exist Obvious approach: scan HD-tree codes in (breadth-first) order Disadvantages: emits RIDs out of order, and (surprisingly) is slower Second approach: scan HD-tree codes sideways (depth-first) Obtain a pointer to the beginning of each tree level s codes Recursively examine codes, descend to next level for each 2 (gray) Advantages: emits RIDs in order, ends up being substantially faster Analysis: Compiler-generated deep for loop to avoid actual recursion Most state can be optimized to hardware registers 6/30/

53 HD-TREE ALGORITHMS: SUMMARY 1. Bulk Construction 4. Better Compression 2. Bulk Set Operations 3. Conversion to RIDs AB CD, EF GJ Encoded RSets Bins Quantized Keys Original Data E 2 E 1 = B 1 = B 1 + B 2 2,7, 0,5,10, B 1 B 2 B ,14,15 1,3,4,6, 8,9,13 AB CD EF GJ C G B K D K J A J G D D B H D C RSet Representation Index Encoding Quantization Index Encoding RSet Representation Quantization Convert R 2 to RIDs (0,5,10,11,14,15) Compute R 2 = E 2 E 1 Want R 2 Want bins CD, EF Query 1: [C, F] Convert R 3 to RIDs (1,3,4,6,8,9,13) Compute R 3 = -E 2 Want R 3 Want bin GJ Query 2: [G,J] 6/30/

54 PROPOSED WORK: HD-TREE ANALYSIS Previous work: theoretical analysis of WAH bitmap compression Used multiple probabilistic models to describe nature of data Models: uniform distribution, Markovian process Solve for expected index size under these models HD-tree compression should have different properties from WAH Compression relative to WAH shown to vary (though always > 1x) Theoretical model would give deeper insight into HD-tree behavior Proposal: solve data models for HD-trees, compare to WAH Though the probabilistic models are given, the solutions for HDtrees should be markedly different from WAH Also, small proposed modification to the Markov model (in text) 6/30/

55 POSSIBLE DIRECTIONS FOR FUTURE WORK Try other quadtree codings (LQT, FBLQ, Huffman-coded CBLQ) All fit nicely into PIQUE as new RSet representations Lossy RSet representations? May need to augment the model Better compression works on HD-tree top levels, too; harder Requires decompression, unlike the way used here. Worth it? Generalize set intersection query method from Zou et al Could be faster in some cases. When? Can this allow multi-variate queries over mixed index types? Experiment with encodings on HD-trees Encodings have never been applied to anything besides bitmaps 6/30/

Chapter 12: Indexing and Hashing. Basic Concepts

Chapter 12: Indexing and Hashing! Basic Concepts! Ordered Indices! B+-Tree Index Files! B-Tree Index Files! Static Hashing! Dynamic Hashing! Comparison of Ordered Indexing and Hashing! Index Definition