FASTER UPPER BOUNDING OF INTERSECTION SIZES

Transcription

1 FASTER UPPER BOUNDING OF INTERSECTION SIZES IBM Research Tokyo - Japan Daisuke Takuma Hiroki Yanagisawa 213 IBM

2 Outline Motivation of upper bounding Contributions Data structure Proposal Evaluation (introduce related work here) Application Evaluation Conclusion

3 Set intersection Computation of the common elements in given sets: A B Applications: Multi-word search JOIN operation : Intersection: A B = {2, 5, 7} Intersection size: A B = 3 Recently the intersection operation is used in a different way.

4 Some apps compute intersections only for their sizes. Top-k term query Input: the hit docs for a search query Output: the k most frequent terms in the hit docs Document sets Top-k terms (k=3) DRIVING DOOR MILES Hit docs for a search query HIGHWAY Ex. ENGINE GEAR MILES DRIVING HIGHWAY Determined by the intersection sizes

5 Intersection sizes are evaluated in inequalities: A B > threshold, but mostly A B threshold. In 8% intersections in top-k term queries: A B <.5 threshold A B threshold Upper bound of A B If this holds, we can skip the computation of the exact intersection

6 Our contributions: We proposed data structures Cardinality Filters (CF) to quickly compute upper bounds of intersection sizes. We accelerated the top-k term query using CFs approximately by factor of 2. CFs are applicable to all the algorithm evaluating A B threshold by adding the condition for the upper bound. upperbound( A B ) threshold AND A B threshold It does not change the logic.

7 Cardinality Filters

8 Requirements for CFs Sets A, B 2 X X A B A B (set intersection) Set size function: CF conversion Φ A B Φ(A) Φ(B) CF size function: Φ(A) Φ(B) Φ(A) Φ(B) (CF intersection) Cardinality filters (CFs) Φ(A), Φ(B) Φ(2 X )

9 SCF: Single Cardinality Filter Φ(A) = (h(a), c(a)) where h(a) hash values, c(a): non-smallest values in collisions Φ(A) A h(a) c(a): 1 14 Φ(A) Φ(B) = Φ(B) h(b) h(a) h(b) =3 + c(a) c(b) = c(b): Bit-wise AND B Small

10 Formal definition of SCF Let X be a universe set (A, B X), and H be a hash value space H={, 1,, [ X /N] - 1}, we define SCF Φ: 2 X (2 H, 2 X ) by Φ(A) = (h(a), c(a)) (A X) where h(a) = { h(x) x A } c(a) = { y A ( x A) (h(x) = h(y), x < y) } Directly used in the proof. The proof has no conditional branches. Thm: A B Φ(A) Φ(B) := h(a) h(b) + c(a) c(b)

11 RCF: Recursive Cardinality Filter The RCF recursively applies the SCF to c(a). A L = 3 H 1 (A), C 1 (A) h 1 (A) c 1 (A) H 2 (A), C 2 (A) h 2 (c 1 (A)) c 2 (c 1 (A)) H 3 (A), C 3 (A) h 3 (c 2 (c 1 (A))) c 3 (c 2 (c 1 (A))) Φ(A) Φ(B) =Σ 1~L H i (A) H i (B) + C L (A) C L (B)

12 We compared CFs with the following algorithms: Exact intersections Linear merge: comparison between sorted lists. Binary search: binary search x A in B ( A < B ). DK algorithm (VLDB 211): Intersect h(a) and h(b) by bit-wise AND (h:hash) For all x h (A) h(b), intersect h -1 (x) A and h -1 (x) B. Naïve upper bounding Candidates Bloom filter: Test x A in the Bloom filter of B: h(b).

13 Performance studies: Size Time [microsec] (A) Large (1M x 1M) - no correlation LM Time Approximation ratio BS DK1H DK2H Asymmetry BF SCF Approx. ratio RCF (D) Asymmetric (1M x 1K) - no correlation LM BS DK1H DK2H BF SCF RCF (B) Middle (1K x 1K) - no correation LM BS DK1H DK2H BF SCF RCF (E) Middle (1K x 1K) - positive correlation LM BS DK1H DK2H BF SCF RCF (C) Small (1K x 1K) - no correlation LM BS DK1H DK2H BF SCF RCF (F) Middle (1K x 1K) - negative correlation LM Correlation BS DK1H DK2H BF SCF RCF traditional DK naïve ours

14 Time complexity and performance break down DK: O( ( A + B )/ w + A B ) RCF: O( ( A + B )/w ) (With a practical setting of h) A B is the barrier for all the exact algorithms. The benefit of considering the upper bounds. Algorithms DK1H DK2H SCF RCF Bit-wise AND Other computations 5 1 Time [microsec] CFs maximize the utilization of bit-wise AND.

15 Improvement of the top-k term query

16 The intersection sizes of CFs are evaluated before evaluation of the exact intersection sizes. S: the doc IDs for the search query Term CFs P[i] The doc IDs for term i (i=, 1, ) h1 Query CF (Candidate CFs) h1 h2 h5 h1 h2 h2 h2 h5 h5 Index The compression ratio for the n-th term: Safe factor P[k-1] / P[n] Intersect the CFs using the same hash functions.

17 Improvement of top-k term queries LM LM+BS DK1H SCF RCF Computation time [millisec] The number of hit documents Ours

18 More than 8% of the exact intersections were skipped. 1 SCF RCF Skip ratio The number of hit documents

19 Summary We proposed Cardinality Filters (SCF, RCF) to quickly compute upper bounds of intersection sizes. Property: A B Φ(A) Φ(B) Performance improvement from the exact algorithms. DK: O( ( A + B )/ w + A B ) RCF: O( ( A + B )/w ) We accelerated the top-k term query using the CFs by factor of 2.

20 Future work Using other algebraic properties of SCF and RCF: Monotonicity: A B Φ(A) Φ(B) Commutative and associative laws of Φ(A) Φ(B) Implementation of CFs having Union-like operation Non-parametric CFs (compression ratio of h)

21 Faster Upper Bounding of Intersection Sizes Thank you!!