Generalizing the Optimality of Multi-Step k-nearest Neighbor Query Processing

Transcription

1 Generalizing the Optimality of Multi-Step k-nearest Neighbor Query Processing SSTD 2007 Boston, U.S.A. Hans-Peter Kriegel, Peer Kröger, Peter Kunath, Matthias Renz Institute for Computer Science University of Munich, Germany

2 Outline 1. Introduction 2. Multi-Step knn Query Processing R Ilu 3. -optimal Multi-Step knn Search 4. Experimental Evaluation 5. Conclusions and Future Work 2

3 Introduction CAD/GIS Graphs Audio/Video Time Series Similarity search on complex objects involves costly distance measures lower-bound (LB) and upper-bound (UB) filter distances 3

4 Lower-Bound & Upper-Bound Filter Distances Lower-Bound & Upper-Bound Filter Distances LB/UB based on reference objects (e.g., M-tree) d min (A,B) = d A -d B d min (A,B) R d max (A,B) d max (A,B) = d A +d B A B LB/UB based on regions (e.g., R-tree) B d min (A,B) d(a,b) d max (A,B) LB exact UB d min (A,B) A d max (A,B) true drops true hits LB/UB based on some application d min (A,B) time series A d max (A,B) time series B conservative approximation of the amplitude values 4

5 Outline 1. Introduction 2. Multi-Step knn Query Processing R Ilu 3. -optimal Multi-Step knn Search 4. Experimental Evaluation 5. Conclusions and Future Work 5

6 Problem Definition Problem Definition Definition: knn-query, knn-distance Let DB be a database, q be a query object and k IN a query parameter. A knn query in DB returns the smallest set NN DB (q, k) DB that contains at least k objects from DB, for which the following condition holds: o NN DB (q, k), o DB \ NN DB (q, k): dist(q, o) < dist(q, o ) q 6

7 Multi-Step Query Processing Multi-Step Query Processing Indexing methods (i.e., single-step solutions) suffer from two drawbacks: the distance function has to be a metric the query predicate is evaluated many times apply a multi-step approach to reduce the number of candidates in a filter step query database multi-step query processor drops candidates hits filter step refinement result 7

8 Multi-Step knn-algorithm based on LB Multi-Step knn-algorithm based on LB The multi-step knn algorithm proposed by [Seidl, Kriegel 98] uses a lower bounding (LB) distance estimation in the filter step: for any query object q and o DB: LB(q, o) dist(q, o) knn-optimal(db, q, k) Ranking = initialize ranking for q; // using the LB filter distance result = ; d max = + ; // stop distance WHILE c = Ranking.getNext() AND LB(q, c) d max DO IF dist(q, c) d max THEN result.insert(dist(q, c), c); IF result.length k THEN d max = result[k].key; remove all entries from result where key > d max ; // true drops END WHILE; RETURN result; The optimality w.r.t. the number of refined candidates is evaluated and formalized by the concept of R-optimality 8

9 Outline 1. Introduction 2. Multi-Step knn Query Processing R Ilu 3. -optimal Multi-Step knn Search 4. Experimental Evaluation 5. Conclusions and Future Work 9

10 Advantages of an Upper-Bound Filter Advantages of an Upper-Bound Filter LB-based knn-search vs. (LB+UB)-based knn-search o 1 o 2 o 3 o 4 o 5 o 6 o 7 o 8 o 9 o 10 o 11 o 12 LB(q,o 1 ) dist(q,o 1 ) o 1 o 2 o 3 o 4 o 5 o 6 o 7 o 8 o 9 o 10 o 11 o 12 LB(q,o 1 ) dist(q,o 1 ) UB(q,o 1 ) distance nn 8 -dist(q) distance nn 8 -dist(q) Using an additional upper-bound filter distance has several advantages: allows to identify true hits without refinement reduces the storage requirements of the priority list of the ranking true hits can be reported immediately to the user 10

11 Generalizing the Optimality Generalizing the Optimality Definition: Generalized Optimiality Given an information class I defining a set of distance approximations available in the filter step, a multi-step knn algorithm is called R I -optimal if it does not produce more candidates in the filter-step than necessary. Before Refinement After Refinement o 1 o 2 o 3 o 4 o 5 o 6 o 7 o 8 o 9 o 10 o 11 o 12 dist(q,o 1 ) LB(q,o 1 ) UB(q,o 1 ) has to be refined o 1 o 2 o 3 o 4 o 5 o 6 o 7 o 8 o 9 o 10 o 11 o 12 dist(q,o 1 ) LB(q,o 1 ) UB(q,o 1 ) refinement not necessary LB k UB k nn 8 -dist(q) distance LB k UB k nn 8 -dist(q) distance Idea: refine only those objects which cover the knn-distance nn k -dist(q) Theorem: there exists always at least one candidate c, where LB k LB(c) and UB k UB(c), i.e. which covers nn k -dist(q) 11

12 R I lu -Optimiality R I lu -Optimiality Theorem: R -Optimiality I lu A multi-step knn algorithm is R I -optimal, iff it refines the candidate set: lu (Case 1) {o DB LB(q, o) nn k -dist(q) UB(q, o)} if there are more than k candidates c DB with LB(q, c) nn k -dist(q), otherwise (Case 2) {o DB LB(q, o) nn k -dist(q) < UB(q, o)} case 1: case 2: case 3: case 4: case 5: LB UB correctness correctness R-optimality case 1: case 2: case 3: case 4: case 5: case 6: case 6: LB UB R-optimality correctness correctness ε < nn k -dist(q) nn k -dist(q) < ε if ε nn k -dist(q), we lose correctness and R I lu -optimiality 12

13 Multi-Step knn-algorithm based on LB & UB Multi-Step knn-algorithm based on LB & UB knn-optimal(db, q, k) Ranking = initialize ranking for q; [Hjaltason, Samet 95] // using the LB filter distance result = ; UB k = + ; LB k = 0; Fetch first k candidates from the ranking; REPEAT // Step 1: fetch next candidate if LB next LB k then c = Ranking.getNext(); insert c into candidates; // only if LB(q,c) UB k endif; update LB k, UB k, LB next ; // Step 2: identify true hits and true drops (Filter Step) for all c candidates do if UB(q,c) LB k then insert c into result; // true hit if LB(q,c) > UB k then remove c from candidates; // true drop end for; // Step 3: refine candidates (Refinement Step) if result + candidates k and LB next > UB k then insert all c candidates into result; // stop condition else refine all c candidates where LB(q,c) LB k UB k UB(q,c), i.e. compute d exakt (q,c) and update LB(q,c) = UB(q,c) = d exakt (q,c); end if; UNTIL( candidates =0 and LB next > UB k ); RETURN result; 13

14 Outline 1. Introduction 2. Multi-Step knn Query Processing R Ilu 3. -optimal Multi-Step knn Search 4. Experimental Evaluation 5. Conclusions and Future Work 14

15 Experimental Evaluation Experimental Evaluation Data Sets used in the Evaluation Road Network Protein Graph CAD data Audio Timeseries Data Set Size 18,263 nodes 1,128 proteins 35,950 voxels 2,400 clips Distance Dijkstra Graph Kernel Euclidean DTW Filter vs. Refinement 1/300 1/ /150 15

16 Relative Number of Unrefined Candidates Relative Number of Unrefined Candidates Road Network Protein Plane Timeseries a significant amount of hits does not need to be refined 16

17 Absolute Number of Refinements Absolute Number of Refinements Road Network Protein Plane Timeseries for high values of k, the number of refinements is reduced by 50% 17

18 Pruning of the Priority Queue of the Ranking Pruning of the Priority Queue of the Ranking Road Network Plane the space requirements of the priority queue can be reduced by more than 50% 18

19 Number of Reported Results vs. Query Iterations Number of Reported Results vs. Query Iterations Plane (k = 1000) Protein (k = 1000) Road Network (k = 250) Road Network (k = 1000) a significant portion of the result can be reported early 19

20 Outline 1. Introduction 2. Multi-Step knn Query Processing R Ilu 3. -optimal Multi-Step knn Search 4. Experimental Evaluation 5. Conclusions and Future Work 20

21 Conclusions and Future Work Conclusions and Future Work Our Approach: Generalization of the traditional notion of R-optimality Multi-step approach which uses both a lower-bound and an upperbound filter distance function Our new approach helps to drastically reduce the refinements, features considerably reduced storage requirements and allows to report first hits very early Future Work: Integration in Data Mining techniques 21

22 ????? 22