Datalog Evaluation. Linh Anh Nguyen. Institute of Informatics University of Warsaw

Transcription

1 Datalog Evaluation Linh Anh Nguyen Institute of Informatics University of Warsaw

2 Outline Simple Evaluation Methods Query-Subquery Recursive Magic-Set Technique Query-Subquery Nets [2/64] Linh Anh Nguyen Datalog Evaluation

3 Simple Evaluation Algorithms Methods to evaluate Datalog program P on database instance I, derived from the different equivalent definitions of the semantics: Model-theoretic definition: Enumerate all subsets J B(P, I) and check modelhood pick smallest such J. Fixpoint definition: Augment I using operator T P until a fixpoint is reached. Proof-theoretic definition: Use SLD-resolution (bottom-up or top-down) [3/64] Linh Anh Nguyen Datalog Evaluation

4 Classes of Datalog Evaluation Two major classes of evaluation approaches: Bottom-Up, Forward Chaining: Proceed in the proof tree from the leaves to the root Apply the datalog rules from body to head (forward) Top-Down, Backward Chaining: Proceed in the proof tree from the root to the leaves Apply the datalog rules from head to body (backward) [4/64] Linh Anh Nguyen Datalog Evaluation

5 Naive Evaluation Follow the bottom-up approach Compute the minimum fixpoint of T P containing I (= T ω P (I)) Given datalog program P, database instance I 1 Start by assuming all idb relations are empty. 2 Repeatedly evaluate the rules using the edb and the previous idb, to get a new idb. 3 End when no change to idb. Disadvantages Relations have to be computed always from scratch, Relations must be copied, Iteration on all relations, even if they do not recursively depend on each other. [5/64] Linh Anh Nguyen Datalog Evaluation

6 Semi-naive Evaluation Since the edb never changes, on each round we only get new idb tuples if we use at least one idb tuple that was obtained on the previous round. Saves work, lets us avoid rediscovering most known facts. [6/64] Linh Anh Nguyen Datalog Evaluation

7 Adornment An adornment for an m-ary predicate p is a string α of length m made up of b (bound) and f (free), let p α be the predicate p adorned by α. The general algorithm for adorning a rule (i) All occurrences of each bound variable in the rule head are bound; (ii) All occurrences of constants are bound; (iii) If a variable X occurs in the rule body, then all occurrences of X in subsequent literals are bound; (iv) The remaining occurrences of variables are free. A different ordering of the rule body would yield different adornments. [7/64] Linh Anh Nguyen Datalog Evaluation

8 Adornment - Example Consider the following positive logic program P: r 1 : ancestor(x, y) parent(x, y) r 2 : ancestor(x, y) parent(x, z), ancestor(z, y) where x, y, z are variables parent is an extensional predicate, ancestor is an intensional predicate, parent(x, y) means x is a parent of y, ancestor(x, y) means x is an ancestor of y. Let the query be ancestor(john, y)?, asking John is an ancestor of whom?. The task is to find all the descendants of John. [8/64] Linh Anh Nguyen Datalog Evaluation

9 Adornment - Example Consider the following positive logic program P: r 1 : ancestor(x, y) parent(x, y) r 2 : ancestor(x, y) parent(x, z), ancestor(z, y) The adorned version (denoted by P ad ) of the program P and the query ancestor(john, y)?: r 1 : ancestor bf (x, y) parent(x, y) r 2 : ancestor bf (x, y) parent(x, z), ancestor bf (z, y) query f (y) ancestor bf (John, y) [9/64] Linh Anh Nguyen Datalog Evaluation

10 Outline Simple Evaluation Methods Query-Subquery Recursive Magic-Set Technique Query-Subquery Nets [10/64] Linh Anh Nguyen Datalog Evaluation

11 Query-Sub-Query Recursive (QSQR) Top-down, direct evaluation; Avoid the calculation of tuples that are not used for deriving answer; Begin with constants in a query pushing them from goals to subgoals; Use sideways information passing to pass constant binding information from one atom to the next in subgoals. [11/64] Linh Anh Nguyen Datalog Evaluation

12 QSQR - Example Reconsider the following adorned program P ad : r 1 : ancestor bf (x, y) parent(x, y) r 2 : ancestor bf (x, y) parent(x, z), ancestor bf (z, y) query f (y) ancestor bf (John, y) [12/64] Linh Anh Nguyen Datalog Evaluation

13 QSQR - Example [13/64] Linh Anh Nguyen Datalog Evaluation

14 QSQR - Example [14/64] Linh Anh Nguyen Datalog Evaluation

15 QSQR - Example [15/64] Linh Anh Nguyen Datalog Evaluation

16 QSQR - Example [16/64] Linh Anh Nguyen Datalog Evaluation

17 QSQR - Example [17/64] Linh Anh Nguyen Datalog Evaluation

18 QSQR - Example [18/64] Linh Anh Nguyen Datalog Evaluation

19 QSQR - Example [19/64] Linh Anh Nguyen Datalog Evaluation

20 QSQR - Example [20/64] Linh Anh Nguyen Datalog Evaluation

21 QSQR - Example [21/64] Linh Anh Nguyen Datalog Evaluation

22 QSQR - Example [22/64] Linh Anh Nguyen Datalog Evaluation

23 QSQR - Example [23/64] Linh Anh Nguyen Datalog Evaluation

24 QSQR - Example [24/64] Linh Anh Nguyen Datalog Evaluation

25 QSQR - Example [25/64] Linh Anh Nguyen Datalog Evaluation

26 QSQR - Example [26/64] Linh Anh Nguyen Datalog Evaluation

27 QSQR - Example [27/64] Linh Anh Nguyen Datalog Evaluation

28 QSQR - Example [28/64] Linh Anh Nguyen Datalog Evaluation

29 QSQR - Example [29/64] Linh Anh Nguyen Datalog Evaluation

30 QSQR - Example [30/64] Linh Anh Nguyen Datalog Evaluation

31 QSQR - Example [31/64] Linh Anh Nguyen Datalog Evaluation

32 QSQR - Example [32/64] Linh Anh Nguyen Datalog Evaluation

33 QSQR - Example [33/64] Linh Anh Nguyen Datalog Evaluation

34 QSQR - Example [34/64] Linh Anh Nguyen Datalog Evaluation

35 Outline Simple Evaluation Methods Query-Subquery Recursive Magic-Set Technique Query-Subquery Nets [35/64] Linh Anh Nguyen Datalog Evaluation

36 Magic-Set technique The Magic-Set technique is a rule-rewriting method that generates from a given set of rules a new set of rules, which is equivalent to the original set w.r.t. the original query. After rewriting, the new program can be evaluated by a simple bottom-up algorithm, usually the (improved) semi-naive evaluation method. This method takes advantages of reducing irrelevant facts and restricting the search space. It combines the pros of top-down and bottom-up methods. The Generalized Supplementary Magic Sets algorithm uses some special predicates called supplementary magic predicates in order to eliminate the duplicate work during the processing. [36/64] Linh Anh Nguyen Datalog Evaluation

37 Magic-Set technique - Example The Magic-Set rule-rewriting corresponding to the adorned program P ad (denoted by P mg ): magic ancestor bf (z) magic ancestor bf (x), parent(x, z) ancestor bf (x, y) magic ancestor bf (x), parent(x, y) ancestor bf (x, y) magic ancestor bf (x), parent(x, z), ancestor bf (z, y) magic ancestor bf (John). [37/64] Linh Anh Nguyen Datalog Evaluation

38 Magic-Set technique - Example Applying improved semi-naive evaluation method for P mg : iteration 1: magic ancestor bf (Ruth), magic ancestor bf (Lois) added. iteration 2: magic ancestor bf (Andy), magic ancestor bf (Mark) added. iteration 3: ancestor bf (John, Lois), ancestor bf (John, Ruth), ancestor bf (Lois, Andy), ancestor bf (Lois, Mark), ancestor bf (John, Andy), ancestor bf (John, Mark) added. iteration 4: fixpoint (no more tuples were added). [38/64] Linh Anh Nguyen Datalog Evaluation

39 Magic-Set technique - Example Generalized Supplementary Magic Sets (denoted by P gmg ): sup magic2 2(x, z) magic ancestor bf (x), parent(x, z) ancestor bf (x, y) magic ancestor bf (x), parent(x, y) ancestor bf (x, y) magic ancestor bf (z) sup magic2 2(x, z), ancestor bf (z, y) sup magic2 2 (x, z) magic ancestor bf (John). [39/64] Linh Anh Nguyen Datalog Evaluation

40 Magic-Set technique - Example Applying improved semi-naive evaluation method for P gmg : iteration 1: sup magic2 2(John, Ruth), sup magic2 2 (John, Lois), magic ancestor bf (Ruth), magic ancestor bf (Lois) added. iteration 2: sup magic2 2(Lois, Andy), sup magic2 2 (Lois, Mark), magic ancestor bf (Andy), magic ancestor bf (Mark) added. iteration 3: ancestor bf (John, Lois), ancestor bf (John, Ruth), ancestor bf (Lois, Andy), ancestor bf (Lois, Mark), ancestor bf (John, Andy), ancestor bf (John, Mark) added. iteration 4: fixpoint (no more tuples were added). [40/64] Linh Anh Nguyen Datalog Evaluation

41 Outline Simple Evaluation Methods Query-Subquery Recursive Magic-Set Technique Query-Subquery Nets [41/64] Linh Anh Nguyen Datalog Evaluation

42 Query-Subquery Nets Make a QSQ-net structure from a program P and use it as a flow control network to choose the processing order of transferring data, in an efficient way. The intention is to increase efficiency of query processing by: eliminating redundant computation, increasing flexibility, reducing the number of accesses to the secondary storage. The framework forms a generic evaluation method called QSQN. It has the following nice properties: the approach is goal-directed, each subquery is processed only once, each supplement tuple, if desired, is transferred only once, operations are done set-at-a-time, any control strategy can be used. [42/64] Linh Anh Nguyen Datalog Evaluation

43 Query-Subquery Nets Definition 1. QSQ-net Structure A QSQ-net structure of a positive logic program P is a tuple (V, E, T ) such that: V is a set of nodes, E is a set of edges, T is a function, called the memorizing type of the net structure. We call the pair (V, E) the QSQ-net topological structure of P. Definition 2. QSQ-net A QSQ-net of P is a tuple N = (V, E, T, C) such that: (V, E, T ) is a QSQ-net structure of P, C is a mapping that associates each node v V with a structure call the content of v. [43/64] Linh Anh Nguyen Datalog Evaluation

44 Query-Subquery Nets Definition 1. QSQ-net Structure A QSQ-net structure of a positive logic program P is a tuple (V, E, T ) such that: V is a set of nodes, E is a set of edges, T is a function, called the memorizing type of the net structure. We call the pair (V, E) the QSQ-net topological structure of P. Definition 2. QSQ-net A QSQ-net of P is a tuple N = (V, E, T, C) such that: (V, E, T ) is a QSQ-net structure of P, C is a mapping that associates each node v V with a structure call the content of v. [43/64] Linh Anh Nguyen Datalog Evaluation

45 Query-Subquery Nets A subquery is a pair of the form (t, δ), where t is a generalized tuple and δ is an idempotent substitution such that dom(δ) Vars(t) =. Subqueries are transferred through edges and processed at nodes. Formally, the processing of a subquery has following properties: every subquery / input tuple / answer tuple subsumed by another one is ignored; every subquery / input tuple / answer tuple with term-depth greater than a fixed bound L is ignored; the processing is divided into smaller steps which can be delayed to maximize flexibility and allow various control strategies; the processing is done set-at-a-time (e.g., for all the unprocessed subqueries accumulated in a given node). [44/64] Linh Anh Nguyen Datalog Evaluation

46 QSQN - Example Reconsider the program P: ancestor(x, y) parent(x, y) ancestor(x, y) parent(x, z), ancestor(z, y). The QSQ-net topological structure of P is constructed as follows: [45/64] Linh Anh Nguyen Datalog Evaluation

47 QSQN - Example [46/64] Linh Anh Nguyen Datalog Evaluation

48 QSQN - Example [47/64] Linh Anh Nguyen Datalog Evaluation

49 QSQN - Example [48/64] Linh Anh Nguyen Datalog Evaluation

50 QSQN - Example [49/64] Linh Anh Nguyen Datalog Evaluation

51 QSQN - Example [50/64] Linh Anh Nguyen Datalog Evaluation

52 QSQN - Example [51/64] Linh Anh Nguyen Datalog Evaluation

53 QSQN - Example [52/64] Linh Anh Nguyen Datalog Evaluation

54 QSQN - Example [53/64] Linh Anh Nguyen Datalog Evaluation

55 Query-Subquery Nets Control strategies are used: Disk Access Reduction (DAR), which tries to reduce the number of accesses to the secondary storage; Depth-First Search (DFS), which gives priority to the order of clauses in the positive logic program defining intensional predicates and thus allows the user to control the evaluation to a certain extent; Improved Depth-First Control Strategy (IDFS), which is an improved version of DFS and the aim is to accumulate as many as possible tuples or subqueries at each node of the QSQ-net before processing it. [54/64] Linh Anh Nguyen Datalog Evaluation

56 QSQN - Example Reconsider the program P: ancestor(x, y) parent(x, y) ancestor(x, y) parent(x, z), ancestor(z, y). The query: ancestor(john, y)? [55/64] Linh Anh Nguyen Datalog Evaluation

57 QSQN - Example [56/64] Linh Anh Nguyen Datalog Evaluation

58 QSQN - Example [57/64] Linh Anh Nguyen Datalog Evaluation

59 QSQN - Example [58/64] Linh Anh Nguyen Datalog Evaluation

60 QSQN - Example [59/64] Linh Anh Nguyen Datalog Evaluation

61 QSQN - Example [60/64] Linh Anh Nguyen Datalog Evaluation

62 QSQN - Example [61/64] Linh Anh Nguyen Datalog Evaluation

63 QSQN - Example [62/64] Linh Anh Nguyen Datalog Evaluation

64 QSQN - Example [63/64] Linh Anh Nguyen Datalog Evaluation

65 QSQN - Example At this point, some edges are active without affecting to the ans ancestor relation. When all the attributes unprocessed, unprocessed subqueries, unprocessed subqueries 2 and unprocessed tuples of the nodes in the net are empty sets, the algorithm terminates and returns the set tuples(ans ancestor) = {(John, Lois), (John, Ruth), (John, Mark), (John, Andy)}. for the query ancestor(john, y)? [64/64] Linh Anh Nguyen Datalog Evaluation