Memoing Evaluation for Constraint Extensions of Datalog *

Transcription

1 Constraints: An International Journal, 2, (1997) c 1997 Kluwer Acaemic Publishers, Boston Manufacture in The Netherlans Memoing Evaluation for Constraint Extensions of Datalog * Department of Computer Science, University of Toronto avi@cstorontoeu Eitor: Raghu Ramakrishnan an Peter Stuckey Abstract This paper proposes an efficient metho for evaluation of euctive queries over constraint atabases The metho is base on a combination of the top-own resolution with memoing an the close form bottom-up evaluation In this way the top-own evaluation is guarantee to terminate for all queries for which the bottom-up evaluation terminates The main avantage of the propose metho is the irect use of the information present in partially instantiate queries without the nee for rewriting of the original program The evaluation algorithm automatically propagates the necessary constraints uring the computation In aition, the top-own evaluation potentially allows the use of compilation techniques, evelope for compilers of logic programming languages, which can make the query evaluation very efficient Keywors: Datalog, constraint class, top-own evaluation, memoing evaluation of logic programs, SLG 1 Introuction We propose a new metho for evaluating euctive queries over constraint atabases (Kanellakis et al, 1995) The evaluation of queries over such atabases is ifferent from the one use in stanar atabase systems The constraints are use as the actual representation of ata store in the atabase rather than mere restrictions of the contents of otherwise groun relations Algorithms for query evaluation over constraint atabases have to satisfy the following criteria: 1 the evaluation algorithm has to terminate for all input queries, 2 the algorithm shoul be able to encompass various classes of constraints over wie range of omains, an 3 partially instantiate queries have to be evaluate efficiently The first requirement is especially ifficult to achieve in the case of constraint atabases: the extents of constraint relations are often infinite There are two main approaches to satisfy the above requirements in the case of Datalog However, neither of them seems to aress all three of the requirements The first approach is base on a fixpoint, bottom-up evaluation of the rules Here the first conition is usually met, eg, for Datalog (Ullman, 1989), Datalog with ense orer constraints (Kanellakis et al, 1995), Datalog with integer constraints (Revesz, 1993, Toman et al, 1994), an sets (Srivastava et al, 1994) However, the evaluation process is not goal-oriente an thus the evaluation of partially instantiate queries is fairly inefficient Application of stanar program transformation techniques, eg, the Magic Rewriting, oes not completely solve the problem (cf Section 4) * A preliminary report on this work appeare in Proc 1995 ILPS, Portlan, OR, (Toman, 1995)

2 338 The secon approach is base on a top-own, resolution-base metho Here the secon an thir conitions are usually met However, the termination guarantees are often sacrifice (Ullman, 1989) in orer to improve the expressiveness an efficiency; an exception is (Swift an Warren, 1994b) where no constraints are allowe On the other han these methos can take full avantage of compilation techniques evelope for other logic programming languages, eg, (Freire et al, 1996, Swift an Warren, 1994a, Warren, 1983) This greatly improves the practical efficiency of query evaluation in the case of Datalog (Swift an Warren, 1994b) We show that similar results can be achieve for constraint extensions of Datalog In this paper we try to combine the avantages of the above two approaches We propose an evaluation metho, Constraint Memoing, applicable to constraint-base extensions of Datalog (Datalog ), that has the following features: Integrate Constraint Representation Constraint Memoing integrates the constraints as first-class ata into the evaluation proceure This approach is ifferent from most CLP systems, where constraints are hanle by a separate constraint solver (Jaffar an Maher, 1994) We propose much tighter integration of constraints into the query evaluation: they are hanle very similarly to stanar groun tuples (or terms in the CLP systems) This is achieve by efining several constraint operations over the representation of the constraints that are use by the query evaluation algorithm (cf Definition 2) Moreover, the same operations are also neee for the bottom-up evaluation (Kanellakis et al, 1995) an thus we can reuse results obtaine in (Kanellakis et al, 1995, Revesz, 1993, Srivastava et al, 1994, Toman et al, 1994) Termination Constraint Memoing guarantees termination of queries for all classes of constraints that have a terminating close-form bottom-up evaluation proceure Also, the complexity bouns of the bottom-up proceure are preserve The expressiveness of the language can be easily extene to accommoate various classes of constraints as long as every class of constraints is equippe with several elementary operations on the unerlying representation of the constraints This step is quite subtle if termination of queries is to be preserve In contrast to bottom-up methos, it is also possible to exten the query language to classes of constraints, where termination is not guarantee Even in those cases the algorithm reuces the possibility of non-termination (Sagonas et al, 1994) The use of a top-own metho allows a fully goal-oriente query evaluation: the information present in partially instantiate queries is use to prune the search space of queries The efficiency achieve by this metho is better than the efficiency of comparable bottom-up methos incluing program rewriting techniques (eg, Magic Set Transformation) The top-own evaluation strategy allows a irect use of the results obtaine in the area of compilation techniques for logic programming languages (Gao an Warren, 1993, Swift an Warren, 1994a, Warren, 1983) Hanling the constraints as first-class ata allows us to use these techniques for query evaluation in constraint atabases In (Ullman, 1989) the bottom-up approach (equippe with a query transformation phase) is shown to be no worse than the top-own approach for restricte classes of Datalog

3 339 programs over groun relations We show that the top-own approach is no worse than the bottom-up approach in the worst case, an in many empirical examples the top-own evaluation is much faster than the bottom-up evaluation of the same query The rest of the paper is organize as follows: section 2 introuces the constraint representation, the abstract constraint operations, an a close form bottom-up evaluation proceure for Datalog in terms of these operations Section 3 escribes the propose evaluation metho, Constraint Memoing, inclues the sounness, completeness, an termination proofs, an iscusses possible optimization techniques specific to the propose metho Section 4 introuces a general Magic Templates transformation (MT ) for Datalog for comparison purposes Section 5 stuies both the analytical complexity of query evaluation using Constraint Memoing an gives results that provie empirical evience of the practicality of the propose evaluation metho Section 6 conclues the presentation with a brief iscussion of the relate work an with possibilities of further improvements an irections for research 2 Preliminaries This section introuces the basic builing blocks in terms of which the evaluation of Datalog queries is efine Also, for reference, the stanar bottom-up query evaluation proceure is introuce in terms of these builing blocks "!$# Let % & be a set of satisfiable atomic constraints We efine % to be the least set of constraints close uner the following rules: 1 true '% 2 % &)( % 3 if *)+-,*/'% an *)+ 1 */ is satisfiable then *2+ 1 */)'3% 4 if *4' % an 56'87:9<;=*?> then there exists a quantifier free formula *<+A@CB-BDBE@ *GF (in DNF) equivalent to Ḧ 5 BI* such that *JK'3% for every satisfiable *LJ where MONQPKRQS 5 if *4' % an T is a renaming of variables then *OT?' % 7U9V;=*?> enotes the set of free variables in * This efinition is similar to the efinition of Constraint Domain (Jaffar an Maher, 1994) However, % contains only satisfiable constraints The elements of % are use as a finite representation of the (possibly infinite) relations store in a constraint atabase The query evaluation over such a representation is base on the following operations: "!QWYX[Z2! \ ] ^ _Z `^ \-\ba Let c be a set of variables A Constraint class is a set of constraints % from Definition 1 equippe with the following (computable) operations: %fe?%gh%2ikjmlon that for every pair of constraints *<+o,*/ computes the conjunction *+p1*/ if the conjunction is satisfiable; otherwise it fails (returns l ) Constraint Conjunction 1

4 R 34 Constraint Projection H finite set of variables 9 ;=c > e?%g ;% > that for every constraint * computes the set j * J n that satisfies the conition * J H 5 + B H 5"! BI* an every for M?N&%VR(' Note that by Definition 1 the function is well efine an always returns a finite subset of % where 5$#O'89 Constraint Subsumption R %e %3g *)+R bool that satisfies following conition: */ implies *2+*)_*/ The first two operations are, in the worl of constraints, the equivalents of relational algebra join an projection operations However, while in the case of groun tuples the projection returns always only one tuple (constraint), in the case of more general constraints the constraint projection may return a set containing more than one constraint representing a isjunction (Toman et al, 1994, Williams, 1976) +, ^-/ `m21 In (Toman et al, 1994) we consiere the following example: Assume that we want to eliminate quantifier H3 from the constraint: ;"H3m> ;=554(6D+N :6 /)N7;1<3 F>=> Clearly we cannot replace it simply by / 4@?fNA; as in the case of gap-orer constraints: we nee to take into account the perioicity constraint 3?F =, ie, we nee to make sure that there is at least one integer of the form j =B4DC S n between 5E4D6 + an ;BF<6 / Thus, the equivalent quantifier-free formula is ;=5>4:6o+G4H?-> FI= 1 5>4&6D+G4&6 ;=5>4:6o+G4(Km> FI= 1 5>4&6D+G4&6 ;=5>4:6 + 4_S > F = 1 5>4&6 + 4&6 / 4_SfN7; It is easy to see that the variable 3 was successfully eliminate an the resulting constraint is a isjunction of conjunctions of perioicity an gap-orer constraints The last operation, the constraint subsumption, replaces the uplicate elimination for groun tuples Note that the L is not unique by efinition an oes not have to imply ) L However, a better approximation of ) relation by the R operation reuces the number of possible uplicate answers an improves the efficiency of the evaluation methos In the following text we omit the superscripts We also use a strict (l -preserving) version of 1 The following efinition states a funamental property of constraint classes on which the termination proofs of bottom-up query evaluation proceures are implicitly base (Revesz, 1993, Srivastava et al, 1994, Toman et al, 1994, Ullman, 1989)

5 R 341 "! YX[Z2! \ ] ^!- / ^ _Z `^ \-\f!z2! \ ] ^ \ba Let % be a con- straint class If for every finite set of variables 9 an for every set ( % such that * ' 3BI7U9V;=*?> ( 9 there is a finite subset ( such that * ' 3BIḦ * ' BI* * (ie, covers with respect to R ) then % is constraint-compact This property plays a central role in the termination proofs of both the bottom-up base query evaluation proceures (cf Section 21) an the top-own query evaluation proceure evelope in Section 3 In general, the above conition coul be weakene to require only that every infinite set of constraints contains a finite cover (where every constraint is covere by possibly several elements of the cover) However, the use of the weaker efinition may require much more expensive subsumption checks (Srivastava, 1993) The two efinitions are equivalent for constraint classes that satisfy the single subsumption property (Maher, 1993) Example 4 (Common Constraint Classes) Stanar Datalog can be efine using the class of constraints generate from the set j 5 ' On where is the set of all constants in the Datalog program (the active omain (Abiteboul et al, 1995)) Allowing general equality may cause problems to the stanar evaluation strategies (rules may not be range-restricte) However, in our case we simply generate the appropriate class of constraints from the set jo5 '?nli jo5 3n The evaluation remains otherwise unchange as we use more general evaluation mechanism Incorporation of more interesting constraints, eg, constraints over integers ;)> is also easy: the gap-orer constraints (Revesz, 1993) are generate from the set %4j 56N 6'?nmij6N 5 '?nmij 54>6 NJ3 6)' Gn Similarly the perioicity constraints are generate from %"! j 5 F 6 6'?n In (Toman et al, 1994) a close form bottomup evaluation proceure for %! %# i6%! was evelope incluing the constraint 6)'<n (gap-orer constraints operations from Definition 2 The ense orer constraints over $ can be incorporate by a slight moification of constraint operations efine in (Kanellakis et al, 1995) All the above constraint classes are constraint-compact However, there are also constraint classes where all the constraint operations are efine, but which are not constraintcompact, eg, the class generate from the set j 5 4(6 NJ3 with possibly negative size of the gap (Revesz, 1993)) or the linear arithmetic constraints (Kanellakis et al, 1995) "! %YX[ ^^ `!& (' ]!)& ] ^-?a Let % be a class of constraints A atom is a preicate symbol with istinct variables as its arguments A Datalog is a set of clauses of the form where an J are atoms an, '3% +*-,, +,DB-BDB,/ F We assume that the extensional atabase (EDB) is represente by a set of unit clauses +*-, as a part of the Datalog program A query over such atabase is a tuple containing an atom an a constraint the returne tuples have to satisfy "! YX2143 ]65 a Let 7 be a Datalog program, 8 an atom, an * ' % We call the tuple ;8f,*?, 7?> a query The answer to the query ;98f,*?, 7?> is a set of valuations T such that 7;: ;98 1 *?>[T A query evaluation proceure is an algorithm that computes an answer to the query A query is partially instantiate if the constraint * * L true) is nontrivial (ie,

6 L Close-form Bottom-up Evaluation The usual approach to query evaluation for Datalog is a variation on the bottom-up evaluation algorithm (Ullman, 1989) In its simplest form a bottom-up evaluation algorithm is efine as follows: "! YX ]/ ] ^ "! Ka Let U;"5 +,DBDB-B,5 F > be an atom an * '_% a constraint such that 7U9?;"*?> ( 7U9V; 2> A pair ; U,E*O> is a constraint atom A (% -)interpretation is a set of constraint atoms Constraint atoms play the role of groun atoms (tuples) store in a stanar relational system The efinition of the operator is now similar to the efinition of the corresponing operator on groun atoms However, in this case all the operations in the efinition of are efine with respect to the chosen class of constraints % (see Definition 2) "! YX - - ^ Z2! \ 3 >/ ] ^! ]Ka Let 7 be a Datalog program an a % -interpretation We efine ; > j ; U,E* > +*-,, +,DB-BDB,/ F ' 7, ; J,E* J >G' for all MON PpR S,, ' %K, *+,41 * + 1 BDB-B1 * F exists, * 'HBI*?, an if * R * for some ; V,* > ' then * # * n ;=7U9V;=5> F7U9V; 2>,E5> The variables in the constraints where H is a shorthan for 5 BIH are rename using the variable names in the associate atoms of the clause The bottom-up evaluation algorithm remains unchange: all the moifications neee for the evaluation of constraint queries are encapsulate in the efinition of the operator Algorithm 9 (Naive Bottom-up Evaluation) Let ;98f,E*O,/7O> be a query The following algorithm computes the answer to this query repeat,, ; m> while return jo*_1, ;8k,/,f> 'n This arrangement also shows how other -base evaluation proceures can be utilize for constraint query evaluation, eg, the semi-naive bottom-up evaluation (Ullman, 1989) Algorithm 9 was shown to be soun an complete for Datalog (Ullman, 1989), Datalog, an Datalog! (Revesz, 1993, Toman et al, 1994) A simple generalization of the proofs in (Toman et al, 1994) shows sounness an completeness of Algorithm 9 for a general class % : Notation 1 Let ( % Then!"#! is the set of valuations T such that T :Q* for some *4' For a constraint interpretation we efine!"$! j 2T ; V,*?> ',T : *Un Theorem 11 (Sounness an Completeness) Let 7 be a Datalog program Then!% '& ; m>(! ) *&p; m>b

7 J 343 Proof: By simultaneous inuction on stages of an The base case hols vacuously Let P M 8 _' ; > Then there is T an extension of an a clause 8 *,, +,-BDB-B, F in 7 such that T :+, an J J + Tk' ; m> Then by the inuction J + hypothesis # T '!% ; m>(! Thus for each # there is a constraint * # such that ; #,*2#o> ' J + ; m> an T :, 1Y* + 1 B-BDB 1 * F By efinition of there is '$Ḧ B, 1*)+ 1YBDB-B18*GF such that +: Therefore ' J!% irection is similar ; >! The other Theorem 12 (Termination) Let % be a constraint-compact class of constraints Then Algorithm 9 terminates for every Datalog query Proof: Immeiate from Definitions 3 an 8 Assume, that Algorithm 9 oes not terminate Then in every iteration it generates a constraint atom that is not subsume by any previously generate constraint atom As there are only finitely many ifferent preicate symbols in every Datalog program, there must be at least one symbol, that occurs infinitely often among the generate atoms However, this is an infinite set of constraints over a fixe finite set of variables an thus it must contain a finite constraint cover by Definition 3; a contraiction All the constraint classes in Example 4 have a close-form terminating bottom-up evaluation proceure (base on Definitions 8 an 9) 22 Goal-oriente Evaluation Strategies There are several stanar improvements to the naive bottom-up evaluation algorithm, eg, the semi-naive algorithm(ullman, 1989) However, these strategies fail to take into account the information containe in a partially instantiate query: they are not goal-oriente There are two major approaches to solving this problem in the case of stanar (groun) Datalog: 1 Rewrite the original program using the Magic Templates (MT) transformation technique (Bancilhon et al, 1986, Ramakrishnan, 1991) an subsequently evaluate the transforme program bottom-up, or 2 Aopt a variant of a top-own evaluation strategy (Clocksin an Mellish, 1987) base on the resolution principle (Lloy, 1987) In this paper we consier mainly the top-own, resolution-base methos However, the MT optimization for constraint euctive query languages is also be introuce for comparison purposes It is well known that the stanar top-own strategies, eg, the SLD-resolution (Clocksin an Mellish, 1987), espite their efficiency, have a major rawback as query evaluation proceures: they lea to nonterminating computations even in the situations, where the bottom-up algorithms are guarantee to terminate Note also that breath-first traversal of a SLD-tree oes not guarantee termination in general The rawback is cause by occurrences of infinite paths in SLD search trees This has been observe in several papers, eg, (Chen an Warren, 1993, Tamaki an Sato, 1986) an an alternative to SLD-resolution was propose (uner various names) The main iea lies in remembering answers for alreay resolve subgoals This approach guarantees

8 X 8 # X 6 ) C ) ' ' C 6 J ' 6 ) C ) termination in the case of function-free logic programs (Swift an Warren, 1994b) We exten this metho to constraint euctive queries while preserving the termination an complexity bouns of the bottom-up evaluation algorithms 3 Top-own evaluation for Datalog In the last section the bottom-up evaluation of Datalog was efine in terms of elementary operations over a given constraint class This section shows how a top-own query evaluation proceure (SLG-resolution (Chen an Warren, 1993)) can be refine using the same operations to hanle constraint queries This approach allows us to buil an efficient topown evaluation proceure for every class of constraints that has a close form bottom-up evaluation proceure Moreover, the termination an complexity bouns of the bottom-up algorithm are preserve The moification of the (positive fragment of the) SLG resolution for Constraint Memoing (SLG resolution) is efine by the following set of rewriting rules: "!$#6YX ] f] &] 3 `m\ a Let % be a constraint class an ;8*?>, ;8 +,-BDB-B, F E*?>, ;98 f,e* / /,DB-BDB,/ F E*O>, an ;8f,22> where 8 is an atom, <+,DB-BDB, F are literals, an *?,*,2 are constraints in %, be noes from which we buil SLG-trees using the following rules: Action/Noe Chilren Conitions Clause resolution Query projection 2 3 "$#&%5 Answer propagation ZY [\"$#&%5 '*ab%5 ] Answer projection! "$#&%('*) + 6 8:9:9:9:8 5 = %('*) + 8:9:9:9:8 5 = %('*) "$#&%i%('*) + -, 2 34"$#&%( :9:9;9:8 -/1 2 34"$#&%(5&C, /X, /, / ZY [\"$#&%5 5<6 =:> %('@?AB6) 5&C =D %'E?AFCG) 6 8:9:9:9;8 6 8 %5^] 8:9:9;9:8 5 = %'*) ZY [\"$#&%5 6 8 %5^] 8:9:9;9:8 5 = %'*) 2 34"$#&%(5 ] 8:9:9:9;8 5 = %'N?Fc 2 34"$#&%(5 ] Y jk:"$#&%lc Y jk:"$#&%lc 8:9:9:9;8 5 = %'N?Fc for all H<IKJ LNM such that #POQAFR 5&R 6 8;9:99:8 5SR = UTV an 'N?WA R is satisfiable for all ' R T_` > 9 ' for all c R Tef "$5 ' a a ) where ' a Lhg ' a a an 'N?Fc R is satisfiable for all c R Tm_n 9 ' where 8f,/ J, an J are atoms, J,*?,*,E*,* J,/, %, an o p qo;9f,* > is the set of answers collecte from the leaves of the SLG-tree roote by ;f,* > (introuce in Notation 16) A SLG-tree is a tree built from a noe ;8*?> by a finite application of the above rules A SLG-forest is a set of SLG-trees

9 345 Note that the Answer propagation an Answer projection rules have to cooperate: every time a new answer an answer not subsume by any answer in the same tree generate earlier is prouce, it is propagate to all the noes that have alreay been resolve using answers from this particular tree Also, the Answer propagation rule is responsible for creating new SLG-trees in the SLG-forest when no tree with a root noe that subsumes the goal to be resolve can be foun In the presente form the Query projection rule implements the left-to-right selection rule (common to most of the LP systems) However, any other goal selection strategy can be implemente here, eg, selection base on specific SIPS Sieways Information Passing Style (Ramakrishnan, 1991) The main ifference between SLG an SLG is in two aitionalrules: the Query Projection an the Answer Projection The Query Projection rule is responsible for etermining what are the goals to be resolve by answer resolution (ie, what is the goal-constraint pair to be looke for among the alreay compute answers) The Answer Projection is responsible for storing the compute answers for a given goal for subsequent lookup an propagating them to the appropriate noes Note the essential use of constraint projection which allows to etermine the relevant constraint for every atom The SLG-resolution can hanle negation using aitional rules (Swift an Warren, 1994b, Toman, 1997) However, our proposal currently allows only positive programs Aing negation is briefly iscusse in Section 6 The SLG rewriting rules are use for the query evaluation as follows: "!$# Let U;8k,E*?> be the SLG-forest generate from the query ;8k,E*O,/7?> as follows: 1 create a SLG-forest containing a single tree j ;98E*O>n 2 expan the leftmost noe using the rules in Definition 13 as long as they can be applie 3 return the set o p qd;8f,*?> as the answer for the query Let q ;8k,E*?> be a SLG-tree roote by the noe ;98E*O> We orer the answers (ie, the noes ;8 E*?> ) in the SLG-forest accoring to the orer in time in which they were erive: "!$#6% Let Z ;98 > an ;8 > be leaves in the SLG-forest generate by the SLG-rules (cf Definition 13) Then we say that ;8 > is oler than Z ;8 l2 > if the noe ;8 > is generate before the noe Z ;84 l2 > We assume that the SLG-trees are orere left-to-rightin the orer they were create The choice of selection strategy oes not affect the sounness/completeness of the algorithm (Lloy, 1987) However, the selection strategy may influence the efficiency of the system (cf Section 6) Notation 16 Let o p qd;8f,*?> be the set of all oler Z ;82 >G' q ;98f,*?> then AL R such that ;9822> ' q ;8k,E*?> an if an Lemma 17 Let Z ;822> be a leaf of the SLG-tree q ;8f,*?> Then for every application of the answer propagation rule along the path ;8*?>Ug BDBDBAg Z ;8 > if Z ;8 2 > was a propagate answer then Z ;98 2 > is oler than Z ;9822>

10 346 Proof: Immeiate from the Definitions 13, 14, an 15 Sounness an completeness of Constraint Memoing is proven by reuction to sounness an completeness of bottom-up evaluation (Algorithm 9) Note that the set o4p q-;98f,e*o> may not be unique an epens on the orer in which the noes Z ;822> are generate However, for our purposes it is sufficient that the set of valuations!uo p qd;8f,*?>(! is unique Lemma 18 Let q ;8f,*?> be a SLG-tree an a valuation Then '!Uo4p q-;98f,e*o>! implies : * Proof: By inuction on the height of the SLG-tree q ;8k,E*?> To prove correctness of the algorithm we show that all the erive answers are also erive in the bottom-up computation: Lemma 19 Let ;8O&,*&, 7?> be a query Then for every q ;8k,E*?>)' every valuation '!Uo p q-;8k,e*?>(!- 8 ' & ; m> B f;98o&,*& > an Proof: Inuctionon the age of answers: Let Z ;8* >G' q ;98f,E*O> such that : * Then there is a path ;8*?> g ;8 / +,-BDBDBE, F * + > g B-BDB g Z ;8* > ' q ;8f,*?> that starts with a Clause Resolution step using a clause 8 *,, V+,-BDBDB, F'+7 (cf Figure 1) By Definition 13, *4 ' Ḧ BI*Q1, 1 O+L1 B-BDB-1 F where )J is an answer propagate from the SLG-trees q ; J,E* J > Thus, there exists T an extension of, such that T :4*Q1, 1 O+p1 BDB-BD1 )F Clearly, T : )J for MkN PGR S By Lemma 17 all the answers Z ; J:2 J"> use along this path have been compute before Z ;8 E* > an thus by the inuction hypothesis we have J TO' & ; > By efinition of an the fact that T :, we have 8OT 8 ' & ; m> Thus, all answers not only for the main query, but also for all subqueries represente by the SLG-trees in the SLG forest generate from the main query are soun Lemma 2 Let 8 be an atom an * +,* / '% constraints Then * + R * /!Uo p q-;8k,e* + >(! ( Proof: Immeiate from the efinition of o p q an Lemma 18!Uo4p q-;98f,* / >(!mb The next thing to show is that the algorithm computes all the answers to the given query This is a little bit more complicate, as the algorithm oes not compute all the answers to an uninstantiate query like the bottom-up evaluation oes However we can show: Lemma 21 Let ;8O&,*&, 7?> be a query Then for every q ;8k,E*?>)' every valuation 8 ' '& ; > 1 : * '!Uo4p q-;98f,*?>!b f;98o&,*& > an

11 J > Operation SLG-tree for goal Other SLG-trees E Clause> resolution * >! " # > # > Query projection %$ > >! > >& & > ' () *# > > #+ % >, > -,/ 1 > Answer propagation Let + > > : >; Answer propagation Let! " #+ #% + new 4 > ' () # =<> : Answer > projection! > & & D (576 * > SLG-tree 3 (576 > > 8, -,/ 9!" *8 : > new 4 SLG-tree 3 (5 6 % Figure 1 SLG g Evaluation of goal? wrt a A Proof: By inuction on stages of Let 8 ' ; m> an : * The claim hols vacuously for P M Let P M Then there is a clause 8 *,, +,DBDB-B, F '7 an an extension T of such that T(:, an J T J + ' ; m> We can construct a path in q ;98f,*?> that ens with a noe Z ;982 > such that : Using the assumption : * we have T : *_1, (this correspons to the application of the Clause Resolution rule) Thus, T :Q* + for at least one element * + of H > *1, By the inuctive hypothesis TU')!Uo p qd; +,* + >! This fact in turn yiels a noe Z ; >'Eq ;9 +,E* + > such that T : + Again, using the previous assumptions, T :Q* 1,_1 + In general, let *4 J be an element of H BI*Y1, BDBDB 1 J + Clearly, by PFJ? applications of the inuction hypothesis, T :Q* J Then by inuction hypothesis on J we have TO'!Uo4p q-;9 J,E* J >(! This exactly correspons to an application of the Query Projection an Answer Propagation rules from Definition 13 After S steps we have T : * 1,_1 + 1 BDBDB 1 F Thus, : for some element of H BI*1, 1 + 1CB-BDB 1 F (this is achieve by the Answer Projection rule), an therefore '!Uo4p q-;98f,e*o>! In the actual algorithm, the application of the Answer Propagation rule oes not nec- J > for answer resolution However, if a ifferent tree J > is use then it is always the case that * J R * J Thus, by Lemma 2, J >(! an no answers can possibly be lost essarily use the tree q ;9 J,E* q ;J,*!Uo4p q-;9 J,E* J >! (!Uo p qd; J,E*

12 348 (1) B initial query noe (2)!" G (4) ' () B B get results for (8)!" 2 B answer from noe (7) (9) ' () B get results for ( + & ) (18)! " 2 answer from noe (17) (2) (5 6 G new answer for (31)! " 2 answer from noe (3) (33) (5 6 G (41)! " 2 answer from noe (4) (44) (5 6 G new answer for (21)!" 2 B answer from noe (2) (22) ' () B get results for (23)! " 2 answer from noe (7) (25) (5 6 G (24)! " 2 answer from noe (2) (26) (5 6 G (46)! " 2 answer from noe (44) (54) (5 6 G (45)!" 2 B answer from noe (44) (48) ' () B get results for ( ) (3)!" B (5) ' () B (6)!" 2 (7) (576 2 new answer for (1) B subquery from (9) (11)!" G (13) ' () B 2 get results for (19)!" 2 B answer from noe (17) (27) ' () B get results for (28)! " 2 answer from noe (7) (3) (5 6 G new answer for (29)! " 2 answer from noe (2) (34) (5 6 G (47)! " 2 answer from noe (44) (55) (5 6 G (32)!" 2 B answer from noe (3) (35) ' () B get results for (36)! " 2 answer from noe (17) (38) (5 6 G (37)! " 2 answer from noe (3) (39) (5 6 G (42)! " 2 answer from noe (4) (56) (5 6 G (43)!" 2 B answer from noe (4) (57) ' () B (12)!" B (14) ' () B (15)!" 2 (17) (576 2 new answer for (16)!" 2 (4) (576 2 new answer for (49) B subquery from (48) (5)!" G (52) ' () B B (51)!" B answer from noe (4) (53) ' () B The First column inicates the orer, in which the iniviual noes have been create, the secon column shows the SLG-trees generate, an the last column escribes how answers have been generate by the SLG-trees; the relation represents the eges in the graph O"! $# Figure 2 Annotate SLG evaluation of query % &('*),+-/ for )132

13 X ' ' C 6 ' ' C 349 By composing the previous Lemmas we have: Theorem 22 (Sounness an Completeness) Let ;8f,*?, 7?> be a query tuple Then for all valuations such that : * 8 6' & ; m> '!Uo p qd;8f,*?>(!mb Proof: Sounness follows from Lemmas 18 an 19, completeness from Lemma 21 The sounness an completeness proof is base on the reuction to the fixpoint computation on groun instances However, to prove termination of the query evaluation algorithms (in both the bottom-up an top-own cases) a finite encoing of a potentially infinite result of the evaluation is neee (Revesz, 1993, Toman et al, 1994): Theorem 23 (Termination) Let % be a constraint-compact class Then the SLG evaluation terminates for all queries ;8k,E*O,/7?> Proof: Let % be a constraint-compact class of constraints Then: 1 The number of trees in the SLG-forest f;98f,e*o> is finite, as there are only finitely many preicate symbols 8 ' 7 an for every preicate symbol the set j * J ;8,* J >G' f;98f,e*o>n is finite by Definitions 3 an 13 2 Every noe has finitely many chilren, as there are only finitely many clauses in 7 3 Every noe has finitely many chilren, as the set H BI* is finite for any *4' % 4 Every noe as only finitely many chilren, as there are only finitely many elements in the set o4p q-;98 =,E* > for any atom 8 an * '3% by Definition 3 5 Every q ;8f,*?> has finite epth, because of finite number of subgoals in the boies of each clause in 7 Therefore the rules from Definition 13 can be applie only finitely many times The termination of the Constraint Memoing algorithm is guarantee in all cases when the bottom-up algorithm terminates computing a finite interpretation & ; > Moreover, it is usually easy to ecompose the original bottom-up evaluation proceure an extract the elementary operations on constraints neee for Constraint Memoing (Definition 2) 31 Optimization To reuce the overhea introuce by the SLG resolution (in comparison to stanar SLD resolution) we explore several possibilities: 1 Solving more general goals than necessary: Action/Noe Chilren Conitions Query projection 2 3 "$#&%5 6 8;9:9:9:8 5 = %'*)1+, / X Y [\"$#&%5 Y [\"$#&% %5 ] %:5 ] 8:9:9;9:8 5 = %('*) 8:9:9;9:8 5 = %'*) _` > 9 ' K' 6 9:9:9 for some ' 6 8;9:9:9:8 C

14 X This moification may reuce the number of SLG-trees in the SLG-forest (in cases where : H > BI* : ' ) However, the propagation of constraints at the time of goal resolution is reuce The sounness an completeness properties are preserve by Lemma 2 The termination is guarantee similarly to Theorem 23 In (Gao an Warren, 1993) the following version of such a moification was consiere: Action/Noe Chilren Conitions Query projection 2 3 "$#&%5 6 8:9;9:9:8 5 = %'*) Y [\"$#&%5 6 8 true %;5 ] 8:9;9:9:8 5 = %'*) none In this case, there is only one SLG-tree per preicate symbol On the other han, no constraints are propagate at the time of goal resolution the constraints are use merely to restrict the returne answers Thus, the computation essentially computes all answers to an uninstantiate query similarly to the bottom-up algorithm, an the performance suffers: The performance is approximately the same as evaluating the uninstantiate query 2 Mixe SLG an SLD resolution (by memoing only subset of the preicate symbols present in the program) Action/Noe Chilren Conitions Non-table resolution 2 3 "$#&%5 6 89:9:9:8 5 = %'*) , / "$#&% :9:9;9:8 5 =:> 6 5 ] 8 8:9;9:9:8 5 = %'K?BA 6 ) for 5 6 not table goal OQAFR 5SR 6 8:9:9:9:8 5&R = TV an 'N?BA R satisfiable 2 3"$#&%5 6 C 8:9:9:9;8 5 =D C 8 5^] 8:9;9:9:8 5 = %'K?BA C ) This is a ifferent way of reucing the number of SLG-trees generate by the algorithm: SLG-trees are generate only for a subset of the preicate symbols in 7 The remaining symbols are always resolve using program clauses, similarly to SLD-resolution Again, sounness an completeness are preserve (by simple moification of Theorem 22) Termination is guarantee if an only if at least one preicate is resolve by the SLG resolution for every cycle in the epenency graph of 7 (this follows by an easy extension of Theorem 23) Otherwise, an infinite branch may appear in some of the SLG-trees This may lea to non-termination similarly to the case of SLD-resolution Also, as there is only a boune number of SLD resolution steps between any two SLG resolution steps, the boies of the non table clauses can be unfole in the boies of their callers This transformation completely eliminates the nee for non table resolution steps 3 Program transformation similar to supplementary magic (Ramakrishnan, 1991) The previous foling transformation may introuce unnecessary recomputation of conjunctions of goals This can be avoie by a technique that fols common parts of boies of the clauses an creates separate clauses Note that the recomputation is avoie by making the heas of such clauses table resolve by the SLG resolution The last two optimizations are base on program transformations However, in contrast to the Magic Transformation, these two transformations are completely query-inepenent 4 Magic Templates Transformation for (MT ) This section escribes a simple version of the program transformation approach to the goal-oriente query evaluation in constraint euctive atabases the Magic Templates

15 q 351 transformation (in the constraint setting the ifference between Magic Sets an Magic Templates is blurre) The transformation has to be slightly moifie in the context of constraint atabases Algorithm 24 (MT program transformation) ;98f,E*O,/7O> j *-,, magic6,/ +,DB-BDB,/ F, magic + *-,, magic6, magic F *,,/ +,DB-BDB, F +, magic i j magic8 * *GJ *GJK'8Ḧ *Un *-,,/ +,-BDBDBE, F ' 7Un where magic, magic +,-BDB-B, magic F are the magic atoms for V, +,-BDB-B, F, respectively Again, for simplicity, only the left-to-rightsips is use This correspons to the selection rule use in Constraint Memoing In both cases, ifferent selection rules may improve the efficiency of query evaluation (Ramakrishnan, 1991) However, in the case on MT, the SIPS is fixe uring the program transformation phase an there are technical ifficulties with combining ifferent SIPS in one program In the case of SLG evaluation, the selection rule can be ajuste uring the evaluation process ynamically while preserving correctness of the answers The Magic Templates transformation is often precee by an aornment transformation (Ramakrishnan, 1991) The aornment phase is esigne to partition the search space accoring to (the statically erivable) information about free an boun arguments of the literals The purpose of this transformation is threefol: 1 The original purpose of the aornments was to project out all the arguments that are not boun an thus guarantee that only groun tuples are generate (in the constraint setting this is not neee) 2 The other important consequence of using aornments is the possibility to reuce the arity of literals in the boies of clauses This leas to more efficient bottom-up evaluation: reuction of arity by one may cause linear speeup (Kemp et al, 199) 3 The aornment partially factors the search space an allows to propagate only the neee restrictions In the SLG case the first use of aornments is not neee (similarly to the bottomup proceure for constraints: Algorithm 9) Thus we implemente the MT without the aornment phase The secon an thir uses are also partially achieve in the top-own evaluation: The factoring technique uses a static preiction of bining patterns of literals to reuce their arity However, at the runtime, these literals have to follow this preiction an thus the effect of factoring is partially achieve using the subsumption check Moreover, the run-time check etects all possible factoring opportunities (for the particular evaluation orer) while the static methos can preict only a subclass of them Also in many cases, the top-own metho groups the answers to particular goals accoring the binings present in these goals This way it propagates only the relevant binings (in the bottom-up metho this effect coul be achieve by builing an goal-base inex on the magic atoms) Note that this grouping of answers can not be achieve by using aornments as it epens on the actual ata in the constraint atabase

16 / / 352 We test the relative performance of the evaluation methos by computing paths in various graphs tc(x,z):-tc(x,y),tc(y,z) tc(x,y):-e(x,y) path(x,z):-x<y<z,path(x,y),path(y,z) path(x,y):-e(x,y) In the gap-orer case (path) we only look for paths where all eges lea from noes with lower number to a noe with higher number We use the tc an path programs to fin paths in the following graphs: chain(n) cycle(n) tree(n) mesh(n) fanchain(m,n) The noes of the graphs are numbere by integers from top to bottom In the case of gap-orer Datalog, the integers are represente using constraints (eg, P is represente by P F7?fN45N4P4@? ) Note that in the case of acyclic graphs the parent noes are always labele with a smaller integer Figure 3 Test Programs 5 Performance In this section, the analytical complexity boun for the Constraint Memoing evaluation is iscusse The complexity of the top-own evaluation epens on the particular class of constraints % ; our analysis is base on the relative comparison to the (time- an space-) complexity of the bottom-up evaluation proceure The analytical results are confirme by experimental results that show the performance gains achieve by Constraint Memoing 51 Theory We show that the complexity of Constraint Memoing is no worse than complexity of the bottom-up evaluation: Theorem 25 Let & ; > be the result of the bottom-up evaluation of the query ;98f,*?, 7?> an ;C > a function such that : & ; m> :' f; ;/: 7 : > > Then 1 the SLG-forest U;8f,*?> has at most f; 2 there are at most f; Proof: > noes > applications of the SLG rules in the evaluation of the query (1) follows from the observation that every SLG-tree in the SLG forest has at most f; > leaves an thus also at most f; > noes as the height of the trees is fixe by the

17 353 number of subgoals in the clauses of 7, an there are at most f; > ifferent SLG trees in the SLG-forest as the size of & ; m> limits the number of possible roots of the SLG-trees (2) follows immeiately from (1) as every application of a rule creates at least one new noe in the SLG-forest A careful implementation nees to store only a single path (of fixe length) in every SLGtree in the SLG forest Thus, the space requirements can be reuce to f; > On the other han, the quaratic number of rule applications cannot be avoie in general However, by analysis of the bottom-up algorithm the number of applications of clauses in 7 is also quaratic with respect to (the bottom-up algorithm can recompute the same element of the interpretation several times, even if it is ae only once to the interpretation & ; > ) 52 Implementation We have implemente experimental versions of the following euctive query evaluation algorithms: 1 the Naive Bottom-up: a straightforwar implementation of Algorithm 9, 2 the Semi-naive Bottom-up: a moification of Algorithm 9 (Ullman, 1989), 3 the Semi-naive Bottom-up with MT, an 4 the Constraint Memoing algorithm SLG The implementation of each of the evaluation algorithms is parametrize on the unerlying class of constraints For each constraint class we provie elementary operations on the constraint representation (cf Definition 2) together with a few aitional auxiliary operations: constraint_new(n,c): Given a number N it returns C as the representation of the constraint true over N variables This operation is use to create a fresh environment for constraints, present uring the evaluation of the iniviual clauses of the Datalog program constraint_an(g,c,co,cn): This operation computes the constraint conjunction of the constraint CO with the constraint C where all the variables in C are rename with respect to the variables of the atom G This operation is use when a constraint erive by a subgoal of a clause is an -e to the overall constraint over the variables in the boy of the clause The operation prouces only consistent constraints; if the conjunction is not satisfiable, the operation fails constraint_qe(g,c,cn): Let G be an atom Then Cn is a finite set (list) of constraints equivalent to the constraint C after all variables not in G are eliminate This operation is use in the Query Projection rule, where we project the constraint on the free variables of the goal to be resolve, an the Answer Projection rule, where we eliminate all variables not present in the hea of the clause constraint_subsumes(c1,c2) is the subsumption checking proceure The operation succees if C1 subsumes C2 We can assume that the constraints C1 an C2 are

18 354 Naive Semi-naive Semi-naive Top-own Query Data Bottom-up Bottom-up with MT tc(x,y) chain(32) tc(1,32) chain(32) tc(1,24) cycle(24) tc(1,48) tree(64) tc(1,36) mesh(6) tc(,37) fanchain(2,18) tc(,37) fanchain(6,6) tc(,37) fanchain(18,2) path(x,y) chain(16) path(1,16) chain(16) path(,13) fanchain(2,6) path(,17) fanchain(4,4) path(,13) fanchain(6,2) Figure 4 Running times of test queries for various evaluation proceures (in msec) over the same set of variables: we only use this operation to ecie if a new constraint atom has been erive by the particular metho or if a new SLG-tree is neee The last two operations are just for the convenience of the user of the system: constraint_pp(c) allows pretty printing of the results of the evaluation, an constraint_rea(t,g,c) allows entering the constraints as formulas, rather than as the actual representation as Prolog terms In aition, we nee to specify the Datalog programs that we inten to evaluate Note that we use the same implementation of the operations on constraints for all the evaluation algorithms Thus the relative performance of these algorithms is not cause by more sophisticate way of manipulating the constraint representation in one of the algorithms 53 Experimental Results Both the bottom-up (incluing the MT optimization) an the Constraint Memoing algorithms have been implemente in Prolog We woul like to emphasize that neither of the implemente evaluation algorithms takes an avantage of Prolog s top-own evaluation strategy all the algorithms are implemente as meta-interpreters operating on a common groun representation of constraints (note that our top-own technique coul have gaine a consierable avantage by using Prolog s evaluation strategy) Figure 4 summarizes the running time of queries over graphs in Figure 3 The first line shows the performance for the uninstantiate case In the constraint cases (path) the integers in the queries are expresse using constraints similarly to constants in Figure 3 The examples of the instantiate queries are those, where the optimization achieves the least effect (in all cases)

19 355 msec Top-own Magic Sets Bottom-Up msec Top-own Magic Sets Bottom-Up Datalog over 24-element chain (tc, chain(24)) Datalog with Gap-orer constraints over 16-element chain (path, chain(16)) We measure the elapse time to verify if there is a path from noe J (x-axis) to noe (y-axis) The elapse time is plotte on the z-axis Note that in the orere case (right graph), the constraint propagation allows more efficient pruning than in the case of stanar Datalog (left graph) Thus the use of constraints may improve efficiency even for stanar queries Figure 5 Elapse time of query evaluation for all possible paths The results show that while the implementation of the various evaluation methos are comparable (the results on uninstantiate queries are approximately the same), the evaluation of instantiate queries is much more efficient using the Constraint Memoing algorithm The boost is inherent to the top-own evaluation metho is not cause by using a more sophisticate implementation The other two experiments (Figures 5 an 6) show that the query evaluation on constraints generally follows the patterns of query evaluation on groun representation: for all possible queries over a given graph (Figure 5 plots the elapse time for all groun queries tc;"p[, %b> over a C -element chain Similar graphs can be prouce for the other structures in Figure 3), an for all shapes of the given graph (Figure 6 plots the elapse time for graphs with varying fanout/fanin an chain lengths of fanchain;"5, 3> efine in Figure 3) Thus we can expect very efficient Constraint Memoing-base query evaluation engines for constraint extensions of Datalog whose performance will be comparable to the top-own engines for groun Datalog (Swift an Warren, 1994a, Swift an Warren, 1994b) 6 Conclusion We have propose a practical approach to query evaluation for generalize constraint atabases Both the analytical an the empirical results show that Constraint Memoing is no worse than comparable bottom-up methos an in many cases the practical performance is much better even when using a very naive implementation The performance

20 356 msec Top-own Magic Sets msec Top-own Magic Sets Fanout (x) Chain length (y) Fanout (x) Chain length (y) 1 Datalog Datalog The above graphs plot the elapse time to fin a path in the fanchain(x,y) graph from the top-most noe to the bottom-most noe (cf Figure 3) On the an axes we plot the parameters of the use graph: the fanout on the -axis an the chain length on the -axis The elapse time is plotte on the -axis Figure 6 Elapse time for varying fanout an chain length of the Constraint Memoing can be booste by utilizing compilation methos evelope in (Freire et al, 1996, Swift an Warren, 1994a, Swift an Warren, 1994b, Warren, 1983) an performance similar to groun Datalog can be expecte In aition, recent work on scheuling of operations in tabling systems (Freire et al, 1997) shows moifications to the tabling strategy that make it efficient even if external storage is involve The scheuling strategies are orthogonal to the extensions introuce for hanling constraints an thus can be immeiately applie to our proposal 61 Relate Work Recently, there have been several other attempts to make query evaluation in the presence of constraints efficient There are two main irections of this research: 1 The first irection has its roots in the euctive atabase community: Techniques for pushing constraints present in the query were propose in (Kemp an Stuckey, 1993, Ramakrishnan an Srivastava, 1993, Stuckey an Suarshan, 1994) However, the goal of these methos is to preprocess the query (ie, the goal an the rules) with respect to the given constraints for subsequent bottom-up evaluation We present a completely ifferent evaluation strategy where the constraints are propagate ynamically without the nee for the preprocessing of the query Also, in the stanar atabase approach, the constraints are consiere to be mere conitions that restrict the otherwise groun answers Constraint Memoing uses constraints as a tool for representing both the ata compute by the queries an store in the atabase itself (ie, non-groun relations are allowe) This ramatically increases the expressive power of the query language while preserving termination an efficiency

21 357 2 The other irection is pursue in the area of (general) Logic Programming: In (Gao an Warren, 1993, Johnson, 1993, Lim an Stuckey, 199) top-own evaluation for constraint logic programs is propose However, in all cases, general constraint solving proceures are use Thus, these methos are not irectly useful for query evaluation in constraint atabases: termination of queries cannot be guarantee The closest to our work is (Gao an Warren, 1993) However, the metho propose there allows propagation of constants only (ie, constraints of the form 5( ); the constraint part of the query is essentially compute bottom-up Our approach allows full propagation of all possible constraints uring the whole evaluation process 62 Directions of Future Research Future research in this area nees to focus on the following issues: 1 Compilation of constraints To achieve an efficient implementation of Constraint Memoing, ata structures for efficient representation of the constraints have to be evelope There are two main ifferences to be aresse: In most cases, the size of the constraint representation is boune with respect to the arity of a literal However, general Logic Programming engines allow unboune terms to be built Exploring this property may lea to an efficient stack-base implementation (ie, without a heap) of the evaluation proceure On the other han, classical Logic Programming assumes that every (logical) variable is either free or boun to a single term (an this biningcan be change only by backtracking) This assumption is no longer vali in the presence of constraints as more restrictive conitions may be erive after a variable is originally boun Also, the constraints specify complex relations between the iniviual variables, which is not possible in the stanar approach Development of such a representation enables builing of very efficient query evaluation engines base on partial evaluation of the atomic constraints in a given class, similarly to the WAM abstract coe (Warren, 1983) 2 Analysis of bining patterns Similarly to the MT transformation, the queries can be analyze to etermine the flow of information in clause boies (Ramakrishnan, 1991) This is a consierably more complicate task in the presence of constraints: it is no longer sufficient to focus on single variables; the relationships between groups of variables have to be taken into consieration (as note in Section 4) Also, the assumption that all EDB relations are groun (ie, after resolution of an EDB goal all variables are boun to constants) is no longer vali the generalize relations store representation of sets of tuples that may be infinite Such an analysis can be use for several purposes: query optimization (MT-like rewriting), optimization of access to the constraint atabase (inexing), goal reorering, etc 3 Interface to an existing RDBMS As the constraints can be finitely encoe, their representation can be store as tuples in a stanar relational atabase system However, query evaluation has to be carrie out with respect to the semantics of such encoing (ie, to perform, eg, a join of two constraint relations, we can not use the join operation