Extending the Well-Founded and Valid Semantics for Aggregation. Computer Sciences Department. University of Wisconsin Madison. Raghu Ramakrishnan

Transcription

1 Extending the Well-Founded and Valid Semantics for Aggregation S. Sudarshan y Rm. 2A-212, AT&T Bell Labs. 600 Mountain Ave. Murray Hill, NJ sudarsha@research.att.com tel: (908) fax: (908) Raghu Ramakrishnan Computer Sciences Department University of Wisconsin Madison Madison, WI raghu@cs.wisc.edu Divesh Srivastava Computer Sciences Department University of Wisconsin Madison Madison, WI divesh@cs.wisc.edu Abstract We present a very general technique for dening semantics for programs that use aggregation. We use the technique to extend the well-founded semantics and the valid semantics, both of which were designed to provide semantics for programs with negation, to handle programs that contain possibly recursive use of aggregation. The generalization is based on a simple but powerful idea of aggregation on three-valued multisets. The use of three-valued multisets makes our extended well-founded semantics, which we call aggregate-well-founded semantics, more intuitive than the extension of well-founded models by Van Gelder [Van92]. Our semantics and Van Gelder's semantics agree on many programs, and on others our semantics provides results that Van Gelder says are intuitive and desirable, but that his semantics does not provide. The extended valid semantics, which we call the aggregate-valid semantics, generalizes the intuition behind several semantics presented earlier for aggregation; for instance, our semantics properly subsumes the semantics dened by Ganguly et al. [GGZ91] for cost-monotonic programs. Keywords: deductive databases, logical extensions, proof theory, non-stratied aggregation The work of Raghu Ramakrishnan and Divesh Srivastava was supported by a David and Lucile Packard Foundation Fellowship in Science and Engineering, a Presidential Young Investigator Award with matching grants from DEC, Tandem and Xerox, and NSF grant IRI The rst author, on behalf of all those alphabetic-order challenged, led a successful crusade to have the names in reverse alphabetical order. The order does not reect on the degree of contribution by the various authors. Extensions planned for the future include height and weight ordering of authors. y All communication may be addressed to the rst author. 0

2 1 Introduction In recent years there has been much interest in dening semantics for deductive databases/logic programs that use negation, aggregation and set-grouping. We call such programs extended logic programs. It is well-known that in the presence of negation, aggregation or set-grouping there is no acceptable unique model semantics based purely on logical implication, and so the meaning of a program is dened either as the result of some `normal form' computation, or as the set of facts in some `intended' model. Early approaches restricted the class of programs to those that are stratied in some fashion [ABW88, Prz88, Ros90, PP88]. More recent approaches, such as the well-founded semantics [VRS91], the three-valued stable model semantics [Prz90] and the valid semantics [BRSS92a] dene semantics for all logic programs with negation and set-grouping. Recently, there have been several proposals to extend the semantics to take into account special properties of aggregate operations [GGZ91, RS92, Van92]. Beeri et al. [BRSS92b] note that the valid semantics can be extended to take into account special properties of some aggregate operations such as min and max. We present a motivating example in Section 1.1, that illustrates the drawbacks of current semantics for nonstratied aggregation. In this paper, we consider how the well-founded semantics and the valid semantics can be extended in a clean fashion to better handle aggregation. The contributions of the paper are as follows. Aggregate-Valid Semantics: This paper denes the aggregate-valid semantics, an extension of the valid semantics to take into account the semantics of aggregate operations. As with the valid semantics, the approach in this paper is based on a `normal form' computation, and is split into two main parts. The rst part denes when a fact `follows from' a rule and a three-valued interpretation (Section 3.3). The denition of `follows from' presented in the paper generalizes that presented in [BRSS92a]. The second part denes when a fact can be assumed false based on default rules, and when a derivation is considered aggregate-valid. An aggregate-valid computation is one where every derivation is aggregate-valid. A complete aggregate-valid computation (i.e., one that cannot be extended to derive new facts) denes the semantics of programs (Section 4). This part of the denition is the same as in [BRSS92a], except for being based on the extended denition of `follows from'. An important part of the extended denition of `follows from' is the novel notion of three-valued multisets and aggregate operations on them. Three-valued sets extend two-valued sets by allowing the membership status of some elements to be undened. Three-valued multisets extend two-valued multisets by allowing true and undened elements to have multiplicities. Since we use three-valued interpretations in dening well-founded and valid semantics, it is quite natural to have aggregation work on three-valued multisets. Three-valued multisets are especially important for aggregate operations such as count greater than 10, or min that can make derivations even when some values are undened. Note that the three-valued multisets need not always be explicitly constructed, but the concept provides a clean way of understanding the use of aggregate operations on partially constructed multisets. Aggregate-Well-Founded Semantics: As shown in [BRSS92a], the well-founded semantics [VRS91] can be dened via well-founded computations, which are a special case of valid computations. By using the extended notion of `follows from' with the denition of well-founded computations, we get a clean extension of the well-founded semantics to aggregation (Section 4.2). We show that the aggregate-well-founded semantics for aggregation is a special case of the aggregate-valid semantics for aggregation, and may be more ecient to compute in general. A Unifying Framework: There have been a number of proposals for dening the declarative semantics of logic programs with aggregation. Some of these semantics are hard to understand, and others are applicable only to special classes of programs. This has led people to question the utility of having a (hard to understand) declarative semantics for logic programs with aggregation. The valid semantics [BRSS92a] provides a framework that unies many semantics proposed earlier, by means of: (1) providing a clean way of understanding the semantics of the program, regardless of the specic details of the special cases, (2) ensuring that the dierent semantics do not conict, and (3) making the task of extending 1

3 semantics to new ways of making inferences very clean by parametrizing the semantics w.r.t. inference rules. These points are demonstrated in this paper by showing how clean and easy it is to extend valid semantics to the aggregate-valid semantics; essentially all that changed was the denition of `follows from'. Once someone has understood the valid semantics, understanding the aggregate-valid semantics is very straightforward since all that has changed is the denition of `follows from'. Part 3 is very important for integrating new features such as quantitative reasoning with the semantics. Ecient Evaluation: Aggregate-well-founded semantics can be computed with eciency similar to well-founded semantics. Evaluation techniques that make use of special properties of special classes of programs, or provide sound approximations to the aggregate-valid semantics, can be computed even more eciently. Several special cases of aggregate-valid semantics, which cover a large class of programs, have been identied and integrated into the CORAL deductive database system ([RSS92b]), and have proven ecient in practice (Section 6). Our semantics has several advantages over earlier semantics. These are briey mentioned in Section 1.1, and discussed in more detail in Section A Motivating Example We consider an example that motivates the results in this paper. We informally describe the notation we use while describing the example; the notation is dened formally later in the paper. On the following example, the semantics proposed earlier either do not apply, or fail to provide intuitive answers and are hard to understand. In Section 7 we present a comparison of our technique with other techniques for dening semantics, and present further examples that illustrate the benet of our technique. Example 1.1 The following program is the extended party invitation program (Example 6.2 in [Van92]). Several people have been invited to a party. Each person has a positive compatibility with some people (values are stored in the relation pos(person1; person2; degree)) and a negative compatibility with others (similarly stored in neg(person1; person2; degree)). A person will come if the total positive compatibility with those who are coming is above a threshold, and the total negative compatibility with those who are coming is below another threshold. The thresholds are stored in a relation thr(person; pos; neg) where the second and third arguments give the positive threshold and the negative threshold respectively. In the following, rel? (a; b; x) is derived if b has accepted, and a has a negative compatibility of x with b; rel+ is similar, but with positive compatibilities. The program is as follows: accept(x) :? goodenough(x); :toobad(x): rel + (X; Y; C) :? accept(y ); pos(x; Y; C): rel? (X; Y; C) :? accept(y ); neg(x; Y; C): goodenough(x) :? groupby(rel + (X; Y; C); [X]; subset(< C >; S)); thr(x; P; N); sum(s) P: toobad(x) :? groupby(rel?(x; Y; C); [X]; subset(< C >; S)); thr(x; P; N); sum(s) > N: Let the database be: thr(a; 1; 0); thr(b; 1; 0); thr(c; 0; 0); pos(a; b; 1); pos(b; a; 1); neg(c; a; 1): The groupby notation is to be understood as follows (formal denitions are presented later). Consider the literal groupby(rel? (X; Y; C); [X]; subset(< C >; S)). For any given binding X=x, (i.e., for the variable in [X] in the literal) consider the facts of the form rel? (x; Y; C). The literal is satised by a binding X=x; Y=y; S=s S if s S is a multiset such that for each element s 2 s S, if the multiplicity of s in s S is k then there are at least k dierent values y such that rel? (x; y; s) is true. Note the `at least' above, which is a result of the use of subset in the groupby literal. Intuitively, the rules, with the given set of facts have the following meaning: a will accept if b does and vice-versa, but neither will accept rst, and c will accept if a does not. In keeping with the intuition behind 2

4 the well-founded semantics for the case of negation (but no aggregation), accept(a) and accept(b) should be false since they are in a \positive cycle". From that it should follow that accept(c) is true. The above semantics is precisely what is provided by the aggregate-valid semantics as well as the aggregate well-founded semantics, which we present in this paper. On the other hand, the semantics of Mumick et al [MPR90], and Ross and Sagiv [RS92] do not work on the above program since it uses aggregation in conjunction with negation. Both the semantics apply only to restricted classes of programs, and in the case of Ross and Sagiv, it is undecidable whether a program falls into their class (i.e., is monotonic) or not. Our semantics applies to all programs. The semantics of Kemp and Stuckey [KS91] also applies to all programs, but leaves all of accept(a); accept(b) and accept(c) undened. The semantics of Van Gelder applies to all programs, as well, but it too makes all of accept(a); accept(b) and accept(c) undened, which Van Gelder says (and we agree) is undesirable. Van Gelder attempts to give semantics to the program through two dierent translations, but both leave all these facts undened. Van Gelder's semantics are, in addition, quite hard to understand since the translations convert aggregation and subset generation into negation, with several levels of cyclic negation being introduced in the process. This example motivates the need for multisets. Both Van Gelder and Ross and Sagiv simulate multisets by assuming that extra arguments are used to distinguish the elements of the set. However, the task is given to the user. Our technique handles multisets in a straightforward way and avoids burdening the user with having to worry about the distinguishing arguments. 2 2 Denitions We assume familiarity with the standard terminology of logic programming, such as variables, constants, function symbols, terms, atoms, facts and rules. A fact that contains no variables is a ground fact. Aggregate operations have traditionally been viewed as functions that map sets of values to values. Examples of aggregate operations include sum, min and max. The sets of values that are aggregated upon are themselves generated from sets of facts, using constructs such as the groupby construct in Example 1.1. The above view of aggregate operations is adequate so long as one can think of a set of facts for a predicate as being the set of true facts for the predicate, and all other facts for the predicate are false, as is the case with the two-valued semantics that have been traditionally used for logic programs. Recently, several very successful proposals have used three-valued models instead of two valued models. The prime candidate is the well-founded semantics [VRS91]. Other proposals include the three-valued stable model semantics [Prz90] and the valid semantics [BRSS92a]. These models map facts to one of the three values true; false or undefined, whereas two-valued models map facts to one of the two values true or false. When using a three-valued model approach, if we generate sets of values using constructs such as the groupby construct, it is but natural to generate three-valued sets of facts, rather than two-valued sets, and to apply aggregate operations on three-valued sets of facts. Doing so provides an elegant way of extending semantics such as the well-founded semantics and the valid semantics to handle aggregate operations. This is precisely what we do in this paper; to the best of our knowledge, the idea has never been proposed before. Another important use of three-valued sets is as concrete representations of partial information about sets that we are trying to construct and apply aggregate operations on. We may have decided on the truth values of some of the elements, but may have yet to decide on the truth values of others these are mapped to undefined. In fact, it is not sucient to deal just with three-valued sets. Aggregate operations are almost always used on multisets of facts, also known as bags, which are basically sets where each element has a multiplicity. For instance, a common operation is to collect values (e.g. salaries in a department) and sum them. If we treated the collection of values as a set, we would eliminate duplicate values and get a wrong sum. Hence it is standard to create multisets of values and aggregate on them. Consequently we need to deal with three-valued multisets of facts. We now dene three-valued multisets. 3

5 A set on domain D can be viewed as a one-to-one mapping from D to ft; Fg. A (two-valued) multiset on a domain D, is a function from D to the cardinals. For simplicity we use N (i.e., the natural numbers) instead of the cardinals in this paper, which is ne with nite universes. Our technique works correctly even with innite universes if we simply take the symbol N to represent the cardinals; thinking in terms of natural numbers helps the intuition. For each element e in a multiset M, M(e) (i.e., the value that e is mapped to) is called its multiplicity. We stress that the multiplicity of an element could be 0. Given two multisets M 1 and M 2 on domain D, their union, M = M1 [ M2 is dened as the multiset M such that 8m 2 D; M(m) = M1(m) + M2(m) We denote the set of all multisets on domain D as M(D). In the special case that the range of a multiset is f0; 1g, the multiset is actually a set we simply think of F as 0 and T as 1. We dene a three-valued set on a domain D as a one-to-one mapping from D to ft; F; Ug, where U stands for `undened'. Intuitively, elements mapped to T are denitely in the set, elements mapped to F are denitely not in the set, and those mapped to U have their inclusion undened. The question now arises as to how to dene three-valued multisets. A rst approach may be to think of them as mappings to N [ fug. An element mapped to 0 would be absent from the set, and an element mapped to U may or may not be in the set. However, this approach loses valuable information. Suppose we use an aggregate operation that counts the number of children of each person. Since we assumed a three-valued semantics is used, the truth value of some facts may be undened. Suppose we know that the following facts are true: parent(mary; john); parent(beth; john); parent(louise; john) and the following facts are undened parent(rama; john); parent(sita; john) and all other facts are false. How can we represent the information about the number of children of john? Using the above approach to dene three-valued sets, we can at best say that john has at least three children, since each value is mapped to only one integer. But we lose the information that john could have at most ve children. Hence we use the following denition. A three-valued multiset on a domain D is a function from D to N N. Let M be a three-valued multiset, and consider an element e that is mapped to hn t ; n u i in M. We interpret the mapping as saying that e has a multiplicity of at least n t, and may have a multiplicity of n u more. We call hn t ; n u i as the three-valued multiplicity of e. We say that e is true in M with multiplicity n t, (or alternatively, e has an true-multiplicity of n t ) and is undened in M with multiplicity n u (or alternatively, e has an undened-multiplicity of n u ). We also say that e is false in M if both n t and n u are 0. In the above example we would map john to h3; 2i. A three-valued set is a special case of three-valued multisets where the range is fh1; 0i; h0; 1i; h0; 0ig. The rst element of the range corresponds to T, the second to U and the third to F. We use M? (D) to denote the set of all three-valued multisets on D. (The motivation for this notation is that? denotes undened.) We have now dened three-valued multisets and sets. We use the following representation for three-valued multisets on domain D: hm T ; M U ; M F i where M T and M U are (two-valued) multisets on domain D, and M F is a set on domain D. The multisets M T and M U, and the set M F are dened follows. Suppose the three-valued multiplicity of an element e is hn t ; n u i. Then the multiplicity of e in M T must be n t, and the multiplicity of e in M U must be n u. Element e is in M F i both n t and n u are 0. 4

6 In the above representation, M F is redundant since it can be derived from M T and M U. We use it to get a triple notation so as to avoid confusion with the pair notation hs T ; S F i which is often used to represent three-valued sets (where S T is the set of elements mapped to T, and S F is the set of elements mapped to F ). Given (two-valued) multisets M1 and M2 on domain D, we say that M1 M2 (or, M1 is contained in M2) i 8m 2 M1; M1(m) M2(m) In other words, the multiplicity of each element in M1 is less than its multiplicity in M2. Containment of three-valued multisets is more complicated, but the intuitive idea is that if M 1 is contained in M2, then M2 can be obtained from M1 by moving (some copies of) some elements from undened to true, and similarly from undened to false. Formally, containment of three-valued multisets is dened as follows. Given three-valued multisets M1 = hm1 T ; M1 U ; M1 F i and M2 = hm2 T ; M2 U ; M2 F i on domain D, we say that M1 M2 i (1) M1 T M2 T and (2) (M2 T [ M2 U ) (M1 T [ M1 U ). The rst condition above is obvious. The second condition ensures that the sum of the true and undened multiplicities of an element in M 2 has to be the same or less than in M 1. Distributing some (copies of) elements from undened to true and false cannot increase the sum of the true and undened multiplicities for any element. It is very important to note that if M1 M2, then M1 F M2 F ; the result follows from the denition of since an element is false i both its true and undened multiplicities are zero. We say that M2 extends M1 (alternatively, M2 is an extension of M1) if M1 M2. We note that is the multiset extension of the information-theoretic containment operator. Given a two-valued multiset M 1 on the domain D, we say that M1 v ht 1; U1; F 1i if M1 T 1. We say that a fact f1 covers fact f2 if f1 subsumes f2. The denition of covers is extended to sets of facts and to three-valued sets of facts in the natural manner a three-valued set of facts covers another three-valued set of facts if it extends it. 1 We assume familiarity with the notion of the universe of a program. Given a program and a universe for the program, the set of all ground facts that can be created using predicates from the program and values from the universe is a base for the program. Suppose for instance that p is the sole predicate symbol in a program, and p has arity 2. If the universe of the program is fa; bg, then the base of the program is fp(a; a); p(a; b); p(b; a); p(b; b)g. A three-valued interpretation of a program maps each ground atom in the base to one of ft; F; Ug. A threevalued interpretation is thus a three-valued set whose domain is the base. An interpretation ht 1; F 1i is 2-valued precisely when F 1 is the complement of T 1 with respect to the base. For example, hfp(a; a); p(b; b)g; fp(a; b)gi is a three-valued interpretation for the base of the example described above, while hfp(a; a); p(b; b)g; fp(a; b); p(b; a)gi is a two-valued interpretation for the same base. Given a set of facts A, A denotes the complement of A w.r.t. the base. Note that we use three-valued sets of facts rather than multisets or three-valued multisets of facts (as opposed to multisets of values, which we do handle). Our semantics can be generalized to handle multisets of facts, but we do not pursue the idea further in this paper. 3 Syntax and Semantics Preliminaries We denote by U the universe of the program. In the initial part of the paper, we shall assume for concreteness that the Herbrand universe of the program along with the database is used. Unless otherwise specied, the domain of a set and a three-valued set is the universe. 1 We do not consider non-ground facts in this paper, but our results can be generalized to handle non-ground facts in a manner similar to [BRSS92b]. 5

7 3.1 Aggregate Functions Traditionally, aggregate functions have been applied to two-valued multisets. We extend aggregate functions to work on three-valued multisets for the following intuitive reason. In the course of a computation for a program, facts start o with their truth status undened, and this status may change to either true or false (but not vice versa). At some stage in the computation, we may want to partially construct a multiset using the groupby construct (dened formally later) and (possibly) aggregate on it. But we don't know the truth value of all facts, so we represent a partially constructed multiset as a three-valued multiset. For each element in the multiset we have a multiplicity for the `true' part (possibly 0), and a multiplicity for the `unknown' part (also possibly 0). As we continue the computation, facts mapped to unknown may change to true or to false. Correspondingly, if we re-evaluate the groupby literal, in the three-valued multiset that we construct, the multiplicity of the unknown part may decrease as facts change from undened to true or false, and the multiplicity of the true part may increase correspondingly. The false part may also grow correspondingly. By operating on three-valued multisets, aggregate operations can use information about `false' values and `undened' values, in addition to information about `true' values. Consider the special case of three-valued sets. Even though some values may be mapped to undened, an aggregate operation such as min may be able to provide an answer. For instance, given a three-valued set of reals S such that 5 is true (i.e., mapped to T ), and every real less than 5 is false (i.e., mapped to F ), the min operator on S can return an answer of 5 the information on what is false is critical for making this derivation. Any extension of the set can only result in some values greater than 5 that were undened in the set (i.e., mapped to U) being mapped to T or F, which does not aect the result of the min operation. If some values less than 5 (the least true value in S) were undened in S, min must return the result? (undened), since dierent extensions of the set have dierent minimum values. The intuition behind the use of min on the particular example above can be generalized to allow aggregate operations to be used on three-valued multisets. But since we allow aggregate operations to be applied before the entire multiset is constructed, we have to be careful to ensure that the result does not change as the multiset gets extended. We therefore introduce a monotonicity condition. Intuitively, the result v1 of an aggregate operation on a three-valued multiset must either be undened (?), or v1 must also be the result of the aggregation on every extension of the three-valued multiset (recall the denition of extension from Section 2). The formal denition of aggregate functions is as follows: Denition 3.1 A function aggfn is dened to be an aggregate function if: 1. The domain of aggfn is M? (U), and the range is U [ f?g (where? stands for undened). 2. If M1 M2, where M1 and M2 are three-valued multisets with domain U, and aggfn(m1) 6=?, then aggfn(m1) = aggfn(m2). 2 The rst condition states that aggregate functions can be partial functions, i.e., they can map some threevalued multisets to? (i.e., undened). The rst condition also requires that the result of the aggregate function should either be in the universe or should be undened (?). The second condition is the monotonicity condition. Note that the domain of the aggregate function is M? (U), the set of three-valued multisets on U. This set can contain elements that are not in the universe, and hence the aggregate function may be applied on multisets that are not themselves in the universe. Aggregate functions return a single value on a given three-valued multiset. However, there are aggregate operations that `return' multiple values given a multiset. For instance, if we want to select the least three elements of a multiset, rather than the minimum element, we might like to have an aggregate operation that returns each of these independently. Similarly, there are many cases when we want to generate subsets of a given multiset of values. Aggregate functions can be generalized to aggregate predicates for this purpose; details are presented in the full version of the paper. An example of an aggregate predicate is subset(t; S), which is satised by a substitution [T=t; S=s] if s is a two-valued multiset, t is a three-valued multiset, and s v t. For a 6

8 given binding T =t there are many bindings for S that satisfy the predicate. Another example is 3val subset(t; S) which is satised by [T=t; S=s] if s t. 3.2 Grouping Atoms A grouping atom has one of the following forms: 2 groupby(p(t); [X]; S = aggfn < Y >) where X and Y appear in t, and aggfn is an aggregate function, or groupby(p(t); [X]; aggpred(< Y >; S)) where X and Y appear in t, and aggpred is an aggregate predicate. Since any aggregate function can be also represented as an aggregate predicate, the rst form is basically a special case of the second form. The semantics of grouping atoms is discussed later, when we dene satisfaction. A literal is either an atom, a negative atom, or a grouping atom. The denitions of rules and facts follow the standard terminology, except for the following restrictions on aggregation: (G1) A grouping literal may appear only in the body of a rule, never in its head, and (G2) A grouping literal may not be negated. As a special case of restriction G1, a grouping atom can never be a fact. Restriction G2 is essentially to keep the discussion simple, and can be worked around easily. 3.3 Satisfaction and Rule Application We now dene when a literal is satised in a given interpretation, and when a fact `follows from' a rule. The denition for rules without aggregation is the same as in [BRSS92b], although we couch it in a dierent notation. Denition 3.2 Satisfaction : Suppose we are given an interpretation I = ht I ; F I i and a substitution. Given a positive non-grouping literal q i (t i ), we say that q i (t i )[] is satised in I if q i (t i )[] is covered by T I. Similarly, for a negative non-grouping literal :q i (t i ), we say that :q i (t i )[] is satised in I if q i (t i )[] is covered by F I. For a grouping literal of the form groupby(p(t); [X]; S = aggfn < Y >), we say that groupby(p(t); [X]; S = aggfn < Y >)[] is satised in I if there is a three-valued multiset hm T ; M U ; M F i with domain U such that: (GF0) : grounds all variables in fx; S; Y g: (GF1) : S[] = aggfn(hm T ; M U ; M F i). (Note that since is a substitution, S[] 6=?). (GF2) : Let be the set of all ground substitutions on variables in t such that T I covers p(t)[] and agrees with on the variables of X. Then M T = [Y []j 2 ] (where [: : :] is the multiset analog of the set constructor f: : :g.) (GF3) : Let 0 be the set of all ground substitutions on variables in t such that F I covers p(t)[] and agrees with on the variables of X. Then M F = fy []j 2 0 g. (GF4) Let U I be the set of undened facts in ht I ; F I i. (U I = B? T I? F I, where B is base of the program.) Let 00 be the set of all ground substitutions on variables in t such that U I covers p(t)[] and agrees with on the variables of X. Then M U = [Y []j 2 00 ] (where [: : :] is the multiset analog of the set constructor f: : :g.) 2 The syntax is dierent from the syntax that we used in [BRSS92a]. The current syntax corresponds closely to that used in [MPR90]. 7

9 The denition of satisfaction for groupby(p(t); [X]; aggpred(< Y >; S))[] is the same as above, except that GF1 is changed to \(hm T ; M U ; M F i; S[]) is in the extension of aggpred". 2 The denitions of M T and M F are monotonic in that if we have two interpretations M1 and M2 such that M1 M2, then M1 T M2 T, and M1 F M2 F, where Mi T and Mi F are respectively the M T and M F multisets generated by Mi, for i = 1; 2. The denition of M U is anti-monotonic. Note that in the above, M T and M U are multisets while M F is a set. We do not require M T, M F, M U or hm T ; M U ; M F i to be in the universe. The above denition diers from earlier denitions of satisfaction with aggregation since it allows the aggregate operation to work on three-valued multisets rather than just two-valued multisets. The generalization is important, since it is often possible to make derivations using aggregate functions even when some elements are undened. We have now dened the notion of satisfaction for all types of literals. We now dene when a fact `follows from' a rule, given an interpretation. It is important to note that the rest of the semantics is parametrized by the denition of `follows from', and is essentially unchanged from [BRSS92a]. Denition 3.3 Follows from : Consider a rule R : p(t):?q 1 (t 1 ); : : :; q n (t n ) We say that the body of the rule is satised in I = ht I ; F I i by a substitution if for each i; 1 i n, q i (t i )[] is satised in I. We say that a fact p(t)[] follows from I by a substitution, using rule R, if the body of R is satised in I by Discussion An important concern with grouping is that even when M T is nite, M F and M U may be innite, making it very hard to generate them before performing the aggregate operation. However, in typical applications we would never generate these multisets explicitly. The multisets M F and M U need be computed only if required, and are often known implicitly. For example, if we know that a multiset cannot have any negative numbers, we know that all negative numbers are in M F, and a test such as sum greater than 10 succeeds if the sum of elements in M T is greater than 10. For the cheapest path problem, if we use Dijkstra's algorithm ([AHU74]), facts are generated in increasing order of cost, and once a fact with a cost 10 is generated we know that all facts with cost less than 10 are false. Here again, the implicit knowledge of false facts lets us make a derivation using the minimum of the current set of true facts. We discuss the idea in more detail in Example 4.1. The denition of `follows from' has an important property, that we call derivation monotonicity if a derivation `follows from' a three-valued interpretation I1, it also `follows from' every interpretation I2 that extends I1. Intuitively, derivations using grouping literals satisfy this property because of the monotonicity requirement on aggregate functions. More formally, we have the following proposition. Proposition 3.1 If a derivation (using a rule R and substitution ) `follows from' an interpretation ht 1; F 1i, and there is an interpretation ht 2; F 2i such that T 2 T 1 and F 2 F 1, then the derivation also `follows from' ht 2; F 2i using R and. 2 As noted in [BRSS92b], this is the only property of `follows from' that is used in the denitions and proofs related to the well-founded semantics and the valid semantics (described later in this paper). This makes the semantics easy to understand as well as easy to extend to allow new ways of making deductions. 4 Computations In this section we dene the notion of `bottom-up' computations, using the notion of `follows from' dened earlier. Informally, a computation is a sequence of derivations. We use the term computation to refer to any such sequence. However, to dene the semantics of a program we want to consider only computations where 8

10 each step follows, in some sense, from earlier ones. Each derivation is made on the basis of some set of facts being true and some set of facts being false. It makes intuitive sense to choose as the set of true facts the facts in the given database along with the facts derived earlier in the computation. But how about the set of false facts? We have no program rules that derive such false facts program rules derive only true facts. Therefore we use some form of default rule to decide which facts are false. There are several possible default rules. One choice leads to the well-founded semantics [VRS91], and another choice leads to the valid semantics [BRSS92a]. Both choices were discussed in [BRSS92a] for the case of programs without aggregation. We now extend the discussion to include aggregation. The basic denitions of the semantics are very similar to those in [BRSS92a] since they were already parametrized by the deduction rules that are used to derive facts. In Section 4.1 we discuss the default rule that leads to the aggregate-valid semantics. In Section 4.2 we discuss the default rule that leads to the aggregate-well-founded semantics. To dene computations of a program P, we consider transnite sequences of triples of the form (R; p(s); ), where R is a rule of the program, p(s) is an instance of R's head, and is a substitution on the variables of R. It is important to not restrict ourselves to nite computations, as discussed in [BRSS92b], although in practice we will only be able to completely evaluate programs that have nite computations. However, in the case of programs with nite universes we can restrict ourselves to nite sequences. A computation C is a mapping from all ordinals that are smaller than some ordinal to the set of such triples. The ordinal is the length of the computation. We call each triple in C a step, and refer to R as the rule of the step, and to p(s) as the fact of the step. We also sometimes refer to the derivation made in the step as a ring of rule R. Given a computation C, and a set of facts T, for each ordinal length(c) we dene M (C; T ) (the set of facts associated with ) inductively as follows: M 0 (C; T ) is dened to be T, M +1 (C; T ) is dened by adding to M (C; T ) the fact derived by step, and for limit ordinals we take the union of facts derived before. A more formal denition is presented in the full version of the paper. If C is a computation of length from a set of facts T, then the set associated with in C is called the result of C on T, and we denote it by M(C; T ). A special case of the denition above is when T is the given database D. Then we have computations from D. We usually omit D, talk about computations rather than computations from a set of facts, and write M(C), rather than M(C; D). 4.1 Valid Computations We now examine the default rule for deriving false facts that leads to the aggregate-valid semantics. We rst dene `lenient extensions' of computations, which can assume a fact to be false if it has not been derived earlier, even if it happens to be derived later. Thus, lenient extensions impose few restrictions on when a fact can be assumed to be false, and there are many possible lenient extensions of a computation. We then dene `valid computations' as computations where a fact is deduced to be false at some step only if there is no lenient extension of the computation up to that step that can derive the fact. The default rule is quite intuitive, and we show that this default rule denes a unique semantics for the program. We also provide examples where the semantics provides intuitive results. A computation C is a lenient computation of program P on a set of facts T if it satises the following condition: (CC1) : If the 'th triple is (R; p(s); ), then p(s) `follows from' hm (C; T ); M (C; T )i by R, using substitution (where M (C; T ) denotes the complement of M (C; T ) with respect to the base of the program). Condition CC1 embodies the requirement that each triple is indeed a step where a rule is used to derive a fact, assuming that each fact that does not unify with any fact derived so far is false. The adjective `lenient' refers to the fact that in such a computation we may use a fact negatively, yet infer it in a subsequent step. Thus, the constraint in the denition of the term `lenient computation' is rather weak; lenient extensions are 9

11 used to provide upper bounds on what can possibly be derived. Notice that each step in a lenient computation depends on the set of facts generated earlier, but not on the exact sequence of earlier derivations. Hence a lenient computation from a set of facts is equivalent to a lenient extension of a computation that generates that set of facts (database facts are assumed to be generated already). Denition 4.1 If C 1 is a lenient computation, and C 2 is a lenient computation from the set of facts M(C 1 ), then we say that C 2 is a lenient extension of C 1 ; we also say that M(C 2 ) is a lenient extension of M(C 1 ). 2 Denition 4.2 F v (T ) : Let T be a set of facts. We associate with it a set of facts, called the set of facts assumed false on the basis of T, denoted F v (T ), dened as follows. 3 A fact p(s) is in F v (T ) i there is no fact that is in T, or is derived in a lenient computation from T, that unies with p(s). 2 The intuition above is that we can assume some facts to be false by default if there is no `reasonable' way of deriving the facts. The interpretation given to `derived in any reasonable way' is `derived in a lenient computation'. It follows from the denition that T \ F v (T ) = ;. Based on the denition of what can be assumed false by default, we can now dene aggregate-valid computations. Denition 4.3 Aggregate-Valid Computations : Let C be a lenient computation (from a database D), and for each step : (R; p(s); ), let C be the prex of C up to (but not including) step. We say that derivation step is aggregate-valid if p(s) follows from hm(c ); F v (M(C ))i using R and. We say that C is an aggregate-valid computation if all the steps in it are aggregate-valid. 2 The denition of aggregate-valid computations is the same as the denition of valid computations in [BRSS92a], except for the fact that an extended notion of `follows from' is being used to deal with aggregation. Example 4.1 Consider the following program used to dene cheapest paths: R1 : cost(x; Y; C) :? edge(x; Y; C): R2 : cost(x; Y; C) :? edge(x; Z; C1); mincost(z; Y; C2); C = C1 + C2: R3 : mincost(x; Y; C) :? groupby(cost(x; Y; C1); [X; Y ]; C = min < C1 >): Also suppose we have the following facts. edge(a; b; 2): edge(b; c; 3): edge(a; c; 6): edge(c; a; 1): To keep the description of computations concise, we omit rule names and substitutions whenever they can be inferred from the context. The following computation uses only rule R1, and it is trivial to see that it is an aggregate-valid computation: C1 : cost(a; b; 2); cost(b; c; 3); cost(a; c; 6); cost(c; a; 1): However, C1 is not `complete' it can be extended. The following are some lenient extensions of C1: L1 : cost(a; b; 2); cost(b; c; 3); cost(a; c; 6); cost(c; a; 1); mincost(a; b; 2): L2 : cost(a; b; 2); cost(b; c; 3); cost(a; c; 6); cost(c; a; 1); mincost(a; c; 6): L3 : cost(a; b; 2); cost(b; c; 3); cost(a; c; 6); cost(c; a; 1); mincost(b; c; 3); cost(a; c; 5): These lenient extensions are obtained by computing min (in rule R3) using the current set of cost facts, assuming every fact that is not currently true is false. (There is no benet to assuming fewer facts to be false doing so does not result in any more derivations, due to the property of derivation monotonicity.) There is a lenient extension (L3) that derives a fact cost(a; c; 5); hence cost(a; c; 5) is not in the F v set, and is undened after computation C1. The step in L2 that computes mincost(a; c; 6) is not aggregate-valid since it does not follow if cost(a; c; 5) is undened. However, the derivation of mincost(a; b; 2) in L1 is agggregate-valid for the following reason. No lenient extension of C1 can derive a fact cost(a; b; C) such that C < 2, and hence all such facts are in the F v set. Given that cost(a; b; 2) is in the T v set, and all the above facts are in the F v set, the 3 F v stands for False under the aggregate-valid semantics. 10

12 min aggregate operation can deduce that the minimum value for the group [a; b] is 2. Since all its steps are aggregate-valid, L1 is an aggregate-valid computation. We can extend L1 to get the following aggregate-valid computation. C2:cost(a; b; 2); cost(b; c; 3); cost(a; c; 6); cost(c; a; 1); mincost(a; b; 2); mincost(b; c; 3); mincost(c; a; 1); cost(a; c; 5); cost(c; b; 3); cost(b; a; 4); mincost(a; c; 5); mincost(c; b; 3); mincost(b; a; 4); cost(a; a; 6); cost(a; a; 7); cost(b; b; 6); cost(c; c; 6); mincost(a; a; 6); mincost(b; b; 6); mincost(c; c; 6): The intuition is as follows. Steps that derive cost facts are trivially valid steps since they don't use aggregation, and make use of database facts and facts derived earlier. Steps that generate mincost facts are valid since no lenient extension can derive a cost fact for the same pair of nodes, but with lesser cost. Derivations of mincost facts as above are not valid by the denition in [BRSS92a], since there are cycles, and we have not determined the truth value of all required cost facts when we make a mincost derivation. The aggregate-valid semantics makes use of the set of false facts, and can make an aggregate derivation even when some facts in the set are undened. 2 In the above example, we have not specied in general how exactly we deduce eciently that no cost fact with lesser cost can be derived; the denition in terms of lenient extensions appears to require constructing every lenient extension and checking if a derivation follows after each of them. For special classes of programs such as the various forms of stratication, and Cost-Monotonic programs [GGZ91] (which includes the cheapest path program), it is possible to deduce that a derivation is aggregate-valid, without computing all lenient extensions (Section 7). Another option is to compute a superset of all facts that can be computed using lenient extensions from a computation; for example, using aggregate-well-founded extensions (see Section 4.2) Aggregate-Valid Semantics We present some properties of aggregate-valid computations, and then dene the semantics of programs based on the facts computed by `complete' aggregate-valid computations. We shall use the notation F v (C), where C is a computation, to denote F v (M(C)), and T v (C) to denote M(C). Obviously, the set of facts in T v for any lenient computation (not necessarily aggregate-valid) grows monotonically as more steps are performed. It turns out this is also true for the set of facts assumed false, and we can prove the following: If C 1 is a prex of a lenient computation C, then F v (C 1 ) F v (C). We can also show that if C 1 ; C 2 are aggregate-valid computations from D, then their concatenation in any order is also an aggregate-valid computation from D. Denition 4.4 Complete aggregate-valid computation : We say that an aggregate-valid computation is complete if in each of its aggregate-valid extensions the fact derived by every step is covered by the set of facts that have already been derived in the aggregate-valid computation. 2 We can show that all complete aggregate-valid computations C of a program on a given database have the same T v (C) sets, and the same F v (C) sets. The fact that the concatenation of two aggregate-valid computations is an aggregate-valid computation is important to show the above result. We can also show that for each program P, and each database D, there exists a complete aggregate-valid computation of P on D. Given these results, we dene a semantics for programs, as follows. Denition 4.5 Aggregate-Valid Semantics : Assume we have a program, a database, and a given generalized universe. Let T C denote the set of facts computed in any complete aggregate-valid computation C of the program, and let F C = F v (C) (i.e., the set of facts that can be assumed false at the end of the computation). We call the pair ht C ; F C i the aggregate-valid semantics of the program (on the given database, in the given universe). 2 Theorem 4.1 The aggregate-valid semantics of a program always exists, and is unique. 2 11

13 Example 4.2 We continue with Example 4.1. Consider the valid computation C2 shown in that example. There is no extension of this computation that generates a new fact for every pair of nodes (x; y) we have derived a mincost fact, and no more mincost facts can be derived. All cost facts that can be derived from the mincost facts have also been derived. Hence C2 is a complete aggregate-valid computation. The complete aggregate-valid computation C2 denes the set of true facts in the aggregate-valid semantics. The set of false facts in general is the set of facts that do not unify with any fact derived either in C2 or in any lenient extension of C2. In this example, there is no lenient extension of C2 that derives any new fact even the grouping rule cannot deduce any more facts, regardless of any assumptions it makes about facts being false. Hence the set of false facts in this case is the complement of the set of true facts. As a result, the semantics is two valued, although in general it can be three valued (the facts that do not unify with any fact in C2 but unify with a fact derived in a lenient extension of C2 would be undened). 2 We have the following result. Theorem 4.2 For programs that do not use groupby, the valid semantics and the aggregate-valid semantics coincide Aggregate-Well-Founded Semantics The well-founded semantics diers from the valid semantics in the default rule that is used, as noted in [BRSS92a]. In this section we dene the default rule using WF-lenient computations instead of lenient computations, and use the default rule to dene aggregate-well-founded computations. Denition 4.6 A computation S is a WF-lenient computation from a set of facts T, if in every step = (R; p(s); ) of S, p(s) follows from hm (S; T ); Ti by R using substitution (where T is the complement of T with respect to the base of the program). The set of facts that can be assumed false w.r.t. to the WF semantics given T (denoted F w (T )), is dened as follows: a fact p(s) is in F w (T ) i there is no fact that is in T, or is derived in a WF-lenient computation from T, that unies with it. 2 A very important point to note in the above denition of WF-lenient extensions, is that M (S; T ) and T may not be disjoint, that is, a fact may be in both sets this is forced on us by the denition of the well-founded model [VRS91]. Thus hm (S; T ); Ti is not an interpretation by our earlier denition. Furthermore, as a result of the above, in the denition of satisfaction for grouping literals (Denition 3.2) M F may contain elements whose multiplicity in either M F or M U is non-zero. Therefore hm T ; M U ; M F i does not represent a three-valued multiset. The denitions of aggregate functions, aggregate predicates and of satisfaction need to be generalized to use extended three-valued multisets that allow an element to be in M F while having non-zero true-multiplicity or undened-multiplicity. We now discuss how to extend the earlier denitions. Extending the denition of aggregate functions (Denition 3.1) requires only extending the denition of M? (U). We dene the three-valued reduction of ht 1; U1; F 1i as ht 1; U1; F 0 i, where F 0 is the set of all elements f in the domain such that T 1(f) = U1(f) = 0. The result of aggregate functions such as min, max and others we discussed in this paper, on an extended three-valued set S are dened to be the results of the aggregate functions on the three-valued reduction of S. Thus the extra false facts are ignored. The denition of aggregate predicates can be extended similarly. The denition of satisfaction can be used unchanged, with the word interpretation taken to mean the extended denition of interpretations. While we could have presented the extended denitions in earlier sections, we postponed the task so as to avoid confusing the reader. The problems caused by the denition of well-founded semantics are intrinsic, and not due to our formulation, and we believe they indicate that the denition is unintuitive in some ways. We dene aggregate-well-founded computations exactly like we dene aggregate-valid computations, except that we use F w (C) instead of F v (C). That is, in an aggregate-well-founded computation, at any step we can 12