Queries with Order-by Clauses and Aggregates on Factorised Relational Data

Queries with Order-by Clauses and Aggregates on Factorised Relational Data Tomáš Kočiský Magdalen College University of Oxford A fourth year project report submitted for the degree of Masters of Mathematics and Computer Science May 21, 2012

Acknowledgements I am truly indebted and thankful to my supervisor Dan Olteanu for his expert guidance and inspiring discussions. I would also like to thank Jakub Zavodny for participating in our discussions, and Nurzhan Bakibayev for his implementation of the FDB that I extended. Finally, I would like to thank my family for constant support and encouragement.

Abstract Factorised representation form a succinct encoding of relational data, whose size can be exponentially smaller than equivalent representation. The nesting structure of the factorised representations is described by factorisation trees. For a query, only valid factorisation trees, characterised previously, can describe factorised representations over any database. This thesis aims to extend the theory of optimisation and evaluation of conjunctive queries with an order-by clause, and a group-by clause with an aggregate. For queries with an order-by clause, I present a characterisation of factorisation trees, such that factorised representations with the nesting structure of these factorisation trees allow enumeration in sorted order with constant delay and constant precomputation time. Moreover, I present a polynomial-time optimisation heuristic for finding one of the characterised factorisation trees in the space of all valid factorisation trees for a given query. Optimisation with this heuristic was experimentally evaluated and produces results very close to the optimal solution. For queries with group-by and an aggregate, I present a characterisation of all factorisation trees whose nesting structure can represent the result of a query over any database.

Contents 1 Introduction 1 1.1 Motivation and Challenges............................ 3 1.2 Contributions................................... 5 1.3 Outline...................................... 6 2 Prerequisites 7 2.1 Factorised Representations and Trees...................... 7 2.2 Dependency Sets and Path Condition..................... 9 2.2.1 Characterisation of Valid F-trees.................... 9 2.2.2 Projections and Their Effect on Dependency Sets........... 10 2.3 Size of F-representations............................. 10 2.4 Operations on F-trees and Normalised F-trees................. 11 2.4.1 Query evaluation............................. 11 2.4.2 The Push-Up Operation......................... 12 2.4.3 Normalised F-trees............................ 12 2.4.4 The Swap Operation........................... 13 2.4.5 The Merge Operation.......................... 13 2.5 Constant Delay and Constant Precomputation Time Enumeration..... 14 3 Queries with Order-by Clause 15 3.1 Parse Tree and Enumeration.......................... 15 3.2 The F-tree Characterisation........................... 17 3.3 Existence of a Valid F-tree............................ 19 3.4 Query Optimisations............................... 20 3.4.1 Full Search: The Brute Force Way................... 20 3.4.2 Greedy-Heuristic Search: The Simple But Effective Way....... 20 3.4.3 Distance-Measure-Heuristic Search: The Failed Attempt....... 21 3.5 Summary..................................... 21 i

4 Implementation of Order-by 23 4.1 Existing FDB Implementation.......................... 23 4.2 Improved Implementation............................ 23 4.3 Evaluation of Order-by Queries......................... 24 5 Queries with Group-by Clause and Aggregates 26 5.1 Accessing the Groups of Tuples......................... 26 5.2 Computation of Aggregates........................... 27 5.3 Dependency Sets of the Aggregate Attribute.................. 27 5.3.1 Overestimating.............................. 27 5.3.2 The Characterisation........................... 28 5.4 Step by Step Aggregate Evaluation....................... 30 5.5 Summary..................................... 30 6 Experimental Evaluation 31 6.1 Experiment One: Evaluation Plan Finding................... 31 6.1.1 Input Data and Parameters....................... 31 6.1.2 Setup................................... 32 6.1.3 Results.................................. 32 6.1.3.1 Plan Cost............................ 32 6.1.3.2 Search Time and Search Space Size............. 33 6.1.4 Conclusion................................ 37 6.2 Experiment Two: Query Evaluation...................... 37 6.2.1 Input Data and Parameters....................... 37 6.2.2 Setup................................... 38 6.2.3 Results.................................. 38 7 Conclusions 40 A Proofs 42 B Computation of Aggregates 45 Bibliography 46 ii

Chapter 1 Introduction Relational databases represent the most commonly used data management paradigm for structured information. It can be found behind major websites, and a lot of systems that need to store, process, and retrieve information. The demand on these systems is often high and, therefore, common requirements are for them to be fast and space efficient. Databases often store data in relations as a list of tuples. However, it is likely there is more hidden structure in the relations, and especially in relations representing results of queries and materialised views. Therefore, a natural question to ask is: can we exploit this structure to obtain more succinct representations of the data? And thereby, also work with it faster? One answer to these questions, that we pursue in this thesis, is that the recently introduced representation system of factorised representations of relations can be both more succinct than relations they encode and allow for faster processing [4]. In particular, we show how this framework of factorised representations can be extended with two key features that are ubiquitous: sorting and aggregation as present in SQL ORDER-BY and GROUP-BY clauses. This is a key contribution since many real-world queries need sorting and aggregation. For instance, all 22 queries in the TPC-H benchmark [1] use at least one of these two features. In the sequel, we exemplify factorised representations and their nesting schemas. Factorised representations (f-representations) can be thought of as an algebraic factorisation of a relation, whereby relations are unions of tuples and each tuple is a product of unary relations representing a single value. Classes of f-representations that share the same nesting structure are defined by so-called factorisation trees (f-trees) undirected rooted forests of trees. Example 1.1. In Figure 1.1 we can see the relation P and one of its equivalent f-representations. For this particular f-representation there is an f-tree; not all f-representations have an f-tree, but converse is true. The f-tree describes the nesting structure of the f-representation starting from the root A, first we have union over values of A, and each of them 1

A B 1 1 1 3 2 1 2 4 (a) Relation P A : 1 ( B : 1 B : 3 ) A : 2 ( B : 1 B : 4 ) (b) f-representation of P A B (c) f-tree for the f-representation Figure 1.1: Example of an f-representation with its f-tree for a given relation. allows values from the subtrees, in this case only from attribute B (see Example 1.2 for more details). There is a precise and complete characterisation of all f-trees for a given database schema and a given conjunctive query, for which there is an f-representation that can represent the result of the query over the database schema for any data contained in the relations [4]. Also an asymptotically tight bound on the size of the f-representation is given that can be inferred from the f-tree. Example 1.2. In Figure 1.2 we have relations P, R, with attributes {A,B P }, {B R,C}. Let s consider the query Q = σ BP =B R (P R). Figure 1.2 shows two possible f-trees T and U describing possible nesting structures of the factorised representation of the query result and also one such representation over the f-tree T. I have omitted the multiplication signs for clarity. Notice that the f-tree T describes the nesting structure of the f-representation first we have a union over values of the equated attributes B P,B R and for each of those there is a product of f-representations of subtrees for these attributes, in particular, product of possible values of A and C for the corresponding B P,B R. The f-tree U (Figure 1.2e) is not an f-tree that can represent result of query Q for all possible data, because B P,B R and C don t factorise for each value of A in general, and this example is an example of that. Given relations and a query, we want to evaluate the query. We can represent the join of the relations using an f-representation with its f-tree, and find a sequence of operations that manipulate the two, to find the result of the query with equalities [2]. These operations, for example, are swapping of two nodes in the f-tree, or merging two nodes (equating the corresponding attributes), with the corresponding adjustment to the f-representation. This thesis further explores operations on these f-representations, and shows how to do order-by and group-by clauses on them. 2

A B P 1 1 1 3 2 1 2 4 (a) Relation P B R C 1 1 3 2 3 3 4 4 (b) Relation R B P,B R B : 1 ( A : 1 A : 2 ) C : 1 B : 3 A : 1 ( C : 2 C : 3 ) B : 4 A : 2 C : 4 (c) f-representation of Q(P, R) A A C B P,B R C (d) good f-tree T (e) bad f-tree U Figure 1.2: Example of an f-representation with good and bad f-trees for query Q. 1.1 Motivation and Challenges After joining relations and selecting from the result, we are often left with too many tuples to display to the user. Choosing only some of them is what we want to do. But which tuples to keep and which to discard? Ordering by, for example by relevancy, is one of fundamental operations in any information retrieval system, and it allows us to easier view the results we are most interested in which we specify by an order. Group-by clause, on the other hand, provides further information on specific classes of tuples, for example, number of items that share a specific location and a manufacturer, and such. For order-by clauses, I focus on the lexicographic ordering by a sequence of attributes. I will precisely characterise f-trees, for which we can enumerate tuples, in the order specified, with constant delay between tuples. By constant delay we will mean that for enumerating each next tuple we want to spend at most time proportional to the size of a tuple, i.e. O( S ). Example 1.3. If we consider the f-representation in Example 1.2, enumerating tuples ordered by B is straightforward, because the elements of the union have the singletons of B at the beginning, which corresponds to B being at the root of the f-tree, and moreover, they are already ordered in ascending order, which will be the case for each list of values of an attribute in f-representations. However, to order by C, we need to first find the smallest value of C, and that is only possible by looking at three lists of ordered values of C, each corresponding to a value of B, i.e. a valuation of ancestors of C. This suggest that not all f-trees are convenient, or even admit the possibility, for ordering by any sequence of attributes with constant-delay enumeration and constant precomputation time, i.e. to enumerate the first tuple it take the same time as the rest. I will later show 3

that enumeration with constant delay and constant precomputation time of tuples ordered by C is not possible in this case in the example. For group-by clauses, we want, for each valuation of group-by attributes, to apply an aggregate function to the remaining attributes, or project them away. To do this effectively, we need an access to the sets of tuples corresponding to each valuation of group-by attributes, and I will show, for which f-trees we can access these tuples, again, with constant delay and constant precomputation time. And I will again precisely characterise f-trees that describe f-representation that can represent the result of a query with a group-by clause. As we have seen in Example 1.2, not all f-trees could be used as a structure to represent results of the query in the example attributes B and C were dependent in a way, by coming from the same relation, which can t be factorised in general, and so they needed to be on the same root-to-leaf path. This is what is restricting the possible f-trees. B A C E F (a) f-tree T D A : 1 (( B : 1 ( E : 1 E : 2 )) ( C : 1 F : 2 ) D : 2 ( B : 2 ( E : 4 E : 5 E : 6 )) ( C : 3 ( F : 2 F : 3 )) D : 2 ) A : 2 (( B : 1 E : 5 ) ( C : 1 ( F : 1 F : 2 )) D : 3 ) (b) an f-representation over T, only some of the times operators shown A : 1 (( B : 1 ( CNT : 2 )) ( C : 1 F : 2 ) D : 2 ( B : 2 ( CNT : 3 )) ( C : 3 ) D : 2 ) A : 2 (( B : 1 CNT : 1 ) ( C : 1 ) D : 3 ) (c) the f-representation after count(e) as the CN T attribute Figure 1.3: The aggregate value in the f-representation. {{A,B,E},{A,C,F},{A,D}}.) (Database schema is Example 1.4. Consider the f-tree T and the f-representation over it from Figure 1.3 for database schema with relations R 1 = {A,B,E}, R 2 = {A,C,F}, R 3 = {A,D}. Let us explore two queries on these relations. First consider the query Q 1 = σ (R 1 R 2 R 3 );group-by = A,B,C,D;count(E). Similarly to Example 1.3 where enumerating tuples in order of a root node attribute was fast, because we did not have to merge lists, we can here access tuples for the group-by attributes easily, because they are all above the non-group-by attributes. And to evaluate the aggregate, for each valuation of A,B,C,D (or just A,B is enough) we count values of E, and we discard values for F. The result of the aggregate can be placed in the f-representation instead of union of values of E (Figure 1.3c), which corresponds to placing the new attribute at the place of E in T. And this works for any input database. 4

However, if we wanted to count(e, F), which is the same as count( ), for the same group-by clause, it is unclear where the resulting number of the count aggregate should be placed in the f-representation. The Chapter 5 explores this case. By having characterised the f-trees we are interested in, we can search for an f-tree which will have the asymptotically smallest f-representation, as defined by the function s(t) [4], which can be inferred from an f-tree. This s(t) is a rational number, a solution to a linear program based on the f-tree. It is related to the fractional edge cover number of a hypergraf of the query. Further, we can search for an optimisation plan path of f-trees where edges represent operations, like swapping, that, for example, minimises the maximal s(t) over the path. And then, we can compare execution time of this plan for f-representations with execution time of the same query with an engine that uses the flat representation of the database. Related Work There has been no previous work on enumeration in sorted order or aggregates on the factorised representations. There has been research into enumeration of flat relations in sorted order. For example, for acyclic conjunctive queries, there are characterised several orders, including the lexicographic, for which it is possible to do enumeration in sorted order with polynomial delay in terms of query and the relation size [3]. In comparison, with the factorised representation of size n we can do 1 this with precomputation time O(nlogn) and then constant delay and, moreover, the n can be exponentially smaller that the relation size in the above. So the factorised representation has an advantage to, basically, do the sorting only on a smaller data set. 1.2 Contributions The contributions of this work are as follows: I will precisely characterise the class of f-trees whose f-representation admits enumeration in the lexicographical order for a given sequence of order-by attributes, schema, and any database over this schema, with constant delay and constant precomputation time. Given a conjunctive query with an order-by clause, the space of all valid f-trees that canrepresenttheresultofthequeryisingeneral exponential andfindinganf-treewith minimal factorised representation that allows constant-delay enumeration is a hard 1 the following complexity is for transforming the input representation into a form enumerable with constant delay, which I will characterise 5

problem. I will therefore propose a polynomial-time greedy heuristic, that explores part of this space, yet can still find almost optimal f-trees. We experimentally verify the effectiveness and efficiency of this heuristic and compare it to the full-space search. I will also compare the actual optimisation plan execution time, i.e. execution time of the path found in the space starting from a given f-tree, on an f-representation to the execution time of the query on a database with flat representation. I have implemented the above two kinds of search and enumeration in sorted order on top of an existing implementation of f-representation and flat representation database systems. For queries with group-by clause, I will characterise the f-trees whose f-representation admit access to the classes of tuples to be aggregated with constant delay between tuples and constant precomputation time. I will precisely characterise all the f-trees whose f-representation can represent the result of a given query with group-by clause, zero or one aggregate function applied to an attribute, given schema, and any database over this schema. Implications of this allow us to do further operations and plan funding on the result relation, in the same manner as before applying the aggregate, and all this, with having the tree dependency sets restricted as little as possible during the application of the aggregate function, and thereby allowing the most room possible for further optimisation. 1.3 Outline Chapter 2 introduces theory of f-trees and f-representations. Chapter 3 introduces enumerations of f-representations, gives statement and proof of the characterisation on f-trees that admit constant-delay enumeration in sorted order with constant precomputation time, and gives proof of existence of an f-representations over these f-trees for a given query. It introduces several search methods for finding of such f-trees. Chapter 4 describes the implementation of evaluation of queries with order-by, and Chapter 6 presents experiments of the optimisation plan search and query with order-by clause evaluation. Chapter 5 gives characterisation of f-trees that admit access to tuples to be aggregated with constant delay and constant precomputation time. And further, explains how to compute the aggregates. It also gives characterisation of f-trees that can be used to represent results of queries with group-by clause and aggregates. 6

Chapter 2 Prerequisites I am going to present some background needed to understand my thesis. All of the definitions and theorems in this chapter closely follow or cite previous work [4, 2]. 2.1 Factorised Representations and Trees This section gives few basic definition about databases and queries considered in this work, and formal definitions of f-representations and f-trees. Let us first have a look at some basic relation database concepts. Definition 2.1 (Schema, Tuple, Relation, Database, Database Schema). A schema S is a set of named attributes. A tuple is a mapping from a schema S to a domain D. Relation R with aschema S is aset of tuples withthis schema. A database D is a collection of relations. A database schema if a set of schemas of a collection of relations. Definition 2.2 (Conjunctive Query, Equi-join Query). A conjunctive query Q on relations R 1,...,R n with schemas S 1,...,S n is Q = π P (σ ϕ (R 1 R n )) with ϕ conjunction of equalities between attributes of the relations, and P i S i is the projection list. Queries with P = i S i are called equi-join queries. As each order-by and group-by clause we will consider an ordered sequence and a set, respectively, of attributes from these relations such that no two are transitively related by equalities in ϕ. Further, we will also consider possibility of adding one aggregate function of an attribute or star. The factorised representation represents tuples of a relation. It uses unions and products, and tuples are product of singletons after the representation is fully expanded. Or it can be enumerated without expanding, as we will see later. Definition 2.3 (Factorised representation [4]). Let S be a relational schema. A factorised representation, or f-representation for short, over S, is a relational algebra expression of the form 7

, the empty relation over schema S,, the relation consisting of the nullary tuple, if S =, A : a, the unary relation with a single tuple with value a if S = A and a is a value in the domain D, (E), where E is an f-representation over S, E 1 E n, where each E i is an f-representation over S, E 1 E n, where each E i is an f-representation over S i and S is the disjoint union of all S i. A factorisation trees represent a nesting structure of some f-representations. We will use them to reason about manipulation of the corresponding f-representation and to infer bound on size of the f-representation. Definition 2.4 (Factorisation tree [4]). A factorisation tree, or f-tree for short, over a possibly empty schema S is an unordered rooted forest with each node labelled by a nonempty subset of S such that each attribute of S occurs in exactly one node. The following definition defines more formally when an f-tree describes the nesting structure of an f-representation, we define an f-representation over an f-tree. Definition 2.5 (f-representation over f-tree [4]). An f-representation E over a given f-tree T is recursively defined as follows: If T is a forest of trees T 1,...,T k, then E = E 1 E k where each E i is an f-representation over T i. If T is a single tree with a root labelled by {A 1,...,A k } and a non-empty forest U of children, then E = a A 1 : a A k : a E a where each E a is an f-representation over U and the union a is over a collection of distinct values a. If T is a single node labelled by {A 1,...,A k }, then E = a A 1 : a A k : a. If T is empty, then E = or E =. For simplicity, we could write instead of equal class of attributes A 1 : a A k : a only A : a, when from context we understand A to represent the attribute class. 8

Remark 2.1. In f-representation we will enforce that values of each attribute are ordered in ascending order, i.e. in each of the unions in the Definition 2.5, for each valuation of its ancestors. This was so in the previous work as well, and I will make use of this later. 2.2 Dependency Sets and Path Condition The concepts of dependency sets and path condition will be important later for restricting f-trees that can represent any database over a given database schema. 2.2.1 Characterisation of Valid F-trees As we saw in Chapter 1, not all f-trees can represent the result of a given query. We saw that, for example, attributes from the same relation can t be siblings, and it turns out that even more needs to be true: they need to be on the same root-to-leaf path. Given a query with a set of equalities, we can define classes of attributes. Let two attributes belong to the same class if they are transitively equated by the equalities in the query. For a given query, let us consider f-trees where attributes from the same class are at the same node. Then the following proposition describes all the valid f-trees for the query. Proposition 2.2 ([4]). Given an equi-join query Q, the result relation has an f-representation over an f-tree T for any database D iff for each relation in Q its attributes lie along a root-to-leaf path in T. We can think of sets of attributes belonging to the same relation as dependency sets. And the requirement for the attributes to lie along the same root-to-leaf path is called the path condition. NB: It is important to keep in mind for later, that the concept of dependency sets together with the path condition were defined to precisely characterise the valid f-trees for the result relation of a query, based on the query. It is these two concepts that we will try to maintain when reasoning about the f-tree and f-representation. There are a few more trees that can represent the result of the query. In particular, trees such that attributes of some of the attribute classes are split into several nodes. In these cases, for each attribute class C, there needs to be a node labelled with some of the attributes of C that is also an ancestor to all other nodes labelled with attributes of C [4]. However, bringing these split nodes together can only decrease the representation size, and therefore, all the interesting f-trees have been described before, and we will consider only those. 9

2.2.2 Projections and Their Effect on Dependency Sets Most of the time I will be considering equi-join queries, but in chapter on aggregates the fact how projections act on dependency sets will be useful. When attributes are projected away, for each attribute A that is projected away and its attribute class A,A 1,...,A n, all dependency sets containing some of the attributes of the class are merged together, i.e. unioned as sets. Example 2.1. In Example 1.2, after the query is evaluated, we have these dependency sets: {{A,B P },{B Q,C}}. After projecting away attributes A, B P = B Q, {B Q,C} we would get, respectively, dependency sets {{B P },{B Q,C}}, {{A,C}} (with a new f-tree with A,C as parent-child is some order), {{A}}. 2.3 Size of F-representations From an f-tree T, it is possible to derive a rational number s(t ) that together with the database size gives asymptotic bounds on size of the corresponding f-representation [4]. For a given query Q, let s(q) be the smallest s(t ) over all valid f-trees T for the query T. In particular, if the size of the query is constant, we get the bound Θ( D s(q) ) on size of the f-representation, where D is the sum of numbers of tuples of all relations in D. Definition 2.6 (F-representation size [4]). The size E of an f-representation E is the number of singletons plus the number of empty relations in E. The following is an aggregated definition of s(q) given in previous work [4]. Definition 2.7 (Theimportantnumbers(Q)). LetQbeagivenqueryover agivendatabase schema S. Let T be a valid f-tree for the query and let L T be the set of all leafs of the forest T. For each leaf l L T and the root-to-leaf path p l ending at l, consider the database schema S l as a restriction of S by removing the attributes outside p l. Define ρ (p l ) as the solution to the linear program with variables {x Ri } for R i S l : minimise i:r i S l x Ri subject to i:a R i S l x Ri 1 for each attribute A on path p l x Ri 0 for all i : R i S l. The idea of this linear program is that it tries to cover the root-to-leaf path attributes with their relations, possibly assigning fractional number to each, and the cover of each node must be at least one. Note that the solution is a rational number. 10

Define s(t ) := max l L T ρ (p l ), s(q) := min valid T for Q s(t). Example 2.2. Consider the f-tree T A B C D E F G with relation schemas {A,B,C}, {A,D}, {D,E}, {E,F}, {D,G}. We need to consider all root-to-leaf paths: A,B,C; A,D,E,F; A,D,E,G. We need 1, 2, 3, relations, respectively, to cover these paths. Maximum being 3, we get s(t) = 3.0. 2.4 Operations on F-trees and Normalised F-trees This section presents operations on f-trees [2]. They are used to find evaluation plan for the query. 2.4.1 Query evaluation An evaluation plan for a query can be found, which doesn t depend on data in the database. It works like this: First, we construct an f-tree for the input database schema that represents the join of the relations. We are not considering the attribute classes as inferred from the equalities in the query at this point all attributes are at their own nodein the tree, and the equally named attributes from different relation, which we are joining on, are considered to be the same attribute, and so are represented by exactly one node in the tree. This is also illustrated by Example 2.2. We can construct from the database schema the f-tree shown. This is described in more detail in the previous work. The f-tree is one of the several possibilities of the result tree for a query with no equalities, which just joins the relations. Further, if the query includes equalities, we can move nodes in the f-tree with the swap operation, described below, and when two attributes that we want to equate are positioned, for example, as siblings, we can use the merge operation, which combines the two nodes, and restricts the data in the f-representation. 11

A B A A B A B A,B B T A T B T A T A B A T B T A T B T A T B T A T AB T B T A T AB (a) push-up ψ B (b) swap χ A,B (c) merge µ A,B Figure 2.1: Operation on f-trees [2]. Push-up on general f-trees, swap and merge act on and produce normalised f-trees. As we can see, this kind of evaluation has different f-trees and their corresponding f-representations as intermediate steps. For each we can calculate the s(t ) and, thereby, get a bound on the intermediate representation size, which we want to minimise. Let s look at some the operations in more detail. They are explained here a bit less formally, for more formal description see previous work [2]. 2.4.2 The Push-Up Operation The push-up operation basically factors out common subexpression. For a node A and its child B, we can push-up B, as shown in Figure 2.1a, if none of the descendant nodes of B, or B itself, depend on A. If this is the case, we can move B with its children to be a sibling of A. Note that, since none of these moved nodes depended on A, we didn t violate the path condition. In the process we decreased the lengths of root-to-leaf path, and so the s(t ) decreased or remained unchanged. Example 2.3. Consider two relation P,R in Figure 2.2 and f-tree T valid for the query Q = P R, a query with no equalities, with its corresponding f-representation. We can push-upnodec, sinceb doesn tdependin it. Andindeed we can see in thef-representation that we can factor out the attribute C. We can see the result in Figure 2.3. Notice also that the representation size decreased. 2.4.3 Normalised F-trees If we can factor out terms, we should do it. It has no immediate advantages to leave the f-tree in this state. And, moreover, pushing up nodes when possible, can t increase the f-representation size, it can only help. We want the f-trees to be as wide and branched as possible, as opposed to deep, because the former have potential for smaller s(t ). So we define normalised f-trees. 12

A B D 1 1 2 1 2 3 2 1 1 A C 1 2 2 1 C A B D A : 1 ( B : 1 C : 2 D : 2 B : 2 C : 2 D : 3 ) A : 2 B : 1 C : 1 D : 1 (a) Relation P (b) Relation R (c) f-tree T (d) f-representation over T Figure 2.2: Example of two relations, valid f-tree for P R, and the corresponding f-representation. A A : 1 C : 2 ( B : 1 D : 2 C B D B : 2 D : 3 ) A : 2 C : 1 B : 1 D : 1 (a) f-tree T (b) f-representation over T Figure 2.3: f-tree and f-representation after a push-up operation on node C from Figure 2.2. Definition 2.8 (Normalised f-tree [2]). An f-tree T is normalised if no node in T can be pushed up without violating the path constraint. Notice that the f-tree in Figure 2.3a is normalised, because we can t push up any of the nodes. (Each of the nodes depends on its, in this case, immediate descendant.) 2.4.4 The Swap Operation The swap operation takes two nodes of a normalised f-tree and produces a normalised f-tree. It is depicted in Figure 2.1b. It swaps the two nodes and places the subtrees based on dependency on the two swapped nodes the subtrees in T ns depend on nodes ns. It requires a little more thought on why the resulting f-tree is actually normalised. One thing to note is that, although the operation in the figure looks like a rotation in a binary tree, it is not. The subtrees in the figure can be forests, and the partition of children of B into T AB and T B needs to be computed. 2.4.5 The Merge Operation The merge operation acts on two sibling nodes of a normalised f-tree. It basically does a cross product of the two attributes and keeps only the pairs with two same values it equates the attributes. This operation is depicted in Figure 2.1c. The result f-tree is normalised, again. 13

There is another operation, called absorb, used for the same purpose. It acts on a node and a descendant, equates them, and removes the descendant. Moreover, it normalises the f-tree with push-up operations. 2.5 Constant Delay and Constant Precomputation Time Enumeration Here are defined concepts that define time complexity of enumeration. Definition 2.9. Constant delay enumeration is enumeration where to output the next tuple we need time O( S ), where S is size of the database schema of thecurrent tree, i.e. number of attributes of the result relation, i.e. size of the tuple itself. This is the best possible time for enumerating one tuple, since it is its size. And constant precomputation time is O(1). We can alternatively have O( S ), but we can absorb the latter into the computation of the first tuple, so we will consider the former. 14

Chapter 3 Queries with Order-by Clause We already know how to evaluate a conjunctive query using f-representations. We would like to evaluate such a query with an order-by clause. In particular, we would like to get an f-representation of the result such that we can enumerate the tuples of the result quickly in the lexicographic order specified by the query. And more precisely, this chapter gives an f-representation for which we can enumerate the result tuples with constant delay and constant precomputation as defined in Definition 2.9. As order-by clause I take an ordered sequence of attributes. Without loss of generality I assume that no two are transitively related by equalities in the query, for if they are, their values are the same for each tuple, so we can trivially ignore one of them. We want to order by the first attribute first and then the rest. For each we could specify whether we want ascending or descending order, but since these cases are so similar, it is enough to solve this problem for ascending orders of each attribute, as we will see later. We can further assume that we have an intermediate result where the equalities of the query are already enforced, and so we get this problem: The problem: Given an f-representation over an f-tree, and a sequence of attributes, we want to find an equivalent f-representation with its f-tree, such that we can enumerate the tuples in lexicographic order, given by the sequence of attributes, each tuple with time complexity O( S ) and constant precomputation time, i.e. the first takes only O( S ) too. We want to do enumeration of an f-representation with specific ordering of the enumerated tuples. Let us first have a look at how to do it with arbitrary order, and then find what properties we need for the enumeration to have the right order and complexity. 3.1 Parse Tree and Enumeration This section first explains how the f-representation is stored the parse tree, and then how to do enumeration in some order. This has already been done before, and also 15

implemented, but let us recall it [4]. Parse tree: The parse tree is a data structure that holds the natural parse tree of the f-representation (which is associated with a f-tree). More precisely we can imagine it as follows. There are two kinds of nodes: union nodes and product nodes. The product nodes hold also a value of some type and on the root-to-leaf paths they alternate. Let me explain this on an example. Example 3.1. For a f-tree with only a single node, the parse tree would be a union node with children product nodes each holding one value for the attribute of the single node of the f-tree. If the f-tree is single tree, i.e. not a forest, with a root node A, then the parse tree is like the above tree with each product node with a value of attribute A in the parse tree having union node children, one for each children attribute of A in the f-tree. Lastly, if the f-tree is a forest, parse tree has as root a product node with no value and each tree in the forest is then under this product node in the form explained above. Enumeration: First for illustration, imagine that we want to enumerate 5 digit numbers in base 3. We do this by setting each digit at its first value, namely zero, and then we advance the last digit. When we have no more further values for the last digit we advance the second to last digit and set the last digit to the first permitted value, as determined by previous digits, which in this case will always be zero. And we continue in this manner until the first digit can t be advanced anymore. For f-representations it is similar. However, the structure of advancing is not linear, but branches, and it is given by the f-tree. The branching gives us a choice which leaf to advance next. And we will encode this choice by a sequence nodes of the f-tree and call it the priority list. Definition 3.1 (Enumeration attribute priority list). Given an f-tree, let a priority list be a sequence of all nodes of the f-tree, such that for each node n, all of its ancestors appear before n in the list. The list can also be of labels of nodes. It is easy to see, that given a priority list, we can enumerate the f-representation by advancing values of the attributes in reverse order of the list. This enumeration outputs each next tuple in O( S ) time. And, moreover, it describes all such orders. For the above base 3 example, the priority list would be digits starting with the most significant digit and up to the least significant. 16

B A C A : 1 ( B : 1 B : 2 )( C : 1 C : 2 ) A : 2 ( B : 1 B : 2 )( C : 1 C : 2 ) (a) f-tree T A B C 1 1 1 1 1 2 1 2 1 1 2 2 2 1 1 2 1 2 2 2 1 2 2 2 (c) Enumeration for priority list [A, B, C]. (b) f-representation over T A B C 1 1 1 1 2 1 1 1 2 1 2 2 2 1 1 2 2 1 2 1 2 2 2 2 (d) Enumeration for priority list [A, C, B]. Figure 3.1: Example of two enumerations for priority lists for given f-tree and f-representation. Example 3.2. Consider the f-tree and f-representation from Figure 3.1. There are precisely two priority lists for this f-tree: [A, B, C],[A, C, B]. Their corresponding enumerations are shown in Figures 3.1c and 3.1d. Notice that, both enumerations are ordered by A; first enumeration is ordered by A,B and A,B,C; and second enumeration is ordered A,C and A,C,B. Proposition 3.1 (Priority list induces lexicographic order). Given an f-representation over an f-tree, and an attribute priority list for the f-tree, then the enumeration of the f-representation according to this priority list is ordered by any prefix sequence of the priority list in ascending order of each of the attributes. Remark 3.2. To get a descending order for an attribute, instead of ascending, traverse the increasingly sorted lists of values of the attribute in the parse tree in reverse. Proof. See Appendix A 3.2 The F-tree Characterisation This section characterises all the f-trees that admit enumeration of each tuple in O( S ) time with constant precomputation time. Theorem 3.3 (The Characterisation). Let T be an f-tree and U a sequence of attributes. F-representation over T can be enumerated in sorted lexicographic order by U, with constant 17

precomputation time and time O( S ) for each tuple if and only if each attribute X of U is at a node in T which is either a root, or a child of a node labelled with an attribute appearing before X in U. The If Part of The Characterisation Theorem 3.3 Let us first observe how the attributes of a priority list are positioned in an f-tree and, thereby, relate them to position of order-by attributes required in the Theorem 3.3. Lemma 3.4 (Prefix of a priority list in the f-tree). Given an f-tree T and a priority list U for this f-tree, then for any prefix of U, each attribute A of the prefix is either a root in T, or a child of a node labelled with an attribute appearing before A in the prefix. In particular, the first attribute of U is a root in T. Conversely, for any sequence of attributes V such that all X V are either a root, or a child of a node labelled with an attribute appearing before X in V, there exists a priority list with prefix V. Proof. Immediate from the definition of a priority list. Proof of the if part of the Theorem 3.3. We are given an f-tree T and a sequence of attributes U. Assume that each attribute X of U is in T either a root, or a child of a node labelled with an attribute appearing before X in U. Then by Lemma 3.4 there is a priority list V for T that has U as a prefix, and by Proposition 3.1 the enumeration with the priority list V is ordered by U and the enumeration can be done in time required. The Only If Part of The Characterisation Theorem 3.3 This section proves the converse: that if we assume the order and the time complexity of the enumeration, then we need the tree structure of attributes of the order-by sequence, which is described in the theorem. Lemma 3.5 (The first attribute at a root). If we can enumerate an f-representation over f-tree T in sorted lexicographic order by attributes of a non-empty sequence U, with constant precomputation time and delay O( S ) for each tuple, then the first attribute of U labels a root in T. Proof. Idea is that there is no way to avoid merging otherwise. See Appendix A. Proof of the only if part of the Theorem 3.3. The proof is by strong induction on (n,m), n m, n number of nodes of an f-tree and m number of elements in the sequence U. We are assuming here that the enumeration can be done in the order and complexity specified in the theorem, and we want to show the structure of the f-tree. 18

Base case (n, 0): Immediate. Induction step (n,m): Assume the hypothesis is true for all pairs of (s,t), s t, s < n, t < m. By Lemma 3.5 the first element of U is a root in the f-tree forest T. Let U be a sequence of attributes of U, except the attribute A, and let T be the restriction 1 of T without the root node labelled with attribute A. We assumed an f-representation over T, call it F, can be enumerated in the order and the complexity, and so for each value a of A the tuples can be enumerated in such way, because we can select the subtree of a in a parse tree of F in O(1), and so each of the restricted f-representations F a over T can be enumerated in the order and the complexity. So we can apply the induction hypothesis to T and U. So for T and U, by induction hypothesis, every node X of U in T is either a root, or a child of a node labelled by an attribute appearing before X in U. Now in T, A U is a root. The attributes X of U, a subsequence of U, that were children of nodes labelled by an attribute appearing before X in U have this property also in context of T and U. The rest, that were roots in T, are going to be roots in T, or children of the node labelled with A, which comes before all of them in U, since A is the first attribute in U. This completes the proof of Theorem 3.3. 3.3 Existence of a Valid F-tree I have described the f-trees that allow enumeration with constant delay. However, for a given query and database schema, as we have seen in examples previously, not all f-trees can represent the result. In particular, only those that satisfy the path condition. Now, the question is, does one of those f-trees that can represent the result have the structure we require for constant delay enumeration? I show this in two ways. Firstly, by an interesting observation of uniqueness of structure of the top parts of the in-sorted-order-enumerable f-tree by constructing it and it will satisfy the path condition by construction. Secondly, by exhibiting f-tree operations that can transform a given f-tree to one that allows the enumeration, and this in Section 3.4.2. The following proof of the following lemma provides a way to construct all f-tree that admit enumeration in sorted order with constant delay and constant precomputation time. It further shows that the relative placement in the f-tree of attributes of the order-by clause is the same among all these f-trees, i.e. they form connected trees at roots of the trees in the f-tree. 1 remove the node with attribute A and promote its subtrees to be trees in the forest T 19

Lemma 3.6. Given a database schema S and a sequence of attributes U that we want to order by, then there is an f-tree that satisfies the condition for the f-tree to be constant-delay enumerable, and the path condition based on dependency sets from S, and all such f-trees have the same tree structure of attributes of U. Proof. See Appendix A 3.4 Query Optimisations Now we are getting to answering the main problem, stated in the introduction of this chapter given f-representation over an f-tree T, we want to find an equivalent f-representation over some f-tree that can be enumerated with constant delay and precomputation time. It is easy to see that from an f-tree that satisfied the path condition for a database schema, we can get to all other f-trees that satisfy the path condition for the database scheme using only the swap operation. (This is a previous result, but was not presented in detail.) Our aim in this section is to find a sequence of such swap operations. This is obviously hard, because the search space is exponential in number of the attributes. The following sections describe several kinds of search for the f-tree we desire. 3.4.1 Full Search: The Brute Force Way I use the same full search as was proposed in the previous work [2], and also implemented. It uses the Dijkstra s algorithm to search the exponential full space of f-trees, generating the following ones using the swap operation. We need just a different goal condition that describes the f-trees enumerable in sorted order with constant delay. 3.4.2 Greedy-Heuristic Search: The Simple But Effective Way I am proposing a simple, but as we will see in Chapter 6, very effective heuristic how to choose nodes for the next swap operation. In turn, for each node in the order-by attribute sequence, say U, we will keep swapping the node with its parent until it becomes a root, or becomes a child of a previously broughtup node, i.e. a node sooner in the sequence of order-by attributes. This method requires only about O(depth(f-tree) U ) f-trees to be explored, which is considerably less that the exponential full-search space size. 20

3.4.3 Distance-Measure-Heuristic Search: The Failed Attempt Since, by Lemma 3.6, the attributes of the order-by attribute sequence U have a predetermined position in the f-tree for a database schema S, I have tried to define a distance measure between the current f-tree T and the target f-tree as d U,S (T ) = currentdepth(a) targetdepth(a). A U Then the search would consider only paths of f-trees where this distance is non-increasing. This has an advantage compared to the greedy-heuristic search, that it could potentially achieve better results by doing some modifications to other parts of the f-tree, other that just the nodes with attributes of U. In fact, we will see in Example 6.1 later that the greedy-heuristic search will not find an optimal solution in general, so some fixes to other parts of the f-tree are really necessary. Moreover, compared to the full search, this limits the search space, which is an advantage when the space is exponential. However, this distance-measure heuristic does not find a solution in all cases. There are cases, where the distance needs to increase first before it can decrease to 0, i.e. get to the goal. Example 3.3. Consider database schema {{A, B},{B, C},{B, D}} and the f-tree T in Figure 3.2. The figure shows all possible swaps on this tree and illustrates that the distance needs to increase for the goal to be reached. I have considered to augment the search criteria to allow an increase in the distance to the target, but in general, the increase required would be about as large as the size of the attribute order-by sequence, which is disadvantageous. 3.5 Summary In this chapter we have seen how to do enumeration on a parse tree and priority lists that specify the enumeration order to some extent. We have seen the characterisation of all the f-trees, for a given order-by attribute sequence U and a given database schema S, that allow enumeration in sorted order by the sequence with delay O( S ) and constant precomputation time. Moreover, we have seen two existence proof for such f-trees: a construction that shows also the necessary structure of attributes of U in such a tree, and a proof by swapping and bringing nodes to the top of the tree. We have also seen two search methods for finding the desired tree, which we will evaluate in Chapter 6. 21

D B D A D C C A B B (a) f-tree T, d = 2 C (b) after χ A,B, d = 3 A (c) after χ B,C, d = 5 B A C D A B C D (d) after χ B,D, d = 3 (e) target for order-by A,B,C, e.g. from (d) by χ A,B, d = 0 Figure 3.2: All possible swaps on f-tree T, a unique target f-tree, d is the distance measure to the target. (Database schema is {{A,B},{B,C},{B,D}}.) 22

Chapter 4 Implementation of Order-by I have implemented plan finding and enumeration for queries with an order-by clause. There are two search methods implemented, the full search and the search with the greedy heuristic, as described in 3.4.1 and 3.4.2, respectively. And also the enumeration in sorted order on both the factorised and flat representations. Let me first describe the implementation of factorised representation database engine that I was extending. 4.1 Existing FDB Implementation Query-evaluation plan finding and also the evaluation for f-representation was previously implemented. Evaluation on flat representation for the same kind of queries was also implemented before. The implementation is written in C++. It has different classes for f-representation, parse tree, few optimisers that find the evaluation plan, a class that executes the plan on an f-representation and thereby evaluates a query. Also a class for computing the s(t ) is present. The optimiser for the full search uses Dijkstra s algorithm. It uses string representations of f-trees as identifiers for the f-trees, and remembers the cost as a double precision floating point number. The optimiser also records various statistics like the number of goals, size of the search space, and such. The query parsing, search algorithm logic, next f-tree computation, goal f-tree detection, and statistics, are all in the same optimiser class and mostly in the same method. 4.2 Improved Implementation I have cleaned up the implementation somewhat by splitting the optimiser into smaller parts. I have implemented from scratch a separate search class, using also the Dijkstra s algorithm but parametrised heavily. It has a template argument Cost, which is expected to 23