Chapter 18 Strategies for Query Processing We focus this discussion w.r.t RDBMS, however, they are applicable to OODBS. 1
1. Translating SQL Queries into Relational Algebra and Other Operators - SQL is most commonly used query language - Translated into equivalent extended relational algebra expression - It is represented as a query tree data structure and optimized - SQL queries are decomposed into query blocks; basic units; translated into expressions and optimized - A query block contains single select-from-where expression including GROUP BY and HAVING - Nested queries within a query are identified as separate blocks Q1: Retrieve the names of employees who earn a salary that is greater than the highest salary in department 5. SELECT Lname, Fname FROM EMPLOYEE WHERE Salary > (SELECT MAX (Salary) FROM EMPLOYEE WHERE (Dno = 5); The Q1 is a nested query, it is decomposed into two sub queries as follows. 2
Q1a: (inner) SELECT MAX (Salary) FROM EMPLOYEE WHERE (Dno = 5) Q1b: (outer) SELECT Lname, Fname FROM EMPLOYEE WHERE Salary > c (c represents the results of Q1a) Q1a can be translated to: (inner) ᵮ MAX Salary (σ Dno=5 (EMPLOYEE)) (Selection operator) Q1b can be translated to: (outer) π Lname, Fname (σ Salary>c (EMPLOYEE)) (Projection operator) - The query optimizer will choose the execution plan for each query block - The inner block needs to be evaluated only once - The outer block has to iterate for each employee - This type of nested queries are called non-correlated queries 3
4
Additional Operators Semi-join and Anti-join Semi-join is generally used for un-nesting EXISTS, IN and ANY subqueries Consider: EMPLOYEE (Ssn, Bdate, Address, Sex, Salary, Dno) DEPARTMENT (Dnumber, Dname, Dmgrssn, Zipcode) Where a department is located in a specific zip code. Q(SJ): Counting the number of departments, that have employees, who make more than 200000 salary. SELECT COUNT(*) FROM DEPARTMENT D WHERE D.Dnumber IN (SELECT E.Dno FROM EMPLOYEE E WHERE E.Salary > 200000); The nested query is joined by the connector IN operator. 5
To remove the nested query: (SELECT E.Dno FROM EMPLOYEE WHERE E.Salary > 200000) is called as un-nesting. This query gives a set of Dno with employees who make more than 200000. It leads to the following query with an operation called semi-join, shown as S= SELECT COUNT(*) FROM EMPLOYEE E, DEPARTMENT D WHERE D.Dnumber S = E.Dno and E.Salary > 200000; Consider another query: Count the number of employees who do not work in department located in Zipcode 30332. Q(AJ): SELECT COUNT(*) FROM EMPLOYEE WHERE EMPLOYEE.Dno NOT IN (SELECT DEPARTMENT.Dnumber FROM DEPATMENT WHERE Zipcode = 30332); 6
SELECT COUNT(*) FROM EMPLOYEE DEPARTMENT WHERE EMPLOYEE.Dno A = DEPARTMENT AND Zipcode=30332 (Antijoin) A row of T1 is rejected as soon as T1.X finds a match with any value of T2.y. A row of T1 is returned, only if T1.x does not match with any value of T2.y. 2.Algorithms for External Sorting External sorting: sorting for large files or records stored on disk that do not fit entirely in memory Sorting most commonly used in databases. Ex. Whenever order by is used, it sorts results. Join, Union, Intersection, also use sorting. Sorting may be avoided if primary or clustering index is available. External sorting algorithms are used to sort large files on disk. A typical sort used for this is sort-merge algorithm. Requires buffer space in main memory, where sorting is done, which is called DBMS cache that is controlled by DBMS. Sort small sub files called runs, merge the sorted files. Sorting Phase: - Read a piece of file into buffer - Sort using internal sort algorithm - Write back to disk (as temporary disk space) 7
Merging Phase: - The sorted runs are merged into one or more merge phases - Each merge phase can have one or more merge steps - One block is used to keep merged results 8
Available buffer space = n B blocks Number of file blocks b If n B = 5 and b = 1024 Number of initial runs (each of size 5 blocks except the last one) = n R = гb/n B = 1024 / 5 = 205 After sort phase, 205 sorted runs are stored as temporary subfields on disk Degree of merging d M is the number of runs that can be merged in each pass d M = (n B 1) or n R (smaller of) d M = 5-1 = 4 (four way merging) n P = гlog dm (n R ) = number of passes = log 4 (205) = 4 Number of block accesses = (2 * b) + (sort phase, once to read and once to write) (2 * (b * log dm (n R ))) (merge phase) = 2 * 1024 + (2 * (1024 * 4))) = 2048+8192 = 10240 (205 -> 51 + 1 ->13 -> 3 +1 -> 1) 9
10
3.Algorithms for SELECT Operation There are many algorithms for SELECT operation, basically a search operation to locate records to satisfy certain conditions. They depend on the access paths. The following operations are used to illustrate the algorithms: OP1: σ ssn= 123456789 (EMPLOYEE) OP2: σ Dnumber > 5 (DEPARTMENT) OP3: σ Dno=5 (EMPLOYEE) OP4: σ Dno=5 AND Salary > 30000 AND Sex= F (EMPLOYEE) OP5: σ Essn= 123456789 AND Pno=10 (WORKS_ON) OP6: An SQL Query SELECT * FROM EMPLOYEE WHERE Dno IN (3, 27, 49) OP7: An SQL Query SELECT First_name, Lname FROM EMPLOYEE WHERE ((Salary * Commission_pct) + Salary) > 15000; 11
Search Methods A number of search algorithms are possible for selecting records from a file, called file scans. (1) S1 Linear Search; read all related disk blocks into memory buffers and search for required conditions (2) S2 Binary Search: if equality condition on a key attribute; binary search is efficient (file is ordered on this attribute), OP1 (3) S3a Using a primary index; equality comparison and a key attribute; OP1; use primary index to retrieve the record; retrieves a single record (4) S3b Using a hash key: equality condition, use the hash key to retrieve a record; OP1 (5) S4 Using a primary index to retrieve multiple records; if the comparison is > >= < <=, key field with a primary index, Dnumber > 5; OP2; find a record satisfying the equality condition, then retrieve all preceding records (6) S5 Using a clustering field to retrieve multiple records: equality condition on a nonkey; OP3; retrieve all records (7) S6 Using a secondary (B+ index) on an equality comparison; single record if a key, multiple records for non-key attribute; can also be used for range queries; leaf nodes provide the ordering of values to select multiple values in a range SKIP other methods Search methods for conjunctive selection Made up of several conditions connected with AND 12
1. S8 Using an individual index: if an attribute has an access path, one of the methods S2 S6 can be used to retrieve the record; check if the retrieved record satisfies other conditions 2. S9 Using a composite index: if a composite index created for one or more conditions, then use it retrieve the records 3. S10 Using intersection of record pointers: if secondary indexes or access paths are available for more than one field, and the indexes include a record pointer, then each record can be retried and checked for a given condition; the intersection of these records can only be retrieved to check for the conjunctive condition Search methods for disjunctive selection OP4 example Uses OR operator in the Selection. Require to retrieve records that satisfy each condition and make a union of records and eliminate duplicates. If one of the attribute does not have an access path, then we need to use linear search to retrieve records. All the above steps for access paths can be used if they exist. Estimating the selectivity of a condition - System catalog can maintain some statistics on database attributes such as number of distinct values, number of records, frequency of data changes etc - The above information can be used in query optimization 13
4.Implementing the JOIN Operation - one of the most time consuming operation - equijoin or natural join are most commonly used operations - two-way JOIN involves joining two files - multi-way JOIN involves multiple files (number of possible ways to execute grows rapidly) - we discuss two-way JOINS here R A=B S A, B are the join attributes, domain compatible OP6: EMPLOYEE OP7: DEPARTMENT Dnumber=Dno DEPARTMENT Mgrssn=Ssn EMPLOYEE Implementing Joins (1) J1 Nested Loop Join (or nested block join): Default, no need to have access paths; test if this condition is true for each record; t[a] = s[b] (2) J2 Index based nested loop Join: use an access structure to retrieve the matching records; if an index or hash value exists for one of the records, B of the S, retrieve each record t in R s[b] = t[a] 14
(3) J3 Sort merge join: The records of R and S are physically sorted by the value of attributes A and B respectively; both files are scanned concurrently and matched for the same values of A and B (4) J4 partitioned has join (or just hash-join): R and S are partitioned into smaller files; the partitioning of each file is done using the same hash function h on the attributes A of S and B of R (partitioning phase); partitioning phase and probe phase R S. hash some records h(a) h(b).... (smaller file will fit into memory) Match records from S Continue the process. Join Performance OP6: EMPLOYEE Dnumber=Dno DEPARTMENT Assume buffer space is available Nested loop approach 15
Available buffer space = n B blocks Each memory buffer is same size as one disk block DEPARTMENT file consists of r D = 50 records stored in b D = 10 disk blocks EMPLOYEE file consists of e D = 6000 records stored in b E = 2000 disk blocks Outer loop file records should be read to memory (as many as possible) One block for the inner loop and one block to write the result and remaining n b -2 blocks for outer loop Read one block at a time for the inner loop file and use its records to probe (that is search) that outer loop blocks that are currently in memory for matching records; The contents of the result block are appended to the result file on the disk In the nested loop join, it makes a difference which file is chosen for the outer loop and which file is chosen for the inner loop; If EMPLOYEE is chosen for outer loop, each block of employee is read once; entire DEPARTMENT file (each of its blocks) read once for each time we read in n B -2 blocks of the EMPLOYEE file. Total number of blocks accessed for outer loop file = b E Number of times (n B -2) blocks of outer file are loaded into memory = гb E /(n B 2) 16
Total number of blocks accessed for inner loop file = b D * гb E /(n B 2) Total number of read accesses = b E + b D * гb E /(n B 2) = 2000 + 10(2000/5) = 6000 block accesses Total number of read accesses (if DEPARTMENT is in outer loop) = b D + b E * гb D /(n B 2) = 10 + 2000(10/5) = 4010 block accesses b RES block accesses are added to the above result for a total block accesses. JOIN Selection Factor and Performance A fraction of records in one file is joined with other file. It is called a join selection factor w.r.t equijoin OP7: DEPARTMENT Mgrssn=Ssn EMPLOYEE r D = 50 (50 records in the department); 50 records will be joined with 6000 employee records, but 5950 will not be matched. Suppose secondary indexes exist on Mgr_ssn and Ssn. The number of index levels are Xssn = 4 and Xmgr_ssn = 2; 17
Retrieve each EMPLOYEE record and then use the index Xmgr_ssn to find a matching record for the DEPARTMENT entries: The number of accesses = b E + (r E * (X mgr_ssn + 1)) = 2000 + (6000 * 3)) = 20000 block accesses The second option is: retrieve each DEPARTMENT record and then use the index on Ssn of EMPLOYEE to find a matching manager: The number of accesses = b D + (r D * (X ssn + 1)) = 10 + (50 * 5)) = 260 SKIP 18.4.4 and 18.4.5 5.Algorithms for PROJECT and Set Operations <attribute-list> ( R) selects columns in the list of attributes SELECT Salary FROM EMPLOYEE; Produces a list of salary for all employees. It is linear search of all employees. If attribute list does not include a key attribute, then you have duplicate values in the result. If duplicate tuples exist, simply sort and remove the duplicates. Set operations UNION, INTERSECTION, DIFFERENCE, CARTESIAN PRODUCT are expensive Sort merge can be used for set operations. Cartesian product should be avoided and use JOIN instead. SKIP 18.5.1 18
SKIP 18.6 7.Combining Operations Using Pipelining A query gets translated into a relational algebraic expression with many operations. A single operation at a time requires temporary files to hold results; these temporary files can be passed to next operation; this is called materialized evaluation. To avoid large temporary files, a combination of operations used to improve query execution. A JOIN can be combined with select operations on two input files and projection on one output file. It is common to generate query execution code directly to implement multiple operations. The output of one operation is fed as input of another operation; a series of pipeline to perform combined operations. Iterator for implementing physical operations - If the operator is implemented in such a way that it outputs one tuple at a time, then it is regarded as a iterator (nested join) - The iterator interfaces consists of the following methods: o Open() o GetNext() o Close() - Iterator concept can also be applied to access methods (B trees) 8.Parallel Algorithms for Query Processing 19
Parallel database architectures (distributed databases, big data and NOSQL) (1) Shared Memory Architectures a. Interconnection network, common memory b. Each processor has access to entire memory c. Contention for memory and not scalable, interference (2) Shared Disk Architectures a. Every processor has its own memory b. Every processor has access all disks through interconnection network c. Every processor may not have disk of its own (SANs, and NAS) (3) Shared-nothing Architectures a. Commonly used b. Its own memory and disk c. Inexpensive to build (racks of storage with processors) d. Allocating more processors and disks can achieve linear speedup (a) Operator-level Parallelism - Partition data across disks - Horizontal partitioning; distribution of tuples across disks based on some method; ith tuple to (i mod n); range partitioning (10 disks, 10 ranges for storing employee records); hash partitioning ith record assigned to h(i) disk - Sorting: sort on age and partition data - Selection based on some condition; range portioning - Projection and duplicate elimination: sorting tuples and discarding; sorting can use partitioning strategies 20
- JOIN: split the relations to be joined; join is divided into multiple smaller joins to perform in parallel on n processors and take a union of the result; Partitions must be non-overlapping on the join key (b) Intraquery Parallelism - A query execution graph can be modeled as a graph of operations - Provide appropriate data in the partition to execute in parallel - If operations do not depend upon one another, they can be run in parallel - If one of the operations can be generated tuple-by-tuple and fed to the other operation, then result is pipelined parallelism (c) Interquery Parallelism - Scale up the parallelism - Avoid cache coherency, concurrency control - Increase the overall rate at which transactions can be processed in parallel by increasing the processors 21
22