Database Management System
|
|
|
- Silvia Posy Robbins
- 5 years ago
- Views:
Transcription
1 Database Management System Lecture Join * Some materials adapted from R. Ramakrishnan, J. Gehrke and Shawn Bowers
2 Today s Agenda Join Algorithm Database Management System
3 Join Algorithms Database Management System
4 Join Algorithms SELECT * FROM Reserves R, Sailors S WHERE R.sid = S.sid Sailors(sid, snam, rating, age) Boats(bid, bname, color) Reserves(sid, bid, day) R S is very common R X S followed by a selection is inefficient... why? So we process joins (rather than cross product) when possible Much effort in query processing invested in join algorithms Database Management System
5 Notations M -- pages in R P R -- tuples per page N -- pages in S P S -- tuples per page table R page tuple table S page tuple tuple tuple Database Management System
6 Join Algorithms Simple nested loops Join Join on i-th column of R and j-th column of S. foreach tuple r in R do. foreach tuple s in S do. if ri== sjthen add <r, s> to result For R S... We call R the outer relation We call S the inner relation Database Management System
7 Join Algorithms For each tuple in the outer relation R, we scan the entire inner relation S tuple-by-tuple... If M = 000 pages in R, P R = 00 tuples/page If N = 00 Pages in S, P S = 0 tuples/page If 00 I/Os per second Cost R S = M + (P R * M) * N = *000*00 I/Os 0,00,000 I/Os 00,00 seconds days! Database Management System
8 Simple Nested Loops Join This example highlights Simple nested loop join isn t very practical We need algorithms that optimize joins There are also the other operations to consider... Database Management System
9 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... Database Management System
10 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... Load st page of Table into memory Load st page of Table into memory Start checking join conditon (e.g., R.sid = S.sid).. Database Management System 0
11 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... Match! query Answer.. Database Management System
12 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... query Answer No Match! discard.. Database Management System
13 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk.... Load nd page of Table into memory query Answer No Match! discard.. Database Management System
14 Simple Nested Loops Join Table on Disk Table on Disk. Memory Buffers.... query Answer.. No Match! discard Load rd page of Table into memory.. Database Management System
15 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... query Answer No Match! discard Go to next tuple in st page of Table (Re)Load st page of Table into memory.. Database Management System
16 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk.... (Re)Load nd page of Table into memory Match! query Answer and so forth.. Database Management System
17 Simple Nested Loops Join Table on Disk. Memory Buffers Table on Disk.... (Re)Load nd page of Table into memory Does this algorithm work for R.sid < S.sid? Does this work for cross for cross product Match! query Answer and so forth.. Database Management System
18 Join Algorithms (Revisited) simple nested loops join Join on i-th column of R and j-th column of S. foreach tuple r in R do. foreach tuple s in S do. if ri== sjthen add <r, s> to result For each tuple in the outer relation R, we scan the entire inner relation S tuple-by-tuple... If M = 000 pages in R, P R = 00 tuples/page If N = 00 Pages in S, P S = 0 tuples/page If 00 I/Os per second Cost R S = M + (P R * M) * N = *000*00 I/Os 0,00,000 I/Os 00,00 seconds days! Database Management System
19 Join Algorithms (Revisited) page-oriented nested loops join Join on i-th column of R and j-th column of S. foreach page of tuples in R do. foreach page of tuples in S do. foreach record r and s in memory. if r i == s j then add <r, s> to result For each page in R, get each page in S... If M = 000 pages in R,N = 00 Pages in S,and 00 I/Os per sec. Cost R S = M + M * N = *00 = 0,000 I/Os Cost S R = N + N * M = *000 = 00,00 I/Os Thus, we typically use smaller relation as outer relation 00,00 I/Os. hours Database Management System
20 Page-Oriented Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... Database Management System 0
21 Page-Oriented Nested Loops Join Table on Disk. Memory Buffers Table on Disk..... Load st page of Table into memory Load st page of Table into memory Check every combination of records in buffers.. Database Management System
22 Page-Oriented Nested Loops Join Table on Disk. Memory Buffers Table on Disk.... Do the same thing for nd page in table.. Database Management System
23 Page-Oriented Nested Loops Join Table on Disk. Memory Buffers Table on Disk Do the same thing for rd page in table.. Database Management System
24 Page-Oriented Nested Loops Join Table on Disk. Memory Buffers Table on Disk Repeat for the nd page of Table and so on.. Database Management System
25 Another Alternative Algorithm: Use Buffer Block nested loops join Join on i-th column of R and j-th column of S. Assume B pages of memory in buffer. Assign one page of memory in buffer to output. Load B- pages of tuples from R. Load page of tuples from S. foreach record r and s in memory. if ri == sj then add <r, s> to result For multiple pages in R, get each page in S... check all pairs and output If M = 000 pages in R,N = 00 Pages in S,B =,and 00 I/Os per sec. Cost R S = M + (M / (B )) * N = (000/)*00,000 I/Os Cost S R = N + N * M = 00 + (00/)*000,00 I/Os,00 I/Os minutes Database Management System
26 Block Nested Loops Join Table on Disk... Memory Buffers.... Table on Disk. Load st page of Table into memory Load as many pages of Table into memory as possible Check every combination of records in buffers.. Database Management System
27 Yet Another Alternative Algorithm: Use Index Index nested loops join Join on i-th column of R and j-th column of S. Assuming there is an index on the j-th column of S. foreach tuple r in R do. find tuples s in S with matching search key r i. for each such s, add <r, s> to result For records in R, use search key to obtain matching S records If M = 000 pages in R, PR = 00 tuples/page, and 00 I/Os per sec. Cost R S = M + (M*PR) * cost of finding matching S tuples = (000*00) * 00,00 I/Os hour Cost S R = 00 + (00*0)* 0,00 I/Os 0 minutes If probing R is I/Os, then minutes Database Management System
28 Page-Oriented Nested Loops Join Table on Disk Table on Disk. Memory Buffers.... Table index Load st page of Table into memory For each tuple in page, probe index of Table Output resulting tuples.. Database Management System
29 And Another Alternative Algorithm: Sort If each relation is sorted on the join attributes... Cost of joining R and S can be reduced to M + N... Compare st in R and st in S If match output <r, s> Otherwise discard smallest and repeat But what if R and S are not sorted? We need to sort them The Challenge: The tables do not fit into memory! The Solution: External Sorting Note that other relational operator algorithms also require sorting Database Management System
30 N-Way External Sorting Employ the merge step in the mergesort algorithm On the first pass: Read pages of file until memory (buffers) full Sort data in buffer pages on (search/sort) key Write result back out to disk Result is a sorted run... A sorted run consists of a (sub-) set of small sorted files Database Management System 0
31 N-Way External Sorting Employ the merge step in the mergesort algorithm Once we have a sorted run Do an N-way merge... rather than a -way merge as in mergesort N = B is the number of available buffers One buffer reserved for output Results in a set of additional passes In each pass we create larger sorted sub-files Database Management System
32 Ex: Buffer Pages New File of sorted sub-files First Pass: load B = pages, sort, and store as sorted sub-files Database Management System
33 Ex: Buffer Pages 0 0 Load Buffer Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
34 Ex: Buffer Pages 0 0 Output st Page Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
35 Ex: Buffer Pages 0 0 Load Buffer again Buffer 0 In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
36 Ex: Buffer Pages 0 0 Output nd Page Buffer 0 In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
37 Ex: Buffer Pages 0 0 Output rd Page Buffer 0 In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
38 Ex: Buffer Pages 0 0 Load Buffer again Buffer 0 In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
39 Ex: Buffer Pages 0 0 Output rd Page Buffer 0 In this case -way merge Merge B of the sorted sub files (sorted runs) Database Management System
40 Ex: Buffer Pages 0 0 Output th Page Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System 0
41 Ex: Buffer Pages 0 0 Load Buffer again Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System
42 Ex: Buffer Pages 0 0 Output th Page Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System
43 Ex: Buffer Pages 0 0 Load Buffer again Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System
44 Ex: Buffer Pages 0 0 Output th Page Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System
45 Ex: Buffer Pages 0 0 Output th Page Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System
46 Ex: Buffer Pages 0 0 Load Buffer again Buffer 0 In this case -way merge Merge B of the sorted sub files (sorted runs) 0 Database Management System
47 Ex: Buffer Pages 0 0 Output th Buffer Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 0 Database Management System
48 Ex: Buffer Pages 0 0 Output th Buffer Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 0 Database Management System
49 Ex: Buffer Pages 0 0 Load Buffer again Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 0 Database Management System
50 Ex: Buffer Pages 0 0 Output th Buffer Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 0 Database Management System 0
51 Ex: Buffer Pages 0 0 Output th Buffer DONE!! Buffer In this case -way merge Merge B of the sorted sub files (sorted runs) 0 0 Database Management System
52 N-Way External Sorting Merge may require multiple passes At each merge pass the number of sub-files is reduced by B initial sort st Merge nd Merge Database Management System
53 N-Way External Sorting The cost: Each pass does *M I/Os (for M pages in table) We read and write the entire file (all pages) in each pass So... how many passes? Number of passes depends on buffer space available Passes = LogB (M/B)... Why M/B? Can sort 00 million pages in passes w/ pages of memory Can sort M pages using B memory pages in passes if M < B (often true) Database Management System
54 Sort Merge Join Sort R on join attribute (if not already sorted) Sort S on join attribute (if not already sorted) Merge R and S Scan of R until R-tuple current S-tuple Then scan S until S-tuple R-tuple Repeat until R-tuple = S-tuple At this point, we have a match, and output Then resume scanning R and S Database Management System
55 Sort Merge Join Outer relation R is scanned once Each time an R-tuple r matches first S-tuple We form a group of S-tuples that match r Each such group is scanned once per matching R tuple Either: This group fits into memory (and the scan is free ) Or we have extra page I/Os (to reread the group) Database Management System
56 Sort Merge Join Best case cost (all matches in memory): Cost to sort R + Cost to sort S + (M+N) Worst case cost (all R and S have same value) Matching group is the entire S relation Cost to Sort R + Cost to sort S + M + M*N... note this is worse than page-oriented nested loops! (since you also have to sort R and S) Database Management System
57 Sort Merge Join For Reserves and Sailors: Reserves has 000 pages Sailors has 00 pages With pages in the buffer, each sorted in passes Best case cost is: *000 + * = 00 I/Os minute... multiply by since it takes passes and each pass reads and writes each page of file Database Management System
58 Sorting using B+ Trees Lets say the table we want to sort has a B+ Tree defined on the sorting attributes Can the B+ Tree help to retrieve records in order? It can help if the B+ Tree is clustered We can retrieve records in order by traversing leaf pages Records either stored in leaf pages or can be obtained from leaf pages It can be a very bad idea if B+ Tree is not clustered Why? Database Management System
59 Sorting via Clustered B+ Trees Cost for clustered case Root to the left-most leaf, then retrieve all pages What if it is unclustered? Additional cost of retrieving data records Each page fetched just once Always better than external sorting! Database Management System
60 Sorting via Clustered B+ Trees Similar to the case of doing a range query unclustered case - data entries Each entry on one leaf page can point to a different page In general, one I/O per data record! Database Management System 0
61 Hash Join Simple case: entire S table fits into main memory Build an in-memory hash index for S ( build phase) recall a hash index maps keys to buckets of records Scan R and find matching S-records ( probe phase) this is identical to the index nested loops join Cost is: Cost to read R (the outer relation) Cost to read S (the inner relation + build index) Each time we read a page in R we find all matches with S So total cost is M + N! Database Management System
62 Hash Join What do we do if S does not fit into memory? Define a hash function h that can be used to partition R and S Each S partition should be small enough to fit into main memory Apply h to R and S and store each resulting partition in a file Do the simple case (index nested loop join) on each pair of matching partitions (files) Database Management System
63 Hash Join R partitions S partitions P P foreach r in R, h(r) P foreach s in S, h(s) P Pn Pn Store each in File Store each in File P P Probe R partitions P P Probe S partitions simple hash join Pn Pn Database Management System
64 Hash Join Partitions: We assume that the number of partitions k < B Each partition may have many pages Cost of Hash Join: *M to partition R (read and write) *N to partition S (read and write) Cost to join partitions: M + N Total cost is: *(M+N) For reserves and sailors: *( ) = 00 I/Os seconds Database Management System
65 Sort Merge Join vs. Hash Join Sort-Merge Join Less sensitive to data skew (e.g., clusters of similar values) Result is sorted (... more on this later) Hash Join Highly parallelizable (join partitions concurrently) For inequality conditions (e.g., R.name < S.name) Hash and Sort-Merge Join not applicable Block nested loops likely to be the best approach Database Management System
66 Comparison of (approximate) costs Join Algorithm I/Os Time Simple Nested Loops Join 0,000,000 days Page Nested Loops Join 00,000. hours Block Nested Loops Join,000 minutes Index Nested Loops Join 0,00 0 minutes Sorted-merge Join 00 (at best) minute (at best) Hash Join 00 seconds Assuming: R has 000 pages, 00 tuples/page S has 00 pages, 0 tuples/page buffer pages 00 I/Os per second Database Management System
67 For Next Week Read Ch. Database Management System
Implementing Relational Operators: Selection, Projection, Join. Database Management Systems, R. Ramakrishnan and J. Gehrke 1
Implementing Relational Operators: Selection, Projection, Join Database Management Systems, R. Ramakrishnan and J. Gehrke 1 Readings [RG] Sec. 14.1-14.4 Database Management Systems, R. Ramakrishnan and
Implementing Joins 1
Implementing Joins 1 Last Time Selection Scan, binary search, indexes Projection Duplicate elimination: sorting, hashing Index-only scans Joins 2 Tuple Nested Loop Join foreach tuple r in R do foreach
Evaluation of Relational Operations
Evaluation of Relational Operations Yanlei Diao UMass Amherst March 13 and 15, 2006 Slides Courtesy of R. Ramakrishnan and J. Gehrke 1 Relational Operations We will consider how to implement: Selection
Evaluation of Relational Operations. Relational Operations
Evaluation of Relational Operations Chapter 14, Part A (Joins) Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Relational Operations v We will consider how to implement: Selection ( )
Implementation of Relational Operations
Implementation of Relational Operations Module 4, Lecture 1 Database Management Systems, R. Ramakrishnan 1 Relational Operations We will consider how to implement: Selection ( ) Selects a subset of rows
Cost-based Query Sub-System. Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications. Last Class.
Cost-based Query Sub-System Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications Queries Select * From Blah B Where B.blah = blah Query Parser Query Optimizer C. Faloutsos A. Pavlo
Evaluation of Relational Operations
Evaluation of Relational Operations Chapter 14 Comp 521 Files and Databases Fall 2010 1 Relational Operations We will consider in more detail how to implement: Selection ( ) Selects a subset of rows from
Evaluation of Relational Operations
Evaluation of Relational Operations Chapter 12, Part A Database Management Systems, R. Ramakrishnan and J. Gehrke 1 Relational Operations We will consider how to implement: Selection ( ) Selects a subset
Announcement. Reading Material. Overview of Query Evaluation. Overview of Query Evaluation. Overview of Query Evaluation 9/26/17
Announcement CompSci 516 Database Systems Lecture 10 Query Evaluation and Join Algorithms Project proposal pdf due on sakai by 5 pm, tomorrow, Thursday 09/27 One per group by any member Instructor: Sudeepa
Evaluation of Relational Operations: Other Techniques. Chapter 14 Sayyed Nezhadi
Evaluation of Relational Operations: Other Techniques Chapter 14 Sayyed Nezhadi Schema for Examples Sailors (sid: integer, sname: string, rating: integer, age: real) Reserves (sid: integer, bid: integer,
Overview of Implementing Relational Operators and Query Evaluation
Overview of Implementing Relational Operators and Query Evaluation Chapter 12 Motivation: Evaluating Queries The same query can be evaluated in different ways. The evaluation strategy (plan) can make orders
Evaluation of relational operations
Evaluation of relational operations Iztok Savnik, FAMNIT Slides & Textbook Textbook: Raghu Ramakrishnan, Johannes Gehrke, Database Management Systems, McGraw-Hill, 3 rd ed., 2007. Slides: From Cow Book
CompSci 516 Data Intensive Computing Systems
CompSci 516 Data Intensive Computing Systems Lecture 9 Join Algorithms and Query Optimizations Instructor: Sudeepa Roy CompSci 516: Data Intensive Computing Systems 1 Announcements Takeaway from Homework
Faloutsos 1. Carnegie Mellon Univ. Dept. of Computer Science Database Applications. Outline
Carnegie Mellon Univ. Dept. of Computer Science 15-415 - Database Applications Lecture #14: Implementation of Relational Operations (R&G ch. 12 and 14) 15-415 Faloutsos 1 introduction selection projection
External Sorting Implementing Relational Operators
External Sorting Implementing Relational Operators 1 Readings [RG] Ch. 13 (sorting) 2 Where we are Working our way up from hardware Disks File abstraction that supports insert/delete/scan Indexing for
Overview of Query Processing. Evaluation of Relational Operations. Why Sort? Outline. Two-Way External Merge Sort. 2-Way Sort: Requires 3 Buffer Pages
Overview of Query Processing Query Parser Query Processor Evaluation of Relational Operations Query Rewriter Query Optimizer Query Executor Yanlei Diao UMass Amherst Lock Manager Access Methods (Buffer
Overview of Query Evaluation
Overview of Query Evaluation Chapter 12 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Overview of Query Evaluation Plan: Tree of R.A. ops, with choice of alg for each op. Each operator
Dtb Database Systems. Announcement
Dtb Database Systems ( 資料庫系統 ) December 10, 2008 Lecture #11 1 Announcement Assignment #5 will be out on the course webpage today. 2 1 External Sorting Chapter 13 3 Why learn sorting again? O (n*n): bubble,
ECS 165B: Database System Implementa6on Lecture 7
ECS 165B: Database System Implementa6on Lecture 7 UC Davis April 12, 2010 Acknowledgements: por6ons based on slides by Raghu Ramakrishnan and Johannes Gehrke. Class Agenda Last 6me: Dynamic aspects of
Overview of Query Evaluation. Overview of Query Evaluation
Overview of Query Evaluation Chapter 12 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Overview of Query Evaluation v Plan: Tree of R.A. ops, with choice of alg for each op. Each operator
CS 4604: Introduction to Database Management Systems. B. Aditya Prakash Lecture #10: Query Processing
CS 4604: Introduction to Database Management Systems B. Aditya Prakash Lecture #10: Query Processing Outline introduction selection projection join set & aggregate operations Prakash 2018 VT CS 4604 2
CSE 444: Database Internals. Section 4: Query Optimizer
CSE 444: Database Internals Section 4: Query Optimizer Plan for Today Problem 1A, 1B: Estimating cost of a plan You try to compute the cost for 5 mins We will go over the solution together Problem 2: Sellinger
15-415/615 Faloutsos 1
Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications Lecture #14: Implementation of Relational Operations (R&G ch. 12 and 14) 15-415/615 Faloutsos 1 Outline introduction selection
Evaluation of Relational Operations: Other Techniques
Evaluation of Relational Operations: Other Techniques Chapter 12, Part B Database Management Systems 3ed, R. Ramakrishnan and Johannes Gehrke 1 Using an Index for Selections v Cost depends on #qualifying
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VI Lecture 14, March 12, 2014 Mohammad Hammoud Today Last Session: DBMS Internals- Part V Hash-based indexes (Cont d) and External Sorting Today s Session:
CAS CS 460/660 Introduction to Database Systems. Query Evaluation II 1.1
CAS CS 460/660 Introduction to Database Systems Query Evaluation II 1.1 Cost-based Query Sub-System Queries Select * From Blah B Where B.blah = blah Query Parser Query Optimizer Plan Generator Plan Cost
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VI Lecture 17, March 24, 2015 Mohammad Hammoud Today Last Two Sessions: DBMS Internals- Part V External Sorting How to Start a Company in Five (maybe
Implementation of Relational Operations: Other Operations
Implementation of Relational Operations: Other Operations Module 4, Lecture 2 Database Management Systems, R. Ramakrishnan 1 Simple Selections SELECT * FROM Reserves R WHERE R.rname < C% Of the form σ
Implementation of Relational Operations. Introduction. CS 186, Fall 2002, Lecture 19 R&G - Chapter 12
Implementation of Relational Operations CS 186, Fall 2002, Lecture 19 R&G - Chapter 12 First comes thought; then organization of that thought, into ideas and plans; then transformation of those plans into
Presenter: Eunice Tan Lam Ziyuan Yan Ming fei. Join Algorithm
Presenter: Eunice Tan Lam Ziyuan Yan Ming fei Join Algorithm Example Table Table Sailors sid sname rating Age 22 Dustin 7 45 28 Yuppy 9 35 31 Lubber 8 55 36 Guppy 6 36 44 rusty 5 35 Table Reserves sid
CS330. Query Processing
CS330 Query Processing 1 Overview of Query Evaluation Plan: Tree of R.A. ops, with choice of alg for each op. Each operator typically implemented using a `pull interface: when an operator is `pulled for
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VII Lecture 15, March 17, 2014 Mohammad Hammoud Today Last Session: DBMS Internals- Part VI Algorithms for Relational Operations Today s Session: DBMS
CSE 444: Database Internals. Sec2on 4: Query Op2mizer
CSE 444: Database Internals Sec2on 4: Query Op2mizer Plan for Today Problem 1A, 1B: Es2ma2ng cost of a plan You try to compute the cost for 5 mins We go over the solu2on together Problem 2: Sellinger Op2mizer
Midterm Review CS634. Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke
Midterm Review CS634 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Coverage Text, chapters 8 through 15 (hw1 hw4) PKs, FKs, E-R to Relational: Text, Sec. 3.2-3.5, to pg.
Evaluation of Relational Operations: Other Techniques
Evaluation of Relational Operations: Other Techniques [R&G] Chapter 14, Part B CS4320 1 Using an Index for Selections Cost depends on #qualifying tuples, and clustering. Cost of finding qualifying data
Administriva. CS 133: Databases. General Themes. Goals for Today. Fall 2018 Lec 11 10/11 Query Evaluation Prof. Beth Trushkowsky
Administriva Lab 2 Final version due next Wednesday CS 133: Databases Fall 2018 Lec 11 10/11 Query Evaluation Prof. Beth Trushkowsky Problem sets PSet 5 due today No PSet out this week optional practice
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VIII Lecture 16, March 19, 2014 Mohammad Hammoud Today Last Session: DBMS Internals- Part VII Algorithms for Relational Operations (Cont d) Today s Session:
Database Systems. Announcement. December 13/14, 2006 Lecture #10. Assignment #4 is due next week.
Database Systems ( 料 ) December 13/14, 2006 Lecture #10 1 Announcement Assignment #4 is due next week. 2 1 Overview of Query Evaluation Chapter 12 3 Outline Query evaluation (Overview) Relational Operator
QUERY EXECUTION: How to Implement Relational Operations?
QUERY EXECUTION: How to Implement Relational Operations? 1 Introduction We ve covered the basic underlying storage, buffering, indexing and sorting technology Now we can move on to query processing Relational
Principles of Data Management. Lecture #9 (Query Processing Overview)
Principles of Data Management Lecture #9 (Query Processing Overview) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Today s Notable News v Midterm
Evaluation of Relational Operations: Other Techniques
Evaluation of Relational Operations: Other Techniques Chapter 14, Part B Database Management Systems 3ed, R. Ramakrishnan and Johannes Gehrke 1 Using an Index for Selections Cost depends on #qualifying
Query Evaluation Overview, cont.
Query Evaluation Overview, cont. Lecture 9 Feb. 29, 2016 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Architecture of a DBMS Query Compiler Execution Engine Index/File/Record
Overview of Query Evaluation. Chapter 12
Overview of Query Evaluation Chapter 12 1 Outline Query Optimization Overview Algorithm for Relational Operations 2 Overview of Query Evaluation DBMS keeps descriptive data in system catalogs. SQL queries
Query Evaluation Overview, cont.
Query Evaluation Overview, cont. Lecture 9 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Architecture of a DBMS Query Compiler Execution Engine Index/File/Record Manager
Examples of Physical Query Plan Alternatives. Selected Material from Chapters 12, 14 and 15
Examples of Physical Query Plan Alternatives Selected Material from Chapters 12, 14 and 15 1 Query Optimization NOTE: SQL provides many ways to express a query. HENCE: System has many options for evaluating
CompSci 516 Data Intensive Computing Systems. Lecture 11. Query Optimization. Instructor: Sudeepa Roy
CompSci 516 Data Intensive Computing Systems Lecture 11 Query Optimization Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements HW2 has been posted on sakai Due on Oct
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing
R & G Chapter 13. Implementation of single Relational Operations Choices depend on indexes, memory, stats, Joins Blocked nested loops:
Relational Query Optimization R & G Chapter 13 Review Implementation of single Relational Operations Choices depend on indexes, memory, stats, Joins Blocked nested loops: simple, exploits extra memory
Evaluation of Relational Operations. SS Chung
Evaluation of Relational Operations SS Chung Cost Metric Query Processing Cost = Disk I/O Cost + CPU Computation Cost Disk I/O Cost = Disk Access Time + Data Transfer Time Disk Acess Time = Seek Time +
Sorting a file in RAM. External Sorting. Why Sort? 2-Way Sort of N pages Requires Minimum of 3 Buffers Pass 0: Read a page, sort it, write it.
Sorting a file in RAM External Sorting Chapter 13 Three steps: Read the entire file from disk into RAM Sort the records using a standard sorting procedure, such as Shell sort, heap sort, bubble sort, Write
RELATIONAL OPERATORS #1
RELATIONAL OPERATORS #1 CS 564- Spring 2018 ACKs: Jeff Naughton, Jignesh Patel, AnHai Doan WHAT IS THIS LECTURE ABOUT? Algorithms for relational operators: select project 2 ARCHITECTURE OF A DBMS query
Overview of Query Processing
ICS 321 Fall 2013 Overview of Query Processing Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 11/20/2013 Lipyeow Lim -- University of Hawaii at Manoa 1
Introduction to Data Management. Lecture 14 (SQL: the Saga Continues...)
Introduction to Data Management Lecture 14 (SQL: the Saga Continues...) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Announcements v HW and
Principles of Data Management. Lecture #12 (Query Optimization I)
Principles of Data Management Lecture #12 (Query Optimization I) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Today s Notable News v B+ tree
Relational Query Optimization. Highlights of System R Optimizer
Relational Query Optimization Chapter 15 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Highlights of System R Optimizer v Impact: Most widely used currently; works well for < 10 joins.
Question 1. Part (a) [2 marks] what are different types of data independence? explain each in one line. Part (b) CSC 443 Term Test Solutions Fall 2017
![Question 1. Part (a) [2 marks] what are different types of data independence? explain each in one line. Part (b) CSC 443 Term Test Solutions Fall 2017](/thumbs/77/75919252.jpg "Question 1. Part (a) [2 marks] what are different types of data independence? explain each in one line. Part (b) CSC 443 Term Test Solutions Fall 2017")
Question 1. [12 marks] Short Answer Question Part (a) [2 marks] what are different types of data independence? explain each in one line Logical data independence: Protection from changes in logical structure
EECS 647: Introduction to Database Systems
EECS 647: Introduction to Database Systems Instructor: Luke Huan Spring 2009 External Sorting Today s Topic Implementing the join operation 4/8/2009 Luke Huan Univ. of Kansas 2 Review DBMS Architecture
Outline. Database Management and Tuning. Outline. Join Strategies Running Example. Index Tuning. Johann Gamper. Unit 6 April 12, 2012
Outline Database Management and Tuning Johann Gamper Free University of Bozen-Bolzano Faculty of Computer Science IDSE Unit 6 April 12, 2012 1 Acknowledgements: The slides are provided by Nikolaus Augsten
Join Algorithms. Lecture #12. Andy Pavlo Computer Science Carnegie Mellon Univ. Database Systems / Fall 2018
Join Algorithms Lecture #12 Database Systems 15-445/15-645 Fall 2018 AP Andy Pavlo Computer Science Carnegie Mellon Univ. 2 ADMINISTRIVIA Project #2 Checkpoint #1 is due TODAY No class on Wednesday October
Operator Implementation Wrap-Up Query Optimization
Operator Implementation Wrap-Up Query Optimization 1 Last time: Nested loop join algorithms: TNLJ PNLJ BNLJ INLJ Sort Merge Join Hash Join 2 General Join Conditions Equalities over several attributes (e.g.,
Query Optimization. Schema for Examples. Motivating Example. Similar to old schema; rname added for variations. Reserves: Sailors:
Query Optimization Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Schema for Examples (sid: integer, sname: string, rating: integer, age: real) (sid: integer, bid: integer, day: dates,
CSIT5300: Advanced Database Systems
CSIT5300: Advanced Database Systems E11: Exercises on Query Optimization Dr. Kenneth LEUNG Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong SAR,
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? v A classic problem in computer science! v Data requested in sorted order e.g., find students in increasing
Chapter 12: Query Processing. Chapter 12: Query Processing
Chapter 12: Query Processing Database System Concepts, 6 th Ed. See www.db-book.com for conditions on re-use Chapter 12: Query Processing Overview Measures of Query Cost Selection Operation Sorting Join
Basic form of SQL Queries
SQL - 1 Week 6 Basic form of SQL Queries SELECT FROM WHERE target-list relation-list qualification target-list A list of attributes of output relations in relation-list relation-list A list of relation
Schema for Examples. Query Optimization. Alternative Plans 1 (No Indexes) Motivating Example. Alternative Plans 2 With Indexes
Schema for Examples Query Optimization (sid: integer, : string, rating: integer, age: real) (sid: integer, bid: integer, day: dates, rname: string) Similar to old schema; rname added for variations. :
Administrivia. Relational Query Optimization (this time we really mean it) Review: Query Optimization. Overview: Query Optimization
Relational Query Optimization (this time we really mean it) R&G hapter 15 Lecture 25 dministrivia Homework 5 mostly available It will be due after classes end, Monday 12/8 Only 3 more lectures left! Next
Relational Query Optimization
Relational Query Optimization Module 4, Lectures 3 and 4 Database Management Systems, R. Ramakrishnan 1 Overview of Query Optimization Plan: Tree of R.A. ops, with choice of alg for each op. Each operator
Relational Query Optimization. Overview of Query Evaluation. SQL Refresher. Yanlei Diao UMass Amherst October 23 & 25, 2007
Relational Query Optimization Yanlei Diao UMass Amherst October 23 & 25, 2007 Slide Content Courtesy of R. Ramakrishnan, J. Gehrke, and J. Hellerstein 1 Overview of Query Evaluation Query Evaluation Plan:
Sorting & Aggregations
Sorting & Aggregations Lecture #11 Database Systems /15-645 Fall 2018 AP Andy Pavlo Computer Science Carnegie Mellon Univ. 2 Sorting Algorithms Aggregations TODAY'S AGENDA 3 WHY DO WE NEED TO SORT? Tuples
CMPUT 391 Database Management Systems. Query Processing: The Basics. Textbook: Chapter 10. (first edition: Chapter 13) University of Alberta 1
CMPUT 391 Database Management Systems Query Processing: The Basics Textbook: Chapter 10 (first edition: Chapter 13) Based on slides by Lewis, Bernstein and Kifer University of Alberta 1 External Sorting
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part V Lecture 13, March 10, 2014 Mohammad Hammoud Today Welcome Back from Spring Break! Today Last Session: DBMS Internals- Part IV Tree-based (i.e., B+
Final Exam Review. Kathleen Durant PhD CS 3200 Northeastern University
Final Exam Review Kathleen Durant PhD CS 3200 Northeastern University 1 Outline for today Identify topics for the final exam Discuss format of the final exam What will be provided for you and what you
Query and Join Op/miza/on 11/5
Query and Join Op/miza/on 11/5 Overview Recap of Merge Join Op/miza/on Logical Op/miza/on Histograms (How Es/mates Work. Big problem!) Physical Op/mizer (if we have /me) Recap on Merge Key (Simple) Idea
Lassonde School of Engineering Winter 2016 Term Course No: 4411 Database Management Systems
Lassonde School of Engineering Winter 2016 Term Course No: 4411 Database Management Systems Last Name: First Name: Student ID: 1. Exam is 2 hours long 2. Closed books/notes Problem 1 (6 points) Consider
EXTERNAL SORTING. Sorting
EXTERNAL SORTING 1 Sorting A classic problem in computer science! Data requested in sorted order (sorted output) e.g., find students in increasing grade point average (gpa) order SELECT A, B, C FROM R
Query Processing: The Basics. External Sorting
Query Processing: The Basics Chapter 10 1 External Sorting Sorting is used in implementing many relational operations Problem: Relations are typically large, do not fit in main memory So cannot use traditional
Query Optimization. Schema for Examples. Motivating Example. Similar to old schema; rname added for variations. Reserves: Sailors:
Query Optimization atabase Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Schema for Examples (sid: integer, sname: string, rating: integer, age: real) (sid: integer, bid: integer, day: dates,
Last Class Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications
Last Class Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications C. Faloutsos A. Pavlo Lecture#12: External Sorting (R&G, Ch13) Static Hashing Extendible Hashing Linear Hashing Hashing
Database Systems CSE 414
Database Systems CSE 414 Lectures 16 17: Basics of Query Optimization and Cost Estimation (Ch. 15.{1,3,4.6,6} & 16.4-5) 1 Announcements WQ4 is due Friday 11pm HW3 is due next Tuesday 11pm Midterm is next
Chapter 12: Query Processing
Chapter 12: Query Processing Database System Concepts, 6 th Ed. See www.db-book.com for conditions on re-use Overview Chapter 12: Query Processing Measures of Query Cost Selection Operation Sorting Join
Tree-Structured Indexes
Tree-Structured Indexes Yanlei Diao UMass Amherst Slides Courtesy of R. Ramakrishnan and J. Gehrke Access Methods v File of records: Abstraction of disk storage for query processing (1) Sequential scan;
Query Processing. Introduction to Databases CompSci 316 Fall 2017
Query Processing Introduction to Databases CompSci 316 Fall 2017 2 Announcements (Tue., Nov. 14) Homework #3 sample solution posted in Sakai Homework #4 assigned today; due on 12/05 Project milestone #2
Advanced Database Systems
Lecture IV Query Processing Kyumars Sheykh Esmaili Basic Steps in Query Processing 2 Query Optimization Many equivalent execution plans Choosing the best one Based on Heuristics, Cost Will be discussed
CSE 544 Principles of Database Management Systems
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 6 Lifecycle of a Query Plan 1 Announcements HW1 is due Thursday Projects proposals are due on Wednesday Office hour canceled
Storage hierarchy. Textbook: chapters 11, 12, and 13
Storage hierarchy Cache Main memory Disk Tape Very fast Fast Slower Slow Very small Small Bigger Very big (KB) (MB) (GB) (TB) Built-in Expensive Cheap Dirt cheap Disks: data is stored on concentric circular
Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M.
![Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M.](/thumbs/75/72297724.jpg "Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M.")
Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M. Patel, 2013 1 Joins The focus here is on equijoins These are very common,
ATYPICAL RELATIONAL QUERY OPTIMIZER
14 ATYPICAL RELATIONAL QUERY OPTIMIZER Life is what happens while you re busy making other plans. John Lennon In this chapter, we present a typical relational query optimizer in detail. We begin by discussing
Chapter 13: Query Processing
Chapter 13: Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions 13.1 Basic Steps in Query Processing 1. Parsing
Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13!
![Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13!](/thumbs/78/78385651.jpg "Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13!")
Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13! q Overview! q Optimization! q Measures of Query Cost! Query Evaluation! q Sorting! q Join Operation! q Other
Query Processing. Debapriyo Majumdar Indian Sta4s4cal Ins4tute Kolkata DBMS PGDBA 2016
Query Processing Debapriyo Majumdar Indian Sta4s4cal Ins4tute Kolkata DBMS PGDBA 2016 Slides re-used with some modification from www.db-book.com Reference: Database System Concepts, 6 th Ed. By Silberschatz,
Course No: 4411 Database Management Systems Fall 2008 Midterm exam
Course No: 4411 Database Management Systems Fall 2008 Midterm exam Last Name: First Name: Student ID: Exam is 80 minutes. Open books/notes The exam is out of 20 points. 1 1. (16 points) Multiple Choice
NOTE: sorting using B-trees to be assigned for reading after we cover B-trees.
External Sorting Chapter 13 (Sec. 13-1-13.5): Ramakrishnan & Gehrke and Chapter 11 (Sec. 11.4-11.5): G-M et al. (R2) OR Chapter 2 (Sec. 2.4-2.5): Garcia-et Molina al. (R1) NOTE: sorting using B-trees to
Review. Relational Query Optimization. Query Optimization Overview (cont) Query Optimization Overview. Cost-based Query Sub-System
Review Relational Query Optimization R & G Chapter 12/15 Implementation of single Relational Operations Choices depend on indexes, memory, stats, Joins Blocked nested loops: simple, exploits extra memory
An SQL query is parsed into a collection of query blocks optimize one block at a time. Nested blocks are usually treated as calls to a subroutine
QUERY OPTIMIZATION 1 QUERY OPTIMIZATION QUERY SUB-SYSTEM 2 ROADMAP 3. 12 QUERY BLOCKS: UNITS OF OPTIMIZATION An SQL query is parsed into a collection of query blocks optimize one block at a time. Nested
CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 9 Index Selection and External Sorting Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements Private project threads created on piazza
Final Exam Review. Kathleen Durant CS 3200 Northeastern University Lecture 22
Final Exam Review Kathleen Durant CS 3200 Northeastern University Lecture 22 Outline for today Identify topics for the final exam Discuss format of the final exam What will be provided for you and what
! A relational algebra expression may have many equivalent. ! Cost is generally measured as total elapsed time for
Chapter 13: Query Processing Basic Steps in Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions 1. Parsing and
Chapter 13: Query Processing Basic Steps in Query Processing
Chapter 13: Query Processing Basic Steps in Query Processing! Overview! Measures of Query Cost! Selection Operation! Sorting! Join Operation! Other Operations! Evaluation of Expressions 1. Parsing and
Announcements. Reading Material. Today. Different File Organizations. Selection of Indexes 9/24/17. CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 9 Index Selection and External Sorting Announcements Private project threads created on piazza Please use these threads (and not emails) for all communications on your