External Sorting Implementing Relational Operators
|
|
|
- Tracey Townsend
- 5 years ago
- Views:
Transcription
1 External Sorting Implementing Relational Operators 1
2 Readings [RG] Ch. 13 (sorting) 2
3 Where we are Working our way up from hardware Disks File abstraction that supports insert/delete/scan Indexing for efficient access 3
4 External sorting Sorting a large amount of data that does not fit in RAM E.g., sort 1TB of data with 16G of RAM Why sorting? Useful building block in a variety of DB queries Eliminate duplicates in a relation Some join implementations Or may just want data in sorted order Benchmarks and competitions See 4
5 Remember merge sort? 5
6 2-Way External Merge Sort Pass 0: Read a page at a time, sort it using your favorite algorithm, write it Only one buffer page used (could use more if we have more) 1 Page Disk Main memory buffers Disk 6
7 Two-Way External Merge Sort: Pass 0 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 Input file 1-page runs Assume input file with N data pages What is the cost of Pass 0 (in terms of # I/Os)? 7
8 Definition: A run is a sorted portion of the file Pass 0 generates some number of runs (how many??) 8
9 2-Way External Merge Sort Now: make more passes to merge runs Pass 1: Merge two runs of length 1 (page) Pass 2: Merge two runs of length 2 (pages) until 1 run of length N Three buffer pages used INPUT 1 INPUT 2 OUTPUT Disk Main memory buffers Disk 9
10 2-Way External Merge Sort 3,4 6,2 9,4 8,7 5,6 3,1 2 Input file 3,4 2,6 4,9 7,8 5,6 1,3 2 4,6 4,4 6,7 8,9 4,7 8,9 1,3 5,6 2 1,2 3,5 6 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 1,2 3,4 4,5 6,6 7,8 9 8-page runs 10
11 2-Way External Merge Sort: Analysis Total I/O cost for sorting file with N pages Cost of Pass 0 = 2N Number of merge passes = Cost of each merge pass = log 2 N 2N 2 log Cost of all merge passes = N 2 N Total cost = ( log N + ) 2 N
12 General External Merge Sort General case where B buffer pages are available Let's see how the calculations change 12
13 2-Way External Merge Sort Pass 0: read a page at a time, sort it using your favorite algorithm, write it Only one buffer page used How can this be modified if B buffer pages are available? 1 Page Disk Main memory buffers Disk 13
14 General External Merge Sort Read B pages at a time, sort B pages in main memory, and write out B pages Length of each run = B pages Assuming N input pages, number of runs = N/B Cost = 2N B Pages Disk Main memory buffers Disk 14
15 General External Merge Sort: Pass 0 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs 4,4 1,1 3,4 15
16 2-Way External Merge Sort Merge passes: Make multiple passes to merge runs Pass 1: Merge two runs of length 1 (page) Pass 2: Merge two runs of length 2 (pages) until 1 run of length N Three buffer pages used How can this be modified if B buffer pages available? INPUT 1 INPUT 2 OUTPUT Disk Main memory buffers Disk 16
17 General External Merge Sort Make multiple passes to merge runs Pass 1: Produce runs of length B(B-1) pages Pass 2: Produce runs of length B(B-1) 2 pages Pass P: Produce runs of length B(B-1) P pages INPUT 1... INPUT OUTPUT Disk INPUT B-1 B Main memory buffers Disk 17
18 Definition Merge fan-in number of runs being merged in each merge pass What is the value of this fan-in? 18
19 General External Merge Sort: Merge # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 Pass 1 Main Memory 19
20 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 1,1 Pass 1 Main Memory 20
21 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 1,1 Pass 1 Main Memory 21
22 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 1,1 Pass 1 Main Memory 22
23 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 1 Pass 1 1 Main Memory 23
24 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 Pass 1 1,1 Main Memory 24
25 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 Pass 1 Main Memory 1,1 25
26 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 Pass 1 Main Memory 1,1 26
27 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 Pass 1 Main Memory 1,1 27
28 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 3 Pass 1 2 Main Memory 1,1 28
29 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 3 3 Pass 1 2,2 Main Memory 1,1 29
30 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 3 3 Pass 1 2,2 Main Memory 1,1 30
31 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3, Pass 1 2 Main Memory 2,2 1,1 31
32 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 3 3 Pass 1 Main Memory 2,2 1,1 32
33 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 3 3 Pass 1 2,2 Main Memory 1,1 33
34 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 4,4 3 3 Pass 1 2,2 Main Memory 1,1 34
35 General External Merge Sort: Phase 2 # buffer pages B = 4 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 4,4 3 3 Pass 1 2,2 Main Memory 1,1 35
36 General External Merge Sort: Analysis Total I/O cost for sorting file with N pages Cost of Pass 0 = 2N If # merge passes is P then: B(B-1) P = N Therefore P = logb 1 N / B Cost of each merge pass = Cost of all merge passes = 2N 2 1 N logb N / B Total cost = ( log N / B 1) 2N 1 + B 36
37 Number of Passes of External Sort N B=3 B=5 B=9 B=17 B=129 B= , , , ,000, ,000, ,000, ,000,000,
38 External Merge Sort: Optimizations Currently, do one page I/O at a time But can read/write a block of pages sequentially! Make each buffer input/output a block of pages 38
39 General External Merge Sort: Phase 2 # buffer pages B = 8 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 4,4 Pass 1 Main Memory 39
40 General External Merge Sort: Phase 2 # buffer pages B = 8 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 4,4 1,1 Pass 1 Main Memory 40
41 General External Merge Sort: Phase 2 # buffer pages B = 8 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 4,4 1,1 3,4 Pass 1 Main Memory 41
42 General External Merge Sort: Phase 2 # buffer pages B = 8 3,4 6,2 9,4 8,7 5,6 3,1 9,2 6,1 8,2 3,4 5,5 6,3 Input file 8,9 6,7 6,9 5,6 6,8 5,5 4-page runs Pass 0 4,4 1,1 3,4 4,4 1,1 3,4 Pass 1 Main Memory 42
43 External Merge Sort: Optimizations Tradeoff: we reduce the merge fan-in So more passes will be needed 43
44 General Merge Sort: Optimizations Double buffering: to reduce I/O wait time, prefetch into `shadow block. Again, potentially more passes; in practice, most files still sorted in 2-3 passes. INPUT 1 INPUT 1' INPUT 2 INPUT 2' OUTPUT OUTPUT' Disk INPUT k INPUT k' b block size Disk B main memory buffers, k-way merge 44
45 Using B+ Trees for Sorting Scenario: Table to be sorted has B+ tree index on sorting column(s). Idea: Can retrieve records in order by traversing leaf pages. Is this a good idea? Cases to consider: B+ tree is clustered Good idea! B+ tree is not clustered Could be a very bad idea! 45
46 Clustered B+ Tree Used for Sorting Cost: root to the leftmost leaf, then retrieve all leaf pages (Alternative 1) If Alternative 2 is used? Additional cost of retrieving data records: each page fetched just once. Index (Directs search) Data Entries Data Records Always better than external sorting! 46
47 Unclustered B+ Tree Used for Sorting Alternative (2) for data entries; each data entry contains rid of a data record. In general, one I/O per data record! Index (Directs search) Data Entries Data Records 47
48 Summary External sorting is important External merge sort minimizes disk I/O cost: Pass 0: Produces sorted runs of size B (# buffer pages). Later passes: merge runs. # of runs merged at a time depends on B In practice, # of passes rarely more than 2 or 3. 48
49 So far Index entries Data entries 49
50 So far SELECT S.sname FROM Reserves R, Sailors S WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5 RA Tree: sname bid=100 rating > 5 sid=sid Reserves Sailors 50
51 Now for the "middle piece" How is a query actually evaluated in your DBMS? Two aspects: How to implement each relational operator Several options depending if indexes available How to evaluate the whole plan Maybe optimize plan first using RA equivalences Choose implementation for each operator based on what's best for the overall plan (not just locally) 51
52 Implementing RA Operators Selection Projection Join Set operations Extended RA: GROUP BY, Aggregates 52
53 Readings [RG] Sec (implementing select) 53
54 Example relations Sailors (sid, sname, rating, age) Reserves(sid, bid, day, rname) Each tuple of Sailors 50 bytes long (80 tuples per page) and have 500 pages Each tuple of Reserves 40 bytes long (100 tuples per page) and have 1000 pages 54
55 Cost model Number of page I/Os incurred by algorithm Ignore things like benefits of sequential access, CPU computation cost, etc. Crude but disk I/O is likely to dominate cost so a good way to compare implementation alternatives 55
56 Select Operator SELECT * FROM Reserves R WHERE R.rname = 'Alice'; 56
57 Various cases Case 1: No index on any selection attribute File could be sorted on selection attr. or not Case 2: Have matching index on all selection attributes Case 3: Have matching index on some (but not all) selection attributes e.g. want to select "name = 'Alice' AND rating > 5" but only have index on name 57
58 No index on any selection attribute If file unsorted, need to scan whole thing 1000 I/Os Plus cost to write out results, but we ignore that aspect Want to compare evaluation costs for different implementations But all of them need to write out result If sorted, can do binary search Logarithmic cost for 1000 pages about 10 I/Os Plus cost to retrieve qualifying tuples (0 to 1000 extra I/Os) 58
59 Using indexes for selection Start with simple case where only one selection attribute (e.g. "name = 'Alice'") Suppose we have a matching index Should we use it? Probably, but not always! Depends on cost of accessing data through index 59
60 Using indexes for selection Tree index cost: Traverse tree from root Done only once 2 to 3 I/Os (trees are short) Scan leaf level pages to find all data entries Retrieve actual data records 60
61 Cost Components Tree Index Index Component 1: Traversing index File 61
62 Cost Components Tree Index Index Component 2: Traversing subset of data entries in index File 62
63 Cost Components Tree Index Index Component 3: Fetching actual data records (alternative 2 or 3) File 63
64 Clustered indexes If index is truly 100% clustered, don't need second component (scan from first leaf) But "clustered" may mean "data is stored close to search key order", not "exactly in search key order" In this case do need to access leaf index pages In your calculations, both are ok but make it very clear which you're choosing and why State your assumptions 64
65 What about hash indexes? Depends on implementation (linear, extendible etc) But typically small Reasonable assumption: 1 or 2 I/Os to find right bucket Plus cost of retrieving actual data records (depends on number of matching records) 65
66 Example Selection on Reserves, condition is "R.rname < 'C%'" Assume uniform distribution of names, so about 10% of relation should be retrieved 10K tuples, 100 pages 66
67 Example Have clustered B+ tree index on rname Traverse index to find first leaf page - 1 or 2 I/Os Start scan and retrieve tuples I/Os Have unclustered B+ tree index on rname Worst case, retrieving each tuple requires a separate I/O so 10K I/Os Much cheaper to just scan all 1000 original pages! Heuristic: scan cheaper than unclustered index if expect to retrieve more than 5% of tuples 67
68 Optimizations are possible Alternative 2 or 3, unclustered index Find qualifying data entries from index Sort the rids of the data entries to be retrieved Remember rid = (page ID, slot #) Fetch rids in order Ensures each data page is read from disk just once! Although number of data pages retrieved still likely to be more than with clustering 68
Implementing Relational Operators: Selection, Projection, Join. Database Management Systems, R. Ramakrishnan and J. Gehrke 1
Implementing Relational Operators: Selection, Projection, Join Database Management Systems, R. Ramakrishnan and J. Gehrke 1 Readings [RG] Sec. 14.1-14.4 Database Management Systems, R. Ramakrishnan and
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VI Lecture 17, March 24, 2015 Mohammad Hammoud Today Last Two Sessions: DBMS Internals- Part V External Sorting How to Start a Company in Five (maybe
Announcement. Reading Material. Overview of Query Evaluation. Overview of Query Evaluation. Overview of Query Evaluation 9/26/17
Announcement CompSci 516 Database Systems Lecture 10 Query Evaluation and Join Algorithms Project proposal pdf due on sakai by 5 pm, tomorrow, Thursday 09/27 One per group by any member Instructor: Sudeepa
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VI Lecture 14, March 12, 2014 Mohammad Hammoud Today Last Session: DBMS Internals- Part V Hash-based indexes (Cont d) and External Sorting Today s Session:
Evaluation of Relational Operations
Evaluation of Relational Operations Yanlei Diao UMass Amherst March 13 and 15, 2006 Slides Courtesy of R. Ramakrishnan and J. Gehrke 1 Relational Operations We will consider how to implement: Selection
Examples of Physical Query Plan Alternatives. Selected Material from Chapters 12, 14 and 15
Examples of Physical Query Plan Alternatives Selected Material from Chapters 12, 14 and 15 1 Query Optimization NOTE: SQL provides many ways to express a query. HENCE: System has many options for evaluating
Overview of Implementing Relational Operators and Query Evaluation
Overview of Implementing Relational Operators and Query Evaluation Chapter 12 Motivation: Evaluating Queries The same query can be evaluated in different ways. The evaluation strategy (plan) can make orders
Overview of Query Evaluation. Chapter 12
Overview of Query Evaluation Chapter 12 1 Outline Query Optimization Overview Algorithm for Relational Operations 2 Overview of Query Evaluation DBMS keeps descriptive data in system catalogs. SQL queries
Evaluation of Relational Operations
Evaluation of Relational Operations Chapter 14 Comp 521 Files and Databases Fall 2010 1 Relational Operations We will consider in more detail how to implement: Selection ( ) Selects a subset of rows from
Evaluation of Relational Operations: Other Techniques. Chapter 14 Sayyed Nezhadi
Evaluation of Relational Operations: Other Techniques Chapter 14 Sayyed Nezhadi Schema for Examples Sailors (sid: integer, sname: string, rating: integer, age: real) Reserves (sid: integer, bid: integer,
R & G Chapter 13. Implementation of single Relational Operations Choices depend on indexes, memory, stats, Joins Blocked nested loops:
Relational Query Optimization R & G Chapter 13 Review Implementation of single Relational Operations Choices depend on indexes, memory, stats, Joins Blocked nested loops: simple, exploits extra memory
Faloutsos 1. Carnegie Mellon Univ. Dept. of Computer Science Database Applications. Outline
Carnegie Mellon Univ. Dept. of Computer Science 15-415 - Database Applications Lecture #14: Implementation of Relational Operations (R&G ch. 12 and 14) 15-415 Faloutsos 1 introduction selection projection
Administriva. CS 133: Databases. General Themes. Goals for Today. Fall 2018 Lec 11 10/11 Query Evaluation Prof. Beth Trushkowsky
Administriva Lab 2 Final version due next Wednesday CS 133: Databases Fall 2018 Lec 11 10/11 Query Evaluation Prof. Beth Trushkowsky Problem sets PSet 5 due today No PSet out this week optional practice
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing
Overview of Query Evaluation. Overview of Query Evaluation
Overview of Query Evaluation Chapter 12 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Overview of Query Evaluation v Plan: Tree of R.A. ops, with choice of alg for each op. Each operator
CS330. Query Processing
CS330 Query Processing 1 Overview of Query Evaluation Plan: Tree of R.A. ops, with choice of alg for each op. Each operator typically implemented using a `pull interface: when an operator is `pulled for
CS542. Algorithms on Secondary Storage Sorting Chapter 13. Professor E. Rundensteiner. Worcester Polytechnic Institute
CS542 Algorithms on Secondary Storage Sorting Chapter 13. Professor E. Rundensteiner Lesson: Using secondary storage effectively Data too large to live in memory Regular algorithms on small scale only
Principles of Data Management. Lecture #9 (Query Processing Overview)
Principles of Data Management Lecture #9 (Query Processing Overview) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Today s Notable News v Midterm
CS 4604: Introduction to Database Management Systems. B. Aditya Prakash Lecture #10: Query Processing
CS 4604: Introduction to Database Management Systems B. Aditya Prakash Lecture #10: Query Processing Outline introduction selection projection join set & aggregate operations Prakash 2018 VT CS 4604 2
Cost-based Query Sub-System. Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications. Last Class.
Cost-based Query Sub-System Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications Queries Select * From Blah B Where B.blah = blah Query Parser Query Optimizer C. Faloutsos A. Pavlo
15-415/615 Faloutsos 1
Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications Lecture #14: Implementation of Relational Operations (R&G ch. 12 and 14) 15-415/615 Faloutsos 1 Outline introduction selection
Overview of Query Evaluation
Overview of Query Evaluation Chapter 12 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Overview of Query Evaluation Plan: Tree of R.A. ops, with choice of alg for each op. Each operator
Database Systems. Announcement. December 13/14, 2006 Lecture #10. Assignment #4 is due next week.
Database Systems ( 料 ) December 13/14, 2006 Lecture #10 1 Announcement Assignment #4 is due next week. 2 1 Overview of Query Evaluation Chapter 12 3 Outline Query evaluation (Overview) Relational Operator
Spring 2017 EXTERNAL SORTING (CH. 13 IN THE COW BOOK) 2/7/17 CS 564: Database Management Systems; (c) Jignesh M. Patel,
Spring 2017 EXTERNAL SORTING (CH. 13 IN THE COW BOOK) 2/7/17 CS 564: Database Management Systems; (c) Jignesh M. Patel, 2013 1 Motivation for External Sort Often have a large (size greater than the available
Implementation of Relational Operations
Implementation of Relational Operations Module 4, Lecture 1 Database Management Systems, R. Ramakrishnan 1 Relational Operations We will consider how to implement: Selection ( ) Selects a subset of rows
Database Management System
Database Management System Lecture Join * Some materials adapted from R. Ramakrishnan, J. Gehrke and Shawn Bowers Today s Agenda Join Algorithm Database Management System Join Algorithms Database Management
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? v A classic problem in computer science! v Data requested in sorted order e.g., find students in increasing
Dtb Database Systems. Announcement
Dtb Database Systems ( 資料庫系統 ) December 10, 2008 Lecture #11 1 Announcement Assignment #5 will be out on the course webpage today. 2 1 External Sorting Chapter 13 3 Why learn sorting again? O (n*n): bubble,
Evaluation of Relational Operations. Relational Operations
Evaluation of Relational Operations Chapter 14, Part A (Joins) Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Relational Operations v We will consider how to implement: Selection ( )
Query Evaluation Overview, cont.
Query Evaluation Overview, cont. Lecture 9 Feb. 29, 2016 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Architecture of a DBMS Query Compiler Execution Engine Index/File/Record
Implementation of Relational Operations. Introduction. CS 186, Fall 2002, Lecture 19 R&G - Chapter 12
Implementation of Relational Operations CS 186, Fall 2002, Lecture 19 R&G - Chapter 12 First comes thought; then organization of that thought, into ideas and plans; then transformation of those plans into
Implementation of Relational Operations: Other Operations
Implementation of Relational Operations: Other Operations Module 4, Lecture 2 Database Management Systems, R. Ramakrishnan 1 Simple Selections SELECT * FROM Reserves R WHERE R.rname < C% Of the form σ
CAS CS 460/660 Introduction to Database Systems. Query Evaluation II 1.1
CAS CS 460/660 Introduction to Database Systems Query Evaluation II 1.1 Cost-based Query Sub-System Queries Select * From Blah B Where B.blah = blah Query Parser Query Optimizer Plan Generator Plan Cost
Relational Query Optimization
Relational Query Optimization Module 4, Lectures 3 and 4 Database Management Systems, R. Ramakrishnan 1 Overview of Query Optimization Plan: Tree of R.A. ops, with choice of alg for each op. Each operator
Evaluation of relational operations
Evaluation of relational operations Iztok Savnik, FAMNIT Slides & Textbook Textbook: Raghu Ramakrishnan, Johannes Gehrke, Database Management Systems, McGraw-Hill, 3 rd ed., 2007. Slides: From Cow Book
Query Evaluation Overview, cont.
Query Evaluation Overview, cont. Lecture 9 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Architecture of a DBMS Query Compiler Execution Engine Index/File/Record Manager
Evaluation of Relational Operations
Evaluation of Relational Operations Chapter 12, Part A Database Management Systems, R. Ramakrishnan and J. Gehrke 1 Relational Operations We will consider how to implement: Selection ( ) Selects a subset
CompSci 516 Data Intensive Computing Systems
CompSci 516 Data Intensive Computing Systems Lecture 9 Join Algorithms and Query Optimizations Instructor: Sudeepa Roy CompSci 516: Data Intensive Computing Systems 1 Announcements Takeaway from Homework
Relational Query Optimization. Highlights of System R Optimizer
Relational Query Optimization Chapter 15 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Highlights of System R Optimizer v Impact: Most widely used currently; works well for < 10 joins.
CSE 444: Database Internals. Section 4: Query Optimizer
CSE 444: Database Internals Section 4: Query Optimizer Plan for Today Problem 1A, 1B: Estimating cost of a plan You try to compute the cost for 5 mins We will go over the solution together Problem 2: Sellinger
EXTERNAL SORTING. Sorting
EXTERNAL SORTING 1 Sorting A classic problem in computer science! Data requested in sorted order (sorted output) e.g., find students in increasing grade point average (gpa) order SELECT A, B, C FROM R
Overview of Query Processing
ICS 321 Fall 2013 Overview of Query Processing Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 11/20/2013 Lipyeow Lim -- University of Hawaii at Manoa 1
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part V Lecture 13, March 10, 2014 Mohammad Hammoud Today Welcome Back from Spring Break! Today Last Session: DBMS Internals- Part IV Tree-based (i.e., B+
Operator Implementation Wrap-Up Query Optimization
Operator Implementation Wrap-Up Query Optimization 1 Last time: Nested loop join algorithms: TNLJ PNLJ BNLJ INLJ Sort Merge Join Hash Join 2 General Join Conditions Equalities over several attributes (e.g.,
Query Processing and Query Optimization. Prof Monika Shah
Query Processing and Query Optimization Query Processing SQL Query Is in Library Cache? System catalog (Dict / Dict cache) Scan and verify relations Parse into parse tree (relational Calculus) View definitions
Announcements. Reading Material. Today. Different File Organizations. Selection of Indexes 9/24/17. CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 9 Index Selection and External Sorting Announcements Private project threads created on piazza Please use these threads (and not emails) for all communications on your
Sorting a file in RAM. External Sorting. Why Sort? 2-Way Sort of N pages Requires Minimum of 3 Buffers Pass 0: Read a page, sort it, write it.
Sorting a file in RAM External Sorting Chapter 13 Three steps: Read the entire file from disk into RAM Sort the records using a standard sorting procedure, such as Shell sort, heap sort, bubble sort, Write
External Sorting. Chapter 13. Comp 521 Files and Databases Fall
External Sorting Chapter 13 Comp 521 Files and Databases Fall 2012 1 Why Sort? A classic problem in computer science! Advantages of requesting data in sorted order gathers duplicates allows for efficient
Implementing Joins 1
Implementing Joins 1 Last Time Selection Scan, binary search, indexes Projection Duplicate elimination: sorting, hashing Index-only scans Joins 2 Tuple Nested Loop Join foreach tuple r in R do foreach
Last Class Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications
Last Class Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications C. Faloutsos A. Pavlo Lecture#12: External Sorting (R&G, Ch13) Static Hashing Extendible Hashing Linear Hashing Hashing
CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 9 Index Selection and External Sorting Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements Private project threads created on piazza
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VII Lecture 15, March 17, 2014 Mohammad Hammoud Today Last Session: DBMS Internals- Part VI Algorithms for Relational Operations Today s Session: DBMS
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part VIII Lecture 16, March 19, 2014 Mohammad Hammoud Today Last Session: DBMS Internals- Part VII Algorithms for Relational Operations (Cont d) Today s Session:
Overview of Query Processing. Evaluation of Relational Operations. Why Sort? Outline. Two-Way External Merge Sort. 2-Way Sort: Requires 3 Buffer Pages
Overview of Query Processing Query Parser Query Processor Evaluation of Relational Operations Query Rewriter Query Optimizer Query Executor Yanlei Diao UMass Amherst Lock Manager Access Methods (Buffer
NOTE: sorting using B-trees to be assigned for reading after we cover B-trees.
External Sorting Chapter 13 (Sec. 13-1-13.5): Ramakrishnan & Gehrke and Chapter 11 (Sec. 11.4-11.5): G-M et al. (R2) OR Chapter 2 (Sec. 2.4-2.5): Garcia-et Molina al. (R1) NOTE: sorting using B-trees to
Why Is This Important? Overview of Storage and Indexing. Components of a Disk. Data on External Storage. Accessing a Disk Page. Records on a Disk Page
Why Is This Important? Overview of Storage and Indexing Chapter 8 DB performance depends on time it takes to get the data from storage system and time to process Choosing the right index for faster access
Evaluation of Relational Operations: Other Techniques
Evaluation of Relational Operations: Other Techniques [R&G] Chapter 14, Part B CS4320 1 Using an Index for Selections Cost depends on #qualifying tuples, and clustering. Cost of finding qualifying data
Query Optimization. Schema for Examples. Motivating Example. Similar to old schema; rname added for variations. Reserves: Sailors:
Query Optimization Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Schema for Examples (sid: integer, sname: string, rating: integer, age: real) (sid: integer, bid: integer, day: dates,
Query Evaluation (i)
ICS 421 Spring 2010 Query Evaluation (i) Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 2/9/2010 Lipyeow Lim -- University of Hawaii at Manoa 1 Query Enumerate
Evaluation of Relational Operations: Other Techniques
Evaluation of Relational Operations: Other Techniques Chapter 14, Part B Database Management Systems 3ed, R. Ramakrishnan and Johannes Gehrke 1 Using an Index for Selections Cost depends on #qualifying
ECS 165B: Database System Implementa6on Lecture 7
ECS 165B: Database System Implementa6on Lecture 7 UC Davis April 12, 2010 Acknowledgements: por6ons based on slides by Raghu Ramakrishnan and Johannes Gehrke. Class Agenda Last 6me: Dynamic aspects of
Principles of Data Management. Lecture #12 (Query Optimization I)
Principles of Data Management Lecture #12 (Query Optimization I) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Today s Notable News v B+ tree
Query Optimization. Schema for Examples. Motivating Example. Similar to old schema; rname added for variations. Reserves: Sailors:
Query Optimization atabase Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Schema for Examples (sid: integer, sname: string, rating: integer, age: real) (sid: integer, bid: integer, day: dates,
Carnegie Mellon Univ. Dept. of Computer Science Database Applications. Why Sort? Why Sort? select... order by
Carnegie Mellon Univ. Dept. of Computer Science 15-415 - Database Applications Lecture 12: external sorting (R&G ch. 13) Faloutsos 15-415 1 Why Sort? Faloutsos 15-415 2 Why Sort? select... order by e.g.,
Evaluation of Relational Operations: Other Techniques
Evaluation of Relational Operations: Other Techniques Chapter 12, Part B Database Management Systems 3ed, R. Ramakrishnan and Johannes Gehrke 1 Using an Index for Selections v Cost depends on #qualifying
CompSci 516 Data Intensive Computing Systems. Lecture 11. Query Optimization. Instructor: Sudeepa Roy
CompSci 516 Data Intensive Computing Systems Lecture 11 Query Optimization Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements HW2 has been posted on sakai Due on Oct
CSE 444: Database Internals. Sec2on 4: Query Op2mizer
CSE 444: Database Internals Sec2on 4: Query Op2mizer Plan for Today Problem 1A, 1B: Es2ma2ng cost of a plan You try to compute the cost for 5 mins We go over the solu2on together Problem 2: Sellinger Op2mizer
Overview of DB & IR. ICS 624 Spring Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa
ICS 624 Spring 2011 Overview of DB & IR Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 1/12/2011 Lipyeow Lim -- University of Hawaii at Manoa 1 Example
Midterm Review CS634. Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke
Midterm Review CS634 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke Coverage Text, chapters 8 through 15 (hw1 hw4) PKs, FKs, E-R to Relational: Text, Sec. 3.2-3.5, to pg.
Relational Query Optimization. Overview of Query Evaluation. SQL Refresher. Yanlei Diao UMass Amherst March 8 and 13, 2007
Relational Query Optimization Yanlei Diao UMass Amherst March 8 and 13, 2007 Slide Content Courtesy of R. Ramakrishnan, J. Gehrke, and J. Hellerstein 1 Overview of Query Evaluation Query Evaluation Plan:
Schema for Examples. Query Optimization. Alternative Plans 1 (No Indexes) Motivating Example. Alternative Plans 2 With Indexes
Schema for Examples Query Optimization (sid: integer, : string, rating: integer, age: real) (sid: integer, bid: integer, day: dates, rname: string) Similar to old schema; rname added for variations. :
Hash-Based Indexing 165
Hash-Based Indexing 165 h 1 h 0 h 1 h 0 Next = 0 000 00 64 32 8 16 000 00 64 32 8 16 A 001 01 9 25 41 73 001 01 9 25 41 73 B 010 10 10 18 34 66 010 10 10 18 34 66 C Next = 3 011 11 11 19 D 011 11 11 19
Data on External Storage
Advanced Topics in DBMS Ch-1: Overview of Storage and Indexing By Syed khutubddin Ahmed Assistant Professor Dept. of MCA Reva Institute of Technology & mgmt. Data on External Storage Prg1 Prg2 Prg3 DBMS
CSIT5300: Advanced Database Systems
CSIT5300: Advanced Database Systems E11: Exercises on Query Optimization Dr. Kenneth LEUNG Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong SAR,
Relational Query Optimization. Overview of Query Evaluation. SQL Refresher. Yanlei Diao UMass Amherst October 23 & 25, 2007
Relational Query Optimization Yanlei Diao UMass Amherst October 23 & 25, 2007 Slide Content Courtesy of R. Ramakrishnan, J. Gehrke, and J. Hellerstein 1 Overview of Query Evaluation Query Evaluation Plan:
Evaluation of Relational Operations. SS Chung
Evaluation of Relational Operations SS Chung Cost Metric Query Processing Cost = Disk I/O Cost + CPU Computation Cost Disk I/O Cost = Disk Access Time + Data Transfer Time Disk Acess Time = Seek Time +
Review. Relational Query Optimization. Query Optimization Overview (cont) Query Optimization Overview. Cost-based Query Sub-System
Review Relational Query Optimization R & G Chapter 12/15 Implementation of single Relational Operations Choices depend on indexes, memory, stats, Joins Blocked nested loops: simple, exploits extra memory
Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M.
![Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M.](/thumbs/75/72297724.jpg "Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M.")
Spring 2017 QUERY PROCESSING [JOINS, SET OPERATIONS, AND AGGREGATES] 2/19/17 CS 564: Database Management Systems; (c) Jignesh M. Patel, 2013 1 Joins The focus here is on equijoins These are very common,
QUERY EXECUTION: How to Implement Relational Operations?
QUERY EXECUTION: How to Implement Relational Operations? 1 Introduction We ve covered the basic underlying storage, buffering, indexing and sorting technology Now we can move on to query processing Relational
RAID in Practice, Overview of Indexing
RAID in Practice, Overview of Indexing CS634 Lecture 4, Feb 04 2014 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke 1 Disks and Files: RAID in practice For a big enterprise
Modern Database Systems Lecture 1
Modern Database Systems Lecture 1 Aristides Gionis Michael Mathioudakis T.A.: Orestis Kostakis Spring 2016 logistics assignment will be up by Monday (you will receive email) due Feb 12 th if you re not
CS330. Some Logistics. Three Topics. Indexing, Query Processing, and Transactions. Next two homework assignments out today Extra lab session:
CS330 Indexing, Query Processing, and Transactions 1 Some Logistics Next two homework assignments out today Extra lab session: This Thursday, after class, in this room Bring your laptop fully charged Extra
Administrivia. Relational Query Optimization (this time we really mean it) Review: Query Optimization. Overview: Query Optimization
Relational Query Optimization (this time we really mean it) R&G hapter 15 Lecture 25 dministrivia Homework 5 mostly available It will be due after classes end, Monday 12/8 Only 3 more lectures left! Next
Overview of Storage and Indexing
Overview of Storage and Indexing Chapter 8 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Data on External Storage Disks: Can retrieve random page at fixed cost But reading several consecutive
Overview of Storage and Indexing
Overview of Storage and Indexing Chapter 8 How index-learning turns no student pale Yet holds the eel of science by the tail. -- Alexander Pope (1688-1744) Database Management Systems 3ed, R. Ramakrishnan
Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13!
![Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13!](/thumbs/78/78385651.jpg "Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13!")
Query Evaluation! References:! q [RG-3ed] Chapter 12, 13, 14, 15! q [SKS-6ed] Chapter 12, 13! q Overview! q Optimization! q Measures of Query Cost! Query Evaluation! q Sorting! q Join Operation! q Other
Query Processing: The Basics. External Sorting
Query Processing: The Basics Chapter 10 1 External Sorting Sorting is used in implementing many relational operations Problem: Relations are typically large, do not fit in main memory So cannot use traditional
Last Class. Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications. Why do we need to sort? Today's Class
Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications C. Faloutsos A. Pavlo Lecture#12: External Sorting (R&G, Ch13) Static Hashing Extendible Hashing Linear Hashing Hashing vs.
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part V Lecture 15, March 15, 2015 Mohammad Hammoud Today Last Session: DBMS Internals- Part IV Tree-based (i.e., B+ Tree) and Hash-based (i.e., Extendible
CSE 444: Database Internals. Lectures 5-6 Indexing
CSE 444: Database Internals Lectures 5-6 Indexing 1 Announcements HW1 due tonight by 11pm Turn in an electronic copy (word/pdf) by 11pm, or Turn in a hard copy in my office by 4pm Lab1 is due Friday, 11pm
Database Applications (15-415)
Database Applications (15-415) DMS Internals- Part X Lecture 21, April 7, 2015 Mohammad Hammoud Last Session: DMS Internals- Part IX Query Optimization Today Today s Session: DMS Internals- Part X Query
Final Exam Review. Kathleen Durant CS 3200 Northeastern University Lecture 22
Final Exam Review Kathleen Durant CS 3200 Northeastern University Lecture 22 Outline for today Identify topics for the final exam Discuss format of the final exam What will be provided for you and what
RELATIONAL OPERATORS #1
RELATIONAL OPERATORS #1 CS 564- Spring 2018 ACKs: Jeff Naughton, Jignesh Patel, AnHai Doan WHAT IS THIS LECTURE ABOUT? Algorithms for relational operators: select project 2 ARCHITECTURE OF A DBMS query
Indexing. Chapter 8, 10, 11. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
Indexing Chapter 8, 10, 11 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Tree-Based Indexing The data entries are arranged in sorted order by search key value. A hierarchical search
QUERY OPTIMIZATION E Jayant Haritsa Computer Science and Automation Indian Institute of Science. JAN 2014 Slide 1 QUERY OPTIMIZATION
E0 261 Jayant Haritsa Computer Science and Automation Indian Institute of Science JAN 2014 Slide 1 Database Engines Main Components Query Processing Transaction Processing Access Methods JAN 2014 Slide
TotalCost = 3 (1, , 000) = 6, 000
156 Chapter 12 HASH JOIN: Now both relations are the same size, so we can treat either one as the smaller relation. With 15 buffer pages the first scan of S splits it into 14 buckets, each containing about
Overview of Storage and Indexing
Overview of Storage and Indexing Chapter 8 How index-learning turns no student pale Yet holds the eel of science by the tail. -- Alexander Pope (1688-1744) Database Management Systems 3ed, R. Ramakrishnan
IMPORTANT: Circle the last two letters of your class account:
Fall 2001 University of California, Berkeley College of Engineering Computer Science Division EECS Prof. Michael J. Franklin FINAL EXAM CS 186 Introduction to Database Systems NAME: STUDENT ID: IMPORTANT:
Why Sort? Data requested in sorted order. Sor,ng is first step in bulk loading B+ tree index. e.g., find students in increasing GPA order
External Sor,ng Outline Exam will be graded a5er everyone takes it There are two,mes to be fair on an exam, when it s wriaen and when it s graded you only need to trust that I ll be fair at one of them.
Administrivia. CS 133: Databases. Cost-based Query Sub-System. Goals for Today. Midterm on Thursday 10/18. Assignments
Administrivia Midterm on Thursday 10/18 CS 133: Databases Fall 2018 Lec 12 10/16 Prof. Beth Trushkowsky Assignments Lab 3 starts after fall break No problem set out this week Goals for Today Cost-based