CS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #5: Hashing and Sor5ng
|
|
|
- Ashlie Adams
- 5 years ago
- Views:
Transcription
1 CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #5: Hashing and Sor5ng
2 (Sta7c) Hashing Problem: find EMP record with ssn=123 What if disk space was free, and 5me was at premium? Prakash 2014 VT CS
3 Hashing A: Brilliant idea: key- to- address transforma5on: #0 page 123; Smith; Main str #123 page #999,999,999 Prakash 2014 VT CS
4 Hashing Since space is NOT free: use M, instead of 999,999,999 slots hash func5on: h(key) = slot- id #0 page 123; Smith; Main str #123 page #999,999,999 Prakash 2014 VT CS
5 Hashing Typically: each hash bucket is a page, holding many records: #0 page 123; Smith; Main str #h(123) M Prakash 2014 VT CS
6 Hashing No5ce: could have clustering, or non- clustering versions: #0 page 123; Smith; Main str. #h(123) M Prakash 2014 VT CS
7 Hashing No5ce: could have clustering, or non- clustering versions: #0 page... EMP file #h(123) ; Johnson; Forbes ave 123; Smith; Main str. M Prakash 2014 VT CS ; Tompson; Fi]h ave...
8 Design decisions 1) formula h() for hashing func5on 2) size of hash table M 3) collision resolu5on method Prakash 2014 VT CS
9 Design decisions - func7ons Goal: uniform spread of keys over hash buckets Popular choices: Division hashing Mul5plica5on hashing Prakash 2014 VT CS
10 Division hashing h(x) = (a*x+b) mod M eg., h(ssn) = (ssn) mod 1,000 gives the last three digits of ssn M: size of hash table - choose a prime number, defensively (why?) Prakash 2014 VT CS
11 Division hashing eg., M=2; hash on driver- license number (dln), where last digit is gender (0/1 = M/ F) in an army unit with predominantly male soldiers Thus: avoid cases where M and keys have common divisors - prime M guards against that! Prakash 2014 VT CS
12 Mul7plica7on hashing h(x) = [ frac-onal- part- of ( x * φ ) ] * M φ: golden ra5o ( = ( sqrt(5)- 1)/2 ) in general, we need an irra5onal number advantage: M need not be a prime number but φ must be irra5onal Prakash 2014 VT CS
13 Other hashing func7ons quadra5c hashing (bad)... Prakash 2014 VT CS
14 Other hashing func7ons quadra5c hashing (bad)... conclusion: use division hashing Prakash 2014 VT CS
15 Size of hash table eg., 50,000 employees, 10 employee- records / page Q: M=?? pages/buckets/slots Prakash 2014 VT CS
16 Size of hash table eg., 50,000 employees, 10 employees/page Q: M=?? pages/buckets/slots A: u5liza5on ~ 90% and M: prime number Eg., in our case: M= closest prime to 50,000/10 / 0.9 = 5,555 Prakash 2014 VT CS
17 Collision resolu7on Q: what is a collision? A:?? Prakash 2014 VT CS
18 Collision resolu7on #0 page 123; Smith; Main str. #h(123) M Prakash 2014 VT CS
19 Collision resolu7on Q: what is a collision? A:?? Q: why worry about collisions/overflows? (recall that buckets are ~90% full) A: birthday paradox Prakash 2014 VT CS
20 Collision resolu7on open addressing linear probing (ie., put to next slot/bucket) re- hashing separate chaining (ie., put links to overflow pages) Prakash 2014 VT CS
21 Collision resolu7on linear probing: #0 page 123; Smith; Main str. #h(123) M Prakash 2014 VT CS
22 Collision resolu7on re- hashing #0 page h1() 123; Smith; Main str. #h(123) h2() M Prakash 2014 VT CS
23 Collision resolu7on separate chaining 123; Smith; Main str. Prakash 2014 VT CS
24 Design decisions - conclusions func5on: division hashing h(x) = ( a*x+b ) mod M size M: ~90% u5l.; prime number. collision resolu5on: separate chaining easier to implement (dele5ons!); no danger of becoming full Prakash 2014 VT CS
25 Problem with sta7c hashing problem: overflow? problem: underflow? (underu5liza5on) Prakash 2014 VT CS
26 Solu7on: Dynamic/extendible hashing idea: shrink / expand hash table on demand....dynamic hashing Details: how to grow gracefully, on overflow? Many solu5ons - One of them: extendible hashing [Fagin et al] Prakash 2014 VT CS
27 Extendible hashing #0 page 123; Smith; Main str. #h(123) M Prakash 2014 VT CS
28 Extendible hashing solu7on: split the bucket in two #0 page 123; Smith; Main str. #h(123) M Prakash 2014 VT CS
29 Extendible hashing in detail: keep a directory, with ptrs to hash- buckets Q: how to divide contents of bucket in two? A: hash each key into a very long bit string; keep only as many bits as needed Eventually: Prakash 2014 VT CS
30 Extendible hashing directory Prakash 2014 VT CS
31 Extendible hashing directory Prakash 2014 VT CS
32 Extendible hashing directory split on 3- rd bit Prakash 2014 VT CS
33 Extendible hashing directory new page / bucket Prakash 2014 VT CS
34 Extendible hashing directory (doubled) new page / bucket Prakash 2014 VT CS
35 Extendible hashing BEFORE AFTER Prakash 2014 VT CS
36 Extendible hashing Summary: directory doubles on demand or halves, on shrinking files needs local and global depth Prakash 2014 VT CS
37 Linear hashing - overview Mo5va5on main idea search algo inser5on/split algo dele5on Prakash 2014 VT CS
38 Linear hashing Mo5va5on: ext. hashing needs directory etc etc; which doubles (ouch!) Q: can we do something simpler, with smoother growth? Prakash 2014 VT CS
39 Linear hashing Mo5va5on: ext. hashing needs directory etc etc; which doubles (ouch!) Q: can we do something simpler, with smoother growth? A: split buckets from le] to right, regardless of which one overflowed ( crazy, but it works well!) - Eg.: Prakash 2014 VT CS
40 Linear hashing Ini5ally: h(x) = x mod N (N=4 here) Assume capacity: 3 records / bucket Insert key 17 bucket- id Prakash 2014 VT CS
41 Linear hashing Ini5ally: h(x) = x mod N (N=4 here) 17 overflow of bucket#1 bucket- id Prakash 2014 VT CS
42 Linear hashing Ini5ally: h(x) = x mod N (N=4 here) 17 overflow of bucket#1 Split #0, anyway!!! bucket- id Prakash 2014 VT CS
43 Linear hashing Ini5ally: h(x) = x mod N (N=4 here) 17 Split #0, anyway!!! Q: But, how? bucket- id Prakash 2014 VT CS
44 Linear hashing A: use two h.f.: h0(x) = x mod N h1(x) = x mod (2*N) 17 bucket- id Prakash 2014 VT CS
45 Linear hashing - a]er split: A: use two h.f.: h0(x) = x mod N h1(x) = x mod (2*N) bucket- id Prakash 2014 VT CS
46 Linear hashing - a]er split: A: use two h.f.: h0(x) = x mod N h1(x) = x mod (2*N) bucket- id overflow Prakash 2014 VT CS
47 Linear hashing - a]er split: A: use two h.f.: h0(x) = x mod N h1(x) = x mod (2*N) split ptr bucket- id overflow Prakash 2014 VT CS
48 Linear hashing - searching? h0(x) = x mod N the splieed ones) (for the un- split buckets) h1(x) = x mod (2*N) (for split ptr bucket- id overflow Prakash 2014 VT CS
49 Linear hashing - searching? Q1: find key 6? Q2: find key 4? Q3: key 8? split ptr bucket- id overflow Prakash 2014 VT CS
50 Linear hashing - searching? Algo to find key k : compute b= h0(k); if b<split- ptr, compute b=h1(k) search bucket b Prakash 2014 VT CS
51 Linear hashing - inser7on? Algo: insert key k compute appropriate bucket b if the overflow criterion is true split the bucket of split- ptr split- ptr ++ (*) Prakash 2014 VT CS
52 Linear hashing - inser7on? no5ce: overflow criterion is up to us!! Q: sugges5ons? Prakash 2014 VT CS
53 Linear hashing - inser7on? no5ce: overflow criterion is up to us!! Q: sugges5ons? A1: space u5liza5on >= u- max Prakash 2014 VT CS
54 Linear hashing - inser7on? no5ce: overflow criterion is up to us!! Q: sugges5ons? A1: space u5liza5on > u- max A2: avg length of ovf chains > max- len A3:... Prakash 2014 VT CS
55 Linear hashing - inser7on? Algo: insert key k compute appropriate bucket b if the overflow criterion is true split the bucket of split- ptr split- ptr ++ (*) what if we reach the right edge?? Prakash 2014 VT CS
56 Linear hashing - split now? h0(x) = x mod N the splieed ones) (for the un- split buckets) h1(x) = x mod (2*N) for split ptr Prakash 2014 VT CS
57 Linear hashing - split now? h0(x) = x mod N the splieed ones) (for the un- split buckets) h1(x) = x mod (2*N) (for split ptr Prakash 2014 VT CS
58 Linear hashing - split now? h0(x) = x mod N the splieed ones) (for the un- split buckets) h1(x) = x mod (2*N) (for split ptr Prakash 2014 VT CS
59 Linear hashing - split now? h0(x) = x mod N the splieed ones) (for the un- split buckets) h1(x) = x mod (2*N) (for split ptr Prakash 2014 VT CS
60 Linear hashing - split now? this state is called full expansion split ptr Prakash 2014 VT CS
61 Linear hashing - observa7ons In general, at any point of 5me, we have at most two h.f. ac5ve, of the form: h n (x) = x mod (N * 2 n ) h n+1 (x) = x mod (N * 2 n+1 ) (aher a full expansion, we have only one h.f.) Prakash 2014 VT CS
62 Linear hashing - dele7on? reverse of inser5on: Prakash 2014 VT CS
63 Linear hashing - dele7on? reverse of inser5on: if the underflow criterion is met contract! Prakash 2014 VT CS
64 Linear hashing - how to contract? h0(x) = mod N (for the un- split buckets) h1(x) = mod (2*N) (for the splieed ones) split ptr Prakash 2014 VT CS
65 Linear hashing - how to contract? h0(x) = mod N (for the un- split buckets) h1(x) = mod (2*N) (for the splieed ones) split ptr Prakash 2014 VT CS
66 Hashing - pros? Prakash 2014 VT CS
67 Hashing - pros? Speed, on exact match queries on the average Prakash 2014 VT CS
68 B(+)- trees - pros? Prakash 2014 VT CS
69 B(+)- trees - pros? Speed on search: exact match queries, worst case range queries nearest- neighbor queries Speed on inser5on + dele5on smooth growing and shrinking (no re- org) Prakash 2014 VT CS
70 Conclusions B- trees and variants: in all DBMSs hash indices: in some (but hashing in useful for joins ) Prakash 2014 VT CS
71 SORTING Prakash 2014 VT CS
72 Why Sort? Prakash 2014 VT CS
73 Why Sort? select... order by e.g., find students in increasing gpa order bulk loading B+ tree index. duplicate eliminalon (select distinct) select... group by Sort- merge join algorithm involves sor5ng. Prakash 2014 VT CS
74 Outline two- way merge sort external merge sort fine- tunings B+ trees for sor5ng Prakash 2014 VT CS
75 2- Way Sort: Requires 3 Buffers Pass 0: Read a page, sort it, write it. only one buffer page is used Pass 1, 2, 3,, etc.: requires 3 buffer pages merge pairs of runs into runs twice as long three buffer pages used. INPUT 1 INPUT 2 OUTPUT Disk Main memory buffers Disk Prakash 2014 VT CS
76 Two- Way External Merge Sort Each pass we read + write each page in file. 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 2,3 4,4 6,7 8,9 4,7 8,9 1,3 5,6 2 1,2 3,5 6 Input file PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 Prakash 2014 VT CS ,2 2,3 3,4 4,5 6,6 7,8 8-page runs
77 Two- Way External Merge Sort Each pass we read + write each page in file. 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 2,3 4,4 6,7 8,9 4,7 8,9 1,3 5,6 2 1,2 3,5 6 Input file PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 Prakash 2014 VT CS ,2 2,3 3,4 4,5 6,6 7,8 8-page runs
78 Two- Way External Merge Sort Each pass we read + write each page in file. 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 2,3 4,4 6,7 8,9 4,7 8,9 1,3 5,6 2 1,2 3,5 6 Input file PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 Prakash 2014 VT CS ,2 2,3 3,4 4,5 6,6 7,8 8-page runs
79 Two- Way External Merge Sort Each pass we read + write each page in file. 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 2,3 4,4 6,7 8,9 4,7 8,9 1,3 5,6 2 1,2 3,5 6 Input file PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 Prakash 2014 VT CS ,2 2,3 3,4 4,5 6,6 7,8 8-page runs
80 Two- Way External Merge Sort Each pass we read + write each page in file. N pages in the file => = log + 2 N 1 So total cost is: ( log N + ) 2 N 1 2 Idea: Divide and conquer: sort subfiles and merge 3,4 6,2 9,4 8,7 5,6 3,1 2 3,4 2,6 4,9 7,8 5,6 1,3 2 2,3 4,6 Input file Prakash 2014 VT CS ,3 4,4 6,7 8,9 4,7 8,9 1,2 2,3 3,4 4,5 6,6 7,8 1,3 5,6 2 1,2 3,5 6 PASS 0 1-page runs PASS 1 2-page runs PASS 2 4-page runs PASS 3 8-page runs
81 External merge sort B > 3 buffers Q1: how to sort? Q2: cost? Prakash 2014 VT CS
82 General External Merge Sort B>3 buffer pages. How to sort a file with N pages? Disk B Main memory buffers Disk Prakash 2014 VT CS
83 General External Merge Sort Pass 0: use B buffer pages. Produce N / B sorted runs of B pages each. Pass 1, 2,, etc.: merge B- 1 runs. INPUT 1... INPUT 2... OUTPUT... Disk INPUT B-1 B Main memory buffers Disk Prakash 2014 VT CS
84 Sor7ng create sorted runs of size B (how many?) merge them (how?) B Prakash 2014 VT CS
85 Sor7ng create sorted runs of size B merge first B- 1 runs into a sorted run of (B- 1) *B,... B Prakash 2014 VT CS
86 Sor7ng How many steps we need to do? i, where B*(B- 1)^i > N How many reads/writes per step? N+N B Prakash 2014 VT CS
87 Cost of External Merge Sort Number of passes: Cost = 2N * (# of passes) log B N / B Prakash 2014 VT CS
88 Cost of External Merge Sort E.g., with 5 buffer pages, to sort 108 page file: Pass 0: 108 / 5 = 22 sorted runs of 5 pages each (last run is only 3 pages) Pass 1: 22 / 4 = 6 sorted runs of 20 pages each (last run is only 8 pages) Pass 2: 2 sorted runs, 80 pages and 28 pages Pass 3: Sorted file of 108 pages Formula check: log 4 22 = à 4 passes Prakash 2014 VT CS
89 Number of Passes of External Sort ( I/O cost is 2N 5mes number of passes) N B=3 B=5 B=9 B=17 B=129 B= , , , ,000, ,000, ,000, ,000,000, Prakash 2014 VT CS
90 Internal Sort Algorithm Quicksort is a fast way to sort in memory. Prakash 2014 VT CS
91 Blocked I/O & double- buffering So far, we assumed random disk access Cost changes, if we consider that runs are wriyen (and read) sequen5ally What could we do to exploit it? Prakash 2014 VT CS
92 Blocked I/O & double- buffering So far, we assumed random disk access Cost changes, if we consider that runs are wriyen (and read) sequen5ally What could we do to exploit it? A1: Blocked I/O (exchange a few r.d.a for several sequen5al ones) A2: double- buffering Prakash 2014 VT CS
93 Double Buffering To reduce wait 5me for I/O request to complete, can prefetch into `shadow block. Poten5ally, more passes; in prac5ce, most files s5ll sorted in 2-3 passes. INPUT 1 INPUT 1' INPUT 2 INPUT 2' OUTPUT OUTPUT' Disk INPUT k b block size Disk INPUT k' Prakash 2014 VT CS
94 Using B+ Trees for Sor7ng Scenario: Table to be sorted has B+ tree index on sor5ng column(s). Idea: Can retrieve records in order by traversing leaf pages. Is this a good idea? Cases to consider: B+ tree is clustered B+ tree is not clustered Prakash 2014 VT CS
95 Using B+ Trees for Sor7ng Scenario: Table to be sorted has B+ tree index on sor5ng column(s). Idea: Can retrieve records in order by traversing leaf pages. Is this a good idea? Cases to consider: B+ tree is clustered Good idea! B+ tree is not clustered Could be a very bad idea! Prakash 2014 VT CS
96 Clustered B+ Tree Used for Sor7ng Cost: root to the le]- most leaf, then retrieve all leaf pages (Alterna5ve 1) Index (Directs search) Data Entries ("Sequence set") Data Records Always beyer than external sor5ng! Prakash 2014 VT CS
97 Unclustered B+ Tree Used for Sor7ng Alterna5ve (2) for data entries; each data entry contains rid of a data record. In general, one I/O per data record! Index (Directs search) Data Entries ("Sequence set") Data Records Prakash 2014 VT CS
98 External Sor7ng vs. Unclustered Index N Sorting p=1 p=10 p= ,000 10,000 1,000 2,000 1,000 10, ,000 10,000 40,000 10, ,000 1,000, , , ,000 1,000,000 10,000,000 1,000,000 8,000,000 1,000,000 10,000, ,000,000 10,000,000 80,000,000 10,000, ,000,000 1,000,000,000 p: # of records per page B=1,000 and block size=32 for sor-ng p=100 is the more realis-c value. Prakash 2014 VT CS
99 Summary External sor5ng is important External merge sort minimizes disk I/O cost: Pass 0: Produces sorted runs of size B (# buffer pages). Later passes: merge runs. Clustered B+ tree is good for sor5ng; unclustered tree is usually very bad. Prakash 2014 VT CS
Carnegie Mellon Univ. Dept. of Computer Science /615 DB Applications. Outline. (Static) Hashing. Faloutsos - Pavlo CMU SCS /615
Faloutsos - Pavlo 15-415/615 Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 DB Applications Faloutsos & Pavlo Lecture#11 (R&G ch. 11) Hashing (static) hashing extendible hashing linear hashing
Carnegie Mellon Univ. Dept. of Computer Science Database Applications. Why Sort? Why Sort? select... order by
Carnegie Mellon Univ. Dept. of Computer Science 15-415 - Database Applications Lecture 12: external sorting (R&G ch. 13) Faloutsos 15-415 1 Why Sort? Faloutsos 15-415 2 Why Sort? select... order by e.g.,
Why Sort? Data requested in sorted order. Sor,ng is first step in bulk loading B+ tree index. e.g., find students in increasing GPA order
External Sor,ng Outline Exam will be graded a5er everyone takes it There are two,mes to be fair on an exam, when it s wriaen and when it s graded you only need to trust that I ll be fair at one of them.
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part V Lecture 13, March 10, 2014 Mohammad Hammoud Today Welcome Back from Spring Break! Today Last Session: DBMS Internals- Part IV Tree-based (i.e., B+
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? A classic problem in computer science! Data requested in sorted order e.g., find students in increasing
CS542. Algorithms on Secondary Storage Sorting Chapter 13. Professor E. Rundensteiner. Worcester Polytechnic Institute
CS542 Algorithms on Secondary Storage Sorting Chapter 13. Professor E. Rundensteiner Lesson: Using secondary storage effectively Data too large to live in memory Regular algorithms on small scale only
Last Class. Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications. Why do we need to sort? Today's Class
Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications C. Faloutsos A. Pavlo Lecture#12: External Sorting (R&G, Ch13) Static Hashing Extendible Hashing Linear Hashing Hashing vs.
Last Class Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications
Last Class Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications C. Faloutsos A. Pavlo Lecture#12: External Sorting (R&G, Ch13) Static Hashing Extendible Hashing Linear Hashing Hashing
CARNEGIE MELLON UNIVERSITY DEPT. OF COMPUTER SCIENCE DATABASE APPLICATIONS
CARNEGIE MELLON UNIVERSITY DEPT. OF COMPUTER SCIENCE 15-415 DATABASE APPLICATIONS C. Faloutsos Indexing and Hashing 15-415 Database Applications http://www.cs.cmu.edu/~christos/courses/dbms.s00/ general
External Sorting. Chapter 13. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
External Sorting Chapter 13 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Why Sort? v A classic problem in computer science! v Data requested in sorted order e.g., find students in increasing
Symbol Table. Symbol table is used widely in many applications. dictionary is a kind of symbol table data dictionary is database management
Hashing Symbol Table Symbol table is used widely in many applications. dictionary is a kind of symbol table data dictionary is database management In general, the following operations are performed on
Lecture 8 Index (B+-Tree and Hash)
CompSci 516 Data Intensive Computing Systems Lecture 8 Index (B+-Tree and Hash) Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 HW1 due tomorrow: Announcements Due on 09/21 (Thurs),
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part V Lecture 15, March 15, 2015 Mohammad Hammoud Today Last Session: DBMS Internals- Part IV Tree-based (i.e., B+ Tree) and Hash-based (i.e., Extendible
Module 3: Hashing Lecture 9: Static and Dynamic Hashing. The Lecture Contains: Static hashing. Hashing. Dynamic hashing. Extendible hashing.
The Lecture Contains: Hashing Dynamic hashing Extendible hashing Insertion file:///c /Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture9/9_1.htm[6/14/2012
Selection Queries. to answer a selection query (ssn=10) needs to traverse a full path.
Hashing B+-tree is perfect, but... Selection Queries to answer a selection query (ssn=) needs to traverse a full path. In practice, 3-4 block accesses (depending on the height of the tree, buffering) Any
5. Hashing. 5.1 General Idea. 5.2 Hash Function. 5.3 Separate Chaining. 5.4 Open Addressing. 5.5 Rehashing. 5.6 Extendible Hashing. 5.
5. Hashing 5.1 General Idea 5.2 Hash Function 5.3 Separate Chaining 5.4 Open Addressing 5.5 Rehashing 5.6 Extendible Hashing Malek Mouhoub, CS340 Fall 2004 1 5. Hashing Sequential access : O(n). Binary
Topics to Learn. Important concepts. Tree-based index. Hash-based index
CS143: Index 1 Topics to Learn Important concepts Dense index vs. sparse index Primary index vs. secondary index (= clustering index vs. non-clustering index) Tree-based vs. hash-based index Tree-based
Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2014/15
Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2014/15 Lecture II: Indexing Part I of this course Indexing 3 Database File Organization and Indexing Remember: Database tables
Hash-Based Indexing 1
Hash-Based Indexing 1 Tree Indexing Summary Static and dynamic data structures ISAM and B+ trees Speed up both range and equality searches B+ trees very widely used in practice ISAM trees can be useful
Hash-Based Indexes. Chapter 11
Hash-Based Indexes Chapter 11 1 Introduction : Hash-based Indexes Best for equality selections. Cannot support range searches. Static and dynamic hashing techniques exist: Trade-offs similar to ISAM vs.
CS143: Index. Book Chapters: (4 th ) , (5 th ) , , 12.10
CS143: Index Book Chapters: (4 th ) 12.1-3, 12.5-8 (5 th ) 12.1-3, 12.6-8, 12.10 1 Topics to Learn Important concepts Dense index vs. sparse index Primary index vs. secondary index (= clustering index
Indexing: Overview & Hashing. CS 377: Database Systems
Indexing: Overview & Hashing CS 377: Database Systems Recap: Data Storage Data items Records Memory DBMS Blocks blocks Files Different ways to organize files for better performance Disk Motivation for
Data and File Structures Chapter 11. Hashing
Data and File Structures Chapter 11 Hashing 1 Motivation Sequential Searching can be done in O(N) access time, meaning that the number of seeks grows in proportion to the size of the file. B-Trees improve
Lecture 13. Lecture 13: B+ Tree
Lecture 13 Lecture 13: B+ Tree Lecture 13 Announcements 1. Project Part 2 extension till Friday 2. Project Part 3: B+ Tree coming out Friday 3. Poll for Nov 22nd 4. Exam Pickup: If you have questions,
TUTORIAL ON INDEXING PART 2: HASH-BASED INDEXING
CSD Univ. of Crete Fall 07 TUTORIAL ON INDEXING PART : HASH-BASED INDEXING CSD Univ. of Crete Fall 07 Hashing Buckets: Set up an area to keep the records: Primary area Divide primary area into buckets
Kathleen Durant PhD Northeastern University CS Indexes
Kathleen Durant PhD Northeastern University CS 3200 Indexes Outline for the day Index definition Types of indexes B+ trees ISAM Hash index Choosing indexed fields Indexes in InnoDB 2 Indexes A typical
Chapter 6. Hash-Based Indexing. Efficient Support for Equality Search. Architecture and Implementation of Database Systems Summer 2014
Chapter 6 Efficient Support for Equality Architecture and Implementation of Database Systems Summer 2014 (Split, Rehashing) Wilhelm-Schickard-Institut für Informatik Universität Tübingen 1 We now turn
Introduction to Data Management. Lecture 15 (More About Indexing)
Introduction to Data Management Lecture 15 (More About Indexing) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Announcements v HW s and quizzes:
Hashing Techniques. Material based on slides by George Bebis
Hashing Techniques Material based on slides by George Bebis https://www.cse.unr.edu/~bebis/cs477/lect/hashing.ppt The Search Problem Find items with keys matching a given search key Given an array A, containing
Access Methods. Basic Concepts. Index Evaluation Metrics. search key pointer. record. value. Value
Access Methods This is a modified version of Prof. Hector Garcia Molina s slides. All copy rights belong to the original author. Basic Concepts search key pointer Value record? value Search Key - set of
Hash-Based Indexes. Chapter 11. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
Hash-Based Indexes Chapter Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Introduction As for any index, 3 alternatives for data entries k*: Data record with key value k
University of California, Berkeley. CS 186 Introduction to Databases, Spring 2014, Prof. Dan Olteanu MIDTERM
University of California, Berkeley CS 186 Introduction to Databases, Spring 2014, Prof. Dan Olteanu MIDTERM This is a closed book examination sided). but you are allowed one 8.5 x 11 sheet of notes (double
CS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #4: Constraints, Storing and Indexes
CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #4: Constraints, Storing and Indexes Constraints in Rela;onal Algebra and SQL Maintaining Integrity of Data Data is dirty. How does an applica>on
Evaluation of Relational Operations
Evaluation of Relational Operations Yanlei Diao UMass Amherst March 13 and 15, 2006 Slides Courtesy of R. Ramakrishnan and J. Gehrke 1 Relational Operations We will consider how to implement: Selection
Module 5: Hash-Based Indexing
Module 5: Hash-Based Indexing Module Outline 5.1 General Remarks on Hashing 5. Static Hashing 5.3 Extendible Hashing 5.4 Linear Hashing Web Forms Transaction Manager Lock Manager Plan Executor Operator
Module 5: Hashing. CS Data Structures and Data Management. Reza Dorrigiv, Daniel Roche. School of Computer Science, University of Waterloo
Module 5: Hashing CS 240 - Data Structures and Data Management Reza Dorrigiv, Daniel Roche School of Computer Science, University of Waterloo Winter 2010 Reza Dorrigiv, Daniel Roche (CS, UW) CS240 - Module
Data Organization B trees
Data Organization B trees Data organization and retrieval File organization can improve data retrieval time SELECT * FROM depositors WHERE bname= Downtown 100 blocks 200 recs/block Query returns 150 records
External Sorting Implementing Relational Operators
External Sorting Implementing Relational Operators 1 Readings [RG] Ch. 13 (sorting) 2 Where we are Working our way up from hardware Disks File abstraction that supports insert/delete/scan Indexing for
Hashed-Based Indexing
Topics Hashed-Based Indexing Linda Wu Static hashing Dynamic hashing Extendible Hashing Linear Hashing (CMPT 54 4-) Chapter CMPT 54 4- Static Hashing An index consists of buckets 0 ~ N-1 A bucket consists
THE B+ TREE INDEX. CS 564- Spring ACKs: Jignesh Patel, AnHai Doan
THE B+ TREE INDEX CS 564- Spring 2018 ACKs: Jignesh Patel, AnHai Doan WHAT IS THIS LECTURE ABOUT? The B+ tree index Basics Search/Insertion/Deletion Design & Cost 2 INDEX RECAP We have the following query:
Database Applications (15-415)
Database Applications (15-415) DBMS Internals- Part IV Lecture 14, March 10, 015 Mohammad Hammoud Today Last Two Sessions: DBMS Internals- Part III Tree-based indexes: ISAM and B+ trees Data Warehousing/
CS 5614: (Big) Data Management Systems. B. Aditya Prakash Lecture #4: Constraints, Storing and Indexes
CS 5614: (Big) Data Management Systems B. Aditya Prakash Lecture #4: Constraints, Storing and Indexes Constraints in Rela;onal Algebra and SQL Maintaining Integrity of Data Data is dirty. How does an applica>on
External Sorting. Chapter 13. Comp 521 Files and Databases Fall
External Sorting Chapter 13 Comp 521 Files and Databases Fall 2012 1 Why Sort? A classic problem in computer science! Advantages of requesting data in sorted order gathers duplicates allows for efficient
Hashing. Given a search key, can we guess its location in the file? Goal: Method: hash keys into addresses
Hashing Given a search key, can we guess its location in the file? Goal: Support equality searches in one disk access! Method: hash keys into addresses key page 1 Types of Hashing What does H(K) point
Introduction to Data Management. Lecture 21 (Indexing, cont.)
Introduction to Data Management Lecture 21 (Indexing, cont.) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Announcements v Midterm #2 grading
4 Hash-Based Indexing
4 Hash-Based Indexing We now turn to a different family of index structures: hash indexes. Hash indexes are unbeatable when it comes to equality selections, e.g. SELECT FROM WHERE R A = k. If we carefully
1. [1 pt] What is the solution to the recurrence T(n) = 2T(n-1) + 1, T(1) = 1
![1. [1 pt] What is the solution to the recurrence T(n) = 2T(n-1) + 1, T(1) = 1](/thumbs/85/92778785.jpg "1. [1 pt] What is the solution to the recurrence T(n) = 2T(n-1) + 1, T(1) = 1")
Asymptotics, Recurrence and Basic Algorithms 1. [1 pt] What is the solution to the recurrence T(n) = 2T(n-1) + 1, T(1) = 1 2. O(n) 2. [1 pt] What is the solution to the recurrence T(n) = T(n/2) + n, T(1)
Tree-Structured Indexes. Chapter 10
Tree-Structured Indexes Chapter 10 1 Introduction As for any index, 3 alternatives for data entries k*: Data record with key value k 25, [n1,v1,k1,25] 25,
BBM371& Data*Management. Lecture 6: Hash Tables
BBM371& Data*Management Lecture 6: Hash Tables 8.11.2018 Purpose of using hashes A generalization of ordinary arrays: Direct access to an array index is O(1), can we generalize direct access to any key
RAID in Practice, Overview of Indexing
RAID in Practice, Overview of Indexing CS634 Lecture 4, Feb 04 2014 Slides based on Database Management Systems 3 rd ed, Ramakrishnan and Gehrke 1 Disks and Files: RAID in practice For a big enterprise
CS 4604: Introduction to Database Management Systems. B. Aditya Prakash Lecture #10: Query Processing
CS 4604: Introduction to Database Management Systems B. Aditya Prakash Lecture #10: Query Processing Outline introduction selection projection join set & aggregate operations Prakash 2018 VT CS 4604 2
CS261 Data Structures. Maps (or Dic4onaries)
CS261 Data Structures Maps (or Dic4onaries) Goals Introduce the Map(or Dic4onary) ADT Introduce an implementa4on of the map with a Dynamic Array So Far. Emphasis on values themselves e.g. store names in
Indexing. Chapter 8, 10, 11. Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1
Indexing Chapter 8, 10, 11 Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Tree-Based Indexing The data entries are arranged in sorted order by search key value. A hierarchical search
CSIT5300: Advanced Database Systems
CSIT5300: Advanced Database Systems L08: B + -trees and Dynamic Hashing Dr. Kenneth LEUNG Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong Kong SAR,
CSc 120. Introduc/on to Computer Programming II. 15: Hashing
CSc 120 Introduc/on to Computer Programming II 15: Hashing Hashing 2 Searching We have seen two search algorithms: linear (sequen;al) search O(n) o the items are not sorted binary search O(log n) o the
Find the block in which the tuple should be! If there is free space, insert it! Otherwise, must create overflow pages!
Professor: Pete Keleher! keleher@cs.umd.edu! } Keep sorted by some search key! } Insertion! Find the block in which the tuple should be! If there is free space, insert it! Otherwise, must create overflow
Topic HashTable and Table ADT
Topic HashTable and Table ADT Hashing, Hash Function & Hashtable Search, Insertion & Deletion of elements based on Keys So far, By comparing keys! Linear data structures Non-linear data structures Time
CSE 544 Principles of Database Management Systems. Magdalena Balazinska Winter 2009 Lecture 6 - Storage and Indexing
CSE 544 Principles of Database Management Systems Magdalena Balazinska Winter 2009 Lecture 6 - Storage and Indexing References Generalized Search Trees for Database Systems. J. M. Hellerstein, J. F. Naughton
EXTERNAL SORTING. Sorting
EXTERNAL SORTING 1 Sorting A classic problem in computer science! Data requested in sorted order (sorted output) e.g., find students in increasing grade point average (gpa) order SELECT A, B, C FROM R
NOTE: sorting using B-trees to be assigned for reading after we cover B-trees.
External Sorting Chapter 13 (Sec. 13-1-13.5): Ramakrishnan & Gehrke and Chapter 11 (Sec. 11.4-11.5): G-M et al. (R2) OR Chapter 2 (Sec. 2.4-2.5): Garcia-et Molina al. (R1) NOTE: sorting using B-trees to
Hashing. Data organization in main memory or disk
Hashing Data organization in main memory or disk sequential, indexed sequential, binary trees, the location of a record depends on other keys unnecessary key comparisons to find a key The goal of hashing
Hash-Based Indexes. Chapter 11 Ramakrishnan & Gehrke (Sections ) CPSC 404, Laks V.S. Lakshmanan 1
Hash-Based Indexes Chapter 11 Ramakrishnan & Gehrke (Sections 11.1-11.4) CPSC 404, Laks V.S. Lakshmanan 1 What you will learn from this set of lectures Review of static hashing How to adjust hash structure
Algorithms and Data Structures
Algorithms and Data Structures (Overflow) Hashing Marius Kloft This Module Introduction 2 Abstract Data Types 1 Complexity analysis 1 Styles of algorithms 1 Lists, stacks, queues 2 Sorting (lists) 3 Searching
CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 9 Index Selection and External Sorting Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements Private project threads created on piazza
Advances in Data Management Principles of Database Systems - 2 A.Poulovassilis
1 Advances in Data Management Principles of Database Systems - 2 A.Poulovassilis 1 Storing data on disk The traditional storage hierarchy for DBMSs is: 1. main memory (primary storage) for data currently
Storage hierarchy. Textbook: chapters 11, 12, and 13
Storage hierarchy Cache Main memory Disk Tape Very fast Fast Slower Slow Very small Small Bigger Very big (KB) (MB) (GB) (TB) Built-in Expensive Cheap Dirt cheap Disks: data is stored on concentric circular
Spring 2017 EXTERNAL SORTING (CH. 13 IN THE COW BOOK) 2/7/17 CS 564: Database Management Systems; (c) Jignesh M. Patel,
Spring 2017 EXTERNAL SORTING (CH. 13 IN THE COW BOOK) 2/7/17 CS 564: Database Management Systems; (c) Jignesh M. Patel, 2013 1 Motivation for External Sort Often have a large (size greater than the available
Dtb Database Systems. Announcement
Dtb Database Systems ( 資料庫系統 ) December 10, 2008 Lecture #11 1 Announcement Assignment #5 will be out on the course webpage today. 2 1 External Sorting Chapter 13 3 Why learn sorting again? O (n*n): bubble,
CS698F Advanced Data Management. Instructor: Medha Atre. Aug 11, 2017 CS698F Adv Data Mgmt 1
CS698F Advanced Data Management Instructor: Medha Atre Aug 11, 2017 CS698F Adv Data Mgmt 1 Recap Query optimization components. Relational algebra rules. How to rewrite queries with relational algebra
Chapter 17. Disk Storage, Basic File Structures, and Hashing. Records. Blocking
Chapter 17 Disk Storage, Basic File Structures, and Hashing Records Fixed and variable length records Records contain fields which have values of a particular type (e.g., amount, date, time, age) Fields
CMSC 341 Hashing. Based on slides from previous iterations of this course
CMSC 341 Hashing Based on slides from previous iterations of this course Hashing Searching n Consider the problem of searching an array for a given value q If the array is not sorted, the search requires
Tree-Structured Indexes
Tree-Structured Indexes CS 186, Fall 2002, Lecture 17 R & G Chapter 9 If I had eight hours to chop down a tree, I'd spend six sharpening my ax. Abraham Lincoln Introduction Recall: 3 alternatives for data
Administrivia. Tree-Structured Indexes. Review. Today: B-Tree Indexes. A Note of Caution. Introduction
Administrivia Tree-Structured Indexes Lecture R & G Chapter 9 Homeworks re-arranged Midterm Exam Graded Scores on-line Key available on-line If I had eight hours to chop down a tree, I'd spend six sharpening
Lassonde School of Engineering Winter 2016 Term Course No: 4411 Database Management Systems
Lassonde School of Engineering Winter 2016 Term Course No: 4411 Database Management Systems Last Name: First Name: Student ID: 1. Exam is 2 hours long 2. Closed books/notes Problem 1 (6 points) Consider
PS2 out today. Lab 2 out today. Lab 1 due today - how was it?
6.830 Lecture 7 9/25/2017 PS2 out today. Lab 2 out today. Lab 1 due today - how was it? Project Teams Due Wednesday Those of you who don't have groups -- send us email, or hand in a sheet with just your
Tree-Structured Indexes
Tree-Structured Indexes Yanlei Diao UMass Amherst Slides Courtesy of R. Ramakrishnan and J. Gehrke Access Methods v File of records: Abstraction of disk storage for query processing (1) Sequential scan;
CS 350 Algorithms and Complexity
CS 350 Algorithms and Complexity Winter 2019 Lecture 12: Space & Time Tradeoffs. Part 2: Hashing & B-Trees Andrew P. Black Department of Computer Science Portland State University Space-for-time tradeoffs
Tree-Structured Indexes
Introduction Tree-Structured Indexes Chapter 10 As for any index, 3 alternatives for data entries k*: Data record with key value k
CS 4604: Introduc0on to Database Management Systems. B. Aditya Prakash Lecture #3: SQL and Rela2onal Algebra- - - Part 1
CS 4604: Introduc0on to Database Management Systems B. Aditya Prakash Lecture #3: SQL and Rela2onal Algebra- - - Part 1 Reminder: Rela0onal Algebra Rela2onal algebra is a nota2on for specifying queries
Announcements. Reading Material. Recap. Today 9/17/17. Storage (contd. from Lecture 6)
CompSci 16 Intensive Computing Systems Lecture 7 Storage and Index Instructor: Sudeepa Roy Announcements HW1 deadline this week: Due on 09/21 (Thurs), 11: pm, no late days Project proposal deadline: Preliminary
Sorting a file in RAM. External Sorting. Why Sort? 2-Way Sort of N pages Requires Minimum of 3 Buffers Pass 0: Read a page, sort it, write it.
Sorting a file in RAM External Sorting Chapter 13 Three steps: Read the entire file from disk into RAM Sort the records using a standard sorting procedure, such as Shell sort, heap sort, bubble sort, Write
Hash Tables. Hash Tables
Hash Tables Hash Tables Insanely useful One of the most useful and used data structures that you ll encounter They do not support many operations But they are amazing at the operations they do support
Fast Lookup: Hash tables
CSE 100: HASHING Operations: Find (key based look up) Insert Delete Fast Lookup: Hash tables Consider the 2-sum problem: Given an unsorted array of N integers, find all pairs of elements that sum to a
Problem. Indexing with B-trees. Indexing. Primary Key Indexing. B-trees: Example. B-trees. primary key indexing
15-82 Advanced Topics in Database Systems Performance Problem Given a large collection of records, Indexing with B-trees find similar/interesting things, i.e., allow fast, approximate queries 2 Indexing
Query Processing: The Basics. External Sorting
Query Processing: The Basics Chapter 10 1 External Sorting Sorting is used in implementing many relational operations Problem: Relations are typically large, do not fit in main memory So cannot use traditional
CSE373: Data Structures & Algorithms Lecture 17: Hash Collisions. Kevin Quinn Fall 2015
CSE373: Data Structures & Algorithms Lecture 17: Hash Collisions Kevin Quinn Fall 2015 Hash Tables: Review Aim for constant-time (i.e., O(1)) find, insert, and delete On average under some reasonable assumptions
Introduction to Database Systems CSE 444, Winter 2011
Version March 15, 2011 Introduction to Database Systems CSE 444, Winter 2011 Lecture 20: Operator Algorithms Where we are / and where we go 2 Why Learn About Operator Algorithms? Implemented in commercial
Lecture 12. Lecture 12: The IO Model & External Sorting
Lecture 12 Lecture 12: The IO Model & External Sorting Lecture 12 Today s Lecture 1. The Buffer 2. External Merge Sort 2 Lecture 12 > Section 1 1. The Buffer 3 Lecture 12 > Section 1 Transition to Mechanisms
Database design and implementation CMPSCI 645. Lecture 08: Storage and Indexing
Database design and implementation CMPSCI 645 Lecture 08: Storage and Indexing 1 Where is the data and how to get to it? DB 2 DBMS architecture Query Parser Query Rewriter Query Op=mizer Query Executor
Data Structure Lecture#22: Searching 3 (Chapter 9) U Kang Seoul National University
Data Structure Lecture#22: Searching 3 (Chapter 9) U Kang Seoul National University U Kang 1 In This Lecture Motivation of collision resolution policy Open hashing for collision resolution Closed hashing
System Structure Revisited
System Structure Revisited Naïve users Casual users Application programmers Database administrator Forms DBMS Application Front ends DML Interface CLI DDL SQL Commands Query Evaluation Engine Transaction
COSC160: Data Structures Hashing Structures. Jeremy Bolton, PhD Assistant Teaching Professor
COSC160: Data Structures Hashing Structures Jeremy Bolton, PhD Assistant Teaching Professor Outline I. Hashing Structures I. Motivation and Review II. Hash Functions III. HashTables I. Implementations
Module 4: Tree-Structured Indexing
Module 4: Tree-Structured Indexing Module Outline 4.1 B + trees 4.2 Structure of B + trees 4.3 Operations on B + trees 4.4 Extensions 4.5 Generalized Access Path 4.6 ORACLE Clusters Web Forms Transaction
CSE 373: Data Structure & Algorithms Comparison Sor:ng
CSE 373: Data Structure & Comparison Sor:ng Riley Porter Winter 2017 1 Course Logis:cs HW4 preliminary scripts out HW5 out à more graphs! Last main course topic this week: Sor:ng! Final exam in 2 weeks!
Database Management System
Database Management System Lecture Join * Some materials adapted from R. Ramakrishnan, J. Gehrke and Shawn Bowers Today s Agenda Join Algorithm Database Management System Join Algorithms Database Management
key h(key) Hash Indexing Friday, April 09, 2004 Disadvantages of Sequential File Organization Must use an index and/or binary search to locate data
Lectures Desktop (C) Page 1 Hash Indexing Friday, April 09, 004 11:33 AM Disadvantages of Sequential File Organization Must use an index and/or binary search to locate data File organization based on hashing
Overview of Query Processing. Evaluation of Relational Operations. Why Sort? Outline. Two-Way External Merge Sort. 2-Way Sort: Requires 3 Buffer Pages
Overview of Query Processing Query Parser Query Processor Evaluation of Relational Operations Query Rewriter Query Optimizer Query Executor Yanlei Diao UMass Amherst Lock Manager Access Methods (Buffer
Lecture 11. Lecture 11: External Sorting
Lecture 11 Lecture 11: External Sorting Lecture 11 Announcements 1. Midterm Review: This Friday! 2. Project Part #2 is out. Implement CLOCK! 3. Midterm Material: Everything up to Buffer management. 1.
Remember. 376a. Database Design. Also. B + tree reminders. Algorithms for B + trees. Remember
376a. Database Design Dept. of Computer Science Vassar College http://www.cs.vassar.edu/~cs376 Class 14 B + trees, multi-key indices, partitioned hashing and grid files B and B + -trees are used one implementation
CS 222/122C Fall 2017, Final Exam. Sample solutions
CS 222/122C Fall 2017, Final Exam Principles of Data Management Department of Computer Science, UC Irvine Prof. Chen Li (Max. Points: 100 + 15) Sample solutions Question 1: Short questions (15 points)