CSE544: Principles of Database Systems. Lectures 5-6 Database Architecture Storage and Indexes

Transcription

1 CSE544: Principles of Database Systems Lectures 5-6 Database Architecture Storage and Indexes 1

2 Announcements Project Choose a topic. Set limited goals! Sign up (doodle) to meet with me this week Homework 1 A few people have not turned in yet: will do by tomorrow. Then we will post solutions Homework 2 Will be posted today; you will receive Paper review for Wednesday Join processing

3 Where We Are Part 1: The relational data model Part 2: Database Systems Part 3: Database Theory Part 4: Miscellaneous CSE544 - Spring,

4 Where We Are The relational data model Motivation of the relational model Older data models and the need for data independence Relational model, E/R model, Normal Forms (we skipped them) Query Languages SQL Relational algebra Relational calculus Non-recursive datalog with negation Database Systems How can we efficiently implement this model? CSE544 - Spring,

5 Outline DBMS Architecture Anatomy of a database system. J. Hellerstein and M. Stonebraker. In Red Book (4th ed). Storage and Indexes Book: Ch. 8-11, and 20 CSE544 - Spring,

6 DMBS Architecture: Outline Main components of a modern DBMS Process models Storage models Query processor CSE544 - Spring,

7 DBMS Architecture Admission Control Connection Mgr Process Manager Parser Query Rewrite Optimizer Executor Query Processor Memory Mgr Disk Space Mgr Replication Services Admin Utilities Access Methods Buffer Manager Lock Manager Log Manager Storage Manager Shared Utilities [Anatomy of a Db System. J. Hellerstein & M. Stonebraker. Red Book. 4ed.]

8 DMBS Architecture: Outline Main components of a modern DBMS Process models Storage models Query processor CSE544 - Spring,

9 Process Model Q: Why not simply queue all user requests, and serve them one at the time?

10 Process Model Q: Why not simply queue all user requests, and serve them one at the time? A: Because of the high disk I/O latency Corollary: in a main memory db you can service transactions sequentially! Alternatives 1. Process per connection 2. Server process (thread per connection) OS threads or DBMS threads 3. Server process with I/O process

11 Process Per Connection Overview DB server forks one process for each client connection Advantages? Drawbacks? CSE544 - Spring,

12 Process Per Connection Overview DB server forks one process for each client connection Advantages Easy to implement (OS time-sharing, OS isolation, debuggers, etc.) Drawbacks Need OS-supported shared memory (for lock table, buffer pool) Not scalable: memory overhead and expensive context switches CSE544 - Spring,

13 Server Process Overview Dispatcher thread listens to requests, dispatches worker threads Advantages?? Drawbacks?

14 Server Process Overview Dispatcher thread listens to requests, dispatches worker threads Advantages Shared structures can simply reside on the heap Threads are lighter weight than processes: memory, context switching Drawbacks Concurrent programming is hard to get right (race conditions, deadlocks) Subtle API thread differences across different operating systems make portability difficult

15 Sever Process with I/O Process Problem: entire process blocks on synchronous I/O calls Solution 1: Use separate process(es) for I/O tasks Solution 2: Modern OS provide asynchronous I/O CSE544 - Spring,

16 DBMS Threads vs OS Threads Why do DBMSs implement their own threads? CSE544 - Spring,

17 DBMS Threads vs OS Threads Why do DBMSs implement their own threads? Legacy: originally, there were no OS threads Portability: OS thread packages are not completely portable Performance: fast task switching Drawbacks Replicating a good deal of OS logic Need to manage thread state, scheduling, and task switching How to map DBMS threads onto OS threads or processes? Rule of thumb: one OS-provided dispatchable unit per physical device See page 9 and 10 of Hellerstein and Stonebraker s paper CSE544 - Spring,

18 Historical Perspective (1981) In 1981: No OS threads No shared memory between processes Makes one process per user hard to program Some OSs did not support many to one communication Thus forcing the one process per user model No asynchronous I/O But inter-process communication expensive Makes the use of I/O processes expensive Common original design: DBMS threads, frequently yielding control to a scheduling routine CSE544 - Spring,

19 Commercial Systems Oracle Unix default: process-per-user mode Unix: DBMS threads multiplexed across OS processes Windows: DBMS threads multiplexed across OS threads DB2 Unix: process-per-user mode Windows: OS thread-per-user SQL Server Windows default: OS thread-per-user Windows: DBMS threads multiplexed across OS threads CSE544 - Spring,

20 DMBS Architecture: Outline Main components of a modern DBMS Process models Storage models Query processor CSE544 - Spring,

21 Storage Model Problem: DBMS needs spatial and temporal control over storage Spatial control for performance Temporal control for correctness and performance Alternatives Use raw disk device interface directly Use OS files CSE544 - Spring,

22 Spatial Control Using Raw Disk Device Interface Overview DBMS issues low-level storage requests directly to disk device Advantages?? Disadvantages? CSE544 - Spring,

23 Spatial Control Using Raw Disk Device Interface Overview DBMS issues low-level storage requests directly to disk device Advantages DBMS can ensure that important queries access data sequentially Can provide highest performance Disadvantages Requires devoting entire disks to the DBMS Reduces portability as low-level disk interfaces are OS specific Many devices are in fact virtual disk devices SAN = storage area network; NAS = network attached device CSE544 - Spring,

24 Spatial Control Using OS Files Overview DBMS creates one or more very large OS files Advantages? Disadvantages? CSE544 - Spring,

25 Spatial Control Using OS Files Overview DBMS creates one or more very large OS files Advantages Allocating large file on empty disk can yield good physical locality Disadvantages Must control the timing of writes for correctness and performance OS may further delay writes OS may lead to double buffering, leading to unnecessary copying DB must fine tune when the log tail is flushed to disk CSE544 - Spring,

26 Historical Perspective (1981) Recognizes mismatch problem between OS files and DBMS needs If DBMS uses OS files and OS files grow with time, blocks get scattered OS uses tree structure for files but DBMS needs its own tree structure Other proposals at the time Extent-based file systems Record management inside OS CSE544 - Spring,

27 Commercial Systems Most commercial systems offer both alternatives Raw device interface for peak performance OS files more commonly used In both cases, we end-up with a DBMS file abstraction implemented on top of OS files or raw device interface CSE544 - Spring,

28 Correctness problems Temporal Control Buffer Manager DBMS needs to control when data is written to disk in order to provide transactional semantics (we will study transactions later) OS buffering can delay writes, causing problems when crashes occur Performance problems OS optimizes buffer management for general workloads DBMS understands its workload and can do better Areas of possible optimizations Page replacement policies Read-ahead algorithms (physical vs logical) Deciding when to flush tail of write-ahead log to disk CSE544 - Spring,

29 Historical Perspective (1981) Problems with OS buffer pool management long recognized Accessing OS buffer pool involves an expensive system call Faster to access a DBMS buffer pool in user space LRU replacement does not match DBMS workload DBMS can do better OS can do only sequential prefetching, DBMS knows which page it needs next and that page may not be sequential DBMS needs ability to control when data is written to disk CSE544 - Spring,

30 Commercial Systems DBMSs implement their own buffer pool managers Modern filesystems provide good support for DBMSs Using large files provides good spatial control Using interfaces like the mmap suite Provides good temporal control Helps avoid double-buffering at DBMS and OS levels CSE544 - Spring,

31 DMBS Architecture: Outline Main components of a modern DBMS Process models Storage models Query processor (will go over the query processor in lectures 6-7) CSE544 - Spring,

32 Outline DBMS Architecture Anatomy of a database system. J. Hellerstein and M. Stonebraker. In Red Book (4th ed). Storage and Indexes Book: Ch. 8-11, and 20 CSE544 - Spring,

33 Arranging Pages on Disk A disk is organized into blocks (a.k.a. pages) blocks on same track, followed by blocks on same cylinder, followed by blocks on adjacent cylinder A file should (ideally) consists of sequential blocks on disk, to minimize seek and rotational delay. For a sequential scan, pre-fetching several pages at a time is a big win! CSE544 - Spring,

34 Issues Managing free blocks Represent the records inside the blocks Represent attributes inside the records CSE544 - Spring,

35 Managing Free Blocks Linked list of free blocks Or bit map CSE544 - Spring,

36 File Organization Linked list of pages: Data page Data page Data page Data page Header page Full pages Data page Data page Data page Data page Pages with some free space 36

37 Better: directory of pages File Organization Header Data page Data page Directory Data page 37

38 Page Formats Issues to consider 1 page = fixed size (e.g. 8KB) Records: Fixed length Variable length Record id = RID Typically RID = (PageID, SlotNumber) Why do we need RID s in a relational DBMS? 38

39 Page Formats Fixed-length records: packed representation Rec 1 Rec 2 Rec N Free space N Problems? 39

40 Page Formats Free space Slot directory Variable-length records 40

41 Record Formats: Fixed Length Product(pid, name, descr, maker) pid name descr maker L1 L2 L3 L4 Base address (B) Address = B+L1+L2 Information about field types same for all records in a file; stored in system catalogs. Finding i th field requires scan of record. Note the importance of schema information! 41

42 To schema Record Header length pid name descr maker L1 L2 L3 L4 header timestamp (e.g. for MVCC) Need the header because: The schema may change for a while new+old may coexist Records from different relations may coexist 42

43 Variable Length Records Other header information header pid name descr maker L1 L2 L3 L4 length Place the fixed fields first: F1 Then the variable length fields: F2, F3, F4 Null values take 2 bytes only Sometimes they take 0 bytes (when at the end) 43

44 BLOB Binary large objects Supported by modern database systems E.g. images, sounds, etc. Storage: attempt to cluster blocks together CLOB = character large object Supports only restricted operations 44

45 File Organizations Heap (random order) files: Suitable when typical access is a file scan retrieving all records. Sorted Files Best if records must be retrieved in some order, or only a `range of records is needed. Indexes Data structures to organize records via trees or hashing. Like sorted files, they speed up searches for a subset of records, based on values in certain ( search key ) fields Updates are much faster than in sorted files. 45

46 Index A (possibly separate) file, that allows fast access to records in the data file The index contains (key, value) pairs: The key = an attribute value The value = one of: pointer to the record secondary index or the record itself primary index Note: key (aka search key ) again means something else CSE544 - Spring,

47 Index Classification Clustered/unclustered Clustered = records close in index are close in data Unclustered = records close in index may be far in data Primary/secondary Meaning 1: Primary = is over attributes that include the primary key Secondary = otherwise Meaning 2: means the same as clustered/unclustered Organization B+ tree or Hash table 47

48 Clustered/Unclustered Clustered Index determines the location of indexed records Typically, clustered index is one where values are data records (but not necessary) Unclustered Index cannot reorder data, does not determine data location In these indexes: value = pointer to data record CSEP Fall

49 Clustered Index File is sorted on the index attribute Only one per table

50 Unclustered Index Several per table

51 Clustered vs. Unclustered Index B+ Tree B+ Tree Data entries (Index File) (Data file) Data entries CLUSTERED Data Records Data Records UNCLUSTERED CSE544 - Spring,

52 Hash-Based Index Good for point queries but not range queries age h2(age) = 00 H h1(sid) = 00 h2(age) = H1 h1(sid) = 11 sid Another example of unclustered/secondary index Another example of clustered/primary index CSEP Fall

53 Alternatives for Data Entry k* in Index Three alternatives for k*: Data record with key value k <k, rid of data record with key = k> <k, list of rids of data records with key = k> 53

54 Alternatives 2 and

55 B+ Trees Search trees Idea in B Trees Make 1 node = 1 block Keep tree balanced in height Idea in B+ Trees Make leaves into a linked list: facilitates range queries CSE544 - Spring,

56 B+ Trees Basics Parameter d = the degree Each node has >= d and <= 2d keys (except root) Keys k < 30 Keys 30<=k<120 Keys 120<=k<240 Keys 240<=k Each leaf has >=d and <= 2d keys: Next leaf

57 B+ Tree Example d = 2 Find the key

58 B+ Tree Example d = 2 Find the key

59 B+ Tree Example d = 2 Find the key <

60 B+ Tree Example d = 2 Find the key < <

61 Exact key values: Using a B+ Tree Start at the root Proceed down, to the leaf Index on People(age) SELECT name FROM People WHERE age = 25 Range queries: As above Then sequential traversal SELECT name FROM People WHERE 20 <= age and age <= 30 CSE544 - Spring,

62 Which queries can use this index? Index on People(name, zipcode) SELECT * FROM People WHERE name = Smith and zipcode = SELECT * FROM People WHERE name = Smith SELECT * FROM People WHERE zipcode = CSE544 - Spring,

63 B+ Tree Design How large d? Example: Key size = 4 bytes Pointer size = 8 bytes Block size = 4096 byes 2d x 4 + (2d+1) x 8 <= 4096 d = 170 CSE544 - Spring,

64 B+ Trees in Practice Typical order: 100. Typical fill-factor: 67% average fanout = 133 Typical capacities Height 4: = 312,900,700 records Height 3: = 2,352,637 records Can often hold top levels in buffer pool Level 1 = 1 page = 8 Kbytes Level 2 = 133 pages = 1 Mbyte Level 3 = 17,689 pages = 133 Mbytes CSE544 - Spring,

65 Insertion in a B+ Tree Insert (K, P) Find leaf where K belongs, insert If no overflow (2d keys or less), halt If overflow (2d+1 keys), split node, insert in parent: parent parent K3 K1 K2 K3 K4 K5 P0 P1 P2 P3 P4 p5 K1 K2 P0 P1 P2 K4 K5 P3 P4 p5 If leaf, keep K3 too in right node When root splits, new root has 1 key only 65

66 Insert K=19 Insertion in a B+ Tree

67 Insertion in a B+ Tree After insertion

68 Now insert 25 Insertion in a B+ Tree

69 After insertion Insertion in a B+ Tree

70 Insertion in a B+ Tree But now have to split!

71 After the split Insertion in a B+ Tree

72 Delete 30 Deletion from a B+ Tree

73 Deletion from a B+ Tree After deleting 30 May change to 40, or not

74 Deletion from a B+ Tree Now delete

75 Deletion from a B+ Tree After deleting 25 Need to rebalance Rotate

76 Deletion from a B+ Tree Now delete

77 Deletion from a B+ Tree After deleting 40 Rotation not possible Need to merge nodes

78 Final tree Deletion from a B+ Tree

79 Practical Aspects of B+ Trees Key compression: Each node keeps only the from parent keys Jonathan, John, Johnsen, Johnson à Parent: Jo Child: nathan, hn, hnsen, hnson, CSE544 - Spring,

80 Practical Aspects of B+ Trees Bulk insertion When a new index is created there are two options: Start from empty tree, insert each key oneby-one Do bulk insertion what does that mean? CSE544 - Spring,

81 Practical Aspects of B+ Trees Concurrency control The root of the tree is a hot spot Leads to lock contention during insert/ delete Solution: do proactive split during insert, or proactive merge during delete Insert/delete now require only one traversal, from the root to a leaf Use the tree locking protocol 81

82 Summary on B+ Trees Default index structure on most DBMS Very effective at answering point queries: productname = gizmo Effective for range queries: 50 < price AND price < 100 Less effective for multirange: 50 < price < 100 AND 2 < quant < 20 CSE544 - Spring,

83 Indexes in Postgres CREATE TABLE V(M int, N varchar(20), P int); CREATE INDEX V1_N ON V(N) CREATE INDEX V2 ON V(P, M) CREATE INDEX VVV ON V(M, N) CLUSTER V USING V2 Makes V2 clustered 83

84 Database Tuning Overview The database tuning problem Index selection (discuss in detail) Horizontal/vertical partitioning (see lecture 3) Denormalization (discuss briefly) CSEP Fall

85 Levels of Abstraction in a DBMS External Schema External Schema External Schema views access control Conceptual Schema Physical Schema Disk CSEP Fall 2011 a.k.a logical schema describes stored data in terms of data model includes storage details file organization indexes 85

86 The Database Tuning Problem We are given a workload description List of queries and their frequencies List of updates and their frequencies Performance goals for each type of query Perform physical database design Choice of indexes Tuning the conceptual schema Denormalization, vertical and horizontal partition Query and transaction tuning CSEP Fall

87 The Index Selection Problem Given a database schema (tables, attributes) Given a query workload : Workload = a set of (query, frequency) pairs The queries may be both SELECT and updates Frequency = either a count, or a percentage Select a set of indexes that optimizes the workload In general this is a very hard problem CSEP Fall

88 Index Selection: Which Search Key Make some attribute K a search key if the WHERE clause contains: An exact match on K A range predicate on K A join on K CSEP Fall

89 Index Selection Problem 1 V(M, N, P); Your workload is this queries: 100 queries: SELECT * FROM V WHERE N=? SELECT * FROM V WHERE P=? Which indexes should we create?

90 Index Selection Problem 1 V(M, N, P); Your workload is this queries: 100 queries: SELECT * FROM V WHERE N=? SELECT * FROM V WHERE P=? A: V(N) and V(P) (hash tables or B-trees) CSE544 - Spring,

91 Index Selection Problem 2 V(M, N, P); Your workload is this queries: 100 queries: queries: SELECT * FROM V WHERE N>? and N<? SELECT * FROM V WHERE P=? INSERT INTO V VALUES (?,?,?) Which indexes CSE544 should - Spring, 2012 we create? 91

92 Index Selection Problem 2 V(M, N, P); Your workload is this queries: 100 queries: queries: SELECT * FROM V WHERE N>? and N<? SELECT * FROM V WHERE P=? INSERT INTO V VALUES (?,?,?) A: definitely V(N) (must B-tree); unsure about V(P) CSE544 - Spring,

93 Index Selection Problem 3 V(M, N, P); Your workload is this queries: queries: queries: SELECT * FROM V WHERE N=? SELECT * FROM V WHERE N=? and P>? INSERT INTO V VALUES (?,?,?) Which indexes CSE544 should - Spring, 2012 we create? 93

94 Index Selection Problem 3 V(M, N, P); Your workload is this queries: queries: queries: SELECT * FROM V WHERE N=? SELECT * FROM V WHERE N=? and P>? INSERT INTO V VALUES (?,?,?) A: V(N, P) 94

95 Index Selection Problem 4 V(M, N, P); Your workload is this 1000 queries: queries: SELECT * FROM V WHERE N>? and N<? SELECT * FROM V WHERE P>? and P<? Which indexes CSE544 should - Spring, 2012 we create? 95

96 Index Selection Problem 4 V(M, N, P); Your workload is this 1000 queries: queries: SELECT * FROM V WHERE N>? and N<? SELECT * FROM V WHERE P>? and P<? A: V(N) secondary, CSE544 - Spring, V(P) 2012 primary index 96

97 The Index Selection Problem SQL Server Automatically, thanks to AutoAdmin project Much acclaimed successful research project from mid 90 s, similar ideas adopted by the other major vendors PostgreSQL You will do it manually, part of homework 5 But tuning wizards also exist CSE544 - Spring,

98 Index Selection: Multi-attribute Keys Consider creating a multi-attribute key on K1, K2, if WHERE clause has matches on K1, K2, But also consider separate indexes SELECT clause contains only K1, K2,.. A covering index is one that can be used exclusively to answer a query, e.g. index R SELECT (K1,K2) K2 covers FROM CSE544 the R WHERE - Spring, query: 2012 K1=55 98

99 To Cluster or Not Range queries benefit mostly from clustering Covering indexes do not need to be clustered: they work equally well unclustered CSEP Fall

100 SELECT * FROM R WHERE K>? and K<? Cost Sequential scan 0 Percentage tuples retrieved 100 CSE544 - Spring,

101 Hash Table v.s. B+ tree Rule 1: always use a B+ tree J Rule 2: use a Hash table on K when: There is a very important selection query on equality (WHERE K=?), and no range queries You know that the optimizer uses a nested loop join where K is the join attribute of the inner relation (you will understand that in a few lectures)

102 Balance Queries v.s. Updates Indexes speed up queries SELECT FROM WHERE But they usually slow down updates: INSERT, DELECTE, UPDATE However some updates benefit from indexes UPDATE R SET A = 7 WHERE K=55

103 Tools for Index Selection SQL Server 2000 Index Tuning Wizard DB2 Index Advisor How they work: They walk through a large number of configurations, compute their costs, and choose the configuration with minimum cost CSE544 - Spring,