Database Design and Programming

Transcription

1 Database Design and Programming Jan Baumbach

2 JDBC Java Database Connectivity (JDBC) is a library similar for accessing a DBMS with Java as the host language >200 drivers available: PostgreSQL, MySQL, Oracle, ODBC,

3 Making a Connection import java.sql.*;... Loaded by forname URL of the database your name, and password go here The JDBC classes Class.forName( org.postgresql.driver ); Connection mycon = DriverManager.getConnection( );... The driver for postgresql; others exist 3

4 URL for PostgreSQL database getconnection(jdbc:postgresql://< host>[:<port>]/<database>?user=<u ser>&password=<password>); Alternatively use getconnection variant: getconnection("jdbc:postgresql:// <host>[:<port>]/<database>", <user>, <password>); DriverManager.getConnection("jdbc :postgresql:// :5434/po stgres", "petersk", "geheim"); 4

5 Statements JDBC provides two classes: 1. Statement = an object that can accept a string that is a SQL statement and can execute such a string 2. PreparedStatement = an object that has an associated SQL statement ready to execute 5

6 Creating Statements The Connection class has methods to create Statements and PreparedStatements Statement stat1 = mycon.createstatement(); PreparedStatement stat2 = mycon.createstatement( SELECT beer, price FROM Sells + ); WHERE bar = C.Ch. createstatement with no argument returns a Statement; with one argument it returns a PreparedStatement 6

7 Executing SQL Statements JDBC distinguishes queries from modifications, which it calls updates Statement and PreparedStatement each have methods executequery and executeupdate For Statements: one argument the query or modification to be executed For PreparedStatements: no argument 7

8 Example: Update stat1 is a Statement We can use it to insert a tuple as: stat1.executeupdate( INSERT INTO Sells + VALUES( C.Ch., Eventyr,30) ); 8

9 Example: Query stat2 is a PreparedStatement holding the query SELECT beer, price FROM Sells WHERE bar = C.Ch. executequery returns an object of class ResultSet we ll examine it later The query: ResultSet menu = stat2.executequery(); 9

10 Accessing the ResultSet An object of type ResultSet is similar to a cursor Method next() advances the cursor to the next tuple The first time next() is applied, it gets the first tuple If there are no more tuples, next() returns the value false 10

11 Accessing Components of Tuples When a ResultSet is referring to a tuple, we can get the components of that tuple by applying certain methods to the ResultSet Method getx (i ), where X is some type, and i is the component number, returns the value of that component The value must have type X 11

12 Example: Accessing Components Menu = ResultSet for query SELECT beer, price FROM Sells WHERE bar = C.Ch. Access beer and price from each tuple by: while (menu.next()) { thebeer = menu.getstring(1); } theprice = menu.getfloat(2); /*something with thebeer and theprice*/ 12

13 Important Details Reusing a Statement object results in the ResultSet being closed Always create new Statement objects using createstatement() or explicitly close ResultSets using the close method For transactions, for the Connection con use con.setautocommit(false) and explicitly con.commit() or con.rollback() If AutoCommit is false and there is no commit, closing the connection = rollback 13

14 Python and Databases many different modules for accessing databases commercial: mxodbc, open source: pygresql, psycopg2, we use psycopg2 install using easy_install psycopg2 import with import psycopg2 14

15 Connection String Database connection described by a connection string Example: con_str = """ host=' ' port=5434 dbname='postgres' user='peter' password='geheim' """ 15

16 Making a Connection With the DB library imported and the connection string con_str available: con = psycopg2.connect(con_str); Function connect in the DB API Class is connection because it is returned by psycopg2.connect( ) 16

17 Cursors in Python Queries are executed for a cursor A cursor is obtained from connection Example: cursor = con.cursor() Queries or modifications are executed using the execute( ) method Cursors can then be used in a for-loop 17

18 Example: Executing a Query Find all the bars that sell a beer given by the variable beer beer = 'Od.Cl. cursor = con.cursor() cursor.execute( "SELECT bar FROM Sells" + WHERE beer = '%s ;" % beer); Remember this variable is replaced by the value of beer 18

19 Example: Tuple Cursors bar = 'C.Ch.' cur = con.cursor() cur.execute("select beer, price" + " FROM Sells" + " WHERE bar = " + bar + ";") for row in cur: print row[0] + for + row[1] 19

20 Caution: SQL Injection SQL queries are often constructed by programs These queries may take constants from user input Careless code can allow rather unexpected queries to be constructed and executed 20

21 Example: SQL Injection Relation Accounts(name, passwd, acct) Web interface: get name and password from user, store in strings n and p, issue query, display account number cur.execute("select acct FROM " + "Accounts WHERE name = '%s' " + AND passwd = '%s';" % (n,p)) 21

22 User (Who Is Not Bill Gates) Types Name: gates -- Comment in PostgreSQL Password: who cares? Your account number is

23 The Query Executed SELECT acct FROM Accounts WHERE name = gates -- AND passwd = who cares? All treated as a comment 23

24 Summary 8 More things you should know: Stored Procedures, PL/pgsql Declarations, Statements, Loops, Cursors, Tuple Variables Three-Tier Approach, JDBC, psycopg2 24

25 Data Storage 25

26 Computer System CPU RAM SATA Secondary & Tertiary Storage 26

27 The Memory Hierarchy a lot/mb Cache 0.3 ns cost 30/GB 8/GB RAM primary Solid-State Disk secondary 1.5 ns 0.1 ms latency 0.4/GB Harddisk tertiary 7.5 ms 27

28 DBMS and Storage Databases typically too large to keep in primary storage Tables typically kept in secondary storage Large amounts of data that are only accessed infrequently are stored in tertiary storage (or even on tape robot) Indexes and current tables cached in primary storage 28

29 Harddisk N rotating magenetic platters 2xN heads for reading and writing track, cylinder, sector, gap 29

30 Harddisk Access access time: how long does it take to load a block from the harddisk? seek time: how long does it take to move the heads to the right cylinder? rotational delay: how long does it take until the head gets to the right sectors? transfer time: how long does it take to read the block? access = seek + rotational + transfer 30

31 Seek Time average seek time = ½ time to move head from outermost to innermost cylinder 31

32 Rotational Delay average rotational delay = ½ rotation head here block to read 32

33 Transfer Time Transfer time = 1/n rotation when there are n blocks on one track to here from here 33

34 Access Time Typical harddisk: Maximal seek time: 10 ms Rotational speed: 7200 rpm Block size: 4096 bytes Sectors (512 bytes) per track: 1600 (average) Average access time: 9.21 ms Average seek time: 5 ms Average rotational delay: 60/7200/2 = 4.17 ms Average transfer time: 0.04 ms 34

35 Random vs Sequential Access Random access of blocks: 1/ s * 4096 byte = 0.42 Mbyte/s Sequential access of blocks: 120/s * 200 * 4096 byte = 94 Mbyte/s Performance of the DBMS dominated by number of random accesses 35

36 On Disk Cache CPU RAM SATA cache cache Secondary & Tertiary Storage 36

37 Problems with Harddisks Even with caches, harddisk remains bottleneck for DBMS performance Harddisks can fail: Intermittent failure Media decay Write failure Disk crash Handle intermittent failures by rereading the block in question 37

38 Detecting Read Failures Use checksums to detect failures Simplest form is parity bit: 0 if number of ones in the block is even 1 if number of ones in the block is odd Detects all 1-bit failures Detects 50% of many-bit failures By using n bits, we can reduce the chance of missing an error to 1/2^n 38

39 Disk Arrays Use more than one disk for higher reliability and/or performance RAID (Redundant Arrays of Independent Disks) logically one disk 39

40 RAID 0 Alternate blocks between two or more disks ( Striping ) Increases performance both for writing and reading No increase in reliability Disk Storing blocks 0-5 in the first three blocks of disk 1 & 2 40

41 RAID 1 Duplicate blocks on two or more disks ( Mirroring ) Increases performance for reading Increases reliability significantly Disk Storing blocks 0-2 in the first three blocks of disk 1 & 2 41

42 RAID 5 Stripe blocks on n+1 disks where for each block, one disk stores parity information More performant when writing than RAID 1 Increased reliability compared to RAID 0 Disk P 2 5 P P 3 4 Storing blocks 0-5 in the first three blocks of disk 1, 2 & 3 42

43 RAID Capacity Assume disks with capacity 1 TByte RAID 0: N disks = N TByte RAID 1: N disks = 1 TByte RAID 5: N disks = (N-1) TByte RAID 6: N disks = (N-M) TByte... 43

44 Storage of Values Basic unit of storage: Byte Integer: 4 bytes Example: 42 is Characters: ASCII, UTF8,... Boolean: and 8 bits

45 Storage of Values Dates: Days since January 1, 1900 DDMMYYYY (not DDMMYY) Time: Seconds since midnight HHMMSS Strings: Null terminated Length given L a r s 4 L a r s 45

46 DBMS Storage Overview Values Records Blocks Files Memory 46

47 Record Collection of related data items (called Fields) Typically used to store one tuple Example: Sells record consisting of bar field beer field price field 47

48 Record Metadata For fixed-length records, schema contains the following information: Number of fields Type of each field Order in record For variable-length records, every record contains this information in its header 48

49 Record Header Reserved part at the beginning of a record Typically contains: Record type (which Schema?) Record length (for skipping) Time stamp (last access) 49

50 Files Files consist of blocks containing records How to place records into blocks? assume fixed length blocks assume a single file 50

51 Files Options for storing records in blocks: 1. Separating records 2. Spanned vs. unspanned 3. Sequencing 4. Indirection 51

52 1. Separating Records Block R1 R2 R3 a.no need to separate - fixed size recs. b.special marker c.give record lengths (or offsets) i. within each record ii. in block header 52

53 2. Spanned vs Unspanned Unspanned: records must be in one block R1 R2 R3 R4 R5 Spanned: one record in two or more blocks R1 R2 R3 (a) R3 (b) R4 R5 R6 R7 (a) Unspanned much simpler, but wastes space Spanned necessary if record size > block 53 size

54 3. Sequencing Ordering records in a file (and in the blocks) by some key value Can be used for binary search Options: a. Next record is physically contiguous R1 Next (R1)... a. Records are linked R1 Next (R1) 54

55 4. Indirection How does one refer to records? a. Physical address (disk id, cylinder, head, sector, offset in block) b. Logical record ids and a mapping table Indirection map 17 Rec ID Physical addr. 2:34:5:742:2340 Tradeoff between flexibility and cost 55

56 Modification of Records How to handle the following operations on the record level? 1. Insertion 2. Deletion 3. Update 56

57 1. Insertion Easy case: records not in sequence Insert new record at end of file If records are fixed-length, insert new record in deleted slot Difficult case: records are sorted Find position and slide following records If records are sequenced by linking, insert overflow blocks 57

58 2. Deletion a. Immediately reclaim space by shifting other records or removing overflows b. Mark deleted and list as free for re-use Tradeoffs: How expensive is immediate reclaim? How much space is wasted? 58

59 3. Update If records are fixed-length and the order is not affected: Fetch the record, modify it, write it back Otherwise: Delete the old record Insert the new record overwriting the tombstones from the deletion 59

60 Data Organizaton There are millions of ways to organize the data on disk Flexibility Space Utilization Complexity Performance 60

61 Summary 9 More things you should know: Memory Hierarchy Storage on harddisks Values, Records, Blocks, Files Storing and modifying records 61

62 Index Structures 62

63 Finding Records How do we find the records for a query? Example: SELECT * FROM Sells Need to examine every block in every file Group blocks into files by relation! Example: SELECT * FROM Sells WHERE price = 20; Need to examine every block in the file 63

64 Finding Records Use of indexes allows to narrow search to (almost) only the relevant blocks Value Index Blocks Holding records Matching records Indexes can be dense or sparse 64

65 Dense Index Dense Index Sequential File

66 Sparse Index 2nd level Sparse Index Sequential File

67 Deletion from Sparse Index Delete

68 Deletion from Sparse Index Delete

69 Deletion from Sparse Index Delete 30 &

70 Insertion into Sparse Index Insert

71 Insertion into Sparse Index Insert

72 Sparse vs Dense Sparse uses less index space per record (can keep more of index in memory) Sparse allows multi-level indexes Dense can tell if record exists without accessing it Dense needed for secondary indexes Primary index = order of records in storage Secondary index = impose different order 72

73 Secondary Index 2nd level Careful when Looking for 20 Secondary Index Sequential File

74 Secondary Index 2nd level Secondary Index Sequential File

75 Combining Indexes SELECT * FROM Sells WHERE beer = Od.Cl. AND price = 20 Beer index Sells Price index OC 20 C.Ch. Just intersect buckets in memory! 75

76 Conventional Indexes Sparse, Dense, Multi-level,... Advantages: Simple Sequential index is good for scans Disadvantage: Inserts expensive Lose sequentiality and balance 76

77 Example: Unbalanced Index overflow area (not sequential) 77