Faloutsos 15-415 Carnegie Mellon Univ. Dept. of Computer Science 15-415 - Database Applications C. Faloutsos Lecture#1: Introduction Outline Introduction to DBMSs The Entity Relationship model The Relational Model SQL: the commercial query language DB design: FD, 3NF, BCNF indexing, q-opt concurrency control & recovery advanced topics (data mining, multimedia) Faloutsos 15-415 2 We ll learn: What are RDBMS when to use them how to model data with them how to store and retrieve information how to search quickly for information Internals of an RDBMS: indexing, transactions Faloutsos 15-415 3 1
Faloutsos 15-415 We ll learn (cnt d) Advanced topics multimedia indexing (how to find similar, eg., images) data mining (how to find patterns in data) Faloutsos 15-415 4 Detailed outline Introduction Motivating example How do DBMSs work? DDL, DML, views. Fundamental concepts DBMS users Overall system architecture Conclusions Faloutsos 15-415 5 What is the goal of rel. DBMSs Faloutsos 15-415 6 2
Faloutsos 15-415 What is the goal of rel. DBMSs Electronic record-keeping: Fast and convenient access to information. Faloutsos 15-415 7 Definitions DBMS = Data Base Management System : the (commercial) system, like: DB2, Oracle, MS SQL-server,... Database system : DBMS + data + application programs Faloutsos 15-415 8 Motivating example Eg.: students, taking classes, obtaining grades; find my gpa <and other ad-hoc queries> Faloutsos 15-415 9 3
Faloutsos 15-415 Obvious solution: paper-based advantages? disadvantages? eg., student folders, alpha sorted Faloutsos 15-415 10 Obvious solution: paper-based advantages? cheap; easy to use disadvantages? eg., student folders, alpha sorted Faloutsos 15-415 11 Obvious solution: paper-based advantages? cheap; easy to use disadvantages? no ad hoc queries no sharing large physical foot-print Faloutsos 15-415 12 4
Faloutsos 15-415 Next obvious solution computer-based (flat) files + C (Java,...) programs to access them e.g., one (or more) UNIX/DOS files, with student records and their courses Faloutsos 15-415 13 Next obvious solution your layout for the student records? Faloutsos 15-415 14 Next obvious solution your layout for the student records? (eg., comma-separated values csv Smith,John,123,db,A,os,B Tompson,Peter,234 Atkinson,Mary,345,os,B,graphics,A Faloutsos 15-415 15 5
Faloutsos 15-415 Next obvious solution your layout for the student records? (many other layouts are fine, eg.: Smith,John,123 Tompson,Peter,234 Atkinson,Mary,345 123,db,A 123,os,B 345,os,B 345,graphics,A Faloutsos 15-415 16 Problems? Faloutsos 15-415 17 Problems? inconvenient access to data (need C++ expertize, plus knowledge of file-layout) data isolation data redundancy (and inconcistencies) integrity problems atomicity problems Faloutsos 15-415 18 6
Faloutsos 15-415 Problems? (cont d)... concurrent-access anomalies security problems Faloutsos 15-415 19 Problems? (cont d) [ why? because of two main reasons: file-layout description is buried within the C programs and there is no support for transactions (concurrency and recovery) ] DBMSs handle exactly these two problems Faloutsos 15-415 20 DBMS solution commercial/freeware DBMS & application programs Faloutsos 15-415 21 7
Faloutsos 15-415 Main vendors/products Commercial Oracle IBM/DB2 MS SQL-server Sybase Informix/IBM (MS Access,...) Open source Postgres (UCB) mysql, msql minibase (Wisc) Predator (Cornell) sqlite (sqlite.org) (www.acm.org/sigmod) Faloutsos 15-415 22 <Demo with sqlite3> Insert student and takes records Find the os class roster Find the GPA of www.cs.cmu.edu/~christos/courses/dbms-f09/files/sqldemo.zip Faloutsos 15-415 23 Detailed outline Introduction Motivating example How do DBMSs work? DDL, DML, views. Fundamental concepts DBMS users Overall system architecture Conclusions Faloutsos 15-415 24 8
Faloutsos 15-415 Pictorially: How do DBs work? select * from student DBMS data and meta-data = catalog = data dictionary Faloutsos 15-415 25 How do DBs work? % sqlite3 miniu.sql sqlite>create table student ( ssn fixed; name char(20) ); student ssn name Smith,John,123,db,A,os,B Tompson,Peter,234 Atkinson,Mary,345,os,B,graphics,A Faloutsos 15-415 26 How do DBs work? sqlite>insert into student values (123, Smith ); sqlite>select * from student; student ssn name 123 Smith Faloutsos 15-415 27 9
Faloutsos 15-415 create table student (ssn fixed, name char(20)); insert into student values(123, "Smith"); insert into student values(234, "Tompson"); insert into student values(345, "Atkinson"); -- see what we have inserted select * from student; ssn name ---------- ---------- 123 Smith 234 Tompson 345 Atkinson Faloutsos 15-415 28 How do DBs work? sqlite>create table takes ( ssn fixed, cid char(10), grade fixed)); takes ssn cid grade Faloutsos 15-415 29 -- register students in classes and give them grades drop table if exists takes; create table takes (ssn fixed, cid char(10), grade fixed); insert into takes values( 123, "db", 4); insert into takes values( 123, "os", 3); insert into takes values( 345, "os", 3); insert into takes values( 345, "graphics", 4); Smith,John,123,db,A,os,B Tompson,Peter,234 Atkinson,Mary,345,os,B,graphics,A Faloutsos 15-415 30 10
Faloutsos 15-415 -- see what we inserted select * from takes; ssn cid grade ---------- ---------- ---------- 123 db 4 123 os 3 345 os 3 345 graphics 4 Smith,John,123,db,A,os,B Tompson,Peter,234 Atkinson,Mary,345,os,B,graphics,A Faloutsos 15-415 31 How do DBs work - cont d More than one tables - joins Eg., roster (names only) for os student ssn name takes ssn cid grade Faloutsos 15-415 32 How do DBs work - cont d sqlite> select name from student, takes where student.ssn = takes.ssn and takes.c-id = os Faloutsos 15-415 33 11
Faloutsos 15-415 -- find the os class roster select name from student, takes where student.ssn = takes.ssn and cid="os"; name ---------- Smith Atkinson Smith,John,123,db,A,os,B Tompson,Peter,234 Atkinson,Mary,345,os,B,graphics,A Faloutsos 15-415 34 Views - a powerful tool! what and why? suppose secy is allowed to see only ssn s and GPAs, but not individual grades -> VIEWS! Faloutsos 15-415 35 takes ssn 123 123 345 345 Views sqlite> create view fellowship as ( select ssn, avg(grade) from takes group by ssn); cid db os os graphics grade 4 3 3 4 ssn avg(grade 123 3.5 345 3.5 Faloutsos 15-415 36 12
Faloutsos 15-415 Views Views = virtual tables Faloutsos 15-415 37 Views sqlite> select * from fellowship; takes ssn 123 123 345 345 cid db os os graphics grade 4 3 3 4 ssn avg(grade 123 3.5 345 3.5 Faloutsos 15-415 38 Views sql> grant select on fellowship to secy; ( grant not supported in sqlite) takes ssn 123 123 345 345 cid db os os graphics grade 4 3 3 4 ssn avg(grade 123 3.5 345 3.5 Faloutsos 15-415 39 13
Faloutsos 15-415 Iterating: advantages over (flat) files logical and physical data independence, because data layout, security etc info: stored explicitly on the disk concurrent access and transaction support Faloutsos 15-415 40 Disadvantages over (flat) files? Faloutsos 15-415 41 Disadvantages over (flat) files Price additional expertise (SQL/DBA) (hence: over-kill for small, single-user data sets But: mobile phones (eg., android) use sqlite) Faloutsos 15-415 42 14
Faloutsos 15-415 Detailed outline Introduction Motivating example How do DBMSs work? DDL, DML, views. Fundamental concepts DBMS users Overall system architecture Conclusions Faloutsos 15-415 43 Fundamental concepts 3-level architecture logical data independence physical data independence Faloutsos 15-415 44 view level logical level physical level 3-level architecture v1 v2 v3 Faloutsos 15-415 45 15
Faloutsos 15-415 3-level architecture view level logical level: eg., tables STUDENT(ssn, name) TAKES (ssn, cid, grade) physical level: how are these tables stored, how many bytes / attribute etc Faloutsos 15-415 46 3-level architecture view level, eg: v1: select ssn from student v2: select ssn, c-id from takes logical level physical level Faloutsos 15-415 47 3-level architecture -> hence, physical and logical data independence: logical D.I.:??? physical D.I.:??? Faloutsos 15-415 48 16
Faloutsos 15-415 3-level architecture -> hence, physical and logical data independence: logical D.I.: can add (drop) column; add/drop table physical D.I.: can add index; change record order Faloutsos 15-415 49 Detailed outline Introduction Motivating example How do DBMSs work? DDL, DML, views. Fundamental concepts DBMS users Overall system architecture Conclusions Faloutsos 15-415 50 naive users casual users Database users application programmers [ DBA (Data base administrator)] Faloutsos 15-415 51 17
Faloutsos 15-415 select * from student Casual users DBMS data and meta-data = catalog Faloutsos 15-415 52 ``Naive users Pictorially: DBMS data app. (eg., report generator) and meta-data = catalog Faloutsos 15-415 53 App. programmers those who write the applications (like the report generator ) Faloutsos 15-415 54 18
Faloutsos 15-415 DB Administrator (DBA) Duties? Faloutsos 15-415 55 DB Administrator (DBA) schema definition ( logical level) physical schema (storage structure, access methods schemas modifications granting authorizations integrity constraint specification Faloutsos 15-415 56 Detailed outline Introduction Motivating example How do DBMSs work? DDL, DML, views. Fundamental concepts DBMS users Overall system architecture Conclusions Faloutsos 15-415 57 19
Faloutsos 15-415 Overall system architecture [Users] DBMS query processor storage manager [Files] Faloutsos 15-415 58 naive app. pgmr casual DBA users app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 59 Overall system architecture query processor DML compiler embedded DML pre-compiler DDL interpreter Query evaluation engine Faloutsos 15-415 60 20
Faloutsos 15-415 Overall system architecture (cont d) storage manager authorization and integrity manager transaction manager buffer manager file manager Faloutsos 15-415 61 Overall system architecture (cont d) Files data files data dictionary = catalog (= meta-data) indices statistical data Faloutsos 15-415 62 Some examples: DBA doing a DDL (data definition language) operation, eg., create table student... Faloutsos 15-415 63 21
Faloutsos 15-415 naive app. pgmr casual DBA users app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 64 Some examples: casual user, asking for an update, eg.: update student set name to smith where ssn = 345 Faloutsos 15-415 65 naive app. pgmr casual DBA users app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 66 22
Faloutsos 15-415 naive app. pgmr casual DBA users app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 67 naive app. pgmr casual DBA users app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 68 Some examples: app. programmer, creating a report, eg main(){... exec sql select * from student... } Faloutsos 15-415 69 23
Faloutsos 15-415 naive app. pgmr casual DBA users pgm (src) app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 70 Some examples: naive user, running the previous app. Faloutsos 15-415 71 naive app. pgmr casual DBA users pgm (src) app. pgm(o) emb. DML DML proc. query eval. DDL int. query proc. trans. mgr buff. mgr file mgr storage mgr. data meta-data Faloutsos 15-415 72 24
Faloutsos 15-415 Detailed outline Introduction Motivating example How do DBMSs work? DDL, DML, views. Fundamental concepts DBMS users Overall system architecture Conclusions Faloutsos 15-415 73 Conclusions (relational) DBMSs: electronic record keepers customize them with create table commands ask SQL queries to retrieve info Faloutsos 15-415 74 Conclusions cont d main advantages over (flat) files & scripts: logical + physical data independence (ie., flexibility of adding new attributes, new tables and indices) concurrency control and recovery Faloutsos 15-415 75 25