Part I: Introduction. Outline. Introduction to Databases & SQL. Motivating Example Flat Files. Relational Databases. Problems with Flat Files

Size: px

Start display at page:

Download "Part I: Introduction. Outline. Introduction to Databases & SQL. Motivating Example Flat Files. Relational Databases. Problems with Flat Files"

Peter Hancock
5 years ago
Views:

1 Outline Introduction to Databases & SQL Dr. Christopher M. Bourke 1. Introduction 2. Creating Tables 3. Manipulating Data 4. Querying Data 5. Extended Demonstration 6. Designing Databases 7. Extended Demonstration Motivating Example Flat Files Consider the following data, stored in a flat file: Part I: Introduction Course Course Name Student NUID waits, tom tomwaits@hotmail.com CSCE 156 Intro to CSII Lou Reed reed@gmail.com CSCE 156 Intro to CSII Tom Waits twaits@ .com CSCE 230 Computer Hardware Student, J jstudent@geocities.com CSCE 156 Intro to CSII Student, John jstudent@geocities.com CSCE 230 Computer Hardware John Student jstudent@geocities.com CSCE 235 Discrete Math Student, John jstudent@geocities.com CSCE 235 Discrete Math Tom Waits twaits@ .com NONE Null Tom Waits twaits@ .com Table: Course Enrollment Data Problems with Flat Files Relational Databases Problems: Formatting Issues Repetition of data Incomplete data Integrity of data Organizational problems: any aggregation requires processing all records Updating information is difficult (must enumerate all possible changes, side effects so that information is not lost) Concurrency Issues Solution: Relational Database Management System (RDBMS) Represents data as tuples (pairs, triples, etc.) Data is grouped into relations E. F. Codd (IBM, 1970)

2 Key Aspects I Key Aspects II Data is stored in tables Tables have columns (or fields) that have a unique name and type (integer, string) Each column stores a single piece of data Each row represents a single record Each row may have a unique primary key (PK) which uniquely identifies each record Records in different tables are related to each other through foreign keys (FK) Order of rows/columns is meaningless Key Aspects III Key Aspects IV studentid firstname lastname nuid courseid name description CSCE Computer Science 1 Tom Waits II CSCE 230 Computer Hardware 2 Lou Reed CSCE 235 Discrete Mathematics 5 John Student id studentid address 8 2 reed@gmail.com 10 1 tomwaits@hotmail.com enrollmentid studentid courseid semester Fall twaits@ .com Fall jstudent@geocities.com Fall Fall Spring Fall 2011 Supports Transactions: an interaction or batch of interactions treated as one unit Constraints Allowing or disallowing NULL Disallowing bad values (ranges) Enforcing formatting (capitalization, precision) Limiting combinations of data fields ACID Principles I RDBMS s Atomicity Data transactions must be an all-or-nothing process Atomic operation: not divisible or decomposable Consistency Transactions will retain a state of consistency All constraints, triggers, cascades preserve a valid state after the transaction has completed Isolation No transaction interferes or is even aware of another; state transitions are equivalent to serial transactions Durability Once committed, a transaction remains so Data is to be protected from catastrophic error (power loss/crash) Commercial Systems: MS Access, MySQL (Oracle), MariaDB, PostgreSQL, Informix, DB2 SQLServer, Oracle, SQLite NoSQL (non relational databases) Key-value databases: MongoDB, Cassandra, Spanner Graph-based: Neo4j, OrientDB Redis Database SaaS: Amazon Web Services (Dynamo)

3 Advantages I Advantages II Data is structured instead of just there Better organization Duplication is minimized (with proper normalization) Updating information is easier Organization of data allows easy access Organization allows aggregation and more complex information Data integrity can be enforced (data types and user defined constraints) Faster Scalable Security Portability Concurrency Structured Query Language Structured Query Language CRUD We interact with RDBMs using Structured Query Language (SQL) Common language/interface to most databases Developed by Chamberlin & Boyce, IBM 1974 Implementations may violate standards: portability issues Comments: # Keywords are case insensitive Create & manage tables: create, alter, drop Basic SQL functionality: CRUD: Create insert new records into existing tables ( insert ) Retrieve get a (subset) of data from specific rows/columns ( select ) Update modify data in fields in specified rows ( update ) Destroy delete specific rows from table(s) ( delete ) MySQL Getting Started Part II: Creating a Database Need access to a database server Need to use a client to connect to the server CLI: mysql MySQL Workbench 1 use databasename; 2 show tables; 3 describe TableName;

4 MySQL CSCE Students Creating Tables Syntax You only have access to a database with the same name as your login Host: cse.unl.edu Port: 3306 User: your CSE login ID Password: go to Do not use the same password as your login More: 1 create table TableName ( 2 columnname columntype [options], ); Options: primary key auto_increment (for primary keys) not null default (value) Column Types Creating Tables Example varchar(n) variable character field of up to n characters integer (or int ) decimal(65,30), float, double Others: aliases, vendor-specific keywords, date/time, other data formats (JSON) 1 create table Book ( 2 bookid integer primary key auto_increment not null, 3 title varchar(255) not null, 4 author varchar(255), 5 isbn varchar(255) not null default, 6 dewey float, 7 numcopies integer default 0 8 ); Primary Keys Keys A primary key gives each record a unique identity At most one primary key per table No two rows can have the same primary key value Must be able to uniquely identify all records (not just those that exist) PKs can be one or more columns (composite key) Should not use/allow NULL values Can/should External identifiers should not be used Should use integers, not varchars or floats Tables can have multiple (non-primary) keys May be a combination of columns (composite key) May be declared non-unique in which case it serves as an index (allows database lookup optimization) Syntax: key(column1, column2,...)

Indices and Unique Constraints Foreign Keys I The keyword index can also be used We can make them unique by using unique Syntax: constraint constraintname unique index(columnname) Multiple columns:.

5 Indices and Unique Constraints Foreign Keys I The keyword index can also be used We can make them unique by using unique Syntax: constraint constraintname unique index(columnname) Multiple columns:... index(c1, c2) Relations between records in different tables can be made with foreign keys A foreign key is a column that references a key (PK or regular key) in another table Inserts cannot occur if the referenced record does not exist Foreign keys establish relationships between tables: One-to-one (avoid) One-to-many relations Many-to-one Many-to-Many relations: requires a Join Table Syntax: foreign key (columnname) references tablename(columnname) Demonstration I Write SQL to define two tables: one for authors and one for books. Simplify the book table, but include an ISBN as well as a constraint to keep it unique. Model the fact that an author may write more than one book. Demonstration I Solution 1 create table Author ( 2 authorid integer primary key not null auto_increment, 3 firstname varchar(255), 4 lastname varchar(255) not null 5 ); 6 7 create table Book ( 8 bookid integer primary key not null auto_increment, 9 authorid integer not null, 10 title varchar(255), 11 isbn varchar(255) not null, 12 numcopies integer default 0, 13 foreign key (authorid) references Author(authorId), 14 constraint uniqueisbn UNIQUE INDEX(isbn) 15 ); Demonstration II Solution Part III: Manipulating Data Figure: Entity-Relation Diagram for the Book/Author tables

6 Inserting Data Inserting Data I Examples Creating data involves inserting new records into a table Values are specified with literals Numerical literals String literals: use single quote characters Syntax: insert into tablename (c1, c2,...) values (v1, v2,...); 1 insert into Author (firstname, lastname) values 2 ( Normal, Mailer ); 3 --or 4 insert into Author (authorid, firstname, lastname) values 5 (1234, Normal, Mailer ); 6 7 insert into Book (title, authorid, isbn) values 8 ( The Naked and the Dead, 1234, ); 9 insert into Book (title, authorid, isbn) values 10 ( The Naked and the Dead, 11 (select authorid from Author where lastname = Mailer ), ); Inserting Data II Examples Updating Data 1 --Errors: 2 /* 3 insert into Book (title, authorid, isbn) values 4 ( The Executioner s Song, 1234, ); 5 */ 6 7 insert into Book (title, authorid, isbn) values 8 ( The Executioner\ s Song, 1234, ); Existing data can be modified using an update statement Should be used in conjunction with clauses Syntax: 1 update tablename set c1 = v1, c2 = v2,... 2 where [condition]; Updating Data Examples Deleting Data 1 --dangerous: 2 update Author set firstname = Norman ; correct: 5 update Author set firstname = Norman 6 where authorid = 1234; Records can be deleted using a delete statement Should be used in conjunction with clauses Unless you really want to delete everything Syntax: delete from tablename where [condition]

7 Deleting Data Examples 1 --we cannot delete norman: 2 delete from Author where lastname = Mailer ; 3 delete from Author where authorid = 1234; 4 5 delete from Book where bookid = 1; 6 delete from Book where title = The Naked and the Dead ; deletes all of Norman s book entries: 9 delete from Book where authorid = 1234; 10 --now we could delete norman Part IV: Querying Data Querying Data Querying Data Examples Data can be retrieved using the select statement Syntax: select c1, c2,... from tablename where [condition]; Can select all columns by using the * wildcard: select * from Book; 1 select * from Book; 2 select * from Book where authorid = 1234; 3 select * from Book where isbn = ; 4 5 select lastname, firstname from Author; 6 select lastname, firstname from Author where authorid = 1234; Querying Data Aliasing Complex where Clause Names of the columns are part of the database SQL allows us to rename them in result of our query using aliasing Syntax: columnname as columnalias Sometimes necessary if column has no name (aggregates) 1 select title as booktitle, 2 numcopies as numberofcopies 3 from Book; Queries can be quantified using the where clause Only records matching the condition will be affected (updated, deleted, selected) Compound conditions can be composed using parentheses and the and and or keywords 1 select * from Book 2 where numcopies > 10 AND 3 (title!= The Naked and the Dead 4 OR authorid = 1234); To check nullity: where title is null, where title is not null

8 distinct Clause like Clause Many records may have the same column value To query only unique values you may use the distinct keyword 1 select distinct lastname from Author; 2 select distinct authorid from Book; String ( varchar ) values can be searched/partially matched using the like clause Used in conjunction with the string wildcard, % 1 select * from Book where isbn like 123% ; 2 select * from Author where lastname like %ailer% ; in Clause order by Clause The in clause allows you to do conditionals on a set of values May be used in conjunction with a nested query 1 select * from Book where isbn in ( , , ); 3 4 select * from Book where authorid in 5 (select authorid from Author where lastname like M% ); In general, the order of the results of a select clause is nondeterministic To impose an order, you can use order by Can order along multiple columns Can order descending ( desc ) or ascending ( asc ) order 1 select * from Book order by title; 2 select * from Book order by authorid, title desc; Aggregate Functions Aggregate Functions Examples Aggregate functions allow us to compute data in the database without processing all the data in code count provides a mechanism to count the number of records Other aggregate functions: max, min, avg, sum null values are ignored/treated as zero 1 --number of Book records: 2 select count(*) as numberoftitles from Book; 3 --most common book: 4 select max(numcopies) from Book; nested queries: 7 select * from Book 8 where numcopies = (select max(numcopies) from Book);

9 GROUP BY clause group by clause Example Illustrated The group by clause allows you to project data with common values into a smaller set of rows Used in conjunction with aggregate functions to do more complicated aggregates Example: find total copies of all books grouped by their author 1 select author, sum(numcopies) as totalcopies 2 from Book group by author; Table content: title author numcopies Naked and the Dead Norman Mailer 10 Dirk Gently s Holistic Detective Agency Douglas Adams 4 Barbary Shore Norman Mailer 3 The Hitchhiker s Guide to the Galaxy Douglas Adams 2 The Long Dark Tea-Time of the Soul Douglas Adams 1 Ender s Game Orson Scott Card 7 GROUP BY clause Example Illustrated group by clause Example Illustrated Grouping: title author numcopies Naked and the Dead Norman Mailer 10 Barbary Shore Norman Mailer 3 Dirk Gently s Holistic Detective Agency Douglas Adams 4 The Hitchhiker s Guide to the Galaxy Douglas Adams 2 The Long Dark Tea-Time of the Soul Douglas Adams 1 Ender s Game Orson Scott Card 7 Projection & aggregation: author totalcopies Norman Mailer 13 Douglas Adams 7 Orson Scott Card 7 having clause Joins A join is a clause that combines records from two or more tables. Projected data can be further filtered using the having clause having is evaluated after group by which is evaluated after any where clause 1 select author, sum(numcopies) as totalcopies 2 from Book where isbn like %9 3 group by author 4 having totalcopies > 10; Result is a set of columns/rows (a table ) Results from tables A, B are joined by shared values in specified columns Common to join via foreign keys Table names can be aliased for convenience Types of joins we ll look at: (inner) join left (outer) join Other types of joins: Self-join, cross join (cartesian product), right outer joins, full outer joins

10 Inner Join I Inner Join II Inner Join: Most common type of join Combines rows of table A with rows of table B for all records that satisfy some predicate Predicate provided by the on clause Common to omit inner and simply use the join keyword 1 select * from Book b 2 join Author a on b.authorid = a.authorid; 3 4 select * from Author a 5 join Book b on a.authorid = b.authorid; 6 7 select * from Book b 8 join Author a on b.authorid = b.authorid 9 where a.title like The% ; select a.firstname as firstname, 12 a.lastname as lastname, 13 b.title as booktitle 14 from Book b 15 join Author a on b.authorid = b.authorid; Inner Join Example Illustrated Inner Join Example Illustrated 1 select s.studentid, s.lastname, e.address 2 from student s 3 join e on s.studentid = e.studentid; studentid lastname firstname 1234 Castro Starlin 5678 Rizzo Anthony 1122 Sveum Dale 9988 Sandberg Ryne (a) Student Table id studentid address rsandberg@cubbies.net rsberg@unl.edu rizzo@cubs.com number13@cubs.com (b) Table studentid lastname address 1234 Castro number13@cubs.com 5678 Rizzo rizzo@cubs.com 9988 Sandberg rsandberg@cubbies.net 9988 Sandberg rsberg@unl.edu Table: Joined tables Left (Outer) Join I Left (Outer) Join II A Left Outer Join joins table A to table B, preserving all records in table A Records in A with no matching records in B will have null values for columns in B Syntax: omit outer and use left join 1 --no difference as a book cannot exist without an author: 2 select * from Book b 3 left join Author a on b.authorid = a.authorid; includes authors even if they have no book records: 6 select * from Author a 7 left join Book b on a.authorid = b.authorid; 8 9 select s.studentid, s.lastname, e.address 10 from student s 11 left join e on s.studentid = e.studentid;

Left (Outer) Join III Other Joins Nice tutorial: http://www.codeproject.com/articles/33052/visual-representation-of-sql-joins student id last name adddress 1234 Castro number13@cubs.

11 Left (Outer) Join III Other Joins Nice tutorial: student id last name adddress 1234 Castro 5678 Rizzo 9988 Sandberg 9988 Sandberg 1122 Sveum null Table: Left-Joined Tables Figure: Types of Joins SQL Query Demonstrations Part V: Extended Demonstration Figure: Video Game Database Database Design Table Design Identify all entities in different tables Part VI: Designing a Database Ask what defines an entity to determine the columns and their types Properly define whether or not a column should be allowed to be null and/or defaults Each table should have a primary key Relations between tables should be identified and defined with foreign keys Security concerns: don t store sensitive information (passwords) in plaintext

12 Normalization I Normalization II Normalizing a database is the process of separating data into different tables to reduce or eliminate data redundancy and reduce anomolies (preserve integrity) when data is changed. It also reduces the amount of book keeping necessary to make such changes. 1 Normal Form: the domain of each attribute in each table has only atomic values: each column value represents a single value Example: Allowing multiple records for one Person Storing these as (say) a CSV in one column is a violation of 1NF Hardcoding a fixed number of columns ( 1, 2, 3) for multiple values is a violation of 1NF Separating records out into another table and associating them via a FK conforms to 1NF 2 Normal Form: 1NF and no non-prime attribute is dependent on any proper subset of prime attributes Mostly relevant for tables with composite PKs (multiple columns) If another, non-prime column is dependent only on a subset of these, its not 2NF Must split it out groups of data into multiple tables and relate them through FKs so that non-prime column is dependent only on the subset of prime columns Example: a purchase record may contain: customerid, storeid, storelocation Using an auto-generated primary key gives you 2NF automatically Normalization III Normalization IV 3 Normal Form: 2NF and no non-prime column is transitively dependent on the key No non-prime column may depend on another non-prime column Example: Storing a price-per-unit, quantity, and total (total can be derived from the other two) Example: OrderId-CustomerId-CustomerName (OrderId is the PK, CustomerId is FK, CustomerName should be derivable from CustomerId) Bottom line: 3NF is common sense; design your database to be normalized to begin with Every non-key attribute must provide a fact about the key (1NF), the whole key (2NF), and nothing but the key (3NF), so help me Codd 1 1 Edgar Codd, inventor of the relational model, Best Practice Tip 1 Primary Keys Good Practice Tip 2 Use consistent naming conventions Primary Keys Should be integers (not varchars nor floats) Best to allow the database to do key management (auto increment) Should not be based on external identifiers that are not controlled by the database (SSNs, ISBNs, etc.) Be consistent in naming conventions ( tablenameid ) Short, simple, descriptive names Limit abbreviations, acronyms Use consistent styling Tables: UpperCamelCase Columns: lowercamelcase Use singular forms, avoid pluralizations Primary key field: tablenameid Foreign key fields should match the fields they refer to End goal: unambiguous, consistent, self-documenting

13 Good Practice Tip 3 Ensure Good Data Integrity Good Practice Tip 4 Keep Business Logic Out! Data can break code, code should not break data. Data/databases are a service to code Different code, different modules can access the same data The database does not use the code! Should do everything you can to prevent bad code from harming data (constraints, foreign & primary keys, etc). Database is your last line of defense against bad code Databases offer programming functionality Triggers, cascades, stored procedures, etc. Use them sparingly!!! RDMSs are for the management and storage of data, not business logic Demarcation of responsibility DBAs should not have to be Application Programmers, and vice versa Good Practice Tip 5 Hard vs Soft Deletes delete removes records entirely ( hard delete) May not want the data to be permanently deleted Tables may be given an active/inactive flag (boolean) Deleting a record: set the flag to inactive ( soft delete) Application is responsible for only querying for active records Extra care must be taken Part VII: Database Design Demonstration Designing A Database Demonstration Designing A Database Design a database to support a course roster system for the University of Nebraska Lincoln system. The database design should be able to model students (with their unique NUIDs), courses, and their relation (ability of students to enroll in courses). The system will also need to students about updates in enrollment, so be sure your model is able to incorporate this functionality. Figure: A normalized database design.

14 End Result Data from flat file Pieces of data are now organized and have a specific type No duplication of data Entities are represented by IDs, ensuring identity (Tom Waits is now the same as t. Waits) Data integrity is enforced (only one NUID per Student) Relations are well-defined A student has (s) A course has student(s) and a student has course(s) studentid firstname lastname nuid courseid name description CSCE Computer Science 1 Tom Waits II CSCE 230 Computer Hardware 2 Lou Reed CSCE 235 Discrete Mathematics 5 John Student id studentid address 8 2 reed@gmail.com 10 1 tomwaits@hotmail.com enrollmentid studentid courseid semester Fall twaits@ .com Fall jstudent@geocities.com Fall Fall Spring Fall 2011

Database Systems: Design, Implementation, and Management Tenth Edition. Chapter 7 Introduction to Structured Query Language (SQL)

Database Systems: Design, Implementation, and Management Tenth Edition Chapter 7 Introduction to Structured Query Language (SQL) Objectives In this chapter, students will learn: The basic commands and