0. Organizational Issues. 0. Why should you be here? 0. Why should you be here? 0. Why should you be here? Relational Database Systems

Transcription

1 0. Organizational Issues Relational Database Systems 1 Wolf-Tilo Balke Christoph Lofi Institut für Informationssysteme Technische Universität Braunschweig Lecture 07. November February :00-17:15h (3 lecture hours!) Exercises, detours, and home work discussion integrated into lecture 4 Credits Exams Written exam on % of exercise points needed to be eligible for the exam Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 2 0. Why should you be here? 0. Why should you be here? Database system are an integral part of most businesses, workflows and software products There is an abundance of jobs for people with good database skills Help yourself to put you into a good position within the job market Prepare for a sunny and wealthy future! Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 3 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 4 0. Why should you be here? Job descriptions are exactly what we do here 0. Why should you be here? Lecture focuses on all basic aspects of database usage relevant for practical application Topics Basic concepts Conceptual modeling Relational algebra and SQL DB application interfaces Additional lectures for DB implementation aspects and special DB applications in master s course (Relational Database Systems II) Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 5 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 6 1

2 0. Why should you be here? For our economically centered listeners 0. Structure of the Course Basic course in databases Relational Databases I (BA) What can we do with an DBMS? Conceptual modeling, data retrieval, relational model, SQL, building applications, basic data models SQL-Praktikum (BA) Advanced Features of SQL and database programming Hands-on experience Relational Databases II (MA) How can we implement a DBMS? Storage models, query optimization, transactions, concurrency control, recovery, data security Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 7 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 8 0. Structure of the Course Advanced courses in databases (MA) Information retrieval & Web search Multimedia databases Distributed databases & P2P data management Knowledge-based systems & deductive databases Data warehousing & OLAP XML databases Bio-information systems Geo-information systems 0. Recommended Literature Fundamentals of Database Systems (EN) Elmasri & Navathe Addison Wesley, ISBN X Database Systems Concepts (SKS) Silberschatz, Korth & Sudarshan McGraw Hill, ISBN Database Systems (GUW) Garcia-Molina, Ullman & Widom Prentice Hall, ISBN Datenbanksysteme (KE) Kemper & Eickler Oldenbourg, ISBN Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 9 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Recommended Literature Database Modeling and Design. Logical Design Teorey, Lightstone & Nadeau Morgan Kaufmann ISBN X SQL Cookbook Anthony Molinaro O'Reilly Media ISBN Using the new DB2 Don Chamberlin AP Professional ISBN Introduction Introduction What is a database? Characteristics of a database Database users Data models History of databases Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 11 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 12 2

3 1.1 What is a database? 1.1 What is a database? Managing large amounts of data is an integral part of most nowadays business and governmental activities Collecting taxes Bank and account management Book keeping Airline reservations Employee organization Databases are needed to manage that vast amount of data A database (DB) is a collection of related data Represents some aspects of the real world Universe of Discourse Data is logically coherent Is provided for an intended group of users and applications EN EN What is a database? As for today, the database industry is one of the most successful branches of computer science Constantly growing since the 60s More than $8 billion revenue per year DB systems found in nearly any application Ranging from large commercial transaction-processing systems to small open-source systems for your Web site 1.1 What is a database? A database management system (DBMS) is a collection of programs to maintain a database, i.e. for Definition of Data and Structure Physical Construction Manipulation Sharing/Protecting Persistence/Recovery EN 1.1 EN Filesystems 1.1 Filesystems A file system is not a database! File management systems are physical interfaces Account Data Customer Data Loans F i l e S y s t e m App 1 App 2 Customer Letters Money Transfer Balance Sheets Advantages Fast and easy access Disadvantages Uncontrolled redundancy Inconsistent data Limited data sharing and access rights Poor enforcement of standards Excessive data and access paths maintenance

4 1.1 Databases 1.1 Databases Databases are logical interfaces Retrieval of data using data semantics Controlled redundancy Data consistency & integrity constraints Effective and secure data sharing Backup and recovery However More complex More expensive data access DBMS replaced previously dominant file-based systems in banking due to special requirements Simultaneous and quick access is necessary Failures and loss of data cannot be tolerated Data always has to remain in a consistent state Frequent queries and modifications Databases control redundancy Same data used by different applications/tasks is only stored once Access via a single interface provided by DBMS Redundancy only purposefully used to speed up data access (e.g. materialized views) Problems of uncontrolled redundancy Difficulties in consistently updating data Waste of storage space Databases are well-structured (e.g. ER-Model) Catalog (data dictionary) contains all meta-data Defines the structure of the data in the database Example: ER-Model Simple banking system firstname lastname ID customer address has account type balance EN EN Databases support Declarative Querying Just specify what you want, not how and from where to get it Queries are abstracted from actual physical organization and storage of data Get the first name of all customers with last name Smith File system: trouble with physical organization of data Load file c:\datasets\customerdata.csv Build a regular expression and iterate over lines: If 2 nd word in line equals Smith, return 3 rd word Stop when EoF is reached Database system: simply query SELECT firstname FROM customerdata WHERE lastname= Smith Databases aim at efficient manipulation of data Physical tuning allows for good data allocation Indexes speed up search and access Query plans are optimized for improved performance Data File Example: Simple Index type balance Index File saving saving checking saving checking checking saving saving Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig

5 Isolation between applications and data Database employs data abstraction by providing data models Applications work only on the conceptual representation of data Data is strictly typed (Integer, Timestamp, VarChar, ) Details on where data is actually stored and how it is accessed is hidden by the DBMS Applications can access and manipulate data by invoking abstract operations (e.g. SQL Select statements) DBMS-controlled parts of the file system are strongly protected against outside manipulation (tablespaces) Example: Schema is changed and table-space moved without an application noticing balance Application SELEC T FROM account WHERE balance>0 DBMS Disk 1 Disk 2 EN Example: Schema is changed and table-space moved without an application noticing balance Application SELEC T FROM account WHERE balance>0 DBMS Disk 1 Disk 2 type balance saving saving checking saving Supports multiple views of the data Views provide a different perspective of the DB A user s conceptual understanding or task-based excerpt of all data (e.g. aggregations) Security considerations and access control (e.g. projections) For the application, a view does not differ from a table Views may contain subsets of a DB and/or contain virtual data Virtual data is derived from the DB (mostly by simple SQL statements, e.g. joins over several tables) Can either be computed at query time or materialized upfront EN Example Views: Projection Saving account clerk vs. checking account clerk Original Table type balance saving saving checking saving checking checking saving saving Saving View balance Checking View balance Sharing of data and support for atomic multiuser transactions Multiple user and applications may access the DB at the same time Concurrency control is necessary for maintaining consistency Transactions need to be atomic and isolated from each other EN

6 Example: Atomic Transactions Program: Transfer X Euro from Account1 to Account2 1. Deduce amount X from Account1 2. Add amount X to Account2 Example: Atomic Transactions Program: Transfer X Euro from Account1 to Account2 1. Deduce amount X from Account1 2. Add amount X to Account2 But what happens if system fails between step 1 and 2? Example: Multi-User Transactions Program: withdraw amount X from Account 1 1. Read old balance from DB 2. New balance := old balance X 3. Write new balance back to the DB Problem: Dirty Read Account 1 has 500 User 1 wants deduce 20 User 2 wants to deduce 80 at the same time Without multi-user transaction, account will have either 480 or 420, but not the correct 400 Persistence of data and disaster recovery Data needs to be persistent and accessible at all times Quick recovery from system crashes without data loss Recovery from natural disasters ( fire, earthquakes, ) 33 EN Database Users 1.1 Database Users A large DBMS system usually involves several groups of persons (many job opportunities for people with DB knowledge ) Persons directly involved with a certain DB Database Administrators Responsible for tuning and maintaining the DBMS system Management of storage space, security, hardware, software, etc Database Designers Identifies the data that needs to be stored and chooses appropriate data structures and representations Covers the needs of all users in the design Application Programmers Identify the requirements of the end-users Develop the software which is used by (naïve) end-users to interact with the DB Cooperate closely with DB designers Persons working behind the scenes DBMS Designer and Implementers Implement the DBMS software Tool developers Develop generic tools extending the DBMS functionalities Operators and Maintenance Personnel Responsible for actually running and maintaining the DBMS hardware EN 1.4 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 35 EN 1.4 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 36 6

7 1.1 Database Users 1.1 Database Users End Users All people using the DB to perform their job Split into Naïve Users Make up most DB users Usually repeat similar tasks over and over Are supported by predesigned interfaces for their tasks Examples: bank tellers, reservation clerks, Sophisticated Users Require complex non-standard operations and views from the DB Are familiar with the facilities of the DBMS Can solve their problems themselves, but require complex tools Examples: engineers, scientists, business analyst, Casual Users Use DB only from time to time, but need to perform different tasks Are familiar with query languages to solve their needs Examples: middle- or high-level managers EN 1.4 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 37 EN 1.4 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 38 In databases the specific semantics of the data matter a lot What is described? What values are sensible/correct? What data belongs together? What data is often/rarely accessed? Example: describe the age of a person Semantic definition: the number of years elapsed since a person s birthday Integer data type Always: 0 age 120 Connected to the person s name, passport id, etc. May often be retrieved, but should be protected Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 39 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 40 A data model is an abstract model that describes how data is represented and accessed E.g., the network model, relational model, objectoriented model, Warning: the term is used ambiguously A data model theory is a formal description of how data may be structured and accessed A data model instance applies a data model theory to create an instance for some particular application To do that a data model needs three parts Structural part Data structures which are used to create databases representing the objects modeled Integrity part Rules expressing the constraints placed on these data structures to ensure structural integrity Manipulation part Operators which can be applied to the data structures, to update and query the data contained in the database Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 41 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 42 7

8 Data models are instanced by schemas which may be one of three kinds (ANSI,1975) A conceptual schema describes the semantics of a domain, i.e. the scope of the model The kinds of facts or propositions that can be expressed using the model A logical schema describes the semantics, as represented by a particular data manipulation technology Tables and columns, object oriented classes, XML tags,... A physical schema describes the physical means by which data are stored Partitions, tablespaces, indices,... In the pure modeling process the designer models only semantics, the rest is automatic Unfortunately, the three tasks are usually integrated conceptual design ER-diagram UML, logical design tables, columns, physical design tablespaces, Indices, Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 43 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 44 The idea behind the three kinds of instances is to maintain their independence Physical independence means that the storage design can be altered without affecting logical or conceptual schemas Logically it does not matter where exactly the data about a person s age is stored, it is still the same data Logical independence means that the logical design can be altered without affecting the data semantics It does not matter whether a person s age is directly stored or computed from the person s birthdate Shortcomings of specific data models Depending on the application modelers will often produce different data models of the same domain Bringing the models of for instance different companies together is difficult Data exchange and integration between organizations is severely hampered Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 45 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 46 Often differences are attributable to different levels of abstraction in the models Different in the kinds of facts that can be instantiated The semantic expressiveness of the models is different Extensions are often necessary, but difficult E.g., when the focus changes or new information about the domain becomes available The model limits what is expressible about a domain Changes sometimes need complete re-modeling of the schema Generic data models are generalizations of conventional data models Definition of standardized general relation types, together with the kinds of things that may be related by such a relation type Similar to the definition of a natural language Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 47 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 48 8

9 Example: a generic data model may define relation types such as classification relation as a binary relation between an individual thing and a kind of thing (a class) part-whole relation, as a binary relation between two things one with the role of part, the other with the role of whole Regardless of the kind of things that are related Current state of the art: Data is best structured in (relational) tables! Modeling data in tables is very natural and efficient There is no real other alternative to do it Think: index card! All data about an object on each single card Ordered by one attribute Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 49 BUT: is this true? We owe this belief to Edgar F. Codd around1970 Before that, people had a very different perspective on what data actually is Databases have an exceptional history of development High synergy between academic, governmental and industry research Much to be learned from it Most popular concepts used today were invented decades ago Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 51 Early Beginnings 1880: U.S. Bureau of Census charges Herman Hollerith to develop a machine for storing census data Result: Punch-card tabulator machine 1880 census took 8 years to complete 1890 census needed just one year Lead to the founding of IBM International Business Machines Data processing machines soon established in book keeping During WW1, many data collection, sorting and reporting tasks in industry and governmental organization was performed with punch-cards 1935: U.S. Social Security Act required continuous report on all 26 million governmental employees World biggest bookkeeping job Mechanical punch card systems not powerful enough IBM develops electric UNIVAC1 punch card processor in

10 In 1959, U.S. dominated the (still highly active) punch card machine market Within the U.S., the Pentagon alone used more than 200 data processing computers, costing $70 million per year In 1964, the term data base appeared for the first time in military computing using time sharing systems Data could be shared among users Data model very close to punch cards Master files bound to a specific application Master Files Each application had its own master files Similar data needed by multiple applications was duplicated Consistency problems when updating data Each file held records of one type represented by a set of fixed fields Inspired by punch cards and optimized for magnetic tapes Relationships between different records or type of records could not be done (restricted by model and available hardware) Master Files Highly hardware oriented approach Data stored in independent flat files Account Data Application 1 Application 2 Customer Data? Customer Data Customer Credibility Master Files had several severe problems To turn stored data into a proper database, following tasks needed to be solved Data consolidation Data needed to be stored in a central place, accessible for all applications Relationships among records required Data independence Data needs to be independent of low level programming languages (e.g. assembler) Data protection Data need to be protected against loss and abuse Data consolidation lead to the development of data models Hierarchical data model Network data model Relational data model Object-oriented data model Semantic data model Data independence lead to the development of query models and high-level languages Relational Algebra, SQL Data protection lead to development of transactions, backup schemes and security protocols Hierarchical Data Model First appearance in the IBM IMS database system designed for the Apollo Program in1966 Still, as of % of all Fortune 1000 companies use IBM IMS in their data backbone Benefits from advances in hardware design Random access main memory and tape media available Data may be organized in a tree structure Initially, tree had maximum depth of two 10

11 Hierarchical Data Model Each type of record has one or multiple own files/tables Hierarchical One-To-Many relationships possible Resembles vaguely a file system organization Customer CustomerNumber Name 1000 Mickey Mouse 1001 Dagobert Duck 1002 Donald Duck 1003 Goofy Accounts - MickeyMouse Balance Accounts Dagobert Duck Balance 2000 $ 934,3243,435, ,543,123, ,432,355,112 Hierarchical Data Model Advantages Allowed for 1:n relationships Could easily be stored on tape media Disadvantages No n:m relationships No data independence Network Data Model In the mid-60th, direct access storage devices (DASD) gained momentum Mainly hard disks More complex storage schemes possible Hierarchical model failed for, e.g. Bill-Of-Material- Processing (BOMP) Many-To-Many Relationships needed Development of the IBM DBOMP System 1960 Result: Network Model Two types of files: Master Files, Chain Files Chain file entries could chain any master file entry to another one Network Data Model The model was standardized by Charles W. Bachman for the CODASYL (Conference of Data Systems Languages) Consortium in 1969 Thus, also called CODASYL Model Allowed for more natural modeling of associations Advantage Many-Many-Relationships Disadvantage No declarative queries Querying using data path What s wrong with all that? Strong degree of hardware orientation No proper abstraction of data and decoupling of its application Each database needed to be handcrafted for its application To change something in the data-schema, a sharplooking guy in a white shirt and black rims had to do the programming by hand No real mathematical foundation no high-level data languages The relational data model Published by Edgar F. Ted Codd in 1970 after several years of work A Relational Model of Data for Large Shared Data Banks, Communications of the ACM, Vol. 13 (6), 1970 Employee of IBM Research IBM ignored his idea for a long time as not practical while pushing it s hierarchical IMS system Other researchers in the field also rejected his theories Finally got the Turing Award in 1981 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 65 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 66 11

12 Basic Idea: Database is seen as a collection of predicates over a finite set of predicate variables Tables correspond to predicate variables The content of the DB is modeled as a set of relations such that all predicates are satisfied Queries are also predicates Queries and data are very similar Allows for declarative querying It s really like a collection of index cards More details during the next weeks Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 67 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 68 Beginning 1977, Lawrence J. Ellison picked up the idea and created Oracle DB (currently version 11g) And became insanely rich long time in the Top 10 of the richest people In 2007 Oracle ranked third on the list of largest software companies in the world, after Microsoft and IBM Oracle also sells a suite of business applications Oracle ebusiness Suite Includes software to perform financial- and manufacturing-related functions, customer relationsship management, enterprise resource planning and human resource management Basically gained from high-value acquisitions beginning in 2003 JD Edwards, PeopleSoft, Siebel Systems, BEA Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 69 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 70 During the 1970ies IBM had also decided to develop a relational database system System R with the first implementation of a the SQL declarative query language (SEQUEL) At first mostly a research prototype, later became the base for IBM DB2 Today, the relational model is the de-facto standard of most modern databases Company Revenue Estimates 2006 (in mio USD) Market Share (in %) Oracle 7, IBM 3, Microsoft 2, Teradata Sybase Other Vendors 1, Total 15, Source: Gartner Research, 2007 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 71 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 72 12

13 Year Event 1880 Hollerith Census Machine 1951 Univac 1 Electrical Data Machine 1959 First CODASYL Conference 1960 Flight Reservation System SABRE 1966 IMS Hierarchical Database 1969 Networkmodel 1971 CODASYL Recommendation for DDL and 3-Layer-Architecture 1975 System R introduces Sequel query language 1976 System R introduces transaction concepts 1976 Peter Chen publishes the Entity Relationship Modeling approach 1980 Orcacle, Informix and others start selling DBMS with SQL support Year Event 1983 Work on ACID-Transactions published by Jim Gray 1986 SQL standardized as SQL 1 ANSI/SQL 1987 SQL internationally standardized as ISO SQL2 standard supporting referential integrity 1991 SQL2 supports domains and key definitions 1993 Object-oriented data model 1995 Preliminary SQL 3 supporting sub-tables, recursion, procedures, and triggers 1996 First object-oriented databases 1999 SQL3-Part1 finalized 2003 SQL2003 finalized with support for object-relational extensions Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 73 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 74 Beyond the relational model Logic-based data models Lead to deductive databases and expert systems Object-oriented data models Main Idea: Object-oriented design (garage metaphor) Very easy integration in OO programming languages Today mostly integrated into the relational model Semi-Structured data models Mainly incarnate as XML Allows a large degree of structural freedom For details see the master s courses on the topic Exercises Weekly exercises are to be done 50% of the maximum exercise score is necessary to be eligible for exams Exercises are usually quite easy we know that Exercises should be completed within teams of 2 students Exercises can be downloaded from our webpage Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 75 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 76 Exercises Turn in your exercise to the mailbox of the IfIS institute before the next lecture on paper Informatikzentrum, 2 nd floor Mark each sheet of paper with your names and Matrikelnummer If you have multiple pages, staple them together Exercises Exercises are graded and corrected by our HiWi s and returned to you in the lecture An example solution is provided at the end of each lecture For any questions regarding the correction, contact the respective HiWi directly Exercises will be marked by the HiWi Contact data will be on the webpage Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 77 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 78 13

14 SQL-Praktikum Parallel to the lecture, we offer a practical lab course SQL Praktikum Students in Bachelor Informatik (and Wirtschaftinformatik if they feel like it) are recommended to participate Awards 4 credits Others may also voluntarily participate, but it is up to their course of study to accept the credits or not SQL-Praktikum Lab course extends the written home works with additional computer based tasks Extended data modeling using model tools ER models UML models Querying and modifying databases Developing of easy up to complex SQL queries Modifying data with SQL Integrating data Using SQL within host languages Using JDBC in Java to interact with databases Using ER-mapping technologies Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 79 Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 80 SQL-Praktikum Lab course starts late: But: Registration is necessary! You can sign up at our webpage: Registration possible until Tasks have to be performed on a weekly basis and are graded 70% of the maximum score necessary to pass Pairs of 2 students each You may choose a preferred partner If you don t, you get a random partner Relational Database Systems 1 Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 81 14