Lecture 1 Database Systems: An Introduction

Size: px

Start display at page:

Download "Lecture 1 Database Systems: An Introduction"

Nelson Quinn
5 years ago
Views:

1 Lecture 1 Database Systems: An Introduction 3 September, 2008 Shuigeng Zhou School of Computer Science Fudan University

Outline Database, Database Management Systems and Database Systems View of Data Database Languages Relational Databases Database Design Object-based and semi-structured databases Data Storage and

2 Outline Database, Database Management Systems and Database Systems View of Data Database Languages Relational Databases Database Design Object-based and semi-structured databases Data Storage and Querying Transaction Management Database Architecture Database Users and Administrators Overall Structure History of Database Systems Challenges & Opportunities September 16, 2009 Dept. of C.S. Fudan Univ. 2

3 What is Database? A very large, integrated collection of data The amount of data is very large The data is structured and interrelated The data is integrated Models real-world enterprise or organization Entities (e.g., students, courses) Relationships (e.g., Li is taking Database and Knowledge) More recently, also includes active components (e.g. business logic ) Databases touch all aspects of our lives September 16, 2009 Dept. of C.S. Fudan Univ. 3

4 Database Applications Banking: all transactions Airlines: reservations, schedules Universities: registration, grades Sales: customers, products, purchases Online retailers: order tracking, customized recommendations Manufacturing: production, inventory, orders, supply chain Human resources: employee records, salaries, tax deductions September 16, 2009 Dept. of C.S. Fudan Univ. 4

5 Database Management System (DBMS) A Database Management System (DBMS) is a software package designed to store and manage databases Functions of DBMS Manages very large amounts of data Supports efficient access to very large amounts of data Supports concurrent access to very large amounts of data Example: bank and its ATM machines Supports secure, atomic access to very large amounts of data Contrast two people editing the same UNIX file last to write wins September 16, 2009 Dept. of C.S. Fudan Univ. 5

6 People working with DBMS End users: query/update databases through application user interfaces (e.g., Amazon.com, DISCOVER, etc.) Database designers: design database schema to model aspects of the real world Database application developers: build applications that interface with databases Database administrators (a.k.a. DBA s): load, back up, and restore data, fine-tune databases for performance DBMS implementors: develop the DBMS or specialized data management software, implement new techniques for query processing and optimization inside DBMS September 16, 2009 Dept. of C.S. Fudan Univ. 6

7 Why Use a DBMS? Data independence and efficient access Reduced application development time Data integrity and security Uniform data administration Concurrent access, recovery from crashes September 16, 2009 Dept. of C.S. Fudan Univ. 7

8 Is Web Search Engine a DBMS? Fairly sophisticated search available crawler indexes pages; Keyword-based search for pages data is mostly unstructured and untyped; search only: can t modify the data can t get summaries, complex combinations of data few guarantees provided for freshness of data, consistency across data items, fault tolerance, Web sites (e.g. e-commerce) typically have a DBMS in the background to provide these functions. September 16, 2009 Dept. of C.S. Fudan Univ. 8

9 Is a File System a DBMS? Thought Experiment 1: You and your partner are editing the same file. You both save it at the same time. Whose changes survive? A) Yours B) Partner s C) Both D) Neither E)??? Thought Experiment 2: You re updating a file. The power goes out. Which of your changes survive? A) All B) None C) All Since last save D)??? September 16, 2009 Dept. of C.S. Fudan Univ. 9

10 Database Systems A Database System (DBS) contains the following components Hardware platform (PC/Workstation/Cluster/Mainframe etc.) DBMS A (number of) database(s) HI that is both convenient and efficient to use September 16, 2009 Dept. of C.S. Fudan Univ. 10

11 Why Study Databases? Datasets increasing in diversity and volume Digital libraries, interactive video, Human Genome project, EOS project Applications need for DBMS exploding Shift from computation to management DBMS encompasses most of CS OS, languages, theory, AI, multimedia, logic September 16, 2009 Dept. of C.S. Fudan Univ. 11

12 What s the Intellectual Content? Representing information data modeling Languages and systems for querying data complex queries with real semantics over massive data sets Concurrency control for data manipulation controlling concurrent access ensuring transactional semantics Reliable data storage maintain data semantics even if you pull the plug September 16, 2009 Dept. of C.S. Fudan Univ. 12

Data Models A collection of tools for describing Data Data relationships Data semantics Data constraints Relational model Entity-Relationship data model (mainly for database design) Object-based data

13 Data Models A collection of tools for describing Data Data relationships Data semantics Data constraints Relational model Entity-Relationship data model (mainly for database design) Object-based data models (Object-oriented and Object-relational) Semi-structured data model (XML) Other older models: Network model (e.g. IDS) Hierarchical model (e.g. IMS) September 16, 2009 Dept. of C.S. Fudan Univ. 13

14 Data Abstraction September 16, 2009 Dept. of C.S. Fudan Univ. 14

15 Levels of Abstraction Physical level: describes how a record (e.g., customer) is stored. Logical level: describes data stored in database, and the relationships among the data. type customer = record customer_id : string; customer_name : string; customer_street : string; customer_city : integer; end; View level: application programs hide details of data types. Views can also hide information (such as an employee s salary) for security purposes. September 16, 2009 Dept. of C.S. Fudan Univ. 15

16 View of Data An architecture for a database system September 16, 2009 Dept. of C.S. Fudan Univ. 16

17 Instances and Schemas Similar to types and variables in programming languages Schema the logical structure of the database Example: The database consists of information about a set of customers and accounts and the relationship between them) Analogous to type information of a variable in a program Physical schema: database design at the physical level Logical schema: database design at the logical level Instance the actual content of the database at a particular point in time Analogous to the value of a variable Physical Data Independence the ability to modify the physical schema without changing the logical schema Applications depend on the logical schema In general, the interfaces between the various levels and components should be well defined so that changes in some parts do not seriously influence others. September 16, 2009 Dept. of C.S. Fudan Univ. 17

18 Data Manipulation Language (DML) Language for accessing and manipulating the data organized by the appropriate data model DML also known as query language Two classes of languages Procedural user specifies what data is required and how to get those data Declarative (nonprocedural) user specifies what data is required without specifying how to get those data SQL is the most widely used query language September 16, 2009 Dept. of C.S. Fudan Univ. 18

19 Data Definition Language (DDL) Specification notation for defining the database schema Example: create table account ( account-number balance char(10), integer) DDL compiler generates a set of tables stored in a data dictionary Data dictionary contains metadata (i.e., data about data) Database schema Data storage and definition language Specifies the storage structure and access methods used Integrity constraints Domain constraints Referential integrity (references constraint in SQL) Assertions Authorization September 16, 2009 Dept. of C.S. Fudan Univ. 19

20 Relational Model Example of tabular data in the relational model Attributes Customer-id customername customerstreet customercity accountnumber Johnson Alma Palo Alto A Smith North Rye A Johnson Alma Palo Alto A Jones Main Harrison A Smith North Rye A-201 September 16, 2009 Dept. of C.S. Fudan Univ. 20

21 A Sample Relational Database September 16, 2009 Dept. of C.S. Fudan Univ. 21

SQL SQL: widely used non-procedural language Example: Find the name of the customer with customer-id 192-83-7465 select customer.customer_name from customer where customer.

22 SQL SQL: widely used non-procedural language Example: Find the name of the customer with customer-id select customer.customer_name from customer where customer.customer_id = Example: Find the balances of all accounts held by the customer with customer-id select account.balance from depositor, account where depositor.customer_id = and depositor.account_number = account.account_number Application programs generally access databases through one of Language extensions to allow embedded SQL Application program interface (e.g., ODBC/JDBC) which allow SQL queries to be sent to a database September 16, 2009 Dept. of C.S. Fudan Univ. 22

23 Database Design The process of designing the general structure of the database: Logical Design Deciding on the database schema. Database design requires that we find a good collection of relation schemas. Business decision What attributes should we record in the database? Computer Science decision What relation schemas should we have and how should the attributes be distributed among the various relation schemas? Physical Design Deciding on the physical layout of the database September 16, 2009 Dept. of C.S. Fudan Univ. 23

objects Described by a set of attributes Relationship: an association among several entities

24 The Entity-Relationship Model Models an enterprise as a collection of entities and relationships Entity: a thing or object in the enterprise that is distinguishable from other objects Described by a set of attributes Relationship: an association among several entities Represented diagrammatically by an entity-relationship diagram: September 16, 2009 Dept. of C.S. Fudan Univ. 24

Object-Relational Data Models Extend the relational data model by including object orientation and constructs to deal with added data types.

25 Object-Relational Data Models Extend the relational data model by including object orientation and constructs to deal with added data types. Allow attributes of tuples to have complex types, including non-atomic values such as nested relations. Preserve relational foundations, in particular the declarative access to data, while extending modeling power. Provide upward compatibility with existing relational languages. September 16, 2009 Dept. of C.S. Fudan Univ. 25

26 XML: Extensible Markup Language Defined by the WWW Consortium (W3C) Originally intended as a document markup language not a database language The ability to specify new tags, and to create nested tag structures made XML a great way to exchange data, not just documents XML has become the basis for all new generation data interchange formats. A wide variety of tools is available for parsing, browsing and querying XML documents/data September 16, 2009 Dept. of C.S. Fudan Univ. 26

27 Storage Management Storage manager is a program module that provides the interface between the low-level data stored in the database and the application programs and queries submitted to the system. The storage manager is responsible to the following tasks: Interaction with the file manager Efficient storing, retrieving and updating of data Issues: Storage access File organization Indexing and hashing September 16, 2009 Dept. of C.S. Fudan Univ. 27

28 1. Parsing and translation 2. Optimization 3. Evaluation Query Processing September 16, 2009 Dept. of C.S. Fudan Univ. 28

29 Query Processing (Cont.) Alternative ways of evaluating a given query Equivalent expressions Different algorithms for each operation Cost difference between a good and a bad way of evaluating a query can be enormous Need to estimate the cost of operations Depends critically on statistical information about relations which the database must maintain Need to estimate statistics for intermediate results to compute cost of complex expressions September 16, 2009 Dept. of C.S. Fudan Univ. 29

30 Transaction Management A transaction is a collection of operations that performs a single logical function in a database application Transaction-management component ensures that the database remains in a consistent (correct) state despite system failures (e.g., power failures and operating system crashes) and transaction failures. Concurrency-control manager controls the interaction among the concurrent transactions, to ensure the consistency of the database. September 16, 2009 Dept. of C.S. Fudan Univ. 30

31 Database Architecture The architecture of a database systems is greatly influenced by the underlying computer system on which the database is running: Centralized Client-server Parallel (multi-processor) Distributed September 16, 2009 Dept. of C.S. Fudan Univ. 31

32 Database Users Users are differentiated by the way they expect to interact with the system Application programmers interact with system through DML calls Sophisticated users form requests in a database query language Specialized users write specialized database applications that do not fit into the traditional data processing framework Naïve users invoke one of the permanent application programs that have been written previously Examples, people accessing database over the web, bank tellers, clerical staff September 16, 2009 Dept. of C.S. Fudan Univ. 32

33 Database Administrator Coordinates all the activities of the database system; the database administrator has a good understanding of the enterprise s information resources and needs. Database administrator's duties include: Schema definition Storage structure and access method definition Schema and physical organization modification Granting user authority to access the database Specifying integrity constraints Acting as liaison with users Monitoring performance and responding to changes in requirements September 16, 2009 Dept. of C.S. Fudan Univ. 33

34 Overall System Structure September 16, 2009 Dept. of C.S. Fudan Univ. 34

35 Application Architectures Two-tier architecture: E.g. client programs using ODBC/JDBC to communicate with a database Three-tier architecture: E.g. web-based applications, and applications built using middleware September 16, 2009 Dept. of C.S. Fudan Univ. 35

36 Major DBMS Today Oracle IBM DB2 (from System R, System R*, Starburst) Microsoft SQL Server NCR Teradata Sybase Informix (acquired by IBM) PostgreSQL (from UC Berkeley s Ingres, Postgres) Tandem NonStop (acquired by Compaq, now HP) MySQL September 16, 2009 Dept. of C.S. Fudan Univ. 36

37 History of Database Systems 1950s and early 1960s: Data processing using magnetic tapes for storage Tapes provide only sequential access Punched cards for input Late 1960s and 1970s: Hard disks allow direct access to data Network and hierarchical data models in widespread use Ted Codd defines the relational data model Won the ACM Turing Award for this work IBM Research begins System R prototype UC Berkeley begins Ingres prototype High-performance (for the era) transaction processing September 16, 2009 Dept. of C.S. Fudan Univ. 37

38 1980s: History (cont.) Research relational prototypes evolve into commercial systems SQL becomes industrial standard Parallel and distributed database systems Object-oriented database systems 1990s: Large decision support and data-mining applications Large multi-terabyte data warehouses Emergence of Web commerce 2000s: XML and XQuery standards Automated database administration Streaming, mobile data management P2P and sensor networks data management Trusted data management (data privacy) Uncertain/probabilistic data management Data-space September 16, 2009 Dept. of C.S. Fudan Univ. 38

39 Milestones in DBMS History (1) Factoring out data management functionalities from applications and standardizing these functionalities is an important first step CODASYL (Conference on Data Systems Language) standard (circa 1960 s) Charles Bachman got a Turing award for this in 1973 Developed the first database management system IDS (Integrated Data Store) September 16, 2009 Dept. of C.S. Fudan Univ. 39

40 Charles W. Bachman (Dec. 11, 1924 Now) He received the ACM Turing Award in 1973 for "his outstanding contributions to database technology" September 16, 2009 Dept. of C.S. Fudan Univ. 40

41 Milestones in DBMS History (2) The relational revolution (1970 ) A simple data model: data is stored in relations (tables) A declarative query language: SQL Provides physical data independence The single most important reason behind the success of DBMS today And a Turing Award for E. F. Codd in 1981 E. F. Codd, A Relational Model of Data for Large Shared Data Banks, CACM 13, No. 6, June September 16, 2009 Dept. of C.S. Fudan Univ. 41

42 Edgar F. Codd (August 23rd, April 18th, 2003) September 16, 2009 Dept. of C.S. Fudan Univ. 42

Milestones in DBMS History (3) Transaction processing in 1980s Jim Gray got a Turing Award for this work in 1998 Received the Turing Award in 1998 "for seminal contributions to database

43 Milestones in DBMS History (3) Transaction processing in 1980s Jim Gray got a Turing Award for this work in 1998 Received the Turing Award in 1998 "for seminal contributions to database and transaction processing research and technical leadership in system implementation." Jim Gray was lost at sea on 28 Jan., ( ) September 16, 2009 Dept. of C.S. Fudan Univ. 43

44 Challenges and Opportunities An overview on the Claremont Report

45 A Great Tradition Over the last twenty years, small groups of database researchers, practitioners and opinionated professionals have periodically gathered to assess the state of the field and propose directions for future research A final report will come out after each meeting, which aims to serve various functions: to foster debate within the database research community, to explain research directions to external organizations, and to help focus community efforts on timely challenges The latest meeting was held in late May, 2008, at the Claremont Resort in Berkeley, California, which is the seventh meeting of this sort in 20 years September 16, 2009 Dept. of C.S. Fudan Univ. 45

46 the Claremont Report The participants of Claremont meeting Rakesh Agrawal, Anastasia Ailamaki, Philip A. Bernstein, Eric A. Brewer, Michael J. Carey, Surajit Chaudhuri, AnHai Doan, Daniela Florescu, Michael J. Franklin, Hector Garcia Molina, Johannes Gehrke, Le Gruenwald, Laura M. Haas, Alon Y. Halevy, Joseph M. Hellerstein, Yannis E. Ioannidis, Hank F. Korth, Donald Kossmann, Samuel Madden, Roger Magoulas, Beng Chin Ooi, Tim O Reilly, Raghu Ramakrishnan, Sunita Sarawagi, Michael Stonebraker, Alexander S. Szalay, Gerhard Weikum September 16, 2009 Dept. of C.S. Fudan Univ. 46

Challenges Breadth of excitement about Big Data the number of communities working with large volumes of data has grown considerably, to include not only traditional enterprise applications and Web

47 Challenges Breadth of excitement about Big Data the number of communities working with large volumes of data has grown considerably, to include not only traditional enterprise applications and Web search, but also e-science efforts (in astronomy, biology, earth science, etc.), digital entertainment, natural language processing, social network analysis, and more Data analysis as a profit center Ubiquity of structured and unstructured data Expanded developer demands Programmer adoption of relational DBMSs and query languages has grown significantly in recent years Architectural shifts in computing At the macro scale, the rise of cloud computing services suggests fundamental changes in software architecture At a micro scale, computer architectures have shifted the focus of Moore s Law from increasing clock speed per chip to increasing the number of processor cores and threads per chip In storage technologies, major changes are underway in the memory hierarchy, due to the availability of more and larger on-chip caches, large inexpensive RAM, and flash memory Power consumption has become an increasingly important aspect of the price/performance metric of large systems September 16, 2009 Dept. of C.S. Fudan Univ. 47

48 Opportunities (1) Revisiting Database Engines designing systems for clusters of many-core processors exploiting remote RAM and Flash as persistent media, rather than relying solely on magnetic disk treating query optimization and physical data layout as a unified, adaptive, self-tuning task to be carried out continuously compressing and encrypting data at the storage layer, integrated with data layout and query optimization; designing systems that embrace non-relational data models, rather than shoehorning them into tables trading off consistency and availability for better performance and scaleout to thousands of machines designing power-aware DBMSs that limit energy costs without sacrificing scalability September 16, 2009 Dept. of C.S. Fudan Univ. 48

49 Opportunities (2) Declarative Programming for Emerging Platforms The Interplay of Structured and Unstructured Data A significant long-term goal for our community is to transition from managing traditional databases consisting of well-defined schemata for structured business data, to the much more challenging task of managing a rich collection of structured, semistructured and unstructured data, spread over many repositories in the enterprise and on the Web. This has sometimes been referred to as the challenge of managing dataspaces September 16, 2009 Dept. of C.S. Fudan Univ. 49

50 Cloud Data Services Opportunities (3) Mobile Applications and Virtual Worlds In the mobile space, 1) the platforms on which to build mobile applications (i.e., the hardware, software and network) are maturing to the point that they have attracted large user bases, and can ubiquitously support very powerful interactions on the go. 2) emergence of mobile search and social networks suggests an exciting new set of mobile applications Virtual worlds like Second Life are growing quickly in popularity September 16, 2009 Dept. of C.S. Fudan Univ. 50

Conclusion DBMS used to maintain, query large datasets Benefits include recovery from system crashes, concurrent access, quick application development, data integrity and security Levels of

51 Conclusion DBMS used to maintain, query large datasets Benefits include recovery from system crashes, concurrent access, quick application development, data integrity and security Levels of abstraction give data independence A DBMS typically has a layered architecture DBAs hold responsible jobs and are well-paid! DBMS R&D is still one of the broadest, most exciting areas in CS September 16, 2009 Dept. of C.S. Fudan Univ. 51

52 Some useful terms Byte Kilobyte 2 10 bytes (i.e bytes) Megabyte 2 20 bytes (i.e kilobytes) Gigabyte 2 30 bytes (i.e megabytes) Terabyte 2 40 bytes (i.e gigabytes) A handful of these for files in EECS Biggest single online DB is Wal-Mart, >100TB Petabyte 2 50 bytes (i.e terabytes) Exabyte 2 60 bytes (i.e petabytes) Zettabyte 2 70 bytes (i.e exabytes) Yottabyte 2 80 bytes (i.e zettabytes) September 16, 2009 Dept. of C.S. Fudan Univ. 52

53 End of Lecture 1

Chapter 1: Introduction

Chapter 1: Introduction Chapter 1: Introduction Purpose of Database Systems Database Languages Relational Databases Database Design Data Models Database Internals Database Users and Administrators Overall