CS 2451 Database Systems

Transcription

1 Introduction to course What is this course about? CS 2451 Database Systems Let me know if you figure it out Spring 2018 Instructor: Dr. Bhagi Narahari Lead TA: Roxana Leontie TA/Grader: Malik Undergrad TA: Victoria Zheng Based on slides Ramakrishnan&Gerhke, R. Lawrence CS 2441: What is it about? 1. Introduction to database design and implementation 2. Relational database design Application development using Relational DBMS (MySQL), with web front end Theory of relational database design 3. Introduction to database models for unstructured data (Big data) Overview of NoSQL database models 4. Working in teams and team software development 5. Improving technical communication skills: Writing in the disciplines (WID) Presentation and discussion skills Why this Course is Important Database systems is a required course because most systems require a database as part of their implementation. As a developer you will need to: Query/update databases SQL and relational algebra are the languages to extract information from pre-existing databases. Program with databases All languages require interfaces for retrieving/updating data from databases into program code for displaying reports or processing transactions. Design databases Building a new application often involves storing data persistently in a database. The ability to design a database is a leading-edge skill. Be aware of data technologies XML, NoSQL, JSON, web services, and a variety of other technologies are used for web and mobile applications. 1

2 What Is a Database? A very large, integrated collection of data. Not arbitrary, unrelated data.but logically related data in some domain - Definition changes slightly in the context of unstructured data (big data) Models real-world enterprise. Entities (e.g., students, courses) Relationships (e.g., Victoria Zheng is registered in cs2541) What is a database management system (DBMS)? A database is a collection of logically related data for a particular domain. A database management system (DBMS) is software designed for the creation and management of databases. e.g. Oracle, DB2, Access, MySQL, SQL Server, MongoDB Bottom line: A database is the data stored and a database system is the software that manages the data. Why Databases? Most CS courses concentrate on code our interest is managing, manipulating and representing data Warning: this course doesn t focus on teaching SQL or how to be an Oracle DBA (though it will get you started) Why use Databases?? Information gathering is first step to analysis Huge amount of data can be collected easily To effectively analyze data: collect relevant data storein manner amenable to efficient access provide programming interface Data analysis methods are current emphasis in the market Introduces new problems in privacy? 2

3 Data Analysis: Examples Data Warehousing Web search engines Data mining (personalization) Analytics on social networks Twitter sentiment analysis Scientific data analysis many more Why use a DBMS? Why do we need a DBMS, instead of coding in Java? Databases in the Real-World Databases are everywhere in the real-world even though you do not often interact with them directly. $25 billion dollar annual industry Examples: Retailers manage their products and sales using a database. - Wal-Mart has one of the largest databases in the world! Online web sites such as Amazon, ebay, and Expedia track orders, shipments, and customers using databases. The university maintains all your registration information and marks in a database. Can you think of other examples? What data do you have? Example Problem Implement a system for managing products for a retailer. Data: Information on products (SKU, name, desc, inventory) Add new products, manage inventory of products How would you do this without a database? What types of challenges would you face? 3

4 Why do we need databases? File Processing Diagram Without a DBMS, your application must rely on files to store its data persistently. A file-based system is a set of applications that use files to store their data. PROGRAM 1 Each application in a file-based system contains its own code for accessing and manipulating files. This causes several problems: Code duplication of file access routines Data is usually highly redundant across files High maintenance costs Hard to support multi-user access to information Difficult to connect information present in different files Difficulty in developing new applications/handling data changes Data Management PROGRAM 2 Data Management PROGRAM 3 Data Management FILE 1 FILE 2 Redundant Data Data Independence and Abstraction Database System Approach The major problem with developing applications based on files is that the application is dependent on the file structure. That is, there is no program-data independence separating the application from the data it is manipulating. If the data file changes, the code that accesses the file must be changed in the application. PROGRAM 1 DBMS Query Processor Transaction Mgr... Integrated Database One of the major advantages of databases is they provide data abstraction. Data abstraction allows the internal definition of an object to change without affecting programs that use the object through an external definition. PROGRAM 2 Storage Mgr Collection of Data Management Functions 4

5 DBMS A database management system provides efficient, convenient, and safe multi-user storage and access to massive amounts of persistent data. Efficient - Able to handle large data sets and complex queries without searching all files and data items. Convenient - Easy to write queries to retrieve data. Safe - Protects data from system failures and hackers. Massive - Database sizes in gigabytes/terabytes/petabytes. Persistent - Data exists after program execution completes. Multi-user - More than one user can access and update data at the same time while preserving consistency.concept of transactions Components of a DBMS A DBMS is a complicated software system containing many components: Query processor - translates user/application queries into low-level data manipulation actions. - Sub-components: query parser, query optimizer Storage manager - maintains storage information including memory allocation, buffer management, and file storage. - Sub-components: buffer manager, file manager Transaction manager - performs scheduling of operations and implements concurrency control algorithms. DBMS Architecture This course is about Database Design Users End-User Programs Direct (SQL) Users Database Administrators Focus is on design of databases Working at the logical level DBMS Parser + Compiler Query Planner Database API Optimizer Result Formatting Execution Engine Query Processor Internals of DBMS is not the focus in this course We will touch upon a few key concepts that make DBMS work Recovery System Transaction Manager Buffer Manager File Manager Storage Manager DB Files Operating System 5

6 Database Architectures Two-Tier Client-Server Architecture There are several different database architectures: File-server (embedded) architecture - files are shared but DBMS processing occurs at the clients (e.g. Microsoft Access or SQLite) Two-Tier client-server architecture - dedicated machine running DBMS accessed by clients (e.g. SQL Server) Clients Server Database Tier 1: Client User Interface Business and Data Logic Tier 2: Database Server Server-side validation Data storage/management Three-Tier client-server architecture - DBMS is bottom tier, second tier is an application server containing business logic, top tier is clients (e.g. Web browser-apache/tomcat-oracle) Advantages: Only one copy of DBMS software on dedicated machine. Increased performance. Reduced hardware and communication costs. Easier to maintain consistency and manage concurrency. Three-Tier Client-Server Architecture Advantages: Database Tier 1: Client (Web/mobile) User Interface Tier 2: Application Server Business logic Data processing logic Tier 3: Database Server Data validation Data storage/management Reduced client administration and cost using thin web clients. Easy to scale architecture and perform load balancing. Database People There are several different types of database personnel: Database administrator (DBA) - responsible for installing, maintaining, and configuring the DBMS software. Data administrator (DA) - responsible for organizational policies on data creation, security, and planning. Database designer - defines and implements a schema for a database and associated applications. - Logical/Conceptual database designer - interacts with users to determine data requirements, constraints, and business rules. - Physical database designer - implements the logical design for a data model on a DBMS. Defines indexes, security, and constraints. DBMS developer - writes the DBMS software code. Application developer - writes code that uses the DBMS. User - uses the database directly or through applications. 6

7 The Database Abstraction Provided by the DBMS We think of databases at two levels: Logical structure: - What users/programmers see program or query interface Physical structure: - Organization on disk, indices, etc. The logical level is further split into: Overall database design (conceptual; seen by the DB designer) Views that various users get to see Levels of Abstraction Many views, single conceptual (logical) schema and physical schema. View 1 View 2 View 3 Conceptual Schema Physical Schema Views describe how users see the data. Conceptual schema defines logical structure Physical schema describes the files and indexes used. Schemas are defined using Data Definition Language (DDL); data is modified/queried using Data Manipulation Lang(DML). COSC Dr. Ramon Lawrence ANSI/SPARC Architecture Data Independence Users External Schema External view External view External view A user of a relational database system should be able to use the database without knowing about how the precisely how data is stored, e.g. Conceptual Schema Conceptual view 3-schema architecture SELECT When, Where FROM Calendar WHERE Who = Jane" Internal Schema Internal view DBMS After all, you don't worry about IEEE floating-point when you do division in a Java program or with a calculator DB Page 27 7

8 More on Data Independence Logical data independence Protects the user from changes in the logical structure of the data: could reorganize the calendar schema without changing how we query it Physical data independence Protects the user from changes in the physical structure of data: could change how calendar is stored in memory without changing how the user would write the query How to Organizing/Model the data Person has attributes SSN GW ID Name.. Student IS a person who: Takes courses at GW Is given grades registers... This is yet another kind of information.. have you seen this in a different context? How to organize the data/information? What is the data needed? Eg: What do we need to store to uniquely identify a student entity? How to store & organize the data? How many attributes are really needed What is an efficient way to organize the data? - This is why we need to study schema design and Normal forms How to query the data and generate reports for the end users? Need a database query language, such as SQL Data Models and data representation All of the data have an implicit data model Basic assumption on what is an item of data, how to interpret it, etc. A data model is a collection of concepts for describing data. Starting point to design of DBMS Provides us with the mathematical basis to prove/assert properties and show correctness of algorithms The relational model was the first model of data that is independent of its data structures and implementation A theory of normalization guides you in designing relations Other data models: network, hierarchical, Object Oriented With explosion in unstructured data and big data, new models emerged NoSQL database models Relational model is observed to be inefficient for many such applications 8

9 How to define and use the database:data Definition and Manipulation Languages data definition language (DDL) to specify database schema What data, and how it is organized (logical level) Data manipulation language (DML) allows users to access or manipulate data as organized by data model procedural DMLs: require user to specify what data and how to get it non-procedural DMLs: require user to specify what data is needed without specifying how to get it. Commercial languages SQL Relational DB Query Languages Formal query languages: Relational algebra, Relational Calculus, Domain calculus Why study formal languages? Commercial query languages: SQL, QUEL SQL: descendent of SEQUEL; mostly relational algebra and some aspects of relational calculus has procedural and non-procedural aspects Course Administrivia Course Objectives Relational database theory and design Concepts of data storage and retrieval Fluency in SQL and database application dev. Working with relational database systems Software integration experience and team experience Design and deploy a large database application Introduce NoSQL database models Technical writing skills and oral communication skills Very Important: You must come prepared to class (read BEFORE class) and you will work on problems during class 9

10 Course Outline: Topics Part 1: Design of Relational Databases: Weeks 1-10 Entity-Relationship Model (similar to UML) Formal Query Languages: Rel. algebra Query languages: SQL Relational Schema Design and Normal Forms, Tuning Overview of DBMS architecture - File manager, transaction processing, query processing Team based term Project Part 2: Intro. to Databases for Semi/Un-structured data: Weeks Social Impact of Computing (class discussions, guest speakers) NoSQL database models Experience working with one NoSQL DB Writing requirements: discussion of papers, term project report Course : Instruction team Lab Instructor/Lead TA: Roxana Leontie Grad TA/Grader: Mahak Malik Undergraduate Teaching Assistant: Victoria Zheng We may have an additional undergraduate teaching fellow TBD Writing WID TA: Madhu Devaraj All grading inquiries on database homeworks directed to Mahak Malik (and then follow up with instructor) All grading inquiries on writing assignments to be directed to Writing TA (and then follow up with instructor) All inquiries on in-lab demos and projects directed to lead TA Roxana and instructor. In-class work You will learn through in-class exercises most Thursdays, and some Tuesdays and labs Must read the material and come to class You need to complete the exercises while working as a group/team Each team is assigned to a table I may ask a team to present solutions to class If you do not come prepared to class (by reading the materials before class), it is not going to work out too well. Course Materials course webpage Course webpage will have links to syllabus, lecture notes, and inclass exercises (or follow link from my homepage) check course announcements page linked from above page Blackboard will be used for: Homeworks and solutions Projects Electronic submission of homeworks Reporting grades Piazza for discussions 10

11 Piazza Online discussion forum The purpose of this: - to encourage students to ask well formed questions - To encourage students to answer each others question Most of the time, you do better than we do! - Be very careful not to border on plagiarism! - Don t post your HW solution to the world, Signup will be sent check your piazza account Do not expect instant response or substitute Piazza for TA office hours! - Piazza is not manned 24 hours/7 days a week - sometimes answer may take couple days! - Enable students to help one another with common questions/clarifications Not a substitute for office hours NO TA can excuse you from anything/or given any extensions Posting on piazza, not the same as telling instructor things - E.g. : I m going to miss the exam! (cannot do this) Textbooks/Software Textbook: Online notes and resources - Suggested readings/resources linked from course webpage (go to Lectures) Reference (if you want to purchase a textbook): two suggested books on the course webpage - You can use any other Database textbook for Relational DB topic MySQL relational DBMS and PHP You can install it locally on your laptop We will use the install on SEAS shell server At least one NoSQL database Explore setting up your own DB services on AWS Course Requirements: Grading Two Exams: 33% In class, closed book. Based on lecture AND lab materials. Approximately Weeks 6,12 Exact percentage of exam weight determined after 4 weeks of class Homeworks (& Lab assignments): 33% Homeworks, Programming (and lab) assignments: 23% Writing assignments: 10% Project and In-class exercises (some team based): 34% In-class team discussions/questions and In-Lab exercises: 10% - Most in-class will be team based; labs may be individual Team based Term Project: 24% -- Team - Demos required: to pass you must have a working project No Final exam BUT there will be a homework mini-project due after end of classes + Project Demo..replaces Final exam. Grades curved (and scaled as percentage of highest score in class) Approximate grading method after curving and scaling A to A-: % B- to B+ : >80 to < 90 C- to C+: > 70 to < 80 D: >60 The Database Project A significant part of your grade for the course is devoted to a major database development project. In the project you will: Design a database using ER/UML diagrams. Write SQL to query and update the database. Develop a web user interface to your database. - In the process you will learn HTML and PHP. These skills are highly valuable to potential employers. Note that limited background will be given on web programming. The project can involve teams of 2 to 4. Larger teams must develop better projects. 11

12 Team Project: Requirements/Expectations Lab Sections A large project requirement with a set of applications will be posted Objective: Build a complete end-to-end working project Team based project with multiple phases Phase 0: each student assigned to an application, and required to provide a design of the database schema. Phase 1: One set of teams build application A, Second set of teams build application B Phase 2: Create new teams, with members from group A and B. Integrate the applications to provide a complete solution. Integration is not = redesign! Integration reflects real world team SW development practice A clear set of minimum requirements will be specified note that meeting minimum requirements only implies a C grade on the project. Clear deadlines for specific phases and steps in the project will be posted no extensions! Lab sections conducted by the TAs: Roxana Leontie (Lead), Victoria, (and Mahak) Lab sections will cover Intro MySQL: SQL, ( JDBC?) Short tutorials including application development using PhP, JDBC (XML?, CSS? HTML?) Intro to a NoSQL DB Clarifications on Programming Assignments Help with analysis of Project (but not in the design of the project) In-class exercises in some weeks: have to implement the queries during the lab; no extensions! Lab Assignments will be posted separate from homeworks you may have BOTH due concurrently Academic Integrity Policy No collaboration (of any sort) on homeworks, programming assignments, and WID homeworks Including external resources Okay to clarify questions Not Okay to share solutions Not okay under any circumstances to share or show Code No collaboration between teams on team projects within team each team member must have clear role -- i.e., clearly partitioned tasks for each team member Academic Integrity You are here to learn so keep that in mind Strictly enforced! no collaboration means none of any kind No asking friends No searching on web for answers Violations will lead to at least a zero on the work and a grade lower than final grade Stay on top of your work and come ask me questions! For team projects (and team based lab assignments), you can work within your team but not across teams PDT: Plagiarism detection software tool I will be running code submissions through software tool 12

13 Expectations Come prepared to class High expectations of preparation for class In-class expectations don t want to sit in class or want to check your phone then better leave the room than disturb others You will need to spend at least 4-6 hours outside class time each week If you do not, then you will encounter rough seas Depending on how much outside the classroom CS you have done, you may need to learn new generic CS skills on your own (Example: HTML) this will add to the total hours per week There will be some open ended aspects in some projects this is an opportunity for you to learn more on your own, and to go above and beyond the minimum project expectations. Generic advice: You will be expected to learn some materials on your own This is only the beginning..things get more demanding when you get to your junior year. Next.. Read Notes for Relational model Attend Lab sessions and get started on learning software skills you will need for your work 13