Basics of Data Management

Transcription

1 Basics of Data Management Chaitan Baru

2 2 2 Objectives of this Module Introduce concepts and technologies for managing structured, semistructured, unstructured data Obtain a grounding in traditional data management

3 3 3 Outline The Data Lifecycle Data Genres Structured, Semi-structured, Unstructured Traditional Data Management A grounding in the basics

4 4 4 Data Roles The Data Owner The Data User The Application/Database programmer The Database Administrator The Systems Administrator (Storage Systems Administrator) The Data Security Officer (Data Quality)

5 5 5 The Data Lifecycle Acquisition: Obtaining the data Modeling : Representing the data Implies a Requirements Analysis step Storage: Storing the data Access and Analysis (includes data integration) Sharing data: Standards for sharing Preserving the data for the long term

6 6 6 Data Management Tasks Obtaining the data Primary data obtained by you Secondary: Data from existing sources, accessed by you and then transformed for your use Accessing existing data resources Modeling the data Data tends to stay around longer than one thinks, or plans for Some amount of formal modeling can be useful Storing data Depends very much on: Available skill sets Plans for how to use the data Storage system selected for the data has a huge impact on downstream tasks

7 7 7 Data Management Tasks 2 Accessing and Analyzing the Data Methods employed depend upon storage choice E.g. files vs DBMS vs HDFS vs Graph Data Stores, etc. Sharing the Data Need accepted norms for structure and semantics of the data Age: Time period passed since birth to now rounded to nearest year Represented by an integer Absorbed Nitrates, Age of Rock,

8 8 8 Data Management Tasks 3 Preserving the Data for the Long-term Users typically want to preserve ALL data! Until cost is factored in Decide on how long long term is Find a service or service-provider who can preserve the data at a cost that you can afford Could be campus libraries.in future Use immutable methods for storing the bits and referring to the bits

9 9 9 Data Genres: Classes of data Structured Spreadsheets, relational databases, graphs? Semi-structured Web logs, XML documents, Unstructured Text documents, Key-Value pairs,

10 10 10 Representing / storing data Structured E.g., Relational databases Created via a requirements analysis and design process. Using a Data Definition Language (DDL) Information about the structure schema is in metadata tables, aka Database Catalog Structured model includes/enables a formal query language, e.g. SQL

11 11 11 Representing / storing data 2 Semi-structured E.g., XML Data An extensible template for representing data, e.g. tree, graph With extensibility built in, e.g. Adding nodes, subtrees, subgraphs, edges Schema specification available for XML XML extensions to relational databases

12 12 12 Representing / storing data 3 Unstructured E.g. Text documents Created by applications in any language using file operations Information about the structure resides with the application

13 13 13 Traditional Data Management Tasks Design and Implement the Logical Schema (The data template ) Design and Implement the Physical Schema(files, tables, indexes) Load the Data Access Data Update Data Index, Save, Backup/Restore Data Update Schema

14 14 14 Traditional Data Management Application-driven design Requirements analysis to obtain the application needs Data Management Steps 1. Design: Conceptual or Logical Schema How the user thinks about the data The Data Model: Data Representation + Query Language 2. Implementation: Physical Schema How the computer internally implements the logical model (using memory, disk files, disk blocks, etc.) 3. Data Access Query processing Data Loading Data updating: Insert, Delete, Update 4. Utilities: Indexing, Statistics, Backup/Restore,

15 15 15 Structured Data: Spreadsheets Commonly used by science users Logical Model Set of rows with a set of columns. Columns can have data types Columns can be easily added/ removed Language GUI-based manipulation Macros in Visual Basic ODBC interface to the data in the spreadsheet SQL Can create linkages / dependencies among columns

16 16 16 Data example Staff members working on Projects A project may involve multiple Staff members A Staff member could work on multiple projects A Staff member works on a Project for a given percentage of time (effort) Excel

17 17 17 Sharing and Archiving Excel Sheets DataONE (2012) DataUp: dataup.org Create metadata Archive dataset GEON (2005) Contribute Resource: portal.geongrid.org Create metadata Convert to relational database Archive dataset

18 18 18 Spreadsheets Limitations Based on physical implementation E.g. 1M rows, 16K columns Number of worksheets depends upon amount of available memory Utilities Data Load: Smart Import facility Indexing, backup/restore, etc: Not available

19 19 19 Considerations in choosing appropriate data management technologies Available skill sets Available software / hardware Nature of the problem Amount of data Number of users Longevity of the project Always longer than expected Data professionals like to take the long view Scientists / users tend to take the short view

20 20 20 Structured Data: Relational Databases Logical Model Set of relations, each is a set of tuples (rows) Each tuple has a set of columns (fields) Physical Model Set of Tables (rows + columns) Data stored in tablespaces Indexes Extended field types, e.g. blobs, text, image,

21 21 21 The Staff-Projects Example Logical/Conceptual Schema E-R Diagram, UML Staff WorksOn Project

22 22 22 The Staff-Projects Data Example Staff SSN Last Name First Name Project Project Index Project Name Funding Agency Start Date End Date WorksOn Staff SSN Project Index %Time Start Date End Date

23 23 23 Relational Databases: Physical Schema Tables are defined in Tablespaces Tablespaces are a collection of Tablespace Containers Tablespace containers can be files or raw devices Different Tablespaces for Data, Indexes, Large Objects (BLOBs) Data are organized by rows Language SQL: Operates on tables Select, Project, Join Aggregate, Group-By, Sort

24 24 24 DBMS Architecture Applications DBMS Filesystem DBMS Operating System Devices

25 25 25 Making Relational Databases Efficient Schema design Can reduce number of tables to reduce number of joins Select Staff.FirstName, Staff.LastName, Project.ProjectName, Effort.Time From Staff, Project, Effort Where Staff.SSN=Effort.StaffSSN and Project.ProjIndex = Effort.ProjIndex and Effort.Time>0.5 Order By Effort.Time, Staff.LastName, Staff.FirstName Define views to simplify SQL queries CREATE VIEW Over50PerCent AS SELECT (Staff.FirstName, Staff.LastName, Project.ProjectName, Effort.Time.FROM WHERE ) Materialized views to make access efficient

26 26 26 Relational Databases: Loading Data Data Loading Bulk loading of data Incremental load (append) Load with Update option Load with no-log option Insert, Update, Delete Operations

27 27 27 Improving Performance Indexing Single column indexes Multicolumn indexes for efficient joins Multidimensional indexing, e.g. spatial (lat/long), spatiotemporal Query processing Data parallel processing: Multiple threads per operator (e.g. select, join) Dataflow processing: Between consecutive operators, e.g. from select to join Tune database parameters (heap sizes, process pools, temp space)

28 28 28 Data Placement For Intra-query Parallelism Staff Effort Tablespace A Tablespace B Distribution across disks in a storage system Parallel I/O: Read two tables at the same time Use the Tablespace abstraction to define disjoint set of tablespace containers Clustering together of similar records from different tables Use Join columns to define clustering Grab multiple rows with same join values across different tables in single (or few) Read operations

29 29 29 Data Placement Across Nodes P P P P... Staff Effort Staff Effort Staff Effort Staff Effort Data distribution schemes Hash partition Staff and Effort on SSN Range partition data E.g. all data belonging to a particular region Round-robin partition

30 30 30 Embedding application logic in the database query User-Defined Functions, UDFs Application Process Space: Where user applications execute UDF s Application Heap -- Memory Database Process Space: Where SQL queries execute

31 31 31 Relational databases The Winner in data management, since 1980 s Have agglomerated functionality over time Support for new data types E.g., strings, large binary objects (in GB s), images, spatial data, video, XML Have become high performance >300,000 transactions/minute! 100 s-1000 s of concurrent users Very fast query processing Robustness and availability Failover support, on-line backups, indexing, loading, But all within a strict consistency model A database transitions from one consistent state to the next An inconsistent database is unusable

32 32 32 Shared-nothing Parallel Databases Invented in the 1980 s Teradata, IBM DB2 Parallel Edition (now DB2 EEE) But Only interface to DBMS was SQL/ODBC No API s at storage distribution level ( HDFS) Nor at process execution level ( MapReduce) Hadoop and its ecosystem has exposed these internals but, wait, things might get better!

33 33 33