1/5/11. Announcements. Topics Lab hour/discussion sec2on: Introduc4on (SDM, e Science, scien2fic workflows) Databases. Assignment 1

Transcription

1 1/5/11 Announcements Topics Lab hour/discussion sec2on: Introduc4on (SDM, e Science, scien2fic workflows) Databases next Thursday 2 3pm, 93 Hutchison Assignment 1 Basic concepts of rela2onal databases (crash course) Metadata, ontologies Hands on exercises with SQLite, MySQL, Python GeBng started with Python & SQLite Out: end of this week, due: end of next week (tbd) Readings: Scien4fic Workflows Two papers posted on web site please read! Send ques2ons to mailing list! Visit site, subscribe to mailing list (=group) sites.google.com/site/ecs166wq11 Basic concepts Hands on exercises with Kepler, Taverna, Python, Data Provenance (theory & examples) Parallel Execu2on (Map Reduce) New reading materials there, addi2onal online book, etc groups.google.com/group/ecs166wq Workflow Inputs Taverna workflow The 4th Paradigm qtl_end_position qtl_start_position chromosome_name genes_in_qtl mmusculus_gene_ensembl Chapter 3, page 137 in the 4th Paradigm book. remove_uniprot_duplicates remove_entrez_duplicates create_report merge_uniprot_ids merge_entrez_genes merge_reports REMOVE_NULLS_2 remove_nulls add_uniprot_to_string add_ncbi_to_string Kegg_gene_ids Kegg_gene_ids_2 concat_kegg_genes regex_2 workflow connects interna2onally distributed datasets to iden2fy candidate genes that could be implicated in resistance to African trypanosomiasis (sleeping sickness) binfo merge_kegg_references remove_duplicate_kegg_genes Get_pathways Workflow Inputs regex gene_ids split_by_regex lister get_pathways_by_genes1 Merge_pathways concat_ids concat_gene_pathway_ids Merge_gene_pathways pathway_desc Merge_pathway_desc Workflow Outputs pathway_genes merge_genes_and_pathways kegg_pathway_release split_gene_ids split_for_duplicates pathway_desc remove_pathway_duplicates pathway_ids merge_pathway_list_1 gene_descriptions merge_genes_and_pathways_2 merge_pathway_desc merge_pathway_list_2 merge_gene_desc merge_genes_and_pathways_3 remove_pathway_nulls remove_pathway_nulls_2 remove_nulls_3 species getcurrentdatabase Workflow Outputs kegg_pathway_release 3 merged_pathways Workflow Outputs Workflow Inputs An_output_port An_input_port pathway_descriptions A_local_service pathway_ids Beanshell gene_descriptions A_Soaplab_service kegg_external_gene_reference String_constant report ensembl_database_release A_Biomart_Service 4 FIGURE 1. A Taverna workflow that connects several internationally distributed datasets to identify candidate genes that could be implicated in resistance to African trypanosomiasis [11]. THE FOURTH PARADIGM Microbial Ecology, Metagenomics: what microbes are in my favorite environment? Assembled con2gs Workflow for Alignment, Taxonomy, Ecology of Ribosomal Sequences (Amber Hartman; Eisen Lab; UC Davis) WATERS: Profile alignment Chimera check Find OTUs (STAP or Infernal) (Mallard) (OTUHunter) +/ cluster +/ cluster View tree: Dendroscope Build phylogene2c tree (RaxML or Quicktree) +/ cipres UniFrac: tree & STAP (ss rrna Taxonomy Assigning Pipeline) environment file D. Wu, A.L. Hartman, N. Ward, J.A. Eisen, PLoS ONE, June 2008 Li Weng et al. Genome Res Metadata 139 Assign Taxonomy (STAP) +/ cluster Visualiza2on tools: Cytoscape networks & Heat map Diversity sta2s2cs: Metadata Metadata Text: OUT list, Chao1, Shannon Graphs: rarefac2on curves, rank abundance curves 6 1

2 1/5/11 Executable WATERS Workflow in Kepler myexperiment.org myexperiment allows users to find, use and share scien2fic workflows and other Research Objects, and to build communi2es. 7 Simple Kepler analysis workflow using 8 Scientific Workflow for Phylogenetic Analysis Dan Higgins, NCEAS Newick Tree Aligned AA-Sequences AA-Sequences Clustal DrawTree Quicktree SciWF ~ executable spec of a scien4fic data analysis method Actors Tokens int, string, record{..}, array[..],.. Data source from EcoGrid (metadata-driven ingestion) R processing script res <- lm(baro ~ T_AIR) res plot(t_air, BARO) abline(res) Channels 9 From Climate Gate to Reproducible Science Ports 10 Provenance and Scientific Workflows prov e nance noun place of origin; derivation For the scien4st (focus on data deriva2on) Evaluate results based on actors and data used, parameter sebngs, etc. Automate metadata crea2on Maintain a record of what was done within a project, etc. Provide a high level view of what a workflow did, dependencies, etc. For the engineer (focus on processing history) Monitor, benchmark, and op2mize workflow performance Record resources used during workflow execu2on Checkpoint and restart workflows Op2miza2on (e.g., minimize unnecessary recomputa2ons) 11 2

3 1/5/1 Provenance questions a scientists might ask Which DNA sequences were input to the workflow? Which phylogene2c trees were created? Which actor created this phylogene2c tree? Which input sequences did this tree depend on? What input sequences were not used to derive any output consensus trees? What sequence alignment was used to infer this tree? Which actors were involved in crea2ng this tree? How can we answer these questions? By recording what happens during the workflow run We open call the result a workflow execu2on trace What is recorded depends on what can be observed during the run What can be observed depends on the model of computa:on (MoC) Some2mes the MoC isn t enough and the observables must be augmented to capture provenance Provenance (Data Lineage) Graphs Workflow (Kepler/COMAD) Scientific Workflows & Data Mining: Kepler/WEKA Provenance Scien4fic workflows: to specify and execute computa2onal pipelines Provenance informa2ons capture data lineage and processing history 16 Topics Introduc4on (SDM, e Science, scien2fic workflows) Databases Basic concepts of rela2onal databases (crash course) Metadata, ontologies Hands on exercises with SQLite, MySQL, Python Scien4fic Workflows Basic concepts Hands on exercises with Kepler, Taverna, Python, Data Provenance (theory & examples) Parallel Execu2on (Map Reduce) Introduction to Data(base) Management Why study data(base) management? Critical to business, government, science, culture, society, Determines success of many corporations (even their existence) Many tech companies built on data management (Google, Amazon, Yahoo!, Facebook, ) or offer database products (Microsoft, IBM, Oracle) Database systems span major areas of computer science Operating systems (file, memory, process management) Theory (languages, algorithms, complexity) Artificial Intelligence (knowledge-based systems, logic, search) Software Engineering (application development) Data structures (trees, hash-tables) and the DB research community continues to be very active 17 18

4 1/5/1 Databases are everywhere ( Every-Ware ) Regularly Structured Data Sets the structure once (e.g., table attributes) and then has many instances (records) that use that structure Examples of regularly structured data Employee, payroll, bank account Data captured on web forms Examples of unstructured a.k.a. loosely or semi-structured data Documents, (heaps of) video, audio, images, maps, We Focus on Regularly Structured Data We focus on relational database management systems (abbreviated: DBMS or RDBMS) Mainly designed to store, manage, and retrieve structured data We use SQL to manage and retrieve (query) data from databases (abbreviated: DB) Unstructured data (e.g., documents) is managed mainly by content management and information retrieval systems Includes search engines on the web Querying involves indexing words in text, ranking results, etc. Includes Web 2.0 features like tagging/labeling * Many DBMSs now support unstructured and semi-structured data too Some Characteristics of Data in Databases Data is persistent One or more applications use the same data Data stored between applications Data often too large to easily manage in-memory DBMSs handle this for free Manually handling data (files) is usually ad hoc (each app. does it differently) and can be inefficient Data may be very large (business, government, science, ) Library of congress > 20 terabytes of print Amazon.com: > 42 terabytes of data Youtube: > 45 terabytes of video AT&T: > 323 terabytes of call records National Energy Research Scientific Computing Center: > 2.8 petabytes * 1 terabyte 1,000,000,000,000 bytes * 1 petabyte 1,000,000,000,000,000 bytes (and there is talk about exabytes at DOE) 2 2 Lots of Data Everywhere From : History: According to Kevin Kelly in The New York Times, "the entire [written] works of humankind, from the beginning of recorded history, in all languages" would amount to 50 petabytes of data. [1] Computer hardware: Teradata Database 12 has a capacity of 50 petabytes of compressed data. [2][3] Telecoms: AT&T has about 16 petabytes of data transferred through their networks each day. [4] Archives: The Internet Archive contains about 3 petabytes of data, and is growing at the rate of about 100 terabytes per month as of March, [5][6] Internet: Google processes about 20 petabytes of data per day. [7] Physics: The 4 experiments in the Large Hadron Collider will produce about 15 petabytes of data per year, which will be distributed over the LHC Computing Grid. [8] P2P networks: As of October 2009, Isohunt has about 9.76 petabytes of files contained in torrents indexed globally. [9] Games: World of Warcraft utilizes 1.3 petabytes of storage to maintain its game. [10] What is a DB? A database (DB) is a (structured) collection of persistent data NB (the picky guy): DB schema vs. DB instance A database management system (DBMS) is a software system that supports the definition, population, and query of a database DBMS DB 2 2

5 1/5/1 Basic Database Architecture Query Processing Web Forms Application Front Ends SQL Interface Web Forms Application Front Ends SQL Interface SQL Commands SQL Commands Query Execution DBMS Plan Executor Operator Evaluator Parser Optimizer Query Evaluation Engine DBMS Plan Executor Operator Evaluator Parser Optimizer Query Evaluation Engine Transaction Lock Concurrency Control File and Access Methods Buffer Disk Space Recovery Transaction Lock Concurrency Control File and Access Methods Buffer Disk Space Recovery Index Files Data Files System Catalog Index Files Data Files System Catalog Computer Science in a Nutshell All computer science students must learn to integrate theory and practice, to recognize the importance of abstraction, and to appreciate the value of good engineering design. Final report of the Joint ACM/IEEE-CS Task Force on Computing Curricula 2005 for Computer Science Computer Science in a Nutshell this course Strong Emphasis Practice Practical concepts Skills Tools Formalization may not exactly match practical concept (often the core, e.g., SQL vs. Relational Algebra) Important Formalizations Theory Formal definitions Mathematical results This is one of the really fun things about studying database systems!!! Only a Bit Engineering Performance tradeoffs Scalability Reliability Focus of the DB research community 27 Introduction to Relational Databases The name of the table (relation) Assume this table has been defined to keep track of bank account Also referred to as a relation

6 1/5/1 The name of the table (relation) The name of the attributes (columns) The schema of the table The schema sets the structure of the table The schema is the defini2on of the table Which generally includes more that what is shown here E.g., data types and constraints 3 3 Rows Each entry in the table is called a row, tuple, or record (often used interchangeably) The instance of the schema is the current set of rows Instance The inten:on of the table W. Yu R. Jones The current extension (or extent) of the table Not used as often in relational databases mainly in deductive (logic-based) and object-oriented databases 3 3 Degree or Arity of a table is the number of auributes Arity of this rela2on is (because there are auributes) W. Yu R. Jones Cardinality of this instance is 6 (because there are 6 rows) Cardinality of a table is the number of rows in the current instance Transac2on id Date 10/22/09 Check number Date 92 10/23/ /24/

7 1/5/1 Transac2on id Date 10/22/ Each table (typically) has a key The values of the key must be unique Transac2on id Date 10/22/ A key consists of one or more attributes We often underline key attributes Check number Date 92 10/23/ /24/ What is the key for the table? Check number Date 92 10/23/ /24/ Is this legal? Transac2on id Date 10/22/ /5/ If not, how do we prevent it from happening? Transac2on id Date 10/22/ /5/ We say that Deposit.Account is a foreign key that references Account.Number i.e., each Deposit (row) must refer to an Account (row) If the DBMS enforces this constraint, we have referential integrity Check number Date /23/ /24/ Are there any foreign keys in the Check table? Yes, Check.Account is a foreign key that references Account.Number Transac2on id Date 10/22/09 Check number Date 92 10/23/ /24/ Foreign keys may or may not be part of the key for the table Deposit.is not part of the key for Check.is part of the key for 4 4 7

8 1/5/1 Consider the following sample data from a table Consider the following sample data from a table Salesperson Company Jones Smith Age Commission $50,000 $60,000 Salesperson Company Jones Smith Age Commission $50,000 $60,000 Can you tell what the key for this table is? Now can you tell what the key for the table is? 4 4 One possibility: Another possibility: Person Table with Id as the key Sales Commission Table, by client company, per day Id Name Jones Smith Age Salary $50,000 $60,000 Salesperson Company Jones Smith Day Commission $50,000 $60,000 Keys, Table Names, A@ribute Names (help to) tell us what the table is ( the meaning, or seman:cs, of the rela:on => a form of metadata) For every attribute of every table, the schema specifies allowable values. For example, Number must be an integer Owner must be a 30-character string Type must be checking or savings The set of allowed values for an attribute is called the domain of the attribute Select the tables, with a name for each table A database schema may have multiple tables Each table has its own schema Select attributes for each table and give the domain for each attribute This is the basis of a relation (or table) schema also: Specify the key(s) for each table There can be more than one key for a table There is only one primary key (more on this later) Specify all appropriate foreign keys

9 1/5/1 Another example database More standard nota2on; Each table has one primary key Teacher(Number, Name, Office, E mail) Course(Number, Name, Descrip2on) Class Offering(Quarter, Course, Sec2on, Teacher, TimeDay) Student(Number, Name, Major, Advisor) Completed(Student, Course, Quarter, Sec2on, Grade) 49 Another example database with some foreign keys shown informally Teacher(Number, Name, Office, E mail) Course(Number, Name, Descrip2on) Class Offering(Quarter, Course, Sec2on, Teacher, TimeDay) Student(Number, Name, Major, Advisor) Completed(Student, Course, Quarter, Sec2on, Grade) What foreign keys are missing? 50 Another example database with some foreign keys shown informally Teacher(Number, Name, Office, E mail) Course(Number, Name, Descrip2on) Class Offering(Quarter, Course, Sec2on, Teacher, TimeDay) Student(Number, Name, Major, Advisor) Completed(Student, Course, Quarter, Sec2on, Grade) 5 Another example database with some foreign keys shown informally Teacher(Number, Name, Office, E mail) Course(Number, Name, Descrip2on) Class Offering(Quarter, Course, Sec2on, Teacher, TimeDay) Student(Number, Name, Major, Advisor) Completed(Student, Course, Quarter, Sec2on, Grade) What are the limita:ons of this schema? 5 9