BINF 6211 SQL. Lecture th, 2008 Dr. Jennifer W. Weller Dr. Andrew Carr. Instructors: Weller, Carr

Transcription

1 BINF 6211 SQL Lecture th, 2008 Dr. Jennifer W. Weller Dr. Andrew Carr

2 A Simple Schema DataFATE

3 SQL: A Brief History The original paper on RDBMS Codd, E.F. (June 1970). "A Relational Model of Data for Large Shared Data Banks". Communications of the ACM 13 (No. 6): pp Association for Computing Machinery. Structured English Query Language (SEQUEL) Donald D. Chamberlin and Raymond F. Boyce of IBM Name changed to SQL UC Berkley pioneered the first open-source query language: Ingres in 1974 Oracle (Relational Software, Inc.) released first commercial package in ANSI adopted SQL as a standard language ISO standard in 1987

4 SQL: History Continued Until 1996 NIST maintained the standard Currently vendors have to self monitor Each vendor uses slightly different extensions to the language. The core set of commands is standard across all SQL compliant languages.

5 SQL: Is easy!? SQL has a relatively small set of key words Not imperative languages like C++ and java. What does this mean? The user does not need to know how the system works underneath. how it is to be done. [pg216 (Rob and Coronel)]» Is this wise? SQL is a declarative language based on set theory. This means that the user can simply ask questions of the database. Caveats: Stupid questions Stupid Answers Even well posed queries can result in poor or erroneous results depending on the design. The answers I don t know doesn t mean the information isn t present.» Typically the user can t or doesn t know how to access it. SQL is a manner in which the user can retrieve information SQL is not a language that allows the construction of menus, windows, widgets Some tools like SQLServer provide additional language tools for added functionality The open source systems MySQL and Postgres provide hooks to scripting languages like Python and Perl. Additionally lower level languages like Java, C++ can by used.

6 SQL: Commands Two Types of commands (pg215 Rob and Coronel) Data Definition Used to create, alter and destroy tables, schemas, views Used to define constraints on columns Data Manipulation Used to pull and push data at the record level. Subsets: Commands Operators Aggregate functions

7 Creating a Database Three ways in Postgres Command Line psql (default is postgres)» If database exists: psql test CREATE DATABASE test At OS prompt >> createdb test PGAdmin Right click on databases Select new database

8 Create a user First step is to create a user Command line>> createuser me s P SQL inside the database CREATE ROLE me SUPERUSER PASSWORD?? PG-Admin Right click on new Login Roles

9 Creating a Tables SQL: CREATE TABLE Requires column definition CREATE TABLE AS (Select * FROM ) ALTER TABLE SQL: CREATE TABLE dataf_cl aa (x_index real, y_index real, hash_index integer, signal_intensity real, spotarea integer, affy_stdv real, df_tablename varchar(80)) WITHOUT OIDS; ALTER TABLE dataf_cl aa OWNER TO "DataFATE";

10 Loading data: INSERT INSERT Used to enter data into table Syntax: INSERT INTO columnname VALUES (value1, value2,, valuen); The insert command does one tuple at a time It will check all system constraints. INSERT INTO dataf_cl aa VALUES SQL: INSERT INTO DF_DataTable_Registry (DF_DataTable_Name, df_experiment_name, DF_QTSet_Name, DF_User, DF_Input_File) VALUES ( test, exp1, me, celfile1.cel ); Python: Qry1 = "INSERT INTO DF_DataTable_Registry (DF_DataTable_Name, df_experiment_name, DF_QTSet_Name, DF_User, DF_Input_File) VALUES (%s,%s,%s,%s,%s)" Data = [ test, exp1, me, celfile1.cel ] Qry.execute(Qry1,Data)

11 Loading Data: COPY COPY FROM Allows fast copy from a delimited, binary or ascii file into a table. The table columns must be in same order and of the same type as the data. No data can be in the table previously SQL: COPY DF_tbnm FROM file.txt Python: self.qrysu.copy_from(io,self.df_tbnm) A quick algorithm for loading data if data exists in the table: Create a temporary table COPY in new data ALTER original TABLE with UNION to temporary table What are the advantages and disadvantages of this?

12 Select SELECT is used to retrieve data Syntax: SELECT [columnlist] FROM [tablename] * is a wild card meaning all columns The result set retrieved by select statements is lost when ever the connection is closed. For semi-permanent: CREATE VIEW AS SELECT ( ) For permanent tables: CREATE TABLE AS SELECT ( ) SQL: SELECT DISTINCT DF_QTSet_Column_Map AS dcm, DF_QTSet_Column_Map_Name AS def FROM DF_WorkSpace WHERE df_experiment_name = Expname AND DF_QTSet_Name = self.qtsetname AND df_procedure_type = 'Data Load ; Python: Note: lst = qry.fetchall() provides an object that stores the query result

13 Rollback and Commit Commit and rollback are used to control the database. COMMIT permanently saves the changes COMMIT occurs when the connection to the database is closed ROLLBACK steps the table or database back to its previous state Using python and psycopg2: connection = " dbname = " + dbname + " user = " + user + " password = " + password try: self.dbconnectionsu = connect(connection) except: self.textlabel_response.settext("error1: Unable to connect.\n Please check user name, password and database information.") # The to commit any changes self.dbconnectionsu.commit() # To rollback any changes self.dbconnectionsu.rollback()

14 UPDATE Update is used to modify specific rows within a table Syntax: UPDATE tablename SET columnname = expression [, columname = expression] [WHERE conditionlist]; SQL: UPDATE df_experiment_qtset_link SET df_default_column_map = mapname WHERE df_experiment_name = self.expname AND DF_QTSet_Name = self.qtsetname ; Python: Qry1 ="UPDATE df_experiment_qtset_link SET df_default_column_map = mapname WHERE df_experiment_name = self.expname AND DF_QTSet_Name = self.qtsetname Qry.execute(Qry1)

15 On update cascade Insures that any changes to a key are passed to all foreign keys. Distribution of database changes. Foreign Key () REFERENCES ON UPDATE CASCADE.

16 ALTER All changes in table structure are made by using ALTER command Followed by keyword that produces specific change Following three options are available: ADD MODIFY DROP ALTER can be used to change data type Some RDBMSs (such as Oracle) do not permit changes to data types unless column to be changed is empty SQL: ALTER TABLE df_qtset_registry ADD CONSTRAINT df_qtset_registry_fkey1 FOREIGN KEY (df_user) REFERENCES df_user_registry(df_user)

17 DELETE and DROP DELETE Deletes a records or table rows Syntax: DELETE FROM tablename [WHERE condition]; DROP Removes a table from database Syntax: DROP TABLE tablename; SQL: DELETE FROM DF_DataTable_Registry WHERE DF_DataTable_Name = self.df_tbnm ; DROP DF_DataTable_Registry;

18 Operators In SQL operators are used to help define conditional statements or apply functions to individual columns, rows, or records. Column operators COUNT returns number of not null rows MIN and MAX SUM AVG Logical operators AND, OR, NOT Special operators BETWEEN IS NULL LIKE _ndr% (Note: the _, % are the SQL string wildcard characters) IN ( this, that, the other ) EXISTS if the resultant subquery is not null DISTINCT returns unique rows ORDER BY GROUP BY (HAVING) Mathematical operators +, -, ^, *, /, =

19 Table Joins Joining tables occurs when data is concatenated from two different tables within the database. Joining tables is the feature that sets RDMS apart from other database systems. Imagine trying to join two text files The most common type of join makes use of the Primary Key and Foreign Key relationships. There are three basic types of join Cross INNER OUTER NOTE: If columns from two separate tables have the same name, they will often need the table name preappended. ( table1.c1)

20 Cross Joins Cross Join is the simplest type of join All with all comparison Syntax: SELECT * FROM table1, table2 or SELECT * FROM table1 CROSS JOIN table2

21 INNER Joins INNER JOIN: Four types Where clause Select () FROM table1, table2 WHERE table1.c1 = table2.c2» Only rows with matching values are returned Natural Inner Join Returns rows with matching values in matching columns» Same column name is necessary with similar object types» If all columns with matching names are used. Join (Using) Similar to a natural join uses column with same name but not all if more than one exists. Join (On) Similar to the WHERE clause returns rows that meet a given criteria. Allows joining tables on columns where names are not the same.

22 Outer Joins Returns not only matching rows Also returns rows with unmatched attribute values for one table or both tables to be joined. Three types Left Gets all rows from data table on left with the matching rows from the right Right Gets all rows from table on the right with matching rows from left Full Returns all rows from both tables aligning those that match and then returning the others with nulls.

23 Subqueries Subqueries Nested queries Order of operations in SQL executes in a hierarchical and parenthetical manner. This means the innermost subquery is executed first» SELECT * FROM (SELECT * FROM ( SELECT * FROM )))

24 Advanced SQL Complex Queries, Triggers and Procedural language are relatively new to the SQL RDMS systems. In 2004 Postgres was the only open source system to support all of these functions. Currently all tools have them at some level.

25 Union and Union All UNION is used to combine two tables with the exact same attribute set. (i.e. the same column set) Union excludes duplicates: If one row occurs in both sets it will only occur once in the resultant set from the union statement UNION ALL will keep duplicate rows.

26 Intersection and MINUS INTERSECT Intersect returns the intersection between the two sets, returning only complete tuples that match between the two sets. MINUS returns the rows from the first set removing the rows that are found to be in the intersection between the sets. Alternative syntax is to use the IN and NOT IN statements IN (SELECT DISTINCT FROM )

27 ANY and ALL ANY and ALL are used with the comparators < and > in conjunction with a subquery that provides a list. example SELECT * FROM table2 WHERE value > ALL (SELECT value from table1 ) will return only lines from table2 that have values greater than all of those in table1. ANY works similarly with the result set being all rows of table2 that have values greater than any of the values in table1. (i.e. greater than the min of table1) = ANY is equivalent to IN

28 Advanced SQL String Functions concatenate To_Char Upper and Lower Substring Other Conversions To_Numeric NVL DECODE This takes a value and casts it to another type DECODE (State, NC, North Carolina )

29 Advanced SQL Triggers, procedures and functions are all tools that are constructed via PL/SQL. These are stored as part of the Schema and work when called Triggers are automated and associated with changes in tables. Functions and procedures can be called when needed. Embedded SQL are static queries specific to the database that are frequently called from within an external language (Python, Perl, Java)

30 Triggers Triggers are associated with specific tables and are executed automatically. Triggers are invoked when a table row is changed. Triggers are SQL code based. CREATE TRIGGER mytigger AFTER INSERT OR UPDATE of C1 ON table1 Begin End; SET C2 = MIN(C1)

31 PL/SQL stored procedures and functions Procedures and functions are complex queries that are stored as part of the schema and executed using the EXEC command Procedures operate to cause and action Functions return a value If more than one value is returned a cursor object is needed. Basic function syntax: CREATE FUNCTION functionname (argument IN data-type, ) RETURN data- type BEGIN PL/SQL statements; RETURN (value or expression); END;

32 Example problem Given a CDF from Affymetrix and a CEL file I want to know all of the match probes whose mean signal intensity is between 200 and 20,000 flourescent units. How do we write the query.

33 Example: Create a Database Create a User Install a Schema Load the data Using datafate. Write a simple query Concatenate the information merge the data from CDF and CEL files.

34 Example continued To get match probes: SELECT * from cdftable WHERE pm_mm_other IN (0,1); To get the signal intensities we want: SELECT * from celtable name WHERE signal_intensity >= 200 and signal_intensity <= 20000; How to we join this information? What happens if there are multiple tables?

35 Example Join First SQL: CREATE TABLE wk_extractiontable1 AS ((SELECT * FROM celtable NATURAL INNER JOIN arraytablewhere (signalrawintensity >= 200 and signalrawintensity <= and pm_mm_other >=0))) Filter First SQL: CREATE TABLE wk_extractiontable AS ((SELECT * FROM (SELECT * FROM celtable WHERE (signalrawintensity >= 200 and signalrawintensity <= 20000)) AS foo NATURAL INNER JOIN arraytable WHERE (pm_mm_other >=0))) Third method is?

36 Example Cont d Script it Python example qryc = "SELECT pm_mm_other AS pm, probeset_index AS psti, probeset_id AS pid" qryc = qryc + " FROM " + str(self.arrayname) +" ORDER by hash_index" t1 = time.time() self.qry.execute(qryc) ADres = self.qry.fetchall() t2 = time.time() # #Read from the table. # qry1a = "SELECT df_tablename as dft, x_index as xinx, y_index as yinx, hash_index as hinx, " qry1a = qry1a + " signal_intensity, spotarea as spta, Affy_STDV as asd FROM " #qry1b = "WHERE (QtCore.SIGNALrawintensity >= 200 and QtCore.SIGNALrawintensity <= and pm_mm_other >=0)" lntblist = len (self.selectedlist) Pbar_adjust = 70.0/lntblist Pbar_v = 0.0 # if lntblist == 0: QtGui.QApplication.restoreOverrideCursor() mssg = "No tables are selected.\n" mssg = mssg +" Please select tables that will be filtered." QtGui.QMessageBox.warning(self,"ProbeFATE ", mssg ) return else: # try: otfl = open('datafatetmp1_1_v1.txt', 'wr') except StandardError, err: QtGui.QApplication.restoreOverrideCursor() mssg = "System unable to open export file\n" mssg = mssg +str(err) QtGui.QMessageBox.warning(self,"ProbeFATE ", mssg ) return #io = StringIO.StringIO() for tbl in xrange(lntblist): #Loop through the tables t1 = time.time() tbnam = str(self.selectedlist[tbl]) #get first table name ##print tbnam qry1 = qry1a + tbnam + " ORDER by hash_index" #already ordered in CEL ##rint qry1 self.qrysu.execute(str(qry1)) res = self.qrysu.fetchall() lnj = len(res[0]) data = [] ##D.A.C print "errchk4:",len(res) if self.pmcheck == 1: #get only perfect match sequences for i in range(len(res)): temp1 = [] temp2 = [] #temp1.append(tbnam) #temp2.append(tbnam) if ADres[i]['pm'] == 1: #if perfect match sig = res[i]['signal_intensity'] #QtCore.SIGNAL intensity if (sig >= minv and sig <= maxv): #for j in range(lnj):#loop through list to convert to strings # temp1.append(str(res[i][j])) temp1.append(str(res[i]['dft'])) temp1.append(str(res[i]['xinx'])) temp1.append(str(res[i]['yinx'])) temp1.append(str(adres[i]['psti'])) temp1.append(str(adres[i]['pm'])) temp1.append("-none-") temp1.append("-none-") tmpln1 = join(temp1,'\t') + '\n' data.append(tmpln1)