Working with Data. L5-1 R and Databases

Transcription

1 Working with Data L5-1 R and Databases R R Open source statistical computing and graphics language Started in 1993 as an alternative to SAS, SPSS and other proprietary statistical packages Originally called S, renamed to R in 1996 R is a client and server bundled together as one executable It is a single user tool It is not multi-threaded Constrained to a single CPU Millions of R users worldwide Thousands of libraries available at Free Milestones: : packages : 9870 packages : 8492 packages : 6400 packages : 6325 packages : 6000 packages : 5000 packages : 4000 packages : 3000 packages : 2000 packages : 1000 packages : 500 packages : 250 packages 1

2 30/10/2017 Using RJDBC library(rjdbc) # Create connection driver and open connectionjdbcdriver <- JDBC(driverClass="oracle.jdbc.OracleDriver", classpath="c:/ojdbc6.jar") jdbcconnection <- dbconnect(jdbcdriver, "jdbc:oracle:thin:@//localhost:1521/orcl", "dmuser", "dmuser") #list the tables in the schema #dblisttables(jdbcconnection) #get the DB connections details - it get LOTS of info - Do not run unless it is really needed dbgetinfo(jdbcconnection) # Query on the Oracle instance name. #instancename <- dbgetquery(jdbcconnection, "SELECT instance_name FROM v$instance") TABLE_NAME1 1 INSUR_CUST_LTV_SAMPLE2 2 OUTPUT_1_2 #print(instancename)tablenames <- dbgetquery(jdbcconnection, "SELECT table_name from user_tables where table_name not like 'DM$%' and table_name not like 'ODMR$%'") print(tablenames) viewnames <- dbgetquery(jdbcconnection, "SELECT view_name from user_views")print(viewnames) 1 MINING_DATA_APPLY_V 2 MINING_DATA_BUILD_V 3 MINING_DATA_TEST_V 4 MINING_DATA_TEXT_APPLY_V 5 MINING_DATA_TEXT_BUILD_V 6 MINING_DATA_TEXT_TEST_V dbdisconnect(jdbcconnection) 2

3 Different ODBC drivers RODBC RJDBC ROracle RMySQL bluer If possible always use the driver for your Database. It will be specifically optimised for your Database. The RJDBC package is based on the database interface (DBI) established in the R community. The DBI package contains virtual classes; it is the responsibility of the underlying driver to implement the classes. RJDBC uses a combination of a JDBC compliant database driver and Java Runtime Environment (JRE) to exchange data between R and the database server. Using RODBC for Oracle is like using an ODBC connection for any database; so long as your platform provides an ODBC manager and drivers, you are OK. On Linux, this means unixodbc, and on Windows, this means the Oracle Data Access Components package. Sometimes it can be difficult to configure. Hence JDBC Step 1 - Download the ODBC driver for your DB First thing is download the ODBC driver for your Database Make sure it is the correct version Save it somewhere on your search Path 3

4 Step 2 Create your Connection Next, make sure you know how to connect to your source database. You ll need the following information for your database listener: Hostname or IP, e.g., database.company.com Port, e.g., 1521 Service name or SID, Username Password Use the connections details you used for SQL Developer ########################################################################## Create the connection to the Oracle schema ########################################################################## library(rjdbc) # Create connection driver and open connection to the database jdbcdriver <- JDBC(driverClass="oracle.jdbc.OracleDriver", classpath="c:/users/oracle/downloads/ojdbc6.jar") # dbconn <- dbconnect(jdbcdriver, "jdbc:oracle:thin:@//<hostname_or_ip:<port_number/<service_name", <username", <password") # Here is an example dbconn <- dbconnect(jdbcdriver, "jdbc:oracle:thin:@//redwood.ict.ad.dit.ie:1521/pdb12c.ict.ad.dit.ie", "demo_student", D ") Step 3 Explore the meta-data Don t use meta-data functions in these packages = very very very very slow Plus they bring back way more information than you would expect Lots and lots of useless info from the DB Use dbgetquery to query the data dictionary of the database. Runs the query in the DB and returns the results as a dataframe. Try for user_tables user_tab_columns user_indexes user_ind_columns mytables <- dbgetquery(dbconn, "select table_name from user_tables") mytables dblistfields(dbconn, "CARS") [1] "MPG" "CYL" "DISP" "HP" "DRAT" "WT" "QSEC" "VS" "AM" "GEAR" "CARB dbexiststable(dbconn, "CARS", "ORE_USER") [1] TRUE 4

5 Step 4 Query the data Return data as an R dataframe empdata <- dbgetquery(dbconn, "select * from emp") empdata empdata2 <- dbgetquery(dbconn, "select * from emp where salary 2000") Read the entire table tabledata <- dbreadtable(dbconn, "WHITE_WINE" )tabledata Prepared Statements dbsendupdate(dbconn, "UPDATE test1 set salary=? where id=?", teachersalary, teacherid) dbsendupdate(dbconn,"insert INTO test1 VALUES (?,?)",teacherid,teachersalary) Step 5 Updating data Updating the data in a table dbsendupdate dbsendupdate(dbconn, "UPDATE CUSTOMERS_USA SET cust_gender = 'X' WHERE cust_last_name = 'Everett'") customers <- dbreadtable(dbconn, "CUSTOMERS_USA") head(customers) # you should now see the updated rows 5

6 Step 6 Inserting data Updating data into a table cars <- mtcars dbwritetable(dbconn, "CARS", cars) carsdata <- dbreadtable(dbconn, "CARS") dim(carsdata) [1] dbwritetable(dbconn, "CARS", cars, append=true, overwrite=false) carsdata <- dbreadtable(dbconn, "CARS") dim(carsdata) [1] Step 7 Deleting data Deleting records in a table dbsendupdate(dbconn, "UPDATE FROM CUSTOMERS_USA WHERE cust_last_name = 'Everett'") 6

7 Step 8 Creating a table Creating a new table based on an R dataframe dbwritetable(dbconn, "CARS", cars) dbwritetable(dbconn, "CARS", cars, overwrite=false) Removing tables dbremovetable(dbconn, "CARS") Step 8 Creating a table Creating a new table based on an R dataframe # Write the Results table to the Database Sys.time() if (dbexiststable(dbconn, MY_TABLE_NAME", DEMO_STUDENT")){ warning("warning: Delete existing table") dbremovetable(dbconn, MY_TABLE_NAME") warning("re-creating table : Starting") dbwritetable(dbconn, MY_TABLE_NAME", df_table_results) dbcommit(dbconn) warning("re-creating table : Finished") } else { warning("info: Creating New table : Starting") dbwritetable(dbconn, MY_TABLE_NAME", df_table_results) dbcommit(dbconn) warning("info: Creating New table : Finished") } Sys.time() 7

8 Step 9 Saving changes Saving changes to the database (persisting changes) dbcommit(dbconn) Rollback statements or transactions dbrollback(dbconn) if dbgetinfo(rs, what="rowsaffected") 200) { warning("something has gone wrong") dbrollback(dbconn) } Step 10 Disconnecting etc What to do when finished your R session dbdisconnect(dbconn) # free resources occupied by result set dbclearresult(res) dbunloaddriver(drv) 8

9 Tip 1 When you get an error downloading the data Sometimes the number of records in a table can cause the creation of a dataframe to fail. Instead, process the data set in chunks Then merge into one dataframe. res<-dbsendquery(dbconn, "select * from sales ) result<-list() i=1 result[[i]]<-dbfetch(res,n=2000) while(nrow(chunk <- dbfetch(res, n=2000))0){ i<-i+1 result[[i]]<-chunk } dataextracted<-do.call(rbind,result) library(roracle) drv <- dbdriver("oracle") # Create the connection string host <- "localhost" port < sid <- "orcl" connect.string <- paste("(description=, "(ADDRESS=(PROTOCOL=tcp)(HOST=", host, ")(PORT=", port, "))", "(CONNECT_DATA=(SID=", sid, ")))", sep = "") con <- dbconnect(drv, username = "dmuser", password = "dmuser",dbname=connect.string) rs <- dbsendquery(con, "select view_name from user_views") # fetch records from the resultset into a data.frame data <- fetch(rs) # extract all rows dim(data) [1] 6 1 data VIEW_NAME 1 MINING_DATA_APPLY_V 2 MINING_DATA_BUILD_V 3 MINING_DATA_TEST_V 4 MINING_DATA_TEXT_APPLY_V 5 MINING_DATA_TEXT_BUILD_V 6 MINING_DATA_TEXT_TEST_V dbcommit(con) dbclearresult(rs) dbdisconnect(con) Using ROracle Needs Oracle Client in the search path Pulls the data to the Client Has a set of R functions tuned for the Oracle DB 9

10 The Challenges Scalability Regardless of the number of cores on your CPU, R will only use 1 on a default build Performance R reads data into memory by default. Easy to exhaust RAM by storing unnecessary data. Typically R will throw an exception at 2GB. Parallelization can be challenge. Is not Default. Packages available Production Deployment Difficulties deploying R in production Typically need to re-code in.. 10