Oracle R Technologies Overview

Transcription

1 <Insert Picture Here> Oracle R Technologies Overview Mark Hornick, Director, Oracle Database Advanced Analytics

2 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remain at the sole discretion of Oracle. 2

3 Mark Hornick Director, Oracle Advanced Analytics Joined Oracle in 1999 through acquisition of Thinking Machines Corp. Recent publications Using R to Unlock the Value of Big Data, Oracle Press Oracle Big Data Handbook, Oracle Press Founding member / Oracle Advisor Business Intelligence, Warehousing and Analytics SIG of IOUG BIWASummit.org at Oracle HQ, Redwood Shores, CA, Jan 14-16, 2014 Blogger: blogs.oracle.com/r Connect on LinkedIn: Mark Hornick 4

4 5

5 Motivation R increasingly recognized as the language of choice Oracle s R technologies scale R to the enterprise Oracle gives back to the R Community Share with the Boston Big Data Community 6

6 R s Popularity We can see that discussion of R has grown the most rapidly and, for the past few years, R is the most discussed software by an almost two-to-one margin. Robert A. Muenchen 7

7 2013 Data Mining Survey Vendors (18%) were excluded from these analyses 8

8 2013 Data Mining Survey 9

9 Oracle s R Technologies Oracle R Distribution ROracle Software available to R Community for free Oracle R Enterprise Oracle R Advanced Analytics for Hadoop 11

10 Agenda What is R? Three concerns: scalability, performance, deployment Oracle s R technologies How to get started 13

11 What is R? 14

12 What is R? R is an Open Source language and environment for statistical computing and graphics Started in 1994 as an alternative to SAS, SPSS, and other proprietary statistical environments An integrated suite of software facilities for data manipulation, analytical calculations, and graphics Over 2 million R users worldwide Widely taught in universities Many corporate analysts know and use R A thriving ecosystem with thousands of open sources packages 15

13 Why statisticians/data analysts use R R environment is.. Powerful Extensible Graphical Extensive statistics OOTB functionality with many knobs but smart defaults Ease of installation and use Free R is a statistics language similar to Base SAS or SPSS Statistics 16

14 Three Concerns for Enterprise Data 20

15 Three concerns for enterprise data analytics Scalability Performance Production Deployment 21

16 A fourth concern Remain in the R language and environment Same paradigm SQL not required Design, code, test, deploy from R 25

17 Oracle s R Technologies 27

18 Oracle R Distribution 28

19 Oracle R Distribution Ability to dynamically load Intel Math Kernel Library (MKL) AMD Core Math Library (ACML) Solaris Sun Performance Library Oracle Support Oracle s redistribution of open source R Enhanced linear algebra performance using Intel s MKL, AMD s ACML, and Sun Performance Library for Solaris Improve R scalability at client and at database server for embedded R execution Enterprise support for customers of Oracle Advanced Analytics option, Big Data Appliance, and Oracle Linux Free download Oracle makes bug fixes and enhancements available for open source R 29

20 Oracle R Distribution (ORD) Performance with MKL 3-node cluster 24 cores 3.07GHz per CPU 47 GB RAM Linux

21 ROracle 31

22 ROracle Oracle Database DBI & SQL R package enabling connectivity to Oracle Database Open source, publicly available on CRAN, free to R community Execute SQL statements from R interface Oracle Database Interface (DBI) for R based on OCI for high performance Supports Oracle R Enterprise database connectivity 32

23 Comparison loading database table to R data.frame ROracle Up to 79X faster than RJDBC Up to 2.5X faster than RODBC Scales across NUMBER, VARCHAR2, TIMESTAMP data types See 35

24 Comparison writing database table from R data.frame ROracle 61X faster for 10 cols x 10K rows than RODBC 630X faster on 10 cols x 10K rows than RJDBC Scales across the remaining data types 37

25 Example connecting to Oracle Database drv <- dbdriver("oracle") # Create the connection string host <- "myhost" port < sid <- "mysid" connect.string <- paste( "(DESCRIPTION=", "(ADDRESS=(PROTOCOL=tcp)(HOST=", host, ")(PORT=", port, "))", "(CONNECT_DATA=(SID=", sid, ")))", sep = "") con <- dbconnect(drv, username = "scott", password = "tiger", dbname = connect.string) 38

26 Example exerting transactional control dbreadtable(con, "EMP") rs <- dbsendquery(con, "delete from emp where deptno = 10") dbreadtable(con, "EMP") if(dbgetinfo(rs, what = "rowsaffected") > 1){ warning("dubious deletion -- rolling back transaction") dbrollback(con) } dbreadtable(con, "EMP") Example from ROracle package documentation 39

27 Oracle R Enterprise 41

28 Traditional R and Database Interaction read Flat Files extract / export Database export load SQL RODBC / RJDBC / ROracle R script cron job R memory limitation data size, call-by-value R single threaded Paradigm shift: R SQL R Access latency, backup, recovery, security Ad hoc script execution or porting code to target environment 42

29 Oracle R Enterprise A comprehensive, database-centric environment for end-to-end analytical processes in R, with immediate deployment to production environments Operationalize entire R scripts in production applications eliminate porting R code Seamlessly leverage Oracle Database as HPC environment for R scripts, providing data parallelism and resource management Execute R scripts through Oracle Database server machine for scalability and performance Enable integration and management through SQL Avoid reinventing code to integrate R results into existing applications Score R models in Oracle Database Transparently analyze and manipulate data in Oracle Database through R using versatile and customizable R functions Eliminate memory constraint of client R engine Get maximum value from your Oracle Database and Exadata Integrate R into the IT software stack, e.g. OBIEE Client R Engine Transparency Layer ORE packages Oracle Database User tables In-db stats SQL Interfaces SQL*Plus, SQLDeveloper, Database Server Machine 43

30 OBIEE Dashboard Integration Parameterized analytics and graph customization Improve time to insight Accommodate diverse consumption paths Deliver analytics that scale with data volumes, variables, techniques Integrate readily with IT infrastructure and software stack Leverage CRAN packages at database server 44

31 Performance In-Database Advantage 49

32 Performance In-Database Advantage 50

33 Time (seconds) Time (seconds) Time (seconds) Scalability In-Database Advantage GLM Model Build (DOP=64) Decision Tree Model Build (DOP=64) SVM Model Build (DOP=64) K 1M 10M 100M 1 100K 1M 10M 100M 1B 1 100K 1M 10M 100M Number of Rows Number of Rows Number of Rows 11 seconds to score 100M records (DOP=64) 92 seconds to score 1B records (DOP=128) 13 seconds to score 1B records (DOP=128) 2466 seconds to score 1B records (DOP=64) Exadata X2-2 half-rack 51

34 Oracle Database 12.1 Parallel Distributed Advanced Analytics Real world proof points Linear Regression (ore.lm) on Exadata X3-2 half-rack Data set: 2.9 billion rows spanning 12 months of data with over 350 predictors Elapsed time ~5 minutes! Logistic Regression (ore.glm) on Exadata X3-2 half-rack Data set: 2.9 billion rows spanning 12 months of data with over 350 predictors Elapsed time ~30 minutes! Neural networks (ore.neural) on T5-4 Solaris Data set: 1 billion rows with 40 columns Elapsed time ~6 minutes with 10 hidden neurons & 421 weights 52

35 Components of Oracle Advanced Analytics Option Fastest Way to Deliver Scalable Enterprise-wide Predictive Analytics Powerful Accelerate rate at which business problems are tackled Improve time to insight Combination of in-database predictive algorithms and open source R algorithms Accessible via SQL, PL/SQL, R and database APIs Scalable, parallel in-database execution of R language Easy to Use Expand user population that can build models Range of GUI and IDE options for business users to data scientists Enterprise-wide Integrated feature of Oracle Database via SQL - R is integrated into SQL Seamless support for enterprise analytical applications / BI environments Oracle R Enterprise + Oracle Data Mining 54

36 What might you want to do on a large data set? large == doesn t fit in memory Data set of 100M rows and 500 columns numeric data ~400 GB in database ~600 GB as CSV file Select data Summarize Aggregate Visualize data - boxplot Sample data Build models, persist models, score data Deploy R scripts in production executed from SQL 56

37 ORE Transparency Layer 57

38 Transparency No need to learn a different programming paradigm or environment Operate on database data as though they were R objects using R syntax Require minimal change to base R scripts for database data Implicitly translates R to SQL for in-database execution, performance, and scalability The Transparency Layer supports in-database data exploration, data preparation, and data analysis en route performing predictive analytics with a mix of in-database and CRAN techniques. 58

39 Establish a connection to Oracle Database library(ore) if (!ore.is.connected()) ore.connect(user="rquser", sid="orcl", host="localhost", password="rquser", all=true) ore.ls() 59

40 ore.ls returns names of ore.frame objects ore.frame objects are proxies for database tables Invoke R functions on ore.frame objects as if data.frames Data and results stay in Oracle Database until needed Example: ONTIME_S table as ore.frame Domestic airline flight data over 22 years Sample of ~220K rows x 26 columns 60

41 Data Selection Column selection df <- ONTIME_S[,c("YEAR","DEST","ARRDELAY")] class(df) head(df) head(ontime_s[,c(1,4,23)]) head(ontime_s[,-(1:22)]) Row selection df1 <- df[df$dest=="sfo",] class(df1) df2 <- df[df$dest=="sfo",c(1,3)] df3 <- df[df$dest=="sfo" df$dest=="bos",1:3] head(df1) head(df2) head(df3) 61

42 Summarize Data res <- summary(ontime_s[,1:13]) class(res) # table res 62

43 Aggregate Data R aggdata <- aggregate(ontime_s$dest, by = list(ontime_s$dest), FUN = length) class(aggdata) head(aggdata) Client R Engine Transparency Layer Oracle R Enterprise SQL select DEST, count(*) from ONTIME_S group by DEST Oracle Database ONTIME_S In-db stats 63

44 Visualize Data Overloaded graphics functions for in-database statistics dat <- LTV value <- CUST_LIFETIME_VALUE$LTV part <- dat$region bd <- split(value, part) boxplot(bd, notch = TRUE, col = "red", cex = 0.5, outline = FALSE, axes = FALSE, main = "Customer LTV by Region Distribution", ylab = "Lifetime Value ($)", xlab = "Region") axis(1, at=1:length(levels(part)), labels=levels(part)) axis(2) 64

45 High performance in-database sampling techniques dat <- ore.pull( ) samp <- dat[sample(nrow(x),size,] samp <- x[sample(nrow(x), size),,] samp <- ore.pull( ) Oracle Database Data Oracle Database Data Simple random sampling Split data sampling Systematic sampling Stratified sampling Cluster sampling Quota sampling Accidental / Convenience sampling via row order access via hashing 66

46 Simple random sampling Select rows at random set.seed(1) samplesize < row.names(ontime_s) <- ONTIME_S$YEAR simplerandomsample <- ONTIME_S[sample(nrow(ONTIME_S), samplesize),, drop=false] class(simplerandomsample) simplerandomsample[1:5,c(1,19:21)] 67

47 R Object Persistence Provide database storage to save/restore R and ORE objects across R sessions Facilitate production deployment Simplify application architecture Use cases Enable access to predictive models for scoring Passing complex arguments to R functions Preserve ORE objects across R sessions Functions: ore.save(), ore.load() ore.load(name="ds1") ls() x1 x2 x1 <- ore.lm(...) x2 <- ore.frame(...) ore.save(x1,x2,name="ds1") R Datastore ds1 {x1,x2} 70

48 ORE Analytics Packages and Functions 71

49 Philosophy for In-Database Analytics Whenever possible leave the data in the database leverage in-database statistics and algorithms do computations where the data reside 73

50 R Interface to In-Database Statistical Functions Special Functions Gamma function Natural logarithm of the Gamma function Digamma function Trigamma function Error function Complementary error function Tests Chi-square, McNemar, Bowker Simple and weighted kappas Cochran-Mantel-Haenzel correlation Cramer's V Binomial, KS, t, F, Wilcox Base SAS equivalents Freq, Summary, Sort Rank, Corr, Univariate Density, Probability, and Quantile Functions Beta distribution Binomial distribution Cauchy distribution Chi-square distribution Exponential distribution F-distribution Gamma distribution Geometric distribution Log Normal distribution Logistic distribution Negative Binomial distribution Normal distribution Poisson distribution Sign Rank distribution Student's t distribution Uniform distribution Weibull distribution Density Function Probability Function Quantile 74

51 R interface to in-database predictive algorithms SVM GLM k-means clustering Naïve Bayes Decision Trees Attribute Importance Neural Networks Linear Regression Stepwise Regression R interface to Oracle Data Mining algorithms 75

52 Build models in R, predict in-database OREpredict package provide a commercial grade scoring engine High performance Scalability Simplified application workflow Use R-generated models to score in-database on ore.frame Maximizes use of Oracle Database as compute engine Function ore.predict S4 generic function Specific method for each model supported by ORE 83

53 ore.predict supported algorithms Class Package Description glm stats Generalized Linear Model negbin MASS Negative binomial Generalized Linear Model hclust stats Hierarchical Clustering kmeans stats K-Means Clustering lm stats Linear Model multinom nnet Multinomial Log-Linear Model nnet nnet Neural Network rpart rpart Recursive Partitioning and Regression Tree 84

54 Example using lm irismodel <- lm(sepal.length ~., data = iris) IRIS <- ore.push(iris) IRISpred <- ore.predict(irismodel, IRIS, se.fit = TRUE, interval = "prediction") IRIS <- cbind(iris, IRISpred) head(iris) Build a typical R lm model Use ore.predict to score data in Oracle Database using ore.frame, e.g., IRIS Model implicitly translated to SQL for in-database scoring 86

55 ORE Embedded R Script Execution 87

56 Execute R script at database server machine res <- ore.doeval( function (num = 10, scale = 100) { ID <- seq(num) data.frame(id = ID, RES = ID / scale) }) Client R Engine ORE R user on desktop class(res) # ore.object res local_res <- ore.pull(res) class(local_res) # data.frame local_res DB R Engine ORE Oracle Database rq*apply interface User tables extproc 88

57 Results 89

58 Embedded R Execution Production deployment of R scripts Store and manage R scripts in Oracle Database Schedule R scripts for automatic execution Execute R scripts via SQL returning structured data, XML, images Scalability Data-parallelism Task-parallelism Performance High performance read/write between server R engine and Oracle Database Leverage powerful database server machine: memory, CPU 90

59 Execute R script data parallel at database server machine modlist <- ore.groupapply( X=LTV, INDEX=LTV$REGION, function(dat) { lm(ltv ~ SALARY+GENDER+AGE+HAS_CHILDREN+ MARITAL_STATUS+HOUSE_OWNERSHIP, dat) }); mod.ne <- ore.pull(modlist$northeast) summary(mod.ne) Client R Engine Transparency Layer ORE Oracle Database Family of embedded R functions ore.doeval ore.tableapply ore.groupapply ore.rowapply ore.indexapply DB R Engine ORE rq*apply () ORE User tables interface extproc extproc DB R Engine ORE 91

60 Results 92

61 Production Deployment store R script in DB repository ore.scriptdrop("simplescript1") ore.scriptcreate("simplescript1", function (num = 10, scale = 100) { ID <- seq(num) data.frame(id = ID, RES = ID / scale) }) res <- ore.doeval(fun.name="simplescript1", num = 20, scale = 1000) 96

62 Production Deployment using SQL API SQL begin sys.rqscriptcreate('example1', 'function() { ID <- 1:10 data.frame(id = ID, RES = ID / 100) }'); end; / select * from table(rqeval(null, 'select 1 id, 1 res from dual', 'Example1')); 97

63 Hello World! XML Example set long set pages 1000 begin sys.rqscriptcreate('example5', 'function() { res <- "Hello World!" res }'); end; / select name, value from table(rqeval( NULL, 'XML', 'Example5')); 99

64 Production Deployment same R function, multiple uses begin sys.rqscriptdrop('randomreddots'); sys.rqscriptcreate('randomreddots', 'function(){ id <- 1:10 plot(1:100,rnorm(100),pch=21,bg="red",cex =2) data.frame(id=id, val=id / 100) }'); end; / select from value table(rqeval( NULL,'XML', 'RandomRedDots ')); select from ID, IMAGE table(rqeval( NULL,'PNG', 'RandomRedDots ')); select * from table(rqeval( NULL, 'select 1 id, 1 val from dual','randomreddots')); 100

65 Results XML result PNG result select 1 id, 1 val from dual result 101

66 Oracle R Advanced Analytics for Hadoop 108

67 Goals Expand user population that can build models on Hadoop Accelerate rate at which business problems are tackled Deliver analytics that scale Data volumes Variables Techniques 110

68 Oracle R Advanced Analytics for Hadoop Big Data Appliance R script Hadoop Cluster ORD {CRAN packages} Hadoop Job Mapper R HDFS R MapReduce R sqoop MapReduce Nodes {CRAN packages} ORD HDFS Nodes R Client Reducer Oracle Database Provide transparent access to Hadoop Cluster Manipulate data in HDFS, database, and file system - all from R Write and execute MapReduce jobs with R Leverage CRAN R packages to work on HDFS-resident data Move from lab to production without requiring knowledge of Hadoop internals, Hadoop CLI, or IT infrastructure 111

69 Text analysis example Count the number of times each word occurs in a corpus of documents map One mapper per block of data Shuffle and Sort reduce One or more reducers combining results Documents divided into blocks in HDFS Outputs each word and its count: 1 each time a word is encountered Key Value The 1 Big 1 Data 1 word 1 count 1 example 1 One reducer receives only the key-value pairs for the word Big and sums up the counts Key Value Big 1... Big 1 It then outputs the final key-value result Key Value Big

70 Mapper and reducer code in ORCH for Word Count corpus <- scan("corpus.dat", what=" ",quiet= TRUE, sep="\n") corpus <- " ", corpus) input <- hdfs.put(corpus) res <- hadoop.exec(dfs.id = input, mapper = function(k,v) { x <- strsplit(v[[1]], " ")[[1]] x <- x[x!=''] out <- NULL for(i in 1:length(x)) out <- c(out, orch.keyval(x[i],1)) out }, reducer = function(k,vv) { orch.keyval(k, sum(unlist(vv))) }, config = new("mapred.config", job.name = "wordcount", map.output = data.frame(key='', val=0), reduce.output = data.frame(key='', val=0) ) ) res hdfs.get(res) Load the R data.frame into HDFS Specify and invoke map-reduce job Split words and output each word Sum the count of each word 113

71 Prepackaged Analytics Functions Function Description Application orch.lm orch.lmf orch.neural orch.nmf Fits a linear model using tall-and-skinny QR (TSQR) factorization and parallel distribution. The function computes the same statistical parameters as the Oracle R Enterprise ore.lm function. Fits a low rank matrix factorization model using either the jellyfish algorithm or the Mahout alternating least squares with weighted regularization (ALS-WR) algorithm. Provides a neural network to model complex, nonlinear relationships between inputs and outputs, or to find patterns in the data. Provides the main entry point to create a nonnegative matrix factorization model using the jellyfish algorithm. This function can work on much larger data sets than the R NMF package, because the input does not need to fit into memory. Regression analysis: predict numeric values e.g., Customer Lifetime Value Feature extraction e.g., recommender systems Regression e.g., predicting inventory demand Feature extraction e.g., recommender systems, dimension reduction 116

72 ORCHhive 117

73 What is Hive? SQL-like abstraction on Hadoop Becoming de facto standard for SQL based apps on Hadoop Converts SQL queries to map-reduce jobs to be run on Hadoop Provides simple query language (HQL) based on SQL Enables non-java users to leverage Hadoop via SQL-like interfaces 118

74 Motivation for ORCH Hive Big data scalability and performance for R users on Hadoop Enable R users to clean, explore, and prepare Hive data transparently Ready data for analytic techniques using ORCH map-reduce framework ORE provides transparent access to database tables and views from R based on SQL mapping Since Hive is SQL-based, natural to provide ORE-type transparency 119

75 Example using ORCHhive # Connect to Hive ore.connect(type="hive") # Attach the current environment into search path of R ore.attach() # create a Hive table by pushing the numeric columns of the campaign data set CMPGN_TABLE <- ore.push(campaigns[1:4]) # Create bins based on Years as Customer CMPGN_TABLE$YRS_AS_CUST = ifelse(cmpgn_table$yrs_as_cust < 2.0, "SHORT", ifelse(cmpgn_table$yrs_as_cust < 4.0, "MEDIUM", ifelse(cmpgn_table$yrs_as_cust < 6.0, "MEDIUM LONG", "LONG"))) # YRS_AS_CUST is now a derived column of the HIVE object names(yrs_as_cust) # Based on the bins, generate summary statistics for each group aggregate(cmpgn_table$yrs_as_cust, by = list(yrs_bins = CMPGN_TABLE$YRS_AS_CUST), FUN = summary) 120

76 Getting Started 123

77 Getting Started Tutorial resources available on the Oracle R Enterprise home page See Oracle software can be freely downloaded and installed for evaluation See our blog post at blogs.oracle.com/r for an ORE Test Drive Let us know about your experience! 124

78 Resources Book: Using R to Unlock the Value of Big Data, by Mark Hornick and Tom Plunkett Blog: Forum: Oracle R Distribution: ROracle: Oracle R Enterprise: Oracle R Advanced Analytics for Hadoop: 125

79 126

80 127