DBI-B311. Revolution R 和微软数据平台 赵利超微软数据平台技术专家

Transcription

1 DBI-B311 Revolution R 和微软数据平台 赵利超微软数据平台技术专家

2 R 以及 R 在企业里的应用

3 Comprehensive R Archive Network

4 = 统计语言 + 数据挖掘 统计分析语言 巨大的算法包支持数据导入, 数据整理, 数据的统计分析和可视化

5 来源 :

6

7 Revolution R

8 数据量 内存限制 基于硬盘的扩展性 分析速度 单线程 多线程 企业级 开源社区支持 商业支持 广度和深度 6000 个算法分析包 6000 个算法分析包 + RevoScaleR 大数据分析包 商业上可行性 开源算法执行风险 商业许可

9 开源 R RRE 在相同的代码下, RevoR 要比开源 R 的性能上要快 3-50 倍 运算 (4 核笔记本 ) 开源 R Revolution R 加速 线性代数 矩阵乘法 176 sec 9.3 sec 18x Cholesky 分解 25.5 sec 1.3 sec 19x Linear Discriminant Analysis 189 sec 74 sec 3x General R Benchmarks 2 R Benchmarks (Matrix Functions) 22 sec 3.5 sec 5x R Benchmarks (Program Control) 5.6 sec 5.4 sec Not appreciable

10 Deployment / Consumption Advanced Analytics ETL Data / Infrastructure

11 Revolution R Open 免费, 而且开源 Revolution Analytics 对开源 R 的改进 Revolution R Enterprise 安全 可扩展支持分布式的 R 包含 RA 的专有组件

12 Revolution R 开放版 Revolution R 企业版 R 语言引擎多核计算支持支持 R 的大数据工具集 企业级并行数据仓库 网格计算 Hadoop 的分布式计算支持 ( 上下文 ) Web Services API 和企业级身份认证 支持 支持 支持

13 DeployR DevelopR ConnectR ScaleR DistributedR

14 数据分析 商业智能 R/ 统计模型专家 DeployR IT 专家 移动网页应用 云 /SaaS

15 并行计算上下文 rxcomputecontext 此外, rxexec 数据处理 Data import Delimited, Fixed, SAS, SPSS, OBDC Variable creation & transformation Recode variables Factor variables Missing value handling Sort, Merge, Split Aggregate by category (means, sums) 统计测试 Chi Square Test Kendall Rank Correlation Fisher s Exact Test Student s t-test 抽样 Subsample (observations & variables) Random Sampling 变量选择 Stepwise Regression Linear, Logistic and GLM 仿真 Monte Carlo Parallel Random Number Generation 聚类分析 K-Means 统计量 预测模型 Sum of Squares (cross product matrix for set Min / Max, Mean, Median (approx.) variables) Quantiles (approx.) Multiple Linear Regression Standard Deviation Generalized Linear Models (GLM) exponential Variance family distributions: binomial, Gaussian, inverse Correlation Gaussian, Poisson, Tweedie. Standard link Covariance functions: cauchit, identity, log, logit, probit. User Sum of Squares (cross product matrix for set defined distributions & link functions. variables) Covariance & Correlation Matrices Pairwise Cross tabs Logistic Regression Risk Ratio & Odds Ratio Classification & Regression Trees Cross-Tabulation of Data (standard tables & long Predictions/scoring for models form) Residuals for all models Marginal Summaries of Cross Tabulations 分类 Decision Trees Decision Forests NEW Stochastic Gradient Boosted Decision Trees 集成能力 Using Revolution rxdatastep and rxexec functions to combine open source R with Revolution R NEW PEMA API works with Open Source R v3

16

17 DeployR ConnectR ScaleR DistributedR Hadoop 企业级数据仓库集群单个服务器云端 Hortonworks, Cloudera 并行数据库仓库 高性能计算 / 网格计算群集 桌面 / 服务器 Microsoft Azure Amazon AWS

18 数据 分析 集成 决策 其他数据源

19 RRE 演示

20

21 Revolution R & 微软数据平台

22

23 SQL Server 2016 安装 Install Advanced Analytics with R feature in setup Install RRO on SQL Server 2016 machine Install RRE on SQL Server 2016 machine Optional: Install R packages on SQL Server 2016 machine 数据库配置 Enable R Language extension in the database Configure path for RRO runtime in the database Grant EXECUTE EXTERNAL SCRIPT permission to users CREATE EXTERNAL EXTENSION [R] USING SYSTEM LAUNCHER WITH (RUNTIME_PATH = 'c:\revolution\bin ) GRANT EXECUTE SCRIPT ON EXTERNAL EXTENSION::R TO DataScientistsRole; /* User-defined role / users */

24 R runtime usage Resource Governance via Resource Pool Monitoring via DMVs Troubleshooting via XEvents / DMVs CREATE RESOURCE POOL R_runtime FOR EXTERNAL EXTENSION WITH MAX_CPU_PERCENT = 20, MAX_MEMORY_PERCENT = 10; select * from sys.dm_resource_governor_resouce_po ols where name = 'R_runtime';

25

26

27 execute sysname Valid values are: R(Scale R, nvarchar(max) R script as a bit Indicates if multiple invocations of R script can be executed in parallel with distinct set of rows. Valid values are 0 1. Default is nvarchar(max) Input data for the R script in the form of a SQL nvarchar(max) {, = ] N'@parameter_name data_type [,...n ]' } {, = ] 'value1' [,...n ] } WITH <execute_option> [,...n ] ][;] <execute_option>::= { { RESULT SETS UNDEFINED } { RESULT SETS NONE } { RESULT SETS ( <result_sets_definition> [,...n ] ) } } List of columns to partition the input data List of input parameter declarations for the R script Values for the input parameters used by the R script <result_sets_definition> ::= { ( { column_name data_type [ COLLATE collation_name ] [ NULL NOT NULL ] } [,...n ] ) AS OBJECT [ db_name. [ schema_name ]. schema_name. ] {table_name view_name table_valued_function_name } AS TYPE [ schema_name.]table_type_name AS FOR XML }

28 IrisPredict <- function(data, model){ library(e1071) predicted_species <- predict(model, data) return(predicted_species) from my_iris_model) (select model library(rodbc) conn <- odbcconnect("mysqlazure", uid = myuser, pwd = mypassword); Iris_data <-sqlfetch(conn, "Iris_Data"); Iris_model <-sqlquery(conn, "select model from my_iris_model"); IrisPredict (Iris_data, model); IrisPredict <- function(data, model){ library(e1071) predicted_species <- predict(model, data) return(predicted_species) } IrisPredict(input_data_1, select * from

29 Sqlsrvr.ex e Launch External Process Named Pipe SQL Server Trusted Launchpad (MSLP$SQL16) Rlauncher.dll Launch RRO RTerm.exe (MSLP$ SQL16) R.dll RxLink.dll Named Pipe RRE BxlServer.exe (MSLP$SQL16 ) SQL/R Reader, Writer, Converter (Service Account ) TCP Data Channel SqlSatellite.dll

30 Example Solutions Fraud detection Sales forecasting Warehouse efficiency Predictive maintenance Analytic Library Native functions T-SQL Interface Relational Data SQL Server Extensibilit y R Integration R? 数据科学家 Interact directly with data Data Developer/DBA Manage data and analytics together 收益 更快的部署机器学习模型 性能更好 ( 将计算更靠近数据 ) 提升了扩展性 贴近数据的分析场景 欺诈监测 忠诚度分析 产品推荐

31 Data Sources Analytics Consumers Machines 传感器数据供应商 Legacy Sources EDW ERP/MRP 事件 输入数据打分流处理评估运算结构化数据 模型 Server Farm, EDW, Data Lake or Cloud Big Data Transformation Aggregation Exploration Modeling Model Evaluation Data Scoring Data Models Execution Math Servers and Clusters Data Models Execution 可视化 得分 Line of Business users (Analytic Apps, Rules Engines, etc.) Business Analysts (Alteryx, Tableau, Qlik, Cognos, Microstrategy, Datameer etc.) Power Analysts (R Studio, DevelopR, etc.)

32 Q&A

33 Microsoft-Ignite-China

34

35