Mike Schulte Data Scientist at the University of Pittsburgh Professor of Economics and Philosophy at Western Michigan University

Size: px

Start display at page:

Download "Mike Schulte Data Scientist at the University of Pittsburgh Professor of Economics and Philosophy at Western Michigan University"

Erick Eaton
6 years ago
Views:

2 Mike Schulte Data Scientist at the University of Pittsburgh Professor of Economics and Philosophy at Western Michigan University

3 Advanced Analytics Introduced Advanced Analytics within SQL Server and Excel R and RStudio Connecting R to SQL Server Solution Examples

6 Summary Statistics Historical View Traditional Business Intelligence work does a lot of this already

7 Fit Mathematical Models Present Day View Captures current capabilities and performances

8 Fit Statistical Models Forward-Looking View Captures likely outcomes for the future based on past and present outcomes

9 SQL Server and basic SQL statements Excel Data Mining Add-In Analysis Services and DMX R and R Services Microsoft Azure Machine Learning

11 library(e1071) nb_model <- naivebayes(class~.,data = products)

12 Wizard interface no programming required! Contained within Excel Limited capabilities Older algorithms

13 More flexible than the Excel add-in Integrates well with the rest of the SQL stack Limited capabilities Older algorithms Requires specialized knowledge of DMX

14 Statistical programming environment Open source Powerful and flexible Large user community Requires specialized knowledge of, well, R!

17 Academic statisticians Pharmaceutical companies Government agencies Professional consultants Business analysts Converted SAS users! More

18 Create a System DSN for connection Connect R to your SQL Database Pull data from SQL to R Analyze the data to create a model Operationalize the model This can still be a useful way to use R with SQL Server.

19 Open Administrative Tools in Control Panel

20 Manage the ODBC Data Sources

21 Create a New System Data Source Name

22 Choose SQL Server Native Client 11.0

23 Choose the SQL Server Installation You Want

24 Recommended: Use Windows Authentication

25 Install RODBC Package in R

26 Issue standard queries Drop, create, and fetch tables List available tables See documentation for more

27 Load RODBC Package and Connect to DSN

28 You can now issue queries from within R!

30 library(rodbc) Bring the Data into R channel <- odbcconnect("rconnection") autodata <- sqlquery(channel, "SELECT id, mpg, cylinders, displacement, horsepower, weight, acceleration FROM [dbo].[autodata];") trainingdata <- autodata[complete.cases(autodata),] missingdata <- autodata[!complete.cases(autodata),]

31 Build a Linear Regression Model and Impute automodel <- lm(mpg~horsepower+weight, data=trainingdata) missingdata$mpg <- round(predict(automodel, newdata=missingdata),1)

33 Update Our Database with Imputed Values for(i in 1:length(missingdata$id)){ string1 <- "UPDATE dbo.autodata SET mpg = " string2 <- as.character(missingdata$mpg[i]) string3 <- " WHERE id = " string4 <- as.character(missingdata$id[i]) querystring <- paste(paste(paste(paste(string1,string2,sep=""), string3,sep=""),string4,sep="")) sqlquery(channel,querystring) }

34 Note that this approach is new with SQL Server 2016.

35 Advantages: Data do not have to move Performance improvement (scale, parallelism) Challenges: Harder to code Harder to set up access

37 There are lots of use cases that fit into several categories: Association Analysis (Market Basket Analysis) Classification Estimation Simulation and Optimization Clustering And more

38 Products often sell well together. Some of these patterns are well established and may only be confirmed by the analysis. More unexpected patterns, like the apocryphal beer and diapers example, might be discovered too, providing additional insight.

39 Explore associations Confirm expected patterns Find unexpected patterns Create actionable insights

40 Set up periodic monitoring of known rules Detect drops in association strength and investigate

41 A charter fishing company wishes to determine the optimal number of boats to have in service. Too many boats will mean wasted resources, while too few boats will mean missed opportunities.

42 Use historical and forecast data to fit a distribution

43 Use the fitted distribution to project revenue for each additional boat. Decide how many boats to keep!

44 We would like to group countries that are economically similar to one another.

45 We begin with data on each country: Median GDP Growth (3 years) Population (in millions) Enabling Trade Index

46 setwd("c:/users/michael/desktop/demos") dfrm <- read.csv(file="clustering-demo-data.csv", header=t,stringsasfactors=f) dfrm$scgdpg <- scale(dfrm$medgdpg,center=t,scale=t) dfrm$scpop <- scale(dfrm$pop13,center=t,scale=t) dfrm$sceti <- scale(dfrm$eti,center=t,scale=t) kmc <- kmeans(dfrm[,5:7],centers=5,nstart=10)

47 Cluster 1: 50 Countries

48 Cluster 2: 42 Countries

49 Cluster 3: 11 Countries

50 Cluster 4: 2 Countries

51 Cluster 5: 33 Countries

52 What sale price should I use for Froot Loops?

53 Use historical data to determine lift for each price point.

54 Use lift to determine relative profit for each price point. Recommend a sale price to your marketing and sales teams!

55 Two Broad Areas of Concern: Jobs Ethics

Data mining overview. Data Mining. Data mining overview. Data mining overview. Data mining overview. Data mining overview 3/24/2014

Data mining overview. Data Mining. Data mining overview. Data mining overview. Data mining overview. Data mining overview 3/24/2014 Data Mining Data mining processes What technological infrastructure is required? Data mining is a system of searching through large amounts of data for patterns. It is a relatively new concept which is