Practical Data Science with R. Matthew Renze CFT Training for AIG

Transcription

1 Practical Data Science with R Matthew Renze CFT Training for AIG

2 Motivation The ability to take data to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it that s going to be a hugely important skill in the next decades, because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it. Hal Varian, Google s Chief Economist The McKinsey Quarterly, Jan 2009

3 Source: Dice 2014 Tech Salary Survey Results

4 A Flood of Data is Coming... Source: Source: Wikipedia Sink or Swim

5 Overview 1. Introduction 2. Transforming Data 3. Descriptive Statistics 4. Data Visualization 5. Statistical Modeling 6. Handling Big Data 7. Machine Learning 8. R in Practice

6 How Does This Apply to Me? As a software developer, I often: Perform log file analysis Analyze software performance Analyze code metrics for code quality Detect anomalies in source data Transform or clean data files to make them usable Help decision makers make decisions based on data

7 How Does This Apply to Me? Deterministic Programming Probabilistic Programming If x and y then z If P(x) > y then z

8 About Me Independent software consultant Education B.S. in Computer Science B.A. in Philosophy Community Pluralsight Author Public Speaker Microsoft MVP ASPInsider Open-Source Software

9 Agenda 1. Introduction 2. Transforming Data 3. Descriptive Statistics 4. Data Visualization 5. Statistical Modeling 6. Handling Big Data 7. Machine Learning 8. R in Practice 1. Transforming Data Project 2. Descriptive Statistics Project 3. Data Visualization Project 4. Statistical Modeling Project 5. Machine Learning Project

10 Schedule Lectures (15 min) Demos (10 min) Labs (20 min) Breaks (10 min)

11 Logistics Pairing for labs is optional Ask questions if needed Come and go as needed Feedback forms at the end

12 Lab Options Type all code Copy and paste Run demo scripts

13 Workshop URL

14 Introduction to Data Science

15 What is Data Science Interdisciplinary field Derive insight from data Transform data into knowledge Source:

16

17 What is a Data Scientist? Performs data science Proper accreditations More than a scientist More than an analyst More than a developer Source:

18

19 The Data Science Toolkit Programming Data transformation Descriptive statistics Data visualization Statistical modeling Big Data Machine learning

20 Source: O Reilly 2015 Data Science Salary Survey

21

22

23 Why is Data Science Important? Lots of data Fast computers Powerful analysis tools Prediction vs. explanation

24 The Data Science Process 1. Ask a question 2. Explore the data 3. Prepare the data 4. Create a model 5. Evaluate the model 6. Deploy the model / results

25 The Data Science Process Iterative process Non-sequential Early termination Established process (CRISP-DM)

26 Introduction to R

27 What is R? Open source Language and environment Numerical and graphical analysis Cross platform

28 What is R? Active development Large user community Modular and extensible extensions

29 FREE!

30 Source:

31

32 Source:

33

34 Code Demo

35 Lab 1 R Programming Basics

36 Transforming Data

37 Data Munging Transforming data Raw data to usable data Data must be cleaned first

38 Data Munging Tasks Renaming variables Data type conversion Encoding, decoding or recoding data Merging data sets Transforming data Handling missing data Handling anomalous values

39 Loading Data in R File-based data Web-based data Databases Statistical data And many more CSV XML

40 Cleaning Data This step is often the Most difficult Most time consuming TIP: Record all steps

41 Data Munging Tools in R plyr dplyr reshape2 tidyr stringr lubridate

42 dplyr Single Data Frame Verbs Select Filter Mutate Summarize Arrange Multiple Data Frame Verbs Inner join Left join Semi join Anti join Union Intersect Set difference

43 Open Movies Database Title Year Rating Runtime (minutes) Movies Genre Critic Score Box Office Awards International The Whole Nine Yards 2000 R 98 Comedy 45% $57.3M No No Cirque du Soleil 2000 G 39 Family 45% $13.4M Yes No Gladiator 2000 R 155 Action 76% $187.3M Yes Yes Dinosaur 2000 PG 82 Family 65% $135.6M Yes No Big Momma's House 2000 PG Comedy 30% $0.5M Yes Yes

44

45

46 1. Column with wrong name 2. Rows with missing values 3. Runtime column has units 4. Revenue in multiple scales 5. Wrong file format

47 Code Demo

48 Lab 2 Transforming Data

49

50 Descriptive Statistics

51 Descriptive Statistics Movie Runtime Statistic Value (minutes) Describe data Provides a summary aka: Summary statistics Minimum 38 1 st Quartile 93 Median 101 Mean rd Quartile 113 Maximum 219

52 Statistical Terms ID Date Customer Product Quantity Observations Variables Qualitative variable Quantitative variable John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

53 Number of Variables Types of Analysis Number of variables 1 - Univariate 2 - Bivariate n - Multivariate Type of variables Categorical - Qualitative Numeric - Quantitative One Categorical Variable Two Categorical Variables Categorical & Numeric Variable Many Variables One Numeric Variable Two Numeric Variables Type of Variable(s)

54 Analyzing One Categorical Variable Movies by Genre Genre Frequency Percentage Action 612 9% Qualitative univariate analysis Frequency of observations Adventure 496 7% Animation 168 2% Comedy % Drama % Horror 269 4%

55 Analyzing One Numeric Variable Quantitative univariate analysis Central tendency Dispersion Shape

56 Visualizing Two Categorical Variables Movies by Genre and Rating Genre G PG PG-13 R Total Qualitative bivariate analysis Joint frequency Marginal frequency Relative frequency Contingency table Action Adventure Animation Comedy Drama Family Total

57 Visualizing Two Categorical Variables Movies by Genre and Rating Genre G PG PG-13 R Total Qualitative bivariate analysis Joint frequency Marginal frequency Relative frequency Contingency table Action Adventure Animation Comedy Drama Family Total

58 Analyzing Two Numeric Variables Quantitative bivariate analysis Predictor vs. outcome Covariance Correlation

59 Analyzing a Numeric Variable Grouped By a Categorical Variable One categorical variable One numeric variable Aggregate measures

60 Number of Variables Analyzing Many Variables One Categorical Variable One Numeric Variable Multivariate analysis Specific meaning in statistics Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

61

62 Cowboys & Space Invaders: The Musical Extended Edition

63 Code Demo

64 Lab 3 Descriptive Statistics

65 Cowboys & Space Invaders: The Musical Extended Edition

66 Data Visualization

67 Data Visualization Visual data representation For human pattern recognition Map dimensions to visual

68 Data Visualization ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

69 Number of Variables Types of Data Visualizations One Categorical Variable One Numeric Variable Number of variables Type of variable(s) Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

70 Visualizing One Categorical Variable Frequency Proportion

71 Visualizing One Numeric Variable Location Spread Shape

72 Visualizing Two Categorical Variables Joint frequency Marginal frequency Relative frequency

73 Visualizing Two Numeric Variables Relationship Correlation Distribution

74 Visualizing Two Numeric Variables Relationship Correlation Distribution

75 Visualizing Time Series Data Values over time Rate of change

76 Visualizing a Numeric Variable Grouped by a Categorical Variable Aggregate Grouped Comparison

77 Visualizing Many Variables

78 Source: Nathan Yau (

79 Plotting Systems in R Base Lattice ggplot2

80 Cowboys & Space Invaders: The Musical Extended Edition

81 Code Demo

82 Lab 4 Data Visualization

83 The Warlord of the Rings : An Unexpected Adventure Feature Length PG

84 Statistical Modeling

85 Statistical Models Representation of reality Mathematical function Parametric models Non-parametric models

86 Types of Statistical Models Probability distribution ANOVA Linear regression Non-linear regression Bayesian network

87 Gaussian Distribution Probability distribution Parametric model Mean Standard deviation Generative model

88 Simple Linear Regression Linear model Relationship between variables Continuous explanatory variable Single scalar dependent variable

89 Simple Linear Regression Linear predictor function y = m x + b Parameters estimated from data Makes certain assumptions

90 Source:

91

92 Photo by Danielle Langlois

93 Code Demo

94 Lab 5 Statistical Models

95 Photos by Radomił Binek, Danielle Langlois, and Frank Mayfield

96 Handling Big Data

97 What is Big Data?

98 Big Data is a Moving Target What is Big Data today Will not be Big Data tomorrow

99 Do I Have Big Data? Can it fit in memory? Can it fit on your hard drive? Can you process it in a day? Does it fit into tables?

100 Big Data Decision Table Source: Michael E Driscoll Note: Last updated in 2010

101 Decision Table for Big Data in R Class Rows Manage with Small Less than 1 million R on desktop computer Medium 1 million to 1 billion R with 3 rd party support Big More than 1 billion R with big data support Source: Jan Wijffels at user!-conference (2013)

102 If you don t have a big data problem, but you think you have a big data problem, then you ve just created a big data problem!

103 How to Handle Big Data in R Sampling Microsoft R Open 3 rd -party medium data packages 3 rd -party big data packages Microsoft R Server

104 Sampling Randomly selected Subset of original data Low-cost solution

105 Microsoft R Open Enhance 64-bit R distribution Multithreaded performance Only helps with CPU constraint Package repository snapshots Source: Microsoft R Open

106 Microsoft R Open Performance Source:

107 3 rd -Party Extension Packages Big Memory ff

108 3 rd -Party Extension Packages pbdr Rhipe Rhive Rhbase Rhdfs, Rmr Source: Wikipedia

109 Microsoft R Server Microsoft R Server Windows SQL Server 2016 SUSE Linux Redhat Linux Teradata Hadoop HD Insight Source: Microsoft R Server

110 No Demo or Lab 6

111 Machine Learning

112 What is Machine Learning? Subtype of artificial intelligence Learning without being programmed Similar to statistical modeling Similar to data mining Source: Futurama

113 How does Machine Learning Work? Uses statistical models Model s parameters are trained Trained with algorithm and data Model is used to predict output Prediction vs. explanation Source: Futurama

114 What Can Machine Learning Do? Classification Regression Clustering Anomaly detection Source: Futurama

115 Types of Machine Learning Supervised Unsupervised Reinforcement Source: Futurama

116 Types of ML Algorithms Decision Trees Naïve Bayes Classifier Linear Regression Logistic Regression Support Vector Machines Neural Networks K-Means Clustering Ensemble Learning Source: Futurama

117 k-means Clustering Unsupervised learning Specify k (# of clusters) Algorithm finds centers Random restarts Source: Wikipedia

118 Decision Tree Classifier Supervised learning Tree of decisions Easy to understand Transparent Source: Wikipedia

119 Naïve Bayes Classifier Supervised learning Simple Bayesian classifier Independence assumption Relatively easy to understand Transparent

120 Neural Network Classifier Supervised learning Model of neurons in brain Layers of neurons Backpropagation Complex Not transparent Source: Wikipedia

121 Overfitting and Regularization Overfit too specialized Underfit too generalized Regularization techniques Early stopping Pruning (trees) Adding noise Parameter tuning Source: Wikipedia

122 The Machine Learning Process 1. Find a question 2. Prepare the data 3. Train the model Monitor the model Find a question Prepare the data 4. Evaluate the model 5. Deploy the model 6. Monitor the model Deploy the model Evaluate the model Train the model

123 Photos by Radomił Binek, Danielle Langlois, and Frank Mayfield

124 Code Demo

125 Lab 7 Machine Learning

126

127 R in Practice

128 How to Use R in Practice? 1. Deploying R to production 2. Best practices 3. Creating reproducible research

129 How to Deploy to Production Export charts (Rstudio) Create documents (Markdown) Create interactive reports (Shiny) Deploy to Server (R Server) Deploy to Cloud (Azure ML)

130 Code Demo

131

132 Start with a Question How many movies were released in each rating category from 2000 to 2015?

133 Use the Right Tool for the Job

134 Know Your Audience

135 Movies Create Clean Data Analyses 1500 Movies by Rating Category G PG PG-13 0 G PG Rating PG-13 R R

136 Avoid Biases and Information Distortions

137 Create Reproducible Research Replication is hallmark of science Big issue in science right now Allows other to verify findings Creates transparency Allows other to build upon work Source:

138 How Do We Create Reproducible Research? Provide raw data and code Script all analysis steps Use source control State all assumptions Use markdown Source:

139 Where to Go Next Coursera: Revolutions: Flowing Data: R-Blogger: R-Seek:

140 My Website Articles Presentations Source Code Videos Workshops

141 Conclusion

142 Conclusion 1. Introduction 2. Transforming Data 3. Descriptive Statistics 4. Data Visualization 5. Statistical Modeling 6. Handling Big Data 7. Machine Learning 8. R in Practice

143 Feedback Feedback is very important to me! One thing you liked? One thing I could improve?

144 Contact Info Matthew Renze Data Science Consultant Renze Consulting Website: Thank You! : )