Hal Varian, Google s Chief Economist The McKinsey Quarterly, Jan 2009

Transcription

1

2

3

4

5 The ability to take data to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it that s going to be a hugely important skill in the next decades, because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it. Hal Varian, Google s Chief Economist The McKinsey Quarterly, Jan 2009

6

7 Job Postings for Data Scientists

8 Top-paying Tech Skills Skill 2016 Change Skill 2016 Change Source: Dice Salary Survey 2017

9 SQL Excel Python R MySQL Python tools ggplot SQL Server Tableau JavaScript Matplotlib Java PostgreSQL Oracle D3 Homegrown Hive Spark Cloudera Visual Basic MongoDB Hadoop SAS C++ PowerPivot Scala SQLite C Pig RedShift Weka Hbase (EMR) Perl SPSS Teradata Share of Respondents 70% 60% Data Science Tools 50% 40% 30% 20% 10% 0% Tool: language, platform, analytics Source: O Reilly 2015 Data Science Salary Survey

10

11

12

13 Overview Introduction to R Working with Data Descriptive Statistics Data Visualization Beyond R and EDA

14 Introduction to R

15 What is R? Open source Language and environment Numerical and graphical analysis Cross platform

16 What is R? Active development Large user community Modular and extensible extensions and best of all

17 FREE

18 FREE

19 Source:

20

21

22

23 Code Demo

24 Working with Data

25 Working with Data

26 Working with Data

27 Working with Data

28 Working with Data

29 Working with Data

30 Working with Data Data munging Data wrangling Data cleaning Data cleansing

31 Loading Data in R

32 Loading Data in R CSV

33 Loading Data in R CSV XML

34 Loading Data in R CSV XML

35 Loading Data in R CSV XML

36 Cleaning Data

37 Cleaning Data Reshape data

38 Cleaning Data Reshape data Rename columns

39 Cleaning Data Reshape data Rename columns Convert data types

40 Cleaning Data Reshape data Rename columns Convert data types Ensure proper encoding

41 Cleaning Data Reshape data Rename columns Convert data types Ensure proper encoding Ensure internal consistency

42 Cleaning Data Reshape data Rename columns Convert data types Ensure proper encoding Ensure internal consistency Handle errors and outliers

43 Cleaning Data Reshape data Rename columns Convert data types Ensure proper encoding Ensure internal consistency Handle errors and outliers Handle missing values

44 Transforming Data

45 Transforming Data Select columns

46 Transforming Data Select columns Select rows

47 Transforming Data Select columns Select rows Group rows

48 Transforming Data Select columns Select rows Group rows Order rows

49 Transforming Data Select columns Select rows Group rows Order rows Merging data sets

50 Exporting Data File-based data Web-based data Databases Statistical data CSV XML

51 Advice for Working with Data Often difficult Time consuming TIP: Record all steps

52 Open Movies Database Title Year Rating Movies Runtime (minutes) Genre Critic Score Box Office The Whole Nine Yards 2000 R 98 Comedy 45% $57.3M Cirque du Soleil 2000 G 39 Family 45% $13.4M Gladiator 2000 R 155 Action 76% $187.3M Dinosaur 2000 PG 82 Family 65% $135.6M Big Momma's House 2000 PG Comedy 30% $0.5M

53

54

55 1. Column with wrong name 2. Rows with missing values 3. Runtime column has units 4. Revenue in multiple scales 5. Wrong file format

56 Code Demo

57

58 Descriptive Statistics

59 Descriptive Statistics Movie Runtime Statistic Value (minutes) Describe data Provides a summary aka: Summary statistics Minimum 38 1 st Quartile 93 Median 101 Mean rd Quartile 113 Maximum 219

60 Statistical Terms ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

61 Statistical Terms Observations ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

62 Statistical Terms ID Date Customer Product Quantity Observations Variables John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

63 Statistical Terms ID Date Customer Product Quantity Observations Variables Categorical variables John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

64 Statistical Terms ID Date Customer Product Quantity Observations Variables Categorical variables Numeric variables John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

65 Number of Variables Types of Analysis One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

66 Number of Variables Analyzing One Categorical Variable One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

67 Analyzing One Categorical Variable Frequency Movies by Genre Genre Frequency Percentage Action 612 9% Adventure 496 7% Animation 168 2% Comedy % Drama % Horror 269 4%

68 Analyzing One Categorical Variable Movies by Genre Genre Frequency Percentage Action 612 9% Frequency Proportion Adventure 496 7% Animation 168 2% Comedy % Drama % Horror 269 4%

69 Number of Variables Analyzing One Numeric Variable One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

70 Analyzing One Numeric Variable Central tendency Dispersion Shape

71 Number of Variables Analyzing Two Categorical Variables One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

72 Analyzing Two Categorical Variables Joint frequency Movies by Genre and Rating Genre G PG PG-13 R Total Action Adventure Animation Comedy Drama Family Total

73 Analyzing Two Categorical Variables Movies by Genre and Rating Genre G PG PG-13 R Total Joint frequency Contingency table Action Adventure Animation Comedy Drama Family Total

74 Analyzing Two Categorical Variables Movies by Genre and Rating Genre G PG PG-13 R Total Joint frequency Contingency table Marginal frequency Action Adventure Animation Comedy Drama Family Total

75 Analyzing Two Categorical Variables Movies by Genre and Rating Genre G PG PG-13 R Total Joint frequency Contingency table Marginal frequency Relative frequency Action Adventure Animation Comedy Drama Family Total

76 Number of Variables Analyzing Two Numeric Variables One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

77 Analyzing Two Numeric Variables Explanatory vs. outcome Covariance Correlation

78 Number of Variables Analyzing a Numeric Variable Grouped by a Categorical Variable One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

79 Analyzing a Numeric Variable Grouped by a Categorical Variable One categorical variable One numeric variable Aggregate measures

80 Number of Variables Analyzing Many Variables One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

81

82 Cowboys & Space Invaders: The Musical Extended Edition

83 Code Demo

84 Cowboys & Space Invaders: The Musical Extended Edition

85 Data Visualization

86 Data Visualization Visual data representation

87 Data Visualization Visual data representation Human pattern recognition

88 Data Visualization Visual data representation Human pattern recognition Map dimensions to visual

89 Data Visualization ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

90 Data Visualization ID Date Customer Product Quantity John Pizza John Soda Jill Salad Jill Milk Miko Pizza Miko Soda Sam Pizza Sam Milk 1

91 Number of Variables Types of Analysis One Categorical Variable One Numeric Variable Two Categorical Variables Categorical & Numeric Variable Two Numeric Variables Many Variables Type of Variable(s)

92 Cowboys & Space Invaders: The Musical Extended Edition

93 Code Demo

94 The Warlord of the Rings : An Unexpected Adventure Feature Length PG

95 Beyond R and EDA

96 This is just the tip of the iceberg! This is just the tip of the iceberg!

97 Advanced Data Analysis with R Cluster Analysis Statistical Modeling Dimensionality Reduction Analysis of Variance (ANOVA)

98 Source: Nathan Yau (

99 Machine Learning with R

100

101 Photos by Radomił Binek, Danielle Langlois, and Frank Mayfield

102 Code Demo

103

104 Where to Go Next R website: RStudio: Revolutions: Flowing Data: R-Blogger: R-Seek:

105 Data Science with R Exploratory Data Analysis with R Data Visualization with R (3-part) Data Science: The Big Picture

106

107 Feedback Very important to me! One thing you liked? One thing I could improve?

108 Conclusion

109 Conclusion Introduction to R Working with Data Descriptive statistics Data visualization Beyond R & EDA

110 Thank You! Matthew Renze Data Science Consultant Renze Consulting Website: