Tomaž Kaštrun. Data Science for Beginners

Transcription

1 Tomaž Kaštrun Data Science for Beginners

2 To all sponsors, thank you!

3 Thanks to all organizers! GetLatestVersion. it

4 About (2.0.1) BI Developer and data analyst SQL Server, SAS, R, Python, C#, SAP, SPSS 15years experience MSSQL, DEV, BI, DM Working at GEN-I Frequent community speaker Avid coffee drinker & Bicycle junkie

5 This talk was born out of frustration! Why?

6 faking it!

7 Can you answer? 1.Explain what regularization is and why it is useful. 2. Explain what precision and recall are. How do they relate to the ROC curve? 3.What is root cause analysis? 4.Are you familiar with pricing optimization, price elasticity, inventory management, competitive intelligence? Give examples. 5.What is statistical power? 6.Explain what resampling methods are and why they are useful. Also explain their limitations. 7.Is it better to have too many false positives, or too many false negatives? Explain. 8.What is selection bias, why is it important and how can you avoid it? 9.Give an example of how you would use experimental design to answer a question about user behavior. 10.How would you screen for outliers and what should you do if you find one? 11.How would you use either the extreme value theory, Monte Carlo simulations or mathematical statistics (or anything else) to correctly estimate the chance of a very rare event? 12.What is a recommendation engine? How does it work? 13.Explain what a false positive and a false negative are. Why is it important to differentiate these from each other? Source:

8 Can you answer? Source:

9 Developers and database people Statisticians Data science People from business

10 Developers and database people Statisticians People from business

11 Developers and database people Statisticians a.k.a. Data scientist People from business

12 What is data science? It s a buzz word!

13 Terms over time statistics Data mining Data science? ~ Examples: - Regression (Stats 101, SSAS, R+MSSQL) - Decision trees (Stats 101, SSAS, R+MSSQL) - Complexity reduction (Stats 101, SSAS, R+MSSQL) - Clustering (Stats 101, SSAS, R+MSSQL) 2010 ~

14 Interfaces over time statistics Data mining Data science ~ ~

15 Who is data scienctist? MacBook Statistician San Francisco

16 So who is a data scienctist? NOOooo!!!! It s a statistician!!! Source: Internet

17 Because what s next???? Internet killing doctors? It s like searching on internet for your symptoms instead of visiting a doctor!

18 Same goes for data science? Is the buzz data science killing statistics? and statisticians?

19 Why data scientist was born? because it was too damn hard to pronounce /ˌstætɪˈstɪʃ(ə)n/ STATISTICIAN

20 and because it s a sexy job According to a lot of articles published in 2015 and 2016 (thanks to decline of research journalism and copy/paste journalist) this was the top paid job, highly appreciated and wanted position. Well good morning! But we had these positions since the 60 s. They were called. f@%#! that word stasti statsi.something well, straccatella

21 and think about the movies.

22 at the end everyone would like to talk about. eventhough they have never seen.

23 at the very end data science is like teenage sex Everybody talks about it. Nobody really knows how to do it. Everybody thinks everyone else is doing it. So everyone claims they are doing it. In reality this looks like

24 I don t want to conclude, but. Having copy of your business logic and data in jammed into 1,048,576 rows by 16,384 column Data Science = Point & Click adventure game in Azure Machine Learning Copy-Paste-from-web advanced skills Huh Totally forgot about that?!

25 Let s do a quick example to support this problem Euclidian distances between friends

26 Let s do a quick example to support this problem Did he just say Eucl???? distance?

27 Let s do a quick example to support this problem dist(samplerestauratns, distance=euclidian)

28 Let s do a quick example to support this problem Simple distance from the center can gives us sensitivity of the ratings

29 Let s do a quick example to support this problem And so called sensitivity?

30 Think about. Data scientist use: Statisticians (and developers) use: dist(samplerestauratns, distance=euclidian) scale(samplerestaurants, center=t, scale=t)

31 Let s do a quick example to support this problem And doing some more goofying around.

32 Let s do a quick example to support this problem but not sure if algorithm preventing goofy pictures exists?

33 How to stop faking it? Start learning it Understand Test, test, test

34 Why you re not a data scientist?! Some Time series business intelligence stack doesn t make you a data scientist. Programming experience with Hadoop, R, Python, Octave, Matlib and Mathematica are data science tools. Tool skills alone don t give you data science cred. The 8-week course you took on Coursera or the Data Science boot camp you attended does not make you a data scientist Evangelizing Big data does not make you a data scientist Having degree in mathematics and statistics without field and applicative knowledge does not make you a data scientists Source:

35 Agenda for today 1) What we do in Data Science 2) Materials, tools and programs 3) Data science in business world

36 1) What we do in Data Science (part 1) Querying relational data Analyzing and visualizing the data Tasks for: Developers and Database people

37 1) What we do in Data Science (part 2) Understanding stastistics Exploring data with R / Python / Julia Understanding core data science concepts Understand Machine Learning Programming with R / Python to manipulate and model data Apply solution Tasks for: statisticians Tasks for: Business people

38 5 Core data science concetps? 1) is this weird? 2) is A better than B, respectively? 3) how much / many of this is needed? 4) this belong to group A? 5) what is next?

39 Think of. Algorithm as a cooking recipe

40 Think of. Your dataset as an ingredients

41 Think of. Your pans and pots as a computer

42 Think of. Statistician Data Scientist as a chef

43 Think of. Results as a finished dish

44 Think of. API as a prepared food

45 Is this weird spot the intruder? Anomaly detection

46 Is it Blue or is it Gold? Classification

47 What will be the temperature / stock? Regression

48 Belong to which group? Clustering

49 what is next? Reinforcement learning algorithms

50 2) Is your data ready for data science? 1) Is it relevant 2) Is data correlated 3) Is data distributed and accurate 4) Do I have enough data (variables, columns) 5) Unwanted correlations (multicolinearity, hyper )

51 2) Is your data ready for data science? Is it relevant

52 2) Is your data ready for data science? Is it related ( or non-empty)?

53 2) Is your data ready for data science? Is it accurate? Source:

54 2) Is your data ready for data science? Do we have enough? Sampling Number or observations vs. Number of variables Type of algorithm

55 3) Ask the right question? 1) Ask SMART 2) Ask in this way that includes target/predicted data 3) Formulate question based on data and algorithm

56 Model data and apply solution 1) Model data 2) Predict data 3) Apply solution

57 2) Materials and tools (Part 1) R consortium Books on statistics, statistical learning and machine learning Microsoft Books on line Microsoft Virtual academy Udemy, Packt,

58 2) Materials and tools (Part 2) R / Python / Julia / Excel Use Microsoft Azure ML Amazon Web service EC2 SQL Server BI stack Many vendors: SAS, IBM, Tibco, SAP, Tableau, Pentaho,Qlick, Microstrategy, Alteryx, etc.

59 3) Data Science in business world Loyalty program churn analysis Frau detection Out-of-stock prediction Customer classification Recommendation stuff Customer behaviour

60 Sources: Microsoft MVA Microsoft Data Science program ( Stats

61 R and SQL Server (SQL Server 2017 CTP 2.x)

62 The behind Architecture

63 Ecosystem RevoScaleR Package

64 R and T-SQL for predictive analytics EXECUTE = N'R',@script = N' library(e1071); irismodel <-naivebayes(iris_data[,1:4], iris_data[,5]); trained_model <- data.frame(payload = as.raw(serialize(irismodel, connection=null)));',@input_data_1 = N'select "Sepal.Length", "Sepal.Width","Petal.Length","Petal.Width","Species" from iris_data',@input_data_1_name = N'iris_data',@output_data_1_name = N'trained_model' WITH RESULT SETS ((model VARBINARY(MAX))); EXECUTE = N'R',@script = N'require("RevoScaleR"); irislinmod <- rxlinmod(sepal.length ~ Sepal.Width + Petal.Length + Petal.Width + Species, data = iris_rx_data); trained_model <- data.frame(payload = as.raw(serialize(irislinmod, connection=null)));',@input_data_1 = N'select "Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width", "Species" from iris_rx_data',@input_data_1_name = N'iris_rx_data',@output_data_1_name = N'trained_model' WITH result SETS ((model VARBINARY(MAX)));

65 Thanks! and learn Statistics!!

66 #sqlsat675 THANKS! Q&A