Dr. SubraMANI Paramasivam. Think & Work like a Data Scientist with SQL 2016 & R

Transcription

1 Dr. SubraMANI Paramasivam Think & Work like a Data Scientist with SQL 2016 & R

2 About the Speaker Group Leader Dr. SubraMANI Paramasivam PhD., MVP, MCT, MCSE (x2), MCITP (x2), MCP, MCTS (x3), MCSA CEO, Principal Solutions DAGEOP (UK)

3 CONTENTS Intro, 5 Demo, 15 About Data Science, 15 Microsoft Tools & R, 10 Data Science Process, 15

4 DATA SCIENCE Data science is the exploration and quantitative analysis of all available structured and unstructured data to develop understanding, extract knowledge, and formulate actionable results. Evolving subject, no single definition and requires a range of skills mathematics statistics information science computer science signal processing probability models machine learning statistical learning data mining database data engineering pattern recognition & learning predictive analytics uncertainty modeling data warehousing data compression computer programming artificial intelligence high performance computing visualization

5 Data Science History The Future of Data Analysis Concise Survey of Computer Methods 1 st KDD Workshop Cover story Database Marketing Conference on Knowledge Discovery & Data Mining (KDD) data science, classification & related methods From Data Mining to Knowledge Discovery in Databases Statistics to be renamed to data science & statisticians to be renamed data scientists convert data into information and knowledge. Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics Statistical Modelling: The Two Cultures Launch of Data Science Journal Competing on Analytics The Research Center for Datalogy and Data Science The Skills, Role & Career Structure of Data Scientists & Curators: Assessment of Current Practice & Future Needs tells the McKinsey Quarterly: Sexy job in the next ten years will be statisticians The Revolution in Astronomy Education: Data Science for the Masses Those who can model, munge and visually communicate data call us statisticians or data geeks are a hot commodity Created the data scientist group on LinkedIn as a companion to his website, datascientists.com Data, Data Everywhere Software programmer + statiscian + storyteller to extract the gold hidden under mountains of data What is Data Science? The Data Science Venn Diagram Data Science, Moore s Las and Moneyball Building Data Science Teams Data Scientist: The Sexiest Job of the 21st Century

6 DATA SCIENTIST data and analytical ability find, merge and interpret rich data sources manage large amounts of data no HW, SW, Bandwidth constraints ensure consistency of datasets create visualizations build mathematical models communicate the data insights/findings produce answers in days rather than months work by exploratory analysis & rapid iteration to present results with dashboards rather than papers/reports

7 Data Scientist Analytics Analytics More Description Descriptive Diagnostic Predictive Prescriptive What happened? Why did it happen? Predicting, what is likely to happen. Recommending, what to do next How much sales we made last Christmas compared to previous? Why last Christmas sales was low compared to previous? How many additional resources required to improve the sales for next Christmas? Hire 5 resources for each branch in Southern states and 3 in northern states.

8 Source: Dr. Nick Tatonetti (Columbia University Medical Center) Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Proportion Sex Female Male Birth Month

9 How to become a Data Scientist?

10 Credits to Swami Chandrasekaran

11 Data Science Process?

12 Knowledge Discovery in Database KDD Process Courtesy: AI Magazine Article: From Data Mining to Knowledge Discovery in Databases

13 CRISP Process- Cross Industry Standard Process for Data Mining Business Understanding Data Understanding Data Preparation Modeling Evaluation Deployment Identify project objectives Collect and review data Select and cleanse data Manipulate data and draw conclusions Evaluate model and conclusions Apply conclusions to business Come up with new questions. Collect the data Explore the data Model the data Communicate and visualize

14 Data Science Process Classification Regression Supervised learning Clustering Recommender systems (Yes/No and classified as 1,-1) How many (Persons = X) will die because of smoking next year (label = Y)? Classification & Regression are supervised learning problems Trying to evaluate the quality of clustering as no ground truth and much harder to evaluate Recommend any patient who is admitted to do the blood test if he/she had any similar symptoms of most dying patients Did any one die (Yes/No) How many times (Y) will a patient (X) turn up to the hospital next year? Balance between accuracy and simplicity Unsupervised (training set has no ground truth labels to learn from) learning problem. Eg., Images with no labels Please Don t Smoke The data is out and the data is supervised problem for the algorithm

15 Statistical Tools

16 R Vs Python Open Source Community Support Scalability Usability Flexibility Good visualization Highly interactive Highly comfortable IDEs Create custom modules with SQL, R and Python to feed ML

17 R Vs Python

18 R Vs Python

19 More tips to think like a Data Scientist Discover new questions and find answers on top of the hypothesis. Your real tool is the data and not any application It is very important to trust your data Identify and understand the limitations of any tools

20 More tips to think like a Data Scientist Influence customer behaviours Identify key business initiatives Creating Actionable Scores Analytics (Descriptive, Diagnostic, Predictive, Prescriptive)

21 Data Warehouse Machine Learning Document DB Tabular Power Pivot Data Mining Data Factory Microsoft Analytics Box Multidim ensional SSAS Polybase Data Lake SS Integration Services HDInsight Power BI APS Analytics Platform System Stream Analytics Table Storage

22 Data Scientist Tools in SQL 2016 Advanced Analytics Extensions To call R Runtime and Execute R code Python Support in SQL Server 2017 To develop and test R Revolution R Open Main Components Revolution R Enterprise Enhanced packages to support Performance In-database analytics

23 R Components in SQL Server Launchpad BxlServer SQL Satellite RLauncher.dll Binary Exchange Language. Communication between SQL Server & R runtime. Extensibility API BxlServer uses SQL Satellite to communicate with SQL Server Custom data format for fast data transfer

24 R Components in SQL Server R scripts executed from SQL Server (in-database) R scripts executed from a remote client

25 SQL Server 2017

26

27

28 DEMO

29 Q & A

30