Think & Work like a Data Scientist with SQL 2016 & R DR. SUBRAMANI PARAMASIVAM (MANI)

Transcription

1 Think & Work like a Data Scientist with SQL 2016 & R DR. SUBRAMANI PARAMASIVAM (MANI)

2 About the Speaker Dr. SubraMANI Paramasivam PhD., MCT, MCSE, MCITP, MCP, MCTS, MCSA CEO, Principal Consultant & DAGEOP (UK)

3 Data Scientist Tried R in SQL 2016 Data Analyst Data Geek Any plans?

4 CONTENTS Intro, 5 Demo, 15 About Data Science, 15 Microsoft Tools & R, 10 Data Science Process, 15

5

6 DATA SCIENCE Data science is the exploration and quantitative analysis of all available structured and unstructured data to develop understanding, extract knowledge, and formulate actionable results. Evolving subject, no single definition and requires a range of skills mathematics statistics information science computer science signal processing probability models machine learning statistical learning data mining database data engineering pattern recognition & learning predictive analytics uncertainty modeling data warehousing data compression computer programming artificial intelligence high performance computing visualization

7 Data Science History The Future of Data Analysis Concise Survey of Computer Methods 1 st KDD Workshop Cover story Database Marketing Conference on Knowledge Discovery & Data Mining (KDD) data science, classification & related methods From Data Mining to Knowledge Discovery in Databases Statistics to be renamed to data science & statisticians to be renamed data scientists convert data into information and knowledge. Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics Statistical Modelling: The Two Cultures Launch of Data Science Journal Competing on Analytics The Research Center for Datalogy and Data Science The Skills, Role & Career Structure of Data Scientists & Curators: Assessment of Current Practice & Future Needs tells the McKinsey Quarterly: Sexy job in the next ten years will be statisticians The Revolution in Astronomy Education: Data Science for the Masses Those who can model, munge and visually communicate data call us statisticians or data geeks are a hot commodity Created the data scientist group on LinkedIn as a companion to his website, datascientists.com Data, Data Everywhere Software programmer + statiscian + storyteller to extract the gold hidden under mountains of data What is Data Science? The Data Science Venn Diagram Data Science, Moore s Las and Moneyball Building Data Science Teams Data Scientist: The Sexiest Job of the 21st Century

8 DATA SCIENTIST data and analytical ability find, merge and interpret rich data sources manage large amounts of data no HW, SW, Bandwidth constraints ensure consistency of datasets create visualizations build mathematical models communicate the data insights/findings produce answers in days rather than months work by exploratory analysis & rapid iteration to present results with dashboards rather than papers/reports

9

10 Data Scientist Analytics Analytics More Description Descriptive Diagnostic Predictive Prescriptive What happened? Why did it happen? Predicting, what is likely to happen. Recommending, what to do next How much sales we made last Christmas compared to previous? Why last Christmas sales was low compared to previous? How many additional resource required to improve the sales for next Christmas? Hire 5 resources for each branch in Southern states and 3 in northern states.

11 How to become a Data Scientist?

12

13 Data Science Process?

14 Knowledge Discovery in Database KDD Process Courtesy: AI Magazine Article: From Data Mining to Knowledge Discovery in Databases

15 CRISP Process- Cross Industry Standard Process for Data Mining Business Understanding Data Understanding Data Preparation Modeling Evaluation Deployment Identify project objectives Collect and review data Select and cleanse data Manipulate data and draw conclusions Evaluate model and conclusions Apply conclusions to business Come up with new questions. Collect the data Explore the data Model the data Communicate and visualize

16 Data Science Process KDD (Knowledge Discovery & Data), CRISP-DM, Big Data => relationship to Data Mining & Machine Learning. Machine Learning (Training set Vs Test set)

17 Classification Regression Supervised learning Clustering Recommender systems (Yes/No and classified as 1,-1) How many (Persons = X) will die because of smoking next year (label = Y)? Classification & Regression are supervised learning problems Trying to evaluate the quality of clustering as no ground truth and much harder to evaluate Recommend any patient who is admitted to do the blood test if he/she had any similar symptoms of most dying patients Did any one die (Yes/No) How many times (Y) will a patient (X) turn up to the hospital next year? Balance between accuracy and simplicity Unsupervised (training set has no ground truth labels to learn from) learning problem. Eg., Images with no labels Please Don t Smoke The data is out and the data is supervised problem for the algorithm

18 Statistical Tools

19 Open Source Community Support Scalability Usability Flexibility Good visualization Highly interactive Highly comfortable IDEs Create custom modules with SQL, R and Python to feed ML R Vs Python

20 R Vs Python

21 More tips to think like a Data Scientist Discover new questions and find answers on top of the hypothesis. Your real tool is the data and not any application It is very important to trust your data Identify and understand the limitations of any tools

22 More tips to think like a Data Scientist Influence customer behaviours Identify key business initiatives Creating Actionable Scores Analytics (Descriptive, Diagnostic, Predictive, Prescriptive)

23 Microsoft Analytics Box Revolution R Azure Data Warehouse Azure Data Factory Azure Machine Learning Azure Data Lake HDInsight Power BI Document DB SSAS (MultiDimensional, Tabular) SSAS Data Mining StreamInsight SS Integration Services Excel - Power Pivot/Query/Map/View APS - Analytics Platform System (PDW)

24 Data Scientist Tools in SQL 2016 Advanced Analytics Extensions To call R Runtime and Execute R code Python/JSON support in future To develop and test R Revolution R Open Main Components Revolution R Enterprise 7.5 Enhanced packages to support Performance In-database analytics

25 SQL Server R- Key scenarios Using SQL Server Via inline code within system stored procedures Embed in any applications Can be used in live production environment Using the R Tools Any R Development tools and in-database to SQL Server Multi-thread, Multi-core, Multi process computation is now possible Enables to work with large datasets

26 DEMO

27 Q & A

28