intro to data science Module 1

Transcription

1 intro to data science Module 1

2 what is data science?

3 [Data science is] The sexiest job of the 21st century Harvard Business Review (2012)

4 The ability to take data to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it's going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. Hal Varian

5 what is data science? v=a8abc9sgvum [Data science is] Ones attempt to work with data to find answers to questions that they are exploring The process of formulating quantitative questions that can be answered with data

6 what is data science? Historically, organizations have made decisions based on qualitative analysis or expert opinions. Data science shifts this traditional thinking away from qualitative analysis in making business decisions to purely quantitative one based on data

7 why do we need data science?

8 why data science? Data contains (mostly) hidden information of great value Over the last 10 years business owners, medical researchers, marketing and advertising specialists, government organizations, designers, retailers, sports teams, etc. have all caught on to the value of the data they collect They ve all realized how they can make much better business decisions using qualitative reasoning based on data in place of quantitative reasoning based on solely expert opinion

9 data science in the real world Data science is being applied in almost every industry imaginable. Here are a few industry/

10 how is data science useful? How about the fashion/design/marketing industries?? There s even an entire conference dedicated to it! kddfashion2016.mybluemix.net/ LA Fashion Industry: A Data Science Perspective: amuletanalytics.files.wordpress.com/2016/09/amulet-analytices-la-fashionindustry.pdf

11 how is data science useful? Question: How can you build a competitive baseball team with only a small budget? Reasoning: Collective wisdom of baseball insiders (managers, coaches, players, etc) is often subjective and therefore flawed. Data considered: Batting average, runs scored, etc. Impact: A field of data analytics is invented called Sabermetrics. It uses data collected about baseball players to try and select the best possible team based on any given budget Boston Red Sox implemented Sabermetrics and win world series Sources: MONEYBALL (SPORTS) (See section Moneyball Recruiting )

12 how is data science useful? Question: How can Netflix improve its movie recommendations for its users? Goal: Improve predictions of user ratings for any given movie. Data: Previous user ratings NETFLIX PRIZE (TECHNOLOGY) Impact: Small team of data scientists devised a method for improving movie recommendations by over 10% (then awarded a $1 million prize) Source:

13 how is data science useful? FASHION METRIC (FASHION DESIGN AND APPAREL) Problem: Apparel sizing is challenging in an online retail environment Goal: Make it easy for shoppers to buy better fitting clothes both instore and online Impact: In 2015 we have made our core technology and solutions available to empower other brands and retailers with more data about their customers than ever before. Source(s):

14 data in our modern world

15 an abundance of data and opportunity Humans collectively create about 1.8 zettabytes of data per year. That is 1,800,000,000,000,000GB (In comparison, your hard drive ~ 500GB - 1,000GB) The amount of data we consume/create each second, daily, monthly, is staggering: Much of this data is what we call unstructured data, meaning that it is grossly unorganized in any way making it mostly unusable to the untrained eye Documents, pictures, tweets, , customer reviews, historical designs trends, Facebook posts, text on webpages

16 data in our modern world THE FOUR V S Every day we are creating data at high velocity (how fast data is being created) volume (how much data is there?) variety (different types of data) veracity (is the data even consistent?) creates unique challenges for data scientists. How can they manage all of these features of Referred to as the Big Data problem Huge rewards for companies and data scientists alike to be able to manage, store, and extract valuable insights from this data!

17 data in our modern world Velocity Overload and saturation of openly and freely available data Volume massive data (500k users, 20k movies, 100m ratings) curse of dimensionality (very high-dimensional problem) Variety So much structured and unstructured data! Veractiy THE ISSUES THAT EACH V PRESENTS Lack of trust in the reliability of information that is presented in the media, due to skepticism and lack of transparency missing data (99% of data missing; not missing at random)

18 types of data UNSTRUCTURED DATA Text off of a web page, Weblogs, Pictures, Restaurant reviews, comments, sales data, Tweets, Facebook posts Ex. Say you are studying how many people in the the US believe in the inevitable occurrence of a zombie apocalypse. There is a ton of great information (in an unstructured state) freely available on social media and websites A data scientist must figure out a way to take the data from all of these sources, and structure it into a useable format

19 types of data STRUCTURED DATA Information in databases or spreadsheets that is categorized making the data you need easily identifiable Data properly categorized and easy to search and retrieve Easier to answer questions like How many people live in the city of Tenderloin?

20 bringing value to any dataset Often times the toughest and most valuable part of a data science study is to bring structure to unstructured data (unstructured data is almost useless in that state)

21 what does a data scientist do?

22 what does a data scientist do? v=vowxaedh1uk Data Scientists help organizations find hidden insight and value in the massive amounts of data they collect to (just to name a few): Help to gain competitive advantages Prevent supply chain problems Predict when maintenance issues may occur

23 what skills does a data scientist need?

24 what skills does a data scientist need? NOT in this class this class this class this class (and other classes you ve taken)

25 what skills does a data scientist need? DATA SCIENTISTS HAVE A UNIQUE SKILLSET Computer programming find and interpret data sources manage large amounts of data and combine data sources ensure consistency of datasets While any computer programming is outside the scope of this course, your experience and skills with Excel will come in handy in homework assignments and class project. Source: career-and-technical-programs/information-technology/ computer-programming/index

26 what skills does a data Mathematics and statistics scientist need? DATA SCIENTISTS HAVE A UNIQUE SKILLSET Perform visual analysis of data Perform statistical analysis on data Create mathematical models using the data Source:

27 what skills does a data scientist need? DATA SCIENTISTS HAVE A UNIQUE SKILLSET Artistic, design, storytelling To present and communicate the data insights/findings in an interesting story Create visualizations to aid in understanding of data Source:

28 what skills does a data scientist need? DATA SCIENTISTS HAVE A UNIQUE SKILL SET Domain expertise (expertise or knowledge about a subject the subject you are studying) Fashion design? Fashion Forecasting? Marketing? Robotics? Sports? Source: sports-psychology/see-how-its-done-6- lessons-on-visualization

29 what skills does a data scientist need? With these powers combined, data scientists uncover powerful insights for businesses from their data!

30 data science studies

31 data science studies How do data scientists conduct data science studies? DATA SCIENCE PROCESS (DSP) is structured approach to conducting a data science study

32 data science process The Data Science Process (DSP) consists of inspecting, cleaning, transforming, and modeling data with the goal of discovering useful information, suggesting conclusions, and supporting decision-making It s the scientific process that goes into discovering knowledge from data Can I predict X? Can I improve my guess about X?

33 data science process COLLECT MORE DATA (IF YOU NEED IT) ASK AN INTERESTING QUESTION COLLECT DATA EXPLORATORY DATA ANALYSIS UNSTRUCTURED > STRUCTURED EXPLORATORY GRAPHS EXPLORATORY STATISTICS COMMUNICATE AND VISUALIZE THE RESULTS MATHEMATICAL MODELING CREATE A DATA PRODUCT

34 step 0: ask a question that interests you

35 0 - ask a question that interests you Typically all scientific inquiries begin with an interesting question What is something you ve always been interested in? Can I predict what fashions trends will be in next Fall? Is there a way I can better predict California Super lotto numbers? Are there some external factors which influence the outcome of lottery numbers? How do celebrities influence consumer fashion trends?

36 0 - ask a question that interests you WHAT TYPE OF QUESTION ARE YOU ASKING? 1.Descriptive (correlation, observational) - Is your goal to describe a phenomenon? EX. Why do UFO sightings increase during the summer months in the United States? 2.Exploratory - Is your focus solely to gain insights and familiarity for later investigation? (Typical of research problems are in a preliminary stage of investigation) EX. Performing a study of all California lottery outcomes over the last 40 years to identify sources of biased outcomes 3. Inferential - Is your goal to make inferences about an entire population? EX. Voter turnout in the US will increase by 12% in the next presidential election

37 0 - ask a question that interests you WHAT TYPE OF QUESTION ARE YOU ASKING? 4.Causal - Is your goal to examine the cause and effect between two variables? EX. How does weather in a specific geographic location affect average male Body Mass Index (BMI)? 5. Predictive - Does your question aim to predict some future outcome? EX. Does student involvement in constructive learning activities along with a rewards system prevent delinquency? 6.Mechanistic - Is your aim to understand the mechanism at which some process is carried out? EX. How does a disease do damage to a body?

38 0 - ask a question that interests you DATA SCIENTISTS HAVE A UNIQUE SKILL SET Spend a significant amount of time thinking about this Specifying and refining the question over time will guide the type of data you need and the type of analysis you will do Figuring out what type of question you're asking and what exactly the question is really influential.

39 step 1: collect data related to your question (then structure it)

40 data collection The collection, cleaning, and sampling of a dataset in order to acquire an informative, structured, and manageable data set

41 how to get the data you need for your study Data scraping from websites (aka data wrangling) Open data repositories online Polling, surveying, and sampling Use your internal corporate, agency, or business data

42 what is data collection? Data scraping from websites (aka data wrangling) Open data repositories online Polling, surveying, and sampling Use your internal corporate, agency, or business data

43 data collection source and techniques DATA SCRAPING (OR WEB SCRAPING) Techniques (usually involving computer programming) used to extract information from websites (unstructured data) and transform it into a structured format like a spreadsheet or database making it usable for further analysis COMPUTER PROGRAMS SCRAPE DATA FROM WEBSITES (UNSTRUCTURED DATA) FORMATS DATA INTO STRUCTURED DATA NOW WE CAN USE IT FOR FURTHER ANALYSIS!

44 data collection source and techniques TONS OF FREE AND OPEN DATASETS AVAILABLE ONLINE Open data project form the US government: Los Angeles open data: American Census Data: Repository of data from public and privately funded clinical trials: Community curated growing list of open datasets Sets of free and interesting datasets

45 data collection source and techniques SAMPLING, POLLING, AND INTERNAL DATA Get out and survey/poll then sample! Businesses, agencies, etc collect massive amounts of data about their sales, employees, etc.

46 data cleansing Data cleansing (or data Munging) is the process of transforming unstructured data into structured data Once data is collected, computer programming, sampling, statistical techniques are applied to the dataset to cleanse it and ensure it s reliability. Removes/identifies bias in the dataset Note that data cleansing techniques are outside the scope of this course and that the datasets you will use for your studies will be clean enough for you to conduct your study. You may need to find other complimentary datasets if those provided do not have all the information you need to properly analyze your thesis Once you are comfortable that you have a clean and reliable dataset you are ready to move on to the next phase of the data science process.

47 exploratory data analysis

48 dsp - exploratory data analysis The exploration of a data set using graphical, mathematical, and statistical techniques to generate hypotheses and intuition about the data Exploratory Graphs Histograms, scatter plots, Tables, charts Summary statistics Measures of central tendency and dispersion, probabilities

49 mathematical modeling

50 dsp - mathematical modeling A representation of a system (dataset) in mathematical terms In data science we aim to create a mathematical model that is good enough to be useful in some way (make predictions) Model weather patterns, make predictions about outcomes of elections

51 data visualization and presentation

52 dsp - data visualization and presentation The communication of results through modern techniques in visualization, stories, and data products Data products are tools and services that use data as the basis for their output Goal is to empower others with tools to make their own informed decisions based on quantitative analysis Examples Google Search Netflix s Recommendation system

53 defining success in a data science study SUCCESSFUL DATA SCIENCE STUDIES ARE CHARACTERIZED BY ONE OR MORE OF THE FOLLOWING New knowledge about the subject you are studying is created Creating a decent usable model Decisions or policies are made based on this model or the outcome of the experiment A data product is created that people can use You learn that the data can t answer the question you came up with

54 course structure and goal This course is broken down into the four main areas of a data science study. We ll explore each one individually throughout the course 1.Specifying questions (Module 1) 2.Data collection (Module 1) 2.Exploratory data analysis (Modules 2-5) 3.Visualizing and presenting results (Module 6) 4.Mathematical Modeling (Modules 7-8)

55 course goal Students will create new insights in the world from a dataset using the Data Science Process!

56 takeaways Tons of valuable insights hidden in vast seas of data. Huge interest and excitement in the business world to find them and use them to gain competitive advantages or increased revenues The Data Science Process is a structured approach to extracting useful insight from this data Data collection. No computer programming in this course but excel will best your BFF.