MATH36032 Problem Solving by Computer. Data Science

Transcription

1 MATH36032 Problem Solving by Computer Data Science

2 NO. of jobs on jobsite NO. of Jobs MATLAB Data Data Science 0 Jan 2016 Jul 2016 Jan

3 What is Data Science? (from Wiki) an interdisciplinary field about processes and systems to extract knowledge or insights from data in various forms, either structured or unstructured, which is a continuation of some of the data analysis fields such as statistics, data mining, and predictive analytics,...

4 What is Data Science? Math & Stat Knowledge: calculus, statistics/probability, linear algebra Hacking skills: number/text manipulation and (vectorized) manipulation, algorithmic thinking,... Substantive expertise/domain knowledge: knowledge related to specific facts

5 Data Science/Big Data: why now?

6 Data Science/Big Data: why now? We are generating more data before

7 Data Science/Big Data: why now? We are generating more data before Technology in data collection and storage are improving

8 Applications: Social media and search engine Personalised webpage How does Amazon know which items to recommend?

9 Applications: Retailers Personalised promotion offers Who should get what kind of offer?

10 Applications: Credit Card Fraud Detection Do these transactions look normal?

11 Applications: Credit Card Fraud Detection Do these transactions look normal? This is my credit card statement. The transactions are made within two hours after I lost the card in Montreal in the summer of 2015.

12 More applications Insurance/ Actuarial Science: how much do you charge your customer Weather/climate forecasting: long term prediction Finance: better prediction of the stock prices.

13 Big data leaked and generated Is the 2.6 terabytes Panama Papers big data? How about the 1 billion accounts leaked from Yahoo s database?

14 Big data leaked and generated Is the 2.6 terabytes Panama Papers big data? How about the 1 billion accounts leaked from Yahoo s database? Data generated daily More than 10 terabytes for most national meteorological center More than 500 terabytes of data processed by Facebook More than 20 petabytes ( bytes) handled by Google

15 Three V s of Big Data Volume: large quantity of data, big size of datasets Variety: many different types and forms of data, e.g. transactional from ATMs, social media site, s, demographics data, tracking data from cell phones, etc. Velocity: data that is coming in at a very fast pace

16 Three V s of Big Data Volume: large quantity of data, big size of datasets Variety: many different types and forms of data, e.g. transactional from ATMs, social media site, s, demographics data, tracking data from cell phones, etc. Velocity: data that is coming in at a very fast pace Require many new softwares/technologies: The support for big data sets is extended to all major technologies in MATLAB (mapreduce, datastore and other toolboxes)

17 Big Data Landscape

18 The first data science application? Kepler s three laws of planetary motion

19 Make a TV show using data?

20 TV made by Amazon and Netflix using big data

21 TV made by Amazon and Netflix using big data Alpha House was not as successful as expected. Netflix also had open competitions for best algorithms to predict user ratings for films (discontinued now for privacy and other reasons).

22 Google flu trends Google made a big splash in the news in 2008

23 Google flu trends Google made a big splash in the news in 2008 and five years later

24 The dark side of data science Data, if used in the right way, can greatly facilitate our life (like the concept of smart cities), but...

25 The dark side of data science Data, if used in the right way, can greatly facilitate our life (like the concept of smart cities), but... Privacy: how the data collected from you are being used?

26 The dark side of data science Data, if used in the right way, can greatly facilitate our life (like the concept of smart cities), but... Privacy: how the data collected from you are being used? Biased data: polls before Brexit or US presidential election in 2016

27 The dark side of data science Data, if used in the right way, can greatly facilitate our life (like the concept of smart cities), but... Privacy: how the data collected from you are being used? Biased data: polls before Brexit or US presidential election in 2016 Biased interpretation:

28 The dark side of data science Data, if used in the right way, can greatly facilitate our life (like the concept of smart cities), but... Privacy: how the data collected from you are being used? Biased data: polls before Brexit or US presidential election in 2016 Biased interpretation: Data shows that people who shop at Waitrose have longer life span than those at Aldi and Asda.

29 Data Science Workflow (in business)

30 Data Science using MATLAB MATLAB is not the best tool for data science. Certain tasks like text processing is better done with Python or R. More tools and data types are introduced in MATLAB for the past few years, mainly to cope with the increased need in data science.

31 MATLAB Data Type 2 We have already seen these (type whos in command window): Numerical Types: double (double precision floating number), uint8 (images), int32, int64,... Symbolic: Defined by syms Logical: true,false Characters and strings: A = string Function as in integral(@(x) sin(x),...) New data types introduced recently : structures (struct), cell arrays (cell), Time Series (timeseries from 2006b), Table (table from 2013b), Categorical Arrays (categorical from 2013b), Date and Time (datetime from 2015b), data-types.html

32 The plan for the rest of the semester In Week 8 (Friday) Review (and introduce) a few new data structures (mainly character strings) Read (and write) different data formats (csv, excel, image,...) Specific topics: Random simulation (week 9) Regression and classification (week 10) Dimension reduction/low rank approximation (week 10) Google Pagerank (week 11) Other related topics (if time permits, week 11)