INTRODUCTION TO DATA SCIENCE

Transcription

1 DATA11001 INTRODUCTION TO DATA SCIENCE EPISODE 2: MINIPROJECT, ARRAY DATA, STORAGE FORMATS & TRANSFORMATIONS

2 TODAY S MENU 1. M I N I P R O J E C T S 2. A R R AY D ATA 3. D ATA T R A N S F O R M AT I O N S 4. F I LT E R I N G A N D I M P U TAT I O N

3 LOGISTICS You should start by forming a team of 3 students If you have trouble, ask around on Piazza Deadline to register groups is this week's Wednesday, September 12th Deadline to register project topics (to get mentoring), next week's Wednesday September 19th

4 MINIPROJECTS Idea is to come up with an idea of a data science solution: starting point A (need, data, possibilities) aim B (added value! not just a technical solution) you need to come up with: + both A and B, as well as + the solution to get from A to B Not good ideas: take clean data and a given task (e.g., prediction) and apply a machine learning method take data and visualize it without any particular aim or goal copy an existing data science project => try to be creative!

5 EXAMPLES These are primarily to give you an idea of suitable topics IT IS NOT ADVISABLE TO ACTUALLY CHOOSE ONE OF THESE Instead, they are just examples to give you the flavour Elements of the project (will be evaluated based on these): Data: sources, wrangling, management Data analysis: statistics, machine learning Communication of results: summarization & visualization Operationalization: added value, end-user point of view

6 EXAMPLE #1: HOW TO BE POPULAR ON SOCIAL MEDIA 1. Create a system that recommends ways to boost social media popularity. 2. Instagram/twitter/facebook data from open APIs. Challenges: extracting data following accepted policies. Finding other relevant data sources. Can involve big data problems. 3. Predicting popularity using machine learning tools. In case of text content, e.g., word clouds, clustering, regression. 4. For example, visualization of the data so that a user user can identify good strategies: e.g., an interactive hashtag exploration tool that highlights common choices based on co-occurrence. 5. The more popularity, the better the system. Usability also important (considering target group, e.g., teenagers wouldn't appreciate boring statistical graphics).

7 EXAMPLE #2: OK BUT FIRST COFFEE 1. Sometimes the coffee maker in Gurula (student space) is out of coffee just when you need it the most. It's easy enough to make more, but you'd like to know beforehand. 2. Create your own data by installing a webcam (or using one that may already be there) pointing at the coffee maker. Be sure not to violate anyone's privacy and be careful about controlling access to the camere feed. Alternative: use user input. 3. Try using computer vision to predict the amount of coffee left. 4. Share information through an API or an app (possibly a simple visualization, or just the number of cups left). 5. No coffee = bad, coffee = good. Also, possibly recommend how many cups to brew based on expected consumption, or predict peaks in demand.

8 EXAMPLE #3: RENT-A-FLAT 1. The demand for rental flats in Helsinki is leading to high, and increasing rental prices. However, the rents per area vary greatly. Which areas are underrated (and "underrented")? 2. Helsinki open Other possible data sources can include info about important services (bars, computer stores,...) including info about their popularity. 3. Summarization of data to generalize to future rents. 4. Visualization of GIS data, possibly over time. In addition to rental prices, the visualization can display, e.g., crime statistics, availability of services, "hipster index" 5. Information that supports individual users (with different preferences) in finding a cheap flat. Could also be used by city officials to support planning.

9 EXAMPLE #4: MONKIE GOES TO HOLLYWOOD 1. Propose 5 blockbuster movie ideas 2. IMDB, sales figures, reviews, Find the most profitable combinations of the given data (e.g. combination of genre-director-lead actor, etc) 4. Visualization of sales data over movie attributes (and combinations of them), presentation of the key selling points of your proposed movies 5. Incorporate your findings into fresh movie ideas (be creative). Convince the money guys as to why choose your proposed movies, over the proposals of your competitors.

10 EXAMPLE #5: CLIMATE CHANGE 1. Study of meteorological and other data to let us see what the world will be like in the future 2. Plenty of open climate data available on temperature changes throughout time. Consequences: water-level rise, agricultural outcomes, weather, everyday life 3. Predict consequences given climate forecasts (under various policy implications) 4. Visualize your results to support your final statement. For example: illustrated scenes or maps about life in various cities with water levels rising 5. Make it feel tangible but based on science (not too cheesy)

11 EXAMPLE #6: SHOULD I BE WORRIED ABOUT IT? 1. Consider various risks: meteorite-wise (NASA), lightnings, epidemics, cancer, car accidents, NASA, climate data, global health data, Predict risks in different areas (GIS). Compare the magnitudes of various risks 4. Remember: risk x damage = expected loss 5. Display risks to alert or pacify the public. Guide for travelers,...

12 PITCHING AND DELIVERABLE Pitch: about 3-5 minute pitch about the project others can give feedback (online) each student will be assigned to submit written feedback for two projects: pay close attention to their pitches three parallel sessions, about 15 teams per session time & place TBA individual projects: similar but a bit later Deliverables: 1. a web/mobile application, visualization, or a static output: doesn't have to be fancy! 2. in addition, user instructions, a blog post, or a Youtube video, etc, to help users (keep the target audience in mind!) 3. a report (for us): - explaining what you did and how, max 5 pages + appendix

13 IMPORTANT DATES, GRADING Miniproject: Sept 12 read instructions & form groups Sept 19 register group and topic proposals Week 41 project pitching sessions Week 42 project delivery Week 43 submit feedback on two (2) other projects (assigned automatically) Grading: exercises 50% + miniproject 50%

14 OR, ALTERNATIVELY: As was stated in the beginning, you can also do an individual project + exam In this case, grading will be based on exercise points (30%), miniproject (35%), exam (35%), or miniproject (50%), exam (50%)

15 PRACTICALITIES We'll be using peergrade.io for handling the project Register yourself (every student) and then your group (one registration per group) Next week, register your project topic on peergrade We'll give quick feedback to adjust but you can keep working -- very unlikely that we'd reject the whole idea

16 HOW TO GET STARTED Instructions on github But I don't know enough python... Hey! Don't worry. You can also use R, Java,... Additional hints: Don't be too ambitious: the workload per person should be similar to that of the weekly exercises x 3 (50% of the work, 6 weeks) + doesn't have to be the most advanced machine learning on big data with a fancy interface: linear regression on reasonable size data and a working interface is what people normally use anyway! Fail fast: + make prototype solutions and have a contingency plan ("plan B") in case there's a problem + even in the worst case scenario where your project would fail to produce a concrete output, we can evaluate based on the idea and the implemented parts, so it wouldn't be the end of the world

17 TODAY S MENU 1. M I N I P R O J E C T S 2. A R R AY D ATA 3. D ATA T R A N S F O R M AT I O N S 4. F I LT E R I N G A N D I M P U TAT I O N

18 DATA FRAMES The most commonly used structured datatype is a data frame: COLUMNS/FIELDS/ FEATURES/VARIABLES ROWS/INSTANCES

19 ARRAY DATA Another equally common datatype is an array A typical example is a 2-dimensional numerical array: m=5 columns n=4 rows The number of dimensions in an array can also be more than two MATH TERMS 0D: scalar 1D: vector 2D: matrix 3D or more: tensor

20 ARRAY DATA The most important operations on arrays include: constructing arrays (reading from input/initializing as empty) reshaping slicing (subsets of rows and/or columns) numerical operations (elements-wise / aggregate) Especially for large arrays, different implementations have huge efficiency differences e.g. python list vs numpy array avoid for-loops like the plague! Exercise 1.1

21 3. DATA TRANS- FORMATIONS

22 T R A N S F O R M AT I O N S FOR EXAMPLE, SQLITE (the most used database engine in the world): sqlite> sqlite> sqlite> sqlite> sqlite>.mode csv.headers on.output table.csv SELECT * FROM table;.output stdout We will not cover databases or SQL on this course. You should already have taken TKT10004 Fundamentals of Databases (Tietokantojen perusteet) or equivalent.

23 T R A N S F O R M AT I O N S Same in python (using SQlite): import sqlite3 database = 'database.sqlite' conn = sqlite3.connect(database) c = conn.cursor() query = "SELECT * FROM table;" for row in c.execute(query): print(row) conn.close()

24 T R A N S F O R M AT I O N S

25 T R A N S F O R M AT I O N S csv => json is easy with python and pandas! import pandas as pd import json data = pd.read_csv("player_stats.csv") print(data.to_json(orient='records', lines=true)) {"player_name":"aaron Appindangoye","height": ,"weight":187,"p_id":505942,"id": 1,"player_fifa_api_id":218353,"player_api_id": ,"date":" :00:00","overall_rating":67.0,"potential": 71.0,"preferred_foot":"right","attacking_work_rate" :"medium","defensive_work_rate":"medium","crossing" :49.0,"finishing":44.0,"heading_accuracy": 71.0,"short_passing":61.0,"volleys": 44.0,"dribbling":51.0,"curve": 45.0,"free_kick_accuracy":39.0,"long_passing": 64.0,"ball_control":49.0,"acceleration": 60.0,"sprint_speed":64.0,"agility": 59.0,"reactions":47.0,"balance":65.0,"shot_power":

26 OTHER TRANSFORMATIONS HTML => e.g. CSV "Scraping!" (dirty business)

27 TRANSFORMATIONS Content transformations: string to numeric, " " > (float) dates (mind the formats, 9/5/2017 vs ) NA/ /0/99/etc can mean missing entries splitting: name = "Teemu Roos" => first = "Teemu", last = "Roos"... Especially for text, it may be important to: downcase: SuperMan > superman remove punctuation stem: 'swimming' > 'swim'

28 4. F I LT E R I N G AND I M P U TAT I O N

29 FILTERING Subsetting (slicing): columns and/or rows (again, Exercise 1.1) Many of these are conveniently done using command-line tools such as grep, cut, awk, sed For big data, it is important to avoid reading all the data in memory before starting: the above tools only store and process the data little by little, so memory consumption is constant

30 IMPUTATION Missing values can be a show-stopper for many analysis methods A simple way is to filter out all records with missing entries This may, however, lose a lot of important data Another option is to impute, i.e., enter "fake" data in the place of the missing entries: average for numeric columns mode (most typical value) for categorical columns also possible to use machine learning to predict the missing entries based on the other features

31 IMPUTATION The usefulness or safety of different ways of deadling with missing data depend on the circumstances If the data is "missing completely at random" (technical term), then it is safe to delete rows with missing elements of course, the amount of data will shrink If, however, missingness depends on a variable of interest, then deleting rows may bias the conclusions e.g., overweight people may neglect to report their weight, so that the average is no longer representative of the whole population less trivially, people who experience side-effects tend to remember better that they took certain drug