Scalable Data Analysis (CIS )

Transcription

1 Scalable Data Analysis (CIS ) Introduction Dr. David Koop

2 NYC Taxi Data [Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance, T. W. Schneider] 2

3 What are your questions about this data? 3

4 NYC Taxi Data [Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance, T. W. Schneider] 4

5 NYC Taxi Data: Day analysis [Ferreira et al., 2013] 5

6 NYC Taxi Data: Region analysis t) in different neighborhoods over the first week of May The plots show that Midtown over the weekend, there is an increased number of dropoffs in Downtown. The figure also ncrease in the ay 5), with big s indicates that ew of city life, qualities. Peohe lack of taxi at their disconude difference her more afflule people take xploring other rising fact: the he other neighat the fare per omic incentive eans to reward Fig. 10. Comparing movement across NYC transportation hubs. On the [Ferreira et al., 2013] s top, we examine trips starting at the two major airports in NYC: JFK and La Guardia. In the bottom, we refine the query to compare trips starting 6 Grand Central) D. Koop, CIS , Fall 2017 at the airports with trips starting at the train stations, Penn Station and

7 Marine Traffic Data + - Leaflet Map data OpenStreetMap contributors 7

8 Marine Traffic Data + - Leaflet Map data OpenStreetMap contributors 8

9 Baseball Data [Deitrich et al., 2014] 9

10 Baseball Data [Deitrich et al., 2014] 10

11 Baseball Data [Deitrich et al., 2014] 11

12 Mobile Data Growth [Cisco Visual Networking Index Mobile, 2017] 12

13 Mobile Video Keeps Growing Note: Figures in parentheses refer to 2016 and 2021 traffic share. [Cisco Visual Networking Index Mobile, 2017] 13

14 Data Science Venn Diagram [D. Conway, The Data Science Venn Diagram, 2013] 14

15 Questions are important! Having data is great, but most of the time it just sits waiting for someone to analyze it The reason data analysis is not completely automated is that there are so many potential questions Humans need to stay involved in the loop Interaction and visualization can be important, especially early in data analysis 15

16 Scalability Big Data - What is big? For whom is it big? - variety, velocity, volume, Lots of data that was big is not an issue now Understanding the scalability of techniques is important There will always be larger datasets, want to understand - how methods scale - performance bounds - storage constraints 16

17 Real-time Analysis Want to have results now How? - Faster machines - Clusters - Progressive techniques 17

18 About Me Research Interests - Visualization - Computational Provenance - Geospatial Analysis Research Projects - VisTrails: - Dataflow Notebooks - Meta-versioning - Marine Traffic Data See my web page for more information

19 About You Previous topics course (CIS 602)? Research Papers? Data Science? Python? Database Experience? Analytics Experience? Cloud Computing Experience? Anything you want to see covered? 19

20 About this course Course web page is authoritative: Schedule, Readings, Assignments will be posted online - Check the web site before ing me Topics course - A current research area the professor works in - A chance to be on the cutting edge of research Requires student participation - Reading responses - Project presentations 20

21 About this course Balance of techniques and research ideas Some background (Python) followed by topic areas and readings Assignments at the beginning of course, project at end Two tests Topic areas: - Exploratory Data Analysis and Visualization - Data Acquisition - Data Storage and Access - Cloud Computing and Scalable Computation - Applications and specific data considerations 21

22 Project Do scalable data analysis of a large dataset - Questions - Analysis - Visualizations - Cloud/Cluster Computing Another option: research-related topic Waypoints: - Proposal - Progress Report - Final Presentation 22

23 About this course Course Registration: - Make sure you have registered in COIN for the course - me if you are not registered but are interested in taking the course Review of course policies: - Plagiarism and academic honesty - If you have any concerns or questions, please me as soon as possible If you are not sure if this course is a good fit, please me or talk to me 23

24 Data What is this data? Semantics: real-world meaning of the data Type: structural or mathematical interpretation Both often require metadata - Sometimes we can infer some of this information - Line between data and metadata isn t always clear 24

25 Data 25

26 Data Types Items - An item is an individual discrete entity - e.g. row in a table, node in a network Attributes - An attribute is some specific property that can be measured, observed, or logged - a.k.a. variable, (data) dimension 26

27 Items & Attributes attribute Field item 22 27

28 Data Types Nodes - Synonym for item but in the context of networks (graphs) Links - A link is a relation between two items - e.g. social network friends, computer network links 28

29 Items & Links Item Links [Bostock, 2011] 29

30 Dataset Types Dataset Types Tables Networks Fields (Continuous) Geometry (Spatial) Attributes (columns) Grid of positions Items (rows) Cell containing value Link Node (item) Cell Attributes (columns) Position Multidimensional Table Trees Value in cell Value in cell [Munzner (ill. Maguire), 2014] 30

31 Attribute Types Attribute Types Categorical Ordered Ordinal Quantitative Ordering Direction Sequential Diverging Cyclic [Munzner (ill. Maguire), 2014] 31

32 Categorial, Ordinal, and Quantitative 1 = Quantitative quantitative 23 2 = Nominal ordinal 3 = Ordinal categorical 32

33 Categorial, Ordinal, and Quantitative 1 = Quantitative quantitative 24 2 = Nominal ordinal 3 = Ordinal categorical 33

34 Semantics The type of data does not tell us what the data means or how it should be interpreted Tables have keys/values, fields have independent/dependent vars Flat Tables Multidimensional Fields [Munzner (ill. Maguire), 2014] 34

35 Analysis Actions Why? Targets Analyze All Data Consume Trends Outliers Features Discover Present Enjoy Produce Annotate Record Derive tag Attributes One Many Distribution Dependency Correlation Similarity Search Extremes Target known Target unknown Location known Location unknown Lookup Locate Browse Explore Network Data Topology Query Identify Compare Summarize Paths Spatial Data Shape What? Why? How? [Munzner (ill. Maguire), 2014] 35

36 Analysis: Consume & Produce Consume - Exploration - Explanation Analyze Consume Discover Present Enjoy - Enjoyment Produce - Annotation - Record - Derivation Produce Leads to new directions/ideas Annotate Record Derive tag [Munzner (ill. Maguire), 2014] 36

37 Analysis: Search and Query Search based on what a user knows Search - Target Target known Target unknown - Location Location known Lookup Browse Query depends on what data matters - One - Some (Often Two) - All Location unknown Query Locate Explore Identify Compare Summarize [Munzner (ill. Maguire), 2014] 37

38 Targets ALL DATA NETWORK DATA Trends Outliers Features Topology Paths ATTRIBUTES One Many Distribution Dependency Correlation Similarity SPATIAL DATA Shape Extremes [Munzner (ill. Maguire), 2014] 38

39 More Reading Listed on course schedule: - Challenges and Opportunities with Big Data, D. Agrawal et al. - Toward Scalable Systems for Big Data Analysis: A Technology Tutorial, H. Hu et al. - Big Data computing and clouds: Trends and future directions, M. D. Assuncao et al. 39

40 Next Class Introduction to/review of Python Download anaconda distribution: I am planning to use Python 3 (3.6) 40