CIS : Scalable Data Analysis

Size: px

Start display at page:

Download "CIS : Scalable Data Analysis"

Timothy Baker
5 years ago
Views:

1 CIS : Scalable Data Analysis Visualization Dr. David Koop

2 Growth of Data 2

3 Usefulness of Data 3

4 Analyzed Data 4

Dishwashers Traffic Lights MRI Scanners NFL Football

5 Example Data Sources Radio Telescopes Twitter Wind Turbine Sensors Surveillance Cameras Cars & Airplanes Dog Collars Dishwashers Traffic Lights MRI Scanners NFL Football Players Farming [Zebra MotionWorks] [CC-SA 2.0, Stephan Trebs] 5

6 Large Synoptic Survey Telescope (LSST) Image every 15 seconds 100PB over 10 years [ 6

7 Large Numerical Simulations Millennium simulation: dark matter, 30TB raw data [V. Springel et al., 2005] 7

8 More Data Sources Awesome Public Datasets Kaggle Datasets Government Data: data.gov Customer Data: (see Internal Business Data 8

9 Dimensions of Data Dimension Categories Question to be answered objective subjective Type Web Crawler, Customizable Crawler, Search What is the type of the core offering? Engine, Pure Data Vendor, Complex Data Vendor, Matching Vendor, Enrichment Tagging, Enrichment Sentiment, Enrichment Analysis, Data Market Place Time Frame Static/Factual, Up To Date Is the data static or real-time? Domain All, Finance/Economy, Bio Medicine, Social Media, What is the data about? Geo Data, Address Data Data Origin Internet, Self-Generated, User, Community, Where does the data come from? Who is the author? Government, Authority Pricing Model Free, Freemium, Pay-Per-Use, Flat Rate Is the offer free, pay-per-use or usable with a flat rate? Data Access API, Download, Specialized Software, Web What technical means are offered to access the data? Interface Data Output XML, CSV/XLS, JSON, RDF, Report In what way is the data formatted for the user? Language English, German, More What is the language of the website? Does it differ from the language of the data? Target Audience Business, Customer Towards whom is the product geared? Trustworthiness Low, Medium, High How trustworthy is the vendor? Can the original data source be tracked or verified? Size of Vendor Startup, Medium, Big, Global Player How big is the vendor? Maturity Research Project, Beta, Medium, High Is the product still in beta or already established? [Schomm et al., 2013] 9

10 Big Data or Small Data? Many companies feel the need to overclaim the amount of data "when you take a normal tech company and sprinkle on data, you get the next Google" [C. O'Neil] Many large datasets are not useful Twitter processes 8TB, but the tweets only take about 30GB Wikipedia can be downloaded onto a USB drive All MP3s can be stored on a moderately sized disk array Can learn a lot from a "small" dataset, e.g. sensors from a single turbine, grocery store, Apple Watch Small data focused on end-user, more timely insights? 10

11 Jobs on a Large Analytics Cluster [R. Appuswamy et al., 2013] 11

12 Reading Quiz 12

Assignment 1 http://www.cis.umassd.edu/~dkoop/ cis602-2017fa/assignment1.

13 Assignment 1 cis fa/assignment1.html Boston Property Assessments - Initial exploratory analysis - Use a Python Notebook - May use pandas - Label subproblems and answers - Show work (even if it's not your final answer) [Google Maps] 13

14 Big Data Visualization (Slides from Dr. Nan Cao via Dr. Ching-Yung Lin)

15 Big Data Visualization What is Visualization and Why Visualization? Big Data Visualization Challenges and Techniques Visualizing Big Data Visual Analytics and Big Data 15

16 Whisper: Tracing Information Diffusion in Real Time 16

17 Customizing Computational Methods for Visual Analytics with Big Data J. Choo and H. Park

18 Complexities of Visual Analytics of Big Data Human perception and large numbers of items - locating items - tracking items Limited screen space: - clutter - overlapping items 18

19 Use Computational Methods Methods: - Dimensionality reduction - Clustering - Machine learning & data mining Issues with using these methods - What's going on? - Waiting time Goal: - Interactive - Faster 19

20 Exploiting Discrepancies Precision: use knowledge of screen resolution to set precision Convergence: Don't worry about minor changes that may be imperceptible - Human perception - Screen resolution constraints 20

21 Changes in Cluster Membership in k-means No. of per-iteration changes/accuracy (%) Accuracy against final solution Per-iteration changes No. of iterations 21

22 Customizing Computations Use lower precision computation Use interactive visualization that shows iterations Refine results iteratively Data scale confinement 22

23 Iteration-level Interactive Visualization Computational module Subroutine 1 Subroutine k Output Visualization/summarization Iterate (a) Interaction Computational module Subroutine 1 Subroutine k Output Visualization/summarization Iterate (b) Interaction 23

24 Next Progressive Visualization Read: How Progressive Visualizations Affect Exploratory Analysis Write: - Critique of Paper - < 1 paragraph summary, 2 paragraphs critique Which ideas in the paper are interesting and why? Which ideas do you have related to the paper Which ideas seem problematic? Can you suggest alternatives? - Turn in via mycourses - Due Tuesday before class 24

INTRODUCTION TO BIG DATA, DATA MINING, AND MACHINE LEARNING

CS 7265 BIG DATA ANALYTICS INTRODUCTION TO BIG DATA, DATA MINING, AND MACHINE LEARNING * Some contents are adapted from Dr. Hung Huang and Dr. Chengkai Li at UT Arlington Mingon Kang, PhD Computer Science,