Introduction to Data Analytics. David Walling

Size: px

Start display at page:

Download "Introduction to Data Analytics. David Walling"

Andrea Glenn
5 years ago
Views:

1 Introduction to Data Analytics David Walling

2 Source:

.. Simulation produces vast quantities (TB, PB) of output.

3 Computational Simulation Model first, given initial conditions at time=0, mathematical model provides conditions at time=t+1, t+2, etc... Simulation produces vast quantities (TB, PB) of output. (x,y,z,t) Often want to turn data into visualization, mostly focused on physical properties. Data is often cheaper to recreate than to store.

4 Data Analytics Data first, model is fit to data in order to provide some understanding, ex. prediction/inference. Data captured from devices, designed experiments, surveys, observation. Irreplaceable. kb - PB Information visualization to summarize and make data comprehensible. Not necessarily tied to physical properties.

5 Computational Models + Data Collection Data collection helps to improve/refine first principle models. Compare reality to simulation. Divergence between the two indicates new insight, perhaps the model is too simple.

gets cheaper = more data collected over wider area Combining datasets from NOAA, USGS, NCDC, etc.

6 Examples Gene Sequencing Massive data enable by sequencing technology. Statistical analysis of differences in gene expression given explanatory variables Ecology Sensors and the ability to retrieve data gets cheaper = more data collected over wider area Combining datasets from NOAA, USGS, NCDC, etc.. Fitting models describing runoff, climate change Social Web/Media Twitter, Facebook, etc.. Advancing Machine Learning algorithms to handle massive datasets New opportunities for pure science. Ex. Linguistics/Sociology/Anthropology

7 Data Analytics Workflow Research Question Acquisition * Exploration 1 * Cleaning * Exploration 2 Analytics Visualization Reports/Dissemination * Often 80% of time spent here

8 Data Sources Formal studies Designed experiments Surveys Measurement devices Ocean buoys Weather balloons Satellites Machine generated Web/system logs 3rd Parties NOAA Researcher s ftp server

9 Data Formats Text/semi-structured: excel, html, log files, JSON Free text: discussion forums, s, field notes REST API Database: Extract with SQL Proprietary/Binary: up to you to figure out how to access

10 Exploration: 1 What do I have? What does this column name mean? What timezone is this from? UTC/CDT Is this kph or mph, lbs or kgs? What is this special code? Best case, partial README

11 ETL I need this field in a different format. This field maps to my database table/column. Combining these two fields gives me a unique key. I need to store this data in a database/csv/hdf5. Often involves extensive scripting

12 Exploration: 2 Summary Statistics Pattern searching Basic information visualization Almost always start with a histogram Divide into groups and look at box-plots Not necessarily asking a specific question

13 Statistical Analysis Don t have data on entire population, want to make inferences based on sample data. Parametric vs Nonparametric Linear/Logistic regression Hypothesis Testing H0: u <= u0 vs Ha: u > u0 Ex. did treatment X improve life expectancy

14 Machine Learning Statistics + Sex Appeal Same methods, different terms parameters vs weights fitting vs learning model vs graph More worried about performance = prediction accuracy. Traditionally more Computer Science/Algorithm based. Ok with blackbox solutions that work.

15 Visualization Information visualization vs physical systems Check model assumptions and validity Convey results Lead to new hypothesis

16 Visualization source: r-bloggers.com

17 Data Analytic Tools Available at TACC R module load Rstats optimized for each system popular/difficult packages pre-installed ability to install other packages in ~home library interactive access via: module load Rstudio Python Larger ecosystem, i.e. can build websites, everyday scripts, deep analytics Many 3rd party modules, not as easy to install as R Highlighted modules: numpy, scipy, pandas, matplotlib, sklearn

18 Data Analytic Tools Not at TACC Proprietary tools: MATLAB, Mathematica, Tableau, etc.. Can often install locally, need to provide your own license for questions

19 David Walling

Data Analyst Nanodegree Syllabus

Data Analyst Nanodegree Syllabus Discover Insights from Data with Python, R, SQL, and Tableau Before You Start Prerequisites : In order to succeed in this program, we recommend having experience working