STA 490H1S Initial Examination of Data

Size: px
Start display at page:

Download "STA 490H1S Initial Examination of Data"

Transcription

1 Initial Examination of Data Alison L. Department of Statistics University of Toronto Winter 2011

2 Course mantra It s OK not to know. Expressing ignorance is encouraged. It s not OK to not have a willingness to learn.

3 Initial Examination of Data Purpose: Understand the structure of the data.

4 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitiative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert scales or binned quantitative), binary

5 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitiative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert scales or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations?

6 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitiative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert scales or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems.

7 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitiative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert scales or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems.

8 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitiative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert scales or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems. Get ideas for more sophisticated analyses.

9 Initial Examination of Data Purpose: Understand the structure of the data. Types of variables: Quantitiative: continuous or discrete Categorical: nominal, ordinal (e.g., Likert scales or binned quantitative), binary Check the quality of the data. Find errors (data cleaning). Check for credibility, consistency, completeness. Identify potential outliers. Are there missing observations? Clear up any problems. Get ideas for more sophisticated analyses. Check on whether or not assumptions of more sophisticated analyses seem reasonable.

10 IDA Should be motivated by original research questions.

11 IDA Should be motivated by original research questions. Avoid data dredging. (Look long enough and you ll find some meaningless pattern.)

12 IDA Should be motivated by original research questions. Avoid data dredging. (Look long enough and you ll find some meaningless pattern.) Trivial? Requires judgment and common sense.

13 Types of Missing Data 1. Missing Completely At Random (MCAR) The probability that a data value is missing does not depend on the missing value, nor on the values of all other variables.

14 Types of Missing Data 1. Missing Completely At Random (MCAR) The probability that a data value is missing does not depend on the missing value, nor on the values of all other variables. 2. Missing At Random (MAR) The probability that a data value is missing, conditional on the values of the other variables for the observation, is not related to the missing value.

15 Types of Missing Data 1. Missing Completely At Random (MCAR) The probability that a data value is missing does not depend on the missing value, nor on the values of all other variables. 2. Missing At Random (MAR) The probability that a data value is missing, conditional on the values of the other variables for the observation, is not related to the missing value. 3. Informative / Non-ignorable (NMAR) Difficult to deal with.

16 Tools for IDA 5 number summary (for all data and for subsets).

17 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d.

18 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms / stem-and-leaf plots.

19 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms / stem-and-leaf plots. Frequency tables (1- and 2-way) for categorical variables

20 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms / stem-and-leaf plots. Frequency tables (1- and 2-way) for categorical variables Scatterplots.

21 Tools for IDA 5 number summary (for all data and for subsets). Other summary statistics, e.g., mean and s.d. Histograms / stem-and-leaf plots. Frequency tables (1- and 2-way) for categorical variables Scatterplots. Correlations.

22 Some More Sophisticated Tools for IDA

23 Some More Sophisticated Tools for IDA Kernel Density Estimation Smoothed function to estimate the density function. Amount of smoothing controlled by the bandwidth. Non-parametric (that is, doesn t make an assumption about the distribution).

24 Some More Sophisticated Tools for IDA Kernel Density Estimation Smoothed function to estimate the density function. Amount of smoothing controlled by the bandwidth. Non-parametric (that is, doesn t make an assumption about the distribution). LOWESS (LOESS): Locally Weighted Scatterplot Smoothing Idea: fit a simple polynomial using regression on a small ranges of the independent variable, and smoothly join up the pieces. Amount of smoothing controlled by a smoothing parameter.

25 Course mantra It s OK not to know. Expressing ignorance is encouraged. It s not OK to not have a willingness to learn.

26 For Thursday: Hand in your meeting summary to your TA advisor. Be ready for a discussion about a plan for the project: Data cleaning / IDA What methods of analysis might be appropriate.

27 For next class (Tuesday, February 1) Read Chapters 11 and 12 in Chatfield. Bring the text to class.

STA490HY1Y. Initial Examination of Data

STA490HY1Y. Initial Examination of Data STA490Y1Y Initial Examination of Data Alison L. Department of Statistical Sciences University of Toronto 2016-17 Course mantra It s OK not to know. Expressing ignorance is encouraged. It s not OK to not

More information

Special Review Section. Copyright 2014 Pearson Education, Inc.

Special Review Section. Copyright 2014 Pearson Education, Inc. Special Review Section SRS-1--1 Special Review Section Chapter 1: The Where, Why, and How of Data Collection Chapter 2: Graphs, Charts, and Tables Describing Your Data Chapter 3: Describing Data Using

More information

JMP 10 Student Edition Quick Guide

JMP 10 Student Edition Quick Guide JMP 10 Student Edition Quick Guide Instructions presume an open data table, default preference settings and appropriately typed, user-specified variables of interest. RMC = Click Right Mouse Button Graphing

More information

Minitab 17 commands Prepared by Jeffrey S. Simonoff

Minitab 17 commands Prepared by Jeffrey S. Simonoff Minitab 17 commands Prepared by Jeffrey S. Simonoff Data entry and manipulation To enter data by hand, click on the Worksheet window, and enter the values in as you would in any spreadsheet. To then save

More information

12. A(n) is the number of times an item or number occurs in a data set.

12. A(n) is the number of times an item or number occurs in a data set. Chapter 15 Vocabulary Practice Match each definition to its corresponding term. a. data b. statistical question c. population d. sample e. data analysis f. parameter g. statistic h. survey i. experiment

More information

Part I, Chapters 4 & 5. Data Tables and Data Analysis Statistics and Figures

Part I, Chapters 4 & 5. Data Tables and Data Analysis Statistics and Figures Part I, Chapters 4 & 5 Data Tables and Data Analysis Statistics and Figures Descriptive Statistics 1 Are data points clumped? (order variable / exp. variable) Concentrated around one value? Concentrated

More information

Nonparametric Approaches to Regression

Nonparametric Approaches to Regression Nonparametric Approaches to Regression In traditional nonparametric regression, we assume very little about the functional form of the mean response function. In particular, we assume the model where m(xi)

More information

CHAPTER 1 INTRODUCTION

CHAPTER 1 INTRODUCTION Introduction CHAPTER 1 INTRODUCTION Mplus is a statistical modeling program that provides researchers with a flexible tool to analyze their data. Mplus offers researchers a wide choice of models, estimators,

More information

Applied Regression Modeling: A Business Approach

Applied Regression Modeling: A Business Approach i Applied Regression Modeling: A Business Approach Computer software help: SAS SAS (originally Statistical Analysis Software ) is a commercial statistical software package based on a powerful programming

More information

Brief Guide on Using SPSS 10.0

Brief Guide on Using SPSS 10.0 Brief Guide on Using SPSS 10.0 (Use student data, 22 cases, studentp.dat in Dr. Chang s Data Directory Page) (Page address: http://www.cis.ysu.edu/~chang/stat/) I. Processing File and Data To open a new

More information

Applied Regression Modeling: A Business Approach

Applied Regression Modeling: A Business Approach i Applied Regression Modeling: A Business Approach Computer software help: SPSS SPSS (originally Statistical Package for the Social Sciences ) is a commercial statistical software package with an easy-to-use

More information

BASIC LOESS, PBSPLINE & SPLINE

BASIC LOESS, PBSPLINE & SPLINE CURVES AND SPLINES DATA INTERPOLATION SGPLOT provides various methods for fitting smooth trends to scatterplot data LOESS An extension of LOWESS (Locally Weighted Scatterplot Smoothing), uses locally weighted

More information

Regression III: Advanced Methods

Regression III: Advanced Methods Lecture 3: Distributions Regression III: Advanced Methods William G. Jacoby Michigan State University Goals of the lecture Examine data in graphical form Graphs for looking at univariate distributions

More information

Statistics Lecture 6. Looking at data one variable

Statistics Lecture 6. Looking at data one variable Statistics 111 - Lecture 6 Looking at data one variable Chapter 1.1 Moore, McCabe and Craig Probability vs. Statistics Probability 1. We know the distribution of the random variable (Normal, Binomial)

More information

Math 183 Statistical Methods

Math 183 Statistical Methods Math 183 Statistical Methods Eddie Aamari S.E.W. Assistant Professor eaamari@ucsd.edu math.ucsd.edu/~eaamari/ AP&M 5880A 1 / 24 Math 183 Statistical Methods Eddie Aamari S.E.W. Assistant Professor eaamari@ucsd.edu

More information

Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University

Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Data Mining Chapter 3: Visualizing and Exploring Data Fall 2011 Ming Li Department of Computer Science and Technology Nanjing University Exploratory data analysis tasks Examine the data, in search of structures

More information

WELCOME! Lecture 3 Thommy Perlinger

WELCOME! Lecture 3 Thommy Perlinger Quantitative Methods II WELCOME! Lecture 3 Thommy Perlinger Program Lecture 3 Cleaning and transforming data Graphical examination of the data Missing Values Graphical examination of the data It is important

More information

Simulation of Imputation Effects Under Different Assumptions. Danny Rithy

Simulation of Imputation Effects Under Different Assumptions. Danny Rithy Simulation of Imputation Effects Under Different Assumptions Danny Rithy ABSTRACT Missing data is something that we cannot always prevent. Data can be missing due to subjects' refusing to answer a sensitive

More information

Machine Learning Techniques for Data Mining

Machine Learning Techniques for Data Mining Machine Learning Techniques for Data Mining Eibe Frank University of Waikato New Zealand 10/25/2000 1 PART VII Moving on: Engineering the input and output 10/25/2000 2 Applying a learner is not all Already

More information

Acquisition Description Exploration Examination Understanding what data is collected. Characterizing properties of data.

Acquisition Description Exploration Examination Understanding what data is collected. Characterizing properties of data. Summary Statistics Acquisition Description Exploration Examination what data is collected Characterizing properties of data. Exploring the data distribution(s). Identifying data quality problems. Selecting

More information

Slides for Data Mining by I. H. Witten and E. Frank

Slides for Data Mining by I. H. Witten and E. Frank Slides for Data Mining by I. H. Witten and E. Frank 7 Engineering the input and output Attribute selection Scheme-independent, scheme-specific Attribute discretization Unsupervised, supervised, error-

More information

Data Mining. CS57300 Purdue University. Bruno Ribeiro. February 1st, 2018

Data Mining. CS57300 Purdue University. Bruno Ribeiro. February 1st, 2018 Data Mining CS57300 Purdue University Bruno Ribeiro February 1st, 2018 1 Exploratory Data Analysis & Feature Construction How to explore a dataset Understanding the variables (values, ranges, and empirical

More information

3. Data Preprocessing. 3.1 Introduction

3. Data Preprocessing. 3.1 Introduction 3. Data Preprocessing Contents of this Chapter 3.1 Introduction 3.2 Data cleaning 3.3 Data integration 3.4 Data transformation 3.5 Data reduction SFU, CMPT 740, 03-3, Martin Ester 84 3.1 Introduction Motivation

More information

2. Data Preprocessing

2. Data Preprocessing 2. Data Preprocessing Contents of this Chapter 2.1 Introduction 2.2 Data cleaning 2.3 Data integration 2.4 Data transformation 2.5 Data reduction Reference: [Han and Kamber 2006, Chapter 2] SFU, CMPT 459

More information

Basic Medical Statistics Course

Basic Medical Statistics Course Basic Medical Statistics Course S0 SPSS Intro November 2013 Wilma Heemsbergen w.heemsbergen@nki.nl 1 13.00 ~ 15.30 Database (20 min) SPSS (40 min) Short break Exercise (60 min) This Afternoon During the

More information

Your Name: Section: INTRODUCTION TO STATISTICAL REASONING Computer Lab #4 Scatterplots and Regression

Your Name: Section: INTRODUCTION TO STATISTICAL REASONING Computer Lab #4 Scatterplots and Regression Your Name: Section: 36-201 INTRODUCTION TO STATISTICAL REASONING Computer Lab #4 Scatterplots and Regression Objectives: 1. To learn how to interpret scatterplots. Specifically you will investigate, using

More information

Econometric Tools 1: Non-Parametric Methods

Econometric Tools 1: Non-Parametric Methods University of California, Santa Cruz Department of Economics ECON 294A (Fall 2014) - Stata Lab Instructor: Manuel Barron 1 Econometric Tools 1: Non-Parametric Methods 1 Introduction This lecture introduces

More information

Table of Contents (As covered from textbook)

Table of Contents (As covered from textbook) Table of Contents (As covered from textbook) Ch 1 Data and Decisions Ch 2 Displaying and Describing Categorical Data Ch 3 Displaying and Describing Quantitative Data Ch 4 Correlation and Linear Regression

More information

Information Visualization in Data Mining. S.T. Balke Department of Chemical Engineering and Applied Chemistry University of Toronto

Information Visualization in Data Mining. S.T. Balke Department of Chemical Engineering and Applied Chemistry University of Toronto Information Visualization in Data Mining S.T. Balke Department of Chemical Engineering and Applied Chemistry University of Toronto Motivation Data visualization relies primarily on human cognition for

More information

Chapter 1. Looking at Data-Distribution

Chapter 1. Looking at Data-Distribution Chapter 1. Looking at Data-Distribution Statistics is the scientific discipline that provides methods to draw right conclusions: 1)Collecting the data 2)Describing the data 3)Drawing the conclusions Raw

More information

CHAPTER 11 EXAMPLES: MISSING DATA MODELING AND BAYESIAN ANALYSIS

CHAPTER 11 EXAMPLES: MISSING DATA MODELING AND BAYESIAN ANALYSIS Examples: Missing Data Modeling And Bayesian Analysis CHAPTER 11 EXAMPLES: MISSING DATA MODELING AND BAYESIAN ANALYSIS Mplus provides estimation of models with missing data using both frequentist and Bayesian

More information

Few s Design Guidance

Few s Design Guidance Few s Design Guidance CS 4460 Intro. to Information Visualization September 9, 2014 John Stasko Today s Agenda Stephen Few & Perceptual Edge Fall 2014 CS 4460 2 1 Stephen Few s Guidance Excellent advice

More information

DSC 201: Data Analysis & Visualization

DSC 201: Data Analysis & Visualization DSC 201: Data Analysis & Visualization Exploratory Data Analysis Dr. David Koop What is Exploratory Data Analysis? "Detective work" to summarize and explore datasets Includes: - Data acquisition and input

More information

Minitab 18 Feature List

Minitab 18 Feature List Minitab 18 Feature List * New or Improved Assistant Measurement systems analysis * Capability analysis Graphical analysis Hypothesis tests Regression DOE Control charts * Graphics Scatterplots, matrix

More information

22s:152 Applied Linear Regression. Chapter 18: Nonparametric Regression (Lowess smoothing)

22s:152 Applied Linear Regression. Chapter 18: Nonparametric Regression (Lowess smoothing) 22s:152 Applied Linear Regression Chapter 18: Nonparametric Regression (Lowess smoothing) When the nature of the response function (or mean structure) is unknown and one wishes to investigate it, a nonparametric

More information

Statistical Graphs & Charts

Statistical Graphs & Charts Statistical Graphs & Charts CS 4460 Intro. to Information Visualization August 30, 2017 John Stasko Learning Objectives Learn different statistical data graphs Line graph, Bar Graph, Scatterplot, Trellis,

More information

Algebra 1, 4th 4.5 weeks

Algebra 1, 4th 4.5 weeks The following practice standards will be used throughout 4.5 weeks:. Make sense of problems and persevere in solving them.. Reason abstractly and quantitatively. 3. Construct viable arguments and critique

More information

/4 Directions: Graph the functions, then answer the following question.

/4 Directions: Graph the functions, then answer the following question. 1.) Graph y = x. Label the graph. Standard: F-BF.3 Identify the effect on the graph of replacing f(x) by f(x) +k, k f(x), f(kx), and f(x+k), for specific values of k; find the value of k given the graphs.

More information

8 Organizing and Displaying

8 Organizing and Displaying CHAPTER 8 Organizing and Displaying Data for Comparison Chapter Outline 8.1 BASIC GRAPH TYPES 8.2 DOUBLE LINE GRAPHS 8.3 TWO-SIDED STEM-AND-LEAF PLOTS 8.4 DOUBLE BAR GRAPHS 8.5 DOUBLE BOX-AND-WHISKER PLOTS

More information

LOESS curve fitted to a population sampled from a sine wave with uniform noise added. The LOESS curve approximates the original sine wave.

LOESS curve fitted to a population sampled from a sine wave with uniform noise added. The LOESS curve approximates the original sine wave. LOESS curve fitted to a population sampled from a sine wave with uniform noise added. The LOESS curve approximates the original sine wave. http://en.wikipedia.org/wiki/local_regression Local regression

More information

STA 570 Spring Lecture 5 Tuesday, Feb 1

STA 570 Spring Lecture 5 Tuesday, Feb 1 STA 570 Spring 2011 Lecture 5 Tuesday, Feb 1 Descriptive Statistics Summarizing Univariate Data o Standard Deviation, Empirical Rule, IQR o Boxplots Summarizing Bivariate Data o Contingency Tables o Row

More information

Chapter 2: Modeling Distributions of Data

Chapter 2: Modeling Distributions of Data Chapter 2: Modeling Distributions of Data Section 2.2 The Practice of Statistics, 4 th edition - For AP* STARNES, YATES, MOORE Chapter 2 Modeling Distributions of Data 2.1 Describing Location in a Distribution

More information

Quick introduction to descriptive statistics and graphs in. R Commander. Written by: Robin Beaumont

Quick introduction to descriptive statistics and graphs in. R Commander. Written by: Robin Beaumont Quick introduction to descriptive statistics and graphs in R Commander Written by: Robin Beaumont e-mail: robin@organplayers.co.uk http://www.robin-beaumont.co.uk/virtualclassroom/stats/course1.html Date

More information

Basic Statistical Graphics in R. Stem and leaf plots 100,100,100,99,98,97,96,94,94,87,83,82,77,75,75,73,71,66,63,55,55,55,51,19

Basic Statistical Graphics in R. Stem and leaf plots 100,100,100,99,98,97,96,94,94,87,83,82,77,75,75,73,71,66,63,55,55,55,51,19 Basic Statistical Graphics in R. Stem and leaf plots Example. Create a vector of data titled exam containing the following scores: 100,100,100,99,98,97,96,94,94,87,83,82,77,75,75,73,71,66,63,55,55,55,51,19

More information

Summary of Last Chapter. Course Content. Chapter 3 Objectives. Chapter 3: Data Preprocessing. Dr. Osmar R. Zaïane. University of Alberta 4

Summary of Last Chapter. Course Content. Chapter 3 Objectives. Chapter 3: Data Preprocessing. Dr. Osmar R. Zaïane. University of Alberta 4 Principles of Knowledge Discovery in Data Fall 2004 Chapter 3: Data Preprocessing Dr. Osmar R. Zaïane University of Alberta Summary of Last Chapter What is a data warehouse and what is it for? What is

More information

Table Of Contents: xix Foreword to Second Edition

Table Of Contents: xix Foreword to Second Edition Data Mining : Concepts and Techniques Table Of Contents: Foreword xix Foreword to Second Edition xxi Preface xxiii Acknowledgments xxxi About the Authors xxxv Chapter 1 Introduction 1 (38) 1.1 Why Data

More information

The basic arrangement of numeric data is called an ARRAY. Array is the derived data from fundamental data Example :- To store marks of 50 student

The basic arrangement of numeric data is called an ARRAY. Array is the derived data from fundamental data Example :- To store marks of 50 student Organizing data Learning Outcome 1. make an array 2. divide the array into class intervals 3. describe the characteristics of a table 4. construct a frequency distribution table 5. constructing a composite

More information

Course on Microarray Gene Expression Analysis

Course on Microarray Gene Expression Analysis Course on Microarray Gene Expression Analysis ::: Normalization methods and data preprocessing Madrid, April 27th, 2011. Gonzalo Gómez ggomez@cnio.es Bioinformatics Unit CNIO ::: Introduction. The probe-level

More information

Contents. Foreword to Second Edition. Acknowledgments About the Authors

Contents. Foreword to Second Edition. Acknowledgments About the Authors Contents Foreword xix Foreword to Second Edition xxi Preface xxiii Acknowledgments About the Authors xxxi xxxv Chapter 1 Introduction 1 1.1 Why Data Mining? 1 1.1.1 Moving toward the Information Age 1

More information

Bar Charts and Frequency Distributions

Bar Charts and Frequency Distributions Bar Charts and Frequency Distributions Use to display the distribution of categorical (nominal or ordinal) variables. For the continuous (numeric) variables, see the page Histograms, Descriptive Stats

More information

Preprocessing Short Lecture Notes cse352. Professor Anita Wasilewska

Preprocessing Short Lecture Notes cse352. Professor Anita Wasilewska Preprocessing Short Lecture Notes cse352 Professor Anita Wasilewska Data Preprocessing Why preprocess the data? Data cleaning Data integration and transformation Data reduction Discretization and concept

More information

Chapter 2: Looking at Multivariate Data

Chapter 2: Looking at Multivariate Data Chapter 2: Looking at Multivariate Data Multivariate data could be presented in tables, but graphical presentations are more effective at displaying patterns. We can see the patterns in one variable at

More information

Missing Data. Where did it go?

Missing Data. Where did it go? Missing Data Where did it go? 1 Learning Objectives High-level discussion of some techniques Identify type of missingness Single vs Multiple Imputation My favourite technique 2 Problem Uh data are missing

More information

Topic (3) SUMMARIZING DATA - TABLES AND GRAPHICS

Topic (3) SUMMARIZING DATA - TABLES AND GRAPHICS Topic (3) SUMMARIZING DATA - TABLES AND GRAPHICS 3- Topic (3) SUMMARIZING DATA - TABLES AND GRAPHICS A) Frequency Distributions For Samples Defn: A FREQUENCY DISTRIBUTION is a tabular or graphical display

More information

LAB 1: Graphical Descriptions of Data

LAB 1: Graphical Descriptions of Data LAB 1: Graphical Descriptions of Data Part I: Before Class 1) Read this assignment all the way through; 2) Know the terms and understand the concepts of: - scatterplots - stemplots - distributions - histograms

More information

SPSS QM II. SPSS Manual Quantitative methods II (7.5hp) SHORT INSTRUCTIONS BE CAREFUL

SPSS QM II. SPSS Manual Quantitative methods II (7.5hp) SHORT INSTRUCTIONS BE CAREFUL SPSS QM II SHORT INSTRUCTIONS This presentation contains only relatively short instructions on how to perform some statistical analyses in SPSS. Details around a certain function/analysis method not covered

More information

Preparing for Data Analysis

Preparing for Data Analysis Preparing for Data Analysis Prof. Andrew Stokes March 27, 2018 Managing your data Entering the data into a database Reading the data into a statistical computing package Checking the data for errors and

More information

CITS4009 Introduc0on to Data Science

CITS4009 Introduc0on to Data Science School of Computer Science and Software Engineering CITS4009 Introduc0on to Data Science SEMESTER 2, 2017: CHAPTER 3 EXPLORING DATA 1 Chapter Objec0ves Using summary sta.s.cs to explore data Exploring

More information

Data 100 Lecture 5: Data Cleaning & Exploratory Data Analysis

Data 100 Lecture 5: Data Cleaning & Exploratory Data Analysis OrderNum ProdID Name OrderId Cust Name Date 1 42 Gum 1 Joe 8/21/2017 2 999 NullFood 2 Arthur 8/14/2017 2 42 Towel 2 Arthur 8/14/2017 1/31/18 Data 100 Lecture 5: Data Cleaning & Exploratory Data Analysis

More information

Data 100. Lecture 5: Data Cleaning & Exploratory Data Analysis

Data 100. Lecture 5: Data Cleaning & Exploratory Data Analysis Data 100 Lecture 5: Data Cleaning & Exploratory Data Analysis Slides by: Joseph E. Gonzalez, Deb Nolan, & Joe Hellerstein jegonzal@berkeley.edu deborah_nolan@berkeley.edu hellerstein@berkeley.edu? Last

More information

VCEasy VISUAL FURTHER MATHS. Overview

VCEasy VISUAL FURTHER MATHS. Overview VCEasy VISUAL FURTHER MATHS Overview This booklet is a visual overview of the knowledge required for the VCE Year 12 Further Maths examination.! This booklet does not replace any existing resources that

More information

Data Preprocessing. Slides by: Shree Jaswal

Data Preprocessing. Slides by: Shree Jaswal Data Preprocessing Slides by: Shree Jaswal Topics to be covered Why Preprocessing? Data Cleaning; Data Integration; Data Reduction: Attribute subset selection, Histograms, Clustering and Sampling; Data

More information

Excel 2010 with XLSTAT

Excel 2010 with XLSTAT Excel 2010 with XLSTAT J E N N I F E R LE W I S PR I E S T L E Y, PH.D. Introduction to Excel 2010 with XLSTAT The layout for Excel 2010 is slightly different from the layout for Excel 2007. However, with

More information

Last time... Coryn Bailer-Jones. check and if appropriate remove outliers, errors etc. linear regression

Last time... Coryn Bailer-Jones. check and if appropriate remove outliers, errors etc. linear regression Machine learning, pattern recognition and statistical data modelling Lecture 3. Linear Methods (part 1) Coryn Bailer-Jones Last time... curse of dimensionality local methods quickly become nonlocal as

More information

University of Florida CISE department Gator Engineering. Visualization

University of Florida CISE department Gator Engineering. Visualization Visualization Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida What is visualization? Visualization is the process of converting data (information) in to

More information

Multivariate Analysis

Multivariate Analysis Multivariate Analysis Project 1 Jeremy Morris February 20, 2006 1 Generating bivariate normal data Definition 2.2 from our text states that we can transform a sample from a standard normal random variable

More information

Orientation Assignment for Statistics Software (nothing to hand in) Mary Parker,

Orientation Assignment for Statistics Software (nothing to hand in) Mary Parker, Orientation to MINITAB, Mary Parker, mparker@austincc.edu. Last updated 1/3/10. page 1 of Orientation Assignment for Statistics Software (nothing to hand in) Mary Parker, mparker@austincc.edu When you

More information

CSC 411: Lecture 05: Nearest Neighbors

CSC 411: Lecture 05: Nearest Neighbors CSC 411: Lecture 05: Nearest Neighbors Raquel Urtasun & Rich Zemel University of Toronto Sep 28, 2015 Urtasun & Zemel (UofT) CSC 411: 05-Nearest Neighbors Sep 28, 2015 1 / 13 Today Non-parametric models

More information

Multiple Imputation for Missing Data. Benjamin Cooper, MPH Public Health Data & Training Center Institute for Public Health

Multiple Imputation for Missing Data. Benjamin Cooper, MPH Public Health Data & Training Center Institute for Public Health Multiple Imputation for Missing Data Benjamin Cooper, MPH Public Health Data & Training Center Institute for Public Health Outline Missing data mechanisms What is Multiple Imputation? Software Options

More information

Kernel Density Estimation (KDE)

Kernel Density Estimation (KDE) Kernel Density Estimation (KDE) Previously, we ve seen how to use the histogram method to infer the probability density function (PDF) of a random variable (population) using a finite data sample. In this

More information

Applied Regression Modeling: A Business Approach

Applied Regression Modeling: A Business Approach i Applied Regression Modeling: A Business Approach Computer software help: SAS code SAS (originally Statistical Analysis Software) is a commercial statistical software package based on a powerful programming

More information

3 Graphical Displays of Data

3 Graphical Displays of Data 3 Graphical Displays of Data Reading: SW Chapter 2, Sections 1-6 Summarizing and Displaying Qualitative Data The data below are from a study of thyroid cancer, using NMTR data. The investigators looked

More information

Making Science Graphs and Interpreting Data

Making Science Graphs and Interpreting Data Making Science Graphs and Interpreting Data Eye Opener: 5 mins What do you see? What do you think? Look up terms you don t know What do Graphs Tell You? A graph is a way of expressing a relationship between

More information

Further Maths Notes. Common Mistakes. Read the bold words in the exam! Always check data entry. Write equations in terms of variables

Further Maths Notes. Common Mistakes. Read the bold words in the exam! Always check data entry. Write equations in terms of variables Further Maths Notes Common Mistakes Read the bold words in the exam! Always check data entry Remember to interpret data with the multipliers specified (e.g. in thousands) Write equations in terms of variables

More information

Learn What s New. Statistical Software

Learn What s New. Statistical Software Statistical Software Learn What s New Upgrade now to access new and improved statistical features and other enhancements that make it even easier to analyze your data. The Assistant Data Customization

More information

Summarising Data. Mark Lunt 09/10/2018. Arthritis Research UK Epidemiology Unit University of Manchester

Summarising Data. Mark Lunt 09/10/2018. Arthritis Research UK Epidemiology Unit University of Manchester Summarising Data Mark Lunt Arthritis Research UK Epidemiology Unit University of Manchester 09/10/2018 Summarising Data Today we will consider Different types of data Appropriate ways to summarise these

More information

Basic Medical Statistics Course

Basic Medical Statistics Course Basic Medical Statistics Course S0 SPSS Intro December 2014 Wilma Heemsbergen w.heemsbergen@nki.nl This Afternoon 13.00 ~ 15.00 SPSS lecture Short break Exercise 2 Database Example 3 Types of data Type

More information

Interactive Math Glossary Terms and Definitions

Interactive Math Glossary Terms and Definitions Terms and Definitions Absolute Value the magnitude of a number, or the distance from 0 on a real number line Addend any number or quantity being added addend + addend = sum Additive Property of Area the

More information

JMP Book Descriptions

JMP Book Descriptions JMP Book Descriptions The collection of JMP documentation is available in the JMP Help > Books menu. This document describes each title to help you decide which book to explore. Each book title is linked

More information

Data Statistics Population. Census Sample Correlation... Statistical & Practical Significance. Qualitative Data Discrete Data Continuous Data

Data Statistics Population. Census Sample Correlation... Statistical & Practical Significance. Qualitative Data Discrete Data Continuous Data Data Statistics Population Census Sample Correlation... Voluntary Response Sample Statistical & Practical Significance Quantitative Data Qualitative Data Discrete Data Continuous Data Fewer vs Less Ratio

More information

Data Preprocessing. Why Data Preprocessing? MIT-652 Data Mining Applications. Chapter 3: Data Preprocessing. Multi-Dimensional Measure of Data Quality

Data Preprocessing. Why Data Preprocessing? MIT-652 Data Mining Applications. Chapter 3: Data Preprocessing. Multi-Dimensional Measure of Data Quality Why Data Preprocessing? Data in the real world is dirty incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data e.g., occupation = noisy: containing

More information

DSC 201: Data Analysis & Visualization

DSC 201: Data Analysis & Visualization DSC 201: Data Analysis & Visualization Exploratory Data Analysis Dr. David Koop Python Support for Time The datetime package - Has date, time, and datetime classes -.now() method: the current datetime

More information

Visualizing Data: Freq. Tables, Histograms

Visualizing Data: Freq. Tables, Histograms Visualizing Data: Freq. Tables, Histograms Engineering Statistics Section 1.2 Josh Engwer TTU 25 January 2016 Josh Engwer (TTU) Visualizing Data: Freq. Tables, Histograms 25 January 2016 1 / 23 Descriptive

More information

Multistat2 1

Multistat2 1 Multistat2 1 2 Multistat2 3 Multistat2 4 Multistat2 5 Multistat2 6 This set of data includes technologically relevant properties for lactic acid bacteria isolated from Pasta Filata cheeses 7 8 A simple

More information

Nonparametric Regression and Generalized Additive Models Part I

Nonparametric Regression and Generalized Additive Models Part I SPIDA, June 2004 Nonparametric Regression and Generalized Additive Models Part I Robert Andersen McMaster University Plan of the Lecture 1. Detecting nonlinearity Fitting a linear model to a nonlinear

More information

Regression on SAT Scores of 374 High Schools and K-means on Clustering Schools

Regression on SAT Scores of 374 High Schools and K-means on Clustering Schools Regression on SAT Scores of 374 High Schools and K-means on Clustering Schools Abstract In this project, we study 374 public high schools in New York City. The project seeks to use regression techniques

More information

Spell out your full name (first, middle and last)

Spell out your full name (first, middle and last) Spell out your full name (first, middle and last) Be ready to share the following counts: Number of letters in your full name. Number of vowels Number of consonants Section 2-1 Organizing Data After completing

More information

DATA WAREHOUING UNIT I

DATA WAREHOUING UNIT I BHARATHIDASAN ENGINEERING COLLEGE NATTRAMAPALLI DEPARTMENT OF COMPUTER SCIENCE SUB CODE & NAME: IT6702/DWDM DEPT: IT Staff Name : N.RAMESH DATA WAREHOUING UNIT I 1. Define data warehouse? NOV/DEC 2009

More information

CSE 158 Lecture 2. Web Mining and Recommender Systems. Supervised learning Regression

CSE 158 Lecture 2. Web Mining and Recommender Systems. Supervised learning Regression CSE 158 Lecture 2 Web Mining and Recommender Systems Supervised learning Regression Supervised versus unsupervised learning Learning approaches attempt to model data in order to solve a problem Unsupervised

More information

ALGEBRA II A CURRICULUM OUTLINE

ALGEBRA II A CURRICULUM OUTLINE ALGEBRA II A CURRICULUM OUTLINE 2013-2014 OVERVIEW: 1. Linear Equations and Inequalities 2. Polynomial Expressions and Equations 3. Rational Expressions and Equations 4. Radical Expressions and Equations

More information

Using Large Data Sets Workbook Version A (MEI)

Using Large Data Sets Workbook Version A (MEI) Using Large Data Sets Workbook Version A (MEI) 1 Index Key Skills Page 3 Becoming familiar with the dataset Page 3 Sorting and filtering the dataset Page 4 Producing a table of summary statistics with

More information

Here is Kellogg s custom menu for their core statistics class, which can be loaded by typing the do statement shown in the command window at the very

Here is Kellogg s custom menu for their core statistics class, which can be loaded by typing the do statement shown in the command window at the very Here is Kellogg s custom menu for their core statistics class, which can be loaded by typing the do statement shown in the command window at the very bottom of the screen: 4 The univariate statistics command

More information

Multiple Linear Regression

Multiple Linear Regression Multiple Linear Regression Rebecca C. Steorts, Duke University STA 325, Chapter 3 ISL 1 / 49 Agenda How to extend beyond a SLR Multiple Linear Regression (MLR) Relationship Between the Response and Predictors

More information

IBMSPSSSTATL1P: IBM SPSS Statistics Level 1

IBMSPSSSTATL1P: IBM SPSS Statistics Level 1 SPSS IBMSPSSSTATL1P IBMSPSSSTATL1P: IBM SPSS Statistics Level 1 Version: 4.4 QUESTION NO: 1 Which statement concerning IBM SPSS Statistics application windows is correct? A. At least one Data Editor window

More information

Data Handling. Moving from A to A* Calculate the numbers to be surveyed for a stratified sample (A)

Data Handling. Moving from A to A* Calculate the numbers to be surveyed for a stratified sample (A) Moving from A to A* A* median, quartiles and interquartile range from a histogram (A*) Draw histograms from frequency tables with unequal class intervals (A) Calculate the numbers to be surveyed for a

More information

Dynamic Thresholding for Image Analysis

Dynamic Thresholding for Image Analysis Dynamic Thresholding for Image Analysis Statistical Consulting Report for Edward Chan Clean Energy Research Center University of British Columbia by Libo Lu Department of Statistics University of British

More information

UNIT 2 Data Preprocessing

UNIT 2 Data Preprocessing UNIT 2 Data Preprocessing Lecture Topic ********************************************** Lecture 13 Why preprocess the data? Lecture 14 Lecture 15 Lecture 16 Lecture 17 Data cleaning Data integration and

More information

DATA. Business Statistics

DATA. Business Statistics DATA Business Statistics CONTENTS The role of data The data matrix Data types Aspects of data Obtaining data Further study THE ROLE OF DATA Data refers to observed facts there are 82 persons in this train

More information

BUSINESS ANALYTICS. 96 HOURS Practical Learning. DexLab Certified. Training Module. Gurgaon (Head Office)

BUSINESS ANALYTICS. 96 HOURS Practical Learning. DexLab Certified. Training Module. Gurgaon (Head Office) SAS (Base & Advanced) Analytics & Predictive Modeling Tableau BI 96 HOURS Practical Learning WEEKDAY & WEEKEND BATCHES CLASSROOM & LIVE ONLINE DexLab Certified BUSINESS ANALYTICS Training Module Gurgaon

More information

To make sense of data, you can start by answering the following questions:

To make sense of data, you can start by answering the following questions: Taken from the Introductory Biology 1, 181 lab manual, Biological Sciences, Copyright NCSU (with appreciation to Dr. Miriam Ferzli--author of this appendix of the lab manual). Appendix : Understanding

More information