Mineração de Dados Aplicada

Size: px
Start display at page:

Download "Mineração de Dados Aplicada"

Transcription

1 Data Exploration August, 9 th 2017 DCC ICEx UFMG

2 Summary of the last session Data mining Data mining is an empiricism; It can be seen as a generalization of querying; It lacks a unified theory; It implies trades-off between quality and computational complexity; It should always be practiced in an ethical way (privacy concerns). 2 / 25

3 Summary of the last session The pattern discovery process The pattern discovery process is iterative; It is interactive, hence statistics and visualization techniques, on both the data and the patterns, are essential; It involves pre-processing steps to get a dataset that: is relevant to the analysis; can be processed by chosen data mining algorithms. 3 / 25

4 Summary of the last session Outline 1 Data structure 2 Data exploration and completion 3 Analyzing one single attribute 4 / 25

5 Data structure Outline 1 Data structure 2 Data exploration and completion 3 Analyzing one single attribute 5 / 25

6 Data structure Classical data structure Most datasets can be represented as tables that describe objects (the rows, whose order is meaningless) with attributes (the columns, whose order is meaningless): a 1 a 2... a n o 1 d 1,1 d 1,2... d 1,n o 2 d 2,1 d 2,2... d 2,n o m d m,1 d m,2... d m,n m is called size of the dataset; n its dimensionality. 6 / 25

7 Data structure Data streams A dataset with an infinite size is a data stream. The complexity of an algorithm processing it cannot depend on the number of objects seen so far. 7 / 25

8 Data structure The dataset is a sample The dataset with a finite size is seen as a sample, i. e., not all objects of study (usually in infinite number) are in the dataset. 8 / 25

9 Data structure The dataset is a sample The dataset with a finite size is seen as a sample, i. e., not all objects of study (usually in infinite number) are in the dataset. It is almost always assumed that, in the sample, the values taken by an attribute are independent and identically distributed and that they follow a known distribution whose parameters are estimated from the sample. 8 / 25

10 Data structure The dataset is a sample The dataset with a finite size is seen as a sample, i. e., not all objects of study (usually in infinite number) are in the dataset. It is almost always assumed that, in the sample, the values taken by an attribute are independent and identically distributed and that they follow a known distribution whose parameters are estimated from the sample. It is important to understand whether (or to what extent) the assumption holds. If not, many analyses do not apply. 8 / 25

11 Data structure More structured datasets Patterns can be searched inside one large word, sequence, graph (directed or not, weighted or not, labeled or not), with timestamps (including sounds), with spacial positions (including pictures), etc. 9 / 25

12 Data structure More structured datasets Patterns can be searched inside one large word, sequence, graph (directed or not, weighted or not, labeled or not), with timestamps (including sounds), with spacial positions (including pictures), etc. Studying and using a method processing such data can be the non-trivial step of your project. 9 / 25

13 Data structure More structured datasets Patterns can be searched inside one large word, sequence, graph (directed or not, weighted or not, labeled or not), with timestamps (including sounds), with spacial positions (including pictures), etc. Studying and using a method processing such data can be the non-trivial step of your project. The structured dataset can be broken into components, objects that are individually described. Attributes can be derived from the initial structure (degree in the graph, distance to a reference object, statistics on neighbors, etc.). 9 / 25

14 Data exploration and completion Outline 1 Data structure 2 Data exploration and completion 3 Analyzing one single attribute 10 / 25

15 Data exploration and completion Difficulties Data usually are: incomplete; 11 / 25

16 Data exploration and completion Difficulties Data usually are: incomplete; inconsistent; 11 / 25

17 Data exploration and completion Difficulties Data usually are: incomplete; inconsistent; uncertain/noisy or even plainly wrong; 11 / 25

18 Data exploration and completion Difficulties Data usually are: incomplete; inconsistent; uncertain/noisy or even plainly wrong; with some exceptions. 11 / 25

19 Data exploration and completion Difficulties Data usually are: incomplete; inconsistent; uncertain/noisy or even plainly wrong; with some exceptions. Never assume the data are perfect. Detect the problems (basic statistics and visualizations help a lot) and understand the limitations of the data generation/acquisition process. 11 / 25

20 Data exploration and completion Looking at specific objects It may be interesting to look at particular objects: those taking uncommon or extreme values; those that you particularly know; those that everybody knows. 12 / 25

21 Data exploration and completion Looking at specific objects It may be interesting to look at particular objects: those taking uncommon or extreme values; those that you particularly know; those that everybody knows. To do so, you can fire SQL queries (if the dataset is in a database) or POSIX commands (if in text files). 12 / 25

22 Data exploration and completion Using background knowledge It is essential to understand the application domain and to complement the dataset with additional attributes: taken from other sources; derived from existing attributes. 13 / 25

23 Data exploration and completion Using background knowledge It is essential to understand the application domain and to complement the dataset with additional attributes: taken from other sources; derived from existing attributes. Spam detection Given an , its date and the IP addresses sending and receiving it,... may help its classification as spam/ham., 13 / 25

24 Data exploration and completion Using background knowledge It is essential to understand the application domain and to complement the dataset with additional attributes: taken from other sources; derived from existing attributes. Spam detection Given an , its date and the IP addresses sending and receiving it, the presence of some words ( viagra, pr0n, etc.), the country of the sender, the match/mismatch between the language of the and that of the two countries, whether the day is holiday, the number of s from the sender in the dataset,... may help its classification as spam/ham. 13 / 25

25 Analyzing one single attribute Outline 1 Data structure 2 Data exploration and completion 3 Analyzing one single attribute 14 / 25

26 Analyzing one single attribute Typology of an attribute In 1946, Stanley Smith Stevens proposed to categorize the attributes into four types: Nominal (Boolean when there are only two categories); 15 / 25

27 Analyzing one single attribute Typology of an attribute In 1946, Stanley Smith Stevens proposed to categorize the attributes into four types: Nominal (Boolean when there are only two categories); Ordinal (with a partial or a total order); 15 / 25

28 Analyzing one single attribute Typology of an attribute In 1946, Stanley Smith Stevens proposed to categorize the attributes into four types: Nominal (Boolean when there are only two categories); Ordinal (with a partial or a total order); Interval-scaled (differences make sense but an arbitrary 0); 15 / 25

29 Analyzing one single attribute Typology of an attribute In 1946, Stanley Smith Stevens proposed to categorize the attributes into four types: Nominal (Boolean when there are only two categories); Ordinal (with a partial or a total order); Interval-scaled (differences make sense but an arbitrary 0); Ratio-scaled (ratios make sense). 15 / 25

30 Analyzing one single attribute Typology of an attribute In 1946, Stanley Smith Stevens proposed to categorize the attributes into four types: Nominal (Boolean when there are only two categories); Ordinal (with a partial or a total order); Interval-scaled (differences make sense but an arbitrary 0); Ratio-scaled (ratios make sense). Identifying the type of every attribute is essential. It tells what statistics and data mining algorithms are applicable and what operations are allowed to derive new attributes. 15 / 25

31 Analyzing one single attribute Centrality, dispersion and skewness Many (but not all) attributes take values that are distributed around one center. 16 / 25

32 Analyzing one single attribute Centrality, dispersion and skewness Many (but not all) attributes take values that are distributed around one center. If the attribute is interval-scaled, the dispersion is a measure of how much its values deviate from the center. 16 / 25

33 Analyzing one single attribute Centrality, dispersion and skewness Many (but not all) attributes take values that are distributed around one center. If the attribute is interval-scaled, the dispersion is a measure of how much its values deviate from the center. If the distribution is asymmetric around the center, it is said skewed. 16 / 25

34 Analyzing one single attribute Skewness Negative Skew Positive Skew c 2008 Rodolfo Hermans (from Wikimedia Commons) These diagrams are licensed under the Creative Commons Attribution ShareAlike 3.0 Unported License. 17 / 25

35 Analyzing one single attribute Basic statistics Estimating from the sample the center and the dispersion of the values taken by an attribute is very useful. The applicable statistics depend on the type of the attribute: nominal the mode, i. e., the most common value; 18 / 25

36 Analyzing one single attribute Basic statistics Estimating from the sample the center and the dispersion of the values taken by an attribute is very useful. The applicable statistics depend on the type of the attribute: nominal the mode, i. e., the most common value; ordinal the mode, the median (50% of the values are smaller, 50% greater), min and max; 18 / 25

37 Analyzing one single attribute Basic statistics Estimating from the sample the center and the dispersion of the values taken by an attribute is very useful. The applicable statistics depend on the type of the attribute: nominal the mode, i. e., the most common value; ordinal the mode, the median (50% of the values are smaller, 50% greater), min and max; interval-scaled all the above plus the arithmetic mean, the range (max min) and the standard deviation. 18 / 25

38 Analyzing one single attribute Basic statistics Estimating from the sample the center and the dispersion of the values taken by an attribute is very useful. The applicable statistics depend on the type of the attribute: nominal the mode, i. e., the most common value; ordinal the mode, the median (50% of the values are smaller, 50% greater), min and max; interval-scaled all the above plus the arithmetic mean, the range (max min) and the standard deviation. ratio-scaled all the above plus the geometric and harmonic means (for rates), the studentized range (difference of the z-scores of the largest and the smallest values), the coefficient of variation (ratio of the standard deviation and the mean), etc. 18 / 25

39 Analyzing one single attribute Robustness of a statistic A statistic is robust if extreme values (called outliers) do not affect it. A non-robust statistic computed from a sample may be very different from the one that would be obtained from all objects. 19 / 25

40 Analyzing one single attribute Robustness of a statistic A statistic is robust if extreme values (called outliers) do not affect it. A non-robust statistic computed from a sample may be very different from the one that would be obtained from all objects. The median is a robust statistic of centrality, whereas the arithmetic mean is not. The trimmed mean is the arithmetic mean computed after discarding the most extreme values (a small fraction to be chosen). It is a robust statistic. 19 / 25

41 Analyzing one single attribute Robustness of a statistic A statistic is robust if extreme values (called outliers) do not affect it. A non-robust statistic computed from a sample may be very different from the one that would be obtained from all objects. The median is a robust statistic of centrality, whereas the arithmetic mean is not. The trimmed mean is the arithmetic mean computed after discarding the most extreme values (a small fraction to be chosen). It is a robust statistic. The range is not a robust statistic of dispersion. The interquartile range (IRQ) is the range for the middle 50% of the values. It is a robust statistics. 19 / 25

42 Analyzing one single attribute Estimating the skewness Several statistics aim to measure the skewness of an interval-scaled attribute. The most common ones are the Pearson s skewness statistics: mean mode standard deviation ; mean median standard deviation ; the Pearson s moment coefficient of skewness (complicated formula with better statistical foundations). 20 / 25

43 Analyzing one single attribute Basic visualizations A histogram graphically represents the distribution of the values taken by an attribute (whatever its type). It requires partitioning the domain into intervals, base of rectangles whose areas are proportional with the number of values in the interval. Do not use pie charts. 21 / 25

44 Analyzing one single attribute Basic visualizations A histogram graphically represents the distribution of the values taken by an attribute (whatever its type). It requires partitioning the domain into intervals, base of rectangles whose areas are proportional with the number of values in the interval. Do not use pie charts. A box plot provides a simpler visualization of the distribution of an interval-scaled attribute. It shows the boundaries of the four quartiles, and either the min/max or the values 1.5IRQ below/above the first/third quartile and the values exceeding those thresholds. Those values are outliers (but their definition can be tuned modifying the 1.5 coefficient). 21 / 25

45 Analyzing one single attribute Histogram 22 / 25

46 Analyzing one single attribute Box plot 23 / 25

47 Analyzing one single attribute Structured attributes Objects can be described with attributes that are matrices/tensors, words, sequences, graphs (directed or not, weighted or not, labeled or not), with timestamps, with spacial positions, pictures, sounds, videos, etc. 24 / 25

48 Analyzing one single attribute Structured attributes Objects can be described with attributes that are matrices/tensors, words, sequences, graphs (directed or not, weighted or not, labeled or not), with timestamps, with spacial positions, pictures, sounds, videos, etc. Studying and using a method processing such data can be the non-trivial step of your project. 24 / 25

49 Analyzing one single attribute Structured attributes Objects can be described with attributes that are matrices/tensors, words, sequences, graphs (directed or not, weighted or not, labeled or not), with timestamps, with spacial positions, pictures, sounds, videos, etc. Studying and using a method processing such data can be the non-trivial step of your project. Properties of a structured attribute (its size, counts of patterns in it, dominant color, BPM, etc.) can substitute it, with loss of information. Metadata (author, creation date, tags, length, etc.) are valuable too. 24 / 25

50 License c These slides are licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. 25 / 25

Getting to Know Your Data

Getting to Know Your Data Chapter 2 Getting to Know Your Data 2.1 Exercises 1. Give three additional commonly used statistical measures (i.e., not illustrated in this chapter) for the characterization of data dispersion, and discuss

More information

Data Statistics Population. Census Sample Correlation... Statistical & Practical Significance. Qualitative Data Discrete Data Continuous Data

Data Statistics Population. Census Sample Correlation... Statistical & Practical Significance. Qualitative Data Discrete Data Continuous Data Data Statistics Population Census Sample Correlation... Voluntary Response Sample Statistical & Practical Significance Quantitative Data Qualitative Data Discrete Data Continuous Data Fewer vs Less Ratio

More information

Part I, Chapters 4 & 5. Data Tables and Data Analysis Statistics and Figures

Part I, Chapters 4 & 5. Data Tables and Data Analysis Statistics and Figures Part I, Chapters 4 & 5 Data Tables and Data Analysis Statistics and Figures Descriptive Statistics 1 Are data points clumped? (order variable / exp. variable) Concentrated around one value? Concentrated

More information

ECLT 5810 Data Preprocessing. Prof. Wai Lam

ECLT 5810 Data Preprocessing. Prof. Wai Lam ECLT 5810 Data Preprocessing Prof. Wai Lam Why Data Preprocessing? Data in the real world is imperfect incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate

More information

Road Map. Data types Measuring data Data cleaning Data integration Data transformation Data reduction Data discretization Summary

Road Map. Data types Measuring data Data cleaning Data integration Data transformation Data reduction Data discretization Summary 2. Data preprocessing Road Map Data types Measuring data Data cleaning Data integration Data transformation Data reduction Data discretization Summary 2 Data types Categorical vs. Numerical Scale types

More information

1.2. Pictorial and Tabular Methods in Descriptive Statistics

1.2. Pictorial and Tabular Methods in Descriptive Statistics 1.2. Pictorial and Tabular Methods in Descriptive Statistics Section Objectives. 1. Stem-and-Leaf displays. 2. Dotplots. 3. Histogram. Types of histogram shapes. Common notation. Sample size n : the number

More information

Chapter 2 Describing, Exploring, and Comparing Data

Chapter 2 Describing, Exploring, and Comparing Data Slide 1 Chapter 2 Describing, Exploring, and Comparing Data Slide 2 2-1 Overview 2-2 Frequency Distributions 2-3 Visualizing Data 2-4 Measures of Center 2-5 Measures of Variation 2-6 Measures of Relative

More information

Data Preprocessing. S1 Teknik Informatika Fakultas Teknologi Informasi Universitas Kristen Maranatha

Data Preprocessing. S1 Teknik Informatika Fakultas Teknologi Informasi Universitas Kristen Maranatha Data Preprocessing S1 Teknik Informatika Fakultas Teknologi Informasi Universitas Kristen Maranatha 1 Why Data Preprocessing? Data in the real world is dirty incomplete: lacking attribute values, lacking

More information

Summarising Data. Mark Lunt 09/10/2018. Arthritis Research UK Epidemiology Unit University of Manchester

Summarising Data. Mark Lunt 09/10/2018. Arthritis Research UK Epidemiology Unit University of Manchester Summarising Data Mark Lunt Arthritis Research UK Epidemiology Unit University of Manchester 09/10/2018 Summarising Data Today we will consider Different types of data Appropriate ways to summarise these

More information

Measures of Dispersion

Measures of Dispersion Measures of Dispersion 6-3 I Will... Find measures of dispersion of sets of data. Find standard deviation and analyze normal distribution. Day 1: Dispersion Vocabulary Measures of Variation (Dispersion

More information

Measures of Central Tendency. A measure of central tendency is a value used to represent the typical or average value in a data set.

Measures of Central Tendency. A measure of central tendency is a value used to represent the typical or average value in a data set. Measures of Central Tendency A measure of central tendency is a value used to represent the typical or average value in a data set. The Mean the sum of all data values divided by the number of values in

More information

CHAPTER-13. Mining Class Comparisons: Discrimination between DifferentClasses: 13.4 Class Description: Presentation of Both Characterization and

CHAPTER-13. Mining Class Comparisons: Discrimination between DifferentClasses: 13.4 Class Description: Presentation of Both Characterization and CHAPTER-13 Mining Class Comparisons: Discrimination between DifferentClasses: 13.1 Introduction 13.2 Class Comparison Methods and Implementation 13.3 Presentation of Class Comparison Descriptions 13.4

More information

Mean Tests & X 2 Parametric vs Nonparametric Errors Selection of a Statistical Test SW242

Mean Tests & X 2 Parametric vs Nonparametric Errors Selection of a Statistical Test SW242 Mean Tests & X 2 Parametric vs Nonparametric Errors Selection of a Statistical Test SW242 Creation & Description of a Data Set * 4 Levels of Measurement * Nominal, ordinal, interval, ratio * Variable Types

More information

Data can be in the form of numbers, words, measurements, observations or even just descriptions of things.

Data can be in the form of numbers, words, measurements, observations or even just descriptions of things. + What is Data? Data is a collection of facts. Data can be in the form of numbers, words, measurements, observations or even just descriptions of things. In most cases, data needs to be interpreted and

More information

Acquisition Description Exploration Examination Understanding what data is collected. Characterizing properties of data.

Acquisition Description Exploration Examination Understanding what data is collected. Characterizing properties of data. Summary Statistics Acquisition Description Exploration Examination what data is collected Characterizing properties of data. Exploring the data distribution(s). Identifying data quality problems. Selecting

More information

Chapter 3 - Displaying and Summarizing Quantitative Data

Chapter 3 - Displaying and Summarizing Quantitative Data Chapter 3 - Displaying and Summarizing Quantitative Data 3.1 Graphs for Quantitative Data (LABEL GRAPHS) August 25, 2014 Histogram (p. 44) - Graph that uses bars to represent different frequencies or relative

More information

STP 226 ELEMENTARY STATISTICS NOTES PART 2 - DESCRIPTIVE STATISTICS CHAPTER 3 DESCRIPTIVE MEASURES

STP 226 ELEMENTARY STATISTICS NOTES PART 2 - DESCRIPTIVE STATISTICS CHAPTER 3 DESCRIPTIVE MEASURES STP 6 ELEMENTARY STATISTICS NOTES PART - DESCRIPTIVE STATISTICS CHAPTER 3 DESCRIPTIVE MEASURES Chapter covered organizing data into tables, and summarizing data with graphical displays. We will now use

More information

Measures of Central Tendency

Measures of Central Tendency Page of 6 Measures of Central Tendency A measure of central tendency is a value used to represent the typical or average value in a data set. The Mean The sum of all data values divided by the number of

More information

3. Data Analysis and Statistics

3. Data Analysis and Statistics 3. Data Analysis and Statistics 3.1 Visual Analysis of Data 3.2.1 Basic Statistics Examples 3.2.2 Basic Statistical Theory 3.3 Normal Distributions 3.4 Bivariate Data 3.1 Visual Analysis of Data Visual

More information

Chapter 1. Looking at Data-Distribution

Chapter 1. Looking at Data-Distribution Chapter 1. Looking at Data-Distribution Statistics is the scientific discipline that provides methods to draw right conclusions: 1)Collecting the data 2)Describing the data 3)Drawing the conclusions Raw

More information

Name Date Types of Graphs and Creating Graphs Notes

Name Date Types of Graphs and Creating Graphs Notes Name Date Types of Graphs and Creating Graphs Notes Graphs are helpful visual representations of data. Different graphs display data in different ways. Some graphs show individual data, but many do not.

More information

CS6220: DATA MINING TECHNIQUES

CS6220: DATA MINING TECHNIQUES CS6220: DATA MINING TECHNIQUES 2: Data Pre-Processing Instructor: Yizhou Sun yzsun@ccs.neu.edu September 10, 2013 2: Data Pre-Processing Getting to know your data Basic Statistical Descriptions of Data

More information

Data Preprocessing. Why Data Preprocessing? MIT-652 Data Mining Applications. Chapter 3: Data Preprocessing. Multi-Dimensional Measure of Data Quality

Data Preprocessing. Why Data Preprocessing? MIT-652 Data Mining Applications. Chapter 3: Data Preprocessing. Multi-Dimensional Measure of Data Quality Why Data Preprocessing? Data in the real world is dirty incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data e.g., occupation = noisy: containing

More information

IAT 355 Visual Analytics. Data and Statistical Models. Lyn Bartram

IAT 355 Visual Analytics. Data and Statistical Models. Lyn Bartram IAT 355 Visual Analytics Data and Statistical Models Lyn Bartram Exploring data Example: US Census People # of people in group Year # 1850 2000 (every decade) Age # 0 90+ Sex (Gender) # Male, female Marital

More information

Data Preprocessing. Slides by: Shree Jaswal

Data Preprocessing. Slides by: Shree Jaswal Data Preprocessing Slides by: Shree Jaswal Topics to be covered Why Preprocessing? Data Cleaning; Data Integration; Data Reduction: Attribute subset selection, Histograms, Clustering and Sampling; Data

More information

STA 570 Spring Lecture 5 Tuesday, Feb 1

STA 570 Spring Lecture 5 Tuesday, Feb 1 STA 570 Spring 2011 Lecture 5 Tuesday, Feb 1 Descriptive Statistics Summarizing Univariate Data o Standard Deviation, Empirical Rule, IQR o Boxplots Summarizing Bivariate Data o Contingency Tables o Row

More information

Chapter 3. Descriptive Measures. Slide 3-2. Copyright 2012, 2008, 2005 Pearson Education, Inc.

Chapter 3. Descriptive Measures. Slide 3-2. Copyright 2012, 2008, 2005 Pearson Education, Inc. Chapter 3 Descriptive Measures Slide 3-2 Section 3.1 Measures of Center Slide 3-3 Definition 3.1 Mean of a Data Set The mean of a data set is the sum of the observations divided by the number of observations.

More information

Day 4 Percentiles and Box and Whisker.notebook. April 20, 2018

Day 4 Percentiles and Box and Whisker.notebook. April 20, 2018 Day 4 Box & Whisker Plots and Percentiles In a previous lesson, we learned that the median divides a set a data into 2 equal parts. Sometimes it is necessary to divide the data into smaller more precise

More information

STA Rev. F Learning Objectives. Learning Objectives (Cont.) Module 3 Descriptive Measures

STA Rev. F Learning Objectives. Learning Objectives (Cont.) Module 3 Descriptive Measures STA 2023 Module 3 Descriptive Measures Learning Objectives Upon completing this module, you should be able to: 1. Explain the purpose of a measure of center. 2. Obtain and interpret the mean, median, and

More information

Vocabulary. 5-number summary Rule. Area principle. Bar chart. Boxplot. Categorical data condition. Categorical variable.

Vocabulary. 5-number summary Rule. Area principle. Bar chart. Boxplot. Categorical data condition. Categorical variable. 5-number summary 68-95-99.7 Rule Area principle Bar chart Bimodal Boxplot Case Categorical data Categorical variable Center Changing center and spread Conditional distribution Context Contingency table

More information

The first few questions on this worksheet will deal with measures of central tendency. These data types tell us where the center of the data set lies.

The first few questions on this worksheet will deal with measures of central tendency. These data types tell us where the center of the data set lies. Instructions: You are given the following data below these instructions. Your client (Courtney) wants you to statistically analyze the data to help her reach conclusions about how well she is teaching.

More information

MATH& 146 Lesson 8. Section 1.6 Averages and Variation

MATH& 146 Lesson 8. Section 1.6 Averages and Variation MATH& 146 Lesson 8 Section 1.6 Averages and Variation 1 Summarizing Data The distribution of a variable is the overall pattern of how often the possible values occur. For numerical variables, three summary

More information

NOTES TO CONSIDER BEFORE ATTEMPTING EX 1A TYPES OF DATA

NOTES TO CONSIDER BEFORE ATTEMPTING EX 1A TYPES OF DATA NOTES TO CONSIDER BEFORE ATTEMPTING EX 1A TYPES OF DATA Statistics is concerned with scientific methods of collecting, recording, organising, summarising, presenting and analysing data from which future

More information

STA Module 4 The Normal Distribution

STA Module 4 The Normal Distribution STA 2023 Module 4 The Normal Distribution Learning Objectives Upon completing this module, you should be able to 1. Explain what it means for a variable to be normally distributed or approximately normally

More information

STA /25/12. Module 4 The Normal Distribution. Learning Objectives. Let s Look at Some Examples of Normal Curves

STA /25/12. Module 4 The Normal Distribution. Learning Objectives. Let s Look at Some Examples of Normal Curves STA 2023 Module 4 The Normal Distribution Learning Objectives Upon completing this module, you should be able to 1. Explain what it means for a variable to be normally distributed or approximately normally

More information

Further Maths Notes. Common Mistakes. Read the bold words in the exam! Always check data entry. Write equations in terms of variables

Further Maths Notes. Common Mistakes. Read the bold words in the exam! Always check data entry. Write equations in terms of variables Further Maths Notes Common Mistakes Read the bold words in the exam! Always check data entry Remember to interpret data with the multipliers specified (e.g. in thousands) Write equations in terms of variables

More information

Table of Contents (As covered from textbook)

Table of Contents (As covered from textbook) Table of Contents (As covered from textbook) Ch 1 Data and Decisions Ch 2 Displaying and Describing Categorical Data Ch 3 Displaying and Describing Quantitative Data Ch 4 Correlation and Linear Regression

More information

Chapter 3: Data Description - Part 3. Homework: Exercises 1-21 odd, odd, odd, 107, 109, 118, 119, 120, odd

Chapter 3: Data Description - Part 3. Homework: Exercises 1-21 odd, odd, odd, 107, 109, 118, 119, 120, odd Chapter 3: Data Description - Part 3 Read: Sections 1 through 5 pp 92-149 Work the following text examples: Section 3.2, 3-1 through 3-17 Section 3.3, 3-22 through 3.28, 3-42 through 3.82 Section 3.4,

More information

Data analysis using Microsoft Excel

Data analysis using Microsoft Excel Introduction to Statistics Statistics may be defined as the science of collection, organization presentation analysis and interpretation of numerical data from the logical analysis. 1.Collection of Data

More information

AND NUMERICAL SUMMARIES. Chapter 2

AND NUMERICAL SUMMARIES. Chapter 2 EXPLORING DATA WITH GRAPHS AND NUMERICAL SUMMARIES Chapter 2 2.1 What Are the Types of Data? 2.1 Objectives www.managementscientist.org 1. Know the definitions of a. Variable b. Categorical versus quantitative

More information

Averages and Variation

Averages and Variation Averages and Variation 3 Copyright Cengage Learning. All rights reserved. 3.1-1 Section 3.1 Measures of Central Tendency: Mode, Median, and Mean Copyright Cengage Learning. All rights reserved. 3.1-2 Focus

More information

LESSON 3: CENTRAL TENDENCY

LESSON 3: CENTRAL TENDENCY LESSON 3: CENTRAL TENDENCY Outline Arithmetic mean, median and mode Ungrouped data Grouped data Percentiles, fractiles, and quartiles Ungrouped data Grouped data 1 MEAN Mean is defined as follows: Sum

More information

Overview of Clustering

Overview of Clustering based on Loïc Cerfs slides (UFMG) April 2017 UCBL LIRIS DM2L Example of applicative problem Student profiles Given the marks received by students for different courses, how to group the students so that

More information

Let s take a closer look at the standard deviation.

Let s take a closer look at the standard deviation. For this illustration, we have selected 2 random samples of 500 respondents each from the voter.xls file we have been using in class. We recorded the ages of each of the 500 respondents and computed the

More information

Basic Statistical Terms and Definitions

Basic Statistical Terms and Definitions I. Basics Basic Statistical Terms and Definitions Statistics is a collection of methods for planning experiments, and obtaining data. The data is then organized and summarized so that professionals can

More information

Preprocessing Short Lecture Notes cse352. Professor Anita Wasilewska

Preprocessing Short Lecture Notes cse352. Professor Anita Wasilewska Preprocessing Short Lecture Notes cse352 Professor Anita Wasilewska Data Preprocessing Why preprocess the data? Data cleaning Data integration and transformation Data reduction Discretization and concept

More information

2. (a) Briefly discuss the forms of Data preprocessing with neat diagram. (b) Explain about concept hierarchy generation for categorical data.

2. (a) Briefly discuss the forms of Data preprocessing with neat diagram. (b) Explain about concept hierarchy generation for categorical data. Code No: M0502/R05 Set No. 1 1. (a) Explain data mining as a step in the process of knowledge discovery. (b) Differentiate operational database systems and data warehousing. [8+8] 2. (a) Briefly discuss

More information

CHAPTER 1. Introduction. Statistics: Statistics is the science of collecting, organizing, analyzing, presenting and interpreting data.

CHAPTER 1. Introduction. Statistics: Statistics is the science of collecting, organizing, analyzing, presenting and interpreting data. 1 CHAPTER 1 Introduction Statistics: Statistics is the science of collecting, organizing, analyzing, presenting and interpreting data. Variable: Any characteristic of a person or thing that can be expressed

More information

Lecture 7: Decision Trees

Lecture 7: Decision Trees Lecture 7: Decision Trees Instructor: Outline 1 Geometric Perspective of Classification 2 Decision Trees Geometric Perspective of Classification Perspective of Classification Algorithmic Geometric Probabilistic...

More information

Data Mining By IK Unit 4. Unit 4

Data Mining By IK Unit 4. Unit 4 Unit 4 Data mining can be classified into two categories 1) Descriptive mining: describes concepts or task-relevant data sets in concise, summarative, informative, discriminative forms 2) Predictive mining:

More information

2.1 Objectives. Math Chapter 2. Chapter 2. Variable. Categorical Variable EXPLORING DATA WITH GRAPHS AND NUMERICAL SUMMARIES

2.1 Objectives. Math Chapter 2. Chapter 2. Variable. Categorical Variable EXPLORING DATA WITH GRAPHS AND NUMERICAL SUMMARIES EXPLORING DATA WITH GRAPHS AND NUMERICAL SUMMARIES Chapter 2 2.1 Objectives 2.1 What Are the Types of Data? www.managementscientist.org 1. Know the definitions of a. Variable b. Categorical versus quantitative

More information

Chapter 2: Descriptive Statistics

Chapter 2: Descriptive Statistics Chapter 2: Descriptive Statistics Student Learning Outcomes By the end of this chapter, you should be able to: Display data graphically and interpret graphs: stemplots, histograms and boxplots. Recognize,

More information

1.3 Graphical Summaries of Data

1.3 Graphical Summaries of Data Arkansas Tech University MATH 3513: Applied Statistics I Dr. Marcel B. Finan 1.3 Graphical Summaries of Data In the previous section we discussed numerical summaries of either a sample or a data. In this

More information

Univariate descriptives

Univariate descriptives Univariate descriptives Johan A. Elkink University College Dublin 18 September 2014 18 September 2014 1 / Outline 1 Graphs for categorical variables 2 Graphs for scale variables 3 Frequency tables 4 Central

More information

Intrusion Detection using NASA HTTP Logs AHMAD ARIDA DA CHEN

Intrusion Detection using NASA HTTP Logs AHMAD ARIDA DA CHEN Intrusion Detection using NASA HTTP Logs AHMAD ARIDA DA CHEN Presentation Overview - Background - Preprocessing - Data Mining Methods to Determine Outliers - Finding Outliers - Outlier Validation -Summary

More information

Create a bar graph that displays the data from the frequency table in Example 1. See the examples on p Does our graph look different?

Create a bar graph that displays the data from the frequency table in Example 1. See the examples on p Does our graph look different? A frequency table is a table with two columns, one for the categories and another for the number of times each category occurs. See Example 1 on p. 247. Create a bar graph that displays the data from the

More information

MATH 112 Section 7.2: Measuring Distribution, Center, and Spread

MATH 112 Section 7.2: Measuring Distribution, Center, and Spread MATH 112 Section 7.2: Measuring Distribution, Center, and Spread Prof. Jonathan Duncan Walla Walla College Fall Quarter, 2006 Outline 1 Measures of Center The Arithmetic Mean The Geometric Mean The Median

More information

Chapter 5snow year.notebook March 15, 2018

Chapter 5snow year.notebook March 15, 2018 Chapter 5: Statistical Reasoning Section 5.1 Exploring Data Measures of central tendency (Mean, Median and Mode) attempt to describe a set of data by identifying the central position within a set of data

More information

3. Data Preprocessing. 3.1 Introduction

3. Data Preprocessing. 3.1 Introduction 3. Data Preprocessing Contents of this Chapter 3.1 Introduction 3.2 Data cleaning 3.3 Data integration 3.4 Data transformation 3.5 Data reduction SFU, CMPT 740, 03-3, Martin Ester 84 3.1 Introduction Motivation

More information

Learning Log Title: CHAPTER 7: PROPORTIONS AND PERCENTS. Date: Lesson: Chapter 7: Proportions and Percents

Learning Log Title: CHAPTER 7: PROPORTIONS AND PERCENTS. Date: Lesson: Chapter 7: Proportions and Percents Chapter 7: Proportions and Percents CHAPTER 7: PROPORTIONS AND PERCENTS Date: Lesson: Learning Log Title: Date: Lesson: Learning Log Title: Chapter 7: Proportions and Percents Date: Lesson: Learning Log

More information

2. Data Preprocessing

2. Data Preprocessing 2. Data Preprocessing Contents of this Chapter 2.1 Introduction 2.2 Data cleaning 2.3 Data integration 2.4 Data transformation 2.5 Data reduction Reference: [Han and Kamber 2006, Chapter 2] SFU, CMPT 459

More information

Common Core Vocabulary and Representations

Common Core Vocabulary and Representations Vocabulary Description Representation 2-Column Table A two-column table shows the relationship between two values. 5 Group Columns 5 group columns represent 5 more or 5 less. a ten represented as a 5-group

More information

Math 120 Introduction to Statistics Mr. Toner s Lecture Notes 3.1 Measures of Central Tendency

Math 120 Introduction to Statistics Mr. Toner s Lecture Notes 3.1 Measures of Central Tendency Math 1 Introduction to Statistics Mr. Toner s Lecture Notes 3.1 Measures of Central Tendency lowest value + highest value midrange The word average: is very ambiguous and can actually refer to the mean,

More information

Descriptive Statistics Descriptive statistics & pictorial representations of experimental data.

Descriptive Statistics Descriptive statistics & pictorial representations of experimental data. Psychology 312: Lecture 7 Descriptive Statistics Slide #1 Descriptive Statistics Descriptive statistics & pictorial representations of experimental data. In this lecture we will discuss descriptive statistics.

More information

IT 403 Practice Problems (1-2) Answers

IT 403 Practice Problems (1-2) Answers IT 403 Practice Problems (1-2) Answers #1. Using Tukey's Hinges method ('Inclusionary'), what is Q3 for this dataset? 2 3 5 7 11 13 17 a. 7 b. 11 c. 12 d. 15 c (12) #2. How do quartiles and percentiles

More information

MATH& 146 Lesson 10. Section 1.6 Graphing Numerical Data

MATH& 146 Lesson 10. Section 1.6 Graphing Numerical Data MATH& 146 Lesson 10 Section 1.6 Graphing Numerical Data 1 Graphs of Numerical Data One major reason for constructing a graph of numerical data is to display its distribution, or the pattern of variability

More information

More Summer Program t-shirts

More Summer Program t-shirts ICPSR Blalock Lectures, 2003 Bootstrap Resampling Robert Stine Lecture 2 Exploring the Bootstrap Questions from Lecture 1 Review of ideas, notes from Lecture 1 - sample-to-sample variation - resampling

More information

The basic arrangement of numeric data is called an ARRAY. Array is the derived data from fundamental data Example :- To store marks of 50 student

The basic arrangement of numeric data is called an ARRAY. Array is the derived data from fundamental data Example :- To store marks of 50 student Organizing data Learning Outcome 1. make an array 2. divide the array into class intervals 3. describe the characteristics of a table 4. construct a frequency distribution table 5. constructing a composite

More information

M7D1.a: Formulate questions and collect data from a census of at least 30 objects and from samples of varying sizes.

M7D1.a: Formulate questions and collect data from a census of at least 30 objects and from samples of varying sizes. M7D1.a: Formulate questions and collect data from a census of at least 30 objects and from samples of varying sizes. Population: Census: Biased: Sample: The entire group of objects or individuals considered

More information

Chapter 6: DESCRIPTIVE STATISTICS

Chapter 6: DESCRIPTIVE STATISTICS Chapter 6: DESCRIPTIVE STATISTICS Random Sampling Numerical Summaries Stem-n-Leaf plots Histograms, and Box plots Time Sequence Plots Normal Probability Plots Sections 6-1 to 6-5, and 6-7 Random Sampling

More information

Frequency Distributions

Frequency Distributions Displaying Data Frequency Distributions After collecting data, the first task for a researcher is to organize and summarize the data so that it is possible to get a general overview of the results. Remember,

More information

HS Mathematics Item Specification C1 TP

HS Mathematics Item Specification C1 TP Task Model 1 Multiple Choice, single correct response S-ID.A.1 Represent data with plots on the real number line (dot plots, histograms, and box plots). 1. The student will be able to represent data on

More information

STA Module 2B Organizing Data and Comparing Distributions (Part II)

STA Module 2B Organizing Data and Comparing Distributions (Part II) STA 2023 Module 2B Organizing Data and Comparing Distributions (Part II) Learning Objectives Upon completing this module, you should be able to 1 Explain the purpose of a measure of center 2 Obtain and

More information

STA Learning Objectives. Learning Objectives (cont.) Module 2B Organizing Data and Comparing Distributions (Part II)

STA Learning Objectives. Learning Objectives (cont.) Module 2B Organizing Data and Comparing Distributions (Part II) STA 2023 Module 2B Organizing Data and Comparing Distributions (Part II) Learning Objectives Upon completing this module, you should be able to 1 Explain the purpose of a measure of center 2 Obtain and

More information

Interactive Math Glossary Terms and Definitions

Interactive Math Glossary Terms and Definitions Terms and Definitions Absolute Value the magnitude of a number, or the distance from 0 on a real number line Addend any number or quantity being added addend + addend = sum Additive Property of Area the

More information

Data Mining and Analytics. Introduction

Data Mining and Analytics. Introduction Data Mining and Analytics Introduction Data Mining Data mining refers to extracting or mining knowledge from large amounts of data It is also termed as Knowledge Discovery from Data (KDD) Mostly, data

More information

An Introduction to Minitab Statistics 529

An Introduction to Minitab Statistics 529 An Introduction to Minitab Statistics 529 1 Introduction MINITAB is a computing package for performing simple statistical analyses. The current version on the PC is 15. MINITAB is no longer made for the

More information

DATA PREPROCESSING. Pronalaženje skrivenog znanja Bojan Furlan

DATA PREPROCESSING. Pronalaženje skrivenog znanja Bojan Furlan DATA PREPROCESSING Pronalaženje skrivenog znanja Bojan Furlan WHY DO WE NEED TO PREPROCESS THE DATA? Raw data contained in databases is unpreprocessed, incomplete, and noisy. For example, the databases

More information

Week 4: Describing data and estimation

Week 4: Describing data and estimation Week 4: Describing data and estimation Goals Investigate sampling error; see that larger samples have less sampling error. Visualize confidence intervals. Calculate basic summary statistics using R. Calculate

More information

Data Preprocessing. Chapter Why Preprocess the Data?

Data Preprocessing. Chapter Why Preprocess the Data? Contents 2 Data Preprocessing 3 2.1 Why Preprocess the Data?........................................ 3 2.2 Descriptive Data Summarization..................................... 6 2.2.1 Measuring the Central

More information

3 Graphical Displays of Data

3 Graphical Displays of Data 3 Graphical Displays of Data Reading: SW Chapter 2, Sections 1-6 Summarizing and Displaying Qualitative Data The data below are from a study of thyroid cancer, using NMTR data. The investigators looked

More information

(Refer Slide Time: 00:50)

(Refer Slide Time: 00:50) Programming, Data Structures and Algorithms Prof. N.S. Narayanaswamy Department of Computer Science and Engineering Indian Institute of Technology Madras Module - 03 Lecture 30 Searching Unordered linear

More information

Use of GeoGebra in teaching about central tendency and spread variability

Use of GeoGebra in teaching about central tendency and spread variability CREAT. MATH. INFORM. 21 (2012), No. 1, 57-64 Online version at http://creative-mathematics.ubm.ro/ Print Edition: ISSN 1584-286X Online Edition: ISSN 1843-441X Use of GeoGebra in teaching about central

More information

Statistical Analysis of Metabolomics Data. Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte

Statistical Analysis of Metabolomics Data. Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte Statistical Analysis of Metabolomics Data Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte Outline Introduction Data pre-treatment 1. Normalization 2. Centering,

More information

Chapter 2: Understanding Data Distributions with Tables and Graphs

Chapter 2: Understanding Data Distributions with Tables and Graphs Test Bank Chapter 2: Understanding Data with Tables and Graphs Multiple Choice 1. Which of the following would best depict nominal level data? a. pie chart b. line graph c. histogram d. polygon Ans: A

More information

How individual data points are positioned within a data set.

How individual data points are positioned within a data set. Section 3.4 Measures of Position Percentiles How individual data points are positioned within a data set. P k is the value such that k% of a data set is less than or equal to P k. For example if we said

More information

MATH11400 Statistics Homepage

MATH11400 Statistics Homepage MATH11400 Statistics 1 2010 11 Homepage http://www.stats.bris.ac.uk/%7emapjg/teach/stats1/ 1.1 A Framework for Statistical Problems Many statistical problems can be described by a simple framework in which

More information

MKTG 460 Winter 2019 Solutions #1

MKTG 460 Winter 2019 Solutions #1 MKTG 460 Winter 2019 Solutions #1 Short Answer: Data Analysis 1. What is a data table and how are the data values organized? A data table stores the data values for variables across different observations,

More information

Chapter 5. Understanding and Comparing Distributions. Copyright 2012, 2008, 2005 Pearson Education, Inc.

Chapter 5. Understanding and Comparing Distributions. Copyright 2012, 2008, 2005 Pearson Education, Inc. Chapter 5 Understanding and Comparing Distributions The Big Picture We can answer much more interesting questions about variables when we compare distributions for different groups. Below is a histogram

More information

What are we working with? Data Abstractions. Week 4 Lecture A IAT 814 Lyn Bartram

What are we working with? Data Abstractions. Week 4 Lecture A IAT 814 Lyn Bartram What are we working with? Data Abstractions Week 4 Lecture A IAT 814 Lyn Bartram Munzner s What-Why-How What are we working with? DATA abstractions, statistical methods Why are we doing it? Task abstractions

More information

Univariate Statistics Summary

Univariate Statistics Summary Further Maths Univariate Statistics Summary Types of Data Data can be classified as categorical or numerical. Categorical data are observations or records that are arranged according to category. For example:

More information

Mean,Median, Mode Teacher Twins 2015

Mean,Median, Mode Teacher Twins 2015 Mean,Median, Mode Teacher Twins 2015 Warm Up How can you change the non-statistical question below to make it a statistical question? How many pets do you have? Possible answer: What is your favorite type

More information

Ex.1 constructing tables. a) find the joint relative frequency of males who have a bachelors degree.

Ex.1 constructing tables. a) find the joint relative frequency of males who have a bachelors degree. Two-way Frequency Tables two way frequency table- a table that divides responses into categories. Joint relative frequency- the number of times a specific response is given divided by the sample. Marginal

More information

ECLT 5810 Clustering

ECLT 5810 Clustering ECLT 5810 Clustering What is Cluster Analysis? Cluster: a collection of data objects Similar to one another within the same cluster Dissimilar to the objects in other clusters Cluster analysis Grouping

More information

1. To condense data in a single value. 2. To facilitate comparisons between data.

1. To condense data in a single value. 2. To facilitate comparisons between data. The main objectives 1. To condense data in a single value. 2. To facilitate comparisons between data. Measures :- Locational (positional ) average Partition values Median Quartiles Deciles Percentiles

More information

Secondary 1 Vocabulary Cards and Word Walls Revised: June 27, 2012

Secondary 1 Vocabulary Cards and Word Walls Revised: June 27, 2012 Secondary 1 Vocabulary Cards and Word Walls Revised: June 27, 2012 Important Notes for Teachers: The vocabulary cards in this file match the Common Core, the math curriculum adopted by the Utah State Board

More information

Grade 6 Math Vocabulary

Grade 6 Math Vocabulary Ratio, Rate, Unit Rate, Rate Unit Grade 6 Math Vocabulary Ratio, Rate, Unit Rate, Rate Unit Representing Equivalent Ratios Double Number Line 0 Ratio: An ordered pair of numbers which are not both zero.

More information

Glossary Common Core Curriculum Maps Math/Grade 6 Grade 8

Glossary Common Core Curriculum Maps Math/Grade 6 Grade 8 Glossary Common Core Curriculum Maps Math/Grade 6 Grade 8 Grade 6 Grade 8 absolute value Distance of a number (x) from zero on a number line. Because absolute value represents distance, the absolute value

More information

Test Bank for Privitera, Statistics for the Behavioral Sciences

Test Bank for Privitera, Statistics for the Behavioral Sciences 1. A simple frequency distribution A) can be used to summarize grouped data B) can be used to summarize ungrouped data C) summarizes the frequency of scores in a given category or range 2. To determine

More information

MATH 1070 Introductory Statistics Lecture notes Descriptive Statistics and Graphical Representation

MATH 1070 Introductory Statistics Lecture notes Descriptive Statistics and Graphical Representation MATH 1070 Introductory Statistics Lecture notes Descriptive Statistics and Graphical Representation Objectives: 1. Learn the meaning of descriptive versus inferential statistics 2. Identify bar graphs,

More information