Visualizing univariate data 1

Transcription

1 Visualizing univariate data 1 Xijin Ge SDSU Math/Stat Broad perspectives of exploratory data analysis(eda) EDA is not a mere collection of techniques; EDA is a new altitude and philosophy as to how we dissect a data set; what we look for; how we look; and how we interpret. It is true that EDA heavily uses the collection of techniques that we call "statistical graphics", but it is not identical to statistical graphics per se. Simply put, the EDA altitude is to play with your data first, visualize you data first, then we can talk about models / inferences etc. Student Activity (15minutes) The data file (data1.txt) contains 1003 numbers as a result of a measurement. Available here: Explore these numbers using whatever software you like. If you are asked to write a one paragraph to describe this data, what will you write? Do not turn to the following pages Graphical EDA techniques for univariate What can you do with a set of numbers? 1. Have a look at the data. Any text editor or Excel (Yes, Excel can open text files). 2. Summary Statistics. I will use my favorite statistical computing software R. a. Read the data and show summary statistics. Make sure to change the default work directory to where the data file is downloaded. File Change Dir > x = scan("data1.txt") #read in data using the scan() function Read 1003 items >x > summary(x) Min. 1st Qu. Median Mean 3rd Qu. Max Plots. The 4-Plot techniques are simple, efficient, and powerful for the routine testing of underlying assumptions: i. run sequence plot (Y i versus i). ii. lag plot (Y i versus Y i-1 ) iii. histogram (counts versus subgroups of Y) iv. normal probability plot (ordered Y versus theoretical ordered Y) 4. It is very easy to generate these plots in R. Try these: >plot(x) # run sequence plot 5. >lag.plot(x) # lag plot 1 1 This workshop is supported by National Science Foundation RII Track I. Page 1 of 5

2 6. >hist(x) # This is default histogram works for most cases, but we could refine it. 7. >hist(x,br=100,xlim=c(0,100) ) 8. >qqnorm(x) # normal probability plot 9. >qqline(x) 10. See results in Figure 3. Figure 1. 4-Plot for random numbers Figure 2. 4-plot of data1. There are some clearly remarkable features in this dataset. Page 2 of 5

3 The four EDA plots above are used to test the underlying assumptions: Fixed Location: If the fixed location assumption holds, then the run sequence plot will be flat and non-drifting. Fixed Variation: If the fixed variation assumption holds, then the vertical spread in the run sequence plot will be the approximately the same over the entire horizontal axis. Randomness: If the randomness assumption holds, then the lag plot will be structureless and random. Fixed Distribution: If the fixed distribution assumption holds, in particular if the fixed normal distribution holds, then the histogram will be bell-shaped, and the normal probability plot will be linear. In our case, the run sequence plot in Figure 3 shows the order that the data are presented in the file is NOT random. Lag plot has some structure, i.e., two big clusters are visible. Histogram shows the presence of outliers. The histogram is not symmetrical. Nor is it looks like a bell-curve. It has two peaks. The normal QQ plot is not a straight line. What might explain the data with distribution like figure 3? To answer this question, we need to connect these plots with the domain knowledge, namely the situation that these data are collected. Here are some possible scenarios: If the numbers represent height of people in some unit and people that as tall as 100 unit is impossible, then in this situation, the outliers are deemed to be errors in measurement or data recording. We will need to remove these data points before ANY further analysis. After removing the outliers, we have two peaks in this datasets. One possible reason is that there are actually two major races in this population. If the numbers are dimensions (length, height, width) of one component that we purchased and are going to be used in assembling cars, then this is a serious problem to have components like this. Is the supplier of this component purchased from two manufacturers? If it is from the same manufacturer, then are there two production lines? Sometimes, defective or biased measuring device can cause this. We might used two inaccurate rulers to measure these products. Related to this, imagine that a thermometer is used to measure temperature in a weather station somewhere. After 5 years usage, one component in this thermometer broke down and caused to drift upward. But we are unaware of this and continue to use it for another 5 years. We the 4-plot will look-like in this situation? Outliers and non-normality are often very serious problems. Identification of outliers is important since they could severely affect the whole analysis if not properly handled. Normality is assumed in almost all statistical tests. Deviation of the observed distribution from normal Page 3 of 5

4 makes many powerful statistics tools useless. Note that some datasets can be transformed to normal distribution. Log-transformation and square-root transformations are two methods we often use to bring data, especially these that span a very large range, close to normality. If data is severely skewed, we could even choose to discritelize the data, or bin it. Student Activity (10minutes) Explore data points in the first peak. 11. Related to the outlier problem is a technique called boxplot, which can be invoked in R by simply: >boxplot(x) Figure 3. A boxplot of dataset Autocorrelation plot is an extension of lag plot by changing the time lag. Autocorrelation can be used to test if there s any time dependency in the data. where C h is the autocovariance function and C o is the variance function Note--R h is between -1 and +1. See figure 5 for examples of autocorrelation plots. To generate autocorrelation plots in R, use > acf(x) Page 4 of 5

5 ACF Figure 4. Examples of autocorrelation plots. Series data Lag Figure 5 Autocorrelation plot for data1. This shows there is strong positive autocorrelation and that the sequence of data points is not random. Page 5 of 5