5b. Descriptive Statistics - Part II

Transcription

1 5b. Descriptive Statistics - Part II In this lab we ll cover how you can calculate descriptive statistics that we discussed in class. We also learn how to summarize large multi-level databases efficiently, and how to export the results as a CSV file. Such a table of summary statistics can be the basis for graphical representation that we cover in the subsequent labs. You will need to download the aspen.csv data from the course website. 5b.1 Descriptive statistics functions in R Load the dataset (aspen.csv with the variables DBH and VOLUME) into R, attach the data, and try out the following functions that will give you summary statistics. Insert the variable name that you are interested in between the brackets: mean() median() max() min() range() which.max() which.min() var() sd() quantile() Percentiles can be calculated with the quantile function by adding a vector of the percentiles that you want. Bold indicates what you may customize for your own purposes: e.g: quantile(dbh, c(0.025,0.05,0.95,0.975)) There are also these handy functions to calculate multiple statistics: fivenum(), summary() 5b.2 Multi-level summaries in R Load the dataset from Lab 2 that you created by merging the protein data in. (i.e. you should have the independent variables FARM and VARIETY and the dependent variables YIELD and PROTEIN, plus an ID). If you had trouble with this, you can also download the combined dataset from the website under today s lab. dat1=read.csv("lentils_with_protein.csv") head(dat1) attach(dat1) Now, we want to do some more complex data summaries. For example we may want to know the mean and standard deviation of yield and protein content for each lentil variety at each farm. With the formulas above, this would require a lot of programming to subset the data and calculate the statistics of interest. Thankfully there is a nice R-package PLYR that does that automatically for us, but you need to install it first: From the menu in R, choose Packages, then Install packages. Next, you get a pop-up window, where you can choose a download site. They have all exactly the same contents, so you may pick something nearby, for example Canada (BC). Then, you get a pop-up window, where you can choose a package. Scroll down and double-click plyr. The first thing you always have to do with additional packages is to load them into computer memory. You have to rerun this library command everytime you re-open R. Only the base-funcrionality is loaded by default, not the additional packages that you may install. library(plyr)

2 Now we are ready to calculate some summaries. The function ddply() first needs to be told which dataset to work with (dat1), then given a list of independent variables to make the summaries by (in parenthesis and preceded by a period). Here we use the summarize function of (it can do other stuff), and, and finally, you provide the function or formula that should be applied to the data. Try: ddply(dat1,.(farm), summarise, myield=mean(yield)) ddply(dat1,.(farm,variety), summarise, myield=mean(yield)) ddply(dat1,.(farm), summarise, mprotein=mean(protein)) The result from the last command may strike you as odd. However, this is how many functions work by default. If you have missing values, the result of the function will be a missing value, unless you add a comment that removes missing values before the calculation. ddply(dat1,.(farm), summarise, mprotein=mean(protein, na.rm=t)) This is true and worth to keep in mind for all statistical functions. Try: mean(dat1$protein) versus mean(dat1$protein, na.rm=t) sd(dat1$protein) versus sd(dat1$protein, na.rm=t) range(dat1$protein) versus range(dat1$protein, na.rm=t) To top things off, let s calculate the mean and standard error of yield and protein content. We have not covered standard errors in class, but I want to demonstrate that you can use ddply() with any formula of interest. To get the standard error, you have to divide the standard deviation by the square root of the number of observations. We can get the number of observation with the function length(). For protein content, we need the number of non-missing observations, which we can get with length(protein [!is.na(protein)]). We ll write the results into a table (dat2) and then export the result as a CSV. dat2 = ddply(dat1,.(farm,variety), summarise, myield = mean(yield), seyield = sd (YIELD)/sqrt(length(YIELD)), mprotein = mean(protein, na.rm=t), seprotein = sd (PROTEIN, na.rm=t) /sqrt(length (PROTEIN[!is.na(PROTEIN)])) ) write.csv(dat2,"lentil_summary.csv") This may be a little complicated if you are not used to programming, but you see that it is a powerful way to quickly summarize your data. It s the numerical equivalent to: boxplot(yield~variety*farm). Just for the record, the R base package has a function that is similar to ddply(), but it s output is a bit ugly, it loses the names of group variables, and it can t do complex functions. The general syntax is aggregate(variable(s), by=list(class variable(s)), FUN=statistic of your choice). For example: aggregate(yield, by=list(farm,variety), FUN=mean) or a two-variable example: aggregate(dat[,c("yield", "PROTEIN")], by=list(farm,variety), FUN=mean, na.rm=t) There is no reason to use this, rather than the PLYR package, however.

3 5b.3 Multi-level summaries in SAS with UNIVARIATE and BOXPLOT SAS has a very useful and powerful procedure to calculate summary statistics. Let s try PROC UNIVARIATE for the aspen.csv dataset. This gives you all statistics you ever wanted to know: proc univariate data=lentils; var YIELD; There is a more useful alternative to specify which statistics you actually want, and to write them into an output file that you can export. Let s try this with the lentil.csv dataset. For this dataset we can also calculate statistics for multiple levels, by specifying one or more by variables. In the code below, we ask the number of non-missing values (n), the average (mean) of yield proc sort data=lentils; by FARM VARIETY; required sorting for by variables! proc univariate data=lentils noprint; var YIELD; by FARM VARIETY; output out=summary_stats1 n=nyield mean=myield ; You can get any number statistics for multiple variables. Again, merge in your protein dataset and try several or all of the following options: N NMISS MAX MIN RANGE SUM MEAN MODE VAR STD CV STDMEAN MEDIAN P1 P5 P10 P90 P95 P99 Q1 Q3 QRANGE SKEWNESS KURTOSIS proc sort data=lentils; by FARM VARIETY; required sorting for by variables! proc univariate data=lentils noprint; var YIELD PROTEIN; by FARM VARIETY; output out=summary_stats2 q1=q1yield q1protein q3=q3yield q3protein ; In the output out= file, you can add as many variables and statistics as you like, but each has to have a different name. A good idea is to choose a combination of the statistic and variable name as above. You don t see these file names in your SAS spreadsheet, but you do when you export your results. To export your table of summary statistics, it s actually fine to use the SAS wizard from the file menu, but you can also code it to save you some clicks: proc export data=summary_stats2; outfile="c:\your path\your file name.csv" dbms=csv replace;

4 You can also do some quick boxplots in SAS, but the graphical quality is poor and customization options are minimal. Don t waste too much time with graphics in SAS. The data also needs to be sorted correctly for this procedure, but sorting incorrectly, for example by both Farm and Variety can trick SAS into displaying multi-level boxplots. proc sort data=lentils; by variety; (also try: proc sort data=lentils; by farm variety; ) goptions reset=all; proc boxplot data=lentils; plot yield*variety; Simple boxplot Add two rows to your lentils.csv file with a mild and a far outlier (for separate Farm*Variety groups) and see how they display with this extended code: proc boxplot data=lentils; Customized boxplot. ID id; This line labels outliers so that you can identify them. plot yield*variety /cboxfill=ligr cboxes=black; insetgroup n mean q1 q3; This line adds statistics of choice for each variety. 5b.4 Pivot tables with Excel One of the rare occasions where Excel beats R and SAS in functionality are Pivot tables. This is the equivalent to PROC UNIVARIATE and the AGGREGATE command. If you just want some quick summary statistics from a database, Pivot tables are often easier to use and you have more flexibility in arranging your table of summary statistics. I highly recommend exploring this functionality with your own data. AGGREGATE and UNIVARIATE are of course still useful if the calculation of summary statistics is part of a larger program, or if you want to calculate statistics for a large number of dependent variables (for example you will quickly get tired of dragging variables to the value field if you have 365 daily measurements over the course of a year). Open lentils.csv in Excel and choose the ribbon Insert > Pivot Table Calculate means for Farm and Variety combinations: 1. drag Yield into the value field and choose Value field settings from the drop-down list, then select average 2. arrange the table in various ways by dragging Farm and Variety into the Rows / Columns / Filter fields. 3. The variables in Filter get a drop down list, where you can choose subsets (e.g. remove Farm1 data by unchecking the box.

5 CHALLENGE: You now have all the statistical tools to manage data tables and calculate summary statistics. In the next labs we will cover graphical presentation of quantitative data, and this will complete the toolkit that you need for the first round of project websites and project presentations. For your project, you should address a real scientific or applied questions (even if you have to make up the data). You are likely on the right track if you can answer the following questions for your project (and you may also answer them for your two datasets that you have come across so far: aspen.csv from Lab 1 and lentils.csv from lab 2): 1. What is the population that the investigators probably want to study? 2. What are your sampling units or experimental units? 3. How would you select a sample from a population in a way that allows you to generalize? 4. What is the predictor variable(s)? 5. What is the response variable(s)? 6. Is the predictor variable manipulated in an experiment or simply observed? 7. What is your idea of the causal relationship between the response variable and the predictor variable (scientific hypothesis)? 8. If you don t have a non-trivial scientific hypothesis, what are the practical implications of the findings? Can you use the relationship to predict or control the response variable? Later we will ask: 9. What is the appropriate statistical method to study the relationship between the response variable and the predictor variable (statistical hypothesis)? 10. Assuming, you find a statistically significant relationship, can you think of a reasonable alternative explanation for the apparent link between the variables? 11. How accurate will the prediction or control be?