3. Data Tables & Data Management

Transcription

1 3. Data Tables & Data Management In this lab, we will learn how to create and manage data tables for analysis. We work with a very simple example, so it is easy to see what the code does. In your own projects the same code can apply to manage and modify large datasets. You may need to merge multiple data sheets, subset data tables to focus your analysis on particular treatments or locations, fix errors in your datasets, transform or modify variables, or re-arrange data that has not been entered in a format fit for analysis. It is good to do all these operations with scripts that leave a trail of changes and corrections that you did. If you change your mind later, none of your original data has been touched. You simply change a bit of code and execute your script again to do a slightly modified analysis (e.g. with or without outliers) Creating a data table Let s practice creating and importing a data table into R and SAS. Here is an experiment, where three lentil varieties (A, B, C) are tested at two farms. What you see below are the lentil yield in kg/ha that were observed in randomized experimental plots at each of two farms. This is the field-layout of the plots: Farm 1 Farm (A) 515 (B) 545 (B) 540 (C) 760 (A) 502 (C) 492 (B) 480 (B) 740 (A) 505 (C) 690 (A) 510 (C) 163 (A) 405 (B) 389 (B) 375 (C) 163 (A) 381 (C) 387 (B) 375 (B) 176 (A) 385 (C) 168 (A) 400 (C) What are the Populations, Samples, Units, Independent Variables, Dependent Variables, Data? (possibly not all of the terms apply) How does this information need to be arranged into a data table, then? Enter your data table in the correct format into Excel. 3.2 Import data to R and SAS To recap what you learned in the first lab, write code that imports your new data table into R and SAS. I would like you to be able to start R or SAS, import your data, look at your data table and be ready for analysis in less than 10 seconds (check the first lab for setup if you don t remember) If you have difficulty setting yourself up to do this in less than 10 seconds, talk to me or the TA!!! If you have designed your data table in the right format, R and SAS will immediately know how to analyze it correctly. In fact, the layout of your data table defines what analysis your software package will do. Often there are different valid options to define Populations, Units, Independent Variables and Dependent Variables, resulting in different data tables, and as a consequence different valid analyses for different research questions. As a check, call for an Analysis of Variance in R: summary(aov(yield~farm*variety)). Note that your variable names and capitalization may be different, and remember that you need an attach()command for your data table to tell R where to find the variables. The sums of square for Farm should be

2 3.3 Sub-setting datasets In many cases, you don t want to analyze your entire dataset in one go, you can select particular rows and columns and get rid of the rest of the data if you want to focus your analysis in a particular aspect of interest. In R, you ask for rows, columns, and individual cells of a table with this extremely important and widely used command: [,]. We will use this a lot in the future, so remember what those simple brackets specify: data table name [ row(s), column(s) ] Let s start by calling individual cells of a data table, where I assume that the data table name is dat. Change dat to whatever name you use: dat[1,2] should return the value of the first row and the second column of the dat table. Check this with fix(dat), and close the spreadsheet to continue dat[5,3] returns the value of the fifth row and the third column. You can also assign this value to a variable: A=dat[5,3], then call A We can also call entire rows or columns by leaving one of the fields between brackets blank: dat[4,] will return the fourth row, dat[,3] will return the third column, and Y=dat[,3], will create a vector from the third column that you can recall later: Y Third, we can get sections of data by specifying an array: dat[1:3,1:3] will return the first three rows of the first three columns dat[,2:3] will return the second and third column, and farm2=dat[13:24,], will return rows 13-24, which probably is the data for farm 2, if you entered the data in this order. We can now use this subset of farm2 for analysis. Finally, you can use the minus sign to drop particular rows or columns. Try dat[-20,] or farm1=dat[-(13:24),]. Note that you need brackets around arrays here. Now, what if your data is not arranged in a nice way to subset it by using row numbers or column numbers? For example, how would we remove a particular lentil variety from the analysis? You can use conditional statements to select rows: The conditional operators: ==,!=, >, <, >=, <= (meaning: equal, non-equal, larger, smaller, larger or equal, smaller or equal). Note that the is it equal? question is symbolized by == not by =. The single = is reserved for the command make it equal to!. You may combine or stack several conditional statements with &,,! (meaning: AND, OR, NOT)

3 and using brackets () to determine the hierarchy of conditional statements. For example, dat[variety!="c",] will remove variety C from the dataset. Note that you need to put the C in quotation marks. That is necessary for character/factor variables. For numerical variables you would leave the quotation marks out. f1vb=dat[(farm=="farm1")&(variety=="b"),] will just select variety B from farm 1 You could now check, say, the mean of yield for this subset mean(f1vb$yield). Note that you can use the $ symbol or specify in which data table R is supposed to look for the variable YIELD (the general format is table$variable). If you just type mean(yield) R will look in the default dataset, which you specified with the attach() command. You can also select columns with variable names, instead of specifying column numbers: Use the vector command c() to specify a list of variables you want to keep, and insert this into the subset command [,]. For example, dat[,c("id","yield")] will just retain the variables ID and YIELD. 3.4 Calculations and transformations in data tables Although there are other options in R, carrying out any type of calculation on your data table should be done with the transform command, which has the syntax: transform(table, operation1, operation2, ) For example we may want to change the units from kg/acre to kg/hectares or make a log-transformation of our YIELD variable: transform(dat, YIELD=YIELD*0.4) transforms the units of yield from kg/acre to kg/hectares and over-writes your original variable values. You could also create a new variable with a different unit and retain the old one: transform(dat, YIELD_HA=YIELD*0.4) You can also do two or more operations in one transform statement: transform(dat, YIELD_HA=YIELD*0.4, LOG_YIELD=log(YIELD)) You can make this change permanent by over-writing your original dataset dat=transform(dat, YIELD_HA=YIELD*0.4, LOG_YIELD=log(YIELD)) But generally, it s good practice to assign the results of transform operation to a new dataset, especially if you over-write variables. I like to number the resulting datasets consecutively, which allows me to go back and fix errors along the way if I do lots of modifications: dat2=transform(dat, YIELD_HA=YIELD*0.4, LOG_YIELD=log(YIELD)) Check your new table with: fix(dat2) You can also do transformations with a conditional command, which has the general syntax: variable=ifelse(condition, value or operation if true, value or operation if false ) To give an imaginary example where this is useful: farmer 1 may have entered the data in kg/acre and farmer 2 in kg/ha and then send the data to you for analysis.

4 Here is how you would correct this by changing the units to kg/ha only in farm1 data: transform(dat, YIELD=ifelse(FARM=="Farm1",YIELD*0.4,YIELD)) You can do many fancy things with if-else statements, such as generating class variables from continuous data, by stacking multiple if-else statements. An example that creates a new class variable, indicating low, medium, and high yields: transform(dat, YIELDCLASS=ifelse(YIELD<300, "Low", ifelse(yield<500, "Medium", "High"))) Another application is the automated deletion of outliers. For example, we may know that yields cannot reasonably exceed 3000 kg/ha. If there are values larger than this, we may assume that they are typos or other errors and then decide to remove them automatically from the database. Use the fix() command to edit the data table and create an unreasonable yield value. Then, clean the database with this command, where NA means missing or not applicable : transform(dat, YIELD=ifelse(YIELD>3000,NA,YIELD)) 3.5 Merging multiple data tables One reason why you always want to use a unique ID for your research subjects (or rows in the data table) is that you can use IDs to reliably add new measurements to an existing database. For example, in addition to the yield measurement, you may also get protein content for your lentil varieties. However, this is an expensive measurement and you could only afford to pay a lab for 6 data points (two for each variety from Farm 1). Here is the lab report: ID: PROTEIN: You can add this data as columns: Enter the data in Excel, import a second data table to R. Then, merge it into the existing database with the following command: merge(dat, newdat, by="id", all.x=t, all.y=f). The options all.x=t, all.y=f means that you want to keep all observations from the first dataset (x refers to dat, T means TRUE) but only matching observations from the second dataset (y is newdat, F means FALSE). This is the most common Left-merge operation, where you have a master dataset on the left and a secondary dataset to add on the right side of the table as additional column(s). Check the results with fix(dat). On very rare occasions, you may use cbind(table/vector1, table/vector2, table/vector3, ) to merge data without a criteria (ID). This is generally dangerous, because the commands will slap the two tables together without checking if the rows match. If the two tables are differently sorted, have missing rows, etc. you are in trouble. If datasets don t match, R simpy repeats data from the beginning or drops rows, and you don t get error messages! Try cbind(dat,newdat) That s not what you want! You may also need to add rows to your data table: Adding rows of two data tables (also known as appending data tables) is executed with rbind(table1, table2, table3, ). As with cbind(), the two tables have to match exactly for a successful merger, but you will get error messages if they don t match. So, rbind() is generally safe to use.

5 If you want to append 2 data tables with different column names you can use rbind.fill(table1,table2,table3, ) which is part of the plyr package. You will need to install this package to use this command. This function will put a NA value in the cells where the columns names do not match between data tables. 3.6 The same in SAS: Here is how to do everything that you have done above in SAS. There is not a repeat of explanations what we are doing and why, so this exercise is only effective if you have done parts 2.1 to 2.5. To make the code a little easier to adapt to your own purposes, I bolded all the pieces of code that are names or things you would customize for a different task. Here are some good practices for running R scripts Execute code in small chunks (highlight a data step, hit run). Before you execute code, have the LOG window visible to see any error messages. After you execute code check the result in the left panel under Explorer-tab >Libraries > Work Only do the next step after checking data tables, and after you closed all table windows Importing and exporting CSV files: proc import out=dat datafile="c:\lab2\lentils.csv" dbms=csv replace; getnames=yes; datarow=2; guessingrows=10000; proc export data=result; outfile="c:\your path\your file name.csv" dbms=csv replace; The data step does (almost) everything: SAS has no equivalent to the [,] function in R, so we start with the data step, that does all the rest (calculations, subsetting, merging, and transformations). The SAS data step creates a new data table y (data y;) from an existing table x (set x;). Next come one or a bunch of commands that modify the new data table and this end of the data step is indicated with a command, which executes the data step. If you use the same name in the data line as in the set line, your original data table will be overwritten. Mathematical and logical operations data dat2; yield_ha=yield*0.4; log_yield=log(yield); Calculations and transformations data dat3; Conditional operations (. means missing ) if yield>700 then yield=.; if farm= "Farm1" then yield=yield*0.4;

6 data dat4; Deleting rows with missing values set dat3; if yield=. then delete; data dat5; Conditional operations with else statement if yield<300 then yieldclass="low"; else if yield<500 then yieldclass="med"; else yieldclass="high"; Subsetting Creating subsets of your database works exactly the same as the conditional statement, because we can follow it up with then delete (which deletes the row), something R unfortunately doesn t allow. Do delete or select variables (columns), you can either use the a keep or a drop statement that do what their name implies. data farm1; Creating subsets (deleting rows) if farm="farm2" then delete; data farm2; if farm ne "Farm2" then delete; data dat6; drop yield; Creating subsets (deleting columns) data dat7; keep id yield; Merging Merging datasets by ID requires that they are both sorted by the ID filed. So you often have to execute a sort procedure first (not necessary in our case because the datasets are already sorted by ID). Here we merge the protein.csv dataset from section 2.5, which you have to import as newdat. proc sort data=data; by ID; proc sort data=newdat; by ID; Merge columns data merged; merge dat newdat; by ID;

7 Merging additional data rows can be done with the update command. You have to have unique IDs to add data as rows. If the second dataset has duplicate IDs, data points will selectively replaced in the old dataset (hence update), which can also be very handy. proc sort data=dat; by ID; proc sort data=newdat; by ID; Merge rows data all; update farm1 farm2; by ID; Remove duplicate With the sort procedure you can also search for and remove duplicates in large databases. This can be based on duplicate records (identical rows) with option NODUPREC. To delete duplicate IDs or duplicates in a combination of variables, use NODUPKEY. The first record that has duplicates specified in the by statement will be kept and all subsequent records will be deleted. Create some duplicate records and duplicate IDs in Excel, re-import, and see how the sort procedure works: proc sort data=test1 out=test1clean dupout=test1dup nodupkey; by ID; proc sort data=test1 out=test1clean dupout=dest1dup noduprec; by ID; BTW you can, of course, sort your data by one or more variables ascending and/or descending. Add descending behind the applicable variable: proc sort data=dat; by variety descending farm; CHALLENGE (in R): Each of these steps can (and should) be completed in a single line of R code. 1. Open a new session of R and open a new script. 2. Create a single-column table with the character values: A1, A2, A3, B1, B2, B3, C1, C2, C3 3. Create a single column table with the numerical values 2 through 18, counting by 2, in order. 4. Merge the two columns into a new two-column data table named data. 5. Rename the columns ID and X. 6. Create a third column named Y which is equal to X multiplied by Calculate the mean of 1) the entire X column, 2) just the first three rows of the X column, 3) the Y values from the A1, B1, and C1 site, 4) the Y values that are greater than Re-order the rows so that they are in reverse-alphabetical of ID (i.e. C3, C2, C1, B3, etc.). 9. Create a new data table with just the A1 to A3 sites. 10. Transpose this table of A sites. 11. Export your complete data table to your working directory as a CSV file.

8 Some useful R commands when dealing with data: Here are some other commands that can be handy on occasion. This gives you a complete set of tools and reference of R commands for data table management and manipulation. Rename variables: names(dat)=c("id", "X", "Y", "Z"). Note that the length of the vector must match the number of variable you have (four in this case). Assign the ID to row names: row.names(dat)=dat$id. Note that the default row names are consecutive numbers, so in the lentil dataset this does not change anything. Transpose your data table: If you only have numerical values in your data table, you can transpose it (switch rows and columns): t_dat=t(dat). This is handy if you have a table where experimental units are in columns, and not in rows as they should be. Row names become variable names, so it s handy to run the row.names() function above first. Find duplicates: unique(dat1) or dat1[!duplicated(dat1),] removes all rows that are exact duplicates. dat1[duplicated(dat1),] returns the duplicate rows. dat1[duplicated(dat1[,c("id")]),] removes all rows with one or more identical IDs. dat1[!duplicated(dat1[,c("id")]),] returns the rows with duplicate IDs. Ordering data: dat[order(x),] orders rows by variable X. dat[order(x,y),] orders rows by variable X, then variable Y. dat[order(x,-y),]. Orders rows by variable X, then DESCENDING by variable Y. Merging by multiple variables: If you don t have a unique ID but a variable combination that is unique, it can be handy to merge data by two or more variables, for example by latitude and longitude values that together make a unique location ID: merge(dat1,dat2,by=c("lat","long")). Advanced merge options: (1) Left merge, (2) right merge, (3) keep all rows, (4) keep only matching rows: Here you specify which is the master data table, which one will be merged in, and whether you keep observations that don t occur in both datasets. Also, if your IDs have different names, you can identify them for each dataset separately: (1) dat3=merge(dat1,dat2,by.x="id",by.y="id",all.x=t,all.y=f) (2) dat3=merge(dat1,dat2,by.x="id",by.y="id",all.x=f,all.y=t) (3) dat3=merge(dat1,dat2,by.x="id",by.y="id",all.x=t,all.y=t) (4) dat3=merge(dat1,dat2,by.x="id",by.y="id",all.x=f,all.y=f) Reference for Import, export, and data table checks: dat1=read.csv("name.csv") to import data write.csv(results,file="myresults.csv") to export data fix(dat1) to check and edit cell with double-click attach(dat1) to set a table as default to look for variables length(dat1) returns the number of columns nrow(dat1) returns the number of rows str(dat1) to check variable types. Alternatively, check/transform objects/variables with: is.numeric(), is.character(), is.vector(), is.matrix(), is.data.frame() as.numeric(), as.character(), as.vector(), as.matrix(), as.data.frame() Dealing with missing values: Sometimes you need to modify missing values. You may want to set zeros to missing, if it really means that the measurement was not taken, or vice versa if missing has a biological meaning of zero, e.g. plant dead means zero productivity. You can use the ifelse() statement and conditional subsets explained above, or these handy functions: dat1[is.na(dat1)]=0 sets missing values to 0 in entire dataset dat2=na.omit(dat1) delete rows with missing values in any variable dat2=transform(dat1, VAR2=ifelse(is.na(VAR1),NA,VAR2))modify a second variable (here: set to missing) based on missing values in a first variable. Take a random subsample of your data: The code below takes 300 random samples without replacement. Can be handy for bootstrapping or to run quick test analyses on very large datasets. sample1<-dat1[sample(1:nrow(dat1), 300,replace=FALSE),]