GenStat Data Manipulation workshop 1

Transcription

1 For 16th edition GenStat Data Manipulation workshop 1 Contents 1. Re-ordering factor levels 2. Transforming data calculating a new column 3. Recalculating a column 4. Duplicating a column 5. Clearing data from a session 6. Restricting or filtering data 7. Reference Course pre-requisites: attendees should have attended a GenStat Introduction workshop (or be familiar with GenStat). 1. Re-ordering factor levels These data come from Ott & Longnecker (2001) (Example 8.4, p.404) and consist of dissolved oxygen (ppm) in seawater at four distances from the mouth of the Mississippi River. To begin, Start GenStat Create a new GenStat spreadsheet using the Excel spreadsheet Distance oxygen.xls Set the columns Distance and sample as factors. Save the spreadsheet as a GenStat spreadsheet (.GSH) called Distance oxygen.gsh Note: Each level of a factor in GenStat has three attributes: an ordinal, a level, and a label. The ordinal is simply a number that indicates the order the levels of the factor will be displayed. The ordinal values start with 1 for the first level displayed. The level is another number - it may take any value and is usually a treatment or dose value. (It can also be the same as the ordinal, i.e., 1, 2, 3,.). The label is a text descriptor for each level of the factor (often an abbreviated treatment name or other identifier e.g., cultivar name). Generate a list of means for each distance: Stats > Summary Statistics > Summary Tables Variate: oxygen Groups: Distance Click [Run] and the following should appear in the Output window: Mean Distance 1 km km km km 4.60 Notice that in the summary statistics above, the order of the Distance factor levels is not very sensible: 5 km is last in the list when (logically) it should be second in the list. 2013, Library, Teaching & Learning, Lincoln University page 1

2 GenStat originally read in the data as text and therefore sorted the levels alphabetically instead of numerically. We can change this order so that it is correct for further output. Re-order the factor levels: Select the spreadsheet Spread > Factor > Reorder Levels or A dialog box will appear (Figure 1). You may now sort the levels manually or automatically. To manually sort the levels: Select 5 km and click the [Up] button twice to move the 5 km into the correct place (Figure 2). To automatically sort the levels: Click the [Sort] button Choose the Sort Type: select either By Numerical Part or Numerical prefix (eg 2A, 10A) and click [OK]. When the order of the distances is correct (i.e., 1 km, 5 km, 10 km, 20 km), click [OK]. Figure 1 Reorder Factor dialog box. The levels are in incorrect numerical order. Figure 2 The 5 km level has been moved up to the correct position. Redo the summary table: Return to the Summary Tables dialog. Click [Run] to calculate the summary again. The distances should now appear in the desired order: Mean Distance 1 km km km km , Library, Teaching & Learning, Lincoln University page 2

3 2. Transforming data calculating a new column During data analysis, it is common to have to transform data. Sometimes this is to meet the assumptions of particular analysis methods. For example, ANOVA and ordinary least squares regression methods both assume that variances are homogeneous. To calculate a new column of data in a spreadsheet: First, select the spreadsheet, then select Spread > Calculate > Column The Calculate dialog box will appear. Click [Functions ]. The Calculate Functions dialog box will appear. From the Function: drop-down list, choose Logarithm to base 10 (Figure 3). Figure 3 Function drop-down list. Specify oxygen as X by double-clicking on oxygen from the Available Data: list (Figure 4): Figure 4 The Calculate Functions dialog box ready for the calculation log 10 (oxygen). Click [OK] to return to the Calculate dialog box and type Log_oxygen in the Save result in: box (Figure 5): Figure 5 The Calculate dialog box showing the log 10 (oxygen) calculation that will be saved as the column Log_oxygen in the original spreadsheet named Distance oxygen.gsh. 2013, Library, Teaching & Learning, Lincoln University page 3

4 Click [Run] and the following warning message will appear: This message appears because one of the values for oxygen is 0 and log(0) is undefined. Any calculation that gives an error will return a missing value to the spreadsheet (Figure 6): calculated column missing value Figure 6 Spreadsheet with the calculated log 10 (oxygen) values. Note (1) that Row 9 contains a missing value (represented by *) because Row 9 has an oxygen value of 0, and log 10 (0) is undefined. Note (2) the yellow shading on Log in the column title Log_oxygen indicating that this is a calculated column. One way to avoid the zeroes problem when calculating log values is to use log(x + 1) instead of log(x). To do this: Go back to the Calculate dialog box (Figure 5). Edit the formula to read LOG10(oxygen+1)and click [Run]. The spreadsheet should now contain the recalculated values of Log_oxygen (Figure 7). Figure 7 Spreadsheet after the successful calculation of log 10 (oxygen+1). Note that Row 9 now contains a Log_oxygen value of , Library, Teaching & Learning, Lincoln University page 4

5 3. Recalculating a column An alternative way to recalculate a spreadsheet column (rather than using the Calculate dialog box) is: Select the spreadsheet then use Spread > Calculate > Recalculate Edit the formula in the dialog box as necessary (Figure 8). Figure 8 The Recalculate Spreadsheet dialog box can be used to edit and recalculate an existing spreadsheet column formula. Note: Unlike Excel, if you change or edit the data in your GenStat spreadsheet, a calculated column will NOT be automatically updated, and will NOT automatically reflect any changes that you make to the data. If you change any data used in calculations, you will need to manually recalculate the calculated column using Spread > Calculate > Recalculate 4. Duplicating a column Sometimes you need to duplicate a column of data perhaps to perform calculations or edits on one column while keeping the original column intact. GenStat provides a simple way of duplicating columns. In this example, we will duplicate the Distance factor column and at the same time change the new column to a variate. Select the Distance column in the spreadsheet Distance oxygen.gsh Use: Spread > Column > Duplicate (or ) to produce the Duplicate Column dialog box. Select Variate in the Create as list at the right of the dialog box. GenStat provides a default New Column Name: of Distance_1 which you may want to change. I suggest something like Distance_v or vdistance so you can see it is a variate for Distance (Figure 9). Figure 9 The Duplicate Column dialog box used for duplicating the factor Distance as a variate named Distance_v. 2013, Library, Teaching & Learning, Lincoln University page 5

6 Click [OK]. You may then see the following warning message appear (Figure 10). Click either [Yes] or [No] depending on whether you want to see the summary in the Output window. Figure 10 A warning message that can appear when converting text labels to numbers. The resulting spreadsheet (Figure 11) shows both the factor Distance (notice the red! alongside the column heading) and the variate Distance_v. Note the following: Note that GenStat has used the numerical part of the Distance label for the values of the variate named Distance_v, (i.e., 1 km => 1, 5 km => 5, etc). When converting a factor to a variate, the way the factor is converted depends on how the factor levels are displayed. If the values of the factor are displayed as ordinals or levels, then the variate will contain these same numbers. If the values of the factor are displayed as text labels (as they are here), then GenStat will attempt to convert the text to numbers for the variate values. The names of data structures in GenStat are case-sensitive. That is, Distance, distance, DISTANCE, and distance etc. would all be considered to be different names (and hence different columns). You can use this property when you create duplicate or related columns, but you do then need to remember which column is which. Figure 11 Spreadsheet after duplicating the factor Distance as a variate named Distance_v. 2013, Library, Teaching & Learning, Lincoln University page 6

7 5. Clearing data and output Sometimes you need to clear all the data from GenStat and start with new data. Clearing data from the GenStat server: Note: Close all spreadsheets (saving them if necessary) Use: Data > Clear All Data or [Ctrl+D] Press [Yes] If you do not close open spreadsheets before clearing the data, then the data from those spreadsheets will be automatically re-loaded back into the server next time you use a menu. This provides an easy way to selectively delete or keep data by closing unwanted spreadsheets and leaving any wanted spreadsheets open. Clearing output from the Output window: Usually when you clear the data, you will also want to clear the output. Go to the Output window. Clear all the text from the Output window by using Edit > Delete from the main menu, then click [Yes]. Alternatively, to remove just some of the text in the Output window select the unwanted output and again use Edit > Delete from the main menu. (Use the deleting it.) button to recover a section of output if you change your mind immediately after 6. Restricting or filtering data One of the most powerful and useful data manipulations in GenStat is the ease with which you can restrict or filter data. Not all statistics packages offer this feature (e.g., Minitab doesn t) and some others only offer a lesser version of it (e.g., Select Cases in Systat and SPSS). Filtering is used to select a subset of the data for subsequent analysis, e.g. to include data from a certain Site only, or to exclude data from a certain Block, or to exclude or include data with missing values. GenStat differs from most other statistics programs in that after applying one restriction to the data, you can apply a second, a third, or even more restrictions, to restrict the data further and further. In the following exercise, you will open a GenStat spreadsheet, summarise the data, apply a restriction, then do a calculation using the restricted data, summarise the data again, and then lastly remove the restriction. First load the data: Load the data file: Long-term plots spider data.gsh. This file contains data collected from some long-term ecology plots at Lincoln University. The experiment consists of four mowing treatments and two nitrogen treatments applied to 4 5 m plots arranged in a randomised block design (McLachlan, 2000). 2013, Library, Teaching & Learning, Lincoln University page 7

8 Summarise the data: Generate a list of means and sample sizes for each treatment: Stats > Summary Statistics > Summary Tables Variate: Number Groups: Mowing Display: Means, No. of Observations Click [Run]. The output should be as follows: Nobservd Mean Mowing M MI CL MI CR MR CL Restrict the data to exclude the MI CR (Mowing infrequently, Clippings removed) treatment. Select the spreadsheet. Select the Mowing column. Select Spread > Restrict/Filter > To Groups (factor levels) from the main menu (or click on the Filter Spreadsheet by Groups button, ) The Restrict Units on Factor Mowing dialog box will appear (Figure 12). Make sure the Factor: box contains Mowing Click on MI CR in the Selected Levels: list. Select Exclude as the Restriction type, then click [OK]. Figure 12 Restriction dialog box showing that the MI CR level of the Mowing factor will be excluded. 2013, Library, Teaching & Learning, Lincoln University page 8

9 The restriction should now be evident in the spreadsheet notice the red horizontal lines at the top left of the speadsheet and that rows (rows 1, 6, ) are missing (Figure 13). Also notice the status bar at the bottom of the screen. When a spreadsheet is active, the status bar shows how many rows and columns there are in the spreadsheet. In this case, the status bar shows to indicate that 24 of the 32 rows have been included (8 rows have been excluded), and that there are 6 columns. Figure 13 Spreadsheet showing data after the MI CR level of the Mowing factor has been excluded. To display the excluded rows in the spreadsheet, click on the at the top right of the spreadsheet (above the scrollbar). There should now be rows displayed in red in the spreadsheet (e.g., row 1, row 6, Figure 14). Click this cross. Excluded rows of data (in red) from the MI CR level of Mowing. Figure 14 Spreadsheet showing data after the MI CR level of the Mowing factor has been excluded and the button has been pressed to reveal the excluded rows in red. Press the button again to hide the excluded rows. 2013, Library, Teaching & Learning, Lincoln University page 9

10 Return to the Summary Tables dialog box (Stats > Summary Statistics > Summary Tables ). Click [Run] to display the summary again: Nobservd Mean Mowing M MI CL MI CR 0 * MR CL With the restriction in place, the MI CR level of the Mowing treatment has 0 observed values and the mean cannot be calculated (shown as a missing value * ). Compare these results to the results from before the restriction. Calculating new columns when restrictions are in place Calculate the log of the Number variable using log10(number). (Refer back to Section 2 for instructions on how to do this). Name the newly transformed column Log_Number. By default GenStat will remove the restriction before calculating the new column but you do get a choice (Figure 15): Figure 15 Restriction Warning dialog. Click on the Help button,, to see detailed information about the options then select either Remove and reapply or Leave restriction. Adding another Restriction GenStat allows multiple restrictions to be used at one time. The data are currently restricted to exclude all those plots with grass clippings removed (Mowing = MI CR). The following exercise shows one way to further restrict the data so that any plots with nitrogen added (Nitrogen = 1) are also excluded. Select the Nitrogen column in the spreadsheet. Spread > Restrict/Filter > To Groups (factor levels) The Restrict Units on Factor Nitrogen dialog appears. Make sure the Factor: box contains Nitrogen. Click on 1 in the Selected levels: list. 2013, Library, Teaching & Learning, Lincoln University page 10

11 Select Restriction type : Exclude Existing restrictions : Combine with New Click [OK] and notice how the spreadsheet display has changed. Also notice the status bar now shows to indicate that only 12 of the 32 rows are currently unrestricted. Click [Run] to display the summary again: Calculate a summary of the unrestricted data Return to the Summary Tables dialog box (Stats > Summary Statistics > Summary Tables ). Variate : Number Groups: Mowing Nitrogen Display: No. of Observations, Means Press [Run] to display the new summary information. Nitrogen 0 1 Nobservd Mean Nobservd Mean Mowing M * MI CL * MI CR 0 * 0 * MR CL * The above data summary shows that the restricted data excludes both data from Mowing = MI CR plots and data from Nitrogen = 1 plots. 2013, Library, Teaching & Learning, Lincoln University page 11

12 Figure 16 The spreadsheet after a second restriction (Nitrogen = 0) was combined with the first restriction (Mowing MI CR). Remove all restrictions Restrictions on data can easily be removed so that you can use all the data again: Select the spreadsheet. Spread > Restrict/Filter > Remove All (or press the Remove Filter button, on the tool bar) To see the result of removing the restriction, reproduce the Summary Table showing Mowing and Nitrogen levels (result below) and notice that all the data are now used. Nitrogen 0 1 Nobservd Mean Nobservd Mean Mowing M MI CL MI CR MR CL , Library, Teaching & Learning, Lincoln University page 12

13 7. Further information GenStat s Spreadsheet Guide (available via Help > GenStat Guides > Spreadsheet. ) will guide you through many more techniques for dealing with GenStat spreadsheets. 8. References McLachlan, A.R.G. (2000). Spider distribution in agroecosystems in Canterbury, New Zealand. (unpublished doctoral thesis). Lincoln University, Lincoln, New Zealand. Ott, R.L., & Longnecker, M. (2001). An introduction to statistical methods and data analysis. (5th ed.). Australia: Duxbury. 9. Acknowledgements Andrew McLachlan (previously of Lincoln University) designed a number of workshops for versions of GenStat available prior to The workshop presented today is based around one of the original workshops, updated for use with GenStat 16 th edition ( 2013). Alison Lister Learning Advisor (Mathematics and Statistics) Library, Teaching and Learning 2013, Library, Teaching & Learning, Lincoln University page 13