Practical Quantitative Analysis

Transcription

1 Manchester University Dealing with Data in R Graeme.Hutcheson@manchester.ac.uk University of Manchester Your MA dissertation data If you are going to be using quantitative analysis for your MA dissertation, you will probably originally record your data in a spreadsheet and save in a.csv text-based format. You may also download secondary data from the web which is also likely to be available in.csv text-based format. Once downloaded, it is useful to save your data in the.rdata format which is specific to R. This document provides an introduction to working with.csv and.rdata files.

2 Full details about how to import data from various formats is available on-line at r-data-import-tutorial Exercise: loading and importing data This exercise provides a basic introduction to loading and importing data into R Loading.RData and.csv files from the net 2. Loading.RData and.csv files from computer media Note: When completing this exercise it is useful to start a new Rnotebook which you can use to record all your working and return to it at a later date.

3 Exercise: Loading data from the web Loading RData files from the web: The simplest method for loading these data directly from the web is to use the load(url()) command. For example, to load and view ExampleData.RData from Exercise: Loading and importing data Once loaded, the dataset is shown in the R-studio environment window (lower-left). Try clicking on the blue arrow and the data filename...

4 Exercise: Loading and importing data Loading.csv files from the web: The simplest method for loading these data directly from the web is to use the Rcmdr menu Data, Import data, from text file, clipboard or URL... Enter the URL Exercise: Loading and importing data The command to load the dataset is shown in the Rcmdr script window... cut and paste this text into your Rnotebook...

5 Exercise: Loading data from the web When this command is run, examplecsvdata will appear in the R-studio environment window from where it can be viewed... Exercise: Loading and importing data Data can also be loaded from computer media (hard drives, USB, DVD) using the Files menu in R-studio....RData files can be opened from the File, Open File... menu....csv files can be opened from the File, Import Dataset..., From Text (base) menu...

6 Exercise: Loading and importing data Use the R-studio menu to load the Arrests.RData and happy.csv datasets that are available on the course website... Download the two data files (Arrests.RData and happy.csv) onto your computer. Use the R-studio menus to load these data. Copy the R commands into your Rnotebook...

7 Exercise: Loading and importing data Make sure that you have documented all your work. Preview your Rnotebook. Try knitting it to an html, or Word file... Save your Rnotebook file so you can return to this document later. Data Frames and Data Coding

8 Data may be represented using vectors, matrices, lists and data-frames. The system proposed here uses data-frames as they are, perhaps, the easiest to deal with and also provide a structure for data that will be familiar to anyone who has used a spreadsheet or a statistics package such as Excel, Gnumeric, SPSS, STATA, S-Plus or SAS. Data-frames are simply matrices where each variable is represented in it s own column. Data-frames are rectangular in shape as they have the same number of observations, or cases, recorded for each variable. Data-frames are particularly useful as they can be used to represent entire data sets and provide a format for easily dealing with data. There are many rules and conventions that can be applied to coding data. The following are a few that can be applied as general rules for data coding. The main principles are that data should... accurately represent the measurement scales (in particular - code categories as categories and not as numbers). be able to code information without the use of any hidden codes or labels. be coded clearly and unabiguously. be of a form that can be easily imported into different software packages (the coded data should be transportable).

9 General Coding Conventions Variable names should be included in the first row. There should be NO EMPTY ROWS OR COLUMNS in the data file (spreadsheet). Avoid spaces, commas, underscores, quotation marks or mathematical signs and other strange characters (eg., $%^&*?/\"!~#+-_) whenever possible... Avoid using highlights, colours, lines or anything else in the data files. No formulas. Coding Measurement Scales Explicitly code the 3 basic measurement scales Unordered categorical Code data using text Ordered categorical Code data using text preceded by a number to explicitly indicate order (so that it orders the data appropriately for graphics and analyses) Numeric Code data using numbers that best represents the information.

10 Coding Missing Data Problems may arise for any coding system which uses numeric, hidden or substitution codes to indicate missingness. Information about missing data is qualitatively different to the information about the variable. Missing data should be indicated using a unique categorical indicator code such as NA which can be ignored by the analysis (SPSS, however, insists on a numeric code for missing data which is identified using a hidden label). If there are multiple missing codes (eg., not applicable, unanswered, spoiled) this information needs to be coded separately using an additional variable. numeric data just numbered Unordered data just named Ordered data numbered and named Subject Age Nationality EconStatus FactorSocial ManGrade ManGradeMiss subject01 23 Russian 2ses NA 1junior answered subject02 24 English 4ses NA spoiled subject03 31 Welsh 1ses junior answered subject04 NA NA NA NA 3upper answered subject05 43 Irish 2ses middle answered subject06 41 German 2ses NA 2middle answered subject07 19 German 3ses upper answered subject08 38 Portuguese 3ses middle answered subject09 59 Spanish NA NA notapplic subject10 24 Scottish 2ses NA notapplic subject11 39 Irish 4ses junior answered missing data coded as NA

11 The data structure above enables the management grade (ManGrade) and the missing data (ManGradeMiss) to be modelled and graphed. Below are shown the relationships between these variables and the FactorSocial variable FactorSocial FactorSocial junior 2middle 3upper ManGrade answered notapplic spoiled ManGrade.miss Data Manipulation and Management

12 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years) re-labelling (change category names) data transformation (log, sqrt, etc.) changing numeric data into categoric (when numbers in the dataset refer to categories; such as year ). All of these operations can be applied using R... Recoding The 5-category variable EconStatus from the ExampleData dataset can be recoded into 3-categories using the Rcmdr...

13 Compute a new variable The Age variable from the ExampleData dataset is recorded in years but can be transformed into months using the Rcmdr... Convert numeric variables to factors The year variable from the Arrests dataset can be re-defined as a categorical variable using the Rcmdr...

14 Reorder factor levels The nationality variable from the ExampleData dataset can be re-ordered using the Rcmdr... Reordering categories is particularly useful for arranging categories for graphical displays and for assigning a specific order to ordered categorical data (eg., low = 1, medium = 2, high = 3) Look at the changes to the dataset - ExampleData should have two new variables EconStatus3 and AgeMonths and Nationality should be ordered with Irish as the first category...

15 Look at the changes to the dataset - Arrests should have a new variable yearcat which is a Factor with 6 levels... Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none 1,2 = minor 3,4 = medium 5,6 = major Using the Rcmdr, draw a bar graph of checkscat change the order of the categories in checkscat so that none appears on left of the bar graph and major appears on the right.

16 Exercise compute a new variable (agemonth) which shows age in months split age up into four groups using the Rcmdr function bin numeric variable... Copy and paste your commands into an Rnotebook. Fully document your work. Save your Rnotebook so that you can add any new data manipulation techniques that you subsequently come across. The problem of data proliferation Data are routinely manipulated and changed by, for example, recoding variables, combining and renaming categories, transforming observations, changing reference categories, dummy-coding variables and changing measurement scales. If these changes are written to the data-frame it quickly increases in size and complexity with individual variables often represented in multiple columns. Recoded data may also be saved into new data sets; For example, Results.csv, ResultsAll.csv, ResultsFinal.csv, ResultsPublish.csv, ResultsFinal2.csv, Results25/07/18.csv, ResultsVeryFinal.csv, ResultsVeryFinal02.csv etc.,.

17 This proliferation of variables and data sets can cause problems... It is confusing as a single attribute may be represented using multiple variables. If data are amended or corrected, all data files and recoded copies of the data also need amending. Analyses may apply to different versions of your data. A solution to data proliferation: Use a master data-frame that contains the most accurate and complete representation of the information available. This data-frame is the only one that is saved to disk and and is the data file that is accessed at the start of all analysis sessions. Any changes to the data (recodes, transformations, renaming etc.,) should be made on a temporary basis and not saved to the master data file (unless absolutely necessary). The master data file should ONLY include the most complete coding of the information with each variable accurately coded according to its measurement scale.

18 This method of working with data has a number of advantages... Data sets and variables do not proliferate. You may only load the data you actually need for each analysis. The command files document accurately how your data have been coded.