Generalized Linear Models

Transcription

1 Generalized Linear Models Summer School Manchester University July 2 6, 2018 Software and Data Graeme.Hutcheson@manchester.ac.uk University of Manchester

2 The slides and R-files for this session are available on the course website... Lecture Slides: Manchester/2018Manchester02SoftwareData.pdf R-notebook: Manchester/2018Manchester02.Rmd...or from the course DVD.

3 This session assumes that you have read the installation notes for R, the R-studio and the Rcmdr... and have managed to install these software packages onto your own laptop or USB drive, or have access to the programmes through the University computer network.

4 This session assumes that you have read the installation notes for R, the R-studio and the Rcmdr... and have managed to install these software packages onto your own laptop or USB drive, or have access to the programmes through the University computer network. If you have not installed the software onto your own computer and it is not available on the University network, please use the software provided on disk (a full copy of R, R-studio and the Rcmdr are provided on the course DVD).

5 This course uses R, which performs statistical analyses, the R-studio, which provides a sophisticated interface for R, and the Rcmdr, which provides a convenient way for using and learning R.

6 This course uses R, which performs statistical analyses, the R-studio, which provides a sophisticated interface for R, and the Rcmdr, which provides a convenient way for using and learning R. R is open-source software developed for multiple platforms (Unix, Linux, MacOS X and Windows) under the GPL licence.

7 This course uses R, which performs statistical analyses, the R-studio, which provides a sophisticated interface for R, and the Rcmdr, which provides a convenient way for using and learning R. R is open-source software developed for multiple platforms (Unix, Linux, MacOS X and Windows) under the GPL licence. The open-source environment is essential for the developement and use of statistics and provides advantages and resources that cannot be matched by commercial packages such as SPSS, STATA, S-PLUS, GenSTAT, Minitab and SAS.

8 R can be installed onto your computer, or directly onto a USB drive or CD and run from any computer. You can, therefore, guarantee access to and control over your software and also take it with you when you travel. This is becoming essential for those of us who use networked computers.

9 R can be installed onto your computer, or directly onto a USB drive or CD and run from any computer. You can, therefore, guarantee access to and control over your software and also take it with you when you travel. This is becoming essential for those of us who use networked computers. R has a vast number of add-on packages that allow a full range of analytical techniques to be employed.

10 R can be installed onto your computer, or directly onto a USB drive or CD and run from any computer. You can, therefore, guarantee access to and control over your software and also take it with you when you travel. This is becoming essential for those of us who use networked computers. R has a vast number of add-on packages that allow a full range of analytical techniques to be employed. It has a simple and automatic installation and updating system.

11 R can be installed onto your computer, or directly onto a USB drive or CD and run from any computer. You can, therefore, guarantee access to and control over your software and also take it with you when you travel. This is becoming essential for those of us who use networked computers. R has a vast number of add-on packages that allow a full range of analytical techniques to be employed. It has a simple and automatic installation and updating system. It has a comprehensive help-system which includes data sets and examples for most packages and a number of books and papers that can also be downloaded for free (look on CRAN, the Comprehensive R Archive Network).

12 An exercise using R This exercise provides a basic introduction to using R.

13 Exercise: A first session using notebooks 1. Start the R-studio From your laptop or from the programmes available on the network... If you are running a windows version of R-studio from a USB or DVD, change to the directory RStudio\bin on this drive and double click on the file rstudio.exe. Note: If your computer has multiple copies of R available select the version you wish to use when asked, or select from the Tools, Global Options... R-studio menu.

14 Exercise: A first session using notebooks 2. Open a new Rnotebook in the R-studio...

15 Exercise: A first session using notebooks

16 Exercise: A first session using notebooks 3. run and edit R code... Plot the graph by running the R-code in the notebook. look at the help available for plot (lower-right R-studio window) try changing the type of graph plot(cars, type="l") plot(cars, type="b") what does changing the aspect ratio do? plot(cars, asp=1) plot(cars, asp=0.01)

17 Exercise: A first session using notebooks Replace plot(cars) with hist(cars) or scatterplotmatrix(cars) Look at the help menu for scatterplotmatrix - try changing the graphs on the diagonal... scatterplotmatrix(cars, diagonal="boxplot") Preview your results...

18 Data An essential skill to master when using R is how to load and manipulate data.

19 There are many ways in which data can be loaded into R. This course uses data recorded in two formats, RData which is a format specifically designed for R and gives easy access to data used in this course, and csv which is a text-format that can be accessed by most, if not all, statistical software.

20 There are many ways in which data can be loaded into R. This course uses data recorded in two formats, RData which is a format specifically designed for R and gives easy access to data used in this course, and csv which is a text-format that can be accessed by most, if not all, statistical software. During this course, we will be loading datasets directly from the internet and from computer media. We will also be loading example data from various R packages. Full details about how to import data from various formats is available on-line at r-data-import-tutorial

21 Exercise: loading and importing data This exercise provides a basic introduction to loading and importing data into R Loading RData and.csv files from the net 2. Loading RData and.csv files from computer media 3. Loading data from packages Note: When completing this exercise it is useful to start a new Rnotebook which you can use to record all your working and return to it at a later date.

22 Exercise: Loading data from the web Loading RData files from the web: The simplest method for loading these data directly from the web is to use the load(url()) command. For example, to load and view ExampleData.RData from

23 Exercise: Loading and importing data Once loaded, the dataset is shown in the R-studio environment window (lower-left). Try clicking on the blue arrow and the data filename...

24 Exercise: Loading and importing data Loading csv files from the web: The simplest method for loading these data directly from the web is to use the Rcmdr...

25 Exercise: Loading and importing data Loading csv files from the web: The simplest method for loading these data directly from the web is to use the Rcmdr... Load the Rcmdr from the packages window. Select Data, Import data, from text file, clipboard or URL...

26 Exercise: Loading and importing data The command to load the dataset is shown in the Rcmdr script window... cut and paste this text into your Rnotebook...

27 Exercise: Loading data from the web When this command is run, examplecsvdata will appear in the R-studio environment window from where it can be viewed...

28 Exercise: Loading and importing data Data can also be loaded from computer media (hard drives, USB, DVD) using the Files menu in R-studio...

29 Exercise: Loading and importing data Data can also be loaded from computer media (hard drives, USB, DVD) using the Files menu in R-studio... RData files can be opened from the File, Open File... menu...

30 Exercise: Loading and importing data Data can also be loaded from computer media (hard drives, USB, DVD) using the Files menu in R-studio... RData files can be opened from the File, Open File... menu... csv files can be opened from the File, Import Dataset..., From Text (base) menu...

31 Exercise: Loading and importing data Data can also be loaded from computer media (hard drives, USB, DVD) using the Files menu in R-studio... RData files can be opened from the File, Open File... menu... csv files can be opened from the File, Import Dataset..., From Text (base) menu... Use the R-studio menu to load the Arrests.RData and happy.csv datasets that are included in the course documentation...

32 Exercise: Loading and importing data Data can also be loaded from computer media (hard drives, USB, DVD) using the Files menu in R-studio... RData files can be opened from the File, Open File... menu... csv files can be opened from the File, Import Dataset..., From Text (base) menu... Use the R-studio menu to load the Arrests.RData and happy.csv datasets that are included in the course documentation... Copy the commands to load these data into your Rnotebook...

33

34 Exercise: Loading and importing data Many R packages provide example data which can be easily loaded from the Rcmdr. To load the ChickWeight dataset which is available in the datasets package...

35 Exercise: Loading and importing data The commands to load the ChickWeight data can be copied from the Rcmdr into your Rnotebook. All files loaded will appear in the list of datasets...

36 Exercise: Loading and importing data Make sure that you have documented all your work. Preview your Rnotebook. Try knitting it to an html, or Word file... Save your Rnotebook file so you can return to this document later.

37 Data Frames and Data Coding

38 Data may be represented using vectors, matrices, lists and data-frames. The system proposed here uses data-frames as they are, perhaps, the easiest to deal with and also provide a structure for data that will be familiar to anyone who has used a spreadsheet or a statistics package such as Excel, Gnumeric, SPSS, STATA, S-Plus or SAS.

39 Data may be represented using vectors, matrices, lists and data-frames. The system proposed here uses data-frames as they are, perhaps, the easiest to deal with and also provide a structure for data that will be familiar to anyone who has used a spreadsheet or a statistics package such as Excel, Gnumeric, SPSS, STATA, S-Plus or SAS. Data-frames are simply matrices where each variable is represented in it s own column. Data-frames are rectangular in shape as they have the same number of observations, or cases, recorded for each variable. Data-frames are particularly useful as they can be used to represent entire data sets and provide a format for easily dealing with data.

40 There are many rules and conventions that can be applied to coding data. The following are a few that can be applied as general rules for data coding. The main principles are that data should... accurately represent the measurement scales (in particular - code categories as categories and not as numbers).

41 There are many rules and conventions that can be applied to coding data. The following are a few that can be applied as general rules for data coding. The main principles are that data should... accurately represent the measurement scales (in particular - code categories as categories and not as numbers). be able to code information without the use of any hidden codes or labels.

42 There are many rules and conventions that can be applied to coding data. The following are a few that can be applied as general rules for data coding. The main principles are that data should... accurately represent the measurement scales (in particular - code categories as categories and not as numbers). be able to code information without the use of any hidden codes or labels. be coded clearly and unabiguously.

43 There are many rules and conventions that can be applied to coding data. The following are a few that can be applied as general rules for data coding. The main principles are that data should... accurately represent the measurement scales (in particular - code categories as categories and not as numbers). be able to code information without the use of any hidden codes or labels. be coded clearly and unabiguously. be of a form that can be easily imported into different software packages (the coded data should be transportable).

44 General Coding Conventions Variable names should be included in the first row.

45 General Coding Conventions Variable names should be included in the first row. There should be NO EMPTY ROWS OR COLUMNS in the data file (spreadsheet).

46 General Coding Conventions Variable names should be included in the first row. There should be NO EMPTY ROWS OR COLUMNS in the data file (spreadsheet). Avoid spaces, commas, underscores, quotation marks or mathematical signs and other strange characters (eg., $%^&*?/\"!~#+-_) whenever possible...

47 General Coding Conventions Variable names should be included in the first row. There should be NO EMPTY ROWS OR COLUMNS in the data file (spreadsheet). Avoid spaces, commas, underscores, quotation marks or mathematical signs and other strange characters (eg., $%^&*?/\"!~#+-_) whenever possible... Avoid using highlights, colours, lines or anything else in the data files.

48 General Coding Conventions Variable names should be included in the first row. There should be NO EMPTY ROWS OR COLUMNS in the data file (spreadsheet). Avoid spaces, commas, underscores, quotation marks or mathematical signs and other strange characters (eg., $%^&*?/\"!~#+-_) whenever possible... Avoid using highlights, colours, lines or anything else in the data files. No formulas.

49 Coding Measurement Scales Explicitly code the 3 basic measurement scales

50 Coding Measurement Scales Explicitly code the 3 basic measurement scales Unordered categorical Code data using text

51 Coding Measurement Scales Explicitly code the 3 basic measurement scales Unordered categorical Code data using text Ordered categorical Code data using text preceded by a number to explicitly indicate order (so that it orders the data appropriately for graphics and analyses)

52 Coding Measurement Scales Explicitly code the 3 basic measurement scales Unordered categorical Code data using text Ordered categorical Code data using text preceded by a number to explicitly indicate order (so that it orders the data appropriately for graphics and analyses) Numeric Code data using numbers that best represents the information.

53 Coding Missing Data Problems may arise for any coding system which uses numeric, hidden or substitution codes to indicate missingness.

54 Coding Missing Data Problems may arise for any coding system which uses numeric, hidden or substitution codes to indicate missingness. Information about missing data is qualitatively different to the information about the variable.

55 Coding Missing Data Problems may arise for any coding system which uses numeric, hidden or substitution codes to indicate missingness. Information about missing data is qualitatively different to the information about the variable. Missing data should be indicated using a unique categorical indicator code such as NA which can be ignored by the analysis (SPSS, however, insists on a numeric code for missing data which is identified using a hidden label).

56 Coding Missing Data Problems may arise for any coding system which uses numeric, hidden or substitution codes to indicate missingness. Information about missing data is qualitatively different to the information about the variable. Missing data should be indicated using a unique categorical indicator code such as NA which can be ignored by the analysis (SPSS, however, insists on a numeric code for missing data which is identified using a hidden label). If there are multiple missing codes (eg., not applicable, unanswered, spoiled) this information needs to be coded separately using an additional variable.

57 Subject Age Nationality EconStatus FactorSocial ManGrade ManGradeMiss subject01 23 Russian 2ses NA 1junior answered subject02 24 English 4ses NA spoiled subject03 31 Welsh 1ses junior answered subject04 NA NA NA NA 3upper answered subject05 43 Irish 2ses middle answered subject06 41 German 2ses NA 2middle answered subject07 19 German 3ses upper answered subject08 38 Portuguese 3ses middle answered subject09 59 Spanish NA NA notapplic subject10 24 Scottish 2ses NA notapplic subject11 39 Irish 4ses junior answered

58 numeric data just numbered Subject Age Nationality EconStatus FactorSocial ManGrade ManGradeMiss subject01 23 Russian 2ses NA 1junior answered subject02 24 English 4ses NA spoiled subject03 31 Welsh 1ses junior answered subject04 NA NA NA NA 3upper answered subject05 43 Irish 2ses middle answered subject06 41 German 2ses NA 2middle answered subject07 19 German 3ses upper answered subject08 38 Portuguese 3ses middle answered subject09 59 Spanish NA NA notapplic subject10 24 Scottish 2ses NA notapplic subject11 39 Irish 4ses junior answered

59 numeric data just numbered Ordered data numbered and named Subject Age Nationality EconStatus FactorSocial ManGrade ManGradeMiss subject01 23 Russian 2ses NA 1junior answered subject02 24 English 4ses NA spoiled subject03 31 Welsh 1ses junior answered subject04 NA NA NA NA 3upper answered subject05 43 Irish 2ses middle answered subject06 41 German 2ses NA 2middle answered subject07 19 German 3ses upper answered subject08 38 Portuguese 3ses middle answered subject09 59 Spanish NA NA notapplic subject10 24 Scottish 2ses NA notapplic subject11 39 Irish 4ses junior answered

60 numeric data just numbered Unordered data just named Ordered data numbered and named Subject Age Nationality EconStatus FactorSocial ManGrade ManGradeMiss subject01 23 Russian 2ses NA 1junior answered subject02 24 English 4ses NA spoiled subject03 31 Welsh 1ses junior answered subject04 NA NA NA NA 3upper answered subject05 43 Irish 2ses middle answered subject06 41 German 2ses NA 2middle answered subject07 19 German 3ses upper answered subject08 38 Portuguese 3ses middle answered subject09 59 Spanish NA NA notapplic subject10 24 Scottish 2ses NA notapplic subject11 39 Irish 4ses junior answered

61 numeric data just numbered Unordered data just named Ordered data numbered and named Subject Age Nationality EconStatus FactorSocial ManGrade ManGradeMiss subject01 23 Russian 2ses NA 1junior answered subject02 24 English 4ses NA spoiled subject03 31 Welsh 1ses junior answered subject04 NA NA NA NA 3upper answered subject05 43 Irish 2ses middle answered subject06 41 German 2ses NA 2middle answered subject07 19 German 3ses upper answered subject08 38 Portuguese 3ses middle answered subject09 59 Spanish NA NA notapplic subject10 24 Scottish 2ses NA notapplic subject11 39 Irish 4ses junior answered missing data coded as NA

62 The data structure above enables the management grade (ManGrade) and the missing data (ManGradeMiss) to be modelled and graphed. Below are shown the relationships between these variables and the FactorSocial variable FactorSocial FactorSocial junior 2middle 3upper ManGrade answered notapplic spoiled ManGrade.miss

63 Data Manipulation and Management

64 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3)

65 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years)

66 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years) re-labelling (change category names)

67 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years) re-labelling (change category names) data transformation (log, sqrt, etc.)

68 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years) re-labelling (change category names) data transformation (log, sqrt, etc.) changing numeric data into categoric (when numbers in the dataset refer to categories; such as year ).

69 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years) re-labelling (change category names) data transformation (log, sqrt, etc.) changing numeric data into categoric (when numbers in the dataset refer to categories; such as year ).

70 Data often require manipulating to run certain analyses and graphics. Some of the more common data transformations that can be applied are: Recoding categories (eg., recode 5 ordered categories into 3) sub-setting data (eg., only include males, or records from the last 5 years) re-labelling (change category names) data transformation (log, sqrt, etc.) changing numeric data into categoric (when numbers in the dataset refer to categories; such as year ). All of these operations can be applied using R...

71 Recoding The 5-category variable EconStatus from the ExampleData dataset can be recoded into 3-categories using the Rcmdr...

72 Recoding The 5-category variable EconStatus from the ExampleData dataset can be recoded into 3-categories using the Rcmdr...

73 Recoding The 5-category variable EconStatus from the ExampleData dataset can be recoded into 3-categories using the Rcmdr... During this course we will be recoding a number of variables, particularly when dealing with ordered categories...

74 Compute a new variable The weight from the ChickWeight dataset can be transformed into the log of weight using the Rcmdr...

75 Compute a new variable The weight from the ChickWeight dataset can be transformed into the log of weight using the Rcmdr...

76 Compute a new variable The weight from the ChickWeight dataset can be transformed into the log of weight using the Rcmdr... The session on data transformation will involve computing new variables...

77 Convert numeric variables to factors The year variable from the Arrests dataset can be re-defined as a categorical variable using the Rcmdr...

78 Convert numeric variables to factors The year variable from the Arrests dataset can be re-defined as a categorical variable using the Rcmdr...

79 Convert numeric variables to factors The year variable from the Arrests dataset can be re-defined as a categorical variable using the Rcmdr... When we analyse the Arrests dataset, year needs to be considered as a categorical variable...

80 Reorder factor levels The nationality variable from the ExampleData dataset can be re-ordered using the Rcmdr...

81 Reorder factor levels The nationality variable from the ExampleData dataset can be re-ordered using the Rcmdr...

82 Reorder factor levels The nationality variable from the ExampleData dataset can be re-ordered using the Rcmdr...

83 Reorder factor levels The nationality variable from the ExampleData dataset can be re-ordered using the Rcmdr... Reordering categories is particularly useful for arranging categories for graphical displays and for assigning a specific order to ordered categorical data (eg., low = 1, medium = 2, high = 3)

84 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels...

85 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none

86 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none 1,2 = minor

87 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none 1,2 = minor 3,4 = medium

88 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none 1,2 = minor 3,4 = medium 5,6 = major

89 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none 1,2 = minor 3,4 = medium 5,6 = major Using the Rcmdr, draw a bar graph of checkscat

90 Exercise Start a new Rnotebook to save examples of data manipulation. Open the Arrests dataset... Convert the numeric variable checks to a categorical variable with 4 categories with the following labels... 0 = none 1,2 = minor 3,4 = medium 5,6 = major Using the Rcmdr, draw a bar graph of checkscat change the order of the categories in checkscat so that none appears on left of the bar graph and major appears on the right.

91 Exercise compute a new variable (agemonth) which shows age in months

92 Exercise compute a new variable (agemonth) which shows age in months split age up into four groups using the Rcmdr function bin numeric variable...

93 Exercise compute a new variable (agemonth) which shows age in months split age up into four groups using the Rcmdr function bin numeric variable... Copy and paste your commands into an Rnotebook. Fully document your work.

94 Exercise compute a new variable (agemonth) which shows age in months split age up into four groups using the Rcmdr function bin numeric variable... Copy and paste your commands into an Rnotebook. Fully document your work. Save your Rnotebook so that you can add any new data manipulation techniques that you subsequently come across.

95 The problem of data proliferation Data are routinely manipulated and changed by, for example, recoding variables, combining and renaming categories, transforming observations, changing reference categories, dummy-coding variables and changing measurement scales. If these changes are written to the data-frame it quickly increases in size and complexity with individual variables often represented in multiple columns.

96 The problem of data proliferation Data are routinely manipulated and changed by, for example, recoding variables, combining and renaming categories, transforming observations, changing reference categories, dummy-coding variables and changing measurement scales. If these changes are written to the data-frame it quickly increases in size and complexity with individual variables often represented in multiple columns. Recoded data may also be saved into new data sets; For example, Results.csv, ResultsAll.csv, ResultsFinal.csv, ResultsPublish.csv, ResultsFinal2.csv, Results25/07/18.csv, ResultsVeryFinal.csv, ResultsVeryFinal02.csv etc.,.

97 This proliferation of variables and data sets can cause problems...

98 This proliferation of variables and data sets can cause problems... It is confusing as a single attribute may be represented using multiple variables.

99 This proliferation of variables and data sets can cause problems... It is confusing as a single attribute may be represented using multiple variables. If data are amended or corrected, all data files and recoded copies of the data also need amending.

100 This proliferation of variables and data sets can cause problems... It is confusing as a single attribute may be represented using multiple variables. If data are amended or corrected, all data files and recoded copies of the data also need amending. Analyses may apply to different versions of your data.

101 A solution to data proliferation: Use a master data-frame that contains the most accurate and complete representation of the information available. This data-frame is the only one that is saved to disk and and is the data file that is accessed at the start of all analysis sessions.

102 A solution to data proliferation: Use a master data-frame that contains the most accurate and complete representation of the information available. This data-frame is the only one that is saved to disk and and is the data file that is accessed at the start of all analysis sessions. Any changes to the data (recodes, transformations, renaming etc.,) should be made on a temporary basis and not saved to the master data file (unless absolutely necessary).

103 A solution to data proliferation: Use a master data-frame that contains the most accurate and complete representation of the information available. This data-frame is the only one that is saved to disk and and is the data file that is accessed at the start of all analysis sessions. Any changes to the data (recodes, transformations, renaming etc.,) should be made on a temporary basis and not saved to the master data file (unless absolutely necessary). The master data file should ONLY include the most complete coding of the information with each variable accurately coded according to its measurement scale.

104 This method of working with data has a number of advantages... Data sets and variables do not proliferate.

105 This method of working with data has a number of advantages... Data sets and variables do not proliferate. You may only load the data you actually need for each analysis.

106 This method of working with data has a number of advantages... Data sets and variables do not proliferate. You may only load the data you actually need for each analysis. The command files document accurately how your data have been coded.