RStudio for Data Management,

Transcription

1 K /1/9 17:35 page 4 #4 Usng R and RStudo for Data Management, Statstcal Analyss and Graphcs Second Edton Ncholas J. Horton Department of Mathematcs and Statstcs Amherst College Massachusetts, U.S.A. Ken Klenman Department of Populaton Medcne Harvard Medcal School and Harvard Plgrm Health Care Insttute Boston, Massachusetts, U.S.A.

2 K /1/9 17:35 page v #7 Contents Lst of Tables Lst of Fgures Preface to the second edton Preface to the frst edton xv xx xx xx 1 Data nput and output Input Natve dataset Fxed format text fles Other fxed fles Comma-separated value (CSV) fles Read sheets from an Excel fle Read data from R nto SAS Read data from SAS nto R Readng datasets n other formats Readng more complex text fles Readng data wth a varable number of words n a feld Read a fle byte by byte Access data from a URL Read an XML-formatted fle Read an HTML table Manual data entry Output Dsplayng data Number of dgts to dsplay Save a natve dataset Creatng datasets n text format Creatng Excel spreadsheets Creatng fles for use by other packages Creatng HTML formatted output Creatng XML datasets and output Further resources v

3 K /1/9 17:35 page v #8 v 2 Data management Structure and metadata Access varables from a dataset Names of varables and ther types Values of varables n a dataset Label varables Add comment to a dataset or varable Derved varables and data manpulaton Add derved varable to a dataset Rename varables n a dataset Create strng varables from numerc varables Create categorcal varables from contnuous varables Recode a categorcal varable Create a categorcal varable usng logc Create numerc varables from strng varables Extract characters from strng varables Length of strng varables Concatenate strng varables Set operatons Fnd strngs wthn strng varables Fnd approxmate strngs Replace strngs wthn strng varables Splt strngs nto multple strngs Remove spaces around strng varables Convert strngs from upper to lower case Create lagged varable Formattng values of varables Perl nterface Accessng databases usng SQL Mergng, combnng, and subsettng datasets Subsettng observatons Drop or keep varables n a dataset Random sample of a dataset Observaton number Keep unque values Identfy duplcated values Convert from wde to long (tall) format Convert from long (tall) to wde format Concatenate and stack datasets Sort datasets Merge datasets Date and tme varables Create date varable Extract weekday Extract month Extract year Extract quarter Create tme varable Further resources Examples Data nput and output

4 K /1/9 17:35 page v #9 v Data dsplay Derved varables and data manpulaton Sortng and subsettng datasets Statstcal and mathematcal functons Probablty dstrbutons and random number generaton Probablty densty functon Quantles of a probablty densty functon Settng the random number seed Unform random varables Multnomal random varables Normal random varables Multvarate normal random varables Truncated multvarate normal random varables Exponental random varables Other random varables Mathematcal functons Basc functons Trgonometrc functons Specal functons Integer functons Comparsons of floatng-pont varables Complex numbers Dervatves Integraton Optmzaton problems Matrx operatons Create matrx from vector Combne vectors or matrces Matrx addton Transpose matrx Fnd the dmenson of a matrx or dataset Matrx multplcaton Fndng the nverse of a matrx Component-wse multplcaton Create a submatrx Create a dagonal matrx Create a vector of dagonal elements Create a vector from a matrx Calculate the determnant Fnd egenvalues and egenvectors Fnd the sngular value decomposton Examples Probablty dstrbutons Programmng and operatng system nterface Control flow, programmng, and data generaton Loopng Condtonal executon Sequence of values or patterns Perform an acton repeatedly over a set of varables

5 K /1/9 17:35 page v #10 v Grd of values Debuggng Error recovery Functons Interactons wth the operatng system Tmng commands Suspend executon for a tme nterval Execute a command n the operatng system Command hstory Fnd workng drectory Change workng drectory Lst and access fles Create temporary fle Redrect output Common statstcal procedures Summary statstcs Means and other summary statstcs Weghted means and other statstcs Other moments Trmmed mean Quantles Centerng, normalzng, and scalng Mean and 95% confdence nterval Proporton and 95% confdence nterval Maxmum lkelhood estmaton of parameters Bvarate statstcs Epdemologc statstcs Test characterstcs Correlaton Kappa (agreement) Contngency tables Dsplay cross-classfcaton table Dsplayng mssng value categores n a table Pearson ch-square statstc Cochran Mantel Haenszel test Cramér s V Fsher s exact test McNemar s test Tests for contnuous varables Tests for normalty Student s t-test Test for equal varances Nonparametrc tests Permutaton test Logrank test Analytc power and sample sze calculatons Further resources Examples Summary statstcs and exploratory data analyss Bvarate relatonshps

6 K /1/9 17:35 page x #11 x Contngency tables Two sample tests of contnuous varables Survval analyss: logrank test Lnear regresson and ANOVA Model fttng Lnear regresson Lnear regresson wth categorcal covarates Changng the reference category Parameterzaton of categorcal covarates Lnear regresson wth no ntercept Lnear regresson wth nteractons Lnear regresson wth bg data One-way analyss of varance Analyss of varance wth two or more factors Tests, contrasts, and lnear functons of parameters Jont null hypotheses: several parameters equal Jont null hypotheses: sum of parameters Tests of equalty of parameters Multple comparsons Lnear combnatons of parameters Model results and dagnostcs Predcted values Resduals Standardzed and Studentzed resduals Leverage Cook s dstance DFFITs Dagnostc plots Heteroscedastcty tests Model parameters and results Parameter estmates Standardzed regresson coe cents Coe cent plot Standard errors of parameter estmates Confdence nterval for parameter estmates Confdence lmts for the mean Predcton lmts R-squared Desgn and nformaton matrx Covarance matrx of parameter estmates Correlaton matrx of parameter estmates Further resources Examples Scatterplot wth smooth ft Lnear regresson wth nteracton Regresson coe cent plot Regresson dagnostcs Fttng a regresson model separately for each value of another varable Two-way ANOVA Multple comparsons

7 K /1/9 17:35 page x #12 x Contrasts Regresson generalzatons and modelng Generalzed lnear models Logstc regresson model Condtonal logstc regresson model Exact logstc regresson Ordered logstc model Generalzed logstc model Posson model Negatve bnomal model Log-lnear model Further generalzatons Zero-nflated Posson model Zero-nflated negatve bnomal model Generalzed addtve model Nonlnear least squares model Robust methods Quantle regresson model Robust regresson model Rdge regresson model Models for correlated data Lnear models wth correlated outcomes Lnear mxed models wth random ntercepts Lnear mxed models wth random slopes More complex random coe cent models Multlevel models Generalzed lnear mxed models Generalzed estmatng equatons MANOVA Tme seres model Survval analyss Proportonal hazards (Cox) regresson model Proportonal hazards (Cox) model wth fralty Nelson Aalen estmate of cumulatve hazard Testng the proportonalty of the Cox model Cox model wth tme-varyng predctors Multvarate statstcs and dscrmnant procedures Cronbach s Factor analyss Recursve parttonng Lnear dscrmnant analyss Latent class analyss Herarchcal clusterng Complex survey desgn Model selecton and assessment Compare two models Log-lkelhood Akake Informaton Crteron (AIC) Bayesan Informaton Crteron (BIC) LASSO model

8 K /1/9 17:35 page x #13 x Hosmer Lemeshow goodness of ft Goodness of ft for count models Further resources Examples Logstc regresson Posson regresson Zero-nflated Posson regresson Negatve bnomal regresson Quantle regresson Ordered logstc Generalzed logstc model Generalzed addtve model Reshapng a dataset for longtudnal regresson Lnear model for correlated data Lnear mxed (random slope) model Generalzed estmatng equatons Generalzed lnear mxed model Cox proportonal hazards model Cronbach s Factor analyss Recursve parttonng Lnear dscrmnant analyss Herarchcal clusterng A graphcal compendum Unvarate plots Barplot Stem-and-leaf plot Dotplot Hstogram Densty plot Emprcal cumulatve probablty densty plot Boxplot Voln plots Unvarate plots by groupng varable Sde-by-sde hstograms Sde-by-sde boxplots Overlad densty plots Bar chart wth error bars Bvarate plots Scatterplot Scatterplot wth multple y values Scatterplot wth bnnng Transparent overplottng scatterplot Bvarate densty plot Scatterplot wth margnal hstograms Multvarate plots Matrx of scatterplots Condtonng plot Contour plots D plots

9 K /1/9 17:35 page x #14 x 8.5 Specal-purpose plots Choropleth maps Interacton plots Plots for categorcal data Crcular plot Plot an arbtrary functon Normal quantle quantle plot Recever operatng characterstc (ROC) curve Plot confdence ntervals for the mean Plot predcton lmts from a smple lnear regresson Plot predcted lnes for each value of a varable Kaplan Meer plot Hazard functon plottng Mean d erence plots Further resources Examples Scatterplot wth multple axes Condtonng plot Scatterplot wth margnal hstograms Kaplan Meer plot ROC curve Pars plot Vsualze correlaton matrx Graphcal optons and confguraton Addng elements Arbtrary straght lne Plot symbols Add ponts to an exstng graphc Jtter ponts Regresson lne ft to ponts Smoothed lne Normal densty Margnal rug plot Ttles Footnotes Text Mathematcal symbols Arrows and shapes Add grd Legend Identfyng and locatng ponts Optons and parameters Graph sze Grd of plots per page More general page layouts Fonts Pont and text sze Box around plots Sze of margns Graphcal settngs

10 K /1/9 17:35 page x #15 x Axs range and style Axs labels, values, and tck marks Lne styles Lne wdths Colors Log scale Omt axes Savng graphs PDF Postscrpt RTF JPEG Wndows Metafle Btmap mage fle (BMP) Tagged Image Fle Format PNG Closng a graphc devce Smulaton Generatng data Generate categorcal data Generate data from a logstc regresson Generate data from a generalzed lnear mxed model Generate correlated bnary data Generate data from a Cox model Samplng from a challengng dstrbuton Smulaton applcatons Smulaton study of Student s t-test Dploma (or hat-check) problem Monty Hall problem Censored survval Further resources Specal topcs Processng by group Means by group Lnear models stratfed by each value of a groupng varable Smulaton-based power calculatons Reproducble analyss and output Advanced statstcal methods Bayesan methods Propensty scores Bootstrappng Mssng data Fnte mxture models wth concomtant varables Further resources

11 K /1/9 17:35 page xv #16 xv 12 Case studes Data management and related tasks Fndng two closest values n a vector Tabulate bnomal probabltes Calculate and plot a runnng average Create a Fbonacc sequence Read varable format fles Plottng maps Massachusetts countes, contnued Bke rde plot Choropleth maps Data scrapng Scrapng data from HTML fles Readng data wth two lnes per observaton Plottng tme seres data Readng tables from HTML URL APIs and truly random numbers Readng from a web API Text mnng Retrevng data from arxv.org Exploratory text mnng Interactve vsualzaton Vsualzaton usng the grammar of graphcs (ggvs) Shny n Markdown Creatng a standalone Shny app Manpulatng bgger datasets Constraned optmzaton: the knapsack problem A Introducton to R and RStudo 211 A.1 Installaton A.1.1 Installaton under Wndows A.1.2 Installaton under Mac OS X A.1.3 RStudo A.1.4 Other graphcal nterfaces A.2 Runnng R and sample sesson A.2.1 Replcatng examples from the book and sourcng commands A.2.2 Batch mode A.3 Learnng R A.3.1 Gettng help A.3.2 swrl A.4 Fundamental structures and objects A.4.1 Objects and vectors A.4.2 Indexng A.4.3 Operators A.4.4 Lsts A.4.5 Matrces A.4.6 Dataframes A.4.7 Attrbutes and classes A.4.8 Optons A.5 Functons A.5.1 Callng functons

12 K /1/9 17:35 page xv #17 xv A.5.2 The apply famly of functons A.5.3 Ppes and connectons between functons A.6 Add-ons: packages A.6.1 Introducton to packages A.6.2 Packages and name conflcts A.6.3 Mantanng packages A.6.4 CRAN task vews A.6.5 Installed lbrares and packages A.6.6 Packages referenced n ths book A.6.7 Datasets avalable wth R A.7 Support and bugs B The HELP study dataset 237 B.1 Background on the HELP study B.2 Roadmap to analyses of the HELP dataset B.3 Detaled descrpton of the dataset C References 243 D Indces 255 D.1 Subject ndex D.2 R ndex