- ST Computer Intensive Statistical Analytics I Appendix: R basics

Transcription

1 ST appendix - Eric Wolsztynski, UCC p.1/40 - ST Computer Intensive Statistical Analytics I Appendix: Eric Wolsztynski

2 ST appendix - Eric Wolsztynski, UCC p.2/40 What is R? R is a free software environment for statistical computing and graphics In short, R allows to analyse data and write programs R is derived from S, a commercialised statistical solution Visit for resources and documentation The basic R distribution contains about 25 packages Many other packages may be installed (or created)

3 ST appendix - Eric Wolsztynski, UCC p.3/40 R documentation and tutorials Disclaimer: the present guide is only a short, practical introduction to R. Parts of this document use and summarise sections of the Introduction to R manual from the CRAN. Web: huge amount of guides, short and long Starting by the CRAN website is recommended cf. in particular the R manuals page on the CRAN Youtube: 3zWo4sUs&list=PLOU2XLYxmsIK9qQfztXeybpHvru-TrqAP Offline: some really good books (see references)

4 ST appendix - Eric Wolsztynski, UCC p.4/40 Launching R... and quitting R R version ( ) Copyright (C) 2010 The R Foundation for Statistical Computing ISBN Platform: i386-apple-darwin9.8.0/i386 (32-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type license() or licence() for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type contributors() for more information and citation() on how to cite R or R packages in publications. Type demo() for some demos, help() for on-line help, or help.start() for an HTML browser interface to help. Type q() to quit R.

5 ST appendix - Eric Wolsztynski, UCC p.5/40 Instructing R One can use the R console (limited) or the R editor Save and open R scripts using the editor (extension file.r) Editor: select and run a block of instructions (lines) at once using Ctrl-R (Windows) or Cmd-Enter (Mac) Other editors (providing coloured keywords, highlighting, etc.) are available for R (TinnR, Emacs, etc.)

6 ST appendix - Eric Wolsztynski, UCC p.6/40 Getting help for and from R Web search (using keyword cran often helps) Browse packages (CRAN): The R Reference Index contains all help files of the R standard and recommended packages in printable form R> help.start() brings up the html help page?name opens up the help on a specific function. Example: R>?plot Browse the See Also section of R help pages

7 ST appendix - Eric Wolsztynski, UCC p.7/40 R syntax and data handling R syntax and data handling

8 ST appendix - Eric Wolsztynski, UCC p.8/40 R syntax and data handling Some short examples to run in the console window R is case sensitive R> a = 2 R> a (displays the value a = 2) R> A (returns an error: A is not a declared variable) Use R as a calculator: R> 3+5*2; exp(log(10)); 2/0 Load a specific library (containing functions and data): R> library(kernsmooth) Load an existing dataset: R> data(airpassengers) R> plot(airpassengers)

9 ST appendix - Eric Wolsztynski, UCC p.9/40 R syntax and data handling General philosophy In R, statistical analysis is usually carried out incrementally (i.e. in steps) Intermediate results are stored in objects that are re-used in further steps The displayed output of a function (e.g. linear regression) often remains succinct, while the more detailed result is stored in an object

10 ST appendix - Eric Wolsztynski, UCC p.10/40 R syntax and data handling Basics of R syntax Assignment: use either = or <- Without assignment, a value is printed and then lost Standard mathematical operators: +,, /, log, exp, etc. Standard logical operators: <, <=,! (NOT), == (equality) Comments must begin with # R objects have a type, which is a named data structure, such as logical, integer, real, vector, list, array, or a more elaborate structure Example: R> a = 2; is.real(a); is.logical(a); is.list(a)

11 ST appendix - Eric Wolsztynski, UCC p.11/40 R syntax and data handling R object types Vectors are uni-dimensional ordered collection of numbers Arrays (including matrices) are multi-dimensional vectors (M[i,j]) Factors are used for categorical data (for use e.g. in GLM) Lists are vectors of elements of different types Data frames are matrix-like structures containing elements of different types Functions implement re-usable sets of instructions from input arguments. Usage: output = function(argument)

12 ST appendix - Eric Wolsztynski, UCC p.12/40 R syntax and data handling R object classes All R objects also have a class Vectors are usually numeric, complex, logical, character or list: try e.g. class(sqrt(-17+0i)) Other possible classes are matrix, array, factor and data.frame It is possible to create new classes, usually as data.frames This allows object-oriented programming with R, like in Java or C++ Some generic R functions are class-specific, such as plot and summary: they behave differently in function of the class of their argument

13 ST appendix - Eric Wolsztynski, UCC p.13/40 R syntax and data handling Vector operations (1) x = c(0.1, 2, 4.3, 3.1, 5) Synopsis: vector[index] x[2] then returns the second value in that vector x[1:3] returns the first three values x[-(1:3)] gives all but the first three elements x[x>3] selects the subset of values in x greater than 3 N = length(x) stores the length of x For assignment, the assigned value must be either a single value or of same length as the index vector: x[is.na(x)] = 0 x[x<0] = -x[x<0] (equivalent to x = abs(x))

14 ST appendix - Eric Wolsztynski, UCC p.14/40 R syntax and data handling Vector operations (2) Standard arithmetic applies to vectors of equal length, e.g.: v <- 2*x + y + 1 sum((x-mean(x))ˆ2)/(length(x)-1) Regular sequences can be generated easily: s1 = seq(-5, 5, by=.2) s2 = seq(from=-5, length=51, by=.2) xx = rep(x, times = 2) xxx = rep(x, each = 3) Vectors may also contain characters or strings: fruit <- c(5, 10, 4) names(fruit) <- c( orange, banana, apple ) lunch <- fruit[c( apple, orange )]

15 ST appendix - Eric Wolsztynski, UCC p.15/40 R syntax and data handling Matrix operations (1) Create a vector x = c(1, 2, 3, 4, 5, 6, 7, 8, 9) Now make it a 3 3 matrix: matrix(x, ncol = 3) If we prefer arranging by rows: M = matrix(x, ncol = 3, byrow = TRUE) The matrix now has dimensions: dim(m); ncol(m); nrow(m) Note the difference in classes: class(x); class(m) Access any of the matrix components: M[2,3]; M[,1]; M[2,]; M[2, c(1:2)] cbind() and rbind() also produce matrices: cbind(c(1,2,3),c(4,5,6))

16 ST appendix - Eric Wolsztynski, UCC p.16/40 R syntax and data handling Matrix operations (2) Transpose matrices: t(m) or aperm(m, c(2,1)) diag(m), det(m), svd(m) (Single Value Decomposition) Matrix arithmetics (on compatible matrices A, B and vector x): Element-by-element product: A * B (if A and B are square) Matrix product: A %*% B Quadratic form using vector x: x %*% A %*% x crossprod(m, x) is the same as t(x) %*% y

17 ST appendix - Eric Wolsztynski, UCC p.17/40 R syntax and data handling Matrix operations (3) Solving linear equation Ax + b = 0: Given A and b such that b = A %*% x, solve(a,b) returns x (solution given up to some accuracy loss) solve(a) returns the inverse of a matrix x <- solve(a) %*% b is inefficient and potentially unstable eigen(s) computes eigenvalues and eigenvectors of symmetric matrix S Note that x %*% x is ambiguous (with x a column-vector). The best is to use crossprod(x) for x x x %o% x for x x (the outer product for an array)

18 ST appendix - Eric Wolsztynski, UCC p.18/40 R syntax and data handling Array operations Make a vector into an array using array(vector, dim=...): h = seq(1, 24, by=1) Z <- array(h, dim=c(3,4,2)) Z[1,1,1]; Z[1,2,2] Arithmetic operations on arrays are carried out element-by-element and respect constraints of dimensions: D <- 2*A*B + C + 1 works if A, B and C are similar arrays Transpose should be done using aperm Other array-specific operators exist, such as the outer product There exist some more tricky issues with arrays

19 ST appendix - Eric Wolsztynski, UCC p.19/40 R syntax and data handling Lists Synopsis: Lst <- list(name 1=object 1,..., name m=object m) Lst[[index]] # to access its components Create a list: Lst <- list(name= Fred, wife= Mary, no.children=3, child.ages=c(4,7,9)) In this example, Lst[[2]] = "Mary" and Lst[[4]][1] = 4 Components may also be accessed by their name using $, e.g. Lst$name = Fred or Lst$child.ages[1] = 4 length(lst) gives the number of top level components

20 ST appendix - Eric Wolsztynski, UCC p.20/40 R syntax and data handling Data frames A data frame is a list of class data.frame It can be seen as a matrix with columns of possibly different types/classes Character vectors are coerced to factors Vector components must all have equal length Matrix structures must all have equal row size Synopsis: D <- data.frame(name 1=arg 1,..., name m=arg m) D$name 1 # accesses its first component

21 ST appendix - Eric Wolsztynski, UCC p.21/40 R syntax and data handling Data frames Example: D <- data.frame(u=homes, v=incomes, w=children) D$w # displays values of vector children A list whose components conform to the restrictions of a data frame may be coerced into one using as.data.frame() read.table() reads in a table of values from an input file and returns an object of type data.frame: D = read.table(some csv file) attach() loads the components of a data frame in the local environment (i.e. memory), which can thus be used as separate variables. E.g. if D contains components u, v and w: attach(d) u+v-w

22 ST appendix - Eric Wolsztynski, UCC p.22/40 R syntax and data handling Data frames In this example, following instruction attach(d), any assignment to u (such as u = 3 or u = v+w) will change the value of u in the local environment (memory) but not in the original data file, nor even in the data frame D To make a change in the data frame itself, one needs to assign a value to the component itself, i.e. D$u: D$u = new vector The new value of this component will not be visible until the data frame is detached and attached again detach() removes the components of the input data frame from the local environment detach() does not remove the data frame from the memory

23 ST appendix - Eric Wolsztynski, UCC p.23/40 R syntax and data handling Formulae symbol example meaning y x y x1 y = ax 1 + b + y x1 + x2 y = ax 1 + bx 2 + c (include variable) + 0 y x1 + 0 y = ax 1 (exclude intercept) - 1 y x1-1 y = ax 1 (exclude intercept also) : y x1 : x2 include interaction between x 1 and x 2 * y x1 * x2 include these variables and their interaction (equiv. x1 + x2 + x1:x2) / y x1 / x2 nesting: include x 2 nested within x 1 ˆ y (x1 + x2)ˆ3 include these variables and all interactions up to 3 way I y I(x1*x2) as is: specify a new variable consisting of the product x 1 x 2 poly y poly(x1,3) polynomial regression: orthogonal polynomials

24 ST appendix - Eric Wolsztynski, UCC p.24/40 Files and plots Files and plots

25 ST appendix - Eric Wolsztynski, UCC p.25/40 Files and plots Reading data from files read.table() reads in a file in table format and returns a data frame object read.csv(), a variant of read.table() is particularly useful for reading CSV files scan() reads data from the console or from an input file into a vector or list scan() is often called within matrix() when uploading data: X <- matrix(scan( light.dat ), ncol=5, byrow=true) readlines() reads lines (all or some) of an input file

26 ST appendix - Eric Wolsztynski, UCC p.26/40 Files and plots Writing data to files write.table() creates a file and writes the data frame content in it write.csv(), a variant of write.table() creates a CSV file More advanced: file may be used to create a connection (or file), in a C-like approach; see?connections and examples within for more information

27 ST appendix - Eric Wolsztynski, UCC p.27/40 Files and plots Plot?plot provides information on the main plotting device Synopsis: plot(x, y,...), where y may be omitted if x is an appropriate structure... are optional arguments (graphical parameters, see?par): main - overall plot title type - type of plot (type="l" for line, "p" for points, etc.) xlab, ylab - x- and y-axis titles xlim, ylim - x- and y-axis limits col - plot colour (use strings ("red"), or else integers (2)) cf.?plot for more details

28 ST appendix - Eric Wolsztynski, UCC p.28/40 Files and plots Some useful tips x11() may be called before plot() to create a new window par() is used to organise (prepare) the plotting window: par(mfrow=c(2,3)) # 2 lines of 3 plots in window Look after line width or point size (respectively lwd and cex), main title, axis labels, etc.: x11() # opens new window par(pty= s ) # makes plot window square plot(x, y, main= plot of sales against investment, xlab= investment, ylab= sales, type= p, cex=2) or plot(x, y, main= plot of sales against investment, xlab= investment, ylab= sales, type= l, lwd=2)

29 ST appendix - Eric Wolsztynski, UCC p.29/40 Files and plots Plot more in the same window points(x, y,...) adds points on existing plot Usage of points() is very similar to that of plot() abline() plots a line: abline(h=3) # horizontal line at y=3 abline(v=2) # vertical line at x=2 abline(a=1,b=4) # draws line 4x+1 par(new=true) keeps current plot and plotting window, while allowing a new call to plot (which will not clean the window) par(new=true) resets axis scales and other settings!!

30 ST appendix - Eric Wolsztynski, UCC p.30/40 Files and plots More plots plot() behaves specifically to the class of its input argument R offers many other plotting solutions See e.g. and hist() draws histograms, density() kernel density estimates boxplot() plots boxplots persp(x, y, z,...) draws perspective plots of surfaces contour(x, y, z,...) creates a contour plot image(x,...) displays a grayscale/colored image

31 ST appendix - Eric Wolsztynski, UCC p.31/40 Files and plots An example with mathematical annotations x<-seq(-10,10,length=400) [from addictedtor.free.fr] y1<-dnorm(x); y2<-dnorm(x,m=3) par(mar=c(5,4,2,1)) plot(x, y2, xlim=c(-3,8), type="n", xlab=quote(z==frac(mu[1]-mu[2], sigma/sqrt(n))), ylab="density") polygon(c(1.96,1.96,x[240:400],10), c(0,dnorm(1.96,m=3),y2[240:400],0), col="grey80", lty=0) lines(x, y2); lines(x, y1) polygon(c(-1.96,-1.96,x[161:1],-10), c(0,dnorm(-1.96,m=0), y1[161:1],0), col="grey30", lty=0) polygon(c(1.96, 1.96, x[240:400], 10), c(0,dnorm(1.96,m=0), y1[240:400],0), col="grey30") legend(4.2,.4, fill=c("grey80","grey30"), legend=expression(p(abs(z)>1.96, H[1])==0.85, P(abs(Z)>1.96,H[0])==0.05), bty="n") text(0,.2, quote(h[0]:~~mu[1]==mu[2])) text(3,.2, quote(h[1]:~~mu[1]==mu[2]+delta))

32 ST appendix - Eric Wolsztynski, UCC p.32/40 Files and plots An example with mathematical annotations Density H 0 : µ 1 = µ 2 H 1 : µ 1 = µ 2 + δ P(Z > 1.96, H 1) = 0.85 P(Z > 1.96, H 0) = Z = µ 1 µ 2 σ n

33 ST appendix - Eric Wolsztynski, UCC p.33/40 Files and plots Plotting to files R allows copy-pasting figures (easier on Windows) jpg() and pdf() can be used to plot directly to a file: pdf(../slides/figs/ecdf plot.pdf ) Make sure to have your plot well calibrated before plotting to a file In particular, double-check the output figure size and readibility

34 ST appendix - Eric Wolsztynski, UCC p.34/40 R programming R programming

35 ST appendix - Eric Wolsztynski, UCC p.35/40 R programming Some advice R programming consists in writing elaborate R instructions in an R script, which can then be re-used at a later stage It is usually better to do it in an R-dedicated editor It is strongly recommended to keep your code clear and tidy Indenting blocks of instructions is quite often time-saving Any code should also be annotated by comments, using # Write code as if for someone else: anyone will struggle to understand his/her own code after some time, if this code is not well commented and organised

36 ST appendix - Eric Wolsztynski, UCC p.36/40 R programming If if is used as follows: if (test) expr 1 else expr 2 R> if(0) print( un ) else print( deux ) R> if(0) print( un ) else {print( deux ); a=3; print(a)} An extended use with large grouped instructions is as follows: if(test){ expr 1 expr 2... } else { expr 3 expr 4... }

37 ST appendix - Eric Wolsztynski, UCC p.37/40 R programming Loops The for loop: for (name in expr 1) expr 2 R> for (i in 1:10) { print(i) } R> x=0; bag=c(1,2,5); for (i in bag) { x=x+i; print(x) } Another example, scanning matrix M: for(i in 1:nrow(M)){ for(j in 1:ncol(M)){ print(m[i,j]) } } Other loops: repeat expr and while (condition) expr break() terminates any loop (must be used in repeat): R> i=10; repeat {i=i-1; if(i==0) break()} next() skips to the next loop cycle

38 ST appendix - Eric Wolsztynski, UCC p.38/40 R programming Functions (1) R functions define a sequence of instructions to be re-used Synopsis: name <- function(arg 1, arg 2,...) expression A call to the function is then name(expr 1, expr 2,...) Some simple examples: R> helloworld <- function(){ print( Hello World! ) } R> helloworld() # displays "Hello World!" R> plusone <- function(a){ return(a+1) } R> b = 2; c = plusone(b); c # displays 3

39 ST appendix - Eric Wolsztynski, UCC p.39/40 R programming Functions (2) It is convenient to write functions in a separate.r file Such definition files can be loaded at any point by source() Example: source( bin/myfuns.r ) loads all functions defined within myfuns.r in the R environment Function arguments can be given a default value: fun1 <- function(data, graph=true, limit=20) {... } Calls to the newly defined function fun1 may then be: ans <- fun1(d) ans <- fun1(d, limit=10)

40 ST appendix - Eric Wolsztynski, UCC p.40/40 R programming Functions (3) Always be cautious of the scope of existing variables! f <- function(x) { y <- 2*x print(x) print(y) print(z) } In this function, x is a formal parameter (defined as a function argument) y is a local variable (defined within the function body) z is a free variable (with a value defined elsewhere) Also, avoid using existing names to define new local variables!