Statistical Programming with R

Transcription

1 Statistical Programming with R Dan Mazur, McGill HPC daniel.mazur@mcgill.ca guillimin@calculquebec.ca May 14,

2 Outline R syntax and basic data types Documentation and help pages Reading in data Data exploration Programming in R Probability, distributions, and pseudorandom numbers Linear Regression Confidence Intervals Statistical Hypothesis Testing Parallel Programming in R

3 What is R R is... A programming language Designed for interactive, statistical programming and plotting Free software. You may... Run the software for any purpose Inspect the source code Share copies with others Modify and improve R Share your improvements with the whole community

4 Exercise 0: Login and Setup Login to Guillimin $ ssh -X class##@guillimin.hpc.mcgill.ca Load software modules $ module add R/3.1.2 $ module add Rstudio (optional) Copy workshop files $ cp -R /software/workshop/r/* ~/. (on Guillimin) or, us git $ git clone

5 Exercise 1: Invitation to R To launch R, type 'R' in the terminal Let's skip hello world and start analyzing data In the prompt, we will load a data table, get a summary of each column, and make scatterplots of every pair of variables > library(datasets) #Import a library full of data sets >?airquality #Access the help tool for one of the datasets > data(airquality) #Load one of the datasets > summary(airquality) #Print summary of dataset > pairs(airquality) #Make scatterplots of all pairs of variables In a few commands, we just learned a lot about our data!

6 The R Assignment Operator In R, the assignment operator is '<-' Looks like a left-pointing arrow a <- 3 In most languages, assignment is done using the '=' symbol R supports both a = 3 a <- 3 Google's R style guide forbids '=' for assignment Consider this ambiguity f(a = 3) - Call f() with named variable a set to 3 f(a <- 3) - Assign 3 to a, call f() with first positional variable set to the value of a

7 R data types Atomic data types (similar to other languages) character, numeric, integer, complex, logical (TRUE and FALSE), factors Lists Vectors Created with the c() function: > a <- c(1,2,3,4,5) Character Strings Use 'single quotes' or double quotes Data Frames Collections of vectors (like a database table or spreadsheet) Tables Frequency tables Arrays and Matrices

8 Factors Non-numeric data In the air quality data, Month is a label, not numeric data By default, the summary() function will compute quantiles for numeric data, but frequencies for factors > summary(airquality$month) Min. 1st Qu. Median Mean 3rd Qu. Max > airquality$month <- as.factor(airquality$month) > summary(airquality$month)

9 Vectors Vector - an ordered sequence of data elements of a single type Created with the c() function: a <- c(1,2,3,4,5) Summary functions: sort(), min(), max(), mean(), var(), median(), quantile(), sum() Operators +,-,*,/,sqrt(),log(),exp() are element-wise > c(1,2,3,4)/5 [1]

10 Lists List - an ordered sequence of data elements of any type A generalized vector a <- list(2, a, TRUE, c(3,4))

11 Indexing in R Lists and arrays are indexed using square brackets > a <- c(2,4,6) > a[2] [1] 4 We can give list elements names, and index by names > a <- list(alice=2, bob=6, carol=12) > a[['carol']] 12 > a$bob 6 use double bracket [[]] notation to select individual items from a list, and single bracket [] notation to select a sublist of elements Use the dollar sign $ notation to select named members You can index lists with logicals > a[ a > 3 ] $bob [1] 6 $carol [1]

12 Data Frames Data frame - Collection of vectors of different types > myframe <- data.frame(col1=c(1,2,3,4), col2=c(2,4,6,8), col3=factor(c("a","b","c","d"))) > myframe col1 col2 col A B C D Our airquality data was imported as a data frame

13 Data Tables Frequency tables > data(titanic) > class(titanic) [1] "table" > Titanic,, Age = Child, Survived = No Sex Class Male Female 1st 0 0 2nd 0 0 3rd Crew 0 0,, Age = Adult, Survived = No

14 Arrays and Matrices Array - A multiply subscripted collection of data entries of a single type Matrix - A 2-dimensional array Vector - a 1-dimensional array # Generate a 2 by 2 by 5 array. > x <- array(1:20, dim=c(2,2,5)) > x[,,2] [,1] [,2] [1,] 5 7 [2,] 6 8 > x[,2,] [,1] [,2] [,3] [,4] [,5] [1,] [2,] # Generate a 3 by 2 array (matrix). > B = matrix( + c(2, 4, 3, 1, 5, 7), + nrow=3, + ncol=2) > B # B has 3 rows and 2 columns [,1] [,2] [1,] 2 1 [2,] 4 5 [3,]

15 Indexing Arrays Indices for each dimension are separated by commas An absent index is treated as 'everything' > x[, 2, 2:4 ] [,1] [,2] [,3] [1,] [2,]

16 Indexing with Logicals I want to set all elements of a vector that are less than 5 to the value 0. Which line of code achieves this? x <- c(7, 2, 9, 5, 2, 7, 8, 1, 0) A) x[x == 0] < 5 B) x[x < 5] <- 0 C) x[x == 5] <- 0 D) x[x < 0 ] <- 5 E) None of these > x [1]

17 Indexing with Logicals I want to set all elements of a vector that are less than 5 to the value 0. Which line of code achieves this? x <- c(7, 2, 9, 5, 2, 7, 8, 1, 0) A) x[x == 0] < 5 B) x[x < 5] <- 0 C) x[x == 5] <- 0 D) x[x < 0 ] <- 5 E) None of these > x [1]

18 Appending Data cbind() and rbind() can append data to a vector, matrix, or data frame > a <- data.frame(one=c( 0, 1, 2),two=c("a","a","b")) > c <- rbind(a,b) > b <- data.frame(one=c(10,11,12),two=c("c","c","d")) > c > a one two one two 1 0 a 1 0 a 2 1 a 2 1 a 3 2 b 3 2 b 4 10 c > b 5 11 c one two 6 12 d 1 10 c > d <- cbind(a,b) 2 11 c > d 3 12 d one two one two 1 0 a 10 c 2 1 a 11 c 3 2 b 12 d

19 Documentation and Help Pages Built-in help system can be accessed using the syntax '?function' try:?mean Official documentation is available on the R- project web site: There are many tutorials and tips on the web

20 What type of data is a? > a <- rbind( c(2,4,6), c(3, 2, 7) ) Hint: look at '?rbind' Rule: Do not run this command A) A vector of length 2 B) A vector of length 6 C) A 3 by 2 matrix D) A 2 by 3 matrix E) Something else

21 What type of data is a? > a <- rbind( c(2,4,6), c(3, 2, 7) ) Hint: look at '?rbind' Rule: Do not run this command A) A vector of length 2 B) A vector of length 6 C) A 3 by 2 matrix D) A 2 by 3 matrix E) Something else

22 Exercise 2: Reading in Data Use less (or any text viewer) to view the contents of titanic.csv $ less titanic.csv use 'q' to exit less Use the read.csv() function to load the titanic data set > titanic <- read.csv('titanic.csv', sep=',') Use ls() to see what variables are defined > ls() [1] "titanic" Use some of the commands we've already seen to explore the data

23 Reading in Data There are many tools for reading data from many sources into R read.csv(), read.table(), scan() RMySQL, hdf5, netcdf, bmp, jpeg, png, Consider data in a table on a website e.g. This code loads a data.frame from a website designed for human eyes library('xml') url <- " zz = readhtmltable(url) if(any(i <- sapply(zz, ncol) == 14)) { zz = zz[[which(i)[1]]] }

24 Print() The print() function can be used to produce output or to inspect variables Each data-type in R can implement its own output under print() > print("hello world") [1] "hello world" > print(c(2,3,4)) [1] > print(airquality) Ozone Solar.R Wind Temp Month Day head() produces the first 6 (default) lines, tail() produces the last

25 Exercise 3: Data Exploration Find commands to perform the following tasks: Print only the first two rows of titanic Print only the last two rows of titanic Compute the total number of survivors (hint: use sum()) Compute the mean age of all passengers (hint: use mean() with the na.rm option) How many passengers have unknown ages? (hint: Missing values are labeled as NA in R. Use the is.na() function to find them)

26 Simple Plots R is capable of making publication-quality plots and charts Disclaimer: This workshop will only cover the basic plotting commands Great for quickly peeking at your data Making publication-quality graphs is outside our scope Other tutorials will teach you how to make plots for publication

27 Exercise 4: Simple Plots Load the airquality data set > library(datasets) > data(airquality) Try the following plot commands and understand what they are doing (use?plot ): > pairs(airquality) > hist(airquality$temp) > boxplot(airquality) > stripchart(airquality) > plot(airquality$solar.r, airquality$ozone)

28 Exercise 5: Subsets In the subset of airquality data where Ozone > 31 and Temp > 90, what is the mean value of Solar.R? A) B) C) D) E) None of these

29 Exercise 5: Subsets In the subset of airquality data where Ozone > 31 and Temp > 90, what is the mean value of Solar.R? A) B) C) D) E) None of these

30 Exercise 5: Solution #Load the airquality data set library(datasets) data(airquality) #Compute the filter vector for the subset filter.vector <- airquality[,"ozone"] > 31 & airquality[,"temp"] > 90 #Compute the mean of Solar.R for the subset mean(airquality[filter.vector, "Solar.R"], na.rm=t) #Alternatively, we can use the subset() function mean(subset(airquality, Ozone > 31 & Temp > 90)$Solar.R)

31 Programming in R The basic programming features in R are For loops While loops Switch statements functions anonymous functions

32 Using Loops in R Use for() to loop over elements of lists Use while() to loop until a condition is satisfied > words = c("these", "are", "the", "words") for (word in words) { print(word) } [1] "These" [1] "are" [1] "the" [1] "words" for (i in 1:3) { print(i) } [1] 1 [1] 2 [1] 3 while(i < 2) { i <- i+1 print(i) } [1] 1 [1]

33 Not Using Loops in R Loops in R tend to be slow It is preferred to avoid them when possible Use vectorized operations as much as possible (more later) Reduce() function iterates a function over a list or vector R provides functions for applying a function repeatedly to sets of data apply() - apply function to sections of array, return array sapply() - apply function to elements of a list, returns a vector, matrix, or list lapply() - apply function to elements of a list, return list Use '?apply', or '?Reduce' for more details

34 Functions in R In R, functions are objects that get assigned variable names foo <- function(bar) { x <- rnorm(bar) mean(x) } foo(1:10) [1]

35 Default values for arguments You may specify default values for optional arguments in functions foo <- function(bar, message=false) { x <- rnorm(bar) if (message) {print( Computing... )} mean(x) } > foo(1:10) [1] > foo(1:10,true) [1] "Hello" [1] > foo(1:10,message=true) [1] "Hello" [1]

36 Anonymous Functions in R You can use a function without giving it a name > (function(x,y)x+y)(5,6) [1] 11 Useful if small function is only needed on one line This is most frequently used with the apply() set of functions > lapply(c(2,4,6), function(x) x**2) [[1]] [1] 4 [[2]] [1] 16 [[3]] [1]

37 What is the output? What is the output of this R code? foo <- function(x) { bar <- function(y) { y+z } z <- 5 x + bar(x) } z <- 10 foo(2) A) 0 B) 9 C) 14 D) 21 E) None of these

38 What is the output? What is the output of this R code? foo <- function(x) { bar <- function(y) { y+z } z <- 5 x + bar(x) } z <- 10 foo(2) A) 0 B) 9 C) 14 D) 21 E) None of these

39 Lexical Scoping foo <- function(x) { bar <- function(y) { y+z } z <- 5 x + bar(x) } z <- 10 foo(2) Which z value will R use inside bar()? 5 or 10? The answer is z = 5. The output will be 9. Lexical scoping specifies that the version of a free variable that is in scope when the function is defined will be used Recursive search through environment of definition A dynamically scoped language would use z = 10 in bar() with output =

40 What is the output? foo<-function(x, y=x) x+y x<-100 foo(2, x) foo(2) A) [1] 102 [1] 102 B) [1] 4 [1] 4 C) [1] 102 [1] 4 D) [1] 4 [1]

41 What is the output? foo<-function(x, y=x) x+y x<-100 foo(2, x) foo(2) A) [1] 102 [1] 102 B) [1] 4 [1] 4 C) [1] 102 [1] 4 D) [1] 4 [1]

42 Closures Lexical Closure: Data structure that stores a function along with its environment Free variables are captured at the time the closure is defined

43 Closures: Example With regular functions (all functions are closures) add1 <- function(x){ x + 1 } add2 <- function(x){ x + 2 } add3 <- function(x){ x + 3 } add1(10) add2(9) add3(8) With closures addn <- function(n) { function(x) { x + n } } add1 <- addn(1) add2 <- addn(2) add3 <- addn(3) add1(10) add2(9) add3(8) Q: How does R know what value of n to use each time? A: Lexical scoping: n gets defined when the closure is defined

44 Superassignment The superassignment operator, <<-, can assign to a variable higher up in the stack Be careful with the superassignment operator Normally, R functions have no side effects, but <<- can cause side effects R functions are always pass-by-value

45 Exercise 6: Counter Closure Write a closure, counter() Sets a local variable, ctr <- 0 Defines a function f() increments ctr (See note below) prints the current value of ctr Returns f() Test your closure > c1 <- counter() > c2 <- counter() > c1() > c2() > c1() > c2() Note: We cannot increment ctr inside of f() with the <- operator because it is in a higher lexical scope The superassignment operator, <<-, will start searching up in scope until it finds a variable to assign We could also use the assign() function

46 Debugging in R traceback() - Prints the sequence of function calls debug(functionname) - Flag a function for debugging When functionname runs, you will be taken to the browser the browser has four basic debug commands n[ext] - Execute current line and print the next c[ontinue] - Execute the rest of the function without stopping Q[uit] - Exit the browser where - Show the call stack May also use other R commands such as print()

47 Exercise 7: lapply() The coefficient of variation of a vector x is defined as: sd(x)/mean(x) Use lapply() with an anonymous function to compute the coefficient of variation for the Wind and Temp columns of airquality simultaneously Don't use any loops

48 Exercise 7: lapply() The coefficient of variation of a vector x is defined as: sd(x)/mean(x) Use lapply() with an anonymous function to compute the coefficient of variation for the Wind and Temp columns of airquality simultaneously Don't use any loops Solution: > lapply(airquality[,c("wind", "Temp")], function(x) sd(x)/mean(x))

49 Exercise 8: Debugging Run the script debug.r source('debug.r') Use traceback() to discover which function has the error Use debug() to flag the appropriate function for debugging > debug(func#) > func1(-1) > Browse[2]> print(x) > Browse[2]> n

50 Probability and Distributions height of the probability density function cumulative density function inverse cumulative density function (quantiles) generates pseudorandom numbers Uniform Gaussian T-distribution Binomial Chi-squared dunif() dnorm() dt() dbinom() dchisq() punif() pnorm() pt() pbinom() pchisq() qunif() qnorm() qt() qbinom() qchisq() runif() rnorm() rt() rbinom() rchisq()

51 Probability and Distributions > plot(1:30,dchisq(1:30, df=10), type='l') > par(new=t) > hist(rchisq(10000, df=10), axes=false, xlab="", ylab="")

52 Exercise 9: Confidence Intervals What are the range of values such that 95% of the area is contained (i.e. the 95% confidence interval, see figure)? Write the R commands to produce the answer hint: use qnorm() This is a Gaussian distribution with mean=0 and standard deviation=1 2.5% 95% 2.5%

53 Linear Regressions Want to model the relationship between a dependent (target) variable, y, and one or more explanatory variables, x The lm() function is used to fit linear models in R Use Wilkinson notation to define statistical models in R [response variable] ~ [predictor variables] e.g.: Species ~ Sepal.Length + Sepal.Width + Petal.Length + Petal.Width

54 Linear Regression > x <- 0:100 > e <- rnorm(101, 0, 10) > y < * x + e > model <- lm(y ~ x) > summary(model) Call: lm(formula = y ~ x) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) x <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 99 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: 2468 on 1 and 99 DF, p-value: < 2.2e

55 Linear Regression > plot(x, y) > abline(model)

56 Linear Regression The predict() function can make predictions about the value of target variable, y, for new data Use se.fit=true to get standard errors > new_data <- data.frame(x = c(40, 50, 60, 120)) > predict(model, new_data, se.fit=true) $fit $se.fit $df [1] 99 $residual.scale [1]

57 Model Formula Notation Include all variables e.g.: These formulae refer to the same model

58 Exercise 10: Linear Regression In the airquality data, use a linear model with Ozone as the dependent variable and Temp and Wind as the predictor variables Include interactions between Temp and Wind in your model Predict the Ozone level on a day when Temp=80 degrees F, and Wind=15 mph hint: Don't use dollar signs in your model formula lm(ozone ~..., airquality) not lm(airquality$ozone ~...) > new_data <- data.frame(temp=80, Wind=15)

59 Linear Regression Solution > library(datasets) > data(airquality) > mymodel <- lm(ozone ~ Temp*Wind, airquality) > new_data <- data.frame(temp=80, Wind=15) > predict(mymodel, new_data, se.fit=true) $fit $se.fit [1] $df [1] 112 $residual.scale [1] A :

60 Hypothesis Testing > titanic <- read.csv('titanic.csv') > mean(titanic$age, na.rm=t) [1] > t.test(titanic$age, mu=30, alternative="greater") One Sample t-test data: titanic$age t = , df = 632, p-value = alternative hypothesis: true mean is greater than percent confidence interval: Inf sample estimates: mean of x t.test() is used for Student's T-Test Sample mean is Test if true mean is greater than 30 We can see T-statistic Number of degrees of freedom P-value of the test Confidence interval

61 Exercise 11: Two-sample testing t.test() can also perform two-sample testing if given two vectors (samples) instead of one Test whether the true mean of the ages of 1 st class passengers differs from the true mean of the ages of 3 rd class passengers Hints Prepare two vectors, one for first class and one for third class ages Use alternative= two.sided (or omit because two.sided is default)

62 Solution: Two-sample testing > ages.first <- titanic[titanic$pclass == "1st", "age"] > ages.third <- titanic[titanic$pclass == "3rd", "age"] > t.test(ages.first, ages.third, alternative="two.sided") Welch Two Sample t-test data: ages.first and ages.third t = , df = , p-value < 2.2e-16 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: sample estimates: mean of x mean of y

63 Vectorization in R Consider a dot-product between two vectors of length 1,000,000 The time required for a for loop is more than 100x longer than the time required for a vectorized operation > a <- rnorm(1e6) > b <- rnorm(1e6) > dotprod <- 0 > system.time(sum(a*b)) user system elapsed > system.time(for (i in 1:length(a)) { dotprod <- dotprod +a[i]*b[i]}) user system elapsed

64 Vectorization in R Two main reasons for the poor performance of the dot product loop The for loop and even the colon (':'), and vector subscripts are implemented as R functions that require overhead to set up stack frames and create the environment Objects in R are pass-by-value => Data must be copied into the environments set up for those functions Many R functions are vectorized sqrt(), log(), exp(), sin(), etc

65 Parallel Programming in R Built-in package 'parallel' Included in versions >= Builds on and merges 'snow' (simple network of workstations) and 'multicore' packages

66 Setting up a parallel system library(parallel) # report the number of available cores detectcores() [1] 4 # Create a cluster with 3 cores cl <- makecluster(3)

67 parlapply() The most common way to parallelize R code is to replace expensive lapply() operations with the parallel parlapply() from the parallel package library(parallel) # report the number of available cores detectcores() [1] 4 # Create a cluster with 3 cores cl <- makecluster(3) > mylist <- seq(1e6) > cl socket cluster with 3 nodes on host localhost # Time lapply > system.time(mylist2 <- lapply(mylist, function(x) x^2)) user system elapsed # Time parlapply > system.time(mylist2 <- parlapply(cl, mylist, function(x) x^2)) user system elapsed

68 Exercise 12: Parallel R The file lapply-serial.r contains a simple serial code where most of the time is spent on an lapply() operation. Parallelize this code using the 'parallel' package in R

69 Summary Today we learned How to launch the R interpreter How to load and use external libraries How to work with R data types such as data frames How to create and call functions How scoping works in R How to work with probability distributions How to perform linear regressions How to compute confidence intervals How to perform a hypothesis test How to parallelize your code using the parallel and snow libraries

70 The End What questions do you have?