Intermediate Programming in R Session 4: Avoiding Loops. Olivia Lau, PhD

Transcription

1 Intermediate Programming in R Session 4: Avoiding Loops Olivia Lau, PhD

2 Outline Thinking in Parallel Vectorization Avoiding Loops with Homogenous Data Structures Avoiding Loops with Heterogenous Data Structures 2

3 Thinking in Parallel Independent operations can be performed in parallel Dependent operations cannot be performed in parallel Loops where the next loop iteration depends on values from the previous iteration e.g., in Bayesian statistics, some Markov Chain Monte Carlo methods These operations are inherently serial Important in some situations where R performs particularly slowly otherwise Opens the door to parallel computing You ve already been thinking in parallel with vectorized functions 3

4 Motivating Example: ID number of missing values Could use a nested loop mv <- rep(0, col(worldbank)) for (i in 1:ncol(worldbank)) { for (j in 1:nrow(worldbank)) { mv[i] <- mv[i] + is.na(worldbank[j, i]) } } mv Or loop through variables mv <- NULL for (i in 1:ncol(worldbank)) { tmp <- is.na(worldbank[[i]]) mv[i] <- sum(tmp) } mv Or avoid loops all together apply(worldbank, 2, function(x) sum(is.na(x))) 4

5 Vectorization Vectorized function take a vector as input and output a vector of the same length, eliminating the need for nested loops when dealing with 2D data structures From the previous example, the vectorized component is tmp <- is.na(worldbank[[i]]) If you are using a vectorized function in a loop and the loop iterations are independent, you can avoid the loop altogether using *apply() functions For homogenous data structures, use apply() For heterogenous data structures, use lapply(), sapply(), tapply(), or aggregate() 5

6 Homogenous Data Structures: Use apply() apply(x, MARGIN, FUN,...) For a matrix, use the same function on every row (MARGIN = 1) or column (MARGIN = 2) Also generalizes to an array Specify the dimensions to keep Sweep over the remaining dimensions Can specify more than one dimension to keep Function output has to have the same attributes for all margins First argument to the function has to be the margin selected from X Additional arguments to FUN can be specified User-defined functions can be used Output has to have the same length or dimension over all margins 6

7 Check In 1 Revisiting Assignment 1, create a co2 matrix (countries x years) from the worldbank data co2 <- matrix(na, nrow = 214, ncol = 7, dimnames = list(null, as.character(2002:2008))) for (i in 2002:2008) { co2[, as.character(i)] <- worldbank[worldbank$year == i, "CO2"] } Calculate the sum of per capita emissions by year globally apply(co2, 2, sum, na.rm = TRUE) Calculate average of yearly emissions by country for the period apply(co2, 1, mean, na.rm = TRUE) 7

8 Heterogenous Data Structures: Options for lists In a simple case, *apply() functions work on each element of a list Assumes that each list element works as input to the function Output does not have to have the same dimension for each application of the function Two options List in and list out lapply(x, FUN,...) List in, simplified output (vector, matrix, or array) sapply(x, FUN,...) 8

9 Options for Data Frames Recall that a data frame is a hybrid between a matrix and list Can behave like a matrix If all the variables in the data frame are of the same class, use apply() by row or column If all the variables in the data frame are not of the same class, subset appropriate variables first ll <- data.frame(row.num = paste0("row", 1:100), a = rnorm(100), b = runif(100)) apply(ll[, -1], 1, mean) Can behave like a list: lapply() or sapply() by variable 9

10 Check In 2 Summarize the worldbank data set by variable using lapply() lapply(worldbank, summary) Identify the numeric variables in the worldbank data set (omitting the Year variable) and summarize them using sapply() idx <- unlist(lapply(worldbank, is.numeric)) idx <- which(idx)[-1] sapply(worldbank[, idx], summary) 10

11 More Options for Data Frames A data frame can also behave like a ragged array Use a discrete (factor) variable to index groups Function is applied over the rows defined by each group Groups can be of different sizes, hence ragged In the most general case, treat the data frame as a list and use tapply(x, INDEX, FUN,...) Works like apply(), but INDEX is a vector with the same length as X, not a margin Difference is that input to the user-specified function is the set of list elements defined by the INDEX vector Special method for data frames: aggregate(formula, data, FUN,...) Takes a formula and data argument Formula has format y ~ x y is the variable to be aggregated x are the factors by which to stratify or group the y for the aggregation function 11

12 Check In 3 Using aggregate(), calculate the sum of CO2 emissions by year from the worldbank data aggregate(co2 ~ Year, data = worldbank, sum) Calculate the average emissions level by country over the period aggregate(co2 ~ Country.Name, data = worldbank, mean) Calculate the mean emissions level by year, for Countries with life expectancy greater than 60 years Countries with life expectancy less than or equal to 60 years aggregate(co2 ~ Year + I(life.expectancy > 60), data = worldbank, mean) 12

13 Assignment Review solution to Assignment 3 Reading for this week: Chapter 6 from the course text, Paul Teetor s R Cookbook Complete Assignment 4 13