Computing with large data sets

Transcription

1 Computing with large data sets Richard Bonneau, spring 009 (week ): introduction to R

2 other notes, courses, lectures about R and S Ingo Ruczinski and Rafael Irizarry (Johs Hopkins Biostat): Roger D. Peng (JHU): Read the manual!!: v.0480: computing with data, Richard Bonneau

3 S history S is a language and system for organizing, visualizing, and analyzing data. S started at Bell Labs since 976. The language has evolved through several major versions to become the most widely used environment for research in data analysis and statistics. In 998, S became the first statistical system to receive the Software System Award, the top software award from the ACM. ( For a great account of the early history of S see the paper on the course website ) v.0480: computing with data, Richard Bonneau

4 R history and facts R is an environment for data analysis and visualization. R is an open source implementation of the S language (S-Plus is a commercial implementation of the S language). The current version of R (September 004) is.9.. The R Core group consists of Doug Bates, John Chambers, Peter Dalgaard, Rober t Gentleman, Kur t Hornik, Stefano Iacus, Ross Ihaka, Friedrich Leisch, Thomas Lumley, Mar tin Maechler, Guido Masarotto, Paul Murrell, Brian Ripley, Duncan Temple Lang, and Luke Tierney. join the R Foundation for Statistical Computing 99 Ross Ihaka and Rober t Gentleman begin work on a project that will ultimately become R. 99 Design and implementation of pre-r. 993 The first announcement of R. 995 R available by ftp under the GPL. 996 A mailing list is star ted and maintained by Martin Maechler at ETH. 997 The R core group is formed. 999 DSC meeting in Vienna, the first time many R core members meet. 000 R.0.0 is released. 009 R is still very actively developed and availiable for all platforms, open source, pervasive in bioinformatics and several other fields. v.0480: computing with data, Richard Bonneau

5 playing around < > y <- 4 <-.0 > y <- c(,4,6) * y [] 4 8 * [] 4 > y * y [] > sqrt( - ) [] NaN Warning message: In sqrt(-) : NaNs produced > sqrt(-+0i) [] 0+i v.0480: computing with data, Richard Bonneau

6 playing around > y <- :0 > y ^^ Error: syntax error > y ^ [] > y [] > y <- jitter( y ) > y [] [0] > class( y ) [] "numeric" > class( x ) [] "numeric" > length( x ) [] > length( y ) [] 0 > dim( y ) NULL > dim( x ) NULL v.0480: computing with data, Richard Bonneau

7 playing around > z <- matrix( sample(y), nrow = 5, ncol = 5) > z [,] [,] [,3] [,4] [,5] [,] [,] [3,] [4,] [5,] > dim(z) [] 5 5 > length( z ) [] 5 > summary( y ) Min. st Qu. Median Mean 3rd Qu. Max v.0480: computing with data, Richard Bonneau

8 playing around Histogram of y > hist( y ) > hist( y, nclass = 0 ) > hist( y ) > pdf("hist.l.pdf") > hist( y ) > dev.off() quartz <- :0 > y <- runif( length( x ) ) > plot( x, y ) > abline(h=0.5, lty=, col="green",lwd=) > pdf("sample-sesion.pdf") > plot( x, y ) > abline(h=0.5, lty=, col="green",lwd=) > dev.off() quartz Frequency y y x v.0480: computing with data, Richard Bonneau

9 using built in examples >? heatmap ### then cut and paste in exmples > require(graphics); require(grdevices) <- as.matrix(mtcars) > rc <- rainbow(nrow(x), start=0, end=.3) > cc <- rainbow(ncol(x), start=0, end=.3) > hv <- heatmap(x, col = cm.colors(56), scale="column", + RowSideColors = rc, ColSideColors = cc, margins=c(5,0), + xlab = "specification variables", ylab= "Car Models", + main = "heatmap(<mtcars data>,..., scale = \"column\")") map(<mtcars data>,..., scale = "column") ## mtcars is a datastructure provided as an example ## of how to use heatmap() ## > str( mtcars ) 'data.frame': 3 obs. of variables: $ mpg : num $ cyl : num $ disp: num $ hp : num $ drat: num $ wt : num $ qsec: num $ vs : num $ am : num $ gear: num $ carb: num >?mtcars ## for description of what it actually is ## v.0480: computing with data, Richard Bonneau cyl am vs carb wt drat gear qsec specification variables mpg hp disp Toyota Corona Porsche 94 Datsun 70 Volvo 4E Merc 30 Lotus Europa Merc 80 Merc 80C Mazda RX4 Wag Mazda RX4 Merc 40D Ferrari Dino Fiat 8 Fiat X 9 Toyota Corolla Honda Civic Merc 450SL Merc 450SE Merc 450SLC Dodge Challenger AMC Javelin Hornet 4 Drive Valiant Duster 360 Camaro Z8 Ford Pantera L Pontiac Firebird Hornet Sportabout Cadillac Fleetwood Lincoln Continental Chrysler Imperial Maserati Bora Car Models

10 dumping functions : example code galore > hist ### type function with no ( ) or argument function (x,...) UseMethod("hist") <environment: namespace:graphics> map(<mtcars data>,..., scale = "column") you get info, but not code if the function is part of the main R code (part of the base or core) > heatmap ### for higher level functions ### or defined functions you ### you get the code function (x, Rowv = NULL, Colv = if (symm) "Rowv" else NULL, distfun = dist, hclustfun = hclust, reorderfun = function(d, w) reorder(d, w), add.expr, symm = FALSE, revc = identical(colv, "Rowv"), scale = c("row", "column", "none"), na.rm = TRUE, margins = c(5, 5), ColSideColors, RowSideColors, cexrow = 0. + /log0(nr), cexcol = 0. + /log0(nc), labrow = NULL, labcol = NULL, main = NULL, xlab = NULL, ylab = NULL, keep.dendro = FALSE, verbose = getoption("verbose"),...) { scale <- if (symm && missing(scale)) "none" else match.arg(scale) if (length(di <- dim(x))!=!is.numeric(x)) stop("'x' must be a numeric matrix")... truncated cyl am vs carb wt drat gear qsec mpg specification variables hp disp Toyota Corona Porsche 94 Datsun 70 Volvo 4E Merc 30 Lotus Europa Merc 80 Merc 80C Mazda RX4 Wag Mazda RX4 Merc 40D Ferrari Dino Fiat 8 Fiat X 9 Toyota Corolla Honda Civic Merc 450SL Merc 450SE Merc 450SLC Dodge Challenger AMC Javelin Hornet 4 Drive Valiant Duster 360 Camaro Z8 Ford Pantera L Pontiac Firebird Hornet Sportabout Cadillac Fleetwood Lincoln Continental Chrysler Imperial Maserati Bora Car Models v.0480: computing with data, Richard Bonneau

11 R basic types / atomic classes of objects <- character() ### char, strings, vectors or both <- "test" [] "test" [] <- "test" [] <- "test" [3] <- "test" [] "test" "test" "test" > class(x) [] "character" <- numeric() ### double floats, vectors of floats <- complex() ### complex numbers complex(0) <- logical() ## logicals, can be used ## to index other objects <- > class(x) [] "numeric" <- L ### force integer > class(x) [] integer <- Inf ## infinity [] Inf <- NA ### missing values are NA or NaN [] NA > is.na( x ) ### built in functions help in dealing ### with NAs [] TRUE <- logical() logical(0) <- NaN > is.na( x ) [] TRUE v.0480: computing with data, Richard Bonneau

12 NA, NaN, empty/missing values Values can be missing for lots of good reasons. Technical: -the measurement failed (it was cloudy that night, the probe for that DNA was synthesized incorrectly) Budgetary/Social: - we could only afford to measure so many points / attributes - people will only answer 5 minutes of questions... Bugs (incorrect explicit type coercion) Values not filled in YET <- Inf ## infinity [] Inf ### this IS a number <- NA ### missing values are NA or NaN [] NA > is.na( x ) ### built in functions help in dealing ### with NAs [] TRUE > ### messed up explicit coercion <- c( "f", "fg" ) > as.numeric ( x ) [] NA NA Warning message: NAs introduced by coercion see also: is.nan(), is.null(), as.null() v.0480: computing with data, Richard Bonneau

13 R basic types vectors Integers <- : > class( x) [] "integer" <- c(l, L, 3L) [] 3 Numeric <- c(,, 3.) [] Logical <- c( TRUE, TRUE, FALSE) [] TRUE TRUE FALSE Logical from conditional statement <- c("azure", "red", "green", "red") [] "azure" "red" "green" "red" == "azure" [] TRUE FALSE FALSE FALSE <- c(,, 3.) [] <. [] TRUE TRUE FALSE Integer indexes from conditionals > which ( x <. ) [] v.0480: computing with data, Richard Bonneau

14 R basic types: vectors <- numeric( 0 ) ## a length 0 numeric vactor ## short for print(x) [] <- character( 0 ) [] "" "" "" "" "" "" "" "" "" "" [ length(x) + ] <- "a" [] "" "" "" "" "" "" "" "" "" "" "a" <- c(x, "b") [] "" "" "" "" "" "" "" "" "" "" "a" "b" > ### attributes > length ( x ) [] > names( x ) NULL > str( x ) chr [:] "" "" "" "" "" "" "" "" "" "" "a" "b" <- :5 ## loading atributes > names( x ) <- c("one", "two", "three", "four", "five") one two three four five > names( x ) [] "one" "two" "three" "four" "five" > class( x ) [] "integer" v.0480: computing with data, Richard Bonneau

15 creative ways of making nasty bugs > ## you can, but shouldn't do nutz stuff like this <- c(, "two" ) [] "" "two" > class(x ) [] "character" > y <- c(,0,true, FALSE) > y [] 0 0 > class( y ) [] "numeric" > y <- c( "true", TRUE, FALSE) ## nuts! > y [] "true" "TRUE" "FALSE" > class( y ) [] "character" > ## creative ways of writing nasty nasty bugs R variables, vectors and matrices assume the type st specified OR loaded. assigning different types later in the code will often override this initial type. for example <- :0 > example.function( x ) ## function returns a charcter vec <- length( x ) * pi <- FALSE x has been 4 types in 4 lines of code v.0480: computing with data, Richard Bonneau

16 factors... Making a factor vector > youare <- as.factor ( c("m", "F", "F", "U" ) ) > youare [] M F F U Levels: F M U > youare <- rep(, 0) > youare [] >?runif > y <- runif( 0 ) > youare[ y > 0.5 ] <- "big" > youare[ y <= 0.5 ] <- "small" > youare [] "big" "big" "big" "big" "big" "small" "big" "small" "big" "big" > as.factor(youare) [] big big big big big small big small big big Levels: big small Factors are integers with a label, but the label is stored much more efficiently (once for the whole vector of factors) Using Factors is better in that they have meaningful attributes... why say,, 3 as integers when you can say male, female, undetermined? Many functions ( functions that aim to classify instances based on vectors of mixed attributes) use factors. v.0480: computing with data, Richard Bonneau

17 forcing type conversions, explicit coercion > ### explicit coercion --- forcing the type <- character( "", "", "3", "4", "0", "0" ) Error in character("", "", "3", "4", "0", "0") : unused argument(s) ("", "3", "4", "0", "0") <- c( "", "", "3", "4", "0", "0" ) > class ( x ) [] "character" [] "" "" "3" "4" "0" "0" <- as.numeric( x ) [] > str( x) num [:6] > as.logical( x ) [] TRUE TRUE TRUE TRUE FALSE FALSE > as.complex( x ) [] +0i +0i 3+0i 4+0i 0+0i 0+0i > as.integer( x ) [] * remember, many times coercion to the type you think is a good way of checking youʼve read in OR computes what you think you have... e.g. coercion of a character to a numeric can often produce NAs that lead you to bugs. so declaring and coercion of type is a good idea even if R doesnʼt strictly require it. v.0480: computing with data, Richard Bonneau

18 coercion of matrix objects <- c(,,3,4,0,0) [] > matrix( x, ncol =, nrow = ) [,] [,] [,] 3 [,] 4 > matrix( x, ncol =, nrow = 3 ) [,] [,] [,] 4 [,] 0 [3,] 3 0 > matrix( x, ncol =, nrow = 4 ) [,] [,] [,] 0 [,] 0 [3,] 3 [4,] 4 Warning message: In matrix(x, ncol =, nrow = 4) : data length [6] is not a sub-multiple or multiple of the number of rows [4] > ### but it still did it!!!!! is this a feature or a bug waiting to happen? <- c(,,3,4,0,0) > dim(x) <- c(3,) [,] [,] [,] 4 [,] 0 [3,] 3 0 > ### but the dim has to match the length? v.0480: computing with data, Richard Bonneau

19 matrix names <- c(na, NA, ) [] NA NA > is.na(x) [] TRUE TRUE FALSE > > y <- matrix( x, ncol =, nrow = 3 ) > > dim(y ) [] 3 > y [,] [,] [,] NA NA [,] NA NA [3,] > y[ is.na(y) ] < > y [,] [,] [,] [,] [3,] > y[,] < > rownames( y ) <- c( "eq", "er", "es") > colnames( y ) <- c("qr", "rq" ) > dimnames( y ) [[]] [] "eq" "er" "es" [[]] [] "qr" "rq" > y qr rq eq er es > v.0480: computing with data, Richard Bonneau

20 matrices > ## matrix are filled starting in the upper left courner and then running down > ## the column. The first indexis the row, and the second is the > > y <- :0 > dim(y) <- c(,5) > y [,] [,] [,3] [,4] [,5] [,] [,] > dim(y) <- c(5,) > y [,] [,] [,] 6 [,] 7 [3,] 3 8 [4,] 4 9 [5,] 5 0 > dim(y) <- c(5,5) ### oops? Error in dim(y) <- c(5, 5) : dims [product 5] do not match the length of object [0] v.0480: computing with data, Richard Bonneau

21 rbind, cbind <- :0 > y <- 0: > z <- c(:5, 5:) yz <- rbind( x,y,z ) yz [,] [,] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,0] x y z yz <- cbind( x,y,z ) yz x y z [,] 0 [,] 9 [3,] [4,] [5,] [6,] [7,] [8,] [9,] 9 [0,] 0 > > ## adding to a matrix one row at a time yz <-rbind( xyz, c( 3,4,5) ) yz x y z [,] 0 [,] 9 [3,] [4,] [5,] [6,] [7,] [8,] [9,] 9 [0,] 0 [,] > ## could do a similar thing with cbind() v.0480: computing with data, Richard Bonneau

22 lists Making a list > p <- list() > ### Declare a list > p$x <- > p$x <-.0 Error in p$x <- : object "p" not found > p$x <-.0 > p <- list() > p$x <- 3.0 > p$y <-.0 > p$y <-.0 > p$y <-.0 > p $x [] $y [] > p $x [] 3 $y [] Making a list of lists > all.p <- list() > all.p[[]] <- p > all.p[[]] <- p > all.p [[]] [[]]$x [] [[]]$y [] [[]] [[]]$x [] 3 [[]]$y [] Naming and accessing lists: > names( all.p ) <- c("p","p") > all.p $p $p$x [] $p$y [] $p $p$x [] 3 $p$y [] > all.p$p $x [] $y [] > all.p$p$x [] > all.p[[]]$x [] > all.p[[]][[]] [] v.0480: computing with data, Richard Bonneau

23 lists are a great way to return and pass data <- rnorm( 0,., 0.5 ) ## 0 draws from a normal N(., 0.5) > hist.x <- hist( x ) > hist.x $breaks [] $counts [] $intensities [] $density [] $mids [] $xname [] "x" $equidist [] TRUE Frequency Histogram of rnorm(000,., 0.5) attr(,"class") [] "histogram" > class(hist.x ) [] "histogram" ## so it is not ʻjustʼ a list ## more on that later rnorm(000,., 0.5) v.0480: computing with data, Richard Bonneau

24 a strange thing lists do... name autocompletion <- rnorm( 0,., 0.5 ) ## 0 draws from a normal N(., 0.5) > hist.x <- hist( x ) > hist.x $breaks [] $counts [] $intensities [] $density [] $mids [] $xname [] "x" $equidist [] TRUE attr(,"class") [] "histogram" > class(hist.x ) [] "histogram" > ## so it is not ʻjustʼ a list > ## more on that later v.0480: computing with data, Richard Bonneau

25 dataframes v.0480: computing with data, Richard Bonneau Data.frames are tables of data, most of the time you get them by reading in tab delimited tables or flat files, read.table() Letʼs look at an example data.frame most installs of R should have loaded. > class( USJudgeRatings ) [] "data.frame" > str(usjudgeratings) 'data.frame': 43 obs. of variables: $ CONT: num $ INTG: num $ DMNR: num $ DILG: num $ RTEN: num > pairs( USJudgeRatings[,,5] ) ## this function knows ## what to do with a ## dataframe > USJudgeRatings$CONT [] > USJudgeRatings[,] [] CONT INTG DMNR DILG CFMG

26 dataframes coerce a data.frame to a matrix <- as.matrix ( USJudgeRatings ) > str( x ) num [:43, :] attr(*, "dimnames")=list of..$ : chr [:43] "AARONSON,L.H." "ALEXANDER,J.M." "ARMENTANO,A.J." "BERDON,R.I.".....$ : chr [:] "CONT" "INTG" "DMNR" "DILG"... v.0480: computing with data, Richard Bonneau

27 reading in code # ~bonneau/v-class/ > cat mean.vec.r ## function to report the mean of a vector mean.vec <- function ( x, na.remove = T ) { } if ( class( x ) == "numeric" class( x) == "integer") { return( mean(x, na.rm = na.remove) ) } else { return( NULL ) ## we could also return a NA } # ~bonneau/v-class/ > R...R startup... >? source >source( file = mean.vec.r ) ## you might need a path... > mean.vec( c(,3) ).5 > mean.vec( c(, 3, NA) ).5 > mean.vec( c( ps, qs ) ) NULL > v.0480: computing with data, Richard Bonneau

28 homework and reading for next time. Read the R manual.. non-graded homework: Make a function that: given a matrix returns a vector containing the means of each row given a list of numeric vectors returns the mean of each vector in the list for test data either use the link to small test expression matrix or use a built in R data object ( like volcano ): > dim( volcano ) [] 87 6 > str( volcano ) num [:87, :6] > dim(volcano ) [] 87 6 >? image use loops, don t worry about NAs for now, this is not a graded assignment, but give it a try to get your feet wet. if you want a hint stay after class next lecture we ll play with plotting and graphics. If you re confused there will be time to catch up next week. v.0480: computing with data, Richard Bonneau