An Exploration of the Dark Arts. reshape/reshape2, plyr & ggplot2

Transcription

1 An Exploration of the Dark Arts reshape/reshape2, & ggplot2

2 Hadley Wickham Won the John M. Chambers Statistical Software Award (2006) ggplot and reshape Author of 14 R packages since 2005 Basic philosopy appears to be unifying very useful functionalities into a common sytax within R

3 The Dark Arts Wickham s wizardry Unified a lot of R s (non-modeling) functionality into three packages Allows very flexible data manipulation and visualization Syntax different, yet logical New and improved (and much faster)

4 Three unforgivable packages Unforgivable if you don t know and (sometimes) use them (split-analyze-combine paradigm for analysis) reshape/reshape2 (Data manipulation and aggregation) ggplot2 (Visualization using Grammar of Graphics)

5 Three unforgivable packages reshape reshape2 ggplot2

6 Long data

7 Long data indexed by variables x1 x2 x3 x4 x5...

8 Long data x1 indexed by variables x2 Split x3 x4 x5

9 Long data x1 indexed by variables x2 Split x3 Do something x4 x5

10 Long data indexed by variables Split x1 x2 x3 x4 x5 Do something... Put back together

11 Do something create summaries Plot by index reshape ggplot2 x1 x2 x3 x4 x5

12 Do something Transform data ddply(dat,.(by), function) Model by index ldply(dat,.(by), function)

13 Do something Transform data ddply(dat,.(by), function) Model by index ldply(dat,.(by), function) Input data.frame Input data.frame ldply Output list ddply Output data.frame

14 Have time series or repeated measurements Want baseline measurements for each subject library() baseline <- ddply(data1,"mrno", MakeBaseline) # baseline <- ddply(data1,.(mrno), MakeBaseline) # baseline <- ddply(data1,~mrno, MakeBaseline) MakeBaseline takes earliest observations from a particular subject

15 Have time series or repeated measurements Want baseline measurements for each subject

16 Want to add variables by id library() newdata <- ddply(data1,"mrno", transform, tempc=5*(temp-32)/9)

17 Want to add variables by id

18 Want to summarize data by id library() ddply(baseball, "id", summarise, duration = max(year) - min(year), nteams = length(unique(team)))

19 Want to summarize data by id

20 Want to run models by index library() ldply(warpbreaks, "tension", function(x) lm(breaks~wool, data=x)) ldply(warpbreaks, tension, function(x) summary(lm(breaks~wool, data=x))$coef)

21 Want to run models by index

22 lapply aggregate apply mapply tapply by with

23

24 Now faster with parallel computing

25 reshape Wide data

26 reshape Long data

27 reshape Two basic functions: melt (wide to long) cast (long to aggregate) Makes splitting data by variable much easier

28 reshape id age sex Dx1 Dx2 Dx3 Dx M F

29 melt(data, id.vars=c( id, age, sex )) # melt(data, id.vars = 1:3) id age sex variable value 1 24 M Dx M Dx M Dx M Dx F Dx F Dx F Dx F Dx4 NA

30 reshape This splitting then allows aggregating using cast cast(melted, variable~sex, length) Dx1 Dx2 Dx3 Dx4 M F

31 reshape This splitting then allows aggregating using cast cast(melted,variable~sex, function(x)sum(is.na(x)) Dx1 Dx2 Dx3 Dx4 M F

32 ggplot2 ggplot2 plays really well with and reshape With you can create strata-specific plot objects With reshape you can easily create grouping variables ldply(dat,index, function(x){ p <- qplot(x,y, data=x) return(p) }) melted <- melt(data, id.vars=1:3) ggplot(melted, aes(x,y,groups=variable)+geom_point()

33 Example data(baseball) demo = baseball[baseball$year==2007, c(1,3,7:9)] id team ab r h francju01 ATL francju01 NYN zaungr01 TOR witasja01 TBA williwo02 HOU wickmbo01 ARI 0 0 0

34 Example data(baseball) demo = baseball[baseball$year==2007, c(1,3,7:9)] demo2 = melt(demo, id=1:2) id team variable value 1 francju01 ATL ab 40 2 francju01 NYN ab 50 3 zaungr01 TOR ab witasja01 TBA ab 0 5 williwo02 HOU ab 59 6 wickmbo01 ARI ab 0

35 Example data(baseball) demo = baseball[baseball$year==2007, c(1,3,7:9)] demo2 = melt(demo, id=1:2) demo3 = cast(demo2, team~variable, mean) team ab r h 1 ARI ATL BAL BOS CHA CHN

36 Example data(baseball) demo = baseball[baseball$year==2007, c(1,3,7:9)] demo2 = melt(demo, id=1:2) demo3 = cast(demo2, team~variable, mean) demo4 = melt(demo3, id=team) demo4$team = as.numeric(as.factor(demo4$team) qplot(team, value, color=variable, data=demo4, geom= line ) team value variable ab 1 18 ab ab ab ab ab ab ab ab ab ab ab