TrelliscopeJS. Ryan Hafen. Modern Approaches to Data Exploration with Trellis Display

Transcription

1 TrelliscopeJS Modern Approaches to Data Exploration with Trellis Display Ryan Hafen Hafen Consulting, LLC Purdue

2 All examples in this talk are reproducible after installing and loading the following packages: install.packages(c("tidyverse", "gapminder", "rbokeh", "visnetwork", "plotly")) devtools::install_github("hafen/trelliscopejs") library(tidyverse) library(gapminder) library(rbokeh) library(visnetwork) library(trelliscopejs)

3 TrelliscopeJS is an htmlwidget TrelliscopeJS is a layout engine for collections of htmlwidgets TrelliscopeJS is a framework for creating interactive displays of small multiples

4 Small Multiples A series of similar plots, usually each based on a different slice of data, arranged in a grid This idea was formalized and popularized in S/S-PLUS and subsequently R with the trellis and lattice packages "For a wide range of problems in data presentation, small multiples are the best design solution." Edward Tufte (Envisioning Information)

5 source: Advantages of Small Multiple Displays Avoid overplotting Work with big or high dimensional data It is often critical to the discovery of a new insight to be able to see multiple things at once Our brains are good at perceiving simple visual features like color or shape or size and they do it amazingly fast without any conscious effort We can tell immediately when a part of an image is different from the rest, without really having to focus on it In my experience, small multiples are much more effective than more flashy things like animation, linked brushing, custom interactive vis, etc.

6 source: Trelliscope: Interactive Small Multiple Display Small multiple displays are useful when visualizing data in detail But the number of panels in a display can be potentially very large, too large to view all at once It can also be difficult to specify a meaningful order in which panels are displayed Trelliscope is a general solution that allows small multiple displays to come alive by providing the ability to interactively sort and filter the panels based on summary statistics, cognostics, automatically computed for each panel

7 TrelliscopeJS JavaScript Library trelliscopejs-lib R Package trelliscopejs Built using React Pure JavaScript Interface agnostic htmlwidget interface to trelliscopejs-lib Evolved from CRAN "trelliscope" package (part of DeltaRho project)

8 Gapminder Example glimpse(gapminder) bservations:, ariables: country fctr fghanistan, fghanistan, fghanistan, fghanistan, fgh... continent fctr sia, sia, sia, sia, sia, sia, sia, sia, sia, s... year int,,,,,,,,,... lifep dbl.,.,.,.,.,.,.,... pop int,,,,,,... gdp ercap dbl.,.,.,.,.,.... Suppose we want to understand mortality over time for each country

9 plot(year, lifep, data gapminder, color country, geom "line") Yikes! There are a lot of countries...

10 plot(year, lifep, data gapminder, color continent, group country, geom "line") I can't see what's going on...

11 plot(year, lifep, data gapminder, color continent, group country, geom "line") facet_wrap( continent, nrow ) That helped a little...

12 p plot(year, lifep, data gapminder, color continent, group country, geom "line") facet_wrap( continent, nrow ) plotly::ggplotly(p) This helps but there is still too much overplotting... (and hovering for additional info is too much work and we can only see more info one at a time)

13 plot(year, lifep, data gapminder) lim(, ) ylim(, ) theme_bw() facet_wrap( country continent)

14 From ggplot2 Faceting to Trelliscope Turning a ggplot2 faceted display into a Trelliscope display is as easy as changing: facet_wrap() or: facet_grid() to: facet_trelliscope()

15 plot(year, lifep, data gapminder) lim(, ) ylim(, ) theme_bw() facet_trelliscope( country continent, nrow = 2, ncol = 7, width = 300) open in new window

16 plot(year, lifep, data gapminder) lim(, ) ylim(, ) theme_bw() facet_trelliscope( country continent, nrow, ncol, width, s plotl = ) open in new window

17 Plotting in the Tidyverse

18 Gapminder Example from "R for Data Science" country_model function(df) lm(lifep year, data df) by_country gapminder group_by(country, continent) nest() mutate( model map(data, country_model), resid_mad map_dbl(model, function( ) mad(resid( )))) One row per group Per-group data and models as "list-columns" by_country tibble: country continent data model resid_mad fctr fctr list list dbl fghanistan sia tibble : lm. lbania urope tibble : lm. lgeria frica tibble : lm. ngola frica tibble : lm. rgentina mericas tibble : lm. ustralia ceania tibble : lm. ustria urope tibble : lm. ahrain sia tibble : lm. angladesh sia tibble : lm. elgium urope tibble : lm.... with more rows Example adapted from "R for Data Science"

19 Plotting the Fit for Each Country Excerpt from "R for Data Science"

20 Plotting the Data and Model Fit for a Group We'll use the rbokeh package to make a plot function and apply it to the first row of our data country_plot function(data, model) figure( lim c(, ), ylim c(, ), tools N ) ly_points(year, lifep, data data, hover data) ly_abline(model) country_plot(by_country data, by_country model )

21 Let's Apply This Function to Every Row! by_country by_country mutate(plot p2 plot(data, model, country_plot)) by_country tibble: country continent data model resid_mad plot fctr fctr list list dbl list fghanistan sia tibble : lm. : rbokeh lbania urope tibble : lm. : rbokeh lgeria frica tibble : lm. : rbokeh ngola frica tibble : lm. : rbokeh rgentina mericas tibble : lm. : rbokeh ustralia ceania tibble : lm. : rbokeh ustria urope tibble : lm. : rbokeh ahrain sia tibble : lm. : rbokeh angladesh sia tibble : lm. : rbokeh elgium urope tibble : lm. : rbokeh... with more rows Plots as list-columns!!!

22 by_country trelliscope(name "by_country_lm", nrow, ncol ) open in new window

23 Recap: TrelliscopeJS in the Tidyverse Create a data frame with one row per group, typically using Tidyverse group_by() and nest() operations Add a column of plots TrelliscopeJS provides purrr map functions map_plot(), map2_plot(), pmap_plot() that you can use to create these You can use any graphics system to create the plot objects (ggplot2, htmlwidgets, lattice) Optionally add more columns to the data frame that will be used as cognostics - metrics with which you can interact with the panels All atomic columns will be automatically used as cognostics Map functions map_cog(), map2_cog(), pmap_cog() can be used for convenience to create columns of cognostics Simply pass the data frame in to trelliscope() With plots as columns, TrelliscopeJS provides nearly effortless detailed, flexible, interactive visualization in the Tidyverse

24 open in new window by_country Order the data frame to set arrange( resid_mad) initial ordering of display trelliscope(name "by_country_lm", nrow, ncol )

25 open in new window by_country Filter the data to only include filter(continent " frica") plots you want in the display trelliscope(name "by_country_africa_lm", nrow, ncol )

26 Images as Panels

27 pokemon read_csv(" mutate_at(vars(matches("_id ")), as.character) mutate(panel img_panel(url_image)) pokemon Show 10 entries Search: pokemon id species_id height weight base_experience type_1 type_2 attack 1 bulbasaur grass poison 49 2 ivysaur grass poison 62 3 venusaur grass poison grass poison charmander fire 52 6 charmeleon fire 64 7 charizard fire flying venusaurmega charizardmega-x charizardmega-y fire dragon fire flying squirtle water 48 Showing 1 to 10 of 801 entries Previous N

28 trelliscope(pokemon, name "pokemon", nrow, ncol, state list(labels c("pokemon", "pokede "))) data source blog post open in new window

29 htmlwidgets as Panels

30 Example: Network Vis with visnetwork htmlwidget 1 library(visnetwork) nnodes nnedges nodes data.frame( id :nnodes, label :nnodes, value rep(, nnodes)) edges data.frame( from sample( :nnodes, nnedges, replace ), to sample( :nnodes, nnedges, replace )) group_by(from, to) summarise(value n()) network_plot function(id, hide_select ) style ifelse(hide_select, "visibility: hidden position: absolute", "") visnetwork(nodes, edges) vis graph ayout(layout "layout_in_circle") visnodes(fi ed, scaling list(min, ma, label list(min, ma, draw hreshold, maisible ))) vis dges(scaling list(min, ma )) vis ptions(highlightnearest list(enabled, degree, hide olor "rgba(,,,. )"), nodes d election list(selected as.character(id), style style)) network_plot(, hide_select )

31 Trelliscope display with one panel per node We create a one-row-per-node data frame with number of nodes connected to and total number of connections as cognostics and add a plot panel column nodedat edges group_by(from) summarise(n_nodes n(), tot_conns sum(value)) rename(id from) arrange( n_nodes) mutate(panel map_plot(id, network_plot)) nodedat tibble: id n_nodes tot_conns panel int int int list : visnetwork : visnetwork : visnetwork : visnetwork : visnetwork : visnetwork : visnetwork : visnetwork : visnetwork : visnetwork... with more rows

32 nodedat arrange( n_nodes) trelliscope(name "connections", nrow, ncol ) open in new window

33 Larger Trelliscope Displays

34

35 instadf %>% arrange(-likes_count) %>% trelliscope(name = "posts", width = 320, height = 320, nrow = 3, ncol = 6, state = list(labels = c("caption", "post_link", "likes_count"))) blog post open in new window

36 Trelliscope Displays as Apps

37 Trelliscope Displays as Apps Trelliscope displays are most useful as exploratory plots to guide the data scientist (because they can be created rapidly) However, in many cases Trelliscope displays can be used as interactive applications for end-users, domain experts, etc. with the bonus that they are much easier to create than a custom app If you have an app that has multiple inputs and produces a plot output, the idea is simply to enumerate all possible inputs as rows of a data frame and add the plot that corresponds to these parameters as column and plot it

38 library(shiny) library(ggplot2) library(gapminder) server <- function(input, output) { output$countryplot <- renderplot({ qplot(year, lifeexp, data = subset(gapminder, country == input$country)) + xlim(1948, 2011) + ylim(10, 95) + theme_bw() }) } Gampinder Life Expectancy Selectcountry: Afghanistan choices <- sort(unique(gapminder$country)) ui <- fluidpage( titlepanel("gampinder Life Expectancy"), sidebarlayout( sidebarpanel( selectinput("country", label = "Select country: ", choices = choices, selected = "Afghanistan") ), mainpanel( plotoutput("countryplot", height = "500px") ) ) ) runapp(list(ui = ui, server = server))

39 Scaling Trelliscope We used the previous implementation of Trelliscope to visualize millions of subsets of terabytes of data Just because you can't look at all panels in a display doesn't mean it isn't useful or practical to make a large display - it's in fact beneficial because you get an unprecedented level of detail in your displays, and every corner of your data can be conceptually viewed One insight is all you need for a display to serve a purpose (provided it is quick to create)

40 What is needed to scale in the Tidyverse? SparklyR is the natural solution But we need a few things... SparklyR support for list-columns (nested data frames and arbitrary R objects) SparklyR support for remote procedure calls (run arbitrary R code on the data) Fast random access to rows of a SparklyR data frame A TrelliscopeJS deferred panel rendering scheme (render on-the-fly rather than all panels up front)

41 What's Next trelliscopejs Automatic cognostics: automatically compute useful cognostics based on the context of what is being plotted (e.g. if a scatterplot has a model fit superposed, add model diagnostics cognostics Automatic handling of axis limits - "same", "sliced", "free" (underway - currently "same" limits need to be hard-coded) When axes are "same", only show axes on plot margins instead of every panel (underway for ggplot2) trelliscopejs-lib More visual filters for cognostics (dates, geographic, bivariate relationships, etc.) Bookmarkable / sharable state View multiple panels side-by-side Support for receiving panels from other endpoints

42 For More Information Blog: Documentation: Github: