Ranking Visualizations of Correlation Using Weber s Law

Transcription

1 Ranking Visualizations of Coelation Using Webe s Law Lane Haison, Fumeng Yang, Steven Fanconei, Remco Chang Abstact Despite yeas of eseach yielding systems and guidelines to aid visualization design, pactitiones still face the challenge of identifying the best visualization fo a given dataset and task. One pomising appoach to cicumvent this poblem is to leveage peceptual laws to quantitatively evaluate the effectiveness of a visualization design. Following peviously established methodologies, we conduct a lage scale (n=1687) cowdsouced expeiment to investigate whethe the peception of coelation in nine commonly used visualizations can be modeled using Webe s law. The esults of this expeiment contibute to ou undestanding of infomation visualization by establishing that: 1) fo all tested visualizations, the pecision of coelation judgment could be modeled by Webe s law, 2) coelation judgment pecision showed stiking vaiation between negatively and positively coelated data, and 3) Webe models povide a concise means to quantify, compae, and ank the peceptual pecision affoded by a visualization. Index Tems Peception, Visualization, Evaluation. 1 INTRODUCTION The theoy and design of infomation visualization has come a long way since Betin s seminal wok on the Semiology of Gaphics [1]. Yeas of visualization eseach has led to systems [17, 18] and guidelines [5, 24] that aid the designe in choosing visual epesentations based on geneal data chaacteistics such as dimensionality and datatype. Unfotunately, many aspects of visualization design still emain moe at than science. Fo example, given a set of a data chaacteistics, thee ae almost always multiple visualizations (foms) that ae theoetically valid and theefoe difficult to choose fom [18]. In addition, beyond selecting a visualization fom, the designe must also take into account many othe aspects of the visualization. Examples include design elements such as colo, shape, glyph size, as well as usage consideations such as context, and use pofile. With so many options, it is temendously difficult fo even expeienced designes to identify the most accuate and appopiate visualization given a dataset. One method fo objectively identifying the best visualization is to conduct multi-facto human-subject expeiments. In these expeiments, each design o usage consideation is incopoated as an expeimental facto, often esulting in a lage numbe of conditions. While these expeiments poduce actionable esults, they ae difficult to genealize beyond the scope of the expeiment and povide limited explanation as to why one visualization is bette than anothe. As visualization becomes moe widely adopted and divesified, it is clea that exhaustive compaative expeimentation cannot fully addess the gowing needs of the infovis community. What is needed then ae quantitative and obust models fo visualizations that ae genealizable beyond one-off compaative studies while still poviding designes with actionable infomation about tadeoffs between valid visualization foms. Although such models challenge conventional wisdom in infomation visualization design [13], ecent eseach has suggested that the use of peceptual laws fom psychology and cognitive science [12, 3] can be applied to model how humans peceive cetain data popeties given a visualization. In paticula, Rensink et al. successfully demonstated that the peception of positive coelation in scatteplots can be modeled using Webe s law [22], which indicates that the human peception of diffeences in coelation and the objective diffeences in data coelation has a linea elationship: Lane Haison, Fumeng Yang, and Remco Chang ae with Tufts Univesity. [lane,fyang,emco]@cs.tufts.edu. Steven Fanconei is with Nothwesten Univesity. fanconei@nothwesten.edu. Manuscipt eceived 31 Mach 2014; accepted 1 August 2014; posted online 13 Octobe 2014; mailed on 4 Octobe Fo infomation on obtaining epints of this aticle, please send to: tvcg@compute.og. d p = k ds S whee d p is the diffeential change in peception, ds is the diffeential incease in the data coelation (change in stimulus), and S is the oveall coelation in the data (stimulus). k is known as the Webe faction, and is deived expeimentally. Taken togethe, this equation and expeimentally-infeed paamete k fom a Webe model fo the peception of positive coelation in scatteplots. What is significant about this finding by Rensink et al. is that it descibes the peception of coelation in a concise, quantitative manne via the deived Webe model. The authos hypothesize that if othe visualizations of coelation could be shown to follow Webe s law, then it might be possible to compae them without exhaustive empiical testing [22]. Anothe significant benefit of establishing peceptual models fo visualization is that it povides a pedictive and falsifiable baseline to investigate the effect of design elements within a visual fom. Fo example, in followup studies Rensink et al. used the oiginal Webe model to study whethe the effectiveness of scatteplots was impacted by changes in design elements such as point colo, bightness, size, canvas aspect atio, and othes [21]. With the Webe model fo scatteplots as a baseline, the authos demonstated that it was possible to detemine when the peception of positive coelation in scatteplots was invaiant to design elements, and did so without esoting exhaustive multi-facto testing. Theefoe, if the peception of coelation in commonly used visualizations can be shown to follow Webe s law, we gain the ability to quantitatively compae and ank the effectiveness of visualizations, as well as a baseline to exploe the effect of design elements on the basis of peceptual laws. In this pape, we confim the hypothesis of Rensink et al. by demonstating that nine commonly used visualizations also follow Webe s law, and that the peception of coelation acoss multiple valid visual foms can be quantified, compaed, and anked using the deived Webe models. Afte adapting the expeimental methodology used by Rensink et al. fo a cowdsoucing envionment, we fist validate ou appoach by eplicating thei oiginal findings with scatteplots on Amazon s Mechanical Tuk. We then apply this methodology to eight othe visualizations commonly used in the infovis community and commecial softwae, including paallel coodinates plots, stacked aea chats, stacked ba chats, stacked line chats, line chats, odeed line chats, ada chats, and donut chats (see Figue 3). The esults of this expeiment contibute to ou undestanding of infomation visualization in seveal ways: We demonstate that the peception of coelation in seveal commonly used visualizations can be modeled using Webe s law. We povide evidence that the effectiveness of most visualizations tested depends significantly on whethe it depicts positively o (1)

2 negatively coelated data (asymmetic pefomance), implying that many visualizations may equie two Webe models to be descibed completely. Using the deived Webe models, we ank the effectiveness of visualizations fo epesenting coelation. In the following section, we discuss elated wok including peceptual studies fo visualization and ecent advances in evaluation methodologies. We then descibe ou fist expeiment, whee we adapt the methodology of Rensink et al. fo a cowdsouced envionment and eplicate thei oiginal findings. Following the eplication, we pesent ou full expeiment evaluating eight othe visualizations, including the esulting analyses and models. Ou discussion highlights the implications of ou findings, such as how ou esults make it possible to quantify, compae, and ank the effectiveness of visualizations fo depicting coelation on the basis of a peceptual law. We also point out seveal supising diffeences and similaities in the visualizations tested, offeing possible explanations based on ecent wok in vision science. 2 RELATED WORK A key component of modeling a peceptual pocess using Webe s law is the need to expeimentally detemine how much a given stimulus must incease/decease befoe humans can eliably detect changes, a quantity called the just-noticeable diffeence () [6]. Although they ae a key pat of the methodology we use in this pape, the notion of using s to advance visualization design is not entiely new. Colo models that appoach peceptual-unifomity, such as the CIELAB space, ae the esult of yeas of expeiments that examine peceptual distances between colos [23]. In these colo spaces, two colos that have diffeent RGB values but ae peceived as being the same ae said to be within one of each othe. Peceptually-diven colo models have led to impotant advances in infovis, fo example the popula ColoBewe tools [9], and moe ecently the development of algoithms that automatically select effective colo schemes fo visualization [15]. Motivated by the success of these appoaches, which quantify the peceptual space of colo, ou wok similaly seeks to exploe and quantify the peceptual space of visualization foms. Many peceptual studies have examined the peception of coelation in scatteplots. Ealy wok fom Cleveland et al. suggested that subjective estimations of coelation could be biased by inceasing the scale of the axes in a scatteplot [4]. Combining peceptual studies of scatteplots and visualization design, Fink et al. integated paticipant pefeences to develop an automatic method fo selecting effective aspect-atios fo scatteplots [8]. Li and van Wijk examined diffeences in the subjective judgments of coelation in scatteplots and paallel coodinates, finding scatteplots to pefom bette [14]. The key diffeence between these appoaches and ous is that they do not explicitly link thei esults to undelying peceptual laws, limiting the genealizability of thei esults. One of the pimay goals of this wok is to povide a means to evaluate and compae visualization effectiveness on the basis of peceptual laws. It is useful, theefoe, to situate ou contibutions in the context of existing appoaches to visualization evaluation. To bette bidge the gap between design goals and evaluation methodologies, Munzne s Nested Model aanges the design and evaluation space into fou inteelated levels [20]. Moe ecently, Lam et al. conducted an extensive suvey of infovis publications, distilling them into seven evaluation scenaios [13]. Subsuming both qualitative and quantitative appoaches, these evaluation models allows eseaches and pactitiones to bette identify the appopiate level(s) at which a given visualization should be evaluated. Capendale has pointed out, howeve, that an inheent limitation of many evaluation appoaches is that they ae difficult to genealize to diffeent usage contexts [2]. A ecent poposal fom Demialp et al. seeks to addess these limitations by developing visualization geneation and evaluation methods that map peceptual distances between visual elements to simila stuctues in data [7]. Ou wok contibutes to these evaluation eseach diections in two ways. (a) (b) Fig. 1: a) A sample stating compaison fom the expeiment: = 0.7 on the left and = 0.6 on the ight. Paticipants wee asked to choose which of the two appeaed to be moe highly coelated. b) The staicase pocedue hones in on the just-noticeable diffeence by gadually making compaisons moe difficult: = 0.7 on the left and = 0.65 on the ight. Fist, we demonstate that visualization effectiveness can be quantified and evaluated using peceptual laws. Additionally, we povide Webe models descibing the peception of coelation in seveal commonly used visualizations, which can be used as a baseline to investigate the impact of individual design elements. 3 EXPERIMENT 1: REPLICATION AND CROWDSOURCING VAL- IDATION Given ou goal of testing a wide ange of visualizations, we tun to a cowdsoucing platfom to ecuit the necessay numbe of paticipants. Howeve, since ou goal is to leveage expeiment methodologies fom vision science, which have not been peviously validated fo cowdsoucing [16], it is necessay to fist eplicate the oiginal expeiment by Rensink and Baldidge on modeling the peception of positive coelation in scatteplots using Webe s law [22]. In paticula, ou expeiment seeks to confim the pecision potion of Rensink and Baldidge s oiginal expeiment. As the authos note in thei pape, pecision and accuacy ae examined though diffeent expeiment methodologies. Pecision, in thei expeiment, efes to the ability of paticipants to detect diffeences between two coelations, even if they ae blind to the actual numeical coelation values. Accuacy, on the othe hand, coesponds to paticipants bias towads systematically ove- o unde-estimating coelation values (see [22] fo moe on these diffeences). The esults of thei expeiments demonstated, howeve, that pecision and accuacy fo the peception of coelation in scatteplots ae systematically linked via Webe s law. Given this esult, we estict the scope of ou expeiment to investigate whethe in-lab esults fo infeing pecision can be eplicated using cowdsoucing. 3.1 Mateials Following the expeimental design of Rensink and Baldidge [22], scatteplots in this expeiment wee all pixels, contained 100 nomally distibuted points along the 45 degee line, used the same point size of 2 pixels, and displayed both the left and bottom axes. To geneate coelated data fo a taget coelation value, n = 100 data points wee fist taken a standad nomal distibution within 2

3 fom above fom below A (d) egession compaison plot A Fig. 2: Expeiment 1 eplicated the esults fom [22] on Amazon s Mechanical Tuk, validating the platfom fo ou lage expeiment. (a) shows the aw data fom pevious studies [22]. (b) shows the aw data in Expeiment 1. (c) shows the data afte adjustment, whee s ae dawn as functions of adjusted coelation A. (d) compaes pevious egession esults [22] and the egession fom Expeiment 1. standad deviations of the mean and nomalized. The coelation coefficient of this stating dataset is then computed and noted as z. Then, each point (x i,y i ) is tansfomed using the same tansfomation in [22]: y i = λx i + (1 λ)y i (2) λ 2 + (1 λ) 2 whee λ is defined as follows: λ = ( z 1)( 2 + z ) + 2 (z 2 1)( 2 1) ( z 1)(2 2 + z 1) Note that ou equation fo λ diffes fom that of Rensink and Baldidge [22]. Specifically, athe than using an estimation appoach fo computing λ, we instead incopoate the coelation value of the stating dataset z. Ou extension of this method uses the same tansfomation and paametes but a) conveges moe quickly, and b) eliminates eo in the oiginal appoach (taget ±0.005). Afte the dataset is geneated, it is e-nomalized and tansfomed to have a mean of 0.5 and standad deviation of 0.2 (following [22]). 3.2 Methodology Following Rensink and Baldidge [22], we use the same adaptive psychophysical method, a staicase pocedue, to infe just-noticeable diffeences (s) fo the peception of coelation. This expeimental pocedue has a 6 2 design in that thee ae six coelation values (0.3, 0.4, 0.5, 0.6, 0.7 and 0.8) and two appoach conditions (above and below). In the staicase pocedue, given a taget value fo coelation,, paticipants ae given two visualization stimuli side-by-side (two scatteplots in this case) and asked to choose which they peceive to have a highe coelation (see Figue 1). With an above appoach, the paticipant is given one visualization with the taget, and anothe with an value highe than the taget. Fo example, if the taget is 0.7, then the second value would be 0.8 (assuming a stating distance of 0.1). Convesely, with an below appoach, the paticipant would be given a visualization with the taget, and anothe that has an value lowe than the taget. In both cases, if a paticipant chooses coectly, the distance in coelation between the two visualizations is deceased by 0.01 while keeping the taget constant (e.g. 0.7 vesus 0.79 in the above condition, o 0.7 vesus 0.61 in the below condition). If a paticipant chooses incoectly, the distance in coelation between two visualizations is inceased by 0.03, making the next judgment easie. The staicase pocedue hones in on the by penalizing incoect choices moe than coect choices. These distance changes (0.01, 0.03) coespond to infeing 75% s, o the minimum diffeence in coelation equied to be eliably disciminated 75% of the time. Afte (3) each selection is made, the position of the taget and vaiable visualization is andomized (i.e., whethe the taget appeas as the left vs. ight visualization), and new datasets ae geneated fo both stimuli. The staicase pocedue ends when eithe 50 individual judgments ae eached o when a convegence citeia is met. Following [22], the convegence condition is detemined based on the last 24 use judgments, and is designed to detemine if the use s ability to disciminate between the two coelation values depicted has stabilized. Specifically, the 24 use judgments ae divided into 3 subgoups, and convegence is eached when thee is no significant diffeence between these thee subgoups via an F-test (F(2, 21); α = 0.1). Finally, afte the staicase pocedue ends, the aveage distance in coelation value in these subgoups is used as the fo the tested values and appoach (above/below). Thee ae two impotant limitations to the staicase pocedue that we include when epoting esults: ceiling effects and the chance bounday. A ceiling effect occus when the paticipant fails to peceive coelation eliably and the adaptive algoithm eaches an uppe limit fo coelation ( = 1.0). Fo example, if the base value is 0.7 and the value of the second stimuli eaches 1.0, yet the paticipant still answes andomly fo the emainde of the judgments (oughly 50% accuacy), the esulting fo = 0.7 will be 0.3. This uppe limit (0.1 fo 0.9, 0.2 fo 0.8, etc.) will be illustated in ou esults. The chance bounday is defined by the paametes of the staicase pocedue (convegence citeia, stating distance, numbe tials, etc.) To obtain this bounday, we an a simulation of the staicase pocedue 10,000 times, simulating a paticipant guessing at chance (50% accuacy). The esulting bounday in ou pocedue was = 0.45, meaning that any esulting s at o above this bounday would indicate that the paticipant did not eliably peceive coelation. Thee ae two possible cases whee a paticipant pefoms at chance thoughout an expeiment. The fist is when a paticipant is simply making andom choices, which is possible given that we ae using a cowdsoucing platfom [11, 19]. The second possibility occus when the actual s fo a given stimuli ae at o nea 0.45, focing the paticipant to guess thoughout most of the judgments. This chance bounday ( = 0.45) is illustated all of ou esults figues, and will be used to establish an exclusion citeia fo pooly pefoming visualizations. 3.3 Pocedue The conditions in this expeiment include the six coelation value tested, and whethe the taget coelations wee appoached fom above o below. Infomed by ealy pilot testing, paticipants wee andomly assigned to two coelation values: one fom [0.3, 0.4, 0.5], and one fom [0.6, 0.7, 0.8]. These goups oughly coespond to had and easy, since high coelations ae moe easily disciminated than low coelations. Fo the two coelation values chosen, paticipants

4 = - 1 = = = 0.3 = 0.8 = 1 scatteplot paallel coodinates (pcp) stackedaea stackedline stackedba donut ada line odeed line Fig. 3: The nine visualizations tested in ou expeiment, at seveal coelation values. Because many of these visualizations appea diffeently when depicting negatively vesus positively coelated data, we test both in ou expeiment. The visualizations wee lage ( pixels) when pesented to paticipants. The colo scheme used is coloblind-safe, chosen fom ColoBewe.

5 complete both the above and below appoach, esulting in a total of fou tials (easy above, easy below, had above and had below). This coesponds to collecting up to 200 individual judgments fo each paticipant. Since paticipants on Amazon s Mechanical Tuk (AMT) come fom divese educational/statistical backgounds [19], both a taining and pactice session wee added befoe the main tials began. The taining session consisted of a shot definition of coelation, including a gid of ten scatteplots showing coelations anging fom 0.1 to 1.0. Afte the taining session, paticipants wee given a pactice session consisting of 30 individual judgments. In the fist 15 judgments, paticipants wee shown the easy (high) coelation they would be woking with; in the second 15 they wee shown the had (low) coelation condition. Afte each judgment, paticipants wee given feedback on whethe they chose coectly. Afte completing the taining and pactice sessions, paticipants began the fou main tials. The ode of the coelation-appoach pais was andomized in this session. Upon completing a tial set (eithe by eaching the convegence citeia o 50 individual judgments), paticipants wee given the option to take a shot beak, and notified as to how many expeiment tials emained. Following the completion of all fou tials, a demogaphics questionnaie was given, which included a question that asked paticipants to descibe the stategy they used to assess coelation. Finally, paticipants wee given a shot debief explaining the pupose of the expeiment. 3.4 Results We ecuited n = 88 paticipants (36 female) fo this expeiment via Amazon s Mechanical Tuk (AMT). It took appoximately two days to gathe all esponses. Paticipants wee paid $2.10 fo thei time, commensuate with the UṠ minimum wage. To avoid possible confounds fom paticipants using mobile devices o tablets, such devices wee pogammatically blocked fom accessing the expeiment. This expeiment adheed to a between-subjects design, since paticipants wee andomly assigned to complete two coelation values (out of six) fo both above and below appoaches. While thee wee n = 20 paticipants fo each coelation-appoach in [22], in ou expeiment we ecuited appoximately n = 30 fo each pai in ode to account fo the inheent vaiability in AMT woke esponses [11, 19]. Ou esults indicate that cowdsoucing effectively eplicates measuements obtained in-lab. Individual and eo data wee not published in [22], eliminating the possibility fo diect statistical compaison. Howeve, we estimated s and eos fom figues, and compae them with ou esults in Figue 2. The geneal tends between esults obtained in-lab and via cowdsoucing ae simila, including both the highe eo-bas fo lowe coelation values, and small s fo highe coelations. To detemine if ou esults also can be modeled using Webe s law, we follow the model-fitting pocedue in [22]. Specifically, each coelation value was moved by half of the aveage fom the above and below appoach. Fo the above appoach, the coelation was moved towads = 1, while the fom the below appoach was moved towads = 0. Specifically, coelation is tansfomed into adjustedcoelation A by: A = ± 0.5 () (4) Figue 2 illustates this intemediate step, showing all points in ou data afte adjustment. Linea models wee then fit to the data, and eos wee computed based on the squae oot of the mean squaes of the esiduals (RMS eo). Following this pocedue, we find the model to be a good fit ( 2 = 0.98), indicating that ou cowdsoucing esults also follow Webe s law. The esulting egession lines ae shown in Figue 2. Although ou esults have a slightly lowe intecept (0.21 cf. 0.25), indicating bette pefomance oveall, the slopes ae also simila ( 0.17 cf. 0.22). Given the similaity in both the and egession esults, we find this to be evidence that the AMT cowdsoucing platfom is appopiate fo ou expeiments. 4 EXPERIMENT 2: EXTENSION TO OTHER VISUALIZATIONS Expeiment 1 established that esults obtained via cowdsoucing eplicate Rensink and Baldidge s oiginal expeiment that modeled the peception of positive coelation in scatteplots using Webe s law. In Expeiment 2, we extend this expeiment to investigate whethe the peception of coelation in othe commonly used visualizations can also be modeled using Webe s law. 4.1 Mateials We chose nine visualizations fo this expeiment based on two main citeia: a) they must be commonly used in eithe infovis o commecial softwae (extenal validity), and b) they must be viable within the constaints of the expeiment methodology. The nine visualizations chosen include: scatteplots, paallel coodinates plots, stacked aea chats, stacked ba chats, stacked line chats, line chats, odeed line chats, ada chats, and donut chats. Scatteplots wee included both because of thei widespead use in the scientific community, and to seve as a baseline fo eplicating the esults of Rensink and Baldidge [22]. Paallel coodinates plots wee included because of thei continued widespead use in the infovis community. Line chats, stacked line chats, stacked aea chats, and stacked ba chats wee included based on the top ecommendations when viewing one of ou datasets (100 points, 2 dimensions) in Micosoft Excel. Despite thei similaities, all of the stacked chats (line, aea, ba) wee included since any significant diffeences in these chats might shed light on the undelying peceptual pocesses people use when judging coelation. Donut chats and ada chats wee included since they ae essentially adial tansfoms of stacked ba chats and line chats, espectively. Compaing these may allow us to undestand the effect of coodinate tansfoms on the peception of coelation. Note that, of all the visualizations tested, scatteplots and paallel coodinates plots ae the only two that ae tuly bivaiate, in that the two quantitative vaiables in the data displayed (X, Y) detemine the exact position of the gaphical elements. In contast, all of the othe visualizations contain an additional explicit vaiable ode. In othe wods, scatteplots and paallel coodinates plots ae odeed by default, while any of the othe visualizations can be odeed in diffeent ways. To test the effect that manipulating ode may have on the peception of coelation, odeed line chats (soted on the X-axis) wee also added to the expeiment. Finally, we hypothesize that the pefomance of these visualizations may be impacted by the fact that many of them appea vey diffeently when depicting positively vesus negatively coelated data (see Figue 3). To test this hypothesis, we test each of these nine visualizations wee twice: once with positively coelated data, and once with negatively coelated data. As in expeiment 1, all visualizations wee pixels, contained 100 data points and displayed datasets geneated fom same algoithm. Fo visualizations that equied moe than one colo, we chose a single colo scheme fom ColoBewe [9] 1. All visualizations used in this expeiment ae illustated fo seveal coelation values in Figue Pocedue The pocedue fo this expeiment follows that of Expeiment 1, except fo the following two changes. To mitigate possible confounds fom exposing some paticipants to negative (vesus positive) coelations in the taining session, we label the coelation gid (0.1 to 1.0) with the same labels egadless of the coelation diection, and povide a shot disclaime explaining the change fo paticipants familia with coelation. Secondly, since ealy pilot testing showed that many of the visualizations had highe s than scatteplots fo low coelations, we allow the staicase pocedue to move above o below = 0.0 if necessay. 1

6 scatteplot positive scatteplot negative paallel coodinates positive paallel coodinates negative stackedaea positive stackedaea negative stackedline positive stackedline negative stackedba positive stackedba negative donut positive donut negative line positive line negative ada positive ada negative odeed line positive odeed line negative Fig. 4: plotted as a function of coelation fo both above (light points) and below (dak points) appoaches. Eo bas show the standadeo of the mean (SEM). Boken lines show the chance and ceiling boundaies defined in Section 3.2. The x-axis is coelation value, the y-axis is. (a) scatteplots and paallel coodinates paallel coodinates scatteplot positive negative (b) stackedaea, stackedline and stackedba stackedaea negative (c) stackedba and donut odeed A A A A A (d) line and ada (e) odeed line and line Fig. 5: Regession esults fo seveal paied compaisons. s ae modeled as functions of adjusted coelation A. The x-axis is adjusted coelation value A, the y-axis is.

7 Table 1: Mann-Whitney-Wilcoxon Tests Results fo condition pais. Significant values denoted by * with α = visualization - diection 1 visualization - diection 2 W p-value scatteplot - negative scatteplot - positive scatteplot - negative paallel coodinates - positive < scatteplot - positive paallel coodinates - positive 8623 < paallel coodinates - negative scatteplot - negative paallel coodinates - negative scatteplot - positive paallel coodinates - negative paallel coodinates - positive < stacked ba - negative stacked line - negative < stacked ba - negative stacked aea - negative < stacked ba - negative donut - negative stacked line - negative stacked aea - negative line - positive ada - positive line - positive odeed line - positive < line - positive odeed line - negative < odeed line - negative odeed line - positive Table 2: Intecepts, Slopes, Coelation Coefficients, 2, and RMS fo s modeled as functions of adjusted coelation A. visualization - diection intecept-b slope-k coelation- 2 RMS scatteplot - positive scatteplot - negative paallel coodinates - positive paallel coodinates - negative stacked line - negative stacked aea - negative stacked ba - negative donut - negative line - positive ada - positive odeed line - positive odeed line - negative Results We ecuited 1,687 paticipants though AMT (834 female) fo this expeiment. It took appoximately two weeks to gathe all esponses. Ou study used a total of nine visualizations, two coelation diections (positive/negative), and six coelation values (0.3 to 0.8) yielding 54 main goups. Since each paticipant was assigned to one visualization, one coelation diection, and two coelation values (above and below), oughly 30 paticipants wee assigned to each visualization diection -value goup. The esulting data wee non-nomally distibuted, so to mitigate the effect of outlies, s that fell outside 3 median-absolute deviations fom the median (within one of the 54 goups) wee excluded fom the following analyses. Because the staicase methodology penalizes incoect esponses and contols fo guessing by defining a convegence citeia (see Section 3.2), this exclusion citeia also mitigates the effect of click-though esponses that often impact cowdsouced expeiments [11, 19]. Figue 4 shows aveage s fo all visualizations, coelation values, and appoaches tested afte filteing. An exlusion citeia was also enfoced fo visualization diection pais that exceeded 20% of values falling on o outside the chance bounday of = 0.45 established peviously. Six of the eighteen pais met this exclusion citeia: stacked aea-positive, stacked ba-positive, stacked line-positive, donut-positive, ada-negative, and line-negative (see Section 5.2 fo moe details on the exclusion citeia). The following analyses include the emaining twelve visualization diection pais. 4.4 Webe Model Fit Following the same model fitting pocedue as Expeiment 1 (and [22]), slopes, intecepts, coelation coefficients and oot-mean-squae eos (RMS) wee computed. These statistics ae included in Table 2 and coesponding egession lines illustated in Figue 6. All visualizations follow a linea elationship between s and adjusted coelation A, based on the high coelation coefficient and small RMS fo each. Regession lines fo the following statistical compaisons ae shown in Figue Statistical Analyses Examining the data alone, thee appea to be lage diffeences in pefomance between many of the visualizations, as well as asymmeties between many of the positive/negative pais. In ode to confim these obsevations, an oveall Kuskal-Wallis test was conducted on the aw s to evaluate whethe thee is an inteaction between visualization and coelation diection conditions. This test confimed thee was an oveall effect fo visualization coelation diection (χ 2 (17) = , p < 0.001,α = 0.05). To exploe futhe, seveal visualization diection pais wee compaed via Mann-Whitney-Wilcoxon tests. Rathe than compae all possible pais, we instead investigate 14 paiings eflecting the oiginal motivations fo choosing the visualizations tested (see the Mateials section 4.1). We use Bonfeonni coection to addess the poblem of multiple compaisons, esulting in an α = equied fo ejecting the null hypothesis. All tests esults and paametes ae epoted in Table 1. To aid visual compaisons, we povide egession lines coesponding to these compaisons in Figue 5. Examining the compaison esults fo scatteplots in Table 1, we find no significant diffeence between scatteplots depicting positively and negatively coelated data (p = 0.54, see Figue 5.a). In addition to thei symmetic pefomance, scatteplots appea to be among the best pefoming visualizations (see Figue 6). We find clea evidence of asymmety in paallel coodinates plots, with those depicting negatively coelated data significantly outpefoming those depicting positively coelated data (p < 0.001, see Figue 5.a). Futhemoe, we find that paallel coodinates plots depicting negatively coelated data wee not significantly diffeent fom scatte-

8 = 0.1 * = 0.3 = 0.5 = 0.7 = 0.9 * oveall scatteplot positive scatteplot negative scatteplot negative scatteplot positive scatteplot positive scatteplot positive scatteplot positive scatteplot positive scatteplot negative scatteplot negative scatteplot negative scatteplot negative odeed odeed odeed odeed odeed odeed bette Fig. 7: Using the infeed Webe models, we can poduce a peceptually-diven anking fo individual coelation () values, as well as an oveall anking (ight column). Pefomance is odeed fom the best (top) to the wost (bottom). The columns denoted by * ae pedicted esponses using the fit models shown in Figue 6. scatteplot paallel coodinates scatteplot, Rensink model fit esults stackedline stackedaea stackedba donut odeed line line ada positive negative A Fig. 6: Regession esults fom Expeiment 2 (including esults fom [22]). s ae modeled as functions of adjusted coelation A. plots depicting eithe positively (p = 0.16) and negatively (p = 0.42) coelated data. Between the stacked chat vaiants (stacked ba, stacked aea, stacked line) depicting negatively coelated data, we find that stacked ba chats significantly outpefom both stacked aea and stacked line chats (both p < 0.001, see Figue 5.b), and that thee was no significant diffeence between stacked aea and stacked line chats (p = 0.014). We also find no significant diffeence between the stacked ba chat and the donut chat (p = 0.037, see Figue 5.c). Between the line chat vaiants (line chat, ada, odeed line) depicting positively coelated data, the odeed line chat significantly outpefomed both the line chat (p < 0.001) and the ada chat (p < 0.001), (see Figue 5.d and 5.e). In fact, the odeed line chat is the only othe chat besides scatteplots to show symmetic pefomance (positive and negative diffeence p = ). The ada chat also pefomed significantly bette than the line chat (p = ). 5 DISCUSSION Ou esults demonstate that thee ae significant diffeences in the peception of coelation acoss visualizations, and that these esults often vay significantly when depicting positively vesus negatively coelated data. Between scatteplots and paallel coodinates, we find that using scatteplots to depict coelation esults in bette pefomance oveall. Howeve, this pefomance diffeence only occus when depicting positively coelated data. In fact, paallel coodinates depicting negatively coelated data appea to pefom as well as scatteplots (see Figue 5.a). Among the stacked chat vaiants (stacked ba chats, stacked aea chats, and stacked line chats), the stacked ba significantly outpefomed both the stacked aea and stacked line (see Figue 5.b). This finding suggests that although these visualizations appea to be simila, the undelying peceptual pocesses that paticipants use when judging coelation in them may diffe substantially (fo moe discussion, see Section 5.3). While thee was no diffeence between the stacked ba chat and the donut chat, the ada chat significantly outpefomed the line chat, indicating that coodinate tansfoms may yield inconsistent pefomance implications. Some of these findings can be diectly applied to infom visualization design. Fo example, because paallel coodinates plots depicting negatively coelated data significantly outpefom those depicting positively coelated data, new layout algoithms could be developed to maximize the numbe of negative coelations depicted by flipping and e-aanging axes. Howeve, since ou expeiment esults establish that the peception of coelation in these visualizations can be modeled using Webe s law, it also becomes possible to ank the effectiveness of the tested visualizations. 5.1 Ranking One of the pimay questions this expeiment sought to exploe is whethe the effectiveness of visualizations fo depicting coelation can be quantified, compaed, and anked on the basis of a peceptual

9 law. Since we have infeed Webe models fo each of the nine visualizations tested, this anking becomes possible. Recall that we tested each of the nine visualizations with both positively and negatively coelated data, fo a total of eighteen visualization coelation-diection pais. Howeve, since six of the eighteen pais met ou exclusion citeia, we include the emaining twelve models in ou anking. Using the infeed Webe models, we poduce a anking fo six coelation values (see Figue 7). Each of the visualization coelationdiection pais is odeed by pefomance, with the best in the top ow, and the wost in the bottom ow. Note that using Webe s law allows us to make pedictions fo the peception of coelation values that wee not explicitly tested in the expeiment (e.g. 0.1 and 0.9). The anking ode fo each of coelation value vaies due to cossings in the Webe models (see Figue 6). While anking visualizations within individual coelation values can be useful fo design, an oveall best anking is also desiable. One staightfowad way to obtain an oveall anking is to identify the visualization which has the lowest on aveage. This can be computed by calculating the aea of the egions between the egession lines and the A -axis [10] (see the ightmost column in Figue 7). This anking has many potential applications. One possible diection is to define a toleable ange of effectiveness, in ode to estict the possible design space to fewe visualizations. Fo example, if a designe needs to eliably communicate coelation fo a given dataset, they can efe to this model and obtain a pecise anking based on the coelation values in thei dataset. Anothe possible application is in visualization system design, whee it may be helpful to use eithe the oveall anking, o to obtain a custom anking based on a ange of coelation values (e.g. a scientific application may equie identifying the most effective visualizations fo coelations above 0.6). 5.2 Limitations of the Methodology Although we wee able to demonstate that each of the nine visualizations followed Webe s law, we found that six visualization coelation-diection pais poduced uneliable esults and theefoe wee excluded fom the est of the analyses. Specifically, the six pais excluded wee: stacked aea-positive, stacked ba-positive, stacked line-positive, donut-positive, ada-negative, and line-negative chats. The exclusion citeia was based on the uppe limit ( chance ) bounday defined in Section 3.2. Recall that this bounday ( = 0.45) is a function of the staicase pocedue paametes such as the stating distance (0.1) and coect/incoect penalties (0.01 and 0.03 espectively). Since the s fo the excluded visualization coelation-diection pais fequently met the uppe limit of ou staicase pocedue, it is possible that eithe coelation is not eliably peceived in these visualization coelation-diection pais, o that a lage stating distance (and coesponding penalties) is equied to infe eliable esults fo these visualizations. While each visualization tested was shown to follow Webe s law fo least one coelation-diection (positive and/o negative), we cannot say fo sue whethe the six excluded visualization coelationdiection pais also follow Webe s law. Examining the undelying peceptual pocesses involved in judging coelation fo each of these visualizations, howeve, may allow us to identify the easons fo the obseved pefomance vaiations. 5.3 Visual Featues Reviewing Figue 3, we obseve that many of the visualizations tested vay significantly when depicting positively vesus negatively coelated data. In most cases, the visualizations appea to have diffeent visual foms fo the two coelation diections but the same absolute coelation value ( ). Fo example, in paallel coodinates, when = 1, the visualization depicts a set of paallel lines and has the shape of a squae; wheeas when = 1, the set of lines intesect at a single point and appea as two tiangles. One exception to this asymmetic elationship in visual foms is the scatteplot. Fo both positive and negatively coelated data, the visual fom of a scatteplot conveges to a single line when appoaches 1. In fact, past studies have hypothesized that the visual featue viewes attend to when making coelation judgments in scatteplots is the width of the bounding box (o ellipse) that suounds the points [4, 14]. Fo example, when = 1, the width of this bounding box is essentially 0. While futhe testing is needed to confim exactly what visual featues ae peceived in scatteplots, it has long been established that peceptual judgments of line length follow Webe s law [12]. Based on these obsevations, ou esults suggest that the eason the peception of coelation in scatteplots follows Webe s law is because the undelying visual featues that vay with coelation follow Webe s law. Reasoning about the undelying visual featues of visualizations may also explain the significant pefomance diffeences between the stacked chat vaiations. Ou esults demonstate that, fo negative coelations, stacked ba chats significantly outpefom both stacked aea chats and stacked line chats. This finding is supising since the visual foms of these thee chats ae simila. Howeve, based on paticipant feedback, we see that the visual featues employed when judging coelation in these chats might in fact be diffeent: I looked fo which aveage value of the oange line was highe, also taking into account which oange line had fewe peaks/valleys... It seemed like the less-spiky chats wee moe coelated. In contast, a paticipant in the stacked ba condition noted: I mostly compaed how much white and oange wee compaed to the blue on each chat. Usually the one with less white was moe coelated. While the visual foms fo the stacked chats vaiants ae simila (see Figue 3), the visual featues that convey coelation diffe. Since visual featues (athe than visual foms) may be the undelying cause that significantly impacts the effectiveness of visualizations, we believe thee ae seveal impotant aeas fo futue wok. These include compaing visual featues poduced by visualizations, identifying how peceptual laws apply to othe common visualization tasks, and investigating these findings with eal-wold datasets and datasets with diffeent chaacteistics and distibutions 2. 6 CONCLUSION In this pape, we descibed a lage scale (n=1687) cowdsouced expeiment to investigate whethe the peception of coelation in nine commonly used visualizations can be modeled using Webe s law. The esults of this expeiment indicate that all visualizations tested can be modeled using Webe s law, but that the effectiveness of many visualizations vaies when depicting negatively o positively coelated data. Futhemoe, using the leaned Webe models, we ank the effectiveness of the tested visualizations on the basis of a peceptual law. We also intoduce the notion of peceptual symmeties (o asymmeties) that emeged fom obseving significant pefomance diffeences in visualizations depicting positively vesus negatively coelated data, and suggest that these symmeties might be elated to the visual featues paticipants attend to when judging coelation. ACKNOWLEDGMENTS This wok was suppoted in pat by a gant fom NSF (IIS ). Any opinions, findings, conclusions, o ecommendations expessed in this pape ae those of the authos and do not necessaily eflect the views of the National Science Foundation. 2 To facilitate futue wok, all expeiment mateials, paticipant data, and analyses scipts ae available on the autho s website:

10 REFERENCES [1] J. Betin. Semiology of gaphics: diagams, netwoks, maps [2] S. Capendale. Evaluating infomation visualizations. In Infomation Visualization, pages Spinge, [3] H. Choo and S. Fanconei. Enumeation of small collections violates webe s law. Psychonomic bulletin & eview, 21(1):93 99, [4] W. S. Cleveland, P. Diaconis, and R. McGill. Vaiables on scatteplots look moe highly coelated when the scales ae inceased. Defense Technical Infomation Cente, [5] W. S. Cleveland and R. McGill. Gaphical peception: Theoy, expeimentation, and application to the development of gaphical methods. Jounal of the Ameican Statistical Association, 79(387): , [6] S. Coen, L. Wad, and J. Enns. Sensation and peception Hatcout Bace, Foth Woth. [7] C. Demialp, C. E. Scheidegge, G. L. Kindlmann, D. H. Laidlaw, and J. Hee. Visual embedding: A model fo visualization. Compute Gaphics and Applications, IEEE, 34(1):10 15, [8] M. Fink, J.-H. Haunet, J. Spoehase, and A. Wolff. Selecting the aspect atio of a scatte plot based on its delaunay tiangulation. Visualization and Compute Gaphics, IEEE Tansactions on, 19(12): , [9] M. Haowe and C. A. Bewe. Colobewe. og: An online tool fo selecting colou schemes fo maps. The Catogaphic Jounal, 40(1):27 37, [10] S. Hecht. The visual discimination of intensity and the webe-fechne law. The Jounal of geneal physiology, 7(2): , [11] J. Hee and M. Bostock. Cowdsoucing gaphical peception: using mechanical tuk to assess visualization design. In Poceedings of the SIGCHI Confeence on Human Factos in Computing Systems, pages ACM, [12] V. A. C. Henmon. The time of peception as a measue of diffeences in sensations. Numbe 8. Science Pess, [13] H. Lam, E. Betini, P. Isenbeg, C. Plaisant, and S. Capendale. Empiical studies in infomation visualization: Seven scenaios. Visualization and Compute Gaphics, IEEE Tansactions on, 18(9): , [14] J. Li, J.-B. Matens, and J. J. Van Wijk. Judging coelation fom scatteplots and paallel coodinate plots. Infomation Visualization, 9(1):13 30, [15] S. Lin, J. Fotuna, C. Kulkani, M. Stone, and J. Hee. Selecting semantically-esonant colos fo data visualization. In Compute Gaphics Foum, volume 32, pages Wiley Online Libay, [16] G. Little, L. B. Chilton, M. Goldman, and R. C. Mille. Tukit: human computation algoithms on mechanical tuk. In Poceedings of the 23nd annual ACM symposium on Use inteface softwae and technology, pages ACM, [17] J. Mackinlay. Automating the design of gaphical pesentations of elational infomation. ACM Tansactions on Gaphics (TOG), 5(2): , [18] J. Mackinlay, P. Hanahan, and C. Stolte. Show me: Automatic pesentation fo visual analysis. Visualization and Compute Gaphics, IEEE Tansactions on, 13(6): , [19] W. Mason and S. Sui. Conducting behavioal eseach on amazon s mechanical tuk. Behavio eseach methods, 44(1):1 23, [20] T. Munzne. A nested model fo visualization design and validation. Visualization and Compute Gaphics, IEEE Tansactions on, 15(6): , [21] R. A. Rensink. On the pospects fo a science of visualization. In Handbook of Human Centic Visualization, pages Spinge, [22] R. A. Rensink and G. Baldidge. The peception of coelation in scatteplots. In Compute Gaphics Foum, volume 29, pages Wiley Online Libay, [23] G. Shama. Digital colo imaging handbook, volume 11. CRC, [24] J. Zacks and B. Tvesky. Bas and lines: A study of gaphic communication. Memoy & Cognition, 27(6): , 1999.