Why visualisation? IRDS: Visualization. Univariate data. Visualisations that we won t be interested in. Graphics provide little additional information

Transcription

1 Why vsualsaton? IRDS: Vsualzaton Charles Sutton Unversty of Ednburgh Goal : Have a data set that I want to understand. Ths s called exploratory data analyss. Today s lecture. Goal II: Want to dsplay data (.e., for publcaton) Wll save ths for later lecture (f tme) Fnd or dsplay relatonshps n the data Ths s a prelude to model buldng (what s most mportant to model?) Major goal s nter-ocular mpact Vsualsatons that we won t be nterested n Unvarate data Graphcs provde lttle addtonal nformaton ! For an nterestng perspectve on ths dfference, see: Gelman and Unwn. Infovs and statstcal graphcs: Dfferent goals, dfferent looks (wth dscusson). Journal of Computatonal and Graphcal Statstcs. 23 [source: Wkpeda]

2 Summares Hstograms Mean 27.7 Std Dev 9.5 Sample mean x = N x Sample standard devaton Mn. Q 2.7 Medan 28. 3Q 33.6 Max 57.3 Medan and quartles skew multmodalty s x = s N (x x) these three have same summary statstcs! Outlers n hstograms Class-Condtonal Hstograms blood pressure =? Blood pressure data set Frequency Frequency Blood Pressure Postve (dabetes) Negatve Pressure Alternatve: Box plot neg Dabetes? pos Quartle Medan Quartle Extreme data UCI ML repostory says no mssng data (well, for 2 years t dd) [Source: Padhrac Smyth] Blood Pressure Maybe for only 2 groups, graphs not necessary. For more vsual comparsons, can be helpful.

3 Effect of bn sze Effect of bn sze Effect of bn sze More msleadng hstograms x x Data: US Post Codes [Source: Padhrac Smyth]

4 Bvarate data Numercal bvarate summares Data are (x,y ), (x 2,y 2 ),...(x N,y N ) Sample covarance: s xy = N (y N ȳ)(x x) Sample correlaton: xy = s xy s x s y = where as before x = N ȳ = N s x = s y = x y s N s N (x x) (y ȳ) Dangers of correlaton Scatterplots x x [Anscombe, 973]

5 Colour n Scatterplots Token score after attack Token score before attack [Nelson et al, 28] Each pont s a word Entre plot: one emal Axes: Spam score Colour: Whether token was part of an attack on the spam flter Colour n Scatterplots Token score after attack Token score before attack [Nelson et al, 28] For our purposes, note: Use of colour to add a categorcal varable Wthout ths colour would not have seen these two outlers Use of y=x lne to add the eye Overplottng x x2 data ponts x x2 data ponts x x2, data ponts samples from bvarate normal also: notce the axes! 96, bank loan applcants appears: later apps older; realty: downward slope (more apps, more varan [Source: Hand, Manla, and Smyth]

6 Ftted lne To fx overplottng, could consder: Jtterng ponts Subsamplng ponts (.e., plot only %) Averagng (f ths makes sense) Add trend lnes (e.g., quantle lnes) Ths ft s from loess (local lnear regresson). Tme Seres Examples Fnancal data Network traffc Energy usage Human traffc Buldng occupancy Vsualzaton trcks nclude: Smoothng (runnng mean, medan) Repeated multples Transformatons Consder powers, logs. Occasonally recprocals (e.g., rates). Also square root ) ) [Oh et al, 26], fgure from [uan and Murphy, 27] Before After

7 Example Transformaton Wat, what f you have categorcal data? Tools here nclude: Colour Contngency tables Multple plots (e.g., class-condtonal hstograms) Why log log here? Hnt: Imagne a sphercal cow [Source: Wllam Cleveland, Vsualzng Data] Three-Dmensonal Data Hgh-Dmensonal Data Generally hard 3-D plots are not usually useful Usually better to use colour on a 2-D plot Or show multple 2D plots for each value of thrd varable Two man optons: Project the data down to 2-D Many technques Prncpal Components Analyss (IAML, MLPR) Multdmensonal scalng Modern nonlnear methods: t-sne, LLE, Isomap, Egenmaps Problem: Sometmes ths wll obscure hgh-d structure and nonlnear structure Another opton: Scatterplot matrx (see next)

8 Scatterplot matrx Scatterplot matrx Maybe want to use transformed varables up here Colour Ths s performance data for (very old) CPUs Colour Mght be worth understandng ponts lke these Contngency tables Important: Scales must be matched Contngency tables Ths row s the varable we want to predct Ths s the predcton accordng to somebody s model (explans strong relatonshp) What are you lookng for? If you really lke ths stuff Anomales. If somethng looks werd, fgure out why. It could be an error n your data. Learn from your data but do not trust t! (Not completely.) Relatonshps. Hypothess-based vsualzaton. What relatonshps do you expect to exst? Can you see them? Use vsualzaton to nform models and vce versa e.g., Can help wth feature constructon, e.g., whether a relatonshp s really nonlnear Fancy 3D graphs meh These technques also useful for the outputs of learnng! Tukey, Exploratory Data Analyss Bll Cleveland, Vsualzng Data Edward Tufte, all books