Basic statistical operations

Transcription

1 COSC 6397 Big Data Analytics Fundamental Analytics Edgar Gabriel Spring 2014 Basic statistical operations Calculating minimum, maximum, mean, median, standard deviation Data typically multi-dimensional -> analytics can be based on one or more dimensions of the data 1

2 Group-by operations Calculate basic operations by group Allows to utilize more than one reducer Grouping based on key of the mapper step Frequency distributions arrangement of the values that one or more variables take in a sample Each entry in the table contains the frequency or count of the occurrences of values within a particular group table summarizes the distribution of values in the sample Example: Analyze the log file of a web server Sort the number of hits received by each URL in ascending order Input Example: [01/Jul/1995:00:00:00: ] GET /shuttle/countdown/countdown.html HTTP/

3 Frequency distributions First MapReduce job counts the number of occurrences of a URL Result of the MapReduce job: a file containing the list of <URL> <no. of occurrences> Second MapReduce job Use the output of first MapReduce job as input Mapper: use <no of occurrences> as key and <URL> as value Reducer: omit the <no of occurrences> in output file (ignoring URL) Sorting done implicitly by the MapReduce framework Example output 3

4 Histograms graphical representation of the distribution of data. estimate of the probability distribution of a continuous variable representation of tabulated frequencies, shown as adjacent rectangles, erected over discrete intervals area proportional to the frequency of the observations in the interval Example: Determine the number of accesses to the web server per hour Histograms Map step uses the hour as the key and one as the value Reducer sums up the number of occurrences for each hour 4

5 Histograms Scatter Plots A scatter plot is using Cartesian coordinates to display values for two variables for a set of data Typically used when a variable exists that is below the control of the experimenter a parameter is systematically incremented and/or decremented by the other, also called the control parameter or independent variable is typically plotted along the horizontal axis The measured or dependent variable is customarily plotted along the vertical axis 5

6 Scatter Plots Example: analyzes the data to find the relationship between the size of the web pages and the number of hits received by the web page Scatter Plots 6

7 Joining Two Datasets 7

8 Data Visualization Goal: produce high quality figures to be easily incorporated into Latex (or Word) documents scalable and not too big in size Spreadsheet software packages can be used to make figures, but there are limitations: scalability readability size conformity Figures, Formats and the right Tools Illustrations or diagrams (vector images): Adobe Portable Document Format (PDF) PostScript (PS) Encapsulated PostScript (EPS) Photography or microscopy (raster images): Tagged Image File Format (TIFF) Gimp ( EPS Adobe Photoshop PS Adobe Fireworks PDF Important: set the resolution to the desired DPI (dots per inch) value before you begin your editing. GNUPLOT Matlab Tecplot 8

9 Resources GNUPLOT homepage Introduction to GNUPLOT and Not so FAQ and Solutions Software Linux from source or packaged in distribution MacOS Windows load Book Philipp K. Janert Gnuplot in Action: Understanding Data with Graphs. Manning Publications Co., Greenwich, CT, USA. Basics GNUPLOT is a freely distributed command-line based interactive plotting program Can be used in different modes: Interactive console Scripts Interactive GUI 9

10 More Basics - Syntacs GNUPLOT can display the manuals for its commands e.g. type help plot to get information on the plot command commands can be shortened e.g. rep instead of replot or p instead of plot reset restores the defaults several GNUPLOT commands in one line have to be separated by ; GNUPLOT comments start with # shell commands (e.g. vi) in GNUPLOT start with! file names have to be enclosed in single or double quotes Output Formats GNUPLOT uses different terminals e.g. latex, tikz, eps, png... Example: set terminal png picsize X Y set output "plot.png" set term postscript enhanced eps color "Helvetica,26 \ linewidth 4 rounded set output "ErrorDistributions.eps" 10

11 Simple Example To plot a sine curve open GNUPLOT and type: f(x) = sin(x) # define a function plot f(x) # plot this function replot f(2*x) # plot another function Customizations: set ytics 0.5; set mytics 5 rep # update plot set xrange [-pi:pi] # x range set xtics ("-pi" -pi, "-pi/2" -pi/2, 0, "pi/2" pi/2, "pi" pi) Scripting For your scientific work you will prefer scripts Store the commands in a text file Load the script in GNUPLOT by typing either: gnuplot SinExample.plt (command-line) load SinExample.plt (in GNUPLOT) 11

12 Plotting Data From Files GNUPLOT can read data from files data columns are separated by a whitespace or tab Lines that begin with "# are ignored data formats can be specified with the using command blank lines in data files can be used to identify individual blocks of data by using the index command plot 'data.txt' using 1:2 title "TIMP" with lines ls 1, \ '' using 1:4 title "TIMF" with lines ls 2, \ '' using 1:3 title "GSR" with lines ls 3 set style line 1 lt 1 lw 4 lc rgb "red" set style line 2 lt 2 lw 4 lc rgb "black" set style line 3 lt 1 lw 4 lc rgb "black" Simple Example: plot data.txt using 1:2 with lines set key top right set nokey place or hide key set a title define axis labels change the number format select zoom manually select range of axis set yrange [*:*]... select zoom of y-axis automatically, color, width and shape of lines/points (linetype / lt, pointtype / pt, linewidth / lw, pointsize / ps) plot multiple data series separated by commas set format y "%1.1f" set title "Subject V001" font "Helvetica-Bold,18" set xlabel "Time [s]" font "Helvetica-Bold,16" set xrange [ 0 : 245 ] set yrange [ 0 : 1.2 ] 12

13 Scatter Plots Fitting data Simple Example: plot data.txt using 1:2 with points Data fitting define a power law fit your data clip the fit to the fitted area add to plot define a label f(x) = m*x + bfit f(x) 'AttemptsData.txt' using 1:2 via m,bplot \ 'AttemptsData.txt' using 1:2 with points lt -1 pt 7 notitle set label "r = 0.371" at graph 0.6, graph 0.43 Vertical Bar Graphs & Boxplots boxwidth 0.3 absolute set style fill solid border -1 set key left top boxwidth 1.0 absolute set style line 1 lt -1 lw 1 set style histogram cluster gap 1 set style data histogram plot 'HistogramData.txt' index 0 using 2 fs solid 0.5 ls 1 title "Novices", \ '' index 1 using 2 fs solid 0.25 ls 1 title "Experienced" 'MPE.txt' index 0 using 1:3:2:6:5 with candlesticks lt -1 lw 2 ti "Scenario 1" whiskerbars fs solid 0.5, \ '' index 0 using 1:4:4:4:4 with candlesticks lt -1 lw 2 notitle 13

14 Logscale [un]set logscale [xy] Other Useful Commands do math on data columns plot./data.txt u ($0*10):($2*10**($1)) w lp short format p data.txt u 1:2 w p pt 1 lt 2 lw 2 Missing data: set datafile missing 'NaN' Set grid # Line style for axes set style line 80 lt 1 set style line 80 lt rgb "# # Line style for grid set style line 81 lt 0 # dashed set style line 81 lt rgb "#808080" # grey set grid back linestyle 81 set border 3 back linestyle 80 set ytics nomirror stack several plot commands scale the plot place the plot leave multi plot mode Multiple Graphs set multiplot set size 1.0,0.9 set origin 0.0,0.9 unset multiplot 14

15 Optimizing Your Plots 1.Focus on the purpose of the figure: What do you want to show? 2.Keep it simple and efficient: Choose good units for the axes. Scale the axes to make good use of the figure s area. 3.Explain what you plot: Label the axes, find a good title, add a key to symbols and write a clear and complete caption (not just re-stating what s on the axes). The plot should be self explanatory the intended audience should understand it even without reading the text of your paper. 4.Show your figure to a colleague: optimally somebody not directly working with you to check if it is clear. [1] Checklist enough information in well chosen title / caption? content of labels (e.g. units)? content of key of symbols? too much information for a single graph? plot type suited for purpose? scatter vs. line graph; error bars; fits is x the independent variable and y the dependent variable? large enough font size of labels? sufficient line width? (e.g. a few pixels for presentations with data projectors) optimized number of tick marks, minor tick marks? plot looks empty? zoom... make good use of the plot area plot format (eps, png, tikz) suited for the purpose? [1] 15