Data Analysis using Spreadsheets 1 I

Transcription

1 ENGG1811 Computing for Engineers Data Analysis using Spreadsheets 1 I Ø Data Analysis Ø Pivot Tables Ø Simple Statistics Ø Histogram Ø Correlation Ø Fitting Equations to Data Ø Presenting Charts Ø Solving Single-Variable Equations Ø Goal Seek examples: black holes and fluid mechanics ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 1

2 I love data and why you should love them too I DATA Data are extremely useful You can use data to Discover hidden patterns Learn about the world Predict the future ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 2

3 Kepler s Laws on planetary movement B C D Johannes Kepler ( ) A Law #1: Ellipse Law #2: equal area in equal time Law #3: Kepler dervied his laws from data! Data: positions of the planets over time Pictures from the late Prof. Jan Willems Chaire Francqui lectures ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 3

4 Data allows hidden patterns to be discovered! The following slide shows the acceleration recorded on the lecturer s smartphone during one of his commutes to UNSW The acceleration pattern during walking and nonwalking are very different Can you guess which part is walking and which part is not? ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 4

5 Acceleration pattern

6 Data Analysis Data analysis techniques allow professionals such as engineers, social scientists and economists to extract meaningful information from a typically vast amount of data. Spreadsheets are widely available, and provide useful features for data analysis. Some features are integrated with charts. This week Pivot tables Simple Statistics Histogram Correlation Curve fitting and regression analysis Goal Seek (solves single-variable equations) Next week Solver (for more general optimisation problems) Matrix calculations (very briefly, other tools are more appropriate) Data tables (ideal for financial modelling) Value lookup operations ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 6

7 Summarising Data Pivot Tables A Pivot Table is an interactive table that can be used to summarise large amounts of data quickly. You can move data sets (named by their column header) between rows, columns and the table body, and use multivalued filters to see different summaries of the source data. To create a pivot table, select the data, including header rows, and choose Data Pivot Table Create Dialogue box appears as shown overleaf To make changes, right-click on table and select Edit Layout

8 Summarising Data Pivot Tables Page fields: overall single-valued filter Row and column fields: filterable data Data fields: function (sum, count, max etc) applied to values at intersecting row/col

9 Pivot Table Filters apply to whole table (single value drop-downs or standard filter) Row and column filters are fully optioned (like Excel s filters) Multiple rows in body if multiple fields selected Usually sum, but also count (= number of orders here) Row and column totals ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 9

10 Statistics in engineering Plenty of applications What is the difference between LEGO and other generic brands? * Precision tools are certainly necessary but quality control is equally important * From Page 20 of LEGO Company Profile "Lego Color Bricks" by Alan Chia - Lego Color Bricks. Licensed under Creative Commons Attribution-Share Alike 2.0 via Wikimedia Commons - ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 10

11 Quality control Measure the actual products to see whether the tolerance is met tolerance Too short Day 1 Exact length required [ ] Too long Day 100 Day 200 ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 11

12 Quality control data The worksheet Machines contains data from two machines manufacturing a certain part For each machines, 200 produced parts were measured The spreadsheet shows the the deviation from the standard length We will use Summary statistics Histogram ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 12

13 Simple Statistics Spreadsheets provide many predefined statistical functions to calculate useful information such as: mean, max, min, median, standard deviation, etc. These can be applied to columns or rows of data Excel provides a tool called Descriptive Statistics that calculates such commonly used statistical functions for a given data set and produces a useful report. OpenOffice Calc doesn t have this, but there are several user-contributed statistics packages that do More advanced statistics functions are available (χ 2, t-test, various distributions, confidence intervals etc), but serious analysis usually requires specialised software such as SPSS, SAS or R, and the knowledge of how to use it. ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 13

14 Quiz Products of machines 1 and 2 have identical mean, standard deviation, max and min The histogram of the length deviation of the products from machine 1 is here. Do you expect machine 2 to have the same or similar histogram since they have same mean, stdev, max and min? [Yes/No] ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 14

15 Histogram A histogram is a graphical representation of a frequency distribution of a single variable, using a column chart. The columns represent contiguous ranges in the variable, called bins or classes, usually equally spaced, and the column height shows how many variable values lie in that range. Frequencies are stored in a table, the first column of which are the bin boundary values, the second is filled in by the software Calc and Excel consider bin thresholds to be upper bounds, so there s an extra unlabelled bin after the last ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 15

16 Histogram: Frequency table Find a spot on the sheet for the frequency table, 10 to 20 bins is about right Estimate the variable range by plotting or simple stats Fill in the bin values, equally spaced and rounded unless the categories are going to be labelled First value should be less than min and last value > max, to make sure the full range is covered ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 16

17 Histogram: completion Click in the first frequency cell, select the Function Wizard (important because this is an array operation). Find FREQUENCY in the function list and double-click classes: the bin threshold range, without the header end freqs are 0 as expected Chart the result, adding titles and labels in case the bin values are misleading ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 17

18 Correlation Correlation is a statistical measure referring to the strength of linear relationship between two or more dependent variables that have the same independent variable (time, position etc) It can vary from -1 (perfect negative correlation) through 0 (no correlation) to +1 (perfect positive correlation) Negative correlation means that larger values in one set are associated with smaller values in the other, positive goes the other way. Uncorrelated data simply shows no significant relationship ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 18

19 Correlation using Spreadsheets Calc provides the CORREL function. It accepts two ranges and returns the correlation coefficient. Alternatively, you can plot a chart for two or more variables and try to visually identify possible correlations between variables. For several variables, you can produce a table by using references carefully to allow fill operations across and down Excel provides a Correlation tool that calculates correlations between two or more variables. It constructs the table for you. ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 19

20 Correlation Table = correl($b$15:$b$58;b$15:b$58) Formula: = correl($b$15:$b$58;c$15:c$58) first range uses absolute addresses second range is mixed to allow fill right For the first row, Enter the correlation formula shown and then fill right For each row after the first, copy formula from the cell above it change column letter on the first range only beware of Calc s formula prompts! ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 20

21 Correlation is Not Necessarily Causation If two data sets are correlated, it doesn t mean that the processes behind one caused the other, they could be influenced by some (often complex) third process such as climate change geographic influences socio-economic factors solar activity There have been celebrated cases of correlations for which no credible explanation is likely, like this one: ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 21

22 Bogus(?) Correlation is Still Interesting Source is the New England Journal of Medicine Article below (a secondary reference) has an interesting discussion on what the influences might be, including possible Swedish parochialism producing one outlying data point. ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 22

23 Proving causation Need to establish the chain of events Cigarettes è (Physiological events) è Bladder cancer ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 23

24 Application of correlation in engineering Plenty! Example: Want to control the pressure in a chemical reactor You may adjust the temperature or concentration of an ingredient, which one is a better choice? The correlation of these variables: Many other examples: Noise control, vibration control, mobile phones, GPS, radar, Correlation is limited to linear relationship, but there are many generalisations "Batch reactor.2" by Echis at English Wikipedia Licensed under Creative Commons Attribution 2.5 via Wikimedia Commons ENGG UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 24

25 Fitting Equations to Data Given an X-Y plot of values derived from a physical system with likely uncertainty in measurements, many lines or curves can be drawn to approximate the data The method of least squares chooses the parameters for a regression line or curve of a given type specified by the user The best-fit curve is the one that minimises the sum of the squares of the residuals (differences between data and the predicted value). Trend Lines apply regression to data on a line chart. They provide the regression equation and R 2 value, and show the regression line superimposed on the data values. The R 2 value (varies from 0 to 1) indicates how well the model fits the data. R 2 = 1 indicates that the regression line (curve) is a perfect fit to the data. ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 25

26 Regression and Spreadsheets Calc can apply linear, exponential, logarithmic and power regression trend lines to a chart, but not polynomials L Excel can also do polynomials Both can display the equation and R 2 value on the chart J Both do so with poor choice of typeface and significance L But you can roll-your-own with drawing elements J Both can extrapolate the trend (use the trend line as a prediction of behaviour outside the range); in Calc you just change the X axis limits. If you just want the numbers, can use the linest, logest, growth functions; trend calculates points on the linear trend line Apart from simple analyses, regression is often done with Matlab or with statistical packages ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 26

27 Fitting Equations to Data When chart is in edit mode, Insert Trend Lines activates the dialogue box Use scatter plot with small unfilled symbol markers, no lines Options to show regression on the chart ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 27

28 Content vs Presentation The result shows that the trend is appropriate, but the chart is not yet ready to be included in a professionally presented document. Defaults are rarely good enough. Y-axis number format is inappropriate There may be a need to point out features Auto-range implies there s a zero day Equation format is very poor (font, exp, precision, spacing are all wrong) Don t need grids as the chart is there to show a trend, not data Axis number font size is too large, the numbers aren t as important as the shape of the plot ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 28

29 Fixing the Presentation Remember to double-click the chart to enter edit mode Feature to improve How to make changes Axis attributes Remove grid Right click on axis, select Format Axis Format Grid All Grids from main menu Edit equation format Add a text box (anywhere, not just on a chart) Format text in box (double click to edit) Lines, arrows, circles Format drawing elements (not supported in Calc, try the next) Pick text from the drawing toolbar Drag (off chart), type initial text General font changes from main toolbar; To apply sub/superscript, select characters and then Format Character Pick from drawing toolbar Right click, select Line/Area/Text; resize by dragging handles ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 29

30 Final Product There s still room for personal taste: you might not like to use fill and outline on the regression box, or prefer different relative font sizes. Always ask yourself: does the chart (etc) convey the message embedded in the data effectively? ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 30

31 Other Presentation Issues Any labels you add overlay the chart, they re not part of it Select in turn with Shift-click, or drag a marquee around them Format Group allows selected objects to be part of a single item The chart can be copied intact to another document such as OpenOffice Writer or Microsoft Word Calc to Writer is reliable, Excel to Word isn t always Other copy options if the other application can t import Click on group, File Export as PDF, use Range Selection, and export graphic images using lossless compression (PNG) If you really have to take a screen dump, expand the view first and always save as PNG, never JPG. PDF PNG, 150ppi JPEG, 150ppi PNG, 75ppi ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 31

32 Trends in Global CO 2 Concentrations For the final example, data for atmospheric concentrations of CO 2 taken at the Mauna Loa Observatory (Hawaii) each month since 1958 is readily available on the web Plot just the last 10 years of the data using thin lines: the result shows the detail known as the Keeling Curve after CO 2 researcher Charles David Keeling Annual pattern is clearly evident, can you explain these? the overall shape, peaking in May each year, trough in September the kink around January Apply trend line, looks pretty clear and ominous doesn t it? on this model, when will or did we reach one of the tipping points, 400ppm, and when will the concentration never fall below that mark again (without extraordinary global effort)? Apply a trend line to the whole data set, is it still linear? ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 32

33 Solving equations: engineering motivation You will be learning how to solve many different types of equations during your engineering program at UNSW Solving equations can be used to discover hidden information Example: Fukushima radioactive emission (2011) When and how much radioactive Xenon was released Problem: It s a past event, can t go back to measure it! Equations can be solved to estimate what you cannot directly measured! ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 33

34 The key ideas A B C Measurements from 15 stations around the world on radioactive Xenon level Knowledge of the dispersion of substance in the atmosphere The Fukushima Inverse Problem ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 34

35 What is company X? The message behind the headline is: The founders of company X came up with a brilliant equation The solution of the equation turned company X into a $25B business in (About $400B in Feb 2014.) A lot of complicated equations can be solved automatically and routinely by computers nowadays Know what computational tools are available Spreadsheet may not be robust for all purposes, there are other tools. More later. Make your own tools only if necessary Difficulty is in coming out with useful and insightful equations that can transform the world ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Excel slide 35

36 Solving Equations in One Variable Finding the roots of an equation can sometimes, but not always, be done analytically, but if not we need other approaches. Method 1: Graphical estimate Assign a range of values to a variable (say x), calculate the corresponding function values (say f (x)), and plot a graph for f (x) vs x. Now try to see where the chart line intersects with the x-axis. This approach is useful provided we can guess finite intervals within which to search for possible root values ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 36

37 Solving Single Equations 3 2 f ( x) = 4x + 12x 64x + 16 ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 37

38 Solving Single Equations Method 2: Goal Seek (Tools Goal Seek) works with three inputs A formula cell A target value that you would like the cell to calculate, and A variable cell that is used, even indirectly, by the formula cell Goal Seek tries to find a value for the variable cell that results in the target value in the formula cell. It uses iterative refinement, beginning with the current value. If an initial guess is not close enough, Goal Seek may not be able to find a solution Goal Seek tries a fixed number of iterations (attempts at getting closer to the goal) and stops after that, even if the equation is not solved. The Solver tool (next week) is more powerful than Goal Seek. The tools are very similar in Calc and Excel ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 38

39 Example 1: Cygnus X-1 Mass Calculation Mass of an black hole in a binary system can be calculated from a formula that is cubic in the unknown mass (see CygnusX1 sheet) This cubic has one positive real root ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 39

40 Example 2: Transcendental equation Note: Fluid mechanics concept are not important here. The key idea is to show you that OO Calc can solve some complicated looking equations Variable to solve: Reynolds number (R) Small R means flow is smooth, otherwise turbulence Parameters are: D = pipe diameter υ = viscosity (Greek letter `nu ) g = acceleration due to gravity R is the solution to 1 p +2log 10 ( 2.51 p )=0 where = 2gD3 R 2 R 2 ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 40

41 Summary of Learning Outcomes With practice you should be able to do the following Create and manipulate a pivot table from multivariate data Construct histograms to analyse large data sets Identify correlation between a small number of variables, using the correl function and correctly interpreting its results Use scatter charts and trend lines to identify the parameters of processes underlying noisy data, and to extrapolate trends Present charts and other visual content professionally Apply graphical methods to solving equations in one variable Set up a simple equation model and use Goal Seek to identify the value of an input parameter that converges the result to a particular target ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 41

42 Application of data in renewable energy The main problem of using wind as a renewable energy source is that it is not always there Computers and data are used to: Predict the availability of wind Predict the maintenance cycles of wind turbines Picture source: MacKay, Sustainable energy without hot air ENGG1811 UNSW, CRICOS Provider No: 00098G Data Analysis using Spreadsheets I slide 42