EM375 STATISTICS AND MEASUREMENT UNCERTAINTY LEAST SQUARES LINEAR REGRESSION ANALYSIS

Transcription

1 EM375 STATISTICS AND MEASUREMENT UNCERTAINTY LEAST SQUARES LINEAR REGRESSION ANALYSIS I this uit of the course we ivestigate fittig a straight lie to measured (x, y) data pairs. The equatio we wat to fit is of the form: Y = ax + b where a is the slope ad b is the itercept of the lie. If we oly have two data pairs we ca fit a uique lie to them. However, whe we have more data pairs we could fit a large umber of differet lies that pass through the scatter of data poits. The problem therefore arises: how do we choose the best straight lie? There are several differet methods. The oe we develop here is the least squares error approach. This is the oe most commoly used i egieerig, ad solvers are readily available i Excel, MATLAB ad may hadheld calculators. Let us assume we have data pairs (xi, yi) with i = 1... We make the assumptio that all x values have o ucertaity they are kow exactly. The oly ucertaity is associated with the y values. If we kew the actual slope ad itercept of the lie, we could predict the expected y-value for ay give x-value. We choose upper case Y to represet the expected values ad lower case y to represet the measured values. For each measuremet there will be a error ei (differece) betwee the expected value ad observed value. Thus: Y i = ax i + b e i = Y i y i = ax i + b y i The values are demostrated i the figure below. EM375: Liear Regressio - 1

2 y error, ei Expected Y i Observed y i x The approach to fidig the least squares best fit lie is to miimize the sum of the squares of the errors, E. E 2 2 = e i = (Y i y i ) 2 = (ax i + b y i ) 2 Remember that at this stage we do ot kow the slope or itercept, so we caot, just yet, do ay umerical calculatios with our data. We wish to miimize the error E by choosig the best slope ad itercept. We do this by differetiatig the error fuctio separately with respect to (wrt) slope ad itercept ad settig the result to zero to miimize for the total error: wrt slope, a: wrt itercept, b: E 2 a = {2(ax i + b y i )x i } = 2 (a (x i ) 2 + bx x i y i ) = 0 E 2 b = {2(ax i + b y i )} = 2(ax + b y ) = 0 We solve these two equatios simultaeously to determie a slope ad itercept. There are may differet presetatios for this solutio, but the oe we use here uses the relatioships: x = 1 x i ad y = 1 y i EM375: Liear Regressio - 2

3 The resultig slope ad itercept equatios are: slope: itercept: xy x y a = x 2 x 2 b = y a x We call the resultig lie Y = a x + b the least-squares best fit lie (or just the best-fit lie ) to the data represeted by the data set (xi, yi). I these equatios a (a-hat) is the best estimate of the actual slope a, ad b is the best estimate of the actual itercept b. Recall that whe we calculated the average of a data set, the resultig umber was a estimate of the populatio average, ad we could ever determie the populatio average. There is a similar cocept with the best fit lie. We ca ever determie the actual slope ad itercept, a ad b. We ca oly calculate estimates, a ad b. UNCERTAINTY IN THE ESTIMATES While the above equatios give the best estimate, how accurate is that estimate? We ivestigate a method that gives us a estimate of the ucertaity i the calculated slope ad itercept. I other words, we wish to determie the cofidece itervals δ SLOPE ad δ INTERCEPT such that, at a chose level of sigificace: a δ SLOPE < a < a + δ SLOPE b δ INTERCEPT < b < b + δ INTERCEPT I this course we quote the ucertaities, but do ot develop them from first priciples. First we calculate the Mea Square Error (MSE) of the data set. I some texts (for example, Wheeler & Gaji) the MSE is called the stadard error of the estimate S y,x. The MSE ca be calculated as: MSE = S y,x = (residuals)2 ( 2) = (y i Y i ) 2 ( 2) = y2 b y a xy ( 2) The cofidece itervals ca be calculated as follows, where t α, 2 is the Studet-t statistic for a level of sigificace α ad (-2) degrees of freedom: 1 δ SLOPE = ±t α, 2 S y,x ( (x 2 i ) x 2 ) EM375: Liear Regressio - 3

4 δ INTERCEPT = ±t α, 2 S y,x 1 + (x x ) 2 ( (x i 2 ) x 2 ) We see that the cofidece iterval for the slope is a sigle umeric quatity. However, the cofidece iterval for the itercept is a fuctio that depeds upo x. I the equatio x is the value of x for which we calculate the cofidece iterval. We ca calculate the cofidece iterval for a sigle value of x, or we ca calculate the cofidece iterval as a cotiuous fuctio of x. A strategy that is ofte sufficiet for the type of problems we ecouter o this course is to determie the cofidece iterval for the special case whe x is the average value of x, or x = x. For this case oly the itercept cofidece iterval reduces to: δ INTERCEPT = ±t α, 2 S y,x 1 = ±t α, 2 (residuals)2 ( 2) oly whe x = x PRESENTING THE FINAL RESULTS OF THE REGRESSION ANALYSIS Wheever you calculate a least squares error regressio lie, your fial result should ever just quote the best fit slope ad itercept, a ad b. You should ALWAYS iclude the cofidece itervals of the slope ad itercept ad the associated level of cofidece. Also, you should ALWAYS geerate at least oe graph that shows the origial data ad the recostructed best-fit lie. The preferece is also to iclude the cofidece iterval bads. The graphs o the ext page give examples of the fial plots you could produce for a liear regressio aalysis. The plots show the raw data as blue circular symbols ad the best fit lie as a solid blue lie. The upper plot shows the itercept ±cofidece iterval. The solid gree lies show the cofidece iterval as a fuctio of x. The red squares show the cofidece iterval calculated whe x = x. The middle plot shows the slope ±cofidece iterval as the red solid lie. You should esure that these lies itersect at the coordiate (x, y ). Sometimes it ca be iformative to make a separate plot showig the residuals. This plot (the bottom plot) ehaces the errors by removig the fitted straight lie tred. Ay uderlyig oliearity ca sometimes be see i this plot of residuals. The MATLAB code used to geerate the cofidece itervals is show at the ed of this hadout. EM375: Liear Regressio - 4

5 Example plots showig the itercept ad slope cofidece itervals, ad the residuals. EM375: Liear Regressio - 5

6 PROCEDURE FOR DETERMINING A BEST-FIT STRAIGHT LINE USING LINEAR CORRELATION AND REGRESSION 1. Plot the raw data (as symbols, ot a joied lie) 2. Make a subjective decisio about the data ad ay treds 3. Calculate the correlatio coefficiet rxy ad compare it to the values i the table. Make a statistical decisio 4. Calculate slope & itercept of the best-fit lie 5. Calculate the MSE (or stadard error of estimate) 6. Calculate cofidece itervals for the slope ad itercept 7. Plot raw data ad overlay the best-fit lie with the cofidece itervals. Make sure the ceter these plots at the coordiate (x, y ). Remember that may data poits will fall iside the CI Expect a few poits to fall outside the CI 8. Plot the residuals, ad make a subjective decisio about liearity FINAL DECISION BASED ON: SUBJECTIVE looks like a straight lie? STATISTICAL correlatio coefficiet above the radom expected value? GRAPHICAL do the recostructed best-fit lie ad the ucertaity (slope ad itercept) lies look OK? o If the ucertaity is large compared with the chage i y-value over the rage of x data, the straight lie approximatio may ot be adequate. o Plottig the residual values versus x ca ofte be helpful i idetifyig oliear treds i the data. EM375: Liear Regressio - 6

7 MATLAB bare boes code %% Liear Regressio code writte by Prof. Ratcliffe 2019 % This code WILL NOT RUN. IT PRODUCES NO PRINTED OUTPUT. % It is "stripped dow" so that it just shows the various calculatios. % i a compact form. alpha=0.1; cof=1-alpha; % level of sigificace, 10% here =legth(x); R=corrcoef(x,y); R=R(1,2); % correlatio coefficiet % calculate the maximum correlatio value for radom data t2=tiv(1-alpha/2,-2); % studet t for (-2) degrees of freedom micorr=t2/sqrt(-2+t2^2); if abs(r)>micorr fpritf('there is isufficiet evidece to say the data are ot correlated.\'); else fpritf('the data are ot correlated.\'); ed %% the actual liear regressio p=polyfit(x,y,1); slope=p(1); itercept=p(2); %% residuals ad MSE Y=slope*x+itercept; resid=y-y; MSE=sqrt(sum(resid.^2)/(-2)); %% calculate the cofidece itervals for itercept ad slope xbar=mea(x);ybar=mea(y); xrage=max(x)-mi(x); xtilde=mi(x):xrage/100:max(x); ytilde=slope.*xtilde+itercept; CI_itercept=MSE*t2*sqrt(1/ +(xtilde-xbar).^2/(sum(x.^2)-*xbar^2)); CI_iterceptXBar=t2*MSE/sqrt(); % calculated at x-tilde=xbar CI_slope=t2*MSE/sqrt(sum(x.^2)-*xbar^2 ); %% plot itercept CI o figure 1 figure(1);clf;hold o plot(x,y,'o'); xlabel('the x-data (Uits)'); ylabel('the y-data (Uits)'); reflie(slope,itercept); % overlay best fit lie plot(xtilde,ytilde+ci_itercept,'g'); % plot upper CI i gree plot(xtilde,ytilde-ci_itercept,'g'); % lower CI plot(xbar,ybar+ci_iterceptxbar,'sr'); %x-tilde=xbar plot plot(xbar,ybar-ci_iterceptxbar,'sr'); % CI slope figure(2);clf;hold o plot(x,y,'o'); xlabel('the x-data (uits)'); ylabel('the y-data (uits)'); reflie(slope,itercept); % overlay best fit lie % wat the two CI lies to itersect at the mea (x, y) coordiate tmpy=(slope + CI_slope)*xbar+itercept-ybar; h=reflie(slope+ci_slope,itercept-tmpy); % plot upper CI set(h,'color','r'); % make it red tmpy=(slope - CI_slope)*xbar+itercept-ybar; h=reflie(slope-ci_slope,itercept-tmpy); set(h,'color','r') EM375: Liear Regressio - 7