ChEn 475 Statistical Analysis of Regression Lesson 1. The Need for Statistical Analysis of Regression

Transcription

1 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso ChE 475 Statstcal Aalss of Regresso Lesso. The Need for Statstcal Aalss of Regresso What do ou do wth dvdual expermetal data pots? How are the useful to ou ad others? Geerall ou wll wat a smooth represetato of the varato of the depedet varable wth respect to the depedet varables. You wat a equato that explctl shows ths realatoshp betwee the depedet ad depedet varables. Ths allows ou to see approprate treds the data ad outlers. More mportatl, t provdes a accurate wa to terpolate ad sometmes (wth cauto) extrapolate our data to fd values of the depedet varable for values of the depedet varable ot drectl measured. To obta the equato, we "regress" the data to obta values of the parameters the model. The model s the equato relatg the depedet varable,, to the depedet varables, x, x, etc. The model ca be of a kow form based o the phscs of the problem or smpl a hpotheszed form such as a polomal. I ether case, regresso aalss s used to obta the "best" parameter values for the model, values that provde the best represetato of our expermetal data. A good regresso ether over represets or uder represets the accurac of the data ad the treds wth respect to the depedet varables. A good regresso also reports statstcs so that the user of the correlato s aware of the accurac of hs/her results. These attrbutes of a good regresso demad that the perso dog the regresso uderstad these subtler ssues ad be able to use statstcal aalss to make decsos o the regresso procedure, the data, ad the fal coeffcets. As a example of the more subtle ssues uderlg a good regresso, cosder the followg llustrato. Suppose the followg surface teso (γ) data have bee reported, ad ou wat to regress the coeffcets a kow model to "best" represet the data. The ssues that we address here are the bass for defg what s meat b "best." EXAMPLE: Regress coeffcets Surface Teso model from followg T, γ data Model: x C ( x) C + C 3 x+ C 4 x + C 5 x 3 = ORIGIN := Note: s surface teso ad x s reduced temperature, T/Tc Tc := data data := Tr := γ := data :=.. rows( data) Tc γ Tr

2 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso To uderstad some of the ssues volved fdg the "best" regresso of these data, let's cosder alteratve approaches that hghlght two of several ssues that arse fttg data. Issue : Nolear vs. Lear Regresso Lear Models: Lear regresso ca be appled to models lear the parameters. Ths meas that the sestvt coeffcets (dervatves wth respect to each of the parameters) cota o parameters. = β + β x + β x β = β = x β = x Whch of the followg models are lear? = A + B x + Cx b C = a exp( b x) = a + + T T = expa + b T + C For the olear model x C ( x) C + C 3 x+ C 4 x + C 5 x 3 = we ca ft the data usg GENFIT whch does olear regresso. You must eter the model ad the sestvt coeffcets a vector. F( x, C) := C ( x) C + C 3 x+ C 4 x + C 5 x 3 ( x) C + C 3 x+ C 4 x + C 5 x 3 C ( x) C + C 3 x + C 4 x + C 5 x 3 l( x) C ( x) C + C 3 x + C 4 x + C 5 x 3 x l( x) C ( x) C + C 3 x + C 4 x + C 5 x 3 x l( x) C ( x) C + C 3 x + C 4 x + C 5 x 3 x 3 l( x) fucto to be ft dervatve of wrt C dervatve of wrt C dervatve of wrt C3 dervatve of wrt C4 dervatve of wrt C5. vector of tal vg := guesses for D := geft( Tr, γ, vg, F) ST := FTr (, D ) coeffcets Ths performs the olear Ths calculates the regresso ad returs the correlated values at coeffcets each Tr value

3 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 3 fal coeffcets: D = γ ST.3.. Results However, we ca also learze the model b takg the atural logarthm of each sde to obta: + C C x 3 = C l x l( ) = l C + + C x + C x l( x) or l( ) l C + + C x l( x) + C x l( x) + C x 3 l( x) I ths case, l() becomes our ew depedet varable ad x l(-x) for =...3 become our ew depedet varables. Because all of the coeffcets tmes ther depedet varables are added learl, we ma use lear regresso (LINFIT) to fd the coeffcets ths learzed equato..8 l( x) xl ( x) ew := l γ ( ) F( x) := D := lft( Tr, ew, F) D = x 5.34 STl := exp F Tr l( x) 7.6 These are the ew x calculated values l( x) for the lear regresso D := exp D The frst coeffcet the learzed verso was l(c) so covert t to C.3 Tr Results ( D ) D = γ ST STl Tr

4 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 4 Note that there s a bg dfferece the coeffcets, D, that we would report the two dfferet cases. Also ote from the plot that the two dfferet regressos gve dfferet results. The sum of the squared errors (SSE) obtaed from the two methods s gve below. Note that the olear equato gves a smaller SSE, so does that mea t s the "best" ft? The lear regresso follows the up-ad-dow tred of the pots, so does that mea that t s the "best"? Is the up-ad-dow tred eve statstcall real? SSE := ST γ SSE = SSE l := STl γ SSE l = Issue : Number of coeffcets I the above "ssue" we ftted all 5 coeffcets, but do we reall eed, or wat, all of those coeffcets? Let's tr refttg wth fewer parameters. F( x) F3( x) F4( x) := fuctos to use for parameters D := lft( Tr, ew, F) D = l( x) ST := exp F Tr ( D ) := l( x) fuctos to use for 3 parameters D3 := lft( Tr, ew, F3) D3 = xl ( x) ST3 := exp F3 Tr := l( x) xl ( x) x l( x) ( D3 ) fuctos to use for 4 parameters D4 := lft( Tr, ew, F4) D4 = ST5 := STl ( D4 ) ST4 := exp F4 Tr Results SSE := ( ST γ) =.7 5 γ ST ST3 ST4 ST Tr SSE 3 := ( ST3 γ) =.46 5 SSE 4 := ( ST4 γ) =.58 5 SSE 5 := ( ST5 γ) = Notce that terms of the SSE, the 3 parameter ft actuall does better tha the 4 parameter. I geeral, oe would expect that creasg the umber of parameters mproves the ft, but f there s ose the data, do ou reall wat to ft that ose? So how ma parameters should ou ft?

5 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 5 ChE 475 Statstcal Aalss of Regresso Lesso. Aalss of Smple Lear Regresso For ease of llustratg the statstcal aalss that ca gude regressos, smple lear regresso wll frst be dscussed. The cocepts are the easl exteded to geeral lear regresso lke llustrato gve above. We wll assume that we wat to ft a set of data to a straght le = a + bx. To fd the "best" le through a set of (x, ) data, we wll use the method of least squares to mmze the sum of the squared resduals. The resdual s the dfferece betwee the expermetal value ad the value calculated from the assumed equato at the value of x x :=.5 := := s umber of pots :=.. s the dex for all pots 9 8 The sum of squared resduals ca be wrtte as: SSE = = ( r ) = x = a + b x We wat to chage a ad b such a wa that SSE s mmzed;.e., such that the partal dervatves w.r.t. to a ad b are zero. d da SSE = ( a bx ) = or a + b x = b = = = d db SSE = ( a bx ) x = or a x b a ( x ) + = = = = = ( x )

6 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 6 Solvg these two equatos smultaeousl for a ad b elds: A x A x A b = a = A ba x A xx A x where A represets a average over all the pots. For example, A x = x A xx ( x ) = A x = ( x ) A = = = = = For the above data set, the A x := x A xx := x x ( ) A x := ( x ) A := = = = = ad we obta: A x A x A b := a := A ba x b =.89 a = 6.44 ft := a + b x A xx A x Ths s fact what MATHCAD does the slope-tercept fuctos: slope( x, ) =.89 tercept( x, ) = 6.44 ft You alread kew how to do ths, ad we troduced ths smpl to troduce the otato ad the averages defed above that wll be used the statstcal aalss. The real questo here s how do we aalze the regresso that we just performed. Oce a equato has bee ftted to some data, we should alwas ask ourselves the questo: "how good s the ft?" The "goodess" of a ft ca be measured terms of a statstcal aalss of the resduals. There are several thgs ou should alwas check. Check : Resduals r = defto of resdual r := a + b x, exp, calc x A plot of the resduals, lke that at the rght, tells ou whether or ot ther s a bas to the fttg. For example, f there s oe resdual that s large ad postve that s offset b all the other resduals beg egatve, the the model equato does ot represet the data ver well - there s a bas to the data. It's lkel that the oe pot has a error t ad that has throw the le off. Normall we would expect to see a radom dstrbuto of resduals. If the resduals are ot radom, the the pot out areas whch there s a sstematc error the model that does ot represet the data. r x

7 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 7 Oe ca also calculate the average of the resdual to fd out f there s a bas (o-zero) to the average. If there s o large bas the model, the the bas should be close to zero. bas := r bas =. 5 Ths shows that the ft s ot based = Check : R-Squared Statstc The R value s a useful statstc, but ot deftve. It tells ou how well the data ft the model. It does ot tell ou f the model s correct. It tells ou how much of the dstrbuto of the data about the mea s descrbed b the model. (, calc av ) ft A ( ) = R = (, exp av ) R := R =.5 A Check 3: Cofdece Iterval for the Parameters = We ca also calculate cofdece tervals for the parameters that we have regressed. Ths s doe from a estmate of the varace, σ. The varace s calculated terms of the followg quattes: = = ( x A x ) = = ( x ) = x := = ( x A x ) ote that Ax s the average of the x values. The varace s most easl obtaed drectl from SEE, the sum of the squared resduals. SSE := r ( ) SSE = 3.6 Var = σ = = SSE p where - p s the degree of freedom (umber of pots mus the umber of parameters) SSE Var := Var =.4 σ := Var σ =.633 The cofdece terval for the a ad b parameters ca be foud usg the Studet t dstrbuto. I partcular, a P% cofdece terval for the parameters ca be foud b: b a σ t k, σ t k, < b < b + Here t s a value of the t dstrbuto wth - degrees of S k, xx freedom. k s gve b -(-P)/ ad P s the desred cofdece level. σ t A k, xx σ t A k, xx < a < a +

8 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 8 So, for ths example that we are workg through, we ca fd a 95% cofdece terval for a ad b the followg maer..95 For the 95% cofdece terval, k := k =.975 qt( k, ) =.6 The fucto qt(α,β) MATHCAD returs the crtcal value for the Studet's t test at the α probablt level for β degrees of freedom σ qt( k, ) a ucertat := A σ qt( k, ) xx b ucertat := a ucertat =.9 b ucertat =.364 We ca thus report at the 95% cofdece terval the plus/mus ucertates of the parameters;.e., 4.3 < a < < b < 3.74 Check 4: Cofdece Level for the Calculated Values Smlarl, we ca estmate the accurac of predcted values obtaed from our equatos. Ths s a cofdece terval o the mea value obtaed at a x value. Ths s aga doe wth a Studet t test the followg maer. ucertat = t σ + k, + x A x where the t statstc s the same as above. Ths equato ca be appled at each desred value of x ad the plus ad mus values geerated to gve a ucertat bad at the desred cofdece level. I the example we are usg as a llustrato, we would geerate the 95% cofdece bad for predcted values the followg maer: x A x ucertat := qt( k, ) σ + + get the hgh ad low values for the ucertat bad at lo := a + b x ucertat h := a + b x + ucertat each desred x ft h lo x 95% cofdece terval s show for the sample problem Notce how the ucertat bad expads at the two eds.

9 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso 9 ChE 475 Statstcal Aalss of Regresso Lesso 3. Aalss of Multple Lear Regresso Most of what we have leared from smple lear regresso also apples to multple lear regresso where we have more tha oe radom depedet varable. Thus, = b + b x + b x +... Ths of course s the stuato that we had the troductor llustrato after we had learzed the surface teso equato;.e., l( ) = l C C l x + + C x l( x) + C x l( x) + C x 3 l( x) To llustrate the statstcal aalss applcable to multple lear regresso, let us use the followg data: := 3 := data := Y := data X := submatrx( data,, 3,, 4) x data := x data 3 := x3 := data We ca quckl arrve at the values of the coeffcets MATHCAD usg the REGRESS fucto: z := regress( X, Y, ) coeffs := submatrx( z, 4, legth( z),, ) coeffs = Yc := coeffs + coeffs x + coeffs x + coeffs x Note that MATHCAD, the last elemet of the coeffs vector s the costat term the regresso. However ths s ONLY TRUE for more tha coeffcets.

10 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso Sce we have multple depedet varables, we wll swtch to a plot of Y vs. Y so that a perfect ft wll le alog the dagoal. 4 Y 3 Yc 3 4 We wll aga perform several statstcal checks o the qualt of the regresso. Y Check : Resduals r := Y Yc bas := r bas = = SSE := r SSE = = r Check : R-Squared A := Y = Yc A ( ) R := = R =.9 Y A = x Check 3: Cofdece Iterval for the Parameters For multple lear regresso problems we wat to exame the varaces of the parameters but also the covarace of oe parameter wth aother. To do so, we form a matrx from the dervatves of the depedet fucto wth respect to the depedet varables, XX, ad use ths matrx to geerate the matrx A as show below:

11 Statstcal-Regresso_hadout.xmcd Statstcal Aalss of Regresso We ca wrte lear equatos matrx form as Y = XB where Y s the vector of depedet varables B s a vector of parameters, ad X s a p x matrx of the depedet varables. The vector of parameters ca the be solved drectl b matrx algebra. B = X T ( X) X T Y X := X := x X := x X := x3,, 3, 4, B X T := X X T Y B = Y calc := XB Now compute the stadard devato, σ: kk := 3 (kk s the umber of depedet varables: here we have x, x ad x3) SSE Var := σ := Var σ =.73 Var = 4.97 j :=.. 4 kk The cofdece terval s the foud for each parameter usg the T tables ad the correspodg dagoal elemet of (X T X) - *Var..95 For the 95% cofdece terval, k := k =.975 qt( k, 4) =.6 Sb X T := X j j, j Var δ Bj := Sb qt( k, 4) δ j B = Check 3: Cofdece Level for the Calculated Values T T Y ucertat = t σ + X X X X ote that kk=3 kkk, Y u qt( k, 4) σ ( X T ) T X T ( X) ( X T ) := + Ylo := Y Y u Yh := Y + Y u Ylo = 9.8 Y = 5.5 Yh = 3.78 Ylo = 6.86 Y = 3. Yh = 36.4 at the 95% cofdece terval etc.