Least-Squares Regression

Transcription

1 Lest-Squres Regresson Curve fttng Iprove eperentl dt Modelng 5 Eple: 7 eperentll derved ponts ehbtng sgnfcnt vrblt Roberto Muscedere It s probble tht t soe pont n our creer ou wll hve to nlze soe tpe of dt. Ths dt wll hve coe fro soe tpe of observed effect, or wll be fro soethng ou hd no control over. As n engneer, ou wll need to nlze ths dt n order to fnd soe prtculr chrcterstc or trend. Wht ou wll do wll depend on the nture of the dt. If for eple the dt ppers to hve soe vrblt to t, ou wll need to reduce ths error soe how. Curve fttng s one w to do ths. Tpcll curve fttng s used to tr to fnd trend n dt so tht t cn be odeled. Lter, wth ths odel, ou cn perfor other clcultons, but don t forget tht t s onl odel. In the eple bove, we hve severl ponts on grph nd we wsh to fnd soe w to odel ths dt Regresson

2 Lest-Squres Regresson Polnol fttng Eple: 6 th order ft Psses through ll ponts Wde osclltons Roberto Muscedere We cn use polnol fttng, but n ths cse the ponts pper to be followng lner trend. Although we cn use polnol fttng, t ll depends on the dt we re gven. Here we see 6th order ft, but ths curve s not vld outsde the rnge gven. The curve fttng we choose depends on our dt Regresson

3 Lest-Squres Regresson Best lne of ftness Better pproton of dt Eple: best lne through ll ponts Vld for generl/pproton clcultons Roberto Muscedere 3 In ths cse, the best tpe of ftness s lner ft. It offers us good pproton of the dt nd t helps us eclude soe of the splng error. Norll we use lnes of best ft for generl clcultons nd pprotons. We should be wre tht ths tpe of fttng not lws stsf the needs of our dt Regresson 3

4 Lner Regresson Splest eple of lest-squres pproton s fttng strght lne to set of pred observtons (, ), (, ),, ( n, n ) Mthetcl epresson for strght lne: Where b s the ntercept, s the slope, ε s the error or resdul b 3-7 Roberto Muscedere 4 We wll be lookng t the splest eple of lest-squre regresson known s lner regresson. Bscll f we re gven set of dt prs (n ths cse nd vlues on grph), we cn deterne lne of best ft, where b s the ntercept nd s the slope. ε s the error of our pproton, soetes lso known s the resdul Regresson 4

5 Error, or resdul s: Lner Regresson b Dscrepnc between the true vlue of nd the pproton b- 3-7 Roberto Muscedere 5 We cn rerrnge our equton of the lne to fnd the error or wht s clled epslon here. Agn, ths s the dfference between the true vlue of nd our pproton b Regresson 5

6 Lner Regresson Mnze the su of the resdul errors for ll the vlble dt: n An strght lne pssng through the dpont results n nu error 3-7 Roberto Muscedere n b Mdpont 6 In order to fnd the lne of best ft, we need to nze the error soe how. There re n ws to do ths, but we re lookng for the best w. One opton s to su up the error of our pproton. Ths opton s not dequte snce n lner pproton whch les on the d pont of our dt wll gve the nu error Regresson 6

7 Lner Regresson Mnze the su of the bsolute resdul errors: n n b An strght lne wthn the dshed lnes results n nu error 3-7 Roberto Muscedere 7 Another optons s to use the su of the bsolute error. However, ths ethod too suffers fro gvng flse pprotons. In ths cse n lne n between our ponts wll nze the error Regresson 7

8 Lner Regresson Mn crteron The lne s chosen tht nzes the u dstnce tht n ndvdul pont flls fro the lne Gves undue nfluence to n outler ( sngle pont wth lrge error) Outler 3-7 Roberto Muscedere 8 The s ethod known s n n optzton theor whch bscll tres dfferent pprotons n the ttept to nze the overll error fro the true dt. Ths ethod lso suffers snce sngle pont wth lrger error (outler) cn severel pct the pproton Regresson 8

9 Lner Regresson A better strteg tht overcoes the shortcongs of the prevousl entoned pproches s to nze the su of the squres of the resduls, E: E n n b Ths crteron elds unque soluton for gven set of dt 3-7 Roberto Muscedere 9 Our best choce s to use the su of the squre resduls. Ths error functon wll lws eld unque soluton for our gven dt. As our dt chnges, our pproton wll chnge. It s not susceptble s the erler error functons Regresson 9

10 Lner Regresson Dfferentte wth respect to ech vlue to deterne vlues of nd b E b E n n b b 3-7 Roberto Muscedere In order to deterne the vlues of nd b, we dfferentl the error functon wth respect to ech coeffcent. Ths dfferentton s frl sple snce the error functons re lner Regresson

11 Lner Regresson Set dervtves to zero wll result n nu E Splf suton notton b b b nb 3-7 Roberto Muscedere Settng the dervtves equl to zero wll result n nu error. Here we lso use splfed suton notton. All the sutons hve the se dt rnge Regresson

12 Lner Regresson Equtons cn be represented s set of two sultneous lner equtons wth two unknowns nb b 3-7 Roberto Muscedere We cn then represent these equtons s set of two sultneous lner equtons wth two unknowns. We cn now solve for nd b Regresson

13 Lner Regresson Cn use tr solvng ethods or Crer s rule n b 3-7 Roberto Muscedere 3 We cn solve these equtons usng ether tr ethods or snce the tr s resonbl sll, we cn lso use Crer s rule Regresson 3

14 Regresson Roberto Muscedere 3-7 Roberto Muscedere Lner Regresson Lner Regresson Cn use tr solvng ethods or Crer s rule n b n n When we solve the we end up wth these equtons. All of the vlues re constnts tht re bsed on our vlues of nd.

15 Eple: Regresson Clss (p.) clss Regresson { prvte: // ll dt s prvte double *, *; unsgned nt n; publc: vod llocte(unsgned nt nu); vod unllocte(); Regresson(unsgned nt nu, double *, double *); Regresson(const Regresson &); ~Regresson(); Regresson &opertor=(const Regresson &rhs); double lner(double &, double &b); ; 3-7 Roberto Muscedere 5 Here s the begnnng of clss tht wll perfor regresson clcultons for us. Snce we cn hve n nuber of prs of nd, we declre vrbles nd s dnc. We wll then llocte the ount of eor we need to store the prs. n s spl the nuber of prs of dt tht re currentl llocted. Snce we hve dnc eor n ths clss we wll hve functon tht wll llocte the eor for us, nd we wll lso hve functon tht wll un-llocte t. Our storge for nd s prvte, so we need to fcltte w to llow other functons to provde us wth the dt prs. Ths wll hppen through the frst constructor. The second constructor s present snce we re usng dnc eor nd hve to provde for w to crete new objects fro prevous ones. The opertor= overlod ests lso to enble the copng of objects. Lstl the lner regresson functon s defned whch wll return the epslon totl error s well s preters nd b Regresson 5

16 Eple: Regresson Clss (p.) vod Regresson::llocte(unsgned nt nu) { n = nu; f (nu == ) { cout << "Dt ept." << endl; = = ; return; = new double[n]; = new double[n]; vod Regresson::unllocte() { f () delete[] ; f () delete[] ; 3-7 Roberto Muscedere 6 Here we hve our dnc eor functons. It reebers the nuber of prs to be stored nd t wll then llocte the eor to hold those prs. Soe error checkng s done to ke sure n s never less thn. Notce the use of unsgned ntegers to vod negtve stutons s well Regresson 6

17 Eple: Regresson Clss (p.3) Regresson::Regresson(unsgned nt nu, double *, double *) { llocte(nu); for (unsgned nt = ; < nu; ++) { [] = []; [] = []; Regresson::Regresson(const Regresson &) { llocte(.n); for (unsgned nt = ; < n; ++) { [] =.[]; [] =.[]; 3-7 Roberto Muscedere 7 The frst constructor lloctes the necessr ount of eor nd then copes t fro the cllng functon. Ths s n es w to get dt nto the clss. The second constructor s for genertng duplcte objects. Ths functon lloctes eor bsed on the sze of the object pssed nd then copes the dt fro one object to nother Regresson 7

18 Eple: Regresson Clss (p.4) Regresson::~Regresson() { unllocte(); Regresson &Regresson::opertor=(const Regresson &rhs) { f (ths!= &rhs) { unllocte(); llocte(rhs.n); for (unsgned nt = ; < n; ++) { [] = rhs.[]; [] = rhs.[]; return *ths; 3-7 Roberto Muscedere 8 The destructor spl clls the unllocte functon. The opertor= overlod frst checks to ke sure we ren t copng fro the se object. If not, we unllocte the current object, llocte spce bsed on the sze of the one we re copng fro nd then copes ll the dt eleents Regresson 8

19 Eple: Regresson Clss (p.5) double Regresson::lner(double &, double &b) { double su =, su =, su =, su = ; f (n == ) { cout << "Dt ept." << endl; return -; for (unsgned nt = ; <n; ++) { su += []; su += []; su += [] * []; su += [] * []; double D = n * su - su * su; = (n * su - su * su) / D; b = (su * su - su * su) / D; double epslon = ; for (unsgned nt = ; <n; ++) { epslon += fbs([] - b - *[]); return epslon; 3-7 Roberto Muscedere 9 Fro our derved equtons, we wll need the su of ll the s, the su of ll the s, the su of ll the s squred nd the su of the s ultpled b the s. We therefore set up locl vrbles to store these clcultons. We clculte these vlues wth loop tht perfor the clcultons fro ech nd pr. Once tht s done, we cn then clculte nd b. Snce the hve the se denontor, we cn pre clculte tht. nd b re clculted bsed on the prevous clcultons on the dt stored n the object. In order to test how good our lne of ft s, we bck substtute the vlues of nd b bck nto the functon to clculte our overll epslon. We ke sure to strp the sgn so we cn see the overll error. Ths error s the return vlue Regresson 9

20 Eple: Lner Regresson - Test Code nt n(vod) { double [] = {,,, 3, 5 ; double [] = {,.4,., 3.5, 4.4 ; unsgned nt n = szeof() / szeof([]); Regresson dt(n,,); double, b, e; e=dt.lner(,b); cout << "Soluton: " << endl; cout << " = " << << endl << "b = " << b << endl; cout << " or " << endl; cout << " = " << << "* " << "+ " << b << endl; cout << "epslon s " << e << endl; 3-7 Roberto Muscedere Our testng code, n, hs our testng prs set up n sttc rr. We crete our object dt wth n rguent of n (whch s deterned b enng the sze of the sttc dt) nd the nd dt sets. Once ths s done, we cll the solve functon to perfor the lner regresson whch returns us the epslon error s well s nd b. We then crete nce output whch shows the vlues for nd b, nd lso the equton of the lne wth the error noted Regresson

21 Eple: Lner Regresson - Output Soluton: = b = or = * epslon s Y X Dt Appro 3-7 Roberto Muscedere Ths s the output of the code. The fgure s not prt of the code, but t s ncluded so tht ou cn see the nput ponts (sold lne) nd the lne of best ft (dotted lne) Regresson

22 Polnol Regresson Lest squres ethod cn be etended to ft dt to n -th order polnol: Su of the squres of the resduls s: E n 3-7 Roberto Muscedere Soetes lner regresson sn t dequte to odel dt. We need to odel ore cople dt. For ths, we cn etend the order of our lner regresson (whch s currentl ) to n order (n ths cse ). In dong ths, we wll crete n pproton whch depends on, where goes fro to. Just lke lner regresson, we clculte the error functon s the su of the squre of the resduls Regresson

23 Regresson Roberto Muscedere 3-7 Roberto Muscedere Polnol Regresson Polnol Regresson Dfferentte wth respect to ech unknown coeffcent: E E E E In order to deterne the coeffcents, we need to dfferentl ech equton wth respect to. We cn see the pttern here, ech lne s essentll the se ecept the power of (t the end of the equtons) s ncresng.

24 Regresson Roberto Muscedere 3-7 Roberto Muscedere Polnol Regresson Polnol Regresson Set error to zero We now set ll the equtons to zero n order to nze the error.

25 Regresson Roberto Muscedere 3-7 Roberto Muscedere Polnol Regresson Polnol Regresson Rerrnge to develop set of equtons: n We cn rerrnge the equtons so tht the coeffcents ( ) re n nce coluns. Wh do we do ths?

26 Regresson Roberto Muscedere 3-7 Roberto Muscedere Polnol Regresson Polnol Regresson Rerrnge nto tr for: n So we cn plce ll of the nforton n tr. Once we hve ths tr, we cn use technques we developed erler (guss-jordn) to solve for the coeffcents. Notce the trend of the coeffcents. The upper dgonl nd lower dgonl of the left sde of the tr re the se we cn trnspose ths tr nd stll hve the se tr. We wll tke dvntge of ths trend when we wrte the code to buld the tr.

27 Eple: Regresson Clss (p.6) // Assue Mtr clss ests clss Regresson { double pol(unsgned nt degree, double *); ; 3-7 Roberto Muscedere 7 Snce we know we wll need tr to perfor the polnol regresson, we wll ssue tht our tr code s prt of ths code. We cn use our estng clss whch we lred setup for lner regresson. He we just dd the functon prototpe for the pol functon. It tkes the polnol degree nd the rr of where to store the coeffcents of. It too returns the totl epslon error Regresson 7

28 Eple: Regresson Clss (p.7) double Regresson::pol(unsgned nt degree, double *v) { unsgned nt, j; double *sue, t; f (n == ) { cout << "Dt ept." << endl; return -; Mtr t(degree +, degree + ); sue = new double[ * degree + ]; for ( = ; <= * degree; ++) { sue[] = ; for (j = ; j<n; j++) { sue[] += pow([j], ); for ( = ; <= degree; ++) { for (j = ; j <= degree; j++) t.set(sue[j + ],, j); delete[] sue; 3-7 Roberto Muscedere 8 Ths s pge of of our solvng functon. We begn b declrng soe vrbles: we wll need nd j s counters snce we wll be fllng up tr wth soe vlues. We frst ke sure we hve dt n our object, otherwse leve wth negtve error. sue s dnc rr of eleents tht wll hold the vlues of the su of the eleents to the power. Therefore, eleent wll be the su of the eleents to the power, or essentll n. Eleent wll be the su of the eleents to the power. Eleent wll be the su of the eleents to the power nd so on. When we look bck to our trget tr we wll see tht we need to go up to power of. So the pssed rr wll hve + eleents. degree s the order of the polnol pproton. Now tht we know degree we cn crete tr. Our tr needs to be degree+ rows nd degree+ coluns. We then llocte the eor for sue. We cn begn clcultng the tr. We ve prevousl llocted eor to hold sue nd now we wll fll t. We crete loop, countng wth, to go fro to * nclusvel. For ech, we clculte the su of ll the s to the th power. To ke our lves lttle eser, we wll use the th lbrr functon pow. It tkes two rguents, the bse nd the eponent. We use j to count up ech to the power. Now tht we hve ll the constnts for the left sde of the tr, we fll the tr. Lookng bck we see trend tht ech row begns wth the su of s to the power of the row. So we fll the eleents bsed on ths. We use s row counter (fro to nclusvel) nd we use j s colun counter (gn fro to nclusvel). At poston,j we plce n eleent j+, whch s the colun nuber plus the row nuber. Once ths s done, we cn un-llocte sue snce we don t need t nore Regresson 8

29 Eple: Regresson Clss (p.8) for ( = ; <= degree; ++) { t = ; for (j = ; j<n; j++) { t += [j] * pow([j], ); t.set(t,, degree + ); cout << "Mtr before guss-jordn s:" << endl << t; t.gussjordn(); // solve the tr for ( = ; <= degree; ++) v[] = t.get(, degree + ); double epslon =, e; for ( = ; < n; ++) { for (j =, t =, e = ; j <= degree; j++, t *= []) e += v[j] * t; epslon += fbs([] - e); return epslon; 3-7 Roberto Muscedere 9 We now hve to fll the rght sde of the tr. Agn, f we look bck to the tr, we wll see tht the rght colun contns the su of tes to the power of the row. So we crete loop to perfor ths clculton. It s slr to the loop we used to setup sue. Once the vlue s clculted, we plce t n the rght prt of the tr. Now our tr s coplete. We prnt out the contents so we cn see wht t looks lke. Then we perfor the guss-jordn ethod on t to solve t. Assung ths works (we ren t checkng for tr sngulrtes) we the cop the soluton nto the pssed rr v (the coeffcents of the polnol). Just lke n the lner regresson code, we wll do full error clculton to see how good our ft s. Ths wll tke nested look s we hve to clculte the error of ech pont. s our dt counter. We frst set epslon to zero. Our nner loop frst sets j, our coeffcent counter, to zero nd t wll loop untl j s equl to the degree of ths polnol nclusvel. t s our representton of [] to the jth power. We sve lttle CPU te here b ultplng t b [] fter ever loop. t strts t, or []^ nd grows wth ech terton. We clculte v[j]*t for ech coeffcent. When the nner loop s done, we subtrct ths ccuulted error fro [] nd then dd the bsolute vlue to epslon nd then go to the net dt pont. We return epslon when levng the functon Regresson 9

30 Eple: Polnol Regresson - Test Code nt n(vod) { unsgned nt ; double e; double [] = {,,, 3, 4, 5, 6, 7, 8 ; double [] = { 4, 5,, 7,, 6,, 3, ; unsgned nt n = szeof() / szeof([]); Regresson dt(n,,); unsgned nt = ; double *s = new double[ + ]; e = dt.pol(, s); cout << "Soluton: " << endl; cout << " = " << showpos; for ( = ; <= ; ++) cout << s[] << "*^" << << " "; cout << endl << "error s " << e << endl; delete[] s; 3-7 Roberto Muscedere 3 Our test code gn s slr to the lner regresson test code. We setup sttc lst of s nd s. We cn clculte n b usng szeof. We set our order to n ths cse, but ou cn chnge ths dependng on the dt we re trng to pprote. We then crete our Regresson object we cll t dt nd we set t up to hve n eleents nd the contents of nd. We the solve the pproton b cllng the pol functon. We conclude b prntng out the pproton equton usng the clculted coeffcents Regresson 3

31 Eple: Polnol Regresson - Output Mtr before guss-jordn s: Soluton: = *^ *^ -.339*^ error s Y X Dt Appro 3-7 Roberto Muscedere Regresson 3