Numerical Methods Lecture 6 - Curve Fitting Techniques

Transcription

1 Numerical Methods Lecture 6 - Curve Fittig Techiques Topics motivatio iterpolatio liear regressio higher order polyomial form expoetial form Curve fittig - motivatio For root fidig, we used a give fuctio to idetify where it crossed zero where does f( x) 0?? Q: Where does this give fuctio f( x) come from i the first place? Aalytical models of pheomea (e.g. equatios from physics) Create a equatio from observed data 1) Iterpolatio (coect the data-dots) If data is reliable, we ca plot it ad coect the dots This is piece-wise, liear iterpolatio This has limited use as a geeral fuctio f( x) Sice its really a group of small f( x) s, coectig oe poit to the ext ad, it does t work very well for data that has built i radom error (scatter) ) Curve fittig - capturig the tred i the data by assigig a sigle fuctio across the etire rage. The example below uses a straight lie fuctio Iterpolatio f(x) ax + b for each lie Curve Fittig f(x) ax + b for etire rage A straight lie is described geerically by f(x) ax + b The goal is to idetify the coefficiets a ad b such that f(x) fits the data Numerical Methods Lecture 6 - Curve Fittig Techiques page 106 of 118

2 other examples of data sets that we ca fit a fuctio to. height of dropped obect Oxyge i soil pore pressure time Profit temperature soil depth paid labor hours Is a straight lie suitable for each of these cases? No. But we re ot stuck with ust straight lie fits. That is ust where we ll start. Liear curve fittig (liear regressio) Give the geeral form of a straight lie f( x) ax + b How ca we pick the coefficiets that best fits the lie to the data? First questio: What makes a particular straight lie a good fit? Why does the blue lie appear to us to fit the tred better? Cosider the distace betwee the data ad poits o the lie Add up the legth of all the red ad blue verticle lies This is a expressio of the error betwee data ad fitted lie The oe lie that provides a miimum error is the the best straight lie Numerical Methods Lecture 6 - Curve Fittig Techiques page 107 of 118

3 Quatifyig error i a curve fit assumptios: 1) positive or egative error have the same value (data poit is above or below the lie) ) Weight greater errors more heavily we ca do both of these thigs by squarig the distace deote data values as (x, y) >> deote poits o the fitted lie as (x, f(x)) sum the error at the four data poits (x,y ) (x 4,y 4 ) (x,f(x )) (x 4,f(x 4 )) err ( d i ) ( y 1 fx ( 1 )) + ( y fx ( )) + ( y 3 fx ( 3 )) + ( y 4 fx ( 4 )) Our fit is a straight lie, so ow substitute f( x) ax + b # data poits err ( y i fx ( i )) # data poits ( y i ( ax i + b) ) i 1 i 1 The best lie has miimum error betwee lie ad data poits This is called the least squares approach, sice we miimize the square of the error. # data poits miimize err ( y i ( ax i + b) ) i 1 time to pull out the calculus... fidig the miimum of a fuctio 1) derivative describes the slope ) slope zero is a miimum > take the derivative of the error with respect to a ad b, set each to zero err x a i ( y i ax i b) 0 i 1 err y b ( i ax i b) 0 i 1 Numerical Methods Lecture 6 - Curve Fittig Techiques page 108 of 118

4 Solve for the a ad b so that the previous two equatios both 0 re-write these two equatios ax i + bx i ( x i y i ax i y i put these ito matrix form x i x i x i b a y i ( x i y ) i what s ukow? we have the data poits ( x i, y i ) for i 1,...,, so we have all the summatio terms i the matrix so ukows are a ad b Good ews, we already kow how to solve this problem remember Gaussia elimiatio?? A so AX x i, X b, B x i x a i B usig built i MATLAB matrix iversio, the coefficiets a ad b are solved >> X iv(a)*b Note: A, B, ad X are ot the same as a, b, ad x Let s test this with a example: y i ( x i y ) i i x y First we fid values for all the summatio terms 6 x i 7.5, y i.5, x i 13.75, x i y i 41.5 Numerical Methods Lecture 6 - Curve Fittig Techiques page 109 of 118

5 Now pluggig ito the matrix form gives us: b a.5 Note: we are usig x i, NOT ( x i ) 41.5 b a iv * or use Gaussia elimiatio... The solutio is b 0 > f( x) 3x + 0 a 3 This fits the data exactly. That is, the error is zero. Usually this is ot the outcome. Usually we have data that does ot exactly fit a straight lie. Here s a example with some oisy data x [ ], y [ ] b a , b iv * , b a a so our fit is f( x) x Here s a plot of the data ad the curve fit: So...what do we do whe a straight lie is ot suitable for the data set? Profit paid labor hours Straight lie will ot predict dimiishig returs that data shows Numerical Methods Lecture 6 - Curve Fittig Techiques page 110 of 118

6 Curve fittig - higher order polyomials We started the liear curve fit by choosig a geeric form of the straight lie f(x) ax + b This is ust oe kid of fuctio. There are a ifiite umber of geeric forms we could choose from for almost ay shape we wat. Let s start with a simple extesio to the liear regressio cocept recall the examples of sampled data height of dropped obect Oxyge i soil pore pressure time Profit temperature soil depth paid labor hours Is a straight lie suitable for each of these cases? Top left ad bottom right do t look liear i tred, so why fit a straight lie? No reaso to, let s cosider other optios. There are lots of fuctios with lots of differet shapes that deped o coefficiets. We ca choose a form based o experiece ad trial/error. Let s develop a few optios for o-liear curve fittig. We ll start with a simple extesio to liear regressio...higher order polyomials Polyomial Curve Fittig Cosider the geeral form for a polyomial of order f( x) + x+ a x + a 3 x a x + a k x k Just as was the case for liear regressio, we ask: k 1 (1) How ca we pick the coefficiets that best fits the curve to the data? We ca use the same idea: The curve that gives miimum error betwee data y ad the fit f( x) is best Quatify the error for these two secod order curves... Add up the legth of all the red ad blue verticle lies pick curve with miimum total error Numerical Methods Lecture 6 - Curve Fittig Techiques page 111 of 118

7 Error - Least squares approach The geeral expressio for ay error usig the least squares approach is err ( d i ) ( y 1 fx ( 1 )) + ( y fx ( )) + ( y 3 fx ( 3 )) + ( y 4 fx ( 4 )) where we wat to miimize this error. Now substitute the form of our eq. (1) f( x) + x+ a x + a 3 x a x + a k x k k 1 ito the geeral least squares error eq. () err 3 y i + x i + a x i + a 3 x i a x i i 1 where: - # of data poits give, i - the curret data poit beig summed, - the polyomial order re-writig eq. (3) err y i a k x k + i 1 k 1 fid the best lie miimize the error (squared distace) betwee lie ad data poits Fid the set of coefficiets a k, so we ca miimize eq. (4) () (3) (4) CALCULUS TIME To miimize eq. (4), take the derivative with respect to each coefficiet, a k k 1,..., zero err y a i a k x k i 1 k 1 err y a i a k x k + x 0 1 i 1 k 1 err y a i a k x k + x 0 i 1 k 1 : : err y a i a k x k + x 0 i 1 k 1 set each to Numerical Methods Lecture 6 - Curve Fittig Techiques page 11 of 118

8 re-write these + 1 equatios, ad put ito matrix form x i x i... x i x i 3 x i x i x i +... x i : : : : 3 4 x i x i x i x i x i x i... x i a : a y i ( x i y ) i x i yi : x i yi where all summatios above are over i 1...,, what s ukow? we have the data poits ( x i, y i ) for i 1,..., we wat, a k k 1...,, We already kow how to solve this problem. Remember Gaussia elimiatio?? A x i x i... x i x i x i x i... x i 3 4 +, X a, B x i x i x i... x i : : : : : a x i x i x i... x i where all summatios above are over i 1...,, data poits y i ( x i y ) i x i yi : x i yi Note: No matter what the order, we always get equatios LINEAR with respect to the coefficiets. This meas we ca use the followig solutio method AX B usig built i MATLAB matrix iversio, the coefficiets a ad b are solved >> X iv(a)*b Numerical Methods Lecture 6 - Curve Fittig Techiques page 113 of 118

9 Example #1: Fit a secod order polyomial to the followig data i x y Sice the order is ( ), the matrix form to solve is x i x i 3 x i x i x i 3 4 x i x i x i Now plug i the give data. Before we go o...what aswers do you expect for the coefficiets after lookig at the data? 6 x i 7.5, y i x i 13.75, x i y i x i 8.15 x i yi x i a y i x i y i x i yi a Note: we are usig x i, NOT ( x i ). There s a big differece usig the iversio method a iv * Numerical Methods Lecture 6 - Curve Fittig Techiques page 114 of 118

10 or use Gaussia elimiatio gives us the solutio to the coefficiets a 0 0 > f( x) 0 + 0*x + 1*x 1 This fits the data exactly. That is, f(x) y sice y x^ Example #: ucertai data Now we ll try some oisy data x [ ] y [ ] The resultig system to solve is: a iv * givig: a So our fitted secod order fuctio is: f( x) x* *x Example #3 : data with three differet fits I this example, we re ot sure which order will fit well, so we try three differet polyomial orders Note: Liear regressio, or first order curve fittig is ust the geeral polyomial form we ust saw, where we use 1, d ad 6th order look similar, but 6th has a squiggle to it. We may ot wat that... Numerical Methods Lecture 6 - Curve Fittig Techiques page 115 of 118

11 Overfit / Uderfit - pickig a iappropriate order Overfit - over-doig the requiremet for the fit to match the data tred (order too high) Polyomials become more squiggly as their order icreases. A squiggly appearace comes from iflectios i fuctio Cosideratio #1: 3rd order - 1 iflectio poit 4th order - iflectio poits th order - - iflectio poits overfit Cosideratio #: data poits - liear touches each poit 3 data poits - secod order touches each poit data poits - -1 order polyomial will touch each poit SO: Pickig a order too high will overfit data Geeral rule: pick a polyomial form at least several orders lower tha the umber of data poits. Start with liear ad add order util treds are matched. Uderfit - If the order is too low to capture obvious treds i the data Profit paid labor hours Straight lie will ot predict dimiishig returs that data shows Geeral rule: View data first, the select a order that reflects iflectios, etc. For the example above: 1) Obviously oliear, so order > 1 ) No iflcetio poits observed as obvious, so order < 3 is recommeded > I d use d order for this data Numerical Methods Lecture 6 - Curve Fittig Techiques page 116 of 118

12 Curve fittig - Other oliear fits (expoetial) Q: Will a polyomial of ay order ecessarily fit ay set of data? A: Nope, lots of pheomea do t follow a polyomial form they may be, for example, expoetial Example : Data (x,y) follows expoetial form >> x -:.5:4; %create x axis >> y 1.6*exp(1.3*x); %create y data values\ >> %use built i fuctio to fid poly coefficiets >> P polyfit(x,y,); >> P3 polyfit(x,y,3); >> %use built i fuctio to get fits >> fx polyval(p,x); >> fx3 polyval(p3,x); >> %plot the results >> plot(x,y,'rd',x,fx,'g',x,fx3,'b') >> leged('data','d','3rd') Note that either d or 3rd order fit really describes the data well, but higher order will oly get more squiggly What to do...what to do... (he asks thoughtfully) We created this sample of data usig a expoetial fuctio. Why ot create a geeral form of the expoetial fuctio, ad use the error miimizatio cocept to idetify its coefficiets Geeral expoetial equatio f( x) Ce Ax Cexp( Ax) Agai with the error: solve for the coefficiets C, A such that the error is miimized: miimize err ( y i ( Cexp( Ax) )) i 1 Problem: Whe we take partial derivatives with respect to EAR equatios with respect to C, A err ad set to zero, we get two NONLIN- So what? We ca t use Gaussia Elimiatio or the iverse MATLAB fuctio aymore. Those methods are for LINEAR equatios oly... Now what? Numerical Methods Lecture 6 - Curve Fittig Techiques page 117 of 118

13 Solutio #1: Noliear equatio solvig methods Remember we used Newto Raphso to solve a sigle oliear equatio? (root fidig) We ca use Newto Raphso to solve a system of oliear equatios. Is there aother way? For the expoetial form, yes there is Solutio #: Liearizatio: Let s see if we ca t do some algebra ad chage of variables to re-cast this as a liear problem... Give: pair of data (x,y) Fid: a fuctio to fit data of the geeral expoetial form y Ce Ax 1) Take logarithm of both sides to get rid of the expoetial l( y) l( Ce Ax ) Ax + l( C) ) Itroduce the followig chage of variables: Y l( y), X x, B l( C) Now we have: Y AX + B which is a LINEAR equatio The origial data poits i the x y plae get mapped ito the X Y plae. This is called data liearizatio. The data is trasformed as: ( x, y) ( X, Y) ( x, l( y) ) Now we use the method for solvig a first order liear curve fit for A ad B, where above Y l( y), ad X x Fially, we operate o B l( C) to solve C e B Ad we ow have the coefficiets for y Ce Ax Example: repeat previous example, add expoetial fit >> x-:.5:4; >> y 1.6*exp(1.3*x); >> % hopefully our curve fit will capture >> % C 1.6, ad A 1.3 >> %set up liearized data matrix >> sx11sum(x.^0); >> sx1sum(x.^1); >> sx1sx1; >> sxsum(x.^); >> sxy1sum(log(y).*x.^0); >> sxysum(log(y).*x.^1); >> AM [sx11 sx1; sx1 sx] >> BV [sxy1 ; sxy] >> SOL iv(am)*bv >> BSOL(1); >> ASOL(); >> Cexp(B); >> fxexpc*exp(a*x); >> plot(x,y,'rd',x,fx,'g',x,fx3,'b',x,fxexp, m ) >> leged('data','d','3rd', exp fit ) X X X B A Y XY Numerical Methods Lecture 6 - Curve Fittig Techiques page 118 of 118