ES-2 Lecture: Fitting models to data

Transcription

1 ES-2 Lecture: Fitting models to data

2 Outline Motivation: why fit models to data? Special case (exact solution): # unknowns in model =# datapoints Typical case (approximate solution): # unknowns in model < # datapoints Defining what best fit means Least squares criterion One way to solve brute-force search Some alternatives to least-squares 2

3 Big picture: turning data into features (and ultimately, information) Problem statement: given a set of noisy measurements, create features (lines, curves, coefficients in an equation) by fitting data to a model Why this is helpful,#1: Using features, information can be stored / transmitted more efficiently than if we stored raw measurements 3 Example: underwater robot is mapping the seabed, and sends back data with a low-speed acoustic modem Example: you want to watch a movie on your phone

4 Big picture: turning data into features (and ultimately, information) Problem statement: given a set of noisy measurements, create features (lines, curves, coefficients in an equation) by fitting data to a model Why this is helpful,#2: We may have a model we trust, but need to find some coefficients. 4 Example: equations of drag force are well known, but drag coefficients need to be estimated from data (

5 Big picture: turning data into features (and ultimately, information) Problem statement: given a set of noisy measurements, create features (lines, curves, coefficients in an equation) by fitting data to a model Why this is helpful,#3: Use models to understand or interpret our data -Does cancer risk grow linearly with # cigs/day, or is another model better? -If linear, what s the slope how fast does risk increase as # cigarettes increases? 5

6 Outline Motivation: why fit models to data? Special case (exact solution): # unknowns in model = # datapoints Typical case (approximate solution): # unknowns in model < # datapoints Defining what best fit means Least squares criterion One way to solve brute-force search Some alternatives to least-squares 6

7 Fitting 2 points to a line (from middle school!) Model: y = m x + b 7 (0.7, 3.1) (3,1)

8 A more flexible-approach: matrices Model: y = m x + b 8 (0.7, 3.1) (3,1)

9 Warning: making a slight detour This week, we re talking about line fits 9 The next few slides will talk about polynomial fits, as a way to reinforce the idea from the last slide

10 Pair work: Fit a quadratic to 3 points: Model rocket velocity vs time Time Speed Fit data to: v(t) = a t 2 + b t + c 10 Write down a matrix version of this problem (same approach as line fit)

11 Clicker: what is the correct matrix equation? Time Speed Fit data to: v(t) = a t 2 + b t + c A) B) C) a b c a b c = = = a b c 11

12 A clicker question for planning upcoming lectures Should I review what standard deviation is, before referring to it in lecture? 12 A. No, thanks very familiar with it B. Yes, please it s been awhile C. Yes, please never covered this

13 Something to notice about fits Solution is exact when # unknowns = # points - If you draw a straight line through 2 points, it exactly goes through those points - Same thing for a quadratic with 3 points 13 END OF DETOUR Back to line fits

14 Outline Motivation: why fit models to data? Special case (exact solution): # unknowns in model = # datapoints Typical case (approximate solution): # unknowns in model < # datapoints 14

15 Think about fitting a straight line to this data (car mpg vs. weight) Weight Miles/gallon Observations: 1. More data is better avoids errors 2. There is no line that perfectly matches this data

16 Outline Motivation: why fit models to data? Special case (exact solution): # unknowns in model = # datapoints Typical case (approximate solution): # unknowns in model < # datapoints Defining what best fit means Least squares criterion One way to solve brute-force search Some alternatives to least-squares 16

17 Definition: Residual error The residual is what s 17 left over after we subtract the fit from the data

18 Least-squares approach 18 Define the best fit to be that which gives the smallest sum of squared errors hence, least squares For a line, this means we want to pick slope m, intercept b to minimize: S r = n å i= 1 e 2 i = n å( y - - ) i b mxi i= 1 2

19 Least-squares approach 19 Least-squares is a Big Idea. It shows up all over the place data processing, stats, etc. We ll spend this week and next on LS how to find the best-fit solution how to extend it to models besides straight lines It s the basic tool you ll use in HW8 HW10

20 Clicker Which fit is best, under least-squares criterion? Data Fit1 Fit2 S r = n å i= 1 e 2 i = n å( y - - ) i b mxi i= A) B) C) D) Fit1 Fit2 Both equally good Not sure

21 Solution approach How do we pick slope, intercept to minimize S r? 21 S r = n å i= 1 e 2 i = n å( y - - ) i b mxi i= 1 2 Most commonly used methods (efficient for big datasets) will be covered on Thursday Simple: exhaustive or brute-force search Pick a range of likely values for b, m Try out each possible fit Tabulate Sr, keep the best answer

22 Outline Motivation: why fit models to data? Special case (exact solution): # unknowns in model = # datapoints Typical case (approximate solution): # unknowns in model < # datapoints Defining what best fit means Least squares criterion One way to solve brute-force search Some alternatives to least-squares and problems with LS 22

23 Problem: Least squares is sensitive to outliers Fit, no outliers Fit with outlier 23 An outlier is a data point that doesn t match the usual pattern Often, it s from bad data (an error in measurement)

24 Problem: Least squares is sensitive to outliers Fit ignoring outlier Fit including outlier Fit with outlier 24 Point # err err^2 err err^ Sr Because we square the error, outlier has big effect

25 Possible fix #1: Use a different definition of best fit Try minimizing sum of absolute values (not ^2!) n å S abs = e i i= 1 Least absolute value fit Point # err err Leastsquares fit sum See outlierexample.m on Trunk

26 Weight Possible fix #2: random sampling of the data Miles/gallon 26 Randomly pick 2 points, and fit a line to them Store the computed slope and intercept Repeat many times Estimated slope = median of all stored slopes; same for intercept That works and helps ignore outliers!