Exercise 1 - Linear Least Squares

Transcription

1 Exercise 1 - Linear Least Squares Outline Course information Hints for Python, plotting, etc. Recap: Linear Least Squares Problem set: Q1-2D data Q2-3D data Q3 - pen and paper

2 Course information Final grade IFF (average homework grade - exam) < 1: Else: final grade = exam Homework: final grade = 0.8*exam + 0.2*homework the solved exercise by due date to: mad_fs18@sympa.ethz.ch use prefix [homework] in the subject of the all files should be in ONE archive named with your nethz login name and the exercise name. Example: for the exercise 1 and a person with nethz login name: janedoe the file name would be: janedoe_ex01.zip (.tar,.tar.gz are also OK) Questions about homework: for questions write to the same mailing list: mad_fs18@sympa.ethz.ch use prefix [QHW1] in the subject of the , 1 stands for 1 exercise set

3 Course information Homework self grading: use the relation ( ) * points/maxpoints for an indication of where your grade should approximately be placed use the points assigned to each exercise solutions to determine your point ratio points/maxpoints we will upload the solution on Wednesdays adhere to the code of honor! Submitting the Homework grade: at the end of the course the grades to: mad_fs18@sympa.ethz.ch use prefix [grading] in the subject of the

4 Course information Submitting the Homework grade: generate a table name the table file with your legi-number, i.e., xx-xxx-xxx.txt (where the x's correspond to the legi-id). The table should be stored in a simple text file (ASCII characters) (not rich text or anything similar This exercise has not been submitted Programing language? we encourage Python, C++ the solutions for the exercises will be in Python exam will involve pseudo code writing questions (pen-and-paper)

5 Hints for Python import numpy as np define array a = np.zeros(10) array of size 10 define matrix A = np.zeros((10,2)) 10x2 matrix matrix manipulation: np.dot(a,b), np.transpose(a), np.linalg.inv(a) import random random.uniform(a,b) produces uniform random number in range [a,b] random.seed(1234) for debugging

6 Hints for Visualization Plot data: gnuplot Python matplotlib Mac users - Plot2 Other users - Excel, Google Charts Any other methods are welcomed (No hand plotting please)

7 Exercise 1 - Linear Least Squares Learn about Linear Least Squares How to implement it How does it behave for noisy data, outliers and 3D data Implementation Option1: Use the numpy package Option2: Write the equations yourself Recommendation: DO BOTH! Note: only optional (advanced) pen-and-paper this time see Exercises in lecture notes on least squares for more pen-and-paper tasks

8 Least Squares Practical intro to Least Squares example : fitting N data points {x i,y i } N i=1 y = f(x) =a + b x + c x 2 A Generate matrix (following (1.4.1) of lecture notes) x 1 x x 2 x x N x 2 N The L2 norm squared of the error: a b c 3 5 p y 1 y 2... y N y Taking derivative we get (see lectures): A T Ap = A T y Now we have to solve the system for the coefficients p Solution provides the least square fit for the coefficients. Requires inversion of the LHS matrix!

9 Least Squares Practical intro to Least Squares example : fitting N data points {x i,y i } N i=1 y = f(x) = x +1.0 x 2 What happens when no noise is present in the data? (in data assimilation this is called committing the inverse crime!) Final set of parameters Asymptotic Standard Error ======================= ========================== a = 0.1 +/ e-08 (3.723e-05%) b = 1 +/ e-08 (1.734e-06%) c = 1 +/ e-09 (1.669e-07% ) We recover the parameters we created the data exactly!

10 Least Squares Practical intro to Least Squares example : fitting N data points {x i,y i } N i=1 y = f(x) = x +1.0 x 2 Now we start introducing different levels of noise. Assuming the noise is gaussian, we present two cases below Final set of parameters Asymptotic Standard Error ======================= ========================== a = / (403.4%) b = / (45.85%) c = / (4.751%) Final set of parameters Asymptotic Standard Error ======================= ========================== a = / (52.82%) b = / (50.41%) c = / (11.22%)

11 Least Squares Practical intro to Least Squares example : fitting N data points {x i,y i } N i=1 y = f(x) = x +1.0 x 2 Finally, assume we have an outlier (all other data points remain the same The least squares solution is sensitive to outliers! Final set of parameters Asymptotic Standard Error ======================= ========================= a = / (501.6%) b = / (86.51%) c = / (103.2% )

12 Question 1 - LSQ on 2D data Setup: V(I) = V0 + R I experimental data Vj, Ij evaluate V0, R with LSQ generic data with 3 cases: data on line, +noise, +outlier compare V0, R with the one used to generate data plot the data and the fit

13 Question 2 - LSQ on surface N data given in 3D: z(x, y) =A + Bx + Cy generate as in Q1 and add noise solve LSQ and compare computed a,b,c with original one E.x : Assume N=100 data from z = g(x, y) =1.0 x py z = g(x, y) =a x 2 + b py Final set of parameters Asymptotic Standard Error ================================================= a = / (8.198%) b = / (5.761%)

14 Question 3 (optional, advanced) Dependency of the LSQ fit on the noise Given: data: (x i,y i ),i=1,...,n, where y i = A 0 + B 0 x i, noisy data: (x i,yi ),i=1,...,n, where y i = y i + N (0, ), model to fit the data: f(x) =A + Bx method to find A and B: Least Squares Question: How does the quantity f(x) f (x) behaves as function of N? Here f and f are the models that correspond to the (x i,y i ) and the (x i,yi ) data, respectively.

15 Question 3 Dependency of the LSQ fit on the noise 1. Consider the model f(x) and f (x) with parameters A, B and A,B, that correspond to the data (x i,y i ) and (x i,yi ), repsectively. 2. Express the di erence f f in terms of di erences in the paramaters, A A and B B. 3. Express A A in terms of elements of the matrix H 1. Do the same for B B. NX NX A = H11 1 yi + H12 1 x i yi i=1 4. Find out how the elements of matrix H 1 behave as a function of N: First find out how the elements of H behaves as a function of N and the use the fact that H 1 H = I. 5. Use the Central Limit Theorem: (informally) If X i N (µ, ) for i =1,...,N then 1 NX X i N (µ, p ) N N i=1 i=1