Pattern Recognition Systems Lab 1 Least Mean Squares

Transcription

1 Patter Recogitio Systems Lab 1 Least Mea Squares 1. Objectives This laboratory work itroduces the OpeCV-based framework used throughout the course. I this assigmet a lie is fitted to a set of poits usig the Least Mea Squares method (liear regressio). Both the iterative solutio (gradiet descet) ad the closed form are preseted.. Theoretical Backgroud You are give a set of data poits of the form (x i, y i ) where i = {1,,, }. Your task is to fid the lie equatio that best fits the data. We will tackle this with liear regressio. The set of poits is cosidered the traiig set ad your task is to lear a model that best fits the data. Model 1 Whe tryig to fit a model to data the first step is to establish the form of the model. Liear regressio adopts a model that is liear i terms of the parameters (icludig a costat term). For the first part let us adopt a simple model that expresses y i terms of x: f(x) = θ 0 + θ 1 x This is the usual way this problem is solved. However, this represetatio caot treat vertical lies. Noetheless, it provides a good itroductio to the method. A vector ca be formed that cotais all the parameters of the model θ = [θ 0, θ 1 ] T (the itercept term ad the liear coefficiet for x). The least squares approach for determiig the parameters states that the best fit to the model will be obtaied whe the quadratic cost fuctio is at its miimum: J(θ) = 1 (f(x i) y i ) Why quadratic? This ca be motivated by makig the assumptio that the error i the data follows a ormal distributio. See referece. Note that, this miimizes the error oly alog the y-axis ad ot the distaces of the poits from the lie. I order to miimize the cost fuctio we take its partial derivatives with respect to each parameter. J(θ) = (f(x θ i ) y i ) 0 J(θ) = (f(x θ i ) y i ) 1 The cost fuctio attais its miimum whe the gradiet becomes zero. Oe geeral approach to fid the miimum is to use gradiet descet. Sice the gradiet x i

2 shows the directio i which the fuctio icreases the most, if we take steps ito the opposite directio we decrease the value of the fuctio. By cotrollig the size of the step we ca arrive at a local miimum of the fuctio. Sice the objective fuctio i this case is quadratic, the fuctio has a sigle miimum ad so gradiet descet will fid it. To apply gradiet descet start from a iitial o-zero guess θ chose radomly. Fid the gradiet i that poit: J(θ) = [ J(θ), J(θ) T ] θ 0 θ 1 The apply the followig update rule util covergece: θ ew = θ α J(θ), where α is the learig rate ad it is chose appropriately to esure the cost fuctio decreases at each iteratio. Whe the chage betwee the parameter values is small eough, the algorithm stops. The gradiet descet approach is appropriate whe the roots of the gradiet are hard to fid. But i this case (lie fittig) a explicit solutio ca be foud. By settig the gradiet compoets equal to 0 we obtai the followig system: θ 0 + θ 1 x i = y i θ 0 x i + θ 1 x i = x i y i { which is a liear system with two equatios ad two ukows ad ca be solved directly to obtai the values for θ: θ 1 = x iy i x i y i x i ( x i ) { θ 0 = 1 ( y i θ 1 x i ) Normal Equatio - A closed form solutio i vector form I geeral the miimizatio problem for this model ca be writte i matrix form: Aθ b = (Aθ b) T (Aθ b) This case arises for model 1, where A is a x matrix with each row cotaiig the values of xi followed by 1 ad b is a x1 colum vector cotaiig the values for y. the the closed form solutio is give directly by: θ opt = (A T A) 1 A T b For more details ad derivatio cosult [1].

3 Model I order to address the issue of vertical lies we itroduce aother model that is capable of dealig with every possible case. Cosider the followig parametrizatio of a lie i D: xcos(β) + ysi(β) = ρ This describes a lie with uit ormal vector [cos(β), si(β)] which is at a distace of ρ from the origi. The cost fuctio we wish to miimize i this case is the sum of squared distaces of each poit from our curret lie. This is give by: J(β, ρ) = 1 (x icos(β) + y i si(β) ρ) Note, that this is the actual error term we wat to miimize ad that i the previous sectio we have cosidered oly the error alog the y-axis, which is icorrect. The compoets of the gradiet eed to be evaluated to perform gradiet descet: J β = (x icos(β) + y i si(β) ρ)( x i si(β) + y i cos(β)) J ρ = (x icos(β) + y i si(β) ρ) A closed form solutio ca also be obtaied, although ot as easily as i the previous case. The solutio is give as: Model 3 β = 1 ata ( x iy i x i y i, (y i x i ) + 1 ( x i) ρ = 1 (cos(β) x i + si(β) y i ) 1 ( y i) ) There is a third possibility for the form of the model. If we adopt a parametrizatio with 3 free parameters for a lie: ax + by + c = 0 The cost fuctio ca be defied as: J(a, b, c) = 1 (ax i + by i + c) which ca be writte i matrix form as the squared orm of a vector: J(a, b, c) = (Aθ) T Aθ where A is a matrix with x3 elemets, each row cotaiig (x i, y i, 1) ad θ = [a, b, c] T is the parameter vector (colum vector with 3 elemets). We eed to miimize this orm to obtai the parameter values. Workig with this model which has 3 parameters has two importat cosequeces: all possible lies ca be modeled ad we will have a family of values which give the same lie. To solve the secod issue we will seek the parameter vector with uit orm. Fidig the ull-space of

4 a matrix A with uit orm is a classical problem ad it is solved with Sigular Value Decompositio. We have: A = USV where U ad V are orthogoal matrices ad S cotais values oly o the mai diagoal (called sigular values). From here the optimal value for the parameter vector will correspod to the last colum of the matrix V (which is the eigevector of A T A correspodig to the smallest eigevalue). The iterested reader ca cosult [] for a demostratio ad further details. 3. Practical Backgroud Readig from a text file: FILE* f = fope( fileame.txt, r ); float x,y; fscaf(f, %f%f, &x,&y); fclose(f); Creatig a color image: Mat img(height, width, CV_8UC3); //8bit usiged 3 chael Accessig the pixel at positio row i ad colum j: Vec3b pixel = img.at<vec3b>(i,j); //byte vector with 3 elemets Modifyig the pixel at positio row i ad colum j: img.at<vec3b>(i,j)[0] = 55; //blue img.at<vec3b>(i,j)[1] = 55; //gree img.at<vec3b>(i,j)[] = 55; //red Draw a lie betwee two poits: lie(img, Poit(x1, y1), Poit(x, y), Scalar(B,G,R)); Viewig the image: imshow( title, img); waitkey();

5 4. Practical Work 1. Read the iput data from the give file. The first lie cotais the umber of poits. Each lie afterwards cotais a (x,y) pair.. Plot the poits o a white 500x500 backgroud image. For better visibility draw circles, crosses or squares cetered at the poits. Be careful to cosider how the coordiate system i the image is defied. Some poits may have egative coordiates, either do t plot them or shift the whole graph. The method is ot affected by poits havig egative coordiates. 3. Optioally, use model 1 ad gradiet descet to fit a lie to the poits. Visualize the lie at each k-th step. Output ad visualize the value of the cost fuctio at each step. Choose the learig rate so that the cost fuctio is decreasig. 4. Use model 1 ad the closed form to calculate the parameters. Visualize both the fial lie from the previous step ad this oe ad compare the parameter values. 5. Optioally, use model ad gradiet descet to fit a lie to the poits. Visualize the lie at each k-th step. Output ad visualize the value of the cost fuctio at each step. Choose the learig rate so that the cost fuctio is decreasig. 6. Use model ad the closed form to calculate the parameters. Compare the results with the previous step. 7. Optioally, draw the errors as perpediculars segmets from the poits to the lie. 8. Optioally, fid the parameters with model 3 ad SVD. 5. Refereces [1] Staford Machie Learig - course otes 1 [] Tomas Svoboda - Least-squares solutio of Homogeeous Equatios - aied_lsq.pdf