CSC 4510 Machine Learning

Transcription

1 5: Mul'variate Regression CSC 4510 Machine Learning Dr. Mary Angela Papalaskari Department of CompuBng Sciences Villanova Course website: The slides in this presentabon are adapted from: Andrew Ng s ML course hnp:// class.org/ 1

2 Regression topics so far IntroducBon to linear regression IntuiBon least squares approximabon IntuiBon gradient descent algorithm Hands on: Simple example using excel How to apply gradient descent to minimize the cost funcbon for regression linear algebra refresher 2

3 What s next? MulBvariate regression Gradient descent revisited Feature scaling and normalizabon SelecBng a good value for α Non linear regression Solving for analybcally (Normal EquaBon) Using Octave to solve regression problems 3

4 What s next? We are not in univariate regression anymore: Size (feet 2 ) Number of bedrooms Number of floors Age of home (years) Price ($1000)

5 Mul'ple features (variables). Size (feet 2 ) Number of bedrooms Number of floors Age of home (years) Price ($1000) Andrew Ng

6 Mul'ple features (variables). Size (feet 2 ) Number of bedrooms Number of floors Age of home (years) Price ($1000) NotaBon: = number of features = input (features) of training example. = value of feature in training example. 6 Andrew Ng

7 Mul'ple features (variables). Size (feet 2 ) Price ($1000) Andrew Ng

8 Hypothesis: Previously: Now: For convenience of notabon, define. Mul$variate linear regression 8

9 Hypothesis: Parameters: Cost func'on: Gradient descent: Repeat (simultaneously update for every ) 9

10 Gradient Descent Previously (n=1): Repeat (simultaneously update ) 10

11 Gradient Descent Previously (n=1): Repeat New algorithm : Repeat (simultaneously update ) for (simultaneously update ) 11

12 Gradient Descent Previously (n=1): Repeat New algorithm : Repeat (simultaneously update ) for (simultaneously update ) 12

13 Feature Scaling Idea: Make sure features are on a similar scale. E.g. = size ( feet 2 ) = number of bedrooms (1 5) size (feet 2 ) Get every feature into range number of bedrooms 13

14 Feature Scaling Idea: Make sure features are on a similar scale. E.g. = size ( feet 2 ) = number of bedrooms (1 5) Mean normaliza'on Replace with to make features have approximately zero mean (Do not apply to ). E.g. 14

15 Gradient descent Debugging : How to make sure gradient descent is working correctly. How to choose learning rate. 15

16 Making sure gradient descent is working correctly. For sufficiently small, should decrease on every iterabon. But if is too small, gradient descent can be slow to converge No. of iterabons Declare convergence if decreases by less than in one iterabon? 16

17 Summary: Choosing If is too small: slow convergence. If is too large: may not decrease on every iterabon; may not converge. To choose, try 17

18 Housing prices predic'on 18 Andrew Ng

19 Polynomial regression Price (y) Size (x) 19 Andrew Ng

20 Choice of features Price (y) Size (x) 20 Andrew Ng

21 Gradient Descent Normal equabon: Method to solve for analybcally. 21 Andrew Ng

22 IntuiBon: If 1D (for every ) Solve for 22 Andrew Ng

23 Examples: Size (feet 2 ) Number of bedrooms Number of floors Age of home (years) Price ($1000) Andrew Ng

24 Examples: Size (feet 2 ) Number of bedrooms Number of floors Age of home (years) Price ($1000) Andrew Ng

25 examples ; features. E.g. If 25 Andrew Ng

26 is inverse of matrix. Octave: pinv(x *X)*X *y 26 Andrew Ng

27 training examples, Gradient Descent features. Normal EquaBon Need to choose. Needs many iterabons. Works well even when is large. No need to choose. Don t need to iterate. Need to compute Slow if is very large. 27 Andrew Ng

28 Notes on Supervised learning and Regression hnp://see.stanford.edu/materials/aimlcs229/cs229 notes1.pdf Octave hnp:// Wiki: hnp:// documentabon: hnp:// 28

29 Exercise For next class: 1. Download and install Octave (AlternaBve: if you have MATLAB, you can use it instead.) 2. Verify that it is working by typing in an Octave command window: x = [ ] y = [ ] plot(x,y) This example defines two vectors, x y and should display a plot showing a straight line (the line y=2x). If you get an error at this point, it may be that gnuplot is not installed or cannot access your display. If you are unable to get this to work, you can sbll do the rest of this exercise, because it does not involve any plorng (just restart Octave). You might refer to the Octave wiki for installabon help but if you are stuck, you can get some help troubleshoobng this on Friday anernoon 3 4pm in the sonware engineering lab (mendel 159). 3. Create a few matrices and vectors, eg: A = [1 2; 3 4; 5 6] V = [ ] 4. Try some of the elementary matrix and vector operabons from our linear algebra slides (adding, mulbplying between matrices, vectors and scalars) 5. Print out a log of your session 29