Machine Learning using Matlab. Lecture 3 Logistic regression and regularization

Transcription

1 Machine Learning using Matlab Lecture 3 Logistic regression and regularization

2 Presentation Date (correction)

3 Project proposals 13 submissions, 21 students in total. Presentation may reduce from 30 mins to 20 mins (15 mins talk, 5 mins questions) in terms of number of projects. Topic including: Computer vision: Face detection, Car detection, Optical character recognition, Predicting political ideology using face detection Natural language processing: Understanding natural text input, Replicate text with a specific style, Identifying tags for Machine Learning related scientific paper, Client side classification Others: Botnets detection, Predict likelihood of repayment of loans, Predict probabilities for outcome of the football game, Predict whether a person makes over $50k a year, phishing websites detection

4 100% Confidence (In my opinion) Face/car detection Optical Character Recognition (OCR) In lab session we will implement models to recognize handwritten digits May extend to recognize alphabet or number in image? Client side classification Predict whether a person makes over $50k a year Identifying tags for Machine Learning related scientific paper Maybe too small for three members?

5 70% Confidence Replicate text with a specific style How to measure your performance? Botnets detection Predict likelihood of repayment of loans Predict probabilities for outcome of the football game Phishing websites detection Flight data machine learning Board state evaluation for online card game Hearthstone What features?

6 50% Confidence Understanding natural text input How to apply ML model to parse text input? Predicting political ideology using face detection Difficult to detect: hair style, glasses, expression and how to dress? How many data can you collect?

7 Outline Gradient descent for logistic regression Advanced optimization algorithms Polynomial model Options on addressing overfitting Regularized linear regression and logistic regression Multiclass classification (one-vs-all)

8 Logistic regression Classification model Start from binary: E.g., tumor: malignant (1)/benign(0)

9 Logistic regression Hypothesis: Parameter: Cost Function: Goal:

10 Gradient descent Given cost function, our objective is Repeat{ (simultaneously update all ) } Identical to linear regression except the hypothesis is different!

11 Advanced optimization Optimization algorithm: you have cost function, your objective is to minimize it, i.e., Solution: given parameter, we have code that can compute: Cost function Partial derivative Gradient descent Repeat{ }

12 Advanced optimization Other advanced algorithms: Conjugate gradient BFGS L-BFGS... Advantages No need to pick learning rate manually Often faster than gradient descent Disadvantages: More complex to implement Implementation is out of scope in the course, but you can still use them in Matlab!

13 Implementation in Matlab 1. Write your own cost function: function [jval, gradient] = costfunction(theta) jval = [...code to compute J(theta)...]; gradient = [...code to compute derivative of J(theta)...]; end 2. Use the function fminunc(): options = optimset('gradobj', 'on', 'MaxIter', 100); [opttheta, functionval, exitflag] = fminunc(@costfunction, initialtheta, options); 3. Example of linear regression

14 Nonlinear What is nonlinear? The change of the output is not proportional to the change of the input. Nonlinear function: Polynomial, Gaussian,...

15 Features and polynomial regression Go back to linear regression with one variable: To improve model, we can add more artificial features: The hypothesis becomes a polynomial function, therefore, we call it polynomial regression instead Question: the more parameters, the better?

16 Example: linear/polynomial regression Underfitting, high bias Just right Overfitting, high variance

17 Underfitting and Overfitting Underfitting (high bias) A phenomenon that the ML model maps poorly to the trend of the data Occurs when your hypothesis is too simple or use too few features Overfitting (high variance) A phenomenon that the ML Model fit the training set very well, but fail to generalize to test set. In other words, an overfitting model performs well in training set, but quit bad in test set. Occurs when your hypothesis is excessively complex, e.g, too many parameters or too many features

18 Address Underfitting and Overfitting Underfitting Overfitting Add more features Use a more complex hypothesis Reduce number of features Manually select which features to keep (still questionable) Model selection algorithm (later in this course) Regularization Keep all the features, but reduce values of parameters Regularization works well when we have lots of slightly useful features

19 Regularization Small values for parameters Simpler hypothesis, thus less prone to overfitting Regularization for linear regression Note j start from 1 not 0! Here lambda is the regularization parameter, control the tradeoff between two different goals: fitting the training set well and keeping the parameter small What would happen if lambda is too small? What would happen if lambda is too large? How to tune it? (Later in this course)

20 Regularized linear regression Gradient descent: Repeat until convergence{ }

21 Regularized linear regression Gradient descent: Repeat until convergence{ } Additional term here

22 Regularized linear regression Normal equation

23 Regularized logistic regression No regularization: With regularization: Regularization term

24 Regularized logistic regression Gradient descent: Repeat until convergence{ }

25 Advanced optimization for regularized logistic regression function [jval, gradient] = costfunction(theta, x, y) jval = [...code to compute J(theta)...]; gradient = [...code to compute derivative of J(theta)...]; end Feed costfunction and data into fminunc()

26 Multiclass Classification In the previous work we assume the labels in logistic regression were binary: In multiclass classification, we expand our definition so that: Example: face recognition: attendance system object categorization: human, car, face, Weather: sunny, cloudy, rain, snow It doesn t matter what the index starts 0 or 1!

27 Binary vs. Multiclass x 2 x 2 x 1 x 1

28 Multiclass classification Two solutions: One-vs-all (one-vs-rest): we can divide multiclass classification problem to K binary classifier Softmax regression (multinomial logistic regression): train a multiclass classifier

29 One-vs-all (one-vs-rest) x 2 Class 1: Class 2: Class 3: x 1

30 One-vs-all (one-vs-rest) x 2 x 1

31 One-vs-all (one-vs-rest) x 2 x 1

32 One-vs-all (one-vs-rest) x 2 x 1

33 One-vs-all (one-vs-rest) Summary: Train a logistic regression classifier for each class to predict the probability To make a prediction on new data, pick the class that has the maximum output Q: what if a new data doesn t belong to any class?

34 Any question?