CS 6140: Machine Learning Spring Final Exams. What we learned Final Exams 2/26/16

Transcription

1 CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Science Northeastern University Webpage: Assignment 2 will be released before next class (02/11). A couple more programming tasks For easier ones, you have to implement the classifiers and all preprocessing by yourselves For difficult ones, you can choose to use off-theshelf toolkits (need to acknowledge it) Also has analy@cal ques@ons. Due on 02/25 Logis@cs Feb 11 Topic: Deep Learning Feb 18 Topic: Deep Learning Feb 25 Topic: Structured Output Predic@on Assignment 2 due Mar 3 Topic: Mixture Models and Expecta@on Maximiza@on Mar 10 (No Class: Spring Break) Project progress report due Mar 17 Topic: Dimensionality Reduc@on Mar 24 Topic: Clustering Assignment 3 due Mar 31 Topic: Exam Final Exams Short descrip@on ques@ons E.g what s the difference between supervised learning and unsupervised learning? Analy@cal ques@ons E.g. given a bunch of training samples (each has 3 features and 1 label), what is the predic@on of a naïve bayes classifier for a certain test sample? E.g. design a decision tree Open ques@ons E.g given a machine learning problem, how do you model it? What features you will use? What models you will use? 2 hours Final Exams What we learned Bayesian Sta@s@cs Open book Computer without Internet is okay Calculator is okay Frequen@st Sta@s@cs Feature Selec@on 1

2 Bayesian Bayesian vs. Feature We oeen want to select a subset of the inputs that are most relevant for predic@ng the output, to get sparse models interpretability, speed, possibly beier predic@ve accuracy Filter methods Compute relevance of each feature to the label marginally Filter methods Compute relevance of each feature to the label marginally (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features Correla,on with Label buy 0.91 class 0.84 learning 0.65 the 0.10 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features Correla,on with Label buy 0.91 class 0.84 learning 0.65 the

3 Wrapper Methods Perform discrete search in model space Wrap search around standard model filng Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features Correla,on with Label buy 0.91 class 0.84 learning 0.65 the Start with {buy} as feature set (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features buy 0.91 class 0.84 learning 0.65 the Start with {buy} as feature set - Then add class Correla,on with Label Wrapper Methods Perform discrete search in model space Wrap search around standard model filng (spam) 1: you should buy the car! (spam) 2: buy one get one free! (regular) 3: the class takes place in the new building. (regular) 4: new learning algorithm is published. Features buy 0.91 class 0.84 learning 0.65 the Start with {buy} as feature set - Then add class - Then add learning Correla,on with Label Today s Outline Perceptron (and kernels) Support Vector Machines Perceptron Biology and Learning Idea 1: Good behavior should be rewarded, bad behavior punished (or not awarded). Raising a dog. Idea 2: Correlated events should be combined. Babies learn language. [Some of the slides are borrowed from Alex Smola s tutorial] 3

4 Biology and Learning Neurons Idea 1: Good behavior should be rewarded, bad behavior punished (or not awarded). Raising a dog. Idea 2: Correlated events should be combined. Babies learn language. Training Mechanisms Behavioral modifica@on of individuals (learning) Feeding the dog, then the dog learns to stand and sit. Hard-coded behavior in the genes (ins@nct) The wrongly coded animal dies. Perceptron Perceptron Weighted combina@on The output of the neuron is a linear combina@on of the inputs Decision Func@on At the end the results are combined into Perceptron Learning Goal: Linear Separa@on An abstract model is to assume that Where w is the weight, x is the feature vectors b is the bias, Biological Interpreta@on The weights w i correspond to the synap@c weights, the mul@plica@on corresponds to the processing of inputs via the synapses, and the summa@on is the combina@on of signals in the cell body (soma). 4

5 Perceptron Algorithm Perceptron Algorithm Nothing happens if we classify (x i, y i ) correctly If we see incorrectly classified observa@on we update w and b Posi@ve reinforcement of observa@ons Perceptron Algorithm About the solu@on Weight vector is linear combina@on of observa@ons x i : Perceptron Algorithm About the solu@on Classifica@on can be wriien in terms of dot products: Pseudocode 5

6 The XOR Problem The XOR Problem Perceptrons cannot learn such linearly inseparable Problem Linear are oeen too simple to provide good Problem Linear are oeen too simple to provide good Idea: Map to a higher dimensional feature space via Replace every by in the perceptron algorithm. 6

7 Perceptron on Features Problems with Features Problems with Features Need to be an expert in the domain (e.g. Chinese characters). Can be expensive to compute. Dimension = 1 Dimension = 2 Polynomial Features Dimension = d (skip proof) Kernels Some choices of kernel func@ons Defini@on A kernel func@on is a symmetric func@on in its arguments for which the following property holds 7

8 Linear Kernel Laplacian Kernel Gaussian Kernel Kernel Perceptron Linear Separators Which of these linear separators is [Some of the slides are borrowed from David Sontag s lecture] 8

9 Support Vector Machine (SVM) Support Vector Machine (SVM) SVMs (Vapnik, 1990 s) choose the linear separator with the largest margin. Hyperplane Reasons: guarantee (skip here) In tasks: SVM became famous when, using images as input, it gave accuracy comparable to neural-network with hand-designed features in a handwri@ng recogni@on task. Support Vector Machine (SVM) Planes How to find the hyperplane? Hyperplane A plane can be specified as the set of points given by Planes Normal to a plane A plane can be specified as the set of points given by Normal Vector: decide the direc@on of the plane Length of the vector 9

10 Scale invariance What is the distance γ? Support vector machines Final result: can maximize margin by minimizing What if the data is not linearly separable? What if the data is not linearly separable? More features 10

11 What if the data is not linearly separable? What if the data is not linearly separable? Old objec,ve Old objec,ve New objec,ve New objec,ve Jointly minimize w.w and number of training mistakes! What if the data is not linearly separable? Allowing for slack: Soe margin SVM Allowing for slack: Soe margin SVM A Demonstra@on of SVMs hips:// 11

12 Popular Tools for SVMs LIBSVM (c++) hips:// SVM light (c) hip://svmlight.joachims.org/ Scikit-learn (python) hip://scikit-learn.org/ Popular Tools for SVMs Torch (LuaJIT) hip://torch.ch/ Spider (Matlab) Weka (Java) How do we the programming Kernels A kernel is a symmetric func@on in its arguments for which the following property holds How do we op@mize the objec@ve? Constrained op@miza@on Quadra@c programming No place to apply the kernel trick 12

13 Constrained Constrained Lagrange Dual variables Lagrange How do we solve with constraints? Lagrange Lagrange Dual variables Back to SVM (hard margin) 13

14 Dual SVM Dual SVM Slater s condi@on from convex op@miza@on guarantees that these two op@miza@on problems are equivalent! (skip proof) Dual SVM deriva@on Dual SVM deriva@on To get w and b Classifica@on rule using dual solu@on Using dual solu@on dot product 14

15 Dual for the non-separable case Dual for the non-separable case How to interpret dual form Back to the What if the data is not linearly separable? For example: Higher order polynomials Dual only depends on dot-products of the features! 15

16 Dual only depends on dot-products of the features! Kernels A kernel is a symmetric func@on in its arguments for which the following property holds Kernel Trick Soe margin SVM with kernel Common kernels for SVM Gaussian RBF kernel Aka Gaussian Radial basis func,on (RBF) kernel 16

17 SVM Python pseudocode Overfilng Huge feature space with kernels: should we worry about overfilng? SVM seeks a solu@on with large margin Good theore@cal guarantee But everything overfits some@mes Overfilng Dual for the non-separable case Huge feature space with kernels: should we worry about overfilng? SVM objec@ve seeks a solu@on with large margin Good theore@cal guarantee But everything overfits some@mes Can control by: Selng C Choosing a beier kernel Varying parameters of the kernels Linear SVM C = 50 Linear SVM C = 50 17

18 Linear SVM C = 50 Linear SVM C = 50 Insights Changing C For clean data C doesn t maier much. For noisy data, large C leads to narrow margin (SVM tries to do a good job at separa@ng, even though it isn t possible) Insights Changing C For clean data C doesn t maier much. For noisy data, large C leads to narrow margin (SVM tries to do a good job at separa@ng, even though it isn t possible) Noisy data Clean data has few support vectors Noisy data leads to data in the margins More support vectors for noisy data Dual for the non-separable case Gaussian RBF Kernel with C =

19 Gaussian RBF Kernel with C = 0.2 Gaussian RBF Kernel with C = 0.4 Gaussian RBF Kernel with C = 0.8 Gaussian RBF Kernel with C = 1.6 Gaussian RBF Kernel with C = 3.2 Gaussian RBF Kernel with C =

20 Gaussian RBF Kernel with C = 12.8 Insights Changing C For clean data C doesn t maier much. For noisy data, large C leads to more complicated margin (SVM tries to do a good job at separa@ng, even though it isn t possible) OverfiNng for large C Noisy data Clean data has few support vectors Noisy data leads to data in the margins More support vectors for noisy data Common kernels for SVM Gaussian RBF with different Aka Gaussian Radial basis func,on (RBF) kernel Gaussian RBF with different Gaussian RBF with different 20

21 Gaussian RBF with different Insights Changing For clean data, doesn t maier much. For noisy data, small leads to more complicated margin (SVM tries to do a good job at separa@ng, even though it isn t possible) Lots of overfilng for small Noisy data Clean data has few support vectors Noisy data leads to data in the margins More support vectors for noisy data Dual for the non-separable case Homework (part of assignment 2) Study the Sequen@al Minimal Op@miza@on algorithm and implement an SVM classifier by yourself References hip://cs229.stanford.edu/materials/smo.pdf Fast Training of Support Vector Machines using Sequen@al Minimal Op@miza@on hip://research.microsoe.com/pubs/68391/smobook.pdf What we learned today Perceptron (and kernels) Homework Read Murphy CH , Support Vector Machines 21