5 Learning hypothesis classes (16 points)

Transcription

1 5 Learning hypothesis classes (16 points) Consider a classification problem with two real valued inputs. For each of the following algorithms, specify all of the separators below that it could have generated and explain why. If it could not have generated any of the separators, explain why not. A B C D E F 1. 1 nearest neighbor 2. decision trees on real valued inputs 13

2 3. standard perceptron algorithm 4. SVM with linear kernel 5. SVM with Gaussian kernel σ ( = 0.25) 6. SVM with Gaussian kernel σ ( = 1) 7. neural network with no hidden units and one sigmoidal output unit, run until convergence of training error 8. neural network with 4 hidden units and one sigmoidal output unit, run until convergence of training error 14

3 6 Perceptron (8 points) The following table shows a data set and the number of times each point is misclassified during a run of the perceptron algorithm, starting with zero weights. What is the equation of the separating line found by the algorithm, as a function x 1, xof 2, and x 3? Assume that the learning rate is 1 and the initial weights are all zero. x 1 x 2 x 3 y times misclassified

4 7 SVMs (12 points) Assume that we are using an SVM with polynomial a kernel of degree. 2You are given the following support vectors: x 1 x 2 y The α values for each of these support vectors are equal to What is the value of b? Explain your approach to getting the answer. 2. What value does this SVM compute for the input point 3) (1, 16

5 8 Neural networks (18 points) A physician wants to use a neural network to predict whether patients have a disease, based on the results of a battery of tests. He has assigned a cost c 01 to of false positives (generating an output of 1 when it ought to have been 0), and a ccost 10 to of generating an output of 0 when it ought to have been 1. The cost of a correct answer is 0. The neural network is just a single sigmoid unit, which computes the following function: g( x) = s(w x) with s(z) being the usual sigmoid function. 1. Give an error function for the whole training set, E(w) that implements this error metric, for example, for a training set of 20 cases, if the network predicts 1 for 5 cases that should have been 0, predicts 0 for 3 cases that should have been 1 and predicts another 12 correctly, the value of the error function should cbe: c Would this be an appropriate error criterion to use for a neural network? Why or why not? 17

6 3. Consider the following error function for the whole training set: E(w) = c x i ) 2 x i ) 2 10 (g( i) y + c 01 (g( i) y {i y i =1} {i y i =0} Describe, in English that is not simply a direct paraphrase of the mathematics, what it measures. 4. What is the gradient of this E with respect to w? 5. Give a complete algorithm for training a single sigmoid unit using this error function. 18

7 4 Machine Learning Continuous Features (20 points) In all the parts of this problem we will be dealing with one dimensional data, that is, a set of points (x i ) with only one feature (called simply x). The points are in two classes given by the value of y i. We will show you the points on the x axis, labeled by their class values; we also give you a table of values. 4.1 Nearest Neighbors i x i y i In the figure below, draw the output of a 1 Nearest Neighbor classifier over the range indicated in the figure. 11

8 2. In the figure below, draw the output of a 5 Nearest Neighbor classifier over the range indicated in the figure. 12

9 4.2 Decision Trees Answer this problem using the same data as in the Nearest Neighbor problem above. Which of the following three tests would be chosen as the top node in a decision tree? Justify your answer. x 1.5 x 5 x 10.5 You may find this table useful. x y (x/y)*lg(x/y) x y (x/y)*lg(x/y)

10 4.3 Neural Nets Assume that each of the units of a neural net uses one of the the following output functions of the total activation (instead of the usual sigmoid s(z)) Linear: This just outputs the total activation: l(z) = z Non Linear: This looks like a linearized form of the usual sigmoid funtion: f(z) = 0 if z < 1 f(z) = 1 if z > 1 f(z) = 0.5(z + 1) otherwise 14

11 Consider the following output from a neural net made up of units of the types described above. 1. Can this output be produced using only linear units? Explain. 2. Construct the simplest neural net out of these two type of units that would have the output shown above. When possible, use weights that have magnitude of 1. Label each unit as either Linear or Non Linear. 15

12 4.4 SVM What are the values for the α i and the offset b that would give the maximal margin linear classifier for the two data points shown below? You should be able to find the answer without deriving it from the dual Lagrangian. i x i y i

13 5 Machine Learning (20 points) Grady Ent decides to train a single sigmoid unit using the following error function: 1 E(w) = (y(x i, w) y i ) β w 2 2 j i j i where y(x i 1, w) = s(x w) with s(z) = 1+ e z being our usual sigmoid function. E 1. Write an expression for wj. Your answer should not involve derivatives. 2. What update should be made to weight w j given a single training example < x, y >. Your answer should not involve derivatives. 17

14 3. Here are two graphs of the output of the sigmoid unit as a function of a single feature x. The unit has a weight for x and an offset. The two graphs are made using different values of the magnitude of the weight vector ( w 2 = j w 2 j ). Which of the graphs is produced by the larger w 2? Explain. 4. Why might penalizing large w 2, as we could do above by choosing a positive β, be desirable? 5. How might Grady select a good value for β for a particular classification problem? 18

15 6 Pruning Trees (20 points) Following are some different strategies for pruning decision trees. We assume that we grow the decision tree until there is one or a small number of elements in each leaf. Then, we prune by deleting individual leaves of the tree until the score of the tree starts to get worse. The question is how to score each possible pruning of the tree. For each possible definition of the score below, explain whether or not it would be a good idea and give a reason why or why not. 1. The score is the percentage correct of the tree on the training set. 2. The score is the percentage correct of the tree on a separate validation set. 3. The score is the percentage correct of the tree, computed using cross validation. 19

16 4. The score is the percentage correct of the tree, computed on the training set, minus a constant C times the number of nodes in the tree. C is chosen in advance by running this algorithm (grow a large tree then prune in order to maximize percent correct minus C times number of nodes) for many different values of C, and choosing the value of C that minimizes training set error. 5. The score is the percentage correct of the tree, computed on the training set, minus a constant C times the number of nodes in the tree. C is chosen in advance by running cross validation trials of this algorithm (grow a large tree then prune in order to maximize percent correct minus C times number of nodes) for many different values of C, and choosing the value of C that minimizes cross validation error. 20

17 Problem 4: Learning (25 points) Part A: (5 Points) Since the cost of using a nearest neighbor classifier grows with the size of the training set, sometimes one tries to eliminate redundant points from the training set. These are points whose removal does not affect the behavior of the classifier for any possible new point. 1. In the figure below, sketch the decision boundary for a 1-nearest-neighbor rule and circle the redundant points What is the general condition(s) required for a point to be declared redundant for a 1- nearest-neighor rule? Assume we have only two classes (+, -). Restating the definition of redundant ("removing it does not change anything") is not an acceptable answer. Hint think about the neighborhood of redundant points. 14

18 Part B: (5 Points) H V Which of H or V would be preferred as an initial split for a decision (identification) tree? Justify your answer numerically. x y -(x/y)*lg(x/y) x y -(x/y)*lg(x/y)

19 Part C: (10 Points) X Y In this network, all the units are sigmoid except unit 5 which is linear (its output is simply the weighted sum of its inputs). All the bias weights are zero. The dashed connections have weights of -1, all the other connections (solid lines) have weights of Given X=0 and Y=0, what are the output values of each of the units? Unit 1 = Unit 2 = Unit 3 = Unit 4 = Unit 5 = 2. What are the δ values for each unit (as computed by backpropagation defined for squared error) assume that the desired output for the network is 4. Unit 1 = Unit 2 = Unit 3 = Unit 4 = Unit 5 = 3. What would be the new value of the weight connecting units 2 and 3 assuming that the learning rate for backpropagation is set to 1? 16

20 Part D: (10 Points) 1. Consider the simple one-dimensional classification problem shown below. Imagine attacking this problem with an SVM using a radial-basis function kernel. Assume that we want the classifier to return a positive output for the + points and a negative output for the points. Draw a plausible classifier output curve for a trained SVM, indicating the classifier output for every feature value in the range shown. Do this twice, once assuming that the standard deviation (σ) is very small relative to the distance between adjacent training points and again assuming that the standard deviation (σ) is about double the distance between adjacent training points. Small standard deviation (σ): SVM output Feature value Large standard deviation (σ): SVM output Feature value 17

21 Would you expect that a polynomial kernel with d=1 would be successful in carrying out the classification shown above? Explain. 3. Assume we use an SVM with a radial-basis function kernel to classify these same data points. We repeat the classification for different values of the the standard deviation (σ) used in the kernel. What would you expect to be the relationship between the standard deviation used in the kernel and the value of the largest Lagrange multiplier (a i ) needed to carry out the classification? That is, would you expect that the max a i would increase or decrease as the standard deviation decreases? Explain your answer. 18

22 Part E: (5 Points) Given a validation set (a set of samples which is separate from the training set), explain how it should be used in connection with training different learning functions (be specific about the problems that are being addressed): 1. For a neural net For a decision (identification) tree 19

23 Problem 1: Classification (40 points) f The picture above shows a data set with 8 data points, each with only one feature value, labeled f. Note that there are two data points with the same feature value of 6. These are shown as two X's one above the other, but they really should have been drawn as two X's on top of each other, since they have the same feature value. Part A: (10 Points) 1. Consider using 1-Nearest Neighbors to classify unseen data points. On the line below, darken the segments of the line where the 1-NN rule would predict an O given the training data shown in the figure above f 2. Consider using 5-Nearest Neighbors to classify unseen data points. On the line below, darken the segments of the line where the 5-NN rule would predict an O given the training data shown in the figure above f 21

24 3. If we do 8-fold cross-validation using 1-NN on this data set, what would be the predicted performance? Settle ties by choosing the point on the left. Show how you arrived at your answer. 22

25 Part B: (8 Points) f Using this same data set, show the decision tree that would be built from this data. Assume that the tests in the tree are of the form f c. For each test show the approximate value of the average disorder for that test. To help you compute this, there's a small table of values of (x/y)*log(x/y) for small integer x and y. x y -(x/y)*lg(x/y) x y -(x/y)*lg(x/y)

26 Part C: (12 Points) f Construct the simplest neural net (using sigmoid units) that can accurately classify this data. Pick a set of appropriate weights for the network. Assume that we will predict O when the network's output is less than 0.5 and X when the output is above 0.5. Whenever possible use weights that are either 1, -1, 10 or

27 Part D: (10 Points) f Consider the simplified data set above and consider using an SVM with a polynomial kernel with d=2. Let's say the data points are specified as: X1 = [-2] Y1= -1 X5=[-3] Y5= -1 X2 = [-1] Y2= 1 X6=[ 3] Y6= -1 X3 = [ 1] Y3= 1 X4 = [ 2] Y4= What are the kernel values? K(x1,x1) K(x1,x2) K(x2,x3) K(x3,x4) 25

28 2. Show a reasonably accurate picture of the transformed feature space. a. label the axes, b. label the data points, c. show the separating line that would be found by the SVM, d. circle the support vectors. 26

29 Problem 2: Overfitting (20 points) For each of the supervised learning methods that we have studied, indicate how the method could overfit the training data (consider both your design choices as well as the training) and what you can do to minimize this possibility. There may be more than one mechanism for overfitting, make sure that you identify them all. Part A: Nearest Neighbors (5 Points) 1. How does it overfit? 2. How can you reduce overfitting? Part B: Decision Trees (5 Points) 1. How does it overfit? 2. How can you reduce overfitting? 27

30 Part C: Neural Nets (5 Points) 1. How does it overfit? 2. How can you reduce overfitting? Part D: SVM [Radial Basis and Polynomial kernels] (5 Points) 1. How does it overfit? 2. How can you reduce overfitting? 28

31 Problem 3: Spaminator (10 points) Suppose that you want to build a program that detects whether an incoming message is spam or not. You decide to attack this using machine learning. So, you collect a large number of training messages and label them as spam or not-spam. You further decide that you will use the presence of individual words in the body of the message as features. That is, you collect every word found in the training set and assign to each one an index, from 1 to N. Then, given a message, you construct a feature vector with N entries and write in each entry a number that indicates how many times the word appears in that message. Part A: (6 Points) If you had to choose between a Nearest Neighbor implementation or an Decision Tree implementation, which would you choose? Justify your answer briefly both in terms of expected accuracy and efficiency of operation. Indicate the strength and weaknesses of each approach. 29

32 Part B: (4 Points) Assume that you wanted to reduce the size of the feature vectors (during training and classification), for each of the approaches below indicate why it might be a good or bad idea. 1. Use only the words that appear in spam messages. 2. Eliminate words that are very common in the whole data set. 30

33 Problem 5: Backprop (10 points) Suppose we want to do regression instead of classification and so we change the (final) output unit of a neural net to be a linear unit, which simply outputs the weighted sum of its inputs (no sigmoid): y = w i x i i All the other units in the network would still retain their sigmoids. How would this change the backpropagation equations? Derive only the needed changes, do not repeat anything in the usual derivation that does not change. 31