CS446: Machine Learning Fall Problem Set 4. Handed Out: October 17, 2013 Due: October 31 th, w T x i w

Transcription

1 CS446: Machine Learning Fall 2013 Problem Set 4 Handed Out: October 17, 2013 Due: October 31 th, 2013 Feel free to talk to other members of the class in doing the homework. I am more concerned that you learn how to solve the problem than that you demonstrate that you solved it entirely on your own. You should, however, write down your solution yourself. Please try to keep the solution brief and clear. Please use Piazza first if you have questions about the homework. Also feel free to send us s and come to office hours. Please present your algorithms in both pseudocode and English. That is, give a precise formulation of your algorithm as pseudocode and also explain in one or two concise paragraphs what your algorithm does. Be aware that pseudocode is much simpler and more abstract than real code. The homework is due at 11:59 PM on the due date. We will be using Compass for collecting the homework assignments. Please submit your answers and proofs via Compass ( Please do NOT hand in a hard copy of your write-up. Contact the TAs if you are having technical difficulties in submitting the assignment. IMPORTANT: You are allowed exactly one submission for each question in this assignment. Unless you are absolutely certain that you wish to submit your final version, press the Save Answer or Save All Answers buttons instead of the Save and Submit button on Compass. IMPORTANT: Please don t upload pictures in your proof except the ones from built-in Compass formula editor, they might not be displayed on Compass when grading. If you use the built-in Compass formula editor, please give permission to TAs. If you paste latex source code, please make sure it can be successfully compiled. Thank you. 1. [Computing Margins - 25 points] The margin of a set of points {x 1, x 2,, x m } with respect to a hyperplane is defined as the distance of the closest point to the hyperplane w T x = 0. Here the last entry of w is the threshold θ. The margin γ is thus: γ = min i w T x i w Assume that the examples are points in {0, 1} 20 (that is, x is a 20-dimensional binary vector). We wish to learn the following concept in this space: f(x) = x 1 x 3 x 5 x 10 x 12 x 14 x 16 The variables that are in the disjunction are referred to as relevant variables and the others as irrelevant variables. 1.1 [5 points] Represent f as a linear threshold function. That is, find w such that sgn(w T x) is equivalent to f for all x {0, 1} 20. Note that the representation of this function is not unique; to simplify things, we ask that you assume that the 1

2 entries of w except θ are all integers and that θ = 0.5 and w is minimum. Note also that we define sgn(x) to be: { 1 if x 0; sgn(x) = 0 otherwise. [Please fill in the coefficients of your w on 1.2 [5 points] Consider a dataset D 1 that is generated as follows: Positive examples: All possible points that have one relevant variable set to one and six irrelevant variables set to one and all other variables set to zero. Negative examples: All possible points that have no relevant variables set to one and six irrelevant variables set to one and all other variables set to zero. Compute the margin of D 1 with respect to your linear threshold function in Problem 1.1. [Multiple choice question on 1.3 [5 points] Consider a dataset D 2 that is similar to D 1, except that for positive examples six relevant variables are set to one in addition to the six irrelevant variables. Compute the margin of D 2 with respect to your linear threshold function in Problem 1.1. [Multiple choice question on 1.4 [5 points] Consider a third dataset D 3 that is similar to D 1. The only difference is that the number of irrelevant variables that are set to one in both positive and negative examples is increased to ten instead of six. Compute the margin of D 3 with respect to your linear threshold function in Problem 1.1. [Multiple choice question on 1.5 [5 points] Compute the Perceptron mistake bounds for D 1, D 2, and D 3 and rank the datasets in terms of ease of learning. [Multiple choice question on 2. [VC Dimension - 30 points] For this problem, assume that all examples are points in two-dimensional space, i.e. x = x 1, x 2 R 2. We specify two concept spaces below. For each concept space, give the VC dimension and prove that your answer is correct. IMPORTANT: Please don t upload pictures in your proof except the ones from built-in Compass formula editor, they might not be displayed on Compass when grading. If you use the built-in Compass formula editor, please give permission to TAs. If you paste latex source code, please make sure it can be successfully compiled. Thank you. 2.1 [15 points] Consider the concept space of circles with arbitrary origin and radius. Hence, a concept h H has three parameters, r R + and the two coordinates of the center x 0 R 2. An example x R 2 is labeled as positive by h if and only if x lies within a circle of radius r centered at x 0 and of radius r, i.e. x x 0 < r. 2

3 2.2 [15 points] The concept space C is the region between two parallel lines, either (x = a, x = b) or (y = a, y = b) for a < b. That is, each concept f C is defined by two numbers, a and b and another Boolean indicator that determines whether the lines are parallel to the x axis or the y-axis. An example (x, y) is positive for the concept (X, a, b) if and only if a x b. An example (x, y) is positive for the concept (Y, a, b) if and only if a y b. Grading note: You will not get any points without proper justification of your answer. 3. [Decision Lists - 30 points] In this problem, we are going to learn the class of k-decision lists. A decision list is an ordered sequence of if-then-else statements. The sequence of if-then-else conditions are tested in order, and the answer associated to the first satisfied condition is returned. k-decision list form a subset of this family of functions, where the statements in each rule are of a bounded size. Formally, for a fixed k, we define: Definition 1 A k-decision list over the variables x 1,..., x n is an ordered sequence L = (c 1, b 1 ),..., (c l, b l ) and a bit b, in which each c i is a conjunction of at most k literals over x 1,..., x n. The bit b i is referred to as the bit associated with condition c i, and b is called the default value. For any input x {0, 1} n, L(x) is defined to take the value b j, where j {1, l} is the smallest index satisfying c j (x) = 1; if no such index exists, then L(x) = b. We denote by k-dl the class of concepts that can be represented by a k-decision list. Figure 1 shows an example of a 2-decision list over six variables, x 1,..., x 6. For this decision tree, L(x 1 = 0, 1, 1, 0, 0, 1 ) = 1 and L(x 2 = 1, 0, 0, 1, 0, 0 ) = [5 points] Given any k-decision list c = (c 1, b 1 ),..., (c l, b l ), b, can its complement c still be represented as a k-decision list? [Multiple choice question on 3.2 [5 points] What s the relations among k-dnf, k-cnf, and k-dl? (k-cnf is conjunctions of any number of clauses where each disjunctive clause has at most k literals and and k-dnf is disjunctions of any number of terms where each conjunctive term has at most k literals.) [Multiple choice question on 3.3 [5 points] In Problem 3.3 and 3.4, we will learn an algorithm that can learn k-dl concepts. Let S be a non-empty set of examples consistent with the target k-dl c. Denote by C k the collection of all conjunctions of size at most k: C k = O(n k ). Note that the empty conjunction is also included in C k. For every c i C k, let T i (S) = {x S c i (x) = 1}, i.e. T i (S) is the set of all examples in S that satisfy c i. For b {0, 1} we say that c i is b-consistent with the sample S if T i (S) and x T i (S), we have that c(x) = b, i.e. the examples in T i (S) (i.e. the ones that satisfy c i ) are either all positive or all negative. Consider the following algorithm: Algorithm 1 3

4 Figure 1: A 2-decision list. i. Start with an empty decision list. ii. For b {0, 1}, search for a c i C k that is b-consistent with S. To do this, iterate through all c i C k, evaluating c i on all the examples in T i (S). If you find that c i is b-consistent, stop; otherwise, move on to c i+1. iii. Add (c i, b) to the decision list. iv. Update S S \ T i (S), i.e. remove elements of T i (S) from S. v. If S is not empty, go to step (ii). Otherwise, return the current decision list. Suppose Algorithm 1 terminates, then is the constructed k-decision list consistent with the sample S? [Multiple choice question on 3.4 [5 points] The problem setting and the algorithm are same as in Problem 3.3. Is there always at least one b-consistent conjunction in C k in step (ii) of Algorithm 1? [Multiple choice question on 3.5 [5 points] Is the class of k-decision lists efficiently PAC-learnable for any constant integer k 1? [Multiple choice question on 3.6 [5 points] Can any 1-decision list be represented as a linear threshold function 1? [Multiple choice question on 4. [Kernels - 15 points] 4.1 [2 points] Which function is the Laplacian kernel? [Multiple choice question on 1 Find a weight vector w that will make the same prediction as a given 1-decision list. 4

5 4.2 [2 points] Which function is the sigmoid kernel? [Multiple choice question on 4.3 [2 points] Which function is the rational quadratic kernel? [Multiple choice question on 4.4 [4 points] Let K 1 (x, z) and K 2 (x, z) be valid kernel functions. Define a new function K(x, z) = αk 1 (x, z) + βk 2 (x, z) where α and β are positive real numbers. Is K(x, z) a kernel function? [Multiple choice question on 4.5 [5 points] Given two examples x R 2 and z R 2, let K(x, z) = (x T z) (x T z) x T z. Prove that K(x, z) is a valid kernel function. 5