1 Version Spaces. CS 478 Homework 1 SOLUTION

Size: px

Start display at page:

Download "1 Version Spaces. CS 478 Homework 1 SOLUTION"

Barnaby Hugo Thomas
5 years ago
Views:

1 CS 478 Hmewrk SOLUTION This is a pssible slutin t the hmewrk, althugh there may be ther crrect respnses t sme f the questins. The questins are repeated in this fnt, while answers are in a mnspaced fnt. Versin Spaces Cnsider the instance space X f pints x = (x, x 2 ) with integer crdinates ([..], [..]) N 2. The task is t learn a cncept c : X {, } which can be described as a rectangle ((a, b), (c, d)) N 2 N 2. (a, b) is the left tp crner f the rectangle, (c, d) is the lwer right crner. An example is labeled psitive, if it lies inside the rectangle r n it s bundary. It is negative, if it lies utside. The hypthesis space H are all rectangles ver the instance space X. Nte that in this setting, a hypthesis h = ((a, b), (c, d)) can be generalized by decreasing a r b and/r increasing c r d. Similarly it can be made mre specific by increasing a r b r decreasing c r d. Part a (5 POINTS) Suppse yu have the hypthesis h = ((4, 5), (4, 5)), after seeing the psitive training example (4, 5). If yu then see the psitive training example x = (6, 3), what is the smallest generalizatin f h that als accepts x? If yu use axes with the rigin in the usual lcatin, the smallest generalizatin that cvers bth examples is h = ((4, 5), (6, 3)). Alternatively, yu can have the rigin in the tp left crner, as implied by the questin, which gives the smallest generalizatin f h = ((4, 3), (6, 5)). Part b (5 POINTS) Cnsider the fllwing training set in X {, }, where the r is the label f each data pint ( indicates negative examples, and indicates psitive examples): {((, 5), ), ((4, 5), ), ((2, 2), ), ((9, 5), ), ((6, 3), ), ((5, 6), ), ((7, ), ), ((5, 8), )} Fr this training set, what is the S bundary f the versin space? What is the G bundary f the versin space? Write ut the hyptheses and draw them int a diagram shwing the pints as well as the bundaries f the versin space.

2 Drawing the datapints with an rigin in the lwer left crner, yu get the fllwing diagram. Nte the S bundary (smallest, thin slid rectangle) and G bundaries (three ther rectangles) There are three G bundaries, that is three rectangles that cntain nly psitive pints and cannt be generalized withut including any negative pints. Nne f these rectangles is cmpletely cntained in anther. We can write the bundaries t be: S = {((4, 6), (6, 3)} G = {((, 7), (8, 3)), ((3, 7), (8, )), ((3, 7), (6, ))} Many f the figures were just barely big enugh t read. is n reasn t make them t small! There Part c (5 POINTS) Imagine the learner des nt get specific training examples, but instead can prpse examples and ask fr the crrect label. Hw shuld the learner pick the example it asks t be labeled 2

3 next? Assume that ne psitive training example is given initially. Describe a strategy that will lead the learner t identify the target rectangle with as few training examples as pssible. Remember that since the target cncept is nt initially knwn t the learner, yur strategy cannt make use f it. Use the target cncept ((4, 6), (6, 4)) and the initial psitive example (5, 5) t illustrate yur strategy. In this questin, many f the slutins were nt fully specified. When runding is invlved, there are tw ways yu can rund, s please say which way yu chse t rund. Smetimes, it can even make a difference in the wrst case cst. Als, many f the lnger answers were harder t understand. Yu shuld try t be succinct and clear. Always describe the intuitin behind yur algrithm befre giving any pseud-cde, it helps us understand what yu are trying t d. The best slutin is a simple binary search in each f the fur directins frm the initial pint (up, dwn, right and left). Fr each f the fur directins, yu always ask fr the midpint f the range f pssible values fr that particular edge f the target cncept. When yu have t rund, always rund away frm the initial pint. As an example, cnsider the start pint (x, y) = (5, 5) and the target cncept ((4, 6), (6, 4)). We want t find this target cncept, the crners f which we call ((x L, y T ), (x R, y B )), as shwn in the diagram belw. (x, y ) L T (x, y) + (x, y ) R B First we find the right hand brder x R f the target cncept. In ur example, it can be at x R [5... ]. The midpint is 7 r 8, but runding away frm the initial pint means we ask fr the label f the pint (8, 5). (8, 5) is negative, s the right hand brder f the target cncept must be x R [5... 7]. The midpint is 6, s we ask fr (6, 5). This is psitive, s we knw x R [6, 7]. The midpint, runded away frm the start pint is 7. S we ask fr (7, 5), which is negative, and then we knw that x R = 6. 3

4 Similarly, t find the left hand brder, we initially knw x L [... 5], with midpint 2 r 3. Runding away frm the initial pint, we ask fr (2, 5), which is negative. Then we knw x L [3... 5], with midpint 4, s we ask fr (4, 5), which is psitive. Hence x L [3, 4]. Finally, we ask fr (3, 5) which is negative. Therefre x L = 4. We d exactly the same sequence f peratins fr the tp and bttm brders (except that fr all the pints we request, we use x = 5 and use the apprpriate y value). This gives us y T = 6 and y B = 4 after three examples each. Hence the target cncept is ((4, 6), (6, 4)). Part d (5 POINTS) Fr yur strategy frm Part c, what is the maximal number f training examples needed t identify any target cncept? Illustrate yur result by cnstructing the wrst case cncept and initital psitive example, s that the number f training examples is maximal. In this prblem, ften there was n justificatin why a given example might be wrst case. We did nt require a prf, but sme srt f diagram r rugh intuitin why the example is wrst case certainly helped, especially when using sme unusual example as wrst case. Fr many f the algrithms, the example given in part (c) was in fact a wrst-case example. Anther gd ne was with the initial pint at (5,5) and the target cncept being the entire hypthesis space, ((, ), (, )). The best slutin pssible is always being able t learn the target cncept with at mst 2 examples, in additin t the ne initially given. This is because with an ptimally implemented binary search, yu can select amng n values in lg 2 (n) steps. If the initial pint is in the middle f the hypthesis space, we need lg 2 (5) = 3 examples t pin dwn each f the fur sides f the target cncept, giving a ttal f 4 3 = 2 examples. If we were t cnsider a starting pint near the edge f the hypthesis space, say at (, ), we need example t check if each f (, ) and (, ) are psitive r negative, and then lg 2 (9) = 4 examples fr the tw ther sides. This gives a ttal f examples, which is therefre nt the wrst case situatin. 4

5 2 Decisin Trees Part a ( POINTS) What is a decisin tree with as few ndes as pssible that represents the blean functin ( A B) (C A) ver the blean attributes A, B, and C? ( A B) (C A) is equivalent t A (B C) using De Mrgan s Law and Assciativity. The fllwing tree represents this functin: A B C Since the minimal equivalent frmula invlves A, B and C, the smallest tree wuld have t have at least three ndes and therefre the given tree has as few ndes as pssible. Part b ( POINTS) What is a decisin tree that represents the blean functin (A XOR B) C ver the blean attributes A, B, and C? A XOR B is equivalent t ( A B) (A B). representing this functin is: S a pssible tree C A B B 5

6 Part c (2 POINTS) Grw the decisin tree fr predicting SP AM fr the fllwing dataset using Infrmatin Gain as the splitting criterin. The attributes nigeria, viagra, and learning indicate whether that particular wrd ccurs in the dcument. Shw yur calculatins at each step. nigeria viagra learning SPAM Entrpy(Rt) = (5/) lg(5/) (5/) lg(5/) = Gain(Rt, Nigeria) = (5/) ( (4/5) lg(4/5) (/5) lg(/5)) (5/) ( (4/5) lg(4/5) (/5) lg(/5)) =.278 Gain(Rt, V iagra) = (3/) ( (2/3) lg(2/3) (/3) lg(/3)) (7/) ( (3/7) lg(3/7) (4/7) lg(4/7)) =.35 Gain(Rt, Learning) = (2/) ( (/2) lg(/2) (2/2) lg(2/2)) (8/) ( (5/8) lg(5/8) (3/8) lg(3/8)) =.236 S N igeria is the apprpriate chice f attribute fr the rt nde. Entrpy(N igeria = ) = (4/5) lg(4/5) (/5) lg(/5) =.722 6

7 Entrpy(N igeria = ) = (4/5) lg(4/5) (/5) lg(/5) =.722 Gain(Nigeria =, Learning) =.722 (/5) ( (/) lg(/) (/) lg(/)) (4/5) ( (4/4) lg(4/4) (/4) lg(/4)) =.722 Gain(Nigeria =, V iagra) =.722 (/5) ( (/) lg(/) (/) lg(/)) (4/5) ( (3/4) lg(3/4) (/4) lg(/4)) =.73 When N igeria =, Learning is the apprpriate chice f attribute t split n. Bth the children f Learning are pure ndes s we stp here. Gain(Nigeria =, Learning) =.722 (/5) ( (/) lg(/) (/) lg(/)) (4/5) ( (/4) lg(/4) (3/4) lg(3/4)) =.73 Gain(Nigeria =, V iagra) =.722 (2/5) ( (/2) lg(/2) (/2) lg(/2)) (3/5) ( (/3) lg(/3) (3/3) lg(3/3)) =.322 When N igeria =, V iagra is the apprpriate chice f attribute t split n. When N igeria = and V iagra = the nde is pure and s we stp there. When N igeria = and V iagra =, we split n the last remaining attribute i.e. n Learning. S the resulting tree lks as fllws: 7

8 Nigeria Learning Viagra Learning 8

INSTALLING CCRQINVOICE

INSTALLING CCRQINVOICE Thank yu fr selecting CCRQInvice. This dcument prvides a quick review f hw t install CCRQInvice. Detailed instructins can be fund in the prgram manual. While this may seem like a