Assume we are given a tissue sample =, and a feature vector

Transcription

1 MA 751 Part 6 Support Vector Machines 3. An example: Gene expression arrays Assume we are given a tissue sample =, and a feature vector x œ F Ð=Ñ $!ß!!! consisting of 30,000 gene expression levels as read by a gene expression array. We wish to determine whether the tissue is cancerous or not. For an x which in fact corresponds to cancerous tissue, we will set the corresponding output variable Cœ" ; otherwise Cœ ".

2 Consider a data set W œöðx 3 ßCÑ 3 3œ", consisting of pairs of feature vectors x 3 and corresponding (correct) diagnoses C3 Ö "ß " Þ Can we find the right function 0" À J Ä U which generalizes the above examples, so that 0Ð " xñœcfor all feature vectors? Easier (see below): Find a 0ÀJÄ, where 0ÐxÑ! if 0 ÐxÑ œ "à 0ÐxÑ! if 0 ÐxÑ œ "Þ " "

3 4. Support vector machine framework Recall the regularization setting: we have examples. x 3 3 with, C œ Ö ". HœÖÐx ßCÑßáßÐx ßC Ñ " ", As mentioned above, we want to find a function 0" À Ä which generalizes the above data so that 0ÐxÑ œ C generalizes the data H..

4 As mentioned there, we will actually want here something more general: a function 0ÐxÑ which will best help us decide the true value of C. It may not need to be that we want 0ÐxÑ œ C, but rather we want œ 0Ð x Ñ >> " if C œ " 0ÐxÑ << " if C œ " ß (2) i.e., 0ÐxÑ is large and positive if the correct answer is C œ " (e.g. a chair) and 0ÐxÑis large and negative if the correct answer is Cœ " (not a chair).

5 Then the decision rule will be to conclude the value of C based on the rule (2). This is made precise as follows. We have the following optimization criterion for the 'right' 0: " 0 œ arg min Z ÐC3ß0Ðx3ÑÑ -m0mo # ß 0 [ 3œ" where m0m œ norm in an RKHS, e.g., O [ m0mo œ me0m # œ ( ÐE0Ñ.BÞ P #

6 Above 'arg min' denotes the 0 which minimizes the above expression. Loss function: hinge loss Z Ð0ÐxÑß CÑ œ Ð" C0ÐxÑÑ, where Ð+Ñ max Ð+ß!ÑÞ 5. More about the hinge loss

7 Consider the error function Z Ð0ÐxÑß CÑ œ Ð" C0ÐxÑÑ max " C0ÐxÑß! Þ small if Cß 0ÐxÑ have same sign œ œ Þ large otherwise

8 This is called the hinge loss function.

9 [Notice margin built in: error! only if C0ÐxÑ " (more stringent requirement than just C0ÐxÑ!Ñ] Thus data error is " /. œ ZÐ0Ðx4ÑßC4Ñ 4œ" What is a priori information? Note surface LÀ0œ! will separate "positive" x with 0ÐxÑ!, and "negative" x with 0ÐxÑ! À

10

11 Fig. 1. Red points have Cœ+1 and blue have Cœ 1 in the space J. L À 0ÐxÑ œ! is the separating surface. Assume some a priori information defined in terms of an RKHS norm m mo so m0mo is small if a priori assumption is satisfied. Let [ be corresponding RHKS. Will specify desirable norm m m O later... Now solve regularization problem for the above norm and loss Z :

12 " # 0! œ arg min Ð" C40Ðx4ÑÑ - m0moþ (1) 0 [ 4œ" 6. Introduction of slack variables Define new variables 04, and note if we find the min over 0 [ and of 0 4 with the constraint " 0 [0 ß 4 4œ" arg min 04 - m0mo # (1a)

13 Slack variables and solution C0Ð 4 x4ñ " !ß we get the same solution 0. ` To see this, note the constraints are 0 4 Ð!ß " C 4 0Ðx 4 ÑÑ œ Ð" C 4 0Ðx 4 ÑÑ max, (1b) which yields the claim. (Clearly in fact in minimizing sum we will end up with œð" C0Ðx ÑÑ )

14 Slack variables and solution From form (1) above by representer theorem: 0ÐxÑ œ + OÐxßx ÑÞ 4œ" 4 4 Ô + " To find a œ ã (see above material): let Õ+ Ø OœÐO34ÑœOÐx3ßx4Ñ Then

15 Slack variables and solution # m0m œ O + 4OÐxx ß 4Ñß + 3OÐxx ß 3Ñ 4 3 œ + + ØOÐxx ß ÑßOÐxx ß ÑÙ 3ß œ + + OÐ ß Ñ œ X x x + + O œ a Oa ß4 3ß4

16 Thus: Slack variables and solution " X a œ arg min 04 - a Oa (2a) a 4œ" with constraint: C + OÐx ßx Ñ " 0 (2b) œ" 0 4!Þ (2c)

17 5. Bias Slack variables and solution Given choice of [, O we have concluded 0ÐxÑ œ + 4OÐxßx4 Ñ (3) 4œ" which optimizes (1), equivalently (2). Now can expand class (2) of allowable 0 ad hoc. We may feel larger class than [ is appropriate. Often adding a constant, is useful.

18 Slack variables and solution Thus change 0ÐxÑ by adding a bias term,: 0ÐxÑ œ + 4OÐxßx4 Ñ,. (4) 4œ" The effect: regularization term unchanged (i.e., we ignore, in the norm m0m O ; remember any a priori assumption is valid if it is useful Ñ. Note this is still a norm on the expanded space of functions of the form (4), but may not be positive definite, i.e., m0m œ! for some 0 of the form (4).

19 Slack variables and solution For example we may have m,m œ!. O But: minimization of (1) using (4) still makes sense and allows possibly richer set of functions than [, as long as the regularization term m0m O still makes sense for such a richer set. In terms of slack variables 0 3, new optimization problem: Ô + " Find a œ ã which minimizes: Õ+ Ø

20 Slack variables and solution " X 04 -a Oa 4œ" with constraints: C 4 + 3OÐx3ßx4Ñ, " 04 Ð4aÑ 3œ" 0 3! ( quadratic programming problem).