Support Vector Machines

Transcription

1 /9/207 MIST.6060 Busness Intellgence and Data Mnng What are Support Vector Machnes? Support Vector Machnes Support Vector Machnes (SVMs) are supervsed learnng technques that analyze data and recognze patterns. SVMs can be appled to both classfcaton and regresson problems. When used for regresson, the technque s called Support Vector Regresson (SVR). The basc dea of SVM for classfcaton s to construct a hyperplane (or a set of hyperplanes) that best separates the data ponts of dfferent classes. Lnearly Separable Case: Decson Boundary and Maxmum Margn Let {( x } N, y ) be a tranng set of N records, where x (,..., ) = = x xm denotes the values of M predctors, X,..., X M, for record, and y s the correspondng value of the class attrbute Y. We consder bnary classfcaton problem. By conventon, let y be ether or. The decson boundary (hyperplane) can be wrtten as or w + w x + + w x + b 0, x 2 2 M M = w x + b = 0, () where w = w,..., w ) and b are parameters of the hyperplane to be estmated. ( M Consder the modfed Admsson data (Admsson2.arff), where 5 records are removed from the orgnal Admsson data, resultng n a dataset whose two classes are lnearly separable. The dataset s gven on the next page. The data ponts are plotted along wth some lnes (hyperplanes n the 2-dmensonal space) that separate the data ponts of the two dfferent classes. It s easy to see that there are (nfntely) many such lnes possble. The objectve of an SVM approach s to fnd the maxmum margn hyperplane, whch s the hyperplane that has the largest dstance to the nearest tranng data pont of any class. Intutvely, t can be argued that the maxmum margn hyperplane wll generalze well when used for classfyng new data ponts. In ths example, the sold lne represents the maxmum margn hyperplane. The nearest square pont s x 6 and the nearest crcle pont s x. We wll explan later the parameter C that s assocated wth each lne. Xaoba L All Rghts Reserved.

2 /9/207 MIST.6060 Busness Intellgence and Data Mnng 2 ID GPA SAT Accept? Normalzed GPA Normalzed SAT yes yes yes yes yes yes yes yes yes no no no no no no no no no no Xaoba L All Rghts Reserved.

3 /9/207 MIST.6060 Busness Intellgence and Data Mnng 3 Let x and x be two ponts located on the decson hyperplane, then Subtractng the two equatons, we have w x + b = 0, w x + b = 0. w ( x x ) = 0, where x x s a vector parallel to the decson hyperplane and ts drecton s from x to x. Because the dot product s zero, the drecton for w must be perpendcular to the decson hyperplane, as shown n the chart on the prevous page. For any square pont x s, whch s located above the decson hyperplane, we have w + b > 0. (2) x s Smlarly, for any crcle pont x c, whch s located below the decson hyperplane, we have w + b < 0. (3) x c Consder the square pont and crcle pont that are nearest to the decson hyperplane (n ths example, they are x 6 and x ). The square pont must satsfy (2) whle the crcle pont must satsfy (3). We can rescale the parameter w and b so that the two hyperplanes parallel to the decson hyperplane whle passng through the nearest square and crcle ponts respectvely are H : w x + b =, (4) s H : w x + b =. (5) c The two nearest ponts can be used to compute the dstance (margn) between the two hyperplanes. In ths example, substtutng x 6 and x nto (4) and (5) respectvely, we have w ( x x ) 2, 6 = w x 6 x cos( w, x 6 x) = 2, w d = 2, 2 d =. (6) w Xaoba L All Rghts Reserved.

4 /9/207 MIST.6060 Busness Intellgence and Data Mnng 4 Data ponts that le on margn hyperplanes are called support vectors. In ths example, and H c are the margn hyperplanes and x 6 and x are the two support vectors. We can see that the margn hyperplanes depend only on the support vectors, whch mples that w and b depend only on the support vectors. Ths sgnfcantly reduces computatonal cost. Lnearly Separable Case: Lnear SVM Model Our task s to fnd the parameters w and b such that w + b, f y =, (7a) x w + b, f y =. (7b) Equatons (7a) and (7b) can be combned nto one as x y ( w x + b). (8) The objectve of the SVM problem s to maxmze the margn d n equaton (6), or equvalently to mnmze /d n (6), subject to constrant (8). That s, the SVM problem can be wrtten as H s mn w, b 2 w subject to 2, (9a) y ( w x + b), =,..., N. (9b) Ths s a quadratc programmng problem, where the objectve functon (9a) s quadratc and constrants (9b) are lnear. There s a standard Lagrange multpler method to solve ths problem for w and b. All data mnng software packages mplement ths method to solve the SVM problem. The detals of ths soluton method s beyond the scope of ths course. Non-separable Case: Lnear SVM Model Real-world data often cannot be fully lnearly separated. For example, the orgnal Admsson data (Admsson.arff) s not fully lnearly separable. There are some msclassfed data ponts when the lnear SVM decson hyperplane s used. The dataset s shown on the next page, and plotted wth some SVM decson hyperplanes (lnes). The two support vectors assocated wth the sold SVM decson lne are #2 and #4, whch are yellow hghlghted n the table. Xaoba L All Rghts Reserved.

5 /9/207 MIST.6060 Busness Intellgence and Data Mnng 5 ID GPA SAT Accept? Normalzed GPA Normalzed SAT yes yes yes yes yes yes yes yes yes yes yes yes no no no no no no no no no no no no Xaoba L All Rghts Reserved.

6 /9/207 MIST.6060 Busness Intellgence and Data Mnng 6 In order for the SVM model to handle data that s not fully lnearly separable, we ntroduce postve slack varables, ξ > 0,,..., N, nto the constrants, as below: = w ξ + b ξ, f y =, (0a) w ξ + b + ξ, f y =. (0b) It can be shown that the dstance from a msclassfed data pont x to the margn hyperplane that represents ts correspondng class y s ξ / w. Ths s llustrated for the msclassfed crcle pont #3 n the chart. So, the slack varables ξ represent the tranng errors assocated wth the SVM model. These errors should be consdered n the SVM model. Thus, the SVM problem for non-separable case s formulated as mn w, b 2 w subject to 2 + C N = ξ, (a) y ( w ξ + b) + ξ 0, =,..., N, (b) where C > 0 s a user-specfed parameter and constrant (b) s obtaned by combnng (0a) and (0b). The frst term n objectve functon (a) represents the margn hyperplane structure of the model, as n the lnearly separable case. The second term n (a), on the other hand, measures the classfcaton error of the model. Parameter C can be vewed as a weght to balance the tradeoff between the two components of the objectve. Therefore, SVM attempts to mnmze both the classfcaton error and structural complexty (to lmt the overfttng problem) of the model. Ths dea, called structural rsk mnmzaton, s essental to all SVM technques. Formulaton () s called the soft margn model whle (9) s called the hard margn model. The SVM algorthm n Weka s developed based on the soft margn model. Recall that we used a very large C value (C = 000) to fnd the maxmum margn hyperplane for the lnearly separable case. A very large C value mposes a very heavy penalty for even a very small error. Consequently, the algorthm focuses almost exclusvely on mnmzng tranng error. As we can see from both charts, an ncrease n C value may cause ether an ncrease or a decrease n the slope of the hyperplane. In general, there s no analytcal method to determne the best C value. The approprate C value s usually determned usng cross valdaton. Agan, problem () s a quadratc programmng problem, whch can be solved by the Lagrange multpler method. Xaoba L All Rghts Reserved.

7 /9/207 MIST.6060 Busness Intellgence and Data Mnng 7 Support Vector Regresson Regresson s a method for descrbng statstcal relatonshps between a target (dependent, output) varable (Y) and a set of predctor (ndependent, nput) varables (X s) by fttng a mathematcal functon, called regresson model, to a gven dataset. For a dataset wth N records and M predctor varables, the lnear regresson model s expressed by (usng tradtonal notaton n statstcs) where b 0 s the ntercept and y = b + b x + b x + + b x, =, N, (2) M M 2, b,b Usng SVM notaton, we can wrte (2) as or y, M are the slope parameters. = w x + w x + + w x + b, =, N, 2 2 M M 2, y = w x + b, =, N. (3), In lnear regresson, any pont that does not exactly ft the regresson model (or le on regresson lne n a 2-dmentonal case) ncurs some error. In SVR, however, there s a (userspecfed) error margn parameter ε. A devaton from the central SVR model s not consdered an error f t s wthn the margn. When the devaton s larger than ε, t s captured by the slack varables ξ +, ξ > 0, =,..., N. Therefore, the SVR problem s formulated as mn w 2 + C + w, b, ξ, ξ = 2 N ( ξ + + ξ ), (4a) subject to w ξ + b y ε + ξ, =,..., N, (4b) + y w ξ b ε + ξ, =,..., N. (4c) Agan, the second term n the objectve functon (4a) measures the ftness (predcton error) of the model. The frst term n (4a) represents the complexty of the model; ntutvely, the smaller the w values, the flatter the regresson plane (.e., the less complex the model). Therefore, SVR attempts to mnmze both the predcton error and model complexty (to lmt the overfttng problem). As mentoned earler, parameter C > 0 can be vewed as a weght to balance the tradeoff between the two components of the objectve. Xaoba L All Rghts Reserved.

8 /9/207 MIST.6060 Busness Intellgence and Data Mnng 8 To llustrate the dea, consder the dataset below, whch ncludes a predctor attrbute, Educaton (X), and a target attrbute, Salary (Y). Let ε = 5. Then the SVR model s The least-square lnear regresson model s Salary = * Educaton Salary = 4.468* Educaton It can be seen from the plot that the SVR model lne s flatter than the lnear regresson lne. No. Educaton (X) Salary (Y) (n Years) (n $000) Salary (Y) Actual Salary SVR SVR + 5 SVR - 5 Lnear Regresson Educaton (X) Xaoba L All Rghts Reserved.

9 /9/207 MIST.6060 Busness Intellgence and Data Mnng 9 Support Vector Machnes n Weka The Admsson2.arff fle (9 records):. Clck Open fle, fnd and open the Admsson2.arff fle. 2. Clck Classfy / Choose / functons / SMO. [SMO stands for Sequental Mnmal Optmzaton. Reference: Platt J (998) Fast tranng of support vector machnes usng sequental mnmal optmzaton. Schölkopf B, Burges C, Smola AJ, eds. Advances n Kernel Methods Support Vector Learnng (MIT Press, Cambrdge, MA), ] 3. Select Use tranng set. Note the default parameter C =. Clck Start. Xaoba L All Rghts Reserved.

10 /9/207 MIST.6060 Busness Intellgence and Data Mnng 0 4. Clck the long horzontal box on the rght of the Choose button. A pop-up weak.gu.genercobjectedtor appears. Enter 0 for the C box. Clck OK. 5. Clck Start to get the results for C = 0. Xaoba L All Rghts Reserved.

11 /9/207 MIST.6060 Busness Intellgence and Data Mnng 6. Change parameter to C = 000, and run the algorthm agan. Examne the results for C =, 0, and 000, along wth the three lnes n the chart on page 2. The Admsson.arff fle (24 records): Run SMO for C =, 5, and 2, respectvely. The results are show below. See the chart on page 5 for the correspondng lnes. Xaoba L All Rghts Reserved.

12 /9/207 MIST.6060 Busness Intellgence and Data Mnng 2 Xaoba L All Rghts Reserved.

13 /9/207 MIST.6060 Busness Intellgence and Data Mnng 3 Support Vector Machnes n Rattle. Clck Data. Select ARFF. In the Flename box, fnd and open the Admsson.arff. Clck Execute. Deselect Partton. Clck Execute. 2. Clck Model. Select SVM. Select Lnear (vanlladot) n the Kernel box. Clck Execute. 3. Clck Evaluate and then Execute. The error rate s the same as that n Weka. Xaoba L All Rghts Reserved.

14 /9/207 MIST.6060 Busness Intellgence and Data Mnng 4 4. Clck Model and enter C=5 (captal C) n the Optons box. Then clck Execute. 5. Clck Evaluate and then Execute. The error rate s dfferent from that n Weka. Smlarly, the error rate when C=2 s dfferent from that n Weka. Xaoba L All Rghts Reserved.

15 /9/207 MIST.6060 Busness Intellgence and Data Mnng 5 6. Clck Data. Enter 75/25 n the Partton box. Clck Execute. 7. Clck Model. Select SVM. Clck Execute. Xaoba L All Rghts Reserved.

16 /9/207 MIST.6060 Busness Intellgence and Data Mnng 6 8. Clck Evaluate. Select Testng and then clck Execute. Support Vector Regresson and Lnear Regresson n Weka. Clck Open fle, fnd and open the Salary.arff fle. 2. Clck Classfy / Choose / functons / SMOreg. 3. Clck the long horzontal box on the rght of the Choose button. A pop-up weak.gu.genercobjectedtor appears. In the fltertype box, select No normalze/standardzaton (usually, the default Normalze tranng data should be used; for ths exercse, we choose No normalze/standardzaton n order to see the context easly). 4. Clck the long horzontal box on the rght of the Choose button for regoptmzer. A pop-up weak.gu.genercobjectedtor appears. Enter 5 n the epslonparameter box. Clck OK twce to close both pop-up wndows. 5. Select Use tranng set. Clck Start. Xaoba L All Rghts Reserved.

17 /9/207 MIST.6060 Busness Intellgence and Data Mnng 7 6. Clck Choose / functons / LnearRegresson. Clck Start. Xaoba L All Rghts Reserved.