Description of some supervised learning algorithms

Transcription

1 Descriptio of some supervised learig algorithms Patrick Keekayoro Statistical Cybermetrics Research Group Uiversity of Wolverhampto 1. Supervised learig Supervised machie learig is also a type of classificatio. The goal of supervised machie learig is to idetify a model that maps istaces i a dataset to their class labels. The assumptio is that if the idetified model accurately maps the kow istaces to their class labels, it will accurately predict the labels of ew istaces with ukow labels. There are several supervised learig algorithms. To apply these algorithms effectively, it is importat to uderstad the algorithms so that the ideal techique that is suitable for the particular problem ca be selected. Aalysis 1

2 K-Nearest eighbours K-earest eighbour classifiers (Cover ad Hart, 1967) are istace-based learig algorithms called learig classifiers because they delay the geeralizatio process util the classificatio is performed (Kotsiatis, 2007), ad thus requires less computatioal time i the traiig process. Istace-based classifiers have a advatage over other learig algorithms because of their simplicity. The K-earest eighbour algorithm has just oe adjustable parameter, k, which cotrols the umber of earest eighbours that are used to defie class membership. k = 7 Figure 1.1 A two class K Nearest eighbour classifier (La et al., 2012) Figure 1.1 shows the schematic of a two class K-earest eighbour classifier. Whe the istace (desigated by star) is classified with either K = 3 or K = 7, it is assiged to the black class because more istaces i the most similar 3 ad 7 eighbours belog to the black class. Similarity betwee eighbours ca be computed with a variety of metrics, the most popular of which are listed i Table 1.1. These metrics ca also be used to determie the quality of clusterig solutios. Aalysis 2

3 A drawback of istace-based classifiers is the computatioal time that it takes for classificatio. The time it takes to classify a istace is proportioal to the umber of traiig istaces ad the umber of features that describe each istace (Guo et al., 2003). Istace-based algorithms like the k earest eighbour classifier are stable, i some cases, the classificatio accuracy does ot drop sigificatly whe up to 80% of traiig istaces are removed (Kotsiatis, 2007). Because it oly has oe parameter k, it may be easy to fid the optimal classificatio models, eve though there is ot a pricipled way to choose k (Kotsiatis, 2007), oly through methods like (Guo et al., 2003) that icreases the already poor computatioal time. Aalysis 3

4 Table 1.1 K Nearest eighbour distace metrics (Kotsiatis, 2007) Mikowsky m 1/r D(x, y) = ( x i y i r ) Mahatta D(x, y) = x i y i m Chebychev D(x, y) = m i = 1 max x i y i Euclidea m D(x, y) = ( x i y i 2 ) 1 2 Camberra D(x, y) = x i y i x i + y i m Kedall s Rak Correlatio D(x, y) = 1 i 1 2 m(m 1) sig(x i x j )sig (y i y j ) m i=j j=1 Aalysis 4

5 Decisio tree iductio Utgoff (1989) formally defied decisio trees as a leaf ode (aswer ode) that cotais a class ame, or a o-leaf ode (decisio ode) that cotais a attribute text with a brach to aother decisio tree for a possible value of the attribute. Decisio tree iductio recursively splits data ito disjoit sets accordig to a criterio. Each ode is a feature that a istace ca have, ad leaf odes cotai output classes for istaces that reach that ode. Classificatio of istaces starts at the root ode, ad the the istaces traverse dow the tree i the directio that meets several criteria util a leaf ode is reached. The value of the leaf ode is the assiged to that istace. Costructig optimal biary decisio trees is a NP-complete problem (Kotsiatis, Zaharakis ad Pitelas, 2006), however several techiques, like the C4.5 algorithm (Quila, 1993) ad CART, a acroym for Classificatio Ad Regressio Trees (Breima et al., 1984) ca be used to build decisio trees. The C4.5 algorithm is implemeted i the popular machie learig toolkit, WEKA (Hall et al., 2009). The J48 algorithm i the WEKA machie learig toolkit is a implemetatio of the C4.5 decisio tree iductio algorithm. Two major phases of some types of decisio tree iductio algorithms are the growth phase ad the pruig phase (Kotsiatis, 2013). The growth phase ivolves splittig the traiig data ito disjoit sets ad the pruig phase reduces the size of the decisio tree to avoid overfittig. Pruig is Aalysis 5

6 ecessary to reduce the size of a over grow decisio tree ad also to avoid overfittig (Bradford et al., 1998). Decisio tree iductio Pseudo Code (Kotsiatis, 2007) 1 For each attribute a 2 Fid the feature that best divides the traiig data 3 Let a_best be the attribute that best splits data 4 Create a decisio ode that splits o a_best 5 Recurse o the sub-lists obtaied by splittig o a_best ad add odes as childre util leaf odes cotai oly oe istace Y N N Y N Y Figure 1.2 A example of a decisio tree classifier A major differece betwee the C4.5 ad CART algorithms is the way that the best feature that separates the traiig data is selected. The attribute with maximum iformatio gai is used to split the traiig set. C4.5 uses etropy to compute the iformatio gai, while CART uses the Gii idex. The etropy is calculated by: Etropy(S) = Freq(C i, S) log(freq(c i, S)) Freq(C i, S) is the relative frequecy of istaces i class Ci. Aalysis 6

7 The Gii idex is computed by: GiiIdex(S) = 1 Freq(C i, S) 2 Ad iformatio gai is computed by: IformatioGai(S, A) = I(S) S i I(S S i ) The formula above computes the iformatio gai of attribute A i data set S where i = 1 are possible values of attribute A, S i S are partitioed subsets of S where attribute A is i, I(S) is the etropy i the C4.5 algorithm ad the Gii idex i the CART algorithm. The advatage decisio trees have over other supervised learig algorithms is that the resultig tree ca be see as a set of rules which makes it easy to uderstad the classificatio model. The Perceptro Algorithm The perceptro algorithm (Roseblatt, 1962) is a well-kow classificatio algorithm. The simplest classificatio problems are dichotomous. The iput x ca belog to oe of two classes (y = 1 or y = 1 ). For a give iput represeted as a vector, x = (x 1 x ), the goal of machie learig is to classify x to either of two classes ( 1, 1). For a give iput x, liear models describe a fuctio g(x) that returs the output y based o a liear fuctio determied from a traiig dataset D = ( x 1 y 1 ),., (x y ). Liear discrimiat fuctios for patter classificatio are described i (Highleyma, 1962). Aalysis 7

8 writte as: The perceptro algorithm (Roseblatt, 1962) is a liear classifier d g(x) = sig ( i=0 w i x i ); x 0 = 1 Here, x is the iput vector ad the goal of the perceptro algorithm is the fid the appropriate values of the weight vector w that accurately classifies the traiig dataset. Y X Figure 1.3 A visual represetatio of the perceptro classifier The perceptro algorithm aims to fid ay lie that ca separate the two classes i a dataset. It fids this lie by iteratively updatig the weight vector util all objects i the dataset are correctly classified, if possible. The perceptro learig algorithm is mathematically prove to get a solutio if the iput space is liearly separable (Roseblatt, 1962). Aalysis 8

9 Figure 1.4 The perceptro liear classifier Perceptro-like algorithms may have advatages whe data has a high umber of features with few relevat oes (Kotsiatis, 2007). However, a simple perceptro algorithm will ot fid a adequate solutio if the data is ot liearly separable ad most real world machie learig problems are ot liearly separable. Multilayer Perceptro Multilayer perceptro (Ruck et al., 1990) is a kid of eural etwork. Neural etworks are models based o the huma ervous system. Essetially, they are black boxes that are able to predict the output class whe they recogize a give iput patter. Sigle perceptros will ot reach a solutio whe the traiig set is ot liearly separable. This ca be overcome by usig artificial eural etworks (Rumelhart, Hito ad Williams, 1986), that combie multiple liear classifiers, hece the ame multilayer perceptro. Aalysis 9

10 Multilayer perceptros are made up three compoets, the iput, hidde layers ad the output. The feed forward ature of the multilayer perceptro meas that the output of the previous layer is the iput of the ext layer. The back propagatio algorithm (Rumelhart, Hito ad Williams, 1986) is widely used to determie the weights of perceptros i each layer of the eural etwork. Traiig of eural etworks ca be improved by pruig (Castellao, Faelli ad Pelillo, 1997; Parekh, Yag ad Hoavar, 2000); which is removig redudat euros or weight vectors. Figure 1.5 Feed forward artificial eural etworks Oe of the mai challeges of the multilayer perceptro is fidig the optimal umber of layers. Uderestimatio ca lead to poor geeralizatio while overestimatio ca lead to overfittig, also traiig with the back propagatio algorithm may fid solutios slower that other machie learig algorithms. Aalysis 10

11 Naïve Bayes The Naïve Bayes algorithm ca be defied as a classificatio algorithm based o Bayes rule that assumes the attributes X1... X (iput sample) are all coditioally idepedet of oe aother, give Y (class label) (Mitchell, 2010). Naïve Bayes classifiers are statistics-based learig algorithms. Bayes classifiers use coditioal probabilities of radom variables i their algorithms. For example, the probability that it is raiig ad cloudy P(x = Raiig y = cloudy) is higher tha the probability that it is raiig ad suy P(x = Raiig y = suy). formula: Classificatio with the aïve Bayes algorithm is determied by the P(class iput) = P(iput class)p(class) P(iput) This essetially is the same as maximizig the umerator because P(iput) is the same for all classes. The probability of class P(class) is computed by the umber of traiig examples i class c divided by the total umber of traiig examples. P(class) = N c N Aalysis 11

12 To estimate the coditioal desities P(iput class), the aïve Bayes classifier assumes that all parameters of the iput vector are coditioally idepedet. If the iput is represeted as a feature vector of iput = P(x 1, x 2, x ), x 1, x are coditioally idepedet. A way to estimate a class coditioal desity is usig a multiomial class coditioally desity. As all iput features are assumed to be coditioally idepedet give the class, P(iput class) is simple the product of the probability of each iput feature give the class. P(x 1, x 2, x class ) = P(x i class) The Naïve Bayes algorithm s computatioal time for traiig is sigificatly shorter tha other machie learig algorithms (Kotsiatis, 2007). Aalysis 12

13 Support vector machies (SVM) Like all supervised machie learig techiques, the goal of support vector machies is to idetify patters i a traiig set ad the use these idetified patters to predict future usee cases. Support vector machies (Cortes ad Vapik, 1995) do this by describig a hyper plae that separates the traiig data whist maximizig the distace separatig the two classes. Figure 1.6 A graphical represetatio of support vector machies. Figure 1.6 is a graphical represetatio of support vector machies. The SVM aim is to idetify the hyper plae "w. x b = 0" that separates the classes i the traiig set by the distace 2M, where M is the maximum distace from ay class to the separatig hyper plae. The iput data o the margis act as support, ad hece are called support vectors, ad oly these poits are required to predict future classificatios which make the Aalysis 13

14 data required for classificatio much smaller tha the size of the traiig set. The task of fidig the optimal separatig hyper plae ca be mathematically represeted as a dual optimizatio problem ad solved usig quadratic programmig. Mathematical represetatio of SVMs From geometry, the distace betwee two hyper plaes; wx b = 1 ad wx b = +1 is 2/ w. Support vector machies aim to maximize this margi whilst meetig the costraits that: w. x b 1 whe class label y i = +1 ad 2.1 w. x b +1 whe class label y i = If equatios 2.1 ad 2.2 are combied ito oe equatio, it results i: y i (w. x i b) Hece the mathematical problem of SVMs is to: Miimize w 2 2 Subject to y i (w. x i b) Aalysis 14

15 This optimizatio problem ca be solved usig Lagragia multiplier method. The SVM problem as a Lagragia is: ζ(w, b, α) = w 2 2 α i [y i (w. x i ) 1] 2.5 α i 0 i Whe the derivative of ζ(w, b, α) with respect to w ad b is computed, it implies that: w = α i y i x i ; α i y i 2.6 Substitutig the weight vector back ito the Lagragia equatio (2.5) results i the SVM dual optimizatio problem: Maximize α i 1 2 y iy j α i α j (x i, x j ) i,j=1 Subject to α i 0 i 2.7 α i y i = 0 The coversio of the origial SVM problem (2.5) usig the Lagragia multiplier to the SVM dual optimizatio problem (2.7) makes the traiig data (x i, x j ) occur oly as dot products. This product is usually substituted though kerel fuctios. For example, i WEKA (Hall et al., Aalysis 15

16 2009) there are several implemetatios of kerel fuctios that ca be used to determie a SVM learig model. Usig differet kerel fuctios may result i better or poorer accuracy, depedig o the machie learig problem. SVMs ca perform either as well as or sigificatly better tha competig methods i most machie learig cotexts (Burges, 1998) ad so are a logical first choice for webometrics. However, the o free luch theorem (Wolpert ad Macready, 1997) suggests that o algorithm ca outperform all others i all machie learig problems. So eve though SVMs most times outperform other machie learig algorithms, it is worth ivestigatig if ay algorithm will be best suitable for a particular learig problem. Refereces Bradford, J., Kuz, C., Kohavi, R., Bruk, C. ad Brodley, C. (1998) Pruig decisio trees with misclassificatio costs, I Machie Learig: ECML- 98 SE - 18, Lecture Notes i Computer Sciece, Nédellec, C. ad Rouveirol, C. (eds.), Spriger Berli Heidelberg, pp , [olie] Available from: Breima, L., Friedma, J., Stoe, C. J. ad Olshe, R. A. (1984) Classificatio ad Regressio Trees, Wadsforth Iteratioal Group. Burges, C. J. C. (1998) A Tutorial o Support Vector Machies for Patter Recogitio, Data Miiig ad Kowledge Discovery, Higham, MA, USA, Kluwer Academic Publishers, 2(2), pp , [olie] Available from: Castellao, G., Faelli, A. M. ad Pelillo, M. (1997) A iterative pruig algorithm for feedforward eural etworks, Neural Networks, IEEE Trasactios o, 8(3), pp Aalysis 16

17 Cortes, C. ad Vapik, V. (1995) Support-Vector Networks, Machie Learig, 20, p. 273; ; Cover, T. ad Hart, P. (1967) Nearest eighbor patter classificatio, Iformatio Theory, IEEE Trasactios o, 13(1), pp Guo, G., Wag, H., Bell, D., Bi, Y. ad Greer, K. (2003) KNN Model-Based Approach i Classificatio, I O The Move to Meaigful Iteret Systems 2003: CoopIS, DOA, ad ODBASE SE - 62, Lecture Notes i Computer Sciece, Meersma, R., Tari, Z., ad Schmidt, D. (eds.), Spriger Berli Heidelberg, pp , [olie] Available from: Hall, M., Frak, E., Holmes, G., Pfahriger, B., Reutema, P. ad Witte, I. H. (2009) The WEKA data miig software: a update, ACM SIGKDD Exploratios Newsletter, New York, NY, USA, ACM, 11(1), pp , [olie] Available from: Highleyma, W. H. (1962) Liear Decisio Fuctios, with Applicatio to Patter Recogitio, Proceedigs of the IRE, 50(6), pp Kotsiatis, S. B. (2013) Decisio trees: a recet overview, Artificial Itelligece Review, Spriger Netherlads, 30(4), pp , [olie] Available from: Kotsiatis, S. B. (2007) Supervised Machie Learig: A Review of Classificatio Techiques, I Proceedigs of the 2007 coferece o Emergig Artificial Itelligece Applicatios i Computer Egieerig: Real Word AI Systems with Applicatios i ehealth, HCI, Iformatio Retrieval ad Pervasive Techologies, Amsterdam, The Netherlads, The Netherlads, IOS Press, pp. 3 24, [olie] Available from: Kotsiatis, S. B., Zaharakis, I. D. ad Pitelas, P. E. (2006) Machie learig: a review of classificatio ad combiig techiques, Artif.Itell.Rev., Norwell, MA, USA, Kluwer Academic Publishers, 26(3), pp , [olie] Available from: La, L., Guo, Q., Yag, D. ad A Cao, Q. (2012) Multiclass Boostig with Adaptive Group-Based knn ad Its Applicatio i Text Categorizatio, Mathematical Problems i Egieerig, [olie] Available from: Mitchell, T. M. (2010) Geerative ad Discrimiative Classifiers: Naive Bayes ad Logistic Regressio, I Machie Learig, pp Aalysis 17

18 Parekh, R., Yag, J. ad Hoavar, V. (2000) Costructive eural-etwork learig algorithms for patter classificatio, Neural Networks, IEEE Trasactios o, 11(2), pp Quila, J. R. (1993) C4.5: programs for machie learig, Sa Fracisco, CA, USA, Morga Kaufma Publishers Ic. Roseblatt, F. (1962) Priciples of eurodyamics; perceptros ad the theory of brai mechaisms, Washigto, Sparta Books. Ruck, D. W., Rogers, S. K., Kabrisky, M., Oxley, M. E. ad Suter, B. W. (1990) The multilayer perceptro as a approximatio to a Bayes optimal discrimiat fuctio., IEEE trasactios o eural etworks / a publicatio of the IEEE Neural Networks Coucil, 1(4), pp , [olie] Available from: Rumelhart, D. E., Hito, G. E. ad Williams, R. J. (1986) Learig represetatios by back-propagatig errors, Nature, 323, pp , [olie] Available from: citeulike-article-id: Utgoff, P. (1989) Icremetal Iductio of Decisio Trees, Machie Learig, Kluwer Academic Publishers-Pleum Publishers, 4(2), pp , [olie] Available from: Wolpert, D. H. ad Macready, W. G. (1997) No free luch theorems for optimizatio, IEEE Trasactios o Evolutioary Computatio, IEEE, 1(1), pp , [olie] Available from: (Accessed 27 May 2014). Aalysis 18