One Against One or One Against All : Which One is Better for Handwriting Recognition with SVMs?

Transcription

1 One Against One or One Against All : Whih One is Better for Handwriting Reognition with SVMs? Jonathan Milgram, Mohamed Cheriet, Robert Sabourin To ite this version: Jonathan Milgram, Mohamed Cheriet, Robert Sabourin One Against One or One Against All : Whih One is Better for Handwriting Reognition with SVMs? Guy Lorette Tenth International Workshop on Frontiers in Handwriting Reognition, Ot 2006, La Baule (Frane), Suvisoft, 2006 <inria > HAL Id: inria Submitted on 5 Ot 2006 HAL is a multi-disiplinary open aess arhive for the deposit and dissemination of sientifi researh douments, whether they are published or not The douments may ome from teahing and researh institutions in Frane or abroad, or from publi or private researh enters L arhive ouverte pluridisiplinaire HAL, est destinée au dépôt et à la diffusion de douments sientifiques de niveau reherhe, publiés ou non, émanant des établissements d enseignement et de reherhe français ou étrangers, des laboratoires publis ou privés

2 One Against One or One Against All : Whih One is Better for Handwriting Reognition with SVMs? Jonathan Milgram Mohamed Cheriet Robert Sabourin Éole de Tehnologie Supérieure, Montréal, Canada milgram@liviaetsmtla mohamedheriet@etsmtla robertsabourin@etsmtla Abstrat The one against one and the one against all are the two most popular strategies for multi-lass SVM; however, aording to the literature review, it seems impossible to onlude whih one is better for handwriting reognition Thus, we ompared these two lassial strategies on two different handwritten harater reognition problems Several post-proessing methods for estimating posterior probability were also evaluated and the results were ompared with the ones obtained using MLP Finally, the one against all strategy appears signifiantly more aurate for digit reognition, while the differene between the two strategies is muh less obvious with upper-ase letters Besides, the one against one strategy is substantially faster to train and seems preferable for problems with a very large number of lasses To onlude, SVMs allow signifiantly better estimation of probabilities than MLP, whih is promising from the point of view of their inorporation into handwriting reognition systems Keywords: Support Vetor Mahine (SVM), Multi-lass Classifiation, Posterior Probability Estimation 1 Introdution In the 1990s, a new type of learning algorithm was developed: the Support Vetor Mahine (SVM) As shown in the Burges tutorial [1], SVM has several interesting properties for pattern reognition Moreover, thanks to improved omputing power and the development of fast learning algorithms, it is now possible to train SVM in real-world appliations However, although SVM has attrated a great deal of attention in the mahine learning ommunity, the multilass SVM is still an ongoing researh issue The existing methods an roughly be divided between two different approahes: the single mahine approah, whih attempts to onstrut a multi-lass SVM by solving a single optimization problem, and the divide and onquer approah, whih deomposes the multilass problem into several binary sub-problems, and builds a standard SVM for eah The most popular deomposing strategy is probably the one against all, whih onsists of building one SVM per lass, trained to distinguish the samples in a single lass from the samples in all remaining lasses Another popular strategy is the one against one, whih builds one SVM for eah pair of lasses On the other hand, more omplex deomposition shemes based on error orreting output odes (ECOC) have been introdued by Diettrih & Bakiri [2] and more reently extended by Allwein et al [3] A omparison of several multi-lass SVM methods (2 single mahine and 3 divide and onquer ) has been realized by Hsu & Lin [4] The results observed are very similar; however, the authors onlude that one against one is more pratial, beause the training proess is quiker Moreover, as to the laim put forward by Allwein et al [3] that one against one and other ECOC are more aurate than the one against all strategy, Rifkin & Klautau [5] disagree, arguing that the one against all strategy is as aurate as any other approah, assuming that the SVMs are well tuned Thus, aording to the literature review, it seems impossible to onlude whih multi-lass SVM is better for handwriting reognition For this reason, we hose to ompare the two most popular strategies, whih are one against all and one against one On the other hand, generally in handwriting reognition appliations, the lassifier only ontributes to a small part of the final deision It is essential, then, that the output of the lassifier is a alibrated onfidene measure, like posterior probability However, although standard SVMs do not provide suh probabilities, a simple post-proessing method for mapping the outputs of a single SVM into posterior probabilities has been proposed by Platt [6] and improved by Lin et al [7] Moreover, while many methods for estimating probabilities with the one against one strategy have been proposed [8, 9, 10], probability estimation with the one against all strategy has not, to the best of our knowledge, been studied The likely reason for this is that, with this strategy, mapping the outputs of eah SVM separately seems suffiient for estimating multilass probabilities; but, as we will see, this is not neessarily the best solution The remainder of this paper is organized as follows Setion 2 presents our experimental protool (database, baseline lassifier, and omparison riteria) Setion 3 desribes the one against all strategy, and ompares two post-proessing methods to estimate posterior probability Setion 4 desribes the one against one strategy, and ompares three methods for ombining the probabilities by eah SVM Finally, setion 5 ompares the two strategies in terms of omplexity and auray, and setion 6 onludes with some pratial suggestions

3 2 Experimental Protool The experiments were onduted on a personal omputer with 19 GHz CPU and 1 Go of RAM All the SVMs were trained with the LIBSVM software [11] We used the C-SVM with a Gaussian kernel The kernel parameter γ and the regularization parameter C were empirially optimized by minimizing the on the validation dataset 21 Database We used the NIST-SD19 database [12], whih ontains the full-page binary images of Handprinted Sample Forms (HSF) from 3,600 writers In our experiments, we used only the images of isolated handwritten digits and upperase letters The number of samples in eah dataset is reported in Table 1 The training datasets ontain exatly the same number of examples in eah lass, whih are the first images from the hsf_{0,1,2,3} orpus The validation datasets are omposed of the remaining images from hsf_{0,1,2,3} for the digit database and of all images from the hsf_4 orpus for the letter database Finally, the testing datasets are omposed of all the images from the hsf_7 orpus Table 1 Number of samples in eah dataset Digit Letter Training 195,000 43,160 Validation 28,123 11,941 Testing 60,089 12,092 We hose to use the same feature extration proedure as Oliveira et al [13] Indeed, this feature spae has been used on the same digit dataset and made it possible to obtain an aurate lassifiation Aording to this method, eah image is divided into six zones: 3 rows and 2 olumns In eah zone, 22 omponents are extrated: 13 onavity measures, 8 orresponding to the histogram of the ontour diretions and one related to the surfae of the harater Finally, we obtained 132 disriminative features, normalized to between 0 and 1 22 Baseline Classifier We have eleted to use a Multi-Layer Pereptron (MLP) as the baseline lassifier beause this type of artifiial neural network makes it possible to estimate aurate posterior probabilities and has been widely inorporated into handwriting reognition systems For our experimentation, we used the same type of topology as Oliveira et al [13], in whih an MLP is used for reognizing handwritten numerial strings The network used has one hidden layer The neurons of the input and the output layers are fully onneted to the neurons of the hidden layer, and the transfer funtion is the sigmoid funtion Furthermore, the network is trained with a sequential gradient desent with momentum applied to a sum-of-squares error funtion The s obtained with MLP are reported in Table 2 and ompared with another lassial lassifier: the k-nearest Neighbor Table 2 Error rate obtained on the testing dataset Digit Letter k-nn 135% 760% MLP 080% 381% Let us note that the number of hidden neurons (h = 80 for digit, and h = 100 for letter) and the number of neighbors (k = 1 for digit, and k = 3 for letter) are fixed using the validation dataset 23 Comparison Criteria To ompare the different approahes in terms of auray, the simplest way would be to evaluate the on the testing dataset, but this value is often not aurate enough For this reason, an error funtion is generally used for omparing the various probability estimates We hose to use the negative log-likelihood: n log ˆP ( k x k ), (1) k=1 where k denotes the label of the sample x k In addition, we propose to use a third measure based on the rejet option Indeed, if the posterior probabilities of the data lasses are known exatly, then, as Chow demonstrated in [14], the optimal rejet option is to rejet a sample x if: max ( P( j x) ) < T (2) j =1,, Then, the threshold T defines the rate of samples rejeted and onsequently the among the samples aepted Thus, a omplete desription of reognition performane is given by the error-rejet tradeoff, whih is obtained by varying T An example is shown in Figure 1 However, in real appliations, suh probabilities are affeted by signifiant estimate errors, and the better the probabilities estimate is, the better the error-rejet tradeoff is Thus, we propose to evaluate the rejetion rate neessary to derease the to a speifi value (01% for digits and 05% for letters) Figure 1: Error-rejet tradeoff obtained with the baseline lassifier on the digit dataset

4 3 The One Against All Strategy 31 Desription The one against all strategy onsists of onstruting one SVM per lass, whih is trained to distinguish the samples of one lass from the samples of all remaining lasses Usually, lassifiation of an unknown pattern is done aording to the maximum output among all SVMs 32 Probability Estimation The most intuitive approah to estimate posterior probability with the one against all strategy is to separately map the outputs of eah SVM into probability using the method proposed by Platt [6], whih onsist of using an additional sigmoid: 1 ˆP ( j f j ) = 1+ exp A j f j + B j, (3) where f j denotes the output of the SVM trained to separate the lass j from all the others Then, for eah sigmoid the parameters A j and B j are optimized by minimizing the loal negative log-likelihood: n t k log( p k ) + ( 1 t k )log( 1 p k ), (4) k=1 where, p k denotes the output of the sigmoid and t k the probability target To solve this optimization problem, Platt [6] proposes using a model-trust minimization algorithm based on the Levenberg-Marquardt algorithm However, Lin et al [7] showed that there are some problems with this algorithm and propose using another minimization algorithm based on Newton s method with baktraking line searh However, nothing guarantees that: ˆP j f j = 1 (5) j=1 For this reason, it seems preferable to normalize the probabilities as follows: ˆP ˆP ( j x) ( j f j ) = (6) ˆP j ' f j ' j '=1 Another approah to estimate posterior probability with the one against all strategy would be to exploit the outputs of all SVMs to estimate overall probabilities In order to do this, we propose using the softmax funtion, whih an be regarded as a generalization of the sigmoid funtion for the multi-lass ase Thus, in the same spirit as Platt s algorithm, we use a parametri form of the softmax funtion: exp( A j f j ( x) + B j ) ˆP( j x) =, (7) exp A j ' f j ' x j '=1 ( + B j ' ) and derive the parameters A j and B j by minimizing the global negative log-likelihood (Eq 1) Thus, it is neessary to onstrut a dataset of SVM outputs, whih will be used to fix the parameters of sigmoid and softmax funtions The easiest way to do this is to use the same training samples used to fit SVMs; but, as pointed out by Platt [6], using the same data twie, an sometimes lead to a disastrously biased estimate Therefore, it is preferable to derive an unbiased training set of the SVM outputs A first solution would be to use a validation dataset; but, in our ase, the number of samples in eah lass is not proportional to the prior probability For this reason, it seems preferable to use ross-validation Then, the training dataset was split into four parts Eah of four SVMs is trained on permutations of three out of four parts, and the SVM outputs are evaluated on the remaining fourth part Finally, the union of all four sets of SVM outputs forms an unbiased dataset, whih an be used to fix the parameters of funtions Furthermore, one the parameters are fixed, the final SVM is trained on the entire training set 33 Experimental Results Firstly, we tested the one against all strategy with the lassial deision making, whih diretly exploits the SVM outputs The s obtained on the testing datasets are 063% with digits and 324% with letters Thereafter, we implemented the two approahes for mapping the SVM outputs into probabilities The results obtained on the testing dataset are reported in Table 3 and Table 4 Considering these results, a number of remarks an be derived First, the two post-proessing methods allow a slight redution of the on the letter dataset, while no improvement is observed on the digit dataset Seond, as we thought, the is not aurate enough in omparing the various estimates Indeed, while the s obtained with the two methods are similar, the rejetion rates and the negative log-likelihood () are signifiantly different Finally, it appears that it is better for posterior probability estimation to optimize globally a softmax funtion than loally several sigmoid funtions Table 3 Results obtained with the one against all strategy on the digit dataset - no rejetion - rejetion rate - 01% of error - sigmoid 064% 373% 1,517 softmax 063% 230% 1,310 Table 4 Results obtained with the one against all strategy on the letter dataset - no rejetion - rejetion rate - 05% of error - sigmoid 317% 1317% 1,570 softmax 318% 1007% 1,375

5 4 The One Against One Strategy 41 Desription The one against one strategy, also known as pairwise oupling, all pairs or round robin, onsists in onstruting one SVM for eah pair of lasses Thus, for a problem with lasses, (-1)/2 SVMs are trained to distinguish the samples of one lass from the samples of another lass Usually, lassifiation of an unknown pattern is done aording to the maximum voting, where eah SVM votes for one lass 42 Probability Estimation After mapping the output of eah SVM into probability with a sigmoid funtion, the task is to express the global posterior probabilities ˆP( j x) as funtions of the loal posterior probabilities, ˆP j where f j, j ' denotes the output of the SVM trained to distinguish lass ω j from lass ω j Various methods have been proposed in the literature We hose to ompare three of these: Method 1 Prie et al [8] onsidered that for all lasses j : P( j, j ' x) ( 2)P( j x) = 1, (8) j '=1, j ' j where j, j ' denotes the union of lasses j and j ' Then, using: P( j x) ˆP j P( j, j ' x), (9) it is possible to derive the following expression: 1 ˆP( j x) = 1 j'=1, j' j ˆP j f j, j' 2 (10) However, sine nothing guarantees that the sum of all the probabilities is 1, we must divide eah estimate ˆP( j x) by ˆP( j x) Method 2 j=1 In a different way, Hastie & Tibshirani [9] proposed using an iterative algorithm to estimate the posterior probabilities = ˆP( j x), whih minimizes the Kullbak-Leibler distane between r jj ' = ˆP j and µ jj ' = + ', that is: n jj ' r jj ' log r jj ' + (1 r jj ' )log 1 r jj ' µ jj ' 1 µ, (11) j< j ' jj ' where n jj ' denotes the number of training samples in the lasses j and j ' To this end, they start with simple non-iterative estimates: j = 2 ( + 1) r jj ' j '=1, j '" j µ, (12) and repeat ( j = 1, 2,,,1, ) until onvergene: 1 µ jj ' + ', j ' j 2! 3 p j '=1 j ' j '=1, j ' j j '=1, j ' j n jj ' r jj ' n jj ' µ jj ' In pratie, we used the following stopping ondition: ( (t) + (t + 1) ) , (13) j=1 where (t) denotes the atual values of and (t 1) the previous values of Method 3 More reently, Hamamura et al [10] proposed a ombination based on the assumption that disriminant funtions are independent of one another Then, sine prior probabilities are all the same, posterior probabilities an be estimated by: ˆP( j x) = j '=1, j ' j j ''=1 j '=1, j ' j '' 43 Experimental Results ˆP j ˆP ( j '' f j '', j ' ) (14) Firstly, we tested the one against one strategy with the lassial voting rule The s obtained on the testing datasets are 071% with digits and 329% with letters Thereafter, we implemented the three methods for ombining probabilities The results obtained on the testing dataset are reported in Table 5 and Table 6 Considering these results, two remarks an be made First, the last method is less aurate than the first two, whih yield signifiantly better rejetion rates and negative log-likelihoods on the two datasets Seond, although the results obtained with the first two methods are omparable, the first method seems slightly more aurate than the seond Indeed, the first method yields better on the letter dataset and better negative log-likelihoods on the two datasets Moreover, the first method has the advantage of being non-iterative and is thus faster than the seond method

6 Table 5 Results obtained with the one against one strategy on the digit dataset - no rejetion - rejetion rate - 01% of error - method 1 070% 349% 1,483 method 2 069% 337% 1,604 method 3 070% 431% 1,825 Table 8 Comparison in terms of auray on the letter dataset - no rejetion - rejetion rate - 05% of error - MLP 381% 2283% 2,923 SVM - OAO 322% 1122% 1,421 SVM - OAA 318% 1007% 1,375 Table 6 Results obtained with the one against one strategy on the letter dataset - no rejetion - rejetion rate - 05% of error - method 1 322% 1122% 1,421 method 2 337% 1136% 1,548 method 3 329% 1389% 2,197 5 Comparison of the two strategies We an now try to answer to the question raised in our title With this in mind, we ompared the two strategies in terms of auray, but also in terms of omplexity 51 Comparison in terms of auray The results obtained on the testing dataset are reported in Table 7 and Table 8, and the error-rejet tradeoffs are shown in Figure 2 and Figure 3 Conerning the one against all strategy (OAA) the softmax funtion is used for probability estimation, while the method 1 is used for ombining the probabilities of the one against one strategy (OAO) Table 7 Comparison in terms of auray on the digit dataset - no rejetion - rejetion rate - 01% of error - MLP 080% 630% 2,591 SVM - OAO 070% 349% 1,483 SVM - OAA 063% 230% 1,310 Figure 2: Comparison of the error-rejet tradeoff obtained on the digit dataset Figure 3: Comparison of the error-rejet tradeoff obtained on the letter dataset Considering these results, two onlusions an be made First, in agreement with the literature, SVMs allow more aurate lassifiation than a lassial MLP Seond, in disagreement with Allwein et al [3], in our experiments, the one against all strategy is more aurate than the one against one Indeed, if the differene between the two strategies is small on the letter dataset, it is signifiant on the digit dataset 52 Comparison in terms of omplexity Two types of omplexity must be onsidered: The omplexity of the training proess It an seem logial that the total training time with the one against one strategy is larger than with the one against all, beause it is neessary to train more binary lassifiers; but it is not true when the binary lassifiers are SVMs Indeed, the training time of an SVM inreases more than linearly with the number of training samples Thus, sine eah sub-problem involves a small number of training samples and is easier to solve, it is quiker to train the (-1)/2 SVMs of the one against one strategy than the SVMs of the one against all strategy In ours experiments, the total training time is approximately divided by 12 with letters (see Table 9) and by 50 with digits (see Table 10) The omplexity of the deision making proess Again, it an seem logial that the deision making with the one against one strategy is more omplex than with the one against all, beause it is neessary to evaluate more deision funtions; but, as previously, it is not neessarily true with SVMs Indeed, the omplexity of an SVM s deision making is diretly linked to the

7 number of support vetors (SVs), and although the deision making is more ompliated in the multi-lass ase, it is reasonable to onsider that the omplexity is proportional to the total number of support vetors 1 However, in our experiments the one against all strategy uses more support vetors than the one against one (48% more for digits, and 21% more for letters) Table 9 Comparison of the two strategies in terms of omplexity on the digit dataset OAO OAA number of SVMs total training time 36 min 32 h 17 number of SVs 5,753 8,514 Table 10 Comparison of the two strategies in terms of omplexity on the letter dataset OAO OAA number of SVMs total training time 4 min 51 min number of SVs 9,152 11,109 6 Conlusion Finally, our answer to the title question will depend on what the problem is! Indeed, it is not reasonable to laim that one strategy is always better that the other; but aording to the appliation onstraints, the number of lasses, and the number of training samples, it will be one or the other of the two strategies that will be more suitable to solve the lassifiation problem Thus, onsidering the onlusions of the previous setion, some suggestions an be made as to whih strategy is best for a speifi problem For problems with few lasses, like digit reognition, the one against all strategy seems signifiantly more aurate; while for problems with more lasses, like Latin letter reognition, the differene of auray between the two strategies seems muh less signifiant Lastly, for problems with a very large number of lasses, like Chinese or Japanese ideogram reognition, we suspet that the unbalane of the number of the samples auses problem with the one against all strategy, espeially when it has few training samples per lass Moreover, the one against one strategy, whih is more modular, is more suitable for speeding up the deision making proess by ombining with other lassifiers [15] Furthermore, if the number of training samples is very large, the training time an 1 Notes that in the multi-lass ase, the total number of support vetors is not neessarily equal to the sum of the number of support vetors of eah SVM, beause a training sample an be a support vetor for several SVMs beome problemati, and then the one against one strategy appears more suitable for pratial use To onlude, we have shown in this paper that appropriate post-proessing make it possible to estimate aurate posterior probabilities with SVMs Thus, these promising results open the way to new perspetives with respet to inorporating SVMs into handwriting reognition systems Referenes [1] CJC Burges, "A tutorial on support vetor mahines for pattern reognition", Data Mining and Knowledge Disovery, vol 2, pp , 1998 [2] TG Dietterih and G Bakiri, "Solving Multilass Learning Problems via Error-Correting Output Codes", Journal of Artifiial Intelligene Researh, vol 2, pp , 1995 [3] EL Allwein, RE Shapire, and Y Singer, "Reduing Multilass to Binary: A Unifying Approah for Margin Classifiers", Journal of Mahine Learning Researh, vol 1, pp , 2000 [4] C-W Hsu and C-J Lin, "A omparison of methods for multi-lass support vetor mahines", IEEE transations on Neural Networks, vol 13, pp , 2002 [5] R Rifkin and A Klautau, "In defene of one-vs-all lassifiation", Journal of Mahine Learning Researh, vol 5, pp , 2004 [6] JC Platt, "Probabilities for SV Mahines", in Advanes in Large Margin Classifiers, A Smola, P Bartlett, B Shölkopf, and D Shuurmans, Eds, MIT Press, pp 61-74, 1999 [7] H-T Lin, C-J Lin, and RC Weng, "A note on Platt's probabilisti outputs for support vetor mahines", Tehnial Report, National Taiwan University, 2003 [8] D Prie, S Knerr, L Personnaz, and G Dreyfus, "Pairwise Neural Network Classifiers with Probabilisti Outputs", in Neural Information Proessing Systems, MIT Press, pp , 1995 [9] T Hastie and R Tibshirani, "Classifiation by pairwise oupling", The Annals of Statistis, vol 26, pp , 1998 [10] T Hamamura, H Mizutani, and B Irie, "A multilass lassifiation method based on multiple pairwise lassifiers", International Conferene on Doument Analysis and Reognition, pp , Edinburgh, Sotland, August 3-6, 2003 [11] C-C Chang and C-J Lin, "LIBSVM: a library for support vetor mahines", Tehnial Report, National Taiwan University, 2001 [12] PJ Grother, "NIST Speial Database 19-Handprinted Forms and Charaters Database", Tehnial Report, National Institute of Standards and Tehnology, 1995 [13] LS Oliveira, R Sabourin, F Bortolozzi, and CY Suen, "Automati reognition of handwritten numerial strings: a reognition and verifiation strategy", IEEE Transations on Pattern Analysis and Mahine Intelligene, vol 24, pp , 2002 [14] CK Chow, "On optimum reognition error and rejet tradeoff", IEEE Transations on Information Theory, vol 16, pp 41-46, 1970 [15] J Milgram, M Cheriet, and R Sabourin, "Speeding Up the Deision Making of Support Vetor Classifiers", International Workshop on Frontiers in Handwriting Reognition, pp 57-62, Tokyo, Japan, Otober 26-29, 2004