Int. J. Sgnal and Imagng Systems Engneerng, Vol. 7, No. 4, 2014 245 bout retranng rule n mult-expert ntellgent system for sem-supervsed learnng usng SVM classfers D. Barbuzz* and G. Prlo Department of Computer Scence, Unversty of Bar ldo Moro, Bar, Italy Emal: donato.barbuzz@unba.t Emal: guseppe.prlo@unba.t *Correspondng author D. Impedovo Department of Electrcal and Electronc Engneerng, Polytechnc of Bar, Bar, Italy Emal: mpedovo@deemal.polba.t bstract: Ths paper proposes three methods n order to retran classfers n a mult-expert scenaro, when new (unknown) data are avalable. In fact, when a mult-expert system s adopted, the collectve behavour of classfers can be used for both recognton ams and selecton of the most proftable samples for system retranng. More specfcally a msclassfed sample for a partcular expert can be used to update the expert tself f the collectve behavour of the multexpert system allows the classfcaton of the sample wth hgh confdence. In addton, ths paper provdes a comparson between the new approach and those avalable n the lterature for semsupervsed learnng usng the SVM classfer by takng nto account four dfferent combnaton technques at abstract and measurement levels. The expermental results, whch have been obtaned usng the handwrtten dgts of the CEDR database, demonstrate the effectveness of the proposed approach. Keywords: feedback-based strateges; sem-supervsed learnng; ntellgent mult-expert system. Reference to ths paper should be made as follows: Barbuzz, D., Prlo, G. and Impedovo, D. (2014) bout retranng rule n mult-expert ntellgent system for sem-supervsed learnng usng SVM classfers, Int. J. Sgnal and Imagng Systems Engneerng, Vol. 7, No. 4, pp.245 251. Bographcal notes: D. Barbuzz receved the Computer Scence degree cum laude n 2011 from the Unversty of Bar ldo Moro. He worked from September to December 2011 as Collaborator n the Interfaculty Center Rete Pugla. Snce 2012, he has been a PhD student at the Department of Computer Scence at the Unversty of Bar. Hs current research nterest s n the feld of mult-expert systems for pattern recognton. G. Prlo receved the Computer Scence degree cum laude n 1986 at the Unversty of Bar, Italy. Snce 1991 he has been ssstant Professor at the same Unversty, where he s currently ssocate Professor. Hs nterests cover the areas of pattern recognton, bometry, computer arthmetc, communcaton and multmeda technologes. He has developed several scentfc projects and publshed over 200 papers. He s assocate edtor of IEEE T-HMS and revewer for T-PMI, T-SMC, T-IP, PR, IJDR, IPL, etc. He was the general co-char of the ICFHR 2012 and EHSP 2013. He s IEEE and IPR member. D. Impedovo receved the MEng degree cum laude n Computer Engneerng n 2005 and the PhD degree n Computer Engneerng n 2009 both from the Polytechnc of Bar (Italy). In 2011 he receved the MSc on Remote Scence Technologes from Bar Unversty. Hs research nterests are n the feld of pattern recognton and bometrcs. He s co-author of more than 20 artcles n both nternatonal journals and conference proceedngs. He receved The Dstncton for the best young student presentaton n May 2009 at the Internatonal Conference on Computer Recognton Systems (CORES endorsed by IPR). He s IPR and IEEE member. Copyrght 2014 Inderscence Enterprses Ltd.
246 D. Barbuzz, G. Prlo and D. Impedovo 1 Introducton pattern recognton system s based on two man processes: enrolment (or tranng) and matchng (or recognton). In the enrolment process, samples of specfc classes are acqured and processed n order to extract relevant features. The features are labelled wth the ground truth and used to generate the model representng the class. However, t may be expensve n terms of space and tme to obtan a large amount of labelled data. On the other hand, t s relatvely easy to collect many unlabelled data that may be used to enlarge the labelled tranng set. Matchng process performs the recognton of the (unknown) nput pattern by comparng t to the enrolled data. Dependng on the specfc scenaro, t has been observed that the recognton accuracy can also be mproved by multexpert fuson (Kttler et al., 1998; Plamondon and Srhar, 2000; Suen and Tan, 2005; Lu et al., 2003; Impedovo et al., 2011; Impedovo et al., 2008; Impedovo et al., 2012; Barbuzz et al., 2012). For the purpose, many approaches have been proposed so far for classfers combnaton, whch dffer n terms of the type of output they combne, system topology and degree of a pror knowledge they use (Kttler et al., 1998; Plamondon and Srhar, 2000; Suen et al., 1992). Furthermore, when a Mult-Expert (ME) system s adopted, as new (unknown) data become avalable, the collectve behavour of classfers can be used not only to recognse patterns wth hgh accuracy but also to select the most proftable samples for updatng the knowledge base of the ndvdual classfers. Ths s partcularly true n supervsed learnng. s already dscussed n the lterature, the achevement of better performance depends on the teraton of the feedback process (Barbuzz et al., 2013), on the combnaton strategy of the ME, but also on data dstrbuton and the smlarty between samples n the feedback set and samples of the testng set (Prlo et al., 2009; Impedovo and Prlo, 2011; Impedovo et al., 2012; Barbuzz et al., 2012; Barbuzz et al., 2013). Ths paper presents a new feedback-based strategy for parallel topology. The basc motvaton of ths work s that the collectve behavour of a set of classfers generally conveys more nformaton than that of each classfer of the set, and ths nformaton can be exploted for system retranng (Suen and Tan, 2005; Prlo and Impedovo, 2011; Prlo et al., 2009; Impedovo and Prlo, 2011). More precsely, as Fgure 1 shows, the approach selects samples that the mult-expert system has classfed wth hgh confdence for updatng the ndvdual experts who have msclassfed the samples themselves. The expermental results, carred out on the handwrtten dgts of the CEDR database, demonstrate the effectveness of the proposed feedback-based strategy also wth respect to the algorthms self-tranng and co-tranng already presented n the lterature (Frnken and Bunke, 2009; Frnken et al., 2011; Blum and Mtchell, 1998; Rattan et al., 2009; Guo et al., 2011). In addton, the expermental results allow us to defne the extent to whch the performance of the new strategy depends on the specfc schema of combnaton decson as well as on data dstrbuton. The paper s organsed as follows. Secton 2 presents the technques of retranng n sem-supervsed learnng. Secton 3 descrbes n detal the new feedback-based strategy. The operatng condtons are presented n Secton 4. The expermental results are reported n Secton 5. Secton 6 presents the concluson of the work and a bref dscusson on future drectons of research. 2 Related work Let us consder a mult-expert system that s already traned and n ts workng phase. Let us suppose that new unlabelled data become avalable over tme; the queston s: how to use these new unknown data? The labellng method s a laborous and costly process. So, new unlabelled data can be used n order to enlarge the tranng set for mprovng the performance of the whole system by sem-supervsed learnng technques. In partcular, two very well known approaches n the lterature for ths purpose are (Frnken and Bunke, 2009; Frnken et al., 2011; Blum and Mtchell, 1998; Rattan et al., 2009; Guo et al., 2011): selftranng and co-tranng. Self-tranng (or self-update; Frnken and Bunke, 2009; Rattan et al., 2009; Guo et al., 2011), as shown n Fgure 2, s based on the concept that a classfer s retraned on ts own most confdent output produced from unlabelled data. The classfer s frst traned on the set of labelled data and, subsequently, several of the new unlabelled nstances that the current classfer has hgh classfcaton confdence are labelled and moved to enlarge ts tranng set. The whole process terates untl some stop rule s satsfed. Co-tranng (or co-update; Blum and Mtchell, 1998; Rattan et al., 2009; Guo et al., 2011), as shown n Fgure 2. Fgure 2 Flow chart for self-update approach (see onlne verson for colours) Fgure 1 x Feedback n the mult-expert parallel system 1 (x t ) 2 (x t ) E(x t ) E (x t ) N (x t ) Fgure 3, s the stuaton under whch two classfers mprove each other. More specfcally, the features set (vew) s parttoned nto two condtonally ndependent subsets (sub-
bout retranng rule n mult-expert ntellgent system 247 vews), so that for a gven class the features used for one classfer must not be correlated wth the features used for the other classfer. In ths manner, the frst expert s updated wth elements confdently recognsed by the other one and vce versa. The process terates untl stopped. Fgure 3 Flow chart for co-update approach (see onlne verson for colours) determnes whether or not the pattern s fed nto the system for updatng the knowledge base of ndvdual classfers. Fgure 4 x Self-update n the mult-expert parallel system 1 (x t ) 2 (x t ) (x t ) E E(x t ) N (x t ) The stoppng crteron n self-tranng and co-tranng s that ether there are no unlabelled nstances left or the maxmum number of teratons has been reached (Guo et al., 2011). Co-tranng and self-tranng have a very strong role, n the current state of the art, n bometrcs template updatng process (Uludag et al., 2004; Rattan et al., 2009); moreover, they have been appled recently even n the feld of handwrtng recognton (Frnken and Bunke, 2009; Frnken et al., 2011). The man result observed for self-tranng on the task of handwrtten word recognton (Frnken and Bunke, 2009) s that the challenge of successful self-tranng les n fndng the optmal trade-off between data qualty and data quantty for retranng. In partcular, f the retranng s done wth only those elements whose correctness can nearly be guaranteed, the retranng set does not change sgnfcantly and the classfer may reman nearly the same, or n other cases t could dscard genune samples whose dstrbuton s far from the one already embedded n the knowledge base, thus resultng n a performance degradaton. Enlargng the retranng set, on the other hand, s only possble at the cost of ncreasng nose,.e. addng mslabelled words to the tranng set. In ths scenaro, the cotranng approach (Frnken et al., 2011) appears to be much more nterestng; n fact, t does not suffer lmtatons of the self-update process, and performance mprovement are more evdent than those observed n the self-tranng case, even f the confdence threshold stll plays a crucal role. Co-tranng can be easly extended from two to n classfers, but the basc observaton s that, once more, even f an ensemble of classfers s avalable, there s no analyss and use of ther common behavour of classfcaton gven the nput to be recognsed. Self-update and co-update n mult-expert parallel system are nvestgated here (see Fgures 4 and 5, respectvely) and compared aganst our approach, where the fnal result E(xt) provded by the mult-expert system, accordng to (xt) provded by the sngle classfer, Fgure 5 x Co-update n the mult-expert parallel system 1 (x t ) 2 (x t ) (x t ) 3 Feedback-based strategy N (x t ) E E(x t ) The new feedback-based strategy for parallel topology s descrbed n the followng. 3.1 Instance selecton from ensemble behavour Let: Cj, for j = 1,2,, M: the set of pattern classes. P xk k 1,2,..., K : the set of pattern to be fed nto the Mult-Expert (ME) system. In ths work, P s consdered to be parttoned nto S subsets: PP,,, P,, P 1 2 s S where Ps xk P k Ns ( s 1) 1, Ns s and Ns K/ S, N s nteger, whch are provded one after the other to the mult-expert system. In partcular, P 1 s used only for learnng and ts data are labelled, whereas P2, P3,, Ps,, PS are used for both classfcaton and learnng (when necessary) and ther data are unlabelled. y s Ω: the label of the pattern x s, C1, C2,..., CM. : the th classfer, = 1,2,,N. F ( k) F k, F k,, F k, F k : the,1,2, r, R numeral feature vector used by for representng the pattern x k P (for the sake of smplcty, here t s assumed that each classfer uses R numeral features).
248 D. Barbuzz, G. Prlo and D. Impedovo KB (k): the knowledge base of after the processng of the subset P k. In partcular, 1 2 M,,, KB k KB k KB k KB k. E: the mult-expert system whch combnes the ndvdual classfer decsons n order to obtan the fnal classfcaton result. Intally, n the frst stage (s = 1), the classfer s traned * usng the patterns xk P P. Therefore, the knowledge 1 base KB (s) of s ntally defned as 1 2 j M,,,,, KB s KB s KB s KB s KB s (1) where for j = 1,2,, M, KB () s ( F (), s F (), s, F (), s, F ()) s (2) j j j j j,1,2, r, R j wth FR, () s beng the set of the rth feature of the th * classfer for the patterns of the class C j that belongs to P. Successvely, the subsets P 2, P 3,, Ps,, PS 1 of unknown samples are provded one after the other to the mult-classfer system both for classfcaton and for learnng. PS s just consdered to be the testng set n order to avod based or too optmstc results. When consderng new data (samples of P 2, P 3,, Ps,, PS 1 ), n order to nspect and take advantage of the common behavour of the ensemble of classfers, the followng smple strategy s proposed and evaluated n ths work: Ex Ext xt t and x t Ps ' : update _ KB scoreme threshold In other words, s updated wth the pattern that t msclassfes consderng the label, obtaned wth a hgh confdence measure, by ME system. It s worth notng that ths strategy (see equaton 3) takes nto account the performance of the ndvdual classfer as well as the performance of the ME system. It s able to select not only samples to be used for the updatng process but also classfers nto whch those samples must be fed n order to mprove the ME performance. 3.2 lgorthm In the feedback-based procedure, each expert s traned on an ntal set of enrolled data, named P 1. batch P f of (unknown) samples s collected over a certan tme. mong all the samples n P f, only those classfed wth hgh confdence by the mult-expert system are used to enlarge the knowledge base of those experts that produced a msclassfcaton. To compute the confdence threshold, P f s used as the valdaton set. The retranng rule computes both the confdence threshold Thr* for each expert and the confdence threshold ThrME for the mult-expert system. Successvely, all samples recognsed wth a confdence equal to or greater than ThrME and msclassfed by a specfc (3) expert are selected. Ths s done to reduce the probablty of mpostor ntroducton (false acceptance) nto the updated tranng set. Ths procedure s presented n lgorthm 1. lgorthm 1: Feedback-based Process 1 Gven: P1 p,, 1 p : M the Intal Tranng Set S -1 Pf Pu pm 1,, pn u2 : the new avalable (unknown) data set KB (k) : the knowledge base of the expert, =1,2,, N 2 For each expert: (a) (u, P 1 ) s the classfcaton of the sample u P f on P 1 (b) thr* s the estmated threshold for End for 3 pply the combnaton rule: E(u,P 1 ) s the classfcaton of the sample u P on P 1 combnng all experts. 4 thrme s the estmated threshold for E 5 Determne for each sample u Pf a classfcaton score of the Mult-Expert system sme Eu, P 1 6 For h = M + 1,, N 7 For each expert (a) If uh, P1 Euh, P1 sme Eu, P1 thrme and (b) Then KB ( k) KB ( k) u End for End for. h However, t s evdent that many new samples wll not be fed nto a specfc classfer and, n general, we could expect to observe performance degradaton f compared to other strateges. Of course, ths phenomenon depends on thresholds, classfer performances and the rato between the new and the old data. However, we have to consder and remark that we are dealng wth already traned and workng classfers, so that ther ntal performances are expected to be on the state of the art. Ths leads to two consderatons: 1 Gven a specfc classfer, the dfference between the confdence values n the case of msclassfcaton and n the case of correct classfcaton could be mputed to the fact that the specfc classfer s unable to represent a specfc class or sample, and no mprovements would be obtaned by ntroducng the new sample n the knowledge base. Ths s partcularly true under the assumpton that strong (not weak) classfers are used. 2 If each classfer n the ensemble were able to recognse exactly the same set of patterns, ther combnaton would be not useful (Impedovo et al., 2010). f
bout retranng rule n mult-expert ntellgent system 249 3 Operatng condtons In ths paper, the handwrtten dgts P { x j 1,2,, 20351} (classes from 0 to 9 ) of the CEDR database (Hull, 1994) have been used. Fgure 6 presents some samples of handwrtten dgts. The DB has been ntally parttoned nto sx subsets: P x, x, x,,, x, 1 1 2 3 12750 P2 x12751,,, x14119, P3 x14120,,, x15488, P4 x15489,,, x16857, P5 x16858,,, x18223, P x,,, x. 6 18224 20351 In partcular, P1P2 P3 P4 P5 represents the set usually ad`opted for tranng when consderng the CEDR DB (Lu et al., 2003). P 6 s the testng data set. Each dgt s zoned nto 16 unform (regular) regons (Prlo and Impedovo, 2011); successvely for each regon the followng set of features were consdered (Lu et al., 2003): 1 Geometrc features: hole, up cavty, down cavty, left cavty, rght cavty, up end pont, down end pont, left end pont, rght end pont, crossng ponts, up extrema ponts, down extrema ponts, left extrema ponts, rght extrema ponts; 2 Contour profles: max/mn peaks, max/mn profles, max/mn wdth, max/mn heght; 3 Intersecton wth lnes: fve horzontal lnes, fve vertcal lnes, fve slant 45 lnes and fve slant +45 lnes. Fgure 6 3.1 Thresholds Samples of handwrtten dgts of the CEDR database To estmate the qualty of the classfcaton output, addtonal nformaton about the classfcaton s necessary to ndcate the degree of confdence of the expert. To compute the confdence measure, the new set of avalable data s used at the same tme to classfy and retran, when necessary. More precsely, the retranng rules compared n ths work are based on ths confdence measure. The retranng rules, for self-tranng and co-tranng, defne a j specfc confdence threshold and all the dgts classfed wth a degree of confdence equal to or greater than ths threshold are added to the expert s knowledge base. The thrd rule, defned at the mult-expert level, selects all the nstances msclassfed, for a partcular expert, and updates t only f that sample produces a hgh classfcaton confdence by the ensemble of experts. The frst two thresholds are set to 0.75 normalsed z- score. The thrd confdence measure s the average between the thresholds by each expert set to 0.75 normalsed z-score. 3.2 Classfer and combnaton technques The expermental system uses three dfferent classfers, named 1, 2 and 3. Each classfer uses a dfferent set of features: 1 uses the feature set no. 1, 2 uses the feature set no. 2 and 1 uses the feature set no. 3. ll classfers perform classfcaton by a Support Vector Machne (SVM). Of course, snce SVM s a bnary (two-class) classfer, mult-class classfcaton s here performed by combnng multple bnary SVMs. In fact, an M-class problem can be decomposed nto M bnary problems wth each separatng one class from all the others (Chang and Ln, 2011). The kernel functon adopted n ths work s the rbf gamma. The performances are certanly nfluenced by the number of features (gamma) and by the tolerance of classfcaton errors n learnng (C) (Lu et al., 2003; Chang and Ln, 2011). Despte the power of the classfer, n mult-expert system the combnaton technque plays a crucal role n the selecton of new patterns to be fed nto the classfer n the proposed approach. In ths work, the followng decson combnaton strateges have been consdered and compared: Majorty Vote (MV), Weghted Majorty Vote (WMV), Sum Rule (SR) and Product Rule (PR). MV consders labels provded by the ndvdual classfers. It s generally adopted f no knowledge s avalable about performance of classfers. WMV can be adopted by consderng weghts related to the performance of ndvdual classfers on a specfc data set (Polkar, 2007). In the case depcted n ths work, t seems to be more realstc. In fact, the behavour of classfers can be evaluated, for nstance, on the new avalable data set. In partcular, let ε be the error rate of the th classfer evaluated on the last avalable tranng set, the weght assgned to s w log 1, where. For the sake of smplcty, let us consder an ME 1 adoptng three base classfers combned by means of a smple majorty vote approach (see Fgure 7). In the case depcted, the knowledge base of 1 wth x wll be updated, whle the second and the thrd experts wll not be updated. In ths case, the 1 performance would be ncreased on the tranng set and on pattern smlar to x, mprovng the overall ME performance. Fgure 7 Examples of updatng requests 1 x y 2 x y E x y 3 x y
250 D. Barbuzz, G. Prlo and D. Impedovo Sum Rule (SR) and Product Rule (PR) take nto account the confdence of each ndvdual classfer gven the nput pattern and the dfferent classes (Kttler et al., 1998). Of course, before combnaton, confdence values provded by dfferent classfers were normalsed by means of z-score. 4 Expermental results In the expermental tests, results are reported n terms of Error Rate (ER). Moreover, for the dfferent learnng strateges the numbers of Selected Samples (SS) from set of new data are reported. The label X-feed refers to the use of the X modalty for the feedback process. Precsely MV, WMV, SR and PR are feedback at ME level adoptng, respectvely, the majorty vote, the weghted majorty vote, the sum rule and the product rule. Table 1 shows the results related to the frst test. In partcular, P 1 s used for tranng and P 6 for testng. P 2, P 3, P 4 and P 5 were ndependently used, one from the other; for feedback learnng, performances were evaluated for each set and the average was fnally reported. The frst column (No-feed) reports results related to the use of P 1 for tranng and of P 6 for testng, wthout applyng any feedback (selected samples are 0). The total amount of new samples s n ths case (about) the 10% of the number of samples of the ntal tranng set (P 1 ). In ths case, expermental results (see Table 1) demonstrate the effectveness of the mult-expert feedback strategy. In MVfeed, WMV-feed and PR-feed, an mprovement of, respectvely, 0.11%, 0.07%, 0.05% can be found compared to self-update technque, whle for MV-feed, WMV-feed, SR-feed and PRfeed, there s an mprovement of, respectvely, 0.12%, 0.07%, 0.04% and 0.01% respect to co-update technque. Despte the few selected samples by the ME, the performance of the whole system outperforms those related self-tranng and co-tranng strateges. Table 2 shows the results related to the second experment. Ths test ams at evaluatng the proposed approach under the condton n whch the number of new pattern s sgnfcant f compared to the ntal tranng one. In partcular, P 1 s used for tranng and P 6 for testng. P 2 P 3 P 4 P 5 s used for feedback learnng. In ths case, the total amount of new samples s 42.86% of the number of samples of the ntal tranng set (P 1 ). Once more, the frst column (No-feed) reports results related to the use of P 1 for tranng and of P 6 for testng, wthout applyng any feedback (selected samples are 0). The behavour observed consderng the use of SVM classfers s smlar to the prevous experment; n fact, MV-feed, WMV-feed and SR-feed, respectvely, saw an mprovement of 0.24%, 0.09% and 0.10% wth respect to the self-update technque, whle for MV-feed, WMV-feed, SR-feed and PR-feed, respectvely, an mprovement of 0.24%, 0.14%, 0.05% and 0.19% can be seen consderng the co-update approach. The results demonstrate that our approach outperforms other approaches n the lterature n terms of both error rate and selected samples requested for the updatng process, when new (unknown) data became avalable. Table 1 SVM feedback P 2, P 3, P 4, P 5 No-feed Self-update Co-update MV-feed WMV-feed SR-feed PR-feed ER ER SS ER SS ER SS ER SS ER SS ER SS 1 2.96 3.01 1256 2.99 2463 2.90 53 2.95 69 2.98 75 3.06 74 2 8.32 8.28 1176 8.19 2542 8.28 104 8.28 120 8.25 129 8.21 129 3 4.09 4.23 1287 4.08 2432 4.17 36 4.17 36 4.20 50 4.21 51 MV 2.63 2.68 2.69 2.57 WMV 1.69 1.79 1.79 1.72 SR 1.46 1.42 1.47 1.43 PR 1.22 1.25 1.21 1.20 Table 2 SVM feedback P 2 P 3 P 4 P 5 No-feed Self-update Co-update MV-feed WMV-feed SR-feed PR-feed ER ER SS ER SS ER SS ER SS ER SS ER SS 1 2.96 3.05 5022 2.68 9852 3.05 212 2,87 277 2.77 299 2.87 294 2 8.32 8.36 4705 8.04 10169 7.99 414 8,08 479 8.04 514 8.08 516 3 4.09 4.32 5147 3.99 9727 4.42 145 4,42 145 4.23 201 4.23 204 MV 2.63 2.82 2.82 2.58 WMV 1.69 1.88 1.93 1.79 SR 1.46 1.46 1.41 X 1.36 PR 1.22 1.22 1.41 1.22
bout retranng rule n mult-expert ntellgent system 251 5 Conclusons and future works Unlabelled data are most often abundant, but obtanng labels s expensve or tme-consumng. Instead of smply labellng all the new (unknown) data or randomly selectng data to be labelled, ths paper shows the possblty of mprovng (selectng specfc nstances) the effectveness of a mult-classfer system by a sutable use of the nformaton extracted from the collectve behavour of the classfers. More precsely, when a new unlabelled data set becomes avalable, the fnal decson (obtaned by combnng the decsons of the ndvdual classfers) has been used to upgrade the knowledge base of the ndvdual classfers, when necessary, accordng to a feedback-based topology. The expermental results (n terms of error rate and accuracy n samples selecton) reported n ths paper demonstrate that the collectve behavour of a set of classfers provdes useful nformaton to mprove system performance and the effectveness of the new feedback-based strategy when compared to other sem-supervsed learnng algorthms n the lterature, as self-tranng and co-tranng. Future works can nspect the possblty of teratve retranng gven a set of new unlabelled samples as well as the possblty to evaluate the method on the varablty of confdence threshold. The applcablty of the strategy n the context of unsupervsed learnng wll be also consdered. References Barbuzz, D., Impedovo, D. and Prlo, G. (2012) Benchmarkng of update learnng strateges on dgt classfer systems, Proceedngs of the 13th Internatonal Conference on Fronters n Handwrtng Recognton, 18 20 September, Captolo, Bar, Italy, pp.35 40. Barbuzz, D., Impedovo, D., Mangn, F.M. and Prlo, G. (2013) Learnng teratve strateges n mult-expert systems usng SVMs for dgt recognton, n Petrosno,. (Ed.): Proceedngs of the 17th Internatonal Conference on Image nalyss and Processng, 11 13 September, Naples, Italy, LNCS, Vol. 8156, pp.121 130. Blum,. and Mtchell, T. (1998) Combnng labeled and unlabeled data wth co-tranng, Proceedngs of the 11th nnual Conference on Computatonal Learnng Theory, 24 26 July, Madson, WI, pp.92 100. Chang, C-C. and Ln, C-J. (2011) LIBSVM: a lbrary for support vector machnes, CM Transactons on Intellgent Systems and Technology, Vol. 2, pp.27:1 27:27. valable onlne at: http://www.cse.ntu.edu.tw/~cjln/lbsvm. Frnken, V. and Bunke, H. (2009) Evaluatng retranng rules for sem-supervsed learnng n neural network based cursve word recognton, Proceedngs of the IEEE 10th Internatonal Conference on Document nalyss and Recognton (ICDR2009), 26 29 July, Catalona, Span, pp.31 35. Frnken, V., Fscher,., Bunke, H. and Fornes,. (2011) Cotranng for handwrtten word recognton, Proceedngs of the IEEE Internatonal Conference on Document nalyss and Recognton (ICDR2011), 18 21 September, Bejng, Chna, pp.314 318. Guo, Y., Zhang, H. and Lu, X. (2011) Instance selecton n semsupervsed learnng, Proceedngs of the 24th Canadan Conference on rtfcal Intellgence, 25 27 May, St. John s, Canada, pp.158 169. Hull, J. (1994) database for handwrtten text recognton research, IEEE Transactons on Pattern nalyss and Machne Intellgence, Vol. 16, No. 5, pp.550 554. Impedovo, D. and Prlo, G. (2011) Updatng knowledge n feedbackbased mult-classfer systems, Proceedngs of the IEEE of Internatonal Conference on Document nalyss and Recognton (ICDR2011), 18 21 September, Bejng, Chna, pp.227 231. Impedovo, D., Prlo, G. and Barbuzz, D. (2012) Supervsed learnng strateges n mult-classfer systems, Proceedngs of 11th Internatonal Conference on Informaton Scence, Sgnal Processng and ther pplcatons (ISSP 2012), 2 5 July, Montreal, Canada, pp.1215 1220. Impedovo, D., Prlo, G. and Petrone, M. (2011) multresoluton mult-classfer system for speaker verfcaton, Expert Systems, Vol. 29, No. 5, pp.442 455. Impedovo, D., Prlo, G. and Refce, M. (2008) Handwrtten sgnature and speech: prelmnary experments on multple source and classfers for personal dentty verfcaton, LNCS, Vol. 5158, pp.181 191. Impedovo, D., Prlo, G., Sarcnella, L. and Stasolla, E. (2010) rtfcal classfer generaton for mult-expert system evaluaton, Proceedngs of the IEEE Internatonal Conference on Fronters n Handwrtng Recognton (ICFHR 2010), 16 18 November, Kolkata, Inda, pp.421 426. Kttler, J., Hatef, M., Dun, R.P.W. and Matas, J. (1998) On combnng classfers, IEEE Transactons on Pattern nalyss Machne Intellgence, Vol. 20, No.3, pp.226 239. Lu, C.L., Nakashma, K., Sako, H. and Fujsawa, H. (2003) Handwrtten dgt recognton: benchmarkng of state-of-the-art technques, Pattern Recognton, Vol. 36, No. 10, pp.2271 2285. Prlo, G. and Impedovo, D. (2011) Fuzzy-zonng-based classfcaton for handwrtten characters, IEEE Transactons on Fuzzy Systems, Vol. 19, No. 4, pp.780 785. Prlo, G., Trullo, C.. and Impedovo, D. (2009) feedback-based mult-classfer system, Proceedngs of the IEEE Internatonal Conference on Document nalyss and Recognton (ICDR 2009), 26 29 July, Barcelona, Span, pp.713 717. Plamondon, R. and Srhar, S.N. (2000) On-lne and off-lne handwrtng recognton: a comprehensve survey, IEEE Transactons on Pattern nalyss and Machne Intellgence, Vol. 22, No. 1, pp.63 84. Polkar, R. (2007) Bootstrap-nspred technques n computatonal ntellgence, IEEE Sgnal Processng Magazne, Vol. 24, No. 4, pp.59 72. Rattan,., Fren, B., Marcals, G.L. and Rol, F. (2009) Template update methods n adaptve bometrc systems: a crtcal revew, Proceedngs of the 3rd Internatonal Conference on Bometrcs, 2 5 June, lghero, Italy, LNCS, Vol. 5558, pp.847 856. Suen, C.Y. and Tan, J. (2005) nalyss of errors of handwrtten dgts made by a multtude of classfers, Pattern Recognton Letters, Vol. 26, No. 3, pp.369 379. Suen, C.Y., Nadal, C., Legault, R., Ma, T.. and Lam, L. (1992) Computer recognton of unconstraned handwrtten numerals, Proceedngs of the IEEE, Vol. 80, No. 7, pp.1162 1180. Uludag, U., Ross,. and Jan,. (2004) Bometrc template selecton and update: a case study n fngerprnts, Pattern Recognton, Vol. 37, No. 7, pp.1533 1542.