Accumulated-Recognition-Rate Normalization for Combining Multiple On/Off-Line Japanese Character Classifiers Tested on a Large Database

Transcription

1 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK Accumulated-Recognton-Rate Normalzaton for Combnng Multple On/Off-Lne Japanese Character Classfers Tested on a Large Database Ondrej Velek, Stefan Jaeger, Masak Nakagawa Graduate School of Technology, Tokyo Unversty of Agrculture and Technology Naka-cho Kogane-sh, Tokyo, , Japan E-mal: velek@hands.e.tuat.ac.jp, stefan@hands.e.tuat.ac.jp, nakagawa@cc.tuat.ac.jp Abstract. Ths paper presents a technque for normalzng lkelhood of multple classfers, allowng ther far combnaton. Our technque generates for each recognzer one general or several stroke-number specfc characterstc functons. A smple warpng process maps output scores nto an deal characterstc. A novelty of our approach s n usng a characterstc based on the accumulated recognton rate, whch makes our method very robust and stable to random errors n tranng data and requres no smoothng pror to normalzaton. In ths paper we test our method on a large database named Kuchbue_d, a publcly avalable benchmark for on-lne Japanese handwrtten character recognton and very often used for benchmarkng new methods. 1. Introducton Combnng dfferent classfers for the same classfcaton problem has become very popular durng the last years [7,9]. In handwrtng recognton, classfer combnatons are of partcular nterest snce they allow brdgng the gap between on-lne and offlne handwrtng recognton. An ntegrated on-lne/off-lne recognton system can explot valuable on-lne nformaton whle off-lne data guarantees robustness aganst stroke order and stroke number varatons. Snce the dfferent nature of on-lne and off-lne data complcates ther combnaton, most approaches combne both types of nformaton ether durng pre-processng;.e. feature computaton [1-3] or, lke ths paper, n post-processng [5-8]. In ths paper, we report experments on the combnaton of on-lne and off-lne recognzers for on-lne recognton of Japanese characters. Multple classfers, especally on-lne and off-lne classfers, very often generate lkelhood values that are ncompatble. The focus of ths paper les on our accumulated-recognton-rate normalzaton technque, whch tres to overcome ths problem by algnng the lkelhood values wth the actual performance of each classfer. At frst we defne the accumulated recognton rate, whch s normalzed to a lnearly growng functon. Usng normalzed lkelhood ensures a far combnaton of classfers. We have ntroduced our warpng technque n [13, 16], and tested ts effcency on a small NTT-AT database. That database s a collecton of patterns wrtten by elderly people, usually by ncorrect wrtng style. We have acheved mprovement of the recognton rate from 89.7% for the best sngle classfer to 94.14% by combnng seven classfers n our combnaton system [13] and n [16] 93.68% by combnng two classfers. However, snce the NTT-AT database s not a collecton of typcal 196

2 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK character patterns and the number of samples and categores s low, a good performance cannot be easly generalzed for common Japanese character patterns. In ths paper we experment wth the benchmark database Kuchbue_d, whch contans onlne Japanese handwrtten characters for whch many researchers have already publshed recognton rates. Ths paper s structured as follows: Secton 2 and 3 outlne the work descrbed n our prevous paper [13] and descrbe the general theoretcal framework of our normalzaton technque based on accumulated recognton rate. In Secton 4 we ntroduce our mproved approach utlzng nformaton about stroke number, frstly publshed n [16]. Secton 5 descrbes sngle classfers and combnaton schemes used n our experments. Secton 6 presents our new results on the Kuchbue_d benchmark set, whch allows comparson of our recognton results on the same benchmark wth other research groups or wth our prevous results on the NTT-AT database (Secton 7). Fnally, Secton 8 concludes ths paper wth a general dscusson of our results. 2. Comparablty of dfferent classfers Let A be a classfer that maps an unknown nput pattern x to one of m possble classes (ω 1,, ω m ), and returns values a = A(x, ω ) denotng ts confdence that x s a member of class ω,. For an deal classfer, each returned value a corresponds to the true probablty of ω gven x: P (ω, x), also called a-posteror probablty, wth a =P (ω, x) 1. In real practce, however, the output values a can merely be approxmatons of the correct a-posteror probabltes. Many classfers do not even output approxmatons but only values related to the a-posteror probabltes. For nstance, A(x, ω ) = a very often denotes the dstance between the nput pattern x and the class ω n a hgh-dmensonal feature space under a gven metrc, sometmes wth a [; ]. In ths case, the best canddate s not the class wth the maxmum value among all a, but the class havng the smallest a ;.e., the class wth the shortest dstance. Also, the scale of output values s generally unknown. For example, from a r = 2a p one cannot predct that class ω r s twce as lkely as class ω p. These nadequaces generally pose lttle problem for a sngle classfer that needs to fnd only k-best canddates. If a sngle classfer rejects patterns wth confdence values below a certan threshold, a relaton between confdence and a-posteror probablty s useful for settng the threshold. However, for combnng several classfers {A, B, C } we necessarly need some relaton among canddates stemmng from dfferent classfers {a =A(x, ω ), b =B(x, ω ), } n order to compare them and to select the best class. To better descrbe the output of classfers, we defne two auxlary functons, n(a k ) and n correct (a k ) countng the overall number of samples and correctly recognzed samples for each output value respectvely: Functon n(a k ) returns the number of test patterns classfed wth output value a k and functon n correct (a k ) returns the number of correctly recognzed test patterns. We begn by countng the number of correctly and ncorrectly recognzed samples for each value: n correct (a k ) and n ncorrect (a k )= n t (a k )- n correct (a k ). Graph 1 and Graph 2 show two exemplary hstograms of these numbers for off-lne and on-lne recognton respectvely. An deal classfer returns only correct answers wth lkelhood values coverng the whole range of possble values. As a matter of fact, most practcal classfers also 197

3 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK return ncorrect answers and do not cover all output values. Graph 1 and Graph 2 llustrate ths for the off-lne classfer - most of the correct output s around a =6 -, whle the on-lne classfer has most of ts correct answers around a=9. The peak of n correct (a) dffers from n ncorrect () n both classfers. Also, both classfers use only a small range of the output nterval [;1] ntensvely. 25 Hstogram of a confdence value a 5 Hstogram of a confdence value a [a ] correct ncorrect [a] correct ncorrect Graph 1: Hstogram for an off-lne recognzer Graph 2: Hstogram for an on-lne recognzer. n correct For a sngle classfer system, only the recognton rate;.e., the number of correctly recognzed patterns () dvded by the number of overall patterns, s of mportance. However, for combnng multple classfers, not only the recognton rate, but also the dstrbuton of n correct ( a k ) and n ncorrect ( a k ) s of nterest. If some classfers provde better recognton rates than the best sngle recognton rate on several sub-ntervals, then we can suppose that by combnng multple classfers the combned recognton rate wll outperform the sngle best rate. Graph 3 and Graph 4 show the recognton rates correspondng to Graph 1 and 2 respectvely. Recognton Rate R(a) [% ] correct nccorect [a ] Recognton Rate R(a) [% ] correct nccorect [a ] Graph 3: Rcg.Rate for an off-lne recognzer. Graph 4: Rcg.Rate for an on-lne recognzer. 3. Normalzaton and Warpng Our goal s to normalze the output of each classfer so that t better reflects the actual classfer performance for each output value and allows drect comparson and combnaton wth outputs from other classfers. The man dea s to turn confdence values nto a monotone ncreasng functon dependng on the recognton rate. After normalzaton, the hghest output should stll defne the most lkely class of an unknown nput pattern x; a > a j should mply that P(ω, x) > P(ω j x). However straghtforward normalzaton of classfer outputs to a monotone ncreasng functon s not possble, because classfer confdence s nether a contnuous nor a monotone functon. 198

4 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK In our new approach we solve ths problem by defnng the accumulated recognton rate (whch s always a contnuous functon) and applyng a warpng process on t. Ths process expands and compresses local subntervals of the horzontal lkelhood-axs to better match the practcal recognton rates acheved, wthout changng the order and wthout addng artfcal error by addtve smoothng. The confdence value s often ether smaller than the value suggested by the recognton rate, whch means that the confdence value s too pessmstc, or t s hgher than the actual recognton rate suggests, whch s then a too optmstc value. We proposed usng a second tranng set to measure the classfer performance for each lkelhood value, gven an approprate quantzaton of lkelhood, and then calbrate lkelhood n a post-processng step accordng to the performance measured. Let A={a,, a,, a max } be a set of lkelhood values wth a beng the lowest and a max beng the hghest lkelhood assgned by the classfer. In our experments, the lkelhood values span the nteger nterval rangng from to 1, and thus A = [;1], max = 1, and a k = k. The calbraton replaces the old lkelhood new values a by ther correspondng recognton rates r = a so that after normalzaton lkelhood and recognton rate are equal: A =R Usng our auxlary functons, we can state ths as follows: new ncorrect ( a ) amax a =. n( a ) Graph 5 and Graph 6 show the accumulated recognton rates before and after normalzaton computed for the off-lne and on-lne recognton rates shown n Graph 3 and Graph 4 respectvely. The error rates depcted n Graph 5 and Graph 6 show the remanng percentage of msclassfed patterns for lkelhood values hgher than a partcular value a. [%].9 Accumulated Recognton Rate.8 Accumulated RR(,1)=Rec.Rate [a ] correct (before) ncorrect (before) correct (after) ncorrect (after) [%].9 Accumulated Recognton Rate.8 Accumulated RR(,1)=Rec.Rate [a ] correct (before) ncorrect (before) correct (after) ncorrect (after) Graph 5: R(<,a >) Accumulated Recognton Rate for an off-lne recognzer before and after normalzaton. Graph 6: R(<,a >) Accumulated Recognton Rate for an on-lne recognzer before and after normalzaton. Note that the accumulated recognton rate s a monotone growng functon over classfer output. Our normalzaton method equals the accumulated lkelhood mass wth the correspondng accumulated recognton rate. Usng the nomenclature ntroduced 199

5 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK above, our normalzaton ensures the followng equaton: as A R =, whch translates 1 1 n( a ) a = n ( a ) for all, wth N beng the overall number of k k correct L k= N k = patterns and L beng the overall lkelhood mass;.e., k max = L n( a k )* a k. For each classfer, we thus normalze the output so that the accumulated probablty functon R(<,a >) becomes a functon proportonal to the classfer output. Accordngly, we adjust each classfer s output by addng the followng adjustment charf(a ): n ( ) k correct a k a = + amax = a = a + charf ( a ) N n ( ) ( ) k correct a k n k correct a = k a ' amax a = = = + amax a = a + charf ( a ) N N where a max s the maxmum possble output of a classfer ( a max = 1 n our experments), N s the number of overall patterns, and r stands for the partally accumulated recognton rate. We call ths classfer-specfc adjustment the characterstc functon [charf ] of a classfer. To compute the [charf ] of a classfer, we need another sample set, whch should be ndependent from the tranng set. Hence, we use data ndependent from the tranng set to ensure a proper evaluaton, although we observed that a characterstc functon depends mostly on the classfer and not the data. To llustrate the effects of normalzaton we now compare the graphs 1-4 wth the correspondng graphs computed after normalzaton;.e., after addng the adjustment. Graph 7 and Graph 8 ncely llustrate the unform dstrbuton of output values after normalzaton, compared to Graph 1 and Graph 2. In both Graphs 7-8, we see that, after normalzaton, the whole output spectrum s used for correct answers. Moreover, ncorrect answers concentrate n the left, low-value part wth a peak near zero. Snce both off-lne and on-lne recognzers show the same output behavor after normalzaton, ths provdes us wth a standard measure for classfer combnaton. k= Hstogram of a confdence value a Hstogram of a confdence value a correct ncorrect [a ] correct ncorrect [a ] Graph 7: Hstogram for an off-lne recognzer after normalzaton. Graph 8: Hstogram for an on-lne recognzer after normalzaton 2

6 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK Fnally, Graph 9 and Graph 1 show the lkelhood-dependent recognton rates for each lkelhood value after normalzaton, whch correspond to the recognton rates shown n Graph 3 and Graph 4 before normalzaton. [% ] Recognton Rate R(a) correct nccorect [a ] [% ] Recognton Rate R(a) correct nccorect [a ] Graph 9: Recognton Rate for an off-lne recognzer after normalzaton. Graph 1: Recognton Rate for an on-lne recognzer after normalzaton In Graph 9 and Graph 1, the recognton performance quckly ncreases to a hgh level, and except for very small lkelhood values, the recognton rate s about the same for each lkelhood value. The number of correctly recognzed patterns s hgher than the number of falsely recognzed ones, from lkelhood values greater than 5 onwards. Agan, both off-lne and on-lne classfers behave smlarly here. 4. Characterstc of On/Off classfers accordng to stroke number Number of strokes s the basc feature of Chnese and Japanese characters. The rght stroke number can be found n a dctonary and vares from 1 to about 3. However, for fluently and quckly wrtten characters, the number of strokes s often lower because some strokes are connected. In some cases the number of strokes can be hgher than they should be. An nterestng characterstc of stroke number varatons s gven n [11]. Graph show the recognton rate accordng to the stroke number for an on-lne and off-lne recognzer respectvely. Even from a bref vew, we see the bg dfference between both classfers. An off-lne recognzer s weaker n recognzng characters wrtten wth a low number of strokes. The recognton rate grows wth ncreasng complexty of patterns. Ths s n accordance wth the well-known fact that for off-lne classfers t s more dffcult to recognze smple patterns lke Kana than dffcult Kanj. On the contrary, on-lne recognzers are very effcent for one-stroke patterns. 21

7 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK [%] Recognton rate for off-lne recognzer [%] Recognton rate for on-lne recognzer number of strokes number of strokes Graph 11: Recognton rate accordng to the stroke number for off-lne recognzer. Graph 12: Recognton rate accordng to the stroke number for on-lne recognzer. From Graph 12 we see that although an average rate of the on-lne recognzer s about 5% worse than that of the off-lne recognzer, for characters wrtten by one, two, or three strokes the recognton rate s better, or at least smlar. Snce classfers effcency depends on the strokes number, we tred to make not only one general characterstc functon, but 14 specfc functons, where the last one s for patterns wrtten by fourteen or more strokes. In Graph 13 we show some examples of stroke-dependent characterstc functons [charf ] for an on-lne recognzer and n Graph 14 for an offlne recognzer. [char ] #1 #14 Characterstc functons [char ] #5 # #1 number of strokes [a ] # 1 # 2 # 5 # 1 # 14 Graph 13: Characterstc functons for onlne recognzer [char ] Characterstc functons [char ] #14 # #5-2 #2 #1 [a number of strokes ] # 1 # 2 # 5 # 1 # 14 Graph 14: Characterstc functons for offlne recognzer 5. On-lne & off-lne classfers and combnaton schemes Our off-lne recognzer represents each character as a 256-dmensonal feature vector. Every nput pattern s scaled to a 64x64 grd by non-lnear normalzaton and smoothed by a connectvty-preservng procedure. Then, a normalzed mage s decomposed nto 4 contour sub-patterns, one for each man orentaton. Fnally, a 64- dmensonal feature vector s extracted for each contour pattern from ts convoluton wth a blurrng mask (Gaussan flter). A pre-classfcaton step precedes the actual fnal recognton. Pre-classfcaton selects the 5 canddates wth the shortest Eucldan dstances between the categores mean vectors and the test pattern. The fnal classfcaton uses a modfed quadratc dscrmnant functon (MQDF2) developed by 22

8 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK Kmura et al. [5] from tradtonal QDF. The off-lne classfer s traned wth the offlne HP-JEITA database (3,36 categores, 58 patterns per category). The on-lne recognzer presented n [12] by Nakagawa et al. employs a two-step coarse classfcaton based on easly extractable on-lne features n the frst step and four-drectonal features n the second step. An effcent elastc matcher explotng herarchcal pattern representatons performs the man classfcaton. The recognzer s traned wth the on-lne Nakayos_t database[6] (163 patterns for each of 4,438 categores). We nvestgate two dfferent combnaton strateges for combnng our on-lne and off-lne recognzers: max-rule and sum-rule. Max-rule takes the class wth the maxmum output value among each classfer, whle the sum-rule adds up the output for each class and selects the one wth the hghest sum[7]. 6. Benchmarkng wth the Kuchbue_d database Ths secton presents experments evaluatng our proposed normalzaton wth respect to the Kuchbue_d database [6], [14], whch s a wdely acknowledged benchmark for Japanese character recognton. It contans handwrtten sentences from a Japanese newspaper. In total, Kuchbue contans more than 1.4 mllon characters wrtten by 12 wrters (11,962 Kanj samples per wrter). Kuchbue covers 3,356 Kanj categores ncludng the two phonetc alphabets Hragana and Katakana, plus alphanumercal characters. Snce our recently publshed results were based on the ETL9B benchmark-database, whch does not cover Katakana and alphanumercal characters, we confne our experments to the 336 categores of Kuchbue (Kanj and Hragana) ncluded n ETL9B. On-lne patterns processed by the off-lne classfer are converted to btmaps by our callgraphc method [15] pantng realstc off-lne patterns. Table 1: Combnaton of an on-lne and an off-lne recognzer, tested on Kuchbue_d. Benchmark: Kuchbue_d wthout normalzaton Normalzaton by 1 characterstc functon Normalzaton by 14 characterstc functon Sngle classfer Combnaton of sngle classfers On-lne Off-lne AND OR Max Sum The effcency of our two sngle recognzers are 91.6% (on-lne) and 94.69% (offlne); Table 1, column 1 and 2. The hgher recognton rate of the off-lne recognzer s partally result of more tranng patterns per category (58 aganst 163) and smaller scope of recognzable categores (336 aganst 4438). The next two columns show the theoretcal worst (AND - a pattern was recognzed by both classfers) and best (OR - a pattern was recognzed at least by one recognzer) combnaton schemes, whch are the theoretcal bounds for any combnaton rule. Especally the OR combnaton scheme s mportant. Although t can never be utlzed n a real applcaton, the best theoretcal boundary s mportant for comparng the effcency of other combnaton schemes. 23

9 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK Column 5 and Column 6 of Table 1 show the recognton rates for the max-rule and sum-rule respectvely. The sum-rule performs better than max-rule. A theoretcal explanaton why the sum-rule outperforms the max-rule s gven n [7]. The sum-rule s also more resstant to problems of non-normalzed lkelhood. We have acheved mprovements of sngle recognton rates from 91.6% and 94.69% up to 97.71%. And what s also mportant, our result s very near to the theoretcal maxmum 98.88% of the OR combnaton scheme. 7. Benchmarkng wth the NTT-AT database In ths secton we compare the results based on Kuchbue_d from Secton 6 wth results based on the NTT-AT database publshed n [16]. The NTT-AT database contans data wrtten by elderly people wth an average age of 7.5 years, wth the oldest wrter beng 86 years old. These patterns are casually wrtten, very often wth an untypcal stroke order. Table 2: Combnaton of on-lne and off-lne recognzers tested on NTT-AT. Benchmark: NTT-AT wthout normalzaton Sngle classfer Combnaton of sngle classfers On-lne Off-lne AND OR Max Sum Normalzaton by 1 characterstc functon Normalzaton by 14 characterstc functon Table 2 shows that our normalzaton sgnfcantly mproves the recognton rate. From 85.67% (on-lne classfer) and 89.7% (off-lne classfer) to 93.68%. 8. Dscusson In ths paper we presented an updated verson of our accumulated-recognton-ratebased normalzaton for combnng multple classfers, whch was ntroduced at IWFHR 21 [13,16]. A smple warpng process algns confdence values accordng to the accumulated recognton rate, so that the normalzed values allow a far combnaton. In our prevous experments on the NTT-AT database, whch contans casually wrtten characters often wrtten n untypcal wrtng order, we mproved the recognton rate from 89.7% for the best sngle recognzer to 93.68%. In ths paper we used the Kuchbue_d database, whch s the well-establshed benchmark of common casually wrtten Japanese characters. Its sze s 23 tmes bgger than that of the NTT-AT database. The mprovement of the recognton rate s from 94.69% for the best sngle recognzer to 96.29% for the sum-rule combnaton pror to our normalzaton, 97.13% when normalzed by one common characterstc functon, and fnally 97.71% f normalzed by 14 stroke-number-specfc characterstc functons. The man advantages of our normalzaton based on the accumulated-recogntonrate are as follows: Frstly, the possblty to combne any number of classfers, wthout havng weaker classfers degrade the fnal result. Secondly, t allows easy comb- 24

10 4 th Internatonal Workshop on Multple Classfer Systems (MCS23) Guldford, UK naton of classfers wth ncompatble confdence values. Thrdly, normalzng s a completely automatc process, wthout emprcal parameter settngs. And fourthly, normalzaton s performed only once and does not consume tme durng recognton. Normalzaton based on stroke numbers better reflects the nature of each recognzer and thus leads to better results; t requres more tranng patterns for computng multple characterstc functons though. Although our accumulated-recognton-rate-based normalzaton was tested on combned on-lne/off-lne classfers for on-lne Japanese handwrtten characters, t should be useful for combnatons n varous felds of pattern recognton. References 1. M. Hamanaka, K. Yamada, J. Tsukumo, On-Lne Japanese Character Recognton Experments by an Off-Lne Method Based on Normalzaton-Cooperated Feature Extracton, Proc. 2 nd ICDAR (1993) M. Okamoto, A. Nakamura, K. Yamamoto, Drecton-Change Features of Imagnary Strokes for On-Lne Handwrtng Character Recognton, 14 th ICPR (1998) S. Jaeger, S. Manke, J. Rechert, A. Wabel, On-Lne Handwrtng Recognton: The Npen++ Recognzer, IJDAR 3(3) (21) F. Kmura, et al., Modfed quadratc dscrmnant functon and the applcaton to Chnese characters, IEEE Pattern Analyss and Machne Intellgence, 9(1), pp , H. Kang, K. Km and J. Km, A Framework for Probablstc Combnaton of Multple Classfers at an Abstract Level, EAAI 1 (4) (1997) Dstrbuton of the on-lne databases Kuchbue_d and Nakayos_t, Nakagawa laboratory, TUAT, Japan: 7. J.Kttler, M.Hatef, R.Dun, J.Matas, On Combnng Classfers, IEEE PAMI 2(3) (1998) H. Tanaka, K. Nakajma, K. Ishgak, K. Akyama, M. Nakagawa, Hybrd Pen-Input Character Recognton System Based on Integraton of On-Lne Off-Lne Recognton, Proc. 5 th ICDAR (1999) T. Ho, J. Hull, S. Srhar, Decson Combnaton n Multple Classfer Systems, IEEE Trans. on Pattern Analyss and Machne Intellgence, vol. 16, no. 1, pp , M. Oberländer, German Patent DE C1 (n German), 1995, on-lne avalable on the webste of the German Patent and Trade Mark Offce: K. Matsumoto, T. Fukushma, M. Nakagawa, Collecton and analyss of on-lne handwrtten Japanese character patterns, Proc. 6 th ICDAR, Seattle, 21, pp M. Nakagawa, K. Akyama, L.V. Tu, A. Homma, T. Hgashyama, Robust and Hghly Customzable Recognton of On-Lne Handwrtten Japanese Characters, Proc. 13 th ICPR (1996), volume III, O. Velek, S. Jaeger, M. Nakagawa, A New Warpng Technque for Normalzng Lkelhood of Multple Classfers and ts Effectveness n Combned On-Lne/Off-Lne Japanese Character Recognton, Proc. 8 th IWFHR(22), pp S. Jaeger, M. Nakagawa, Two On-Lne Japanese Character Databases n Unpen Format, Proc. 6 th ICDAR(21), pp O. Velek, M. Nakagawa, C.-L.Lu, Vector-to-Image Transformaton of Character patterns for On-lne and Off-lne Recognton, Internatonal Journal of Computer Processng of Orental Languages, Vol.15, No2 (22) O. Velek, M.Nakagawa, Usng Stroke-Number Characterstcs for Improvng Effcency of Combned On-Lne and Off-Lne Japanese Character Classfers, 5 th DAS (22),