Foreground and Background Information in an HMM-based Method for Recognition of Isolated Characters and Numeral Strings

Transcription

1 Foreground and Background Informaton n an HMM-based Method for Recognton of Isolated Characters and Numeral Strngs Alceu de S. Brtto Jr a,b, Robert Sabourn c,d, Flavo Bortolozz a,chng Y. Suen d a Pontfíca Unversdade Católca do Paraná (PUC-PR), R. Imaculada Conceção, Curtba (PR) Brazl b Unversdade Estadual de Ponta Grossa (UEPG), Pr. Santos Andrade S/N, Centro - Ponta Grossa (PR) Brazl c École de Technologe Supéreure (ETS), 1100 Rue Notre Dame Ouest - Montreal (QC) H3C 1K3 - Canada d Centre for Pattern Recognton and Machne Intellgence (CENPARMI), 1455 de Masonneuve Blvd. West, Sute GM Montreal (QC) H3G 1M8 - Canada Abstract In ths paper we combne complementary features based on foreground and background nformaton n an HMM-based classfer to recognze handwrtten solated characters and numeral strngs. A zonng scheme based on column and row models provdes a way of dvdng the character nto zones wthout makng the features sze varant. Ths strategy allows us to avod the character normalzaton, whle t provdes a way of havng nformaton from specfc zones of the character. The expermental results on 10 dgt classes, 52 character classes and 6 classes of numeral strngs of dfferent lengths have shown that the proposed features are hghly dscrmmnant. 1. Introducton Many approaches to solvng the handwrtten character recognton problem have been proposed n recent years due to numerous possble applcatons. Drawng up a taxonomy of these approaches s dffcult, snce ther methodologes overlap. However, research n ths feld has bascally consdered nvestgatng: a) feature extracton methods; b) classfcaton methods; and c) system archtectures based on dfferent strateges, such as combnatons of multple classfers, the use of multple templates, and the use of verfcaton modules. The nvestgaton of feature extracton methods has ganed consderable attenton snce a dscrmnatve feature set s consdered the most mportant factor n achevng hgh recognton performance. In [1] a survey of feature extracton methods for off-lne recognton of segmented characters s presented. The authors descrbe mportant aspects that must be consdered before selectng a specfc feature extracton method. In general, the feature extracton methods for numeral recognton reported n the lterature have been based on two types of features: statstcal and structural. The statstcal features are derved from statstcal dstrbutons of ponts, such as zonng, moments, projecton hstograms or drecton hstograms [2,3]. Structural features are based on topologcal and geometrcal propertes of the character, lke strokes and ther drectons, end-ponts, or ntersectons of segments and loops [4,5]. Many researchers have explored the ntegraton of structural and statstcal nformaton to hghlght dfferent character propertes, snce these types of features are consdered to be complementary. In [5], structural and statstcal nformaton s ntegrated nto a classfer based on Hdden Markov Model (HMM). The authors use stateduraton adapted transton probablty dstrbuton and macro-states to overcome the weakness of the HMMs n modelng structural features. The recognton rate s 96.16% n 2,711 dgt samples extracted from the CEDAR database. Another multfeature-based system s proposed n [6]. In ths work, a combnaton of seven dfferent famles of features s proposed n order to arrve at a complete character descrpton. These features are dvded nto global features (nvarant moments, projectons and profles) and local features (ntersectons wth straght lnes, holes and concave arcs, extremtes, end-ponts and junctons). A set of 53,324 dgts extracted from the NIST database s used to test the system. The recognton, rejecton and substtuton rates are 90.82%, 8.93% and 0.25% respectvely. In [7], a MLP-based classfer based on concavty features acheved a recognton rate of over 99.13% n 60,089 samples of handwrtten dgts of the NIST SD19 database. In ths paper, we combne features extracted from the foreground and the background of character mages n an HMM-based system. The challenge s to acheve recognton rates close to that presented n [7] wth MLPs, however, usng an HMM-based system. HMM have been used to model solated characters, whch are appled to recognze words or numeral strngs. The reason s that HMM can model specfc handwrtng knowledge related to the nteracton between adjacent characters n words or numeral strngs easer than MLPs. Moreover, HMM has been successfully used to provde mplct segmentatonbased methods to recognze words and numeral strngs as

2 n [8]. Wth such an approach t s possble to avod a pror segmentaton of the strng or word nto characters. 2. Feature extracton method The extracton method conssts of scannng the character mage from left-to-rght (column-based features) and from bottom-to-top (row-based features). Foreground and background nformaton are combned n a vector of 47 features: 34 foreground plus 13 background features. 2.1 Foreground features (FF) The FF vector conssts of local and global features calculated takng nto account the foreground pxels of the mage columns or rows. The local features are based on transtons from background to foreground pxels and vce versa. For each transton, the mean drecton and correspondng varance are obtaned by means of statstc estmators. These estmators are more sutable for drectonal observatons, snce they are based on a crcular scale. For nstance, gven the drectonal observatons α 1 = 1 o and α 2 = 359 o, they provde a mean drecton ( α ) of 0 o nstead of 180 o calculated by conventonal estmators. Let α1,..., α,..., α N be a set of drectonal F and sze N. observatons wth dstrbuton ( ) α vectors ( OP,...,,..., ). 1 OP OPN the N coordnates ( cos( α ), sn( α ) ) s defned as: 1 N C = cos N = 1 The center of gravty ( S ) ( α ) or S = sn( α ) 1 N N = 1 C, of These coordnates are used to estmate the mean sze of R, as: 2 2 R = ( C + S ) (3) Then, the crcular mean drecton can be obtaned by solvng one of the followng equatons: C cos ( α ) =, ( α ) R (2) S sn = (4) R Fnally, the crcular varance of α s calculated as: Sα = 1 R 0 S α 1 To estmate α and S α for each transton of a numeral mage, we have consdered { 0, 45, 90, 135, 180, 225, 270, 315 } as the set of F s computed by drectonal observatons, whle ( ) α countng the number of successve black pxels over the drecton α from a transton untl the encounter of a whte pxel. In Fgure 2 the transtons n a column of numeral 5 are enumerated from 1 to 6, and the possble drectonal observatons from transtons 3 and 6 are shown. (5) Fgure 1. Crcular mean drecton α and varance S α for a dstrbuton F ( α ) Fgure 1 shows that α represents the angle between the unt vector OP and the horzontal axs, whle P s the ntersecton pont between OP and the unt crcle. The cartesan coordnates of P are defned as: ( ( ), sn( )) cos α α (1) The crcular mean drecton α of the N drectonal observatons on the unt crcle corresponds to the drecton of the resultng vector ( R ) obtaned by the sum of the unt Fgure 2. Transtons n a column mage of numeral 5, and the drectonal observatons to estmate the mean drecton for transtons 3 and 6 In addton to ths drectonal nformaton, we have calculated two other local features: a) relatve poston of each transton, takng nto account the top of the dgt boundng box, and b) whether the transton belongs to the outer or nner contour, whch shows the presence of loops n the numeral mage. Snce for each column we consder 8 possble transtons, at ths pont our feature vector s composed of 32 features.

3 The global features are based on vertcal projecton (VP) of black pxels for each column, and the dervatve of VP between adjacent columns. Ths consttutes a total of 34 features normalzed between 0 and Background features (BF) The BF vector s based on concavty nformaton. These features are used to hghlght the topologcal and geometrcal propertes of the character classes. Each concavty feature represents the number of whte pxels that belong to a specfc concavty confguraton. The label for each whte pxel s chosen based on the Freeman code wth four drectons. Each drecton s explored untl the encounter of a black pxel or the lmts mposed by the dgt boundng box. A whte pxel s labeled f at least two consecutve drectons fnd black pxels. Thus, we have 9 possble concavty confguratons. Moreover, we consder four more confguratons, n order to detect more precsely the presence of loops. The total length of ths feature vector s then 13. The concavty vector s normalzed between 0 and 1, by the total of the concavty codes computed for each column or row of the character mage. Fgure 3 shows the 9 concavty confguratons and also 4 confguratons (A,B,C,D) for false loops A B C D - a black pxel was found n ths drecton - no black pxel was found n ths drecton Fgure 3 Concavty confguratons 2.3 Column and row-based features The feature vector composed of foreground and background features s extracted from each column and row of the character mage. Each feature vector s mapped to one of 256 possble dscrete symbols avalable n a 9 codebook prevously constructed by usng the K-means algorthm [9]. Thus, the output of the feature extracton method conssts of two sequences of dscrete observatons for each dgt: column-based and row-based sequences. 3. Hdden Markov models In the proposed classfer each character class s represented by two HMMs: one based on columns ( λc, λc,..., λ c ) and other based on rows ( λ r, λ r,..., λ r ) of the character mage. These column- and row-based models provde a way of combnng foreground and background features n the zonng scheme as shown n Fgure 4. s s 22222XXXXXXX XXXXXXXXXX XXXXXXXXXXXXX XXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXX XXXXXDDDDDDDDXXXXXXX XXXDDDDDDDDDDDXXXXXX XXXXXX XXXXXX XXXXXXX XXXXXXX XXXXXXX XXXXXXXX XXXXXXXXX XXXXXXXXXX XXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXXXXXXXXXXX XXXXXXXXDDDXXXXX XXXXXXDDDDDDXXXX XXXXDDDDDDDDXXXX XXXX XXXXX XXXXX XXXXXX XXXXXX XXXXXX XXXXXXX XXXXXXXXXXXX4 333XXXXXXXXXXXXXXXXXXXXXXX4 333XXXXXXXXXXXXXXXXXXXXXX44 333XXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXX XXXXXX XX Fgure 4. Implct zonng scheme provded by combnng column and row-based models The topology of the character models s defned takng nto account the recognton of handwrtten text. Ths means a left-rght model wth number of states defned as descrbed n [10], whch defnes the possble number of states (N) of the HMMs takng nto account duratonal statstcs calculated from the tranng database. Table 1 presents the range (mnmum and maxmum number of states) for each numeral model calculated on the tranng set (50,000 solated dgts 5,000 per class). In addton, the mean length value s also calculated. The fnal number of states of each dgt model was expermentally defned as that correspondng to the mnmum value n Table 1.

4 Table 1. Mnmum, maxmum and mean number of states by dgt class Numeral model Column based models Row based Models Mn Mean Max Mn Mean Max Expermental results Three set of experments were carred out takng nto account solated dgts, solated characters and numeral strngs of dfferent lengths. In all the experments a zerolevel rejecton was used. 4.1 Experments on solated dgts The experments undertaken durng the course of development of the proposed method were done usng solated numerals from the NIST SD19. We use 50,000 numeral samples for tranng, 10,000 for valdaton and 10,000 for testng. A fnal experment was done usng a more robust protocol based on 195,000 samples for tranng, 28,000 for valdaton and 60,089 for testng Evaluaton of the number of states The gap between the number of states usually found n the lterature for HMMs used to represent characters (5 or 6 states) [11,12] and those n Table 1 estmated usng the scheme proposed [10] s very large. For ths reason, we decde at ths tme to evaluate, for the column-based models, confguratons wth 6, 8 and 12 states. Table 2 - Experments consderng dfferent number of states n the numeral HMMs Column models Row models Number of states Vald. (%) Test (%) Vald. (%) Test (%) Mnmum values Mean values Table 2 shows the recognton results consderng dfferent number of states for the column and row numeral models. The best results were obtaned by usng the mnmum values presented n Table 1. The maxmum values were not evaluated snce we have observed a loss n terms of recognton rates for the mean values Evaluaton of the codebook szes The codebook sze was expermentally optmzed. We have evaluated codebooks composed of 64, 128, 192, 256 and 320 entres. The codebook composed of 256 entres provded the best results (see Table 3). The recognton rate of the row models consderng a codebook wth 64 entres were not calculated, snce we have observed that 256 entres provded better results (based on the columnbased models). Table 3 Experments usng dfferent codebook szes Column models Row models Sze Vald. (%) Test. (%) Vald. (%) Test. (%) Combnaton of column and row models The experments have shown that combnng columnand row-based models to represent each dgt class provdes an nterestng recognton performance. In Table 4, experment (a), only the column-based model usng the foreground feature (FF) vector was evaluated. In the experment (b), we observed a sgnfcant mprovement n the recognton performance when we combne foreground and background features n the column-based model. Table 4. Combnaton of column and row models Vald.(%) Testng (%) (a) Column (FF vector) 96,79 94,00 (b) Column (FF + BF vectors) (c) Row (FF + BF vectors) (d) Combnaton (column and row models used n (b) and (c), respectvely) Smlar results were obtaned n the experment (c) for the row-based models. Fnally, the experment (d) has shown that column and row-based models are really complementary. The models were combned by summng the log of ther fnal probablty calculated usng Vterb s algorthm. Table 5 shows the confuson matrx related to the experment (a) presented n Table 4. We can observe many confusons between dgt classes: 0-6, 2-7, 3-5, 4-9, 6-0, 8-6, 9-0 and 9-4.

5 Table 6 shows the fnal confuson matrx related to the experment (d) presented n Table 4. We can observe that the use of complementary nformaton (foreground and background features + column and row models) has shown to be a promsng way to reduce those confusons shown n Table 5. However, there s stll some confuson between classes 2-7, 4-9, and 9-4. Table 5. Confuson matrx: column model (FF vector) Table 6. Confuson matrx: combnaton of column and row models (FF + BF vectors) In a fnal experment consderng solated dgts, we have used a more robust expermental protocol, n whch the database s composed of 195,000 samples for tranng, 28,000 for valdaton and 60,000 for testng. The recognton rate for the testng set was 97.9%. 4.2 Experments on solated characters The solated characters avalable on the NIST SD19 were used n these experments. The hsf_0, hsf_1, hsf_2 and hsf_3 seres were used for tranng. The hsf_7 and hsf_4 seres were used as valdaton and testng sets, respectvely. Table 7 shows the expermental protocol and recognton rates of the proposed method and related works evaluated on the same database. As we can see, t s dffcult to compare snce most of the tme the authors dd not consder 52 classes. We have used the same expermental protocol than Koerch [13] snce the author has consdered the complete expermental protocol. When both, upper and lowercase, are consdered n the same experment the proposed features have shown to be better. The reason s that, the features based on columns and rows have shown to be more sutable to dstngush classes where the dfference s just the sze, such as C and c. Table 7. Expermental protocol and recognton rates on solated characters Class # Tran. # Vald. # Test. Recog. Rates % Proposed method uppercase (26 classes) 37,440 12,092 11, lowercase (26 classes) 37,440 11,578 12, upper/lower (52 classes) 74,880 23,670 23, Koerch [13] uppercase (26 classes) 37,440 12,092 11, lowercase (26 classes) 37,440 11,578 12, upper/lower (52 classes) 74,880 23,670 23, Oh et al [14] 26,000 11, uppercase (26 classes) Dong et al [15] lowercase (26 classes) 23,937 10, Experments on handwrtten numeral strngs We have also used the proposed models and features n a method for numeral strng recognton. It can be categorzed as a segmentaton-free approach that avods a pror segmentaton of the strng nto dgts by usng an mplct segmentaton strategy. In ths method the challenge conssts of fndng the best compromse between segmentaton and recognton. To deal wth ths problem, we propose a two-stage system n [8]. It allows the use of two sets of features and numeral models: one takng nto account of both segmentaton and recognton aspects, and another consderng just the recognton aspects. The frst stage, called the Strng Contextual- Based Stage (SCB), fnds the N best segmentatonrecognton paths for a gven numeral strng. For ths purpose, a dynamc programmng s used to match dgt Hdden Markov Models (HMMs) aganst to a gven strng. The 10 column-based HMMs used n ths stage are traned on solated dgts, but consderng contextual nformaton regardng strng slant and ntra-strng sze varaton. In addton, features extracted from the foreground pxels of the strng mage columns contemplate both segmentaton and recognton processes. The second stage, called Verfcaton Stage, re-ranks the N best segmentatonrecognton paths provded by the frst system stage usng a powerful solated dgt recognzer. The proposed set of features combnng foreground and background nformaton s used n order to mprove the recognton performance of the numeral HMMs. Moreover, 10 addtonal numeral HMMs based on the rows of the numeral mages are combned wth the column-based models durng the dgt recognton process. As we can see, the proposed strng recognton method s totally based on complementary nformaton

6 (foreground + background features, column + row-based models). Ths strategy has been shown to make a sgnfcant contrbuton to strng recognton performance. The recognton rate on 12,802 unknown length strngs composed of 2, 3, 4, 5, 6, and 10 dgts (NIST SD19) were: 94.8%, 91.6%, 91.2%, 88.3%, 89.0%, and 86.9% respectvely. In addton, the recognton rate on 2,069 touchng dgt pars also extracted from NIST SD19 was 89.6%. 5. Concluson In ths paper we have combned complementary features extracted from both foreground and background of character mages. These features were combned n an HMM-based classfer composed of column-based and row-based models. A zonng scheme based on column and row models provdes a way of dvdng the character nto zones wthout makng the features sze nvarant. The experments have shown that HMMs can provde hgh recognton performance close to those provded by the use of neural networks. Ths s very mportant snce HMMs have shown to be more approprate to model handwrtng knowledge related to the nteracton between adjacent characters n words or numeral strngs. Durng the experments t s possble to observe that HMMs and the combnaton of complementary features are a promsng strategy to recognze solated dgts or even to contemplate both segmentaton and recognton aspects n a numeral strng recognton method. Further work can be done n order to develop new features based on structural nformaton. In addton, the system wll be evaluated on cursve words. 6. References [1] Trer O.D., Jan A.K. and Taxt T. Feature Extracton Methods for Character recognton - a Survey. Pattern Recognton, Vol 29, No. 4, pp , [2] Kmura F. and Shrdhar M. Segmentaton-recognton algorthm for handwrtten numeral strngs. Machne Vson Applcatons, No. 5, pp , [3] Cheung K. and Yeung D. A Bayesan Framework for Deformable Pattern Recognton wth Applcaton to Handwrtten Character Recognton. IEEE Trans. on Pattern Analyss and Machne Intellgence, Vol. 20, No. 12, pp , Recognton. Proc. of the Fourteenth Int. Conf. on Pattern Recognton, Vol. I, pp , [6] Heute L., Paquet T., Moreau J.V., Lecourter, Y. and Olver, C. A structural/statstcal feature-based vector for handwrtten character recognton. Pattern Recognton Letters, 19, pp , [7] Olvera L. S., Sabourn R., Bortolozz F., and Suen C.Y. Feature Selecton Usng Mult-Objectve Genetc Algorthms for Handwrtten Dgt Recognton, 16th Int. Conf. on Pattern Recognton, vol I, pp , [8] Brtto Jr, A.S., Sabourn R., Bortolozz F. and Suen C.Y. The Recognton of handwrtten numeral strngs usng a twostage HMM-based method. Int. Journal on Document Analyss and Recognton. Vol. 5, Number 2-3, pp , [9] Makhoul J., Roucos S., and Gsh H. Vector Quantzaton n Speech Codng. Proc. of the IEEE, vol. 73, pp , [10] Wang X. Duratonally constraned tranng of HMM wthout explct state duraton PDF. Inst. of Phonetc Scences, Unversty of Amsterdam, Proceedngs 18, pp , [11] Procter S. and Elms A. J. The recognton of handwrtten dgt strngs of unknown length usng hdden Markov models. Proc. of the Fourteenth Int. Conf. on Pattern Recognton, pp , [12] Rabner, L. R. A Tutoral on Hdden Markov Models and Selected Applcatons n Speech Recognton. Proc. of the IEEE, Vol. 77, No. 2, pp , [13] Koerch, A.L. Large Vocabulary Off Lne Handwrtten Word Recognton. (PhD. Thess) École the Technologa Supéreure (Unversté du Québec). Montreal, 314p, [14] Oh I.S. and Suen C.Y. Dstance features for neural network based recognton of handwrtten characters. Internatonal Journal on Document Analyss and Recognton, 1(2):73 88, [15] Dong J., Krzyzak A., and Suen C.Y. Local learnng framework for recognton of lowercase handwrtten characters. In Proc. Int. Workshop on Machne Learnng and Data Mnng n Pattern Recognton, Lepzg, Germany, pp , [4] Hrano T., Okada Y. and Yoda, F. Structural Character Recognton Usng Smulated Annealng. Proc. of the Fourth Internatonal Conference on Document Analyss and Recognton, Vol. 2, pp , [5] Ca J. and Lu Z. Integraton of Structural and Statstcal Informaton for Unconstraned Handwrtten Numeral