2013 12th Internatonal Conference on Document Analyss and Recognton Onlne Handwrtten Cursve Word Recognton Usng Segmentaton-free n Combnaton wth P2DBM-MQDF Blan Zhu 1, Art Shvram 2, Srrangaraj Setlur 2, Venu Govndaraju 2 and Masak akagawa 1 1 Department of Computer and Informaton Scences, Tokyo Unversty Agrculture and Technology, Tokyo, Japan 2 Center for Unfed Bometrcs and Sensors, Unversty at Buffalo, Buffalo, US 1 {zhublan, nakagawa}@cc.tuat.ac.jp, 2 {ashvram, setlur, govnd}@buffalo.edu Abstract Ths paper descrbes an onlne handwrtten Englsh cursve word recognton method usng a segmentaton-free Markov random feld () model n combnaton wth an offlne recognton method whch uses pseudo 2D b-moment normalzaton (P2DBM) and modfed quadratc dscrmnant functon (MQDF). It extracts ture ponts along the pen-tp trace from pen-down to pen-up and uses the ture pont coordnates as unary tures and the dfferences n coordnates between the neghborng ture ponts as bnary tures. Each character s modeled as a and word s are constructed by concatenatng character s accordng to a tre lexcon of words durng recognton. Our method expands the search space usng a character-synchronous beam search strategy to search the segmentaton and recognton paths. Ths method restrcts the search paths from the tre lexcon of words and precedng paths, as well as the lengths of ture ponts durng path search. We also combne t wth a P2DBM-MQDF recognzer that s wdely used for Chnese and Japanese character recognton. Keywords Word Recognton, Segmentaton-free,, MQDF, Tre Lexcon, Beam Search I. Introducton Onlne handwrtten character recognton s recevng ncreased attenton due to the development and prolferaton of pen-based or touch-based nput devces such as tablet termnals, smart phones, electronc whteboards, and dgtal pens (e.g., the Anoto pen). Realzng onlne handwrtten character recognton wth hgh performance s vtal, especally for applcatons such as the natural nput of text on smart phones, to provde a satsfactory user experence. Hdden Markov models (HMMs) have been the domnant technque for onlne Englsh word recognton [1-2]. HMMs probablstcally treat a sequence of ture vectors n wrtng or poston order. Whle they can use the neghborhood relatonshps between the successvely adjacent ture vectors n wrtng or poston order (the so-called one-dmensonal neghborhood relatonshps), two-dmensonal neghborhood relatonshps, such as those among the geometrcally neghborng ture vectors are not explctly expressed. Even wth one-dmensonal neghborhood relatonshps, HMMs only use the state transton probabltes and unary tures, and bnary tures are not well utlzed. An addtonal drawback s that the neghborhood relatonshps among more than two neghborng ture vectors, such as ternary tures, cannot be used. Although some HMMs apply bnary tures, they only merge the bnary tures nto unary tures and use a vector of larger dmenson because HMMs do not take a new vew of the bnary tures, whch lmts recognton accuracy [3]. The model s descrbed usng an undrected graph n whch a set of random varables have the Markov property, and s can be used to effectvely ntegrate nformaton among neghborng ture vectors, such as bnary and ternary tures, and two-dmensonal neghborhood relatonshps [4]. Therefore, s have been effectvely appled to strokeanalyss-based structural offlne character recognton [5] and mage processng [4]. However, s had not been appled to onlne character recognton untl our pror work on Japanese character recognton [6]. Current onlne handwrtten character recognton approaches tend to use HMMs (note that HMMs can be vewed as specfc cases of s). s have more degrees of freedom than HMMs for explctly expressng relatons among multple ture vectors. Character segmentaton of cursve handwrtng s dffcult due to the fact that segmentaton ponts between characters are not obvous. Wthout character recognton cues and lngustc context, characters cannot be segmented unambguously. A sble way to overcome the ambguty of segmentaton s called ntegrated segmentaton and recognton, whch s classfed nto segmentaton-free and over-segmentaton-based methods [3]. The segmentaton-free method s also called as the word-based method [7], and a global ture vector s extracted from an nput word pattern and matched aganst a stored dctonary of prototype words rather than recognzng ndvdual characters. Ths word recognton method avods problems assocated wth segmentaton. HMM-based word recognton tends to use the segmentaton-free method [1-2] explotng the advantages that one-dmensonal structural methods (one-dmensonal neghborhood relatonshps) such as HMMs have to offer lke the ablty to concatenate character HMMs to construct word HMMs based on the provded lexcon of words durng recognton. On the other hand, the over-segmentaton-based method attempts to splt cursve words nto character patterns at ther true boundares and label the splt character patterns [8-9]. A word pattern s over-segmented nto prmtve segments such that each segment comprses of a sngle character or part of a character. The segments are combned to generate canddate character patterns (formng a canddate lattce), whch are evaluated usng character recognton, ncorporatng geometrc and lngustc contexts. Onlne handwrtten word recognton usng a Tme-Delay eural etwork [7] or a recurrent neural network [10] 1520-5363/13 $26.00 2013 IEEE DOI 10.1109/ICDAR.2013.77 349
performs word recognton by contnuously movng the nput wndow of the network across the frame sequence representaton of a word, thus generatng actvaton traces at the output of the network. These output traces are subsequently examned to determne the ASCII strng(s) best representng the word mage. Snce words have obvous lngustc context, lexcondrven methods can decrease recognton errors due to smlar characters, resultng n the wdespread use of lexcon-drven methods for word recognton. However, many prevous studes usng lexcon-drven methods have focused only on recognton models or ture extracton and have not placed much emphass on the matchng schemes and memory utlzaton whch are crtcal factors for handwrtng recognton systems runnng on personal hand-held devces such as tablets and moble phones. On these relatvely small devces, t s vtal for a handwrtng recognton system to have as small a memory footprnt as possble whle mantanng hgh accuracy. Therefore, we need to refne the matchng schemes and memory storage methods to mprove the effectveness for word recognton. Lexcon matchng schemes can be roughly dvded nto two classes: matchng the nput mage wth one lexcon entry n ts entrety each tme [8] and matchng wth all lexcon entres smultaneously [1, 9]. Although t s sad that the former scheme s sutable for a small vocabulary, we thnk that t s not effectve, even for a small vocabulary, and requres more recognton tme and memory storage. The latter often uses a tre lexcon, whch s effectve n reducng processng tme and memory storage sze. We have proposed an onlne handwrtten Japanese character recognton method usng a model and demonstrated that the model results n a hgher recognton accuracy compared to usng HMM [6]. In ths paper, we apply a segmentaton-free model of one-dmensonal structure for onlne handwrtten Englsh cursve word recognton. It extracts ture ponts along the pen-tp trace from pen-down to pen-up. It uses the ture pont coordnates as unary tures and the dfferences n coordnates between the neghborng ture ponts as bnary tures. Each character s modeled as a, and word s are constructed by concatenatng character s accordng to a tre lexcon of words durng recognton. It expands the search space usng a character-synchronous beam search strategy to search the segmentaton and recognton paths. Ths method restrcts the search paths from the tre lexcon of words and precedng paths, as well as the lengths of ture ponts durng path search. We then combne ths approach wth a P2DBM-MQDF recognzer [11-13] that s wdely used for Chnese and Japanese character recogntons. Expermental results from the new IBM_UB_1 dataset demonstrate the superorty of our method. The rest of ths paper s organzed as follows: Secton 2 begns wth the descrpton of the preprocessng steps. Secton 3 descrbes our recognton method, Secton 4 presents the expermental results, and Secton 5 presents our conclusons. Preprocessng II. We pre-process each onlne word pattern as follow. (1) Extractng ture ponts. To compute baselnes and normalze slant effcently, we frst extract ture ponts usng the method developed by Ramner [14]. The start and end ponts of every stroke are pcked up as ture ponts. Then, the most dstant pont from the straght lne between adjacent ture ponts s selected as a ture pont f the dstance to the straght lne s greater than a threshold value that s decded from the word heght. Ths selecton s done recursvely untl no more ture ponts are selected. Ths ture pont extractng process s shown n Fg. 1(a). We also reserve the orgnal nput ponts for recognton usng the P2DBM-MQDF recognzer, and use the extracted ture ponts for computng baselnes, normalzng slant and for recognton by the model. (a) (c) Fg. 1. Examples showng preprocessng of word pattern (2) Computng baselnes. We compute baselnes usng lnear regresson lnes that approxmate the local mnma (baselne) or the local maxma (corpus lne) of the trajectory as shown n Fg.1 (b). (3) ormalzng rotatons. Rotatons of words are corrected by rotatng wth an angle computed from the two lnes as shown n Fg.1 (b). (4) ormalzng slant. We normalze the dfferent slants of words, by shearng every word accordng to ts slant as shown n Fg.1 (c). The slant s determned by a hstogram over all angles subtended by the lnes connectng two successve ture ponts of the trajectory and the horzontal lne. The computed angles are weghted wth the dstance of every par of successve ponts. A smple search for the maxmum entry n the hstogram provdes us wth the slant of the word. (5) Removng delayed strokes. Delayed strokes, such as the crossng of t or the dot of, are troublesome n onlne handwrtng recognton. These strokes ntroduce addtonal temporal varaton and complcate onlne recognton. We detect such strokes and remove them before word recognton. A delayed stroke s usually a short sequence wrtten n the upper regon of the wrtng pad, above already wrtten parts of (b) (d) 350
a word, and accompaned by a pen movement to the left. We use some smple threshold values for characterzng these tures of delayed strokes as shown n Fg. 1 (d). (6) ormalzng sze. To ensure that the same characters have approxmately the same heght and the same Y coordnate poston for every handwrtten word, we transform every word to a gven corpus heght, where the corpus heght s the dstance between the baselne and the corpus lne, and set the baselne and the corpus lne to fxed Y coordnate postons. Recognton Method III. In ths secton, we present our segmentaton-free recognton method. A. Constructon of Tre Lexcon Frst, we construct a tre lexcon from a word database, as shown n Fg. 2. When tranng character s, we can count the range of the number (length) of ture ponts for each character class. For nstance, the length range of ture ponts for the character O s from 5 to 50 as shown n Fg. 2. Then, the length range of ture ponts to the termnal for each of the tre can be calculated accordng to the length range of ture ponts for each character class, as shown n Fg. 2, where the numbers shown n parentheses of each box are the length range of ture ponts to the termnal for the. We can restrct the searched paths by the length ranges resultng n mproved recognton accuracy. Fg. 2. Porton of tre lexcon of words B. Constructons of Character Recognzers We create two character recognzers: a recognzer and a P2DBM-MQDF recognzer. For the recognzer, we extract character patterns from true segmentaton ponts of preprocessed tranng word patterns and shft the X coordnates of each character pattern so that the mnmum of X coordnate s 0 whle the Y coordnates are unchanged. Here, we employ ture ponts extracted n the preprocessng. We use the ture ponts of obtaned character patterns to tran the model of each character category [6]. For the P2DBM-MQDF recognzer, we extract character patterns from the orgnal nput ponts of tranng word patterns and normalze each character pattern to a standard sze when extractng the tures. Ths s because the character recognton-based approach depends on the performance of each character recognton and latest normalzaton mproves t sgnfcantly. From each character pattern (a sequence of stroke coordnates), we extract drectonal tures: hstograms of normalzed stroke drecton [11]. For coordnate normalzaton, we apply P2DBM [12]. The local stroke drecton s decomposed nto eght drectons, and from the ture map of each drecton, 8x8 values are extracted by Gaussan blurrng so that the dmensonalty of ture vectors s 512. To mprove the Gaussanty of ture dstrbuton, each value of the 512 tures s transformed by the Box-Cox transformaton (also called varable transformaton). The nput ture vector s reduced from 512D to nd by the Fsher lnear dscrmnant analyss (FLDA) [3]. Then we use the nd ture vectors to create a MQDF model [13]. Fg. 3. Extracted ture ponts of word Offer C. Path Search We expand the search space usng a character-synchronous method and apply the beam search strategy to match the ture ponts of each nput pattern wth the states of word models, and obtan a smlarty for each word. Fg. 3 shows an example of the extracted ture ponts of a word pattern. We match the ture ponts F={f 1, f 2, f 3,,f 56 } wth the states of word models. We show an example of a search usng a character-synchronous lexcon-drven method n Fg. 4 to descrbe our process flow. We conduct the search and expanson from the expanson depth d 1 to d 5. We frst search the tre lexcon from ts start s. In Fg. 2, the start s are [O], [p], and [], and based on these we expand the root and set ts chldren s 1-1 wth a character category [O], 1-2 wth a character category [p] and 1-3 wth a character category [], where each chld has a character category C and a start pont of a ture sequence. In ths, they share the start pont f 1. For each, the length from the start pont f 1 to the termnal pont f 56 s 56 so that the 1-3 s erased because the length range to the termnal of the tre [] s 9-44 and does not satsfy the length from the start pont to the termnal pont. For each of 1-1 and 1-2, we match the ture ponts from each start pont f to the pont f +M wth the states S={s 1, s 2, s 3,,s J } of the character model of the correspondng C, where M s the maxmum number of the ture ponts of C and t s calculated from tranng patterns. Fg. 5 shows an example of the matchng process. Before matchng, we shft the X coordnates of the ture ponts from f to f +M so that the mnmum of X coordnate s 0. We apply the Vterb search to match. Then we can get paths at the [End] state whch correspond to some end ponts such as f 16, f 17 and f 18 for 1-1.We sort the scores of the paths (note that each score S s normalzed by the number of the ture ponts from the start pont to the end pont of the path to S / ), and select top paths. In Fg. 4, s set as three and three paths (from f 1 to f 16, from f 1 to f 17, from f 1 to f 18 )of 1-1, and three paths (from f 1 to f 12, from f 1 to f 15, from f 1 to f 18 )of 1-2 are selected, and we call them sub-s. ode 1-1 has three sub-s ( 1-1 -1, 1-1 -2 and 1-1 -3) whle 1-2 has three sub-s ( 1-2 -1, 1-2 -2 and 1-2 -3). 351
Fg. 4. Search and Recognton Fg. 5. Matchng between ture ponts and character For all the sub-s up to the depth d 1, we evaluate all search paths accordng to the path evaluaton crteron, sort them, then only select several top paths and erase others. The number of the selected top paths s called the beam band. In Fg. 4, the beam band s set as two, and two paths endng at 1-1 -1 and 1-1 -3 are selected for d 1. We try the followng three path evaluaton crtera: E E 1 S P2DBM MQDF 1 S 1 P2DBM-MQDF 1 E P2DBM-MQDF S SP2DBM-MQDF P2DBM-MQDF 1 1 where E, E P2DBM-MQDF and E +P2DBM-MQDF stand for the evaluaton crteron usng the score of the recognzer alone, that of usng the score of the P2DBM-MQDF recognzer alone and that of usng both scores respectvely, S, S P2DM-MQDF,,, σ and σ P2DBM-MQDF are the score of the recognzer for sub- sub,the score of the P2DBM-MQDF recognzer for sub, the number of the ture ponts from the start pont to the end pont of sub,the number of sub-s n the path, the varance of the recognzer scores, and the varance of the P2DBM-MQDF recognzer scores, respectvely. We use the number of the ture ponts n each path to normalze the path evaluaton crteron of the path. Scnce S has been normalzed by when matchng ture ponts wth a character, we need to multply S by. We consder that the score of the P2DBM-MQDF recognzer reflects the shape of the combnaton of the all ture ponts of, the more the ture ponts are, the lower the score. Therefore, we do not multply t by. After selectng the end ponts and decdng the sub-s for each k-j by matchng as shown n Fg. 5, we can extract the orgnal nput ponts from the start pont to the end pont of each sub- sub, and nput them to (1) the P2DBM-MQDF recognzer to obtan the score S P2DM- MQDF. To combne the two scores S, S P2DM-MQDF, we normalze them by ther respectve score varances. Then, we apply the same method to process the next depth. Fnally, the expanson reaches to the depth d 5. For all s up to the depth d 5, we evaluate all paths accordng to the path evaluaton crteron, sort them, and then select the optmal path as the recognton result. For each k-j wth a character category C and a start pont f, we need to execute matchng to decde ts end ponts and sub-s. After that we need to extract tures to recognze each sub- by the P2DBM-MQDF recognzer. Dfferent s may be requred for the same par of C and f. Therefore, for each par of C and f, once ts end ponts and sub-s are decded and the scores of ts sub-s are calculated, we store them and use them for other s from the second tme. For each par of start and end ponts, we also store the tures for the P2DBM-MQDF recognzer, and use them for the second pass. Ths can greatly mprove recognton tme. We call ths storage strategy of scores and tures (SSSF). Experments IV. We evaluate the word recognzers usng the new IBM_UB_1 dataset orgnally collected by IBM and beng released by the Unversty at Buffalo (CUBS). IBM_UB_1 contans cursve onlne handwrtng n Englsh wrtten by 43 wrters. A set of 10 topc scrpts were generated each of whch conssted of 100 or more than 100 documents relatng to the correspondng topc. For each document wrtten by a specfc wrter, there s a summary text and a correspondng query text. The summary text contans one or two pages of wrtng on a partcular topc, whle the query text contans approxmately 25 words that encapsulate the summary text and thus may be used to retreve the correspondng summares. Our experments use the query text from 43 wrters. ether IBM_UB_1 nor UIPE database [15] has word patterns wth both word labels and character-level segmentaton labels. We need both the labels to tran our character recognzers. Fortunately, we found that n tran r01 v07 of the UIPE database, Benchmark#3 (solated characters) and Benchmark#6 (solated words) share four common sets ( art data wth 6 wrters, cea data wth 6 wrters, ceb data wth 4 wrters and, ka data wth 28 wrters). We combned these sets to obtan a set wth the word labels and character-level segmentaton labels. The data set contans 14,691 characters and 2,127 words for 44 wrters. At the begnnng, we used the data set to tran the and the P2DBM-MQDF recognzers. Then we set the tre lexcon to have only one word (the word strng label of the recognzed word pattern n IBM_UB_1), and used the traned recognzer to recognze each word pattern of IBM_UB_1 and set the recognzed segmentaton result as the character segmentaton labels of the pattern. After that, we obtaned character-level segmentaton labels for the word patterns of IBM_UB_1. We selected four pages at random for each of 20 wrters as testng data, and used the remanng data as tranng data. We 352
used the tranng data to tran the and the P2DBM- MQDF recognzers. Then, we tested the recognton rate for the testng data. The tranng data has 61,105 words and 355,895 characters of 62 categores (dgt, uppercase and lowercase Latn alphabet), whle the testng data has 1,795 words and 10,987 characters of 62 categores. The constructed tre lexcon contans 5,000 words. The experments were mplemented on an Intel(R) Core(TM) 2 Quad CPU Q9550 @ 2.83GHz 2.83 GHz wth 4.00 GB memory. The ture vector s reduced from 512D to 61D by the FLDA for the P2DBM-MQDF recognzer. s set as 100 and the beam band s set as 1,000. Table 1 shows the results. TABLE I. RECOGITIO RESULTS Method Performance E EP2DBM- MQDF E+P2DBM- MQDF E+P2DBM-MQDF wthout length range restrctng Word rec. rate (%) 55.4 13 76.89 71.65 From these results, we can see that E +P2DBM-MQDF sgnfcantly mproves the recognton accuracy by combnng model wth a P2DBM-MQDF recognzer. The model apples a structural method that s weak at collectng global character nformaton, but robust aganst character shape varatons. The 2DBM-MQDF recognzer uses an unstructural method that s robust aganst noses but weak aganst character shape varatons. By combnng a structural method wth an un-structural method, the recognton accuracy mproves snce they compensate for ther respectve dsadvantages. Restrctng the searched paths by the length range can brng hgher recognton accuracy. The recognton rate for E P2DBM-MQDF s very low and we beleve ths s because the P2DBM-MQDF recognzer renormalzes each character pattern so that t loses the nformaton of character szes and character postons. We have realzed a word recognton rate of 76.89% whch s comparable to the performance reported usng a recurrent neural network [10]. The memory consumpton of our recognzer s about 8.5MB. We also nvestgated the results of dfferent beam bands. Table 2 shows the results, where the recognton tme s the average for recognzng a word. TABLE II. RECOGITIO RESULTS FOR BEAM BADS Performance Beam band 100 300 500 700 1000 1500 Word rec. rate (%) 68.31 72.60 74.10 75.83 76.89 78.17 Recognton tme (s) 1.00 1.21 1.71 2.12 3.55 6.26 We can see that the larger the beam band, the hgher the recognton rate and the longer the recognton tme. The recognton speeds are stll low and we need to ncorporate our system wth a real-tme recognton approach [16] to mprove the recognton speed. We also evaluate the SSSF and got a result that t can mprove recognton speed about 89 tmes when not usng SSSF for beam band 300. V. Concluson Ths paper presented a method for onlne handwrtten Englsh cursve word recognton usng segmentaton-free model. We restrcted the search paths from the tre lexcon of words and precedng paths, as well as the lengths of ture ponts durng path search by the character-synchronous beam search strategy. We combned t wth a P2DBM- MQDF recognzer beng wdely used for Chnese and Japanese character recognton. Expermental results demonstrate a sgnfcant mprovement n recognton accuracy by combnng our segmentaton-free model wth a P2DBM-MQDF recognzer, and by restrctng the searched paths consderng the length range. We have already shown that combnng model wth P2DBM-MQDF acheves very hgh recognton rate for onlne handwrtten Japanese text [17]. In ths paper, we have shown ths combnaton s also very effectve and successful for onlne Englsh word recognton. Acknowledgment Ths work s beng supported by the Grant-n-Ad for Scentfc Research (B)-224300095. References [1] S. Jaeger, S. Manke, J. Rechert and A. Wabel, Onlne handwrtng recognton: the pen++ recognzer, IJDAR, 3(1), pp.69-180, 2001. [2] M. Lwck and H. Bunke, HMM-based on-lne recognton of handwrtten whteboard notes, Proc. 10th IWFHR, pp. 595-599, 2006. [3] M. Cheret,. Kharma, C.-L. Lu and C. Y. Suen, Character recognton systems, A Gude for Students and Practoners, John Wley & Sons, Inc., Hoboken, ew Jersey, 2007. [4] S. Z. L, Markov random feld modelng n mage analyss, Sprnger, Tokyo, 2001. [5] J. Zeng and Z.-Q. Lu, Markov random feld-based statstcal character structure modelng for handwrtten Chnese character recognton, IEEE Trans. PAMI, 30(5), 2008. [6] B. Zhu and M. akagawa, On-lne handwrtten Japanese characters recognton usng a model wth parameter optmzaton by CRF, Proc. 11th ICADR, pp. 603-607, 2011. [7] G. Sen., Large vocabulary recognton of on-lne handwrtten cursve words, PhD thess, Department of Computer Scence of the State Unversty of ew York at Buffalo,.Y., USA, 1995. [8] G. Km and V. Govndaraju, A lexcon drven approach to handwrtten word recognton for real-tme applcatons, IEEE Trans. PAMI, 19(4), pp. 366-379, 1997. [9] C.-L. Lu, M. Koga and H. Fujsawa, Lexcon-drven segmentaton and recognton of handwrtten character strngs for Japanese address readng, IEEE Trans. PAMI, 24(11), pp. 1425-1437, 2002. [10] M. Lwck, A. Graves, S. Fern andez, H. Bunke, and J. Schmdhuber, A novel approach to on-lne handwrtng recognton based on bdrectonal long short-term memory networks, In Proc. 9th ICDAR, pp.367-371, 2007. [11] C.-L. Lu and X.-D. Zhou, Onlne Japanese character recognton usng trajectory-based normalzaton and drecton ture extracton, Proc. 10th IWFHR, pp.217-222, 2006. [12] C.-L. Lu and K. Marukawa, Pseudo two-dmensonal shape normalzaton methods for handwrtten Chnese character recognton, Pattern Recognton, 38(12), pp. 2242-2255, 2005. [13] F. Kmura, Modfed duadratc dscrmnant functon and the applcaton to Chnese characters, IEEE Trans.PAMI, 9 (1), pp.149-153, 1987. [14] U. Ramer, An teratve procedure for the polygonal approxmaton of plan closed curves, Computer Graphcs and Image Processng, 1(3), pp. 244-256, 1972. [15] I. Guyon, L. Schomaker, R. Plamondon,M. Lberman, and S. Janet. Unpen project of on-lne data exchange and recognzer benchmarks, Proc. 12 th ICPR, pp. 29 33, 1994. [16] D.-H. Wang, C.-L. Lu, and X.-D. Zhou, An approach for real-tme recognton of onlne Chnese handwrtten sentences, Pattern Recognton, (45), pp. 3661-3675, 2012. [17] B. Zhu, J. Gao and M. akagawa, Objectve functon desgn for MCEbased combnaton of on-lne and off-lne character recognzers for onlne handwrtten Japanese text recognton, Proc. 11th ICDAR, pp.594-599, 2011. 353