An Approach to Real-Tme Recognton of Chnese Handwrtten Sentences Da-Han Wang, Cheng-Ln Lu Natonal Laboratory of Pattern Recognton, Insttute of Automaton of Chnese Academy of Scences, Bejng 100190, P.R. Chna E-mal : {dhwang, lucl}@nlpr.a.ac.cn Abstract: Wth the advances of handwrtng capturng devces and computng power of moble computers, pen-based Chnese text nput s movng from character-based nput to sentence-based nput. Ths paper proposes an approach to real-tme recognton of Chnese handwrtten sentences. The man feature of the approach s a dynamcally mantaned segmentaton-recognton canddate lattce. Whenever a new stroke s produced, canddate characters are generated and recognzed to assgn canddate classes, whle the wrtng process contnues. When the pen lft tme exceeds a threshold, the system searches the canddate lattce for the result of sentence recognton. Snce the recognton of canddate characters consumes the majorty of computng and s performed durng wrtng process, sentence recognton s obtaned mmedately after a long pen lft. Prelmnary experments demonstrate the promse of the proposed approach. Keywords Onlne Chnese handwrtten sentence recognton, real-tme recognton, dynamc canddate lattce, path search. 1. Introducton There have been growng applcatons of pen-based Chnese text. Wth the advances of handwrtng capturng devces and computng power of moble computers, sentence-based nput becomes possble. Compared to character-based nput, sentence-based nput s more natural and enables faster and more accurate nput va handwrtten sentence recognton utlzng contexts. Handwrtten sentence (character strng) recognton s a dffcult contextual classfcaton problem nvolvng character segmentaton and recognton. To acheve fast recognton, t s crucal to perform character segmentaton and recognton durng the wrtng process such that the whole recognton result s obtaned mmedately after completng a sentence. There have been many efforts towards the mprovement of the performance of handwrtten character strng recognton [1-7]. Most methods adopt the ntegrated segmentaton-recognton strategy to overcome the ambguty of character segmentaton: canddate character patterns are generated based on over-segmentaton and are recognzed by a character classfer to assgned canddate character classes, and the optmal path of segmentaton-recognton s searched for to gve the result. These methods, however, performs character segmentaton and recognton after the sentence wrtng s fnshed. To accelerate sentence recognton, character segmentaton and recognton should be performed durng the wrtng process, such that the result can be obtaned mmedately after the completon of wrtng. In ths paper, we propose an approach to real-tme recognton of Chnese handwrtten sentences usng a dynamcally mantaned segmentaton-recognton canddate lattce. Whenever a new stroke s produced, canddate characters are generated and recognzed to assgn canddate classes, whle the wrtng process contnues. The newly generated canddate characters and assgned classes are stored n the canddate lattce. When the pen lft tme exceeds a threshold, the system searches the canddate lattce for the result of sentence recognton by dynamc programmng (DP). Snce the recognton of canddate characters consumes the majorty of computng and s performed durng wrtng process, sentence recognton s obtaned mmedately after a long pen lft. We also developed some edtng functons to manually correct segmentaton and recognton errors. Our prelmnary experments demonstrate the promse of the proposed approach. 2. Character Strng Recognton We customze an exstng character recognton method to real-tme recognton. Before descrbng the real-tme recognton approach, we descrbe the character
strng recognton method below. In the ntegrated segmentaton-recognton framework, a character strng s frst over-segmented nto prmtve segments. Then one or several segments are merged to generate canddate characters that wll be recognzed by a character classfer to assgn canddate classes. The canddate characters and assgned classes are stored n a canddate lattce, and the optmal path can be searched for by DP or beam search [8][9]. The path evaluaton crteron ntegratng the character classfcaton score, geometrc context and language model s crucal to the accuracy and search effcency of strng recognton. We use the path evaluaton crteron proposed n [9]. Denote by X = x1l xn as a sequence of canddate characters of a strng, and each canddate character s assgned canddate classes (denoted as c ). Then the strng recognton result s a character strng C = c1 L cn. The canddate segmentaton-recognton path ( X, C ) s evaluated by a functon: n f ( XC, ) = { λ log Pc ( c ) + k[ λ log P( x c), (1) + log ( ) + log (, )]} 1 1 2 = 1 1 2 λ3 P g c λ4 P g c c 1 1 where P( x c), Pc ( c 1), P( g c) and 2 P( g c, c 1) are the lkelhood scores of character classfer, language model, unary and bnary geometrc contexts, respectvely, { λ 1, λ 2, λ 3, λ 4 } are four weghts that can be traned on a character strng dataset or determned emprcally, and k s the number of prmtve segments composng the canddate character x. The multpler k enables approxmate optmal path search by DP. 3. System Overvew The real-tme recognton system s dagrammed as n Fgure 1. It conssts of four man modules: real-tme recognton module, sentence recognton module, sentence edton module and language assocaton module. The modules of real-tme recognton and sentence recognton are the core of our approach, whle the other two modules make the system more usable. The real-tme recognton module acts whenever a new stroke s produced. The system determnes whether the stroke forms a new prmtve segment (stroke block) or not accordng to ts overlap wth prevous segments. If the stroke belongs to one prevous segment due to heavy x overlap, t s merged nto the prevous segment. Otherwse, t starts a new segment. Updated segments or newly created segments are merged wth prevous segments to generate new canddate characters, whch are recognzed by a character classfer to assgn canddate classes. The new canddate characters and assgned classes are added to the canddate segmentaton-recognton lattce. The detals of real-tme recognton wll be gven n Secton 4. Fgure 1. Flow chart of the system. Fgure 2. A canddate lattce (upper) and the updated one due to a new stroke (lower).
Fgure 2 shows an ntermedate canddate lattce and ts updated form due to a new stroke. After real-tme recognton on a new stroke, f the pen lft tme exceeds a threshold (adjustable by the user, e.g., 0.5s), the result of sentence recognton s obtaned by path search n the updated canddate lattce, performed by the sentence recognton module. The sentence recognton result may have errors of character segmentaton or recognton. A sentence edtng module s thus desgned to correct such errors. Character splt error can be corrected by drawng a crcle embracng the splt parts. Character merge error can be corrected by drawng a lne to separate the merged characters. After manual merge or splt, the merged or splt parts are re-combned nto canddate characters and re-assgned canddate classes, and the updated canddate lattce are re-searched for sentence recognton result. For character recognton error, canddate classes wll dsplayed when clckng on the character area, and the user can select the correct class. If the correct class s not n the top ranks, the user can erase the character and re-wrte. The language assocaton module ams to accelerate the wrtng process by automatcally enterng successve characters assocated wth the recognzed partal sentence. Ths module has not been mplemented yet. 4. Real-Tme Recognton Module In the followng, we focus on the mechansm of dynamc updatng of canddate segmentaton-recognton lattce. On a gven canddate lattce, optmal path search (sentence recognton) s performed based on the method descrbed n Secton 2. Specfcally, we use a tme-synchronous DP search algorthm for obtanng the sentence recognton result whenever the canddate lattce s updated upon a new stroke. Fgure 3 shows the flow chart of the real-tme recognton module. To make the sentence-based nput more convenent, the system allows the user to wrte multple lnes of sentences, and delayed strokes (nserted to a prevous wrtten part) are allowed. Algorthm 1 llustrates the real-tme process of a new stroke. Denote the stroke by strk. In the algorthm, segnum s the ndex of prmtve segment that the new stroke belongs to, and segnum=-1 ndcates that the stroke forms a new segment. pos s the poston (ndex) of the updated segment n the sequence. lnehe s the heght of text lne estmated from stroke blocks. The functon SameSegment(strk, s ) judges whether the stroke belongs to s or not, SortSegments( s m + 1, s1, s2,..., s m ) re-sorts the order of segments, and CanddateCharacters(pos, s1, s2,..., s m ) generates new canddate characters usng the updated segment sequence and the specfc segment poston. Fgure 3. Flow chart of the real-tme recognton module. Algorthm 1. Real-tme process of a new stroke Input: prmtve segment sequence s1, s2,..., s m a new stroke strk Intalzaton: set segnum=-1 For =m to 1 merge the boundng boxes of strk and s, denote the wdth and heght of the merged box as w and h, respectvely, f ( w> 2* lnehe OR h > 2* lnehe ) contnue; else f (SameSegment(strk, s ) ==TRUE ) segnum=, and pos=, break; End for. If (segnum>0) Merge strk nto s for updatng s Else Create a new segment s m + 1 usng strk, pos= SortSegments( s m + 1, s1, s2,..., s m ), m=m+1. End f. CanddateCharacters(pos, s1, s2,..., s m ), Recognze canddate characters, Update the canddate lattce. End.
4.1 Lne heght estmaton The estmated heght lnehe of a character strng s useful for judgng the plausblty of canddate characters. The lne heght s estmated by computng the average heght of stroke blocks. Frst, all the strokes are sorted n ascendng order of heght, and the half of strokes wth larger heght are used to estmate the lne heght. In the begnnng of wrtng, the system uses all the strokes for estmatng lne heght. Whle the wrtng proceeds, the estmate s updated usng the new stroke. The estmate becomes more precse when several strokes have been wrtten. 4.2 Relatonshp between stroke and segment The functon SameSegment(strk, s ) s computed as follows. Denote the wdth of the stroke s boundng box as w 1, and the wdth of the prmtve segment as w 2. Suppose the horzontal overlap between the two boundng boxes s w 0, the degree of overlap s defned by w o d =. (2) mn( w1, w2) The stroke s judged to belong to the prmtve segment when 1) d>0.4 or 2) d>0.2 and the stroke ntersects one stroke n the segment. 4.3 Sortng segment sequence If the new stroke belongs to one prevous segment (pos=), we need not to re-sort the segment sequence but only update the segment. If the stroke forms a new segment, we need to re-sort the segments n cases of multple lnes and delayed strokes. See Fgure 4 for an examples. The sentence conssts of two lnes, contanng prmtve segments s1s2,... s 9. Pos1, Pos2, and Pos3 are possble postons where the new stroke s nserted. Pos1 s most probable poston, and Pos2 and Pos3 correspond to delayed strokes. Delayed strokes also happen when a character s deleted n edton and a new character s re-wrtten n the same poston. Prmtve segments n one lne are ordered from left to rght accordng to the left boundary. For multple lnes, we use a smple rule to sort segments correctly. For segments that are not n the same lne wth the new segment s m + 1 (as the condton w > 2* lnehe OR h > 2* lnehe n Algorthm 1), reorderng s not needed. From segments whch are n the same lne wth s m + 1, we choose the segment whch s nearest to s m + 1 and left to s m + 1, and place s m + 1 mmedately after ths chosen segment. Fgure 4. Example of segment sequence nserted a new stroke. 4.4 Generatng new canddate characters On sortng the prmtve segments, generaton of canddate characters s straghtforward. Fgure 4 shows examples of new canddate characters. The segment wth red frame ndcates the one updated or formed by a new stroke, and the blue frame embraces the canddate characters that start from or end at the red segment. In ths example, the maxmum number of segments composng a canddate character s set as 3 for convenence. Lne 1 Lne 2 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 Fgure 5. Examples of new canddate characters. The generaton of canddate characters s subject to some heurstc rules for reducng the number of canddate characters whle guaranteeng ncludng true characters: a) The number of segments n a character does not exceed a maxmum number (6 n our case). b) Segments n dfferent lnes are not combned to canddate character. c) Canddate characters wth wdth large than a threshold ( 3*lneHe n our case) are pruned. d) Two successve segments wth horzontal dstance larger than a threshold ( 2*lneHe n our case) are not allowed to be merged. The newly generated canddate characters and ther canddate classes as well as classfcaton scores are added to the canddate segmentaton-recognton lattce.
5. Experments To justfy the flexblty of the proposed real-tme recognton approach, we have done prelmnary experments on a tablet PC. We used a character classfer and a character-level b-gram language model for path evaluaton of canddate lattce. The ncorporaton of geometrc context n the future s expected to mprove the recognton performance. The character classfer used n the system s an modfed quadratc dscrmnant functon (MQDF) classfer [10]. Each canddate character pattern s represented as 512D feature vector by trajectory-based moment normalzaton and local drecton hstogram feature extracton [11]. The feature dmensonalty s reduced from 512 to 160 by Fsher lnear dscrmnant analyss (LDA). The classfer was traned on a publcly avalable dataset of onlne handwrtten Chnese characters: CASIA-OLHWDB1 1 [12]. In the path evaluaton crteron (1), the weghts of language model λ 1 and character classfer λ 2 were emprcally set as 0.95 and 0.05, respectvely. We set the maxmum number of segments n a canddate character as 6, and the number of canddate classes n character classfcaton as 8. segmentaton-recognton lattce. From the trajectores of 化 and 研 n the thrd row of pctures, we can see that the dstance between two characters s smaller than that between the two radcals of 研, but the characters are stll segmented and recognzed correctly. Ths justfes the advantage of the ntegrated segmentaton-recognton framework. Fgure 7. Wrtng process and recognton results. To demonstrate more advantages of sentence-based nput over character-based nput, we show the canddate classes of the character 究 n Fgure 8. In the 8 canddate classes, the truth class of 究 ranks the ffth rather than the frst, but correct class s selected automatcally n sentence recognton due to the context. Specfcally, the b-gram 研究 s much more probable than the one 研乞. Obvously, by character-based nput, the character 究 s ms-recognzed, and needs human nteracton to select the correct class. Fgure 6. Interface of the recognton system. Fgure 6 shows the nterface of our recognton system, where the left sub-wndow dsplays the trajectores of handwrtng and the boundares of character segmentaton after sentence recognton, and the rght sub-wndow gves the text of sentence recognton result. Fgure 7 shows the wrtng process and ntermedate recognton results of a sentence. Whenever the pen lft tme s long enough, the partal sentence recognton result s gven by path search n the canddate 1 The database CASIA-OLHWDB1 was recently renamed as CASIA-OLHWDB1.0. Fgure 8. Canddate classes of a character. 6. Concluson and Future Works To fulfll the needs of the steadly expandng applcatons of sentence-based nput of handwrtten
characters, ths paper proposes an approach to real-tme recognton of Chnese handwrtten sentences. The man feature of the approach s a dynamcally mantaned segmentaton-recognton canddate lattce. The generaton and recognton of canddate characters, whch consume the majorty of computng, are performed durng the wrtng process. Thus, the sentence recognton result can be obtaned mmedately after the completon of wrtng or at a long-tme pen lft. Our prelmnary experments demonstrated the promse of the proposed approach. To make the system accepted by end users, more efforts are needed to mprove the recognton performance and edtng capablty. For segmentaton and recognton, we need to elaborate the procedures of text lne segmentaton, lne heght estmaton, prmtve segment updatng and sortng, and canddate character generaton rules. The path evaluaton crteron needs to ncorporate geometrc context as well as the language model and character classfcaton scores. The combnng weghts need to be optmzed usng a dscrmnatve tranng framework such as condtonal random feld (CRF) [6]. For applcaton on hand-held devces such as moble phones, low-complexty classfers should replace the MQDF for character classfcaton. The sentence recognton performance needs to be evaluated quanttatvely on a handwrtng dataset. The nearest prototype classfer can be a good choce [13][14]. The mplementaton of language assocaton module would vastly promote the text nput effcency, because many characters can be nput automatcally wthout wrtng. Acknowledgements Ths work s supported by the Natonal Natural Scence Foundaton of Chna (NSFC) under grants no.60775004, no.60825301 and no.60933010. A patent based on ths work has been documented and s pendng for grantng. References [1] C.-L. Lu, S. Jaeger, and M. Nakagawa, Onlne handwrtten Chnese character recognton: The state of the art, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 26, no. 2, pp. 198-213, 2004. [2] H. Murase, Onlne recognton of free-format Japanese handwrtngs, Proc. 9th ICPR, 1988, Vol.2, pp.1143-1147. [3] M. Nakagawa, B. Zhu, and M. Onuma, A model of on-lne handwrtten Japanese text recognton free from lne drecton and wrtng format constrants, IEICE Trans. Informaton and Systems, vol.e88-d, no.8, pp.1815-1822, 2005. [4] X.-D. Zhou, J.-L. Yu, C.-L. Lu, T. Nagasak, and K. Marukawa, Onlne handwrtten Japanese character strng recognton ncorporatng geometrc context, Proc. 10th ICDAR, 2007, Curtba, Brazl, pp. 48 52. [5] B. Zhu, X.-D. Zhou, C.-L. Lu, and M. Nakagawa, A robust model for on-lne handwrtten Japanese text recognton, Document Recognton and Retreval XVI (DRR), 2009, San Jose, USA, pp. 1 10. [6] X.-D. Zhou, C.-L. Lu, and M. Nakagawa, Onlne handwrtten Japanese character strng recognton usng condtonal random felds, Proc. 11th ICDAR, 2009, Barcelona, Span, pp. 521 525. [7] X.-D. Zhou, Methods for Onlne Handwrtten Japanese Document Analyss (In Chnese), PhD thess, Insttute of Automaton, Chnese Academy of Scences, Bejng, Chna, 2009. [8] M. Cheret, N. Kharma, C.-L. Lu, and C.Y. Suen, Character Recognton Systems: A Gude for Students and Practtoners, John Wley & Sons, 2007. [9] J.-L. Yu, X.-D. Zhou, and C.-L. Lu, Path search strateges for handwrtten character strng recognton, Chnese Journal of Pattern Recognton and Artfcal Intellgence, vol.22, no.2, pp. 182-187, 2009. [10] F. Kmura, K. Takashna, S. Tsuruoka, and Y. Myake, Modfed quadratc dscrmnant functons and the applcaton to Chnese character recognton, IEEE Trans. Pattern Analyss and Machne Intellgence, vol.9, no.1, pp.149-153, 1987. [11] C.-L. Lu and X.-D. Zhou, Onlne Japanese character recognton usng trajectory-based normalzaton and drecton feature extracton, Proc. 10th IWFHR, 2006, La Baul, France, pp.217-222. [12] D.-H. Wang, C.-L. Lu, J.-L. Yu, and X.-D. Zhou, CASIA-OLHWDB1: A database of onlne handwrtten Chnese characters, Proc. 10th ICDAR, 2009, Barcelona, Span, pp.1206-1210. [13] C.-L. Lu, One-vs-all tranng of prototype classfer for pattern classfcaton and retreval, Proc. 20th ICPR, 2010, Istanbul, Turkey, pp.3328-3331. [14] H. Zhang, D.-H. Wang, and C.-L. Lu, Keyword spottng from onlne Chnese handwrtten documents usng One-Vs-All traned character classfer, Proc. 12th ICFHR, 2010, Kolkata, Inda.