Online Character Segmentation Method for Unconstrained Handwriting Strings Using Off-stroke Features

Transcription

1 Onlne Character Segmentaton Method for Unconstraned Handwrtng Strngs Usng Off-stroke Features Naohro Furukawa, Junko Tokuno, Hsash Ikeda To cte ths verson: Naohro Furukawa, Junko Tokuno, Hsash Ikeda. Onlne Character Segmentaton Method for Unconstraned Handwrtng Strngs Usng Off-stroke Features. Guy Lorette. Tenth Internatonal Workshop on Fronters n Handwrtng Recognton, Oct 006, La Baule (France, Suvsoft, 006. <nra > HAL Id: nra Submtted on 6 Oct 006 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.

2 Onlne Character Segmentaton Method for Unconstraned Handwrtng Strngs Usng Off-stroke Features Naohro Furukawa Junko Tokuno Hsash Ikeda Central Research Laboratory, Htach, Ltd. -80 Hgash-Kogakubo, Kokubunj-sh, Tokyo, Japan E-mal: {naohro.furukawa.qv, Center for Innovaton and Intellectual Property, Tokyo Unversty of Agrculture and Technology -4-6 Naka-cho, Kogane-sh, Tokyo, Japan E-mal: Abstract In ths paper, an onlne character segmentaton method for unconstraned strngs s proposed. To recognze unconstraned-exsson strngs h as those n phrases and nformal notatons, we desgned physcal features of segmented patterns, n partcular, off-stroke features. Segmented-pattern lkelhood was also defned from these features usng a probablstc model. Evaluatons usng a dgtal pen system showed that the character segmentaton rates were 97.8%, 9.7%, and 75.6% of numerals, Japanese characters, and all characters (numerals, alphabets, symbols, and Japanese characters, respectvely. Keywords: character segmentaton, onlne recognton, box-free recognton, off-stroke feature, dgtal pen.. Introducton Paper s one of the most famlar meda to people and has many advantages, h as beng easy to read and wrte on. It s thus stll wdely used, even n today s nformaton socety. For example, when an end user wants to use servces at government offces, banks, or other h offces, he/she hands an applcaton form to an employee n the approprate secton, and that employee then ntates the applcaton processes accordng to the nformaton on the forms. We defne the sequences of all these processes between fllng n forms and startng the task as data entry. Many data entry systems usng pen and paper have been appled to busness tasks, h as notfcatons to government offces, remttances at banks, nsurance applcatons, patent-record updates n hosptals, mantenance records for utltes servces, and nventory management n warehouses. Onlne data entry systems h as a dgtal pen system [] [] have been ntroduced, and wth them, temporal nformaton can be used. To mprove user convenence, the rate of forms that are free of boxes that have to be flled n ( wrtng-box-free s ncreasng more and more. However, a system capable of recognzng not only constraned strngs h as addresses and names, but also unconstraned strngs h as phrases and nformal notatons s needed. Most wrtng-box-free recognton systems need a character segmentaton process that can determne correct segmented patterns from an nput strng. In general, the character segmentaton process s as follows. ( Pre-segmentaton: each character canddate pattern (segmented pattern from an nput strng s segmented; ths hypothess s output as a segmentaton graph. ( Feature extracton: feature vectors are extracted from each segmented pattern. (3 Calculatng segmented-pattern lkelhood: segmented-pattern lkelhood s calculated from segmented-pattern features. (4 Searchng a segmentaton path: a path s found on the segmentaton graph n whch the sum of segmented-pattern lkelhood has the best value. In the -segmentaton step, the segmentaton poston may not be unquely decded usng physcal nformaton. Thus, multple segmented-pattern canddates are created n the -segmentaton step and obtaned as a drected acyclc graph (DAG whch s called a segmentaton graph. Fgure shows an example of h a segmentaton graph for a text lne. Every segmented pattern n the segmentaton graph s tagged wth a segmented-pattern lkelhood, whch ndcates whether that pattern seems to have been correctly segmented. Fgure. Example of segmentaton graph.

3 Here, there are manly three types of nformaton for usng character segmentaton, as follows. ( Physcal nformaton: geometrc and temporal features h as wdth/heght of segmented patterns and wrtng tme of segmented pattern. ( Statstcal nformaton: co-occurrence frequency of concatenated characters, n-gram probablty, and so on. (3 Lngustc nformaton: usng a language model or a morphologcal analyss, etc. For an example of usng physcal nformaton, one method [3] uses eght shape features. Ths method can evaluate the character pattern canddates by lnear transformaton of ther feature vectors. An example of usng statstcal nformaton s a method [4] that uses the transton probablty of two characters (b-gram probablty. Ths method also uses physcal nformaton. An example of usng lngustc nformaton s a method [5] usng a TRIE-structure to obtan the exsson knowledge. Another example s usng a recursve transton network (RTN [6]. These methods can permt character-classfcaton falure and charactersegmentaton falure. In general, statstcal and lngustc nformaton are useful for constraned strng segmentaton. However, these types of nformaton cannot be appled or are ncorrect wth unconstraned strng segmentaton. Thus, for unconstraned strng segmentaton, t s mportant to use physcal nformaton. Even f for constraned strng segmentaton, physcal nformaton s used as a basc factor for statstcal and lngustc approaches. Some vously proposed methods [3] and [4] use physcal nformaton h as the wdth/heght of segmented patterns, the aspect rate of segmented patterns, gaps between segmented patterns, etc. The gap between segmented patterns s an especally useful feature. If all the ptches between characters n a strng are wde enough, h strngs can be segmented usng only the gaps. However, when ptches are narrow, or lkewse, some characters are touchng, a strng cannot be segmented correctly usng only tradtonal features. Therefore, we focused on the features of the offstroke. Even f some characters are touchng, the dstance and the tme perod of the off-stroke between the last stroke of the vous character and the frst stroke of the next character should both be bgger than those of the off-strokes that occur wthn characters. Consequently, we propose new off-stroke features. Segmented-pattern lkelhood combned wth these new features and tradtonal features by usng a probablstc model s also proposed. Ths paper reports our character segmentaton method and ts evaluaton tests.. Proposed Method.. Pre-segmentaton Strngs are usually flled n from left to rght. In ths case, f each segmented pattern s segmented accordng to the stroke order, the segmented pattern can be correctly determned. Wth touchng characters, t s only necessary to segment accordng to stroke order. However, a user sometmes nserts addtonal characters between vously wrtten characters. In ths case, usng the stroke order only would not work because any added characters would be regarded as the characters at the end of the strng. Thus, t s mportant to utlze not only temporal nformaton, but also geometrcal nformaton. The dgtal pen can obtan three varables ( x, y, t of samplng pont p : ( x-axs; ( y-axs; and (3 wrtng tme. A crteron for mergng strokes s calculated from these varables, and the segmented pattern s created accordng to the crteron. We focused on lnearly located characters, and a lnear sum functon of the samplng pont was defned as: ( x, y, t w0 + w x + w y + w t f 3 =. ( Ths functon converts three varables to the crteron. Here, varables of a whole stroke ( x, y, t were defned as varables of a samplng pont that had the smallest f of all samplng ponts. Thus, the lnear sum functon of a stroke was defned as follows. ( x y, t mn( f ( x, y t f, =,. (.. Segmented-pattern feature extracton We desgned 5 features for segmented patterns. These nclude the geometrcal and temporal features h as the wdth/heght of the segmented pattern, the aspect rate of the segmented pattern, and the wrtng tme of the segmented pattern (Table. In ths secton, we explan n detal our proposed features, the off-stroke features wthn segmented patterns and between segmented patterns. Table. Lst of segmented-pattern features. # Segmented-pattern features No. A Shape features of segmented pattern 6 B Poston features of segmented pattern 4 C Gap features wthn segmented pattern and 3 between segmented patterns D Length of strokes R Character classfcaton results F Off-stroke features wthn segmented pattern G Off-stroke features between segmented patterns 8

4 ... Off-stroke features wthn segmented pattern The dgtal pen we used can capture the tme perod used to wrte strokes and the wrtng order of strokes. Therefore, off-stroke nformaton can be obtaned from the last samplng pont of the vous stroke and the frst samplng pont of the next stroke. Both the dstance and tme of off-strokes wthn a correct segmented pattern have a relatvely small value. Usng ths property, we defned two segmented-pattern features (Table. Here, W e s the estmaton of the wdth of a correct segmented pattern and s defned as W e = {the maxmum wdth of segmented pattern n a strng}. N s the number of strokes n a segmented pattern. The other parameters are descrbed n Fgure.... Off-stroke features between segmentedpatterns Both the dstance and tme of off-strokes between segmented patterns have a relatvely large value f the patterns are correct. Here, when a focused segmented pattern s correct, two off-strokes, from the cedng stroke to the current stroke, and from the current stroke to the ceedng stroke, are both correct. In most correct off-strokes between segmented patterns, ther start ponts may be located n the bottom/rght poston, and ther end ponts may be located n the top/left poston because the start ponts of characters are manly n the top/left sde of the pattern, and the end ponts n the bottom/rght poston. Thus, we thnk that the angle of the off-stroke between segmented patterns s one of the most mportant factors. Usng ths property, we defned eght segmented-pattern features (Table 3. Table. Lst of off-stroke features wthn segmentedpattern. # Defntons Average dstance of off-strokes wthn segmented pattern: N F ( EndX StartX + ( EndY StartY N = W e Average tme of off-strokes wthn segmented pattern: F N ( StartTme EndTme N = ( EndX, EndY, EndTme ( StartX, StartY, StartTme Fgure. Parameters for calculaton of pattern features. Table 3. Lst of off-stroke features between segmented patterns. # Defntons Dstance of off-stroke from cedng stroke to current stroke: G ( EndX StartX + ( EndY StartY W e Tme of off-stroke from cedng stroke to G current stroke: StartTme EndTme Sne of the angle of off-stroke from cedng stroke to current stroke: G3 StartY EndY StartX EndX + ( StartY EndY ( Cosne of the angle of off-stroke from cedng stroke to current stroke: G4 StartX EndX StartX EndX + ( StartY EndY G5 G6 ( Dstance of the off-stroke from current stroke to ceedng stroke: ( StartX EndX + ( StartY EndY W e Tme of the off-stroke from current stroke to ceedng stroke: StartTme EndTme Sne of the angle of off-stroke from current stroke to ceedng stroke: G7 StartY EndY ( StartX EndX + ( StartY EndY Cosne of the angle of off-stroke from current stroke to ceedng stroke: G8 StartX EndX ( StartX EndX + ( StartY EndY

5 .3. Segmented-pattern lkelhood calculaton In ths paper, we suppose that each feature dstrbuton fts a normal dstrbuton. Segmented-pattern lkelhood can be calculated from means and varances of features usng a probablstc model. Let μ be a mean of a feature and σ be a varance. Then the probablty of x s P ( x μ, σ. Let x be the - th feature of a segmented-pattern. Here, the -th feature s segmented-pattern lkelhood L s as follows, L = log P( x μ, σ μ ( x = log(πσ. (3 σ We defned segmented-pattern lkelhood as the sum of the lkelhood of all features: L. (4 = L.4. Searchng for a segmentaton path Ths step fnds a path from the startng node to the endng node on the segmentaton graph n whch the sum of segmented-pattern lkelhood s the best value. We appled dynamc programmng (DP to ths step, as reported elsewhere [7] [8] [9]. Here, the -th node s score s defned as: L, S max { } { S } k + Lk, k L =. (5 k, L 4,5 L 5,6 N N N3 N 4 N5 N6 L,3 L,4 L 3,4 Fgure 3. Example of node score. 3. Expermental results { S + L S L } S + 4 = max,4, 3.. Evaluaton of the -segmentaton Frst, we collected,66 samples usng dgtal pens for the evaluaton of our -segmentaton method. These samples conssted of several knds of characters ncludng numerals, letters, symbols, and Japanese characters. There were 98 partcpants. Usng our -segmentaton method, we obtaned the segmentaton results and checked vsually to determne f there was a correct segmented pattern n the results. Table 4 shows ths evaluaton result. The accuracy of the -segmentaton was 00%. Samples wth touchng characters were segmented correctly. 3 3,4 However, more than 95 samples (3.4% had mstakes, for example, a wrong character wth a double lne as a correcton mark. In that case, our -segmentaton method was not able to segment characters correctly because of extra strokes. These samples were excluded from the evaluaton n ths paper. From a practcal pont of vew, a correcton mark recognton functon s necessary n the future. Table 4. Evaluaton result of -segmentaton. Correct Error 00% (,66 0% ( Evaluaton of the character segmentaton The samples for character segmentaton were collected usng dgtal pens from 387 people. We dvded the collected samples nto two data equally, for learnng and evaluaton. Table 5 gves detals of the sample sets for evaluaton. Usng the learnng sample sets, we pursued each mean and varance of segmented pattern feature. The evaluaton results are lsted n Table 6. For strng-based evaluaton, f even one segmented pattern n the path was not a correct pattern, the strng was counted as a falure. Evaluaton results showed that the character-based character segmentaton rates were 97.8%, 9.7%, and 75.6% of numerals, Japanese characters, and all characters, respectvely. Fgure 4 shows examples of correct segmentaton samples. Samples h as (a and (b have lttle character spacng; other samples h as (c, (d, and (e have touchng characters. Samples (f, (g, and (h have dfferent font szes and/or bg gaps wthn characters. Our proposed method was able to segment these samples correctly. For example, sample (e has two pars of touchng characters, and, and and. In ths case, our proposed method segmented them correctly. Moreover, the character whch has wde nternal gaps was also segmented correctly. However, some samples were segmented ncorrectly; these are shown n Fgure 5. There were manly fve types of falure: (a an over-segmented pattern; (b overmerged pattern; (c character classfcaton falure; (d small character msmerge; and (e a mxture of character szes. For example, n sample (a, 5 s over-segmented because of the wde nternal gap n the 5. Ths s especally lkely to occur when an nternal gap n the character wdens, because nternal gaps n numerals rarely exst. In addton, the falures n the over-segmented and over-merged patterns, h as n (a, (b, and (b, sometmes occur n strngs consstng of dfferent character szes, lke a mxture of Kanj characters and numerals. Ths s because the average pattern sze n the strng s dfferent from the sze of the Kanj characters

6 and the numerals. For nstance, (b contans many Kanj characters; thus the average pattern sze s bgger than the Hragana characters (e.g., enclosed n boxes. In ths case, the occurrence of over-merged patterns s slghtly hgher. In the case of (c, when a correct pattern s rejected by character classfcaton, character segmentaton tends to result n a mstake due to the use of the character classfcaton results as the pattern feature. The case of (d s a specal type of over-merged falures. Small characters h as commas and hyphens are sometmes msmerged wth neghborng characters. The case of (e s a falure accordng to mxture of character szes. In general, the wdth of Kanj characters s several tmes as large as the wdth of numerals and letters. Thus occurrence of numerals/letters over-merged or Kanj characters over-segmented s hgher, n ths case. To mprove h falures, we should use for the feature normalzaton not only global nformaton but also local nformaton h a mean wthn a short range. Table 7 lsts the comparatve experments of the character segmentaton methods usng physcal nformaton. The top lne n shows the result of the method usng physcal features descrbed by Senda, et al. [3]. The result of the second lne s from the method usng physcal features descrbed by Fukushma and Nakagawa [4]. From these results, we confrmed that our desgned off-stroke features are effectve nformaton for character segmentaton. Table 5. Sample sets for character segmentaton evaluaton. Set No. of char. No. of str. Char. /str. Strng propertes Set_N 0,588,077 Numerals, hyphens and 9.9 commas only. Set_K, Japanese characters only. All characters: numerals, Set_M 50,37 3, alphabets, symbols, and Japanese characters. Table 6. Evaluaton results of character segmentaton. Set Character based Strng based Set_N 97.8% (0, % (,87 Set_K 9.3% (,65 7.4% ( 35 Set_M 75.6% (38, % ( 736 All sets 8.3% (59, % (, Concluson We developed an onlne character segmentaton method for box-free strngs. To recognze strngs of unconstraned exssons h as phrases and nformal notatons, we desgned physcal features of segmented patterns, notably off-stroke features. Segmented-pattern lkelhood was also defned from these features usng a probablstc model. Evaluatons usng a dgtal pen system showed that the respectve character segmentaton rates were 97.8%, 9.7%, and 75.6% for numerals, Japanese characters, and all characters (numerals, alphabets, symbols, and Japanese characters. Moreover, from the comparatve experments, we confrmed that our desgned off-stroke features are effectve nformaton for character segmentaton. References [] [] N. Furukawa, H. Ikeda, Y., Kato, and H. Sako, D-Pen: a dgtal pen system for publc and busness enterprses, Proc. of 9th IWFHR, pp , 004. [3] S. Senda, M. Hamanaka, and K. Yamada, An Onlne Handwrtten Character Segmentaton Method of whch Parameters can be Decded by Learnng, Techncal Report of IEICE, PRMU97-9, pp.7-4, 998 (n Japanese. [4] T. Fukushma, and M. Nakagawa, On-lne Wrtng-boxfree Recognton of Handwrtten Japanese Text Consderng Character Sze Varatons, Proc. of ICPR 00, 000. [5] M. Koga, R. Mne, H. Sako, and H. Fujsawa, Lexcal Search Approach for Character-Strng Recognton, Proc. of DAS'98, Nov. 9, pp. 37-5, 998. [6] H. Ikeda, N. Furukawa, M. Koga, H. Sako, and H. Fujsawa, Context-Free Grammar-Based Language Model for Strng Recognton, Internatonal Journal of Computer Processng of Orental Languages, Vol. 5, No., pp , 00. [7] H. Bunke, A Fast Algorthm for Fndng the Nearest Neghbor of a Word n a Dctonary, Report of Insttut fur Informatk und Angewandte Mathematk, Unverstat Bern, 993. [8] F. Kmura, M. Shrdhar, and Z. Chen, Improvements of a lexcon drected algorthm for recognton of unconstraned handwrtten words, Proc. of nd ICDAR, pp. 8-, 993. [9] N. Furukawa, A. Imazum, M. Fujo, and H. Sako, Document Form Identfcaton Usng Constellaton Matchng, Techncal Report of IEICE, PRMU 00-5, Vol. 0, No. 4, pp. 85-9, 00 (n Japanese. Table 7. Comparatve experments usng all sets. Method Character-based Strng-based Usng features from [3] 69.% 3.0% Usng features from [4] 53.4% 5.4% Proposed method 8.3% 46.%

7 (a (b (c (d (e (f (g (h (a (a (b (b (c (d (e Fgure 5. Examples of segmentaton samples wth errors. Fgure 4. Examples of correct segmentaton samples.