On-line cursive letter recognition using sequences of local minima/maxima. Robert Powalka

Transcription

1 O-lie cursive letter recogitio usig sequeces of local miima/maxima Summary Robert Powalka 19 th August 1993 This report presets the desig ad implemetatio of a o-lie cursive letter recogizer usig sequeces of local extrema. Recogitio tests were performed o sigle writer s data. The results for top five alteratives exceeded 90% (Table 4). The matchig is performed i three stages: letter size ormalisatio (Figure 1); limitig the recogitio domai usig sequeces of local extrema ad iexact matchig (Figure 3) with a database; evaluatig the goodess of match for each elemet of the obtaied limited recogitio domai. Two methods of letter size ormalisatio were cosidered ad tested: uiform (Figure 1b); o-uiform (Figure 1c). The uiform ormalisatio approach proved to be better (Figures 5, 6 ad 7). Three differet criteria are used for the last stage, resultig i three differet little recogizers : positio of the local extrema (Figure 4a); characteristic loci (Figure 4b); loci agles (Figure 4c). The results of the three recogizers could be combied together i order to improve the recogitio rate. The recogizer eeds to be traied. The databases are built i two mai stages: creatig the database of sequeces of local extrema; derivig detailed feature data for rakig the alteratives. The first stage is curretly maual ad may ot lead to the best local extrema sequece database. The secod stage ca still eed some supervisio, depedig o the specific extrema sequece database. The recogizer possesses a potetial for icremetal learig. At ru time both ew sequeces of local extrema ca be added, ad the detailed features used for the rakig ca be adjusted to better reflect a particular writig style. Further work is ecessary: tests with other writers data; coversio of the differece betwee patters ito the cofideces of the match; combiatio of results of differet classifiers; itegratio ito the o-lie cursive script recogitio system, testig; compariso with the existig letter recogizer usig vector directio ecodig. Page 1

2 1. Itroductio Curretly developed by the author o-lie cursive script recogitio system uses vector ecodig [1][5] to perform letter segmetatio ad recogitio. While the system provides promisig results [8], it is clear that it eeds to be improved. Oe area for improvemet ivolves the letter recogizer curretly used. The letter recogizer curretly implemeted uses Freema-like vector directio ecodig of iput data, strict matchig with a database of kow shapes ad multi-scale recogitio [5]. The multi-scale recogitio approach results i icreased robustess, but it ca also geerate superfluous letter alteratives. This report describes a prototype of aother o-lie cursive letter recogizer which uses differet methods to achieve the recogitio. The methods used are described ad prelimiary results reported. 2. Recogitio The letter recogizer uses sequeces of local miima ad maxima [6][7]. These are matched with a database of kow local extrema sequeces to limit the recogitio domai. Further aalysis is the performed withi obtaied recogitio domai i order to rak the obtaied letter alteratives. Three differet methods are used at this stage: extrema positio; characteristic loci; loci agles. This approach effectively results i three letter recogizers, each capable of producig differet results. The recogitio process cosists of three mai stages: letter ormalisatio; matchig; refiig matchig results. The described recogizer eeds to be traied, that is appropriate iformatio ecessary for the recogitio eeds to be obtaied ad stored i a database. This process curretly ivolves a maual stage Normalisatio All matched letters are size ormalised. This process is ecessary for the refiemet of the matchig results stage as positio ad size depedet features are used. Two approaches to letter size ormalisatio have bee cosidered: uiform - all the letters are ormalised to the same size; o-uiform - differet letters are ormalised to differet sizes, depedig o their characteristic. The uiform scalig appears obvious. However, lower case letters of lati alphabet are iheretly of differet size (e.g. Figure 1a). Thus scalig all of them to the same size seems to cotradict the letter defiitio ad is likely to distort their shapes (see Figure 1b). This is the reaso for itroducig the o-uiform letter scalig, which seems to better reflect differeces i the letter sizes ad better preserve the letter shape (Figure 1c). Page 2

3 a) b) c) Figure 1: Size ormalisatio of letter patters: a) iheret size differeces; b) uiform ormalisatio; c) o-uiform ormalisatio. For both approaches ecessary scalig factors will likely be differet i X ad Y axes. Whe the differece becomes high, letters ca be sigificatly distorted (e.g. letter l looks more like e after scalig i Figure 1b). O the other had, usig idetical scalig factors i both axes is likely to result i poorly ormalised patters, as idetical letters ca have differet height/ width proportios depedig o the writer. A compromise was chose. A attempt is made to ormalise every letter patter i both axes. However, if the ratio of scalig factors i differet axes is too high, the X axis scalig factor is adjusted so it does ot exceed the threshold. This approach results i letter patters well ormalised i the Y axis (the height is cosidered more importat) ad possibly less well ormalised i the X axis. The threshold for the scalig factors ratio has bee arbitrarily set to +/-50%. The o-uiform ormalisatio uses four differet ormal sizes. Each letter of the alphabet is assiged the most suitable ormal size (Figure 2). a) b) c) d) 2 2 1/2 1/2 Figure 2: No-uiform size ormalisatio: a) letters a, c, e, m,, o, s, u, w, x, z; b) letters b, d, g, h, k, p, q, y; c) letters f, j, l, t; d) letters i, r, v. Agai, a compromise was ecessary: letters m ad w should be wider tha others of their class ad the letter f should probably be taller tha others of its class. This, however, would icrease the ecessary umber of ormal size classes. Whe a ukow patter is matched, all the possible ormalisatios are performed ad the the appropriate versio is selected for each database patter. Experimets were performed usig both uiform ad o-uiform ormalisatio. Page 3

4 2.2. Matchig The matchig process uses sequeces of local extrema. The local extrema are located by previously described method [7]. They are combied ito a sequece accordig to the local extrema occurrece alog the path of the pe. The ukow sequeces of local extrema are matched with a database of kow sequeces. As the variability of the local extrema ecodig scheme is observed to be rather high, a iexact matchig process is used. The ukow sequece ca have oe extra extremum betwee each two cosecutive extrema i the database sequece. The umber of the omitted extrema (referred to as the match distace) is calculated ad ca provide some estimate of the goodess of match. Figure 3 presets examples of the matchig process ad the priciples of calculatig the match distace distace = distace = distace = distace = 5 Figure 3: Examples of relaxed sequeces matchig. Bottom sequece is the referece. For the four elemet log sequece the match distace ca vary from 0 (ideal match) to 5 (poorest match). The iexact sequece matchig process is oly capable of limitig the recogitio domai. It usually produces up to te letter alteratives, occasioally cosiderably more. Further refiemet is ecessary i order to fid the best letter alterative Refiig matchig results The results of matchig the local extrema sequece eed to be refied i order to estimate the cofidece of the match ad thus choose the best letter alterative. The local extrema are used agai, but this time differet, more specific features are exploited: positio of the extrema; characteristic loci; loci agles. Average values of differeces betwee these features are calculated. These represet the similarity betwee the ukow ad the database patters. The smaller the result the closer the match is obtaied. Page 4

5 d a) b) c) α β D d dist d = dist D d = dist = α β Figure 4: Three criteria for refiig the matchig results: a) local extrema positio; b) characteristic loci; c) loci agles Extrema positio This classifier calculates the average distace betwee the appropriate local extrema poits i the ukow ad i the database patters (Figure 4a) Characteristic loci This classifier calculates the average differece betwee the loci of the local extrema poits i the ukow ad i the database patters (Figure 4b). The loci are calculated as distaces betwee the cetre of gravity of the patter ad the local extrema. The cetre of gravity is calculated usig oly the coordiates of the local extrema poits Loci agles This classifier calculates the average differece betwee the loci agles of the local extrema poits i the ukow ad i the database patters (Figure 4c). The loci agles are the agles betwee the horizotal lie ad the lie marked by the cetre of gravity ad the extrema poits Dealig with the local extrema ot preset i the database patter Figure 4 depicts the match refiemet process for patters where match distace is zero. If the match distace is greater, the extra local extrema are igored i the goodess of the match calculatio. However, the match distace does ifluece the patter goodess. The goodess of the patter is icreased proportioally to the value of the match distace. Curretly the followig formula is applied: PatterGoodess = PatterGoodess MatchDist Potetial for results combiatio The use of three differet criteria for evaluatig the goodess of match results effectively i three differet letter recogizers. Each of the recogizers is capable of producig results differet to those of other recogizers. It is expected that the combiatio of the recogizers could provide better results tha idividual recogizers [2][4]. Page 5

6 2.4. Traiig The letter recogizers eed to be traied. This process is performed i two stages: creatig database of local extrema sequeces; collectig iformatio about the derived features used for the refiemet of the extrema sequece matchig Extrema sequeces Buildig the database of the extrema sequeces is curretly a maual process. This is due to the fact, that the database eeds to be both compact ad geeric. Hece it is ecessary to abstract from particular local extrema sequeces that ca easily be obtaied from the traiig data set. Curretly, the abstractio is performed by the author ad is very much a hit ad miss process. It is however expected that correlatig differet sequeces for the same letter ad locatig most ivariat parts of the sequeces over a large traiig data set should produce the ecessary database. This would eed to be verified. O the other had, sice the database is to be abstract ad geeric, it is expected that a comprehesive database could be built. This would eed to be verified, too Derived features Derived features are calculated automatically. The features are calculated for each patter i the traiig set ad accumulated for the appropriate database patters. Upo the completio of the traiig all the features are averaged ad stored i the database. If the extrema sequece database cotais ay sequeces which are ot ecoutered i the traiig set, such database etries have ull values. Further treatmet of such etries may vary: supply extra traiig data; remove these etries from the database; use values from similar etries to fill i blaks. Curretly the last approach is used ad this process is performed maually Potetial for icremetal learig Described letter recogitio method possesses a potetial for icremetal learig. The database of the extrema sequeces ca be expaded by user-specific sequeces at ru-time. The derived features ca be modified at ru-time to closer reflect user-specific hadwritig style. 3. Results A experimet was performed to asses the effectiveess of the described approach. Three data sets were used: oe for traiig (A) ad two for testig (B ad C). All the data have bee filtered usig a cross product filter [3][5], idetical to the oe used by the o-lie cursive script recogitio system developed by the author. All data were obtaied from oe writer. Data sets A ad B were writte as sigle letters ad the ligatures were maually removed, data set C cosists of letters separated from cursive words (~rkp/data/cr/200words/rkp/*). Table 1 presets the data details. Data set Number of letter samples Data set locatio A 517 ~rkp/data/cr/letters/rkpltr1.ik B 511 ~rkp/data/cr/letters/rkpltr2.ik Commets writte as sigle letters Table 1: Details of data used for the experimet. Page 6

7 Data set The experimet cosisted of two mai stages: traiig; recogitio tests. Followig software was used: local extrema sequece extractio: ~rkp/script/features/xfeatures versio 1.6 of 12/8/1993, optio trai; extrema sequece recogizer: ~rkp/script/extrema/exsrecog versio 1.0 of 16/8/1993; traiig program: ~rkp/script/extrema2/extrai versio 1.0 of 16/8/1993; letter recogizer: ~rkp/script/extrema2/extrecog versio 1.0 of 16/8/1993. Followig databases were obtaied as a result of traiig ad used for the recogitio experimets: database for o-uiform ormalisatio approach: ~rkp/script/extest/ex1mod.db SCCS revisio 1.1; database for uiform ormalisatio approach: ~rkp/script/extest/ex1mod.uiform.db SCCS revisio Traiig Number of letter samples Data set locatio C 684 ~rkp/data/cr/letters/ltrdata.ik separated from cursive words Table 1: Details of data used for the experimet. The traiig was performed i four stages: geeratig sequeces of local extrema for all the traiig data (automatic); abstractig the results of the previous stage to create the database of codig sequeces (maual); derivig detailed feature data for the sequece database usig the traiig data (automatic); geeratig iformatio for database patters that did ot occur i the traiig data (maual). Creatig the database of codig sequeces is the most importat stage. It is ecessary to obtai possibly small yet as comprehesive as possible set of geeralised extrema sequeces that ca be matched with the specific sequeces. The process was performed iteratively. A umber of sequeces were ituitively created ad the matched with the traiig data to fid out how much of the traiig data such created database ca recogise. The extrema sequece recogizer was used. The both urecogised sequeces, ad the cotets of the database were aalysed ad the database expaded. The process was repeated. A small umber of sequeces were decided too uusual to be icluded i the database. This is the reaso for less tha 100% recogitio results for the traiig data set (Table 4). A effort was made to create possibly comprehesive extrema sequece database. As a result a umber of sequeces were created, which could ot be derived from the traiig data. Obviously, it was impossible to obtai the detailed feature data for such sequeces. This problem was solved by maually choosig similar sequeces ad copyig their data Variability ad ambiguity of the database Commets I order to estimate variability of the sequeces database, the distributio of the umber of patters represetig each letter was calculated ad compared with aalogous results for the traiig data. Table 2 presets the results. Note that for some letters ( k, x ) there are more patters i the database tha were ecoutered i the traiig data. This is due to may differet Page 7

8 ways of writig these letters, which were catered for. Number of sequeces per letter is cosiderably lower i the sequece database tha i the traiig data (most otably for letters f, g, m, z ). The decrease of variability of the database was possible due to the use of iexact matchig. Letter Number of letters i the data sample Number of patters per letter traiig data sequeces database a b c d e f g h i j k l m o p q r s t u v w x y z Number of differet patters Table 2: Number of sequeces for each letter of the alphabet. I order to estimate ambiguity of the obtaied database, the distributio of the umber of patters ecodig specified umber of letters was obtaied. Table 3 presets the results. Over half of the sequeces withi the database idetify letters uiquely ad there are oly two patters which ecode more tha five letters. The local extrema sequeces database exhibits reasoably low ambiguity ad variability. This however icreases cosiderably because a iexact matchig techique is used. Page 8

9 Number of letters ecoded by a patter 3.3. Recogitio Number of patters ecodig specified umber of letters traiig data sequeces database Total Table 3: Number of sequeces that ecode specified umber of letters. Recogitio tests were performed for three data sets. Table 4 presets the results whe top five letter alteratives are take ito cosideratio. The recogizer was tested usig both uiform ad o-uiform ormalisatio approach. It ca be observed that the ormalisatio method used has very little ifluece over the top five alteratives recogitio results. The results for the traiig data set are ot 100% due to the fact that a few extrema sequeces were deliberately omitted durig the traiig. Criterio Extrema positio Characteristic loci Loci agles Normalisatio o-uiform uiform o-uiform uiform o-uiform uiform Data set rkpltr1 99.3% 99.2% 98.8% 99.3% 97.7% 97.7% rkpltr2 94.3% 94.3% 92.4% 93.7% 92.8% 92.6% ltrdata 91.5% 91.0% 90.5% 89.4% 87.2% 87.7% Table 4: Results of the recogizer for top five letter alteratives, differet classifiers ad ormalisatio methods. Figures 5, 6 ad 7 preset more detailed recogitio results. It ca be observed that, i geeral, the uiform size ormalisatio provides better results for the top letter alterative. The extrema positio criterio provides least ambiguous results i all the tests. It ought to be remembered that oly sigle writer data was used for the tests. The results might differ whe tests are performed o data geerated by multiple writers. Page 9

10 [%] Figure 5: Recogitio results for the traiig data set A. Shades represet rakig of the correct result. [%] Extrema positio Characteristic loci Loci agles Extrema positio Characteristic loci Loci agles Figure 6: Recogitio results for the test data set B. Shades represet rakig of the correct result. Page 10

11 [%] Results for the test data set C are lower tha for the other data sets. This is most probably the result of differet way this data set was obtaied. However, the coditios i which these data were obtaied are earer to the real workig coditios of the recogizer. It is expected that the database of local extrema sequeces eeds to be expaded i order to better accommodate for the variatios occurrig i cursive writig. The filterig thresholds applied to letters i the data set C were derived usig etire cursive words, with o prior kowledge of the zoig. I case of data sets A ad B, the zoig kowledge was explicitly used ad the filterig thresholds derived usig idividual letters. Hece differeces i data filterig results are likely. It is expected that the letter recogizer ca adapt to the particular hadwritig style ad icrease the recogitio rate. 4. Discussio Extrema positio Characteristic loci Loci agles Figure 7: Recogitio results for the test data set C. Shades represet rakig of the correct result. The preseted results are very promisig. Letter recogitio rates over 90% for oe writer are achieved usig very simple methods. It is expected that the combiatio of the results of three differet classifiers will provide further improvemet. Algorithms for combiig recogitio results eed to be ivestigated. The results combiatio is ecessary both to provide sigle output from the three differet classifiers, ad to combie the results of the described recogizer with those of the recogizer usig vector directio sequeces. Two differet scearios for combiig together the recogizers ca be evisaged: multi-layered: first combie results withi the local extrema recogizer, the combie the output with the results of the vector ecodig recogizer; sigle-layered: treat the results of all four recogizers equally ad combie them together i oe stage. It is ecessary to test the performace of the vector ecodig recogizer. The results should help to decide whether the recogizers should be treated equally or they should be assiged differet priorities. Curretly, tests have bee performed oly o sigle writer s data. Further tests eed to be performed i order to asses the effectiveess of the recogizer for multiple writers data. Page 11

12 The recogizer does ot calculate cofideces of letter alteratives as yet. Curretly, it raks the results usig the distace calculated betwee the ukow ad the database patters. Methods to covert this distace ito a match cofidece eed to be ivestigated. Cotrary to the expectatios, the uiform letter size ormalisatio provides better results. At the same time it is much simpler to implemet. This ormalisatio method will be chose for the fial implemetatio. A geeric extrema sequece database is eeded. It is believed that a comprehesive sequece database ca be obtaied. A compromise will probably be ecessary betwee the comprehesiveess ad the compactess. More data eed to be processed to achieve this aim. Automatio of the extrema sequece database buildig is desired. 5. Future work This report presets iitial results. Further work is ecessary: tests with other writers data eed to be performed; method for covertig the differece betwee patters ito the cofideces of the match eeds to be desiged; combiatio of results of differet classifiers eeds to be ivestigated ad implemeted; the local extrema recogizer eeds to be itegrated ito the o-lie cursive script recogitio system developed by the author. The etire system the eeds to be tested ad the results compared with the already obtaied oes; strategies for usig two letter recogizers withi the recogitio system eed to be ivestigated: two differet letter recogizers at the same time, sequetially, or oe of them exclusively; the existig letter recogizer usig vector directio ecodig eeds to be separated from the recogitio system ad tested separately, usig the same data sets as i the described experimet. This will make the compariso of the two letter recogizers possible. Refereces [1] H. Freema. Computer processig of lie-drawig images. Computig Surveys, Vol. 6, No. 1, pp , 1974 [2] T.K. Ho, J.J. Hull, S.N. Srihari. A regressio approach to combiatio of decisios by multiple character recogitio algorithms. Machie Visio Applicatios i Character Recogitio ad Idustrial Ispectio, Proc. SPIE 1661, pp , 1992 [3] J.S. Lipscomb. A traiable gesture recogizer. Patter Recogitio, Vol. 24, No. 9, pp , 1991 [4] T. Matsui, I. Yamashita, T. Wakahara, M. Yoshimuro. State of the art of hadwritte umeral recogitio i Japa - the results of The First Character Recogitio Competitio. First Europea Coferece o Postal Techology JET POSTE 93, Nates, Frace, pp. 3-10, Jue 1993 [5] R. Powalka. Aual Report, Departmet of Computig, Nottigham Polytechic, 7th October [6] R. Powalka. Usig sequeces of local miima/maxima for o-lie cursive letter recogitio. Iteral Report, Departmet of Computig, The Nottigham Tret Uiversity, 3rd Jue [7] R. Powalka. Differet method of extractig sequeces of local miima/maxima for o-lie cursive letter recogitio. Iteral Report, Departmet of Computig, The Nottigham Tret Uiversity, 2d July [8] R.K. Powalka, N. Sherkat, L.J. Evett, R.J. Whitrow. Multiple word segmetatio with iteractive look-up for cursive script recogitio. To be preseted at Secod IAPR Coferece o Documet Aalysis ad Recogitio, October 1993, Tsukuba Sciece City, Japa. Page 12