Statistical classification of spatial relationships among mathematical symbols

Transcription

1 th Interntionl Conference on Document Anlysis nd Recognition Sttisticl clssifiction of sptil reltionships mong mthemticl symbols Wl Aly, Seiichi Uchid Deprtment of Intelligent Systems, Kyushu University 744 Motook, Nishi-ku, Fukuok-shi, Jpn Akio Fujiyoshi Deprtment of Computer nd Informtion Sciences, Ibrki University Nknrusw, Hitchi, Ibrki, , Jpn Mskzu Suzuki Deprtment of Mthemtics, Kyushu University , Hkozki, Higshi-ku, Fukuok-shi, Jpn Abstrct In this pper, sttisticl decision method for utomtic clssifiction of sptil reltionships between ech djcent pir is proposed. Ech pir is composed of mthemticl symbols nd/or lphbeticl chrcters. Specil tretment of mthemticl symbols with vrible size is importnt. This clssifiction is importnt to recognize n ccurte structure nlysis module of mth OCR. Experimentl results on very lrge dtbse showed tht the proposed method worked well with n ccurcy of 99.57% by two importnt geometric feture reltive size nd reltive position. 1 Introduction Automtic recognition of mthemticl expressions is considered s bsic process in converting scientific nd engineering documents into n electronic form. This process is composed of two prts; structure nlysis nd recognition of mthemticl symbols. There hve been mny ttempts to recognize mthemticl documents, n overview of previous ttempts re found in [1]. In this pper, we consider the structure nlysis prt, tht is, the utomtic clssifiction of sptil reltionships between ech djcent pir (herefter, simply clled discrimintion tsk). The sptil reltionships re composed of bseline, subscript, superscript, upper, nd lower reltions. Determintion of sptil reltionships is very importnt to recognize mthemticl expressions becuse the sme set of symbols convey different mening depending on the sptil reltionships. For exmple, b, b, b hvethe sme symbols but introduce different mening. Throughout this pper, we ssume the correct ctegory is given for every chrcters nd symbols; tht is, we ssume tht recognition of chrcters nd mthemticl symbols hs done lredy. This ssumption is rther relistic when we focus on the structure nlysis prt; in most mth OCRs, in fct, the structure nlysis is done fter recognizing individul chrcters nd symbols. The structure nlysis prt is discussed by mny reserchers strted from Anderson [7], who used purely syntctic pproch for prsing mthemticl expressions. Okmoto nd his collegues [8, 9] determined the sptil reltionships using geometric informtions. Znibbi etl. [10] presented system for recognizing typeset nd hndwritten mthemticl expressions. They used clsses of symbols to determine the reltionships. Suzuki etl. [11] introduced n OCR system clled INFTY to recognize mthemticl expression. They used geometric fetures to determine the sptil reltionships. Grin nd Chudhuri [12, 13] proposed n pproch for understnding mthemticl expressions, in which they used geometric informtion to determine sptil reltionships. Unfortuntely, most of the previous ttempts did not give detils bout the discrimintion tsk (e.g. [2, 3, 4, 5, 6]); they gve only the totl performnce of the system nd specified neither quntittive nor qulittive nlysis of their results. This my be becuse (i) the discrimintion tsk is one module of lrge mth OCR system, (ii) it employs mny heuristics whose detils re often hidden from reders, nd /09 $ IEEE DOI /ICDAR

2 Figure 2. Exmples of mthemticl expressions. Figure 1. The proposed method. Figure 3. Detection the type of symbol. (iii) it should be evluted with lrge-scle dtbse. The min contribution of this pper is to tckle the discrimintion tsk by sttisticl decision method grounded by huge dtbse. In the proposed method, the importnce of using document-dependent processing nd symbol types will be fully emphsized. In this method we will use two fetures clled reltive size nd reltive position. These fetures re very importnt to specify the sptil reltionships. For exmple, the reltive size H is used to discriminte between bseline nd non bseline clsses nd the reltive position D is used to discriminte mong subscript, superscript, upper, nd under clsses. Experimentl results reveled tht the discrimintion cn be done lmost perfectly ( 99.57%). Figure 1 shows n overview of the proposed method. Our tsk is the clssifiction of sptil reltionships between ech djcent pir (prent-child), where prent is the first symbol of ech pir nd child is the second symbol. The fetures reltive size H nd reltive position D will form distribution mps which plot the fetures in two dimensionl spce. The sptil reltionships re determined using Byesin clssifier which clssifies ech point in the distribution mps into one of 5 clsses. The proposed method introduces severl new techniques; for exmple, symbol types is used to compenste the vrition of symbol sizes; ech symbol hs type depending on its size nd position. These types re used to discriminte the distribution mps nd therefor ech symbol type is clssified in different distribution mp. Beside, document-dependent processing is introduced to improve the performnce of the discrimintion tsk. Furthermore, very lrge dtbses re used in the discrimintion tsk, these dtbses re suitble for the evlution of the usefulness of the proposed fetures. Our initil work on this tsk ws reported in [14]. In this work only the sptil reltionships mong chrcters were focused. Thus, the subscript pir of nd p nd the horizontl pir of D nd ( in Fig 2 were out of its scope. The present work is lrge extension of the previous work. In this work, we consider both symbols nd chrcters. The reminder of this pper is orgnized s follows. Section 2 introduces the fetures which were used in the distribution mp. Section 3 presents document dependent processing. Section 4 shows experimentl results with very lrge dtbses. Finlly, Section 5 presents conclusion. 2 Feture extrction for discriminting the sptil reltionships 2.1 Symbols types A type for ech symbol is defined to compenste the vrition in positions nd heights of symbols. Figure 3 shows n exmple of determining the type of symbol. To specify the type of symbol ; first the top nd bse of its prent chrcter is clculted using the hight rtio of three regions clled X:Y:Z regions. Figure 4 shows these regions. Then the type is decided ccording to the vlue of X-prt nd Z- prt. Figure 4. X, Y nd Z regions. 1351

3 Tble 1. Symbol types. Types Exmples of symbols X+Y+Z ] X+Y < > Y = Y+Z X â ǎ ă ã ȧ Z }{{} Figure 6. Symbol in different documents. α α () (b) k h 2 c 1 -c 2 (c) Figure 5. () Actul bounding box for chrcter α. (b) Normlized bounding box for chrcter α. (c) Normlized sizes (, h 2 ), normlized centers (c 1, c 2 ). Symbols will hve 6 types ccording to X:Y :Z regions such s ( X, X + Y, X + Y + Z, Y, Y + Z, Z ). Tble 1 shows some exmples of symbol types. This estimtion of symbol types improves the performnce of the proposed method. These types re used to discriminte the distribution mps nd therefore ech type will be clssified in different distribution mps. 2.2 Feture Extrction As stted in [14], to estimte the proposed fetures, the normlized bounding boxes re used for chrcters insted of ctul bounding boxes. Figure 5 () shows the ctul bounding box for chrcter α nd Fig 5 (b) shows the normlized bounding box for chrcter α. The normlized bounding boxes re estimted by dding virtul scender or virtul descender or both depending on the chrcter ctegory. Unfortuntely, this technique does not vlid for symbols; symbols hve more vrition thn chrcters. For exmple, some symbols such s, = hve smller hight thn Y hight nd others such s [, hve longer heights thn X + Y + Z heights. Therefore, the ctul bounding boxes re used for symbols. c 2 c 1 Let nd h 2 denote the heights of bounding box of the prent nd child, respectively. Similrly, let c 1 nd c 2 denote the centers of the bounding box of those pirs. Figure 5 (c) shows these prmeters. The reltive size H nd the reltive position D cn be extrcted for ech djcent pir s follows: H = h 2, D = c 1 c 2. (1) 3 Document dependent processing Ech document hs its own chrcteristics which differers from the other document. Observing these chrcteristics improved the performnce of the proposed method. 3.1 Privte X:Y:Z We estimte X:Y:Z rtio for bseline chrcters nd X:Y:Z rtio for non-bseline chrcters becuse they hve different font shpes 1. These rtios re common for ll chrcters in ech document nd therefore we herefter cll them privte X:Y:Z rtios. In contrst, we lso cn estimte the X:Y:Z rtios common for ny document by using ll the chrcters of lrge-scle multi-document dtbse nd therefore we herefter cll the rtios common X:Y:Z rtios. PrivteX:Y:Z rtio outperforms common X:Y:Z rtio s will be shown in the experimentl results. 3.2 Irregulr symbols nd chrcters After determining the symbol types, we noticed tht, some symbols hve different types in different document. These symbols will clled irregulr symbols. For exmple, 1 Reders my be confused by the fct tht we need to discriminte between bseline chrcters nd non-bseline chrcters for estimting their own X:Y:Z rtio during the process towrd our finl gol. For this discrimintion we used X:Y:Z which clculted from ll chrcters contined in the dtbse. Of course, the result from this discrimintion includes some errors. These errors do not ffect the estimtion seriously becuse we use the verge of the heights. 1352

4 Figure 7. Exmples of irregulr chrcters. symbol occupies only Y region in some documents nd occupies X + Y regions in nother documents, yet occupies the entire X + Y + Z regions in the other documents nd therefor they hve 3 different types. Figure 6 shows these types. As stted in [14], there re some chrcters which hve different sizes nd occupy different X, Y, Z regions in different documents. These chrcters re clled irregulr chrcters. Figure 7 shows some exmples of these chrcters. Specil tretment is pplied for irregulr chrcters/symbols, in which ech irregulr chrcters/symbols ws discriminted into mny cses depending on its heights nd positions. This specil tretment improved the performnce s will be shown in the experimentl results. 4 Experimentl results 4.1 Dtbse The discrimintion tsk ws pplied on 158,308 djcent pir of symbols nd chrcters of mthemticl expressions. For exmple, 37,263 djcent pirs hve chrcterschrcters type, 52,057 djcent pirs hve symbolschrcters type, 54,924 djcent pirs hve chrcterssymbols type, nd 14,064 djcent pirs hve symbolssymbols type. To the uthors best knowledge, these dtbses re the lrgest of those used in pst ttempts on the discrimintion tsk. For exmple, they re lrger thn the dtbse used in [18], which consists of 297 pges. Such lrge dtbses re extremely well suited to derive universl properties (e.g., the discrimintion tsk) of mthemticl expressions. 4.2 Distribution mps nlysis Figure 8 () shows the distribution mp without using symbol types. Hevy overlps between the clsses cn be Tble 2. Discrimintion ccurcy (%) by qudrtic clssifier on H-D spce. No. of Doc. dependent Accurcy symbol processing rte type irregulr privte tretment X:Y:Z rtio 1 X X X O O X O O X X X O O X O O observed on this mp. These overlps come from the vrition of the sizes nd positions of the ctul bounding boxes of symbols. For exmple, horizontl symbols which occupy X + Y regions my overlpped with superscript symbols becuse their centers will be up. Figure 8 (b) shows the distribution mp fter using symbols types. The overlps were drsticlly decresed becuse we voided the vrition of the sizes of the symbols; symbols with the region ( X, X+Y, X+Y +Z, Y, Y +Z, Z ) were clssified in different distribution mps. Thus we cn conclude tht, using the types of symbols is very powerful for the discrimintion tsk with (H, D)-fetures. Tble 2 shows the ccurcy rte using simple Byesin clssifier. From this tble, we notice tht, the results improved very much fter using symbol types. The best ccurcy rte occurred when pplying document-dependent processing, in which privte X:Y:Z rtio nd specil tretment for irregulr chrcters/symbols were pplied. These results prove the importnce of using both symbol types nd document-dependent processing in the discrimintion tsk. 5 Conclusion In this pper, the sptil reltionships between ech djcent pir of mthemticl expressions is clssified into one of five clsses (horizontl clss, subscript clss, superscript clss, upper clss, nd lower clss) for relizing n ccurte structure nlysis module of mth OCR. In this tsk very lrge dtbses re used which re suitble to covey the geometric informtion of chrcters nd symbols of mthemticl expressions. Experimentl results shows tht, symbol types nd document dependent processing improved the performnce of the proposed method by observing the distribution mps which defined by two fetures reltive size nd reltive position. These two points were overlooked in the 1353

5 ()Without using symbol types for ll symbol-symbol pirs. (b) Using symbol types for symbol-symbol pirs, symbols hve the X + Y + Z type. Figure 8. Distribution mps for different cses. The curves show the decision boundries by qudrtic clssifier. The vlues of H nd D re multiplied by pst ttempts, while they give us n importnt spect tht documents-dependent processing re necessry on the structure nlysis. References [1] K. Chn, nd D. Yeung, Mthemticl expression recognition: A survey, Int. J. Document Anlysis nd Recognition, vol. 3, no. 1, pp. 3 15, [2] J. H, R. M. Hrlick, nd I. T. Phillips, Understnding mthemticl expressions from document imges, Proc. 3rd Int. Conf. Document Anlysis nd Recognition, vol. 2, pp , [3] J. -Y. Toumit, S. Grci-Slicetti, nd H. Emptoz, A hierrchicl nd recursive model of mthemticl expressions for utomtic reding of mthemticl documents, Proc. 5th Int. Conf. Document Anlysis nd Recognition, pp , [4] Y. Guo, L. Hung, C. Liu, nd X. Jing, An utomtic mthemticl expression understnding system, Proc. 9th Int. Conf. Document Anlysis nd Recognition, vol. 2, pp , [5] H. J. Lee, M.c. Lee, Understnding mthemticl expressions using procedure-oriented trnsformtion, Int. Journl of Pttern Recognition, vol. 27, no. 3, pp , pp , [6] D. Blostein, nd A. Grbvec, Recognition of mthemticl nottion, In Hndbook of Chrcter Recognition nd Document Imge Anlysis, pp , [7] R.H. Anderson, Syntx-directed recognition of hndprinted two-dimensionl mthemtics, in Interctive Systems for Experimentl Applied Mthemtics, M.Klerernd J. Reinfelds, Eds. Acdemic Press, pp , [8] M. Okmoto, nd B. Mio, Recognition of mthemticl expressions by using the lyout structure of symbols, Proc. 1st Int. Conf. Document Anlysis nd Recognition, pp , [9] H. Twkyondo, nd M. Okmoto, Structure nlysis nd recognition of mthemticl expressions, Proc. 3th Int. Conf. Document Anlysis nd Recognition, pp , [10] R. Znibbi, D. Blostein, nd J.R. Cordy, Recognizing mthemticl expressions using tree trnsformtion,, vol. 24, no. 11, Int. J. Pttern nlysis nd mchine intelligence, pp , [11] M. Suzuki, F. Tmri, R. Fukud, S. Uchid, nd T. Knhori, INFTY- An integrted OCR system for mthemticl documents, Proc. Int. Conf. ACM Symposium on Document Engineering, pp , [12] U. Grin, nd B. B. Chudhuri, A syntctic pproch for processing mthemticl expressions in printed documents, Proc. Int. Conf. Pttern Recognition, vol. 4, pp , [13] U. Grin, nd B. B. Chudhuri, An pproch for recognition nd interprettion of mthemticl expressions in printed documents, Proc. Int. Conf. Spring-Verlg London, vol. 3, pp , [14] A. Wl, S. Uchid, nd M. Suzuki, A Lrge-Scle Anlysis of Mthemticl Expressions for n Accurte Understnding of Their Structure, Proc. 8th Int. Document Anlysis Systems, pp , [15] M. Suzuki, S. Uchid, nd A. Nomur, A Ground-truthed mthemticl chrcter nd symbol imge dtbse, Proc. 8th Int. Conf. Document Anlysis nd Recognition, pp , [16] S. Uchid, A. Nomur, nd M. Suzuki, Quntittive nlysis of mthemticl documents, Int. J. Document Anlysis nd Recognition, vol. 7, no. 4, pp , [17] M. Suzuki, C. Mlon, nd S. Uchid, Dtbses of mthemticl documents, Reserch Reports on Informtion Science nd Electricl Engineering of Kyushu University, vol. 12, no. 1, pp. 7 14, [18] U. Grin nd B. B. Chudhuri, A corpus for OCR reserch on mthemticl expressions, Int. Journl Document Anlysis nd Recognition, vol. 7, no. 4, pp ,