Improving online handwritten mathematical expressions recognition with contextual modeling

Transcription

1 Improvng onlne handwrtten mathematcal expressons recognton wth contextual modelng Ahmad-Montaser Awal, Harold Mouchère, Chrstan Vard-Gaudn IRCCyN/IVC UMR CNRS 6597 Ecole polytechnque de l unversté de Nantes - Nantes France {ahmad-montaser.awal, harold.mouchere, Chrstan.Vard-Gaudn }@unv-nantes.fr Abstract We propose n ths paper a new contextual modellng method for combnng syntactc and structural nformaton for the recognton of onlne handwrtten mathematcal expressons. Those models are used to fnd the most lkely combnaton of segmentaton/recognton hypotheses proposed by a 2D segmentor. Models are based on structural nformaton concernng the layouts of symbols. They are learned from a mathematcal expressons dataset to prevent the use of heurstc rules whch are fuzzy by nature. The system s tested wth a large base of synthetc expressons and also wth a set of real complex expressons. 1. Introducton For scentsts, nothng s better than modelng a gven problem usng mathematcal notaton. Almost all felds of scence ncludng human scences use mathematcs n less or more complex ways. To understand even more the mportance of mathematcal expressons (MEs), we have extracted all MEs from web pages of the French Wkpeda. Almost expressons were found n 7000 web pages. Thus, MEs are unversal communcaton tools among scentsts. Furthermore, the tendency n scentfc communtes to use dgtal proceedngs ncreased remarkably n the last few years. There are many tools to nput MEs nto dgtal documents. However, those tools requre specal sklls to be used effcently. Latex and MathML, for example, requre knowledge of predefned sets of key words. Other tools, such as Math Type, depend on a vsual envronment to add symbols usng the mouse and though needs lot of tme. Recent advances n the doman of dgtal pens and touch screens allow to wdespread the use of handwrtng nput tools. Of course, these tools present an nterestng alternatve to nput mathematcal expressons nto dgtal documents. Hence, t s essental to develop systems able to convert expressons from the natural handwrtten way to a dgtal format. However, handwrtten MEs recognton s more challengng than text recognton [1]. Unlke handwrtten text whch s a smple left to rght sequence of characters; a ME s a complex 2D layout of mathematcal symbols. The number of these symbol (~220 symbols) s by tself another challenge that requres powerful classfyng tools. Moreover, the two dmensonal layout causes many ambgutes n symbol roles, spatal relatons, more examples can be found n [2][3]. Many researches have been done recently n ths doman wth promsng results. Most of these researches consder expresson recognton as a sequence of ndependent subtasks. Ths decomposton smplfes the problem, but errors nherted from one step cannot be easly corrected. Our research focuses on the recognton of onlne handwrtten MEs. The advantage of our proposton s to perform a smultaneous segmentaton, recognton and nterpretaton of MEs under the restrcton of a language model. Specfcally, the classfer used to recognze symbols s based on a global learnng method allowng to learn symbols drectly from MEs performng at the same tme the segmentaton, and the nterpretaton.. The contrbuton of ths paper s to propose a new method to model contextual nformaton between symbols. Contextual models are learned drectly from a ME database. These models serve not only n recognzng expresson structure, but also n boostng the capacty of the symbols classfer by consderng the n best class canddates.

2 2. State of the art Generally, ME recognton takes place n three man steps [4]: segmentaton, recognton and nterpretaton. Consderng an onlne handwrtten sgnal, the prmtve unt, whch allows to segment t, s a stroke. A stroke beng a trace drawn between a pen down and a pen lft. However, n most cases a sngle symbol s composed of several strokes. Conversely, we wll assume that one pen lft exsts between consecutve symbols. A good segmentaton s the key pont of a good recognton and nterpretaton. Hence the segmentaton step consst n groupng strokes belongng to the same symbol. Early systems consdered symbol segmentaton as an ndependent step [5][6]. More recently, symbol segmentaton and recognton are consdered as one step. Thus, the segmentaton s lead by symbol recognton [7][8][9], where recognton scores serve to choose groupngs that are more lkely to represent symbols. In order to decrease the complexty of ths smultaneous optmzaton «best frst search» [7], or CYK [9] algorthms are used. The geometrcal structure of a ME s usually more complex than that of a normal text. Whle a text s systematcally wrtten from left to rght, math symbols can be wrtten n almost all drectons, see Fgure 1. Therefore, MEs nterpretaton conssts of analyzng geometrcal structures of the expresson and applyng syntactc analyss. The objectve of ths nterpretaton s to fnd the dervaton tree of the expresson. Fgure 1 Wrtng drectons n a normal text and a mathematcal expresson Spatal relatons between symbols are crucal for good nterpretaton. Even f all symbols are correctly segmented and recognzed, a 2D analyss s requred to correctly nterpret the expresson. A method based on a Defnte Clause Grammar s proposed n [4]. The effcency of ths DCG s ncreased by usng left factored rules. More recently, Garan [10] proposed a context free grammar. A structure s bult by dvdng the expresson recursvely nto horzontal and vertcal bands. When reachng the level of atomc elements, grammar producton rules are appled accordng to the type of spatal relatons. In [11], the authors present an approach called Fuzzy Shft-Reduce Parsng. Ths method uses a descendng analyss assurng effcent verfcaton. A probablstc grammar has been proposed n [9]. Each producton rule of the grammar s assocated to a logcal relaton n addton to the probablty of ths rule. Thus, the recognton of an expresson s transformed nto a search of rules that maxmze the probablty of obtanng the result expresson. 3. Proposed recognton system The proposed archtecture ams at handlng the recognton of MEs as a smultaneous optmzaton of segmentaton, symbol recognton and nterpretaton problems. The tranng of the system and also the recognton of expressons are done usng the global archtecture we proposed n [17], see Fgure 2. Fgure 2 Expresson recognzer archtecture The System s traned n two dstnct stages n a global learnng schema. Furthermore, the system s traned drectly from mathematcal expressons nstead of tranng t wth solated symbols or usng heurstc values for structural analyss. It takes nto account the ground truth of the gven ME (deal segmentaton and correspondng labels of symbols and spatal relatons among those symbols) and the best current nterpretaton resultng from a specfc segmentaton, and correspondng recognzed symbols. The archtecture s detaled n the followng sectons Symbol hypothess generator The number of all possble segmentatons s defned by the bell number. On a smple example shown n Fgure 3, consderng 7 strokes B 7 = 877 dfferent segmentaton. Bell numbers are calculated usng the followng recursve formula : n n n Bn+ 1 = Bk ; s a bnomal coeffcent. k = 0 k k Fgure 3 Example of groupng hypotheses

3 Hypothess generator lsts a number of possble combnatons of strokes. Each group of strokes s called a symbol hypothess (sh). From a computng perspectve, t can be consdered as a Dynamc Programmng (DP) algorthm, whch s well adapted to ths knd of decson makng problems [12]. However, the key pont s that ths s not a standard 1D-DP but we adopt an extenson to a 2D-DP. To avod the combnatory exploson of the search space, some constrants are added lmtng the maxmum number of strokes n one symbol and of hypothess Symbol recognzer The symbol recognzer provdes, n addton to the best n canddates of each hypothess, a recognton score of each canddate that wll be used to calculate the recognton cost. We used a tme delayed neural network (TDNN) [13] for ts nterestng propertes of beng nsenstve to poston shfts. However, an addtonal layer s added n order to convert the classfer outputs nto probabltes. For a gven hypothess p c j sh ) denotes the probablty of the ( hypothess sh beng the class c j ; where p( c sh ) j = 1 (1) j Some methods consders N best canddate of the symbol classfer [9]. Smlarly, others delay the decson of labelng ambguous symbols to be resolved by the global context [7]. In order to determne the number of canddates retaned for each sh. We retan k canddates wth a max number N (k <=N). k s chosen under the k 1 condton: p ( c j sh ) Thresh to keep only j= 0 canddates wth hgh confdence and avod to keep all N canddates systematcally. In the frst tranng stage, the system s traned wth the tranng expressons wthout consderng contextual nformaton. The goal of ths stage s to tran the symbol classfer. Furthermore, ths stage gves the classfer the ablty to dentfy wrong hypotheses durng the segmentaton usng an addtonal class that we call the junk class [17] Language model A mathematcal expresson s produced by a 2D language. Context free grammars are effcently used for parsng 1D languages (such as programmng languages). However, parsng 2D languages s ntractable and requres specal algorthms and constrants to reduce complexty and ambguty [3]. Graph grammar was ntroduced n [15], transferrng parsng an ME to a graph rewrtng problem. Snce two-dmensonal grammars are faced to performance ssues, we have descrbed one as a set of one-dmensonal rules on both vertcal and horzontal axes. Vertcal rules (VR) and horzontal rules (HR) are appled successvely untl elementary symbols are reached to perform a bottom up parsng algorthm. Table 1 Example of a smple grammar Rule sym x, y, 1, 2, op +, -, x, formule subexp op sym subexp subexp op sym subexp sym sym subexp sym Relaton type (operator) [HR] (operator) [HR] (superscrpt) [VR] As shown n Table 1, each producton rule of the grammar s assocated to a spatal relaton that descrbes the layout between the elements of the rule. On the other hand, each relaton s assocated to a cost functon (see sectons 3.4 and 3.5) that penalzes more or less the rule accordng to structural nformaton of ts elements. Ideal postons and szes are dffcult to be predefned because of the fuzzy nature of relatons. Therefore, we try to model these relatons nstead of predefnng them usng some heurstc rules Structural analyzer Structural analyss requres extractng spatal nformaton of each symbol hypothess to perform contextual evaluaton. In order to avod ambgutes, spatal nformaton are extracted dfferently dependng on the type of the recognzed symbol as shown n Fgure 4. Extracted nformaton s the base lne y and hypothess heght h. Furthermore, the spatal nformaton of a produced sub-expresson s calculated from ts elements consderng the spatal relaton type. Fgure 4 Structural nformaton of dfferent symbol types When a relaton R s bult, the value of y and h of the correspondng sub-expresson s computed from the N chldren components: y h y = f ( y,..., y ) ; h = f ( h,...,h ) SE R SE1 SE N SE R SE1 SEN

4 The number of chldren N dffers accordng to the type of the relaton R; wth N = {2, 3, 4}. For example, relatons as superscrpt and subscrpt" have two chldren. The operator relaton has three chldren, whle the ntegral relaton has four. A sub-expresson (SE) can denote three possble nterpretatons: a symbol hypothess, a sub-expresson, or the fnal result expresson. Then, we defne the normalzed poston and sze dfferences (dh, dy) of a sub-expresson (SE ) as follows: dh = ( hse hse ) / h ; SE dy = ( yse yse ) / h SE The second stage of the tranng ams at modelng spatal relatons wthout consderng the classfer. All occurrences of (dy, dh) of each sub-expressons for each knd of relatons are computed usng the ground truth of the tranng dataset of MEs. Then, two Gaussan models are constructed for each element of a relaton. The frst model reflects the pertnence of the sze of ths element for the sub-expresson produced by the producton rule assocated to ths relaton, usng dh. The second one reflects the qualty of the algnment of the consdered elements accordng to the produced sub-expresson, usng dy. Thus, for each relaton R, 2N Gaussan models are constructed usng the n occurrences of dh, dy of each sub-expresson. The mean µ and varance σ of each model are calculated as follows ( = 1..n): dx µ = ; 1 2 σ = ( dx µ ) n n 1 For example, the relaton superscrpt, whch nvolves two elements, s assocated to four models. Two models for sze dfferences (dh) of each element, and another two for the algnment dfferences (dy). Fgure 5 llustrates the structural nformaton of the superscrpt relaton and shows an example of Gaussan models of the poston dfferences (dh). Though, for a gven stuaton we defne the probablty of a sub-expresson beng correctly postoned and szed regardng the produced SE by a Gaussan functon: g ( x ) 2 µ ) 2 2σ SE, dx ( x = e (2) Hence, the probablty of a sub-expresson beng produced by the relaton R s: p( SE R ) = g ( dh ).g ( dy ) (3) SE,dh SE, dy Back to the example of N sub-expressons producng a new sub-expresson SE by the relaton R. The probablty that SE s produced by the N subexpressons s: p( SE R ) = N = 1 p( SE R ) (4) But, from the defnton of Bayes rule, the probablty of a relaton that produces a sub-expresson knowng ts Gaussan model s: p( SE R ).p( R ) p ( R SE ) = (5) p( SE ) but the term p(se) can be gnored, because t s a constant for all relatons, and by usng the equaton (4) we obtan the probablty of a relaton R: p( R SE ) N = 1 p( SE R ).p( R ) (6) where p(r) s the pror probablty of the relaton R calculated durng the learnng stage Decson maker dh (b) Fgure 5 (a) Illustraton of the relaton superscrpt, (b) Gaussan models of poston dfferences (dh) (a) As shown n the example n Fgure 6, a canddate expresson produced by the grammar s represented by a relatonal tree. Each node represents a hypothess of stroke groupng. The root of ths tree contans n addton to the orgnal nk sgnal: a proposed soluton wth ts global recognton cost and the nature of the lnk wth ts chldren. In a smlar way, each non termnal node represents a sub-expresson, whch contans the same nformaton. Fnally, termnals are symbol hypotheses proposed by the hypothess generator wth ther recognton costs computed by the symbol recognzer.

5 Fgure 7 Examples of real and synthetc expressons Fgure 6 Relatonal trees of two canddates of a mathematcal expresson Snce recognton and structural scores are n form of probablty (p), we use a logarthmc functon to transform those scores nto costs: cost = log( p) The decson maker organzes all resultng relatonal trees, and selects the one that mnmzes a global cost functon. The cost of non-termnal nodes n the relatonal tree s defned by the recursve formula: Costreco ( sh );f SE s a termnal (7) Cost( SE ) = α.cost struct( RSE SE ) + Cost( SE );Otherwse where: Cost ( R SE ) = log( p( R SE ) s the struct SE probablty that the sub-expresson SE s produced by the relaton R. The cost of termnals s the recognton cost of the hypothess sh : Cost ( sh ) = log( p( c sh )) reco j Fnally, the global cost of a canddate expresson wth n symbols hypotheses lnked by r relatons s: Cost (exp) = Cost ( SE root ) Where α s a weghtng factor between recognton and structural costs. The choce of alpha value n the equaton (7) s very mportant to balance the dfferences between structural and recognton costs, t has been set expermentally to α = Experments The expressons corpus s extracted from the base Aster [14]. A set of 36 expressons s chosen coverng a majorty of mathematcal domans. Each expresson contans n average 11 symbols, where the total number of dstnct symbols s 34 classes. Each expresson s artfcally generated from a base of solated symbols collected from 280 wrters [16], therefore ths dataset contans 10,080 mathematcal expressons. In addton, each of those 36 expressons s wrtten twce resultng a total of 72 expressons wrtten by 10 wrters, examples shown n Fgure 7. Further detals about databases can be found n [17]. SE It s napproprate to evaluate the system only at a global level, especally when dealng wth long and complex expressons. In consequence, we have chosen three measures smlar to those used recently n [7] and [9] n order to be more precse and to allow comparng our results. Denotng total number of symbols: N total, we used the followng measures: SegRate=(correctly segmented symbols)/n total RecoRate=(correctly recognzed symbols)/n total ExpRate= (correctly recognzed expressons)/exp total 4.1. Results The number of canddates consdered as a result of symbol hypotheses classfyng s an mportant factor. The curve Fgure 8 shows expresson recognton rate evoluton on the synthetc DB when varyng the average of consdered canddates per symbol hypothess. Fgure 8 Evoluton of test expressons recognton rate Vs average canddates number per symbol Ths curve s obtaned by fxng the max number of canddates to 5, and varyng the threshold (see secton 3.2). Consderng only one recognton result for a hypothess lmts the classfer capacty. On the other hand, retanng a large number of canddates gves more freedom to the structure analyzer. Ths freedom allows to ncrease segmentaton rate wth a rsk to overlap correct expresson nterpretaton. Thus, results n Table 2 are obtaned by consderng at maxmum the frst 5 canddates wth the sum of ther

6 recognton probabltes beng below We compare obtaned results usng ths confguraton and the contextual modelng proposed n ths paper wth those of the system proposed n [17] based on geometrc modelng of spatal relatons. Table 2 Test Expresson recognton rates Synthetc DB SegRate RecoRate ExpRate Geometrc modelng [17] Gaussan modelng Real DB SegRate RecoRate ExpRate Geometrc modelng [17] Gaussan modelng Table 2 shows the nterest of usng Gaussan models for the contextual modelng. Ths latter ncreases expressons recognton rate on the synthetc DB from 64.2% to 64.9%. Ths mprovement s not generalzed on real expressons. ExpRate decreases from (28.6% to 27.9%) compared to 29.2% n the system proposed by Ha et al. [6], notcng that they use a dfferent database and t s not possble to do a drect comparson of results. Ths general fall n performance compared to the geometrc model comes prncpally from the strong dependence of learned models wth the tran database, whch s synthetc. Thus, we assume that t s ndspensable to use large database of real expressons n order to ncrease the generalzaton capacty of usng Gaussan models. 5. Concluson and perspectve We presented n ths paper a new method for contextual modelng n a system for onlne handwrtten mathematcal expressons recognton. Contextual models are learned drectly from ME database. Learned models partcpate n penalzng wrong locatons of hypotheses. Another mprovement comes from consderng the best n canddates of the symbols classfer. We beleve that the performance can be mproved by ntegratng models of smlar relatons and gnore some others that cause ambgutes at relaton level. The system has been tested wth a large set of synthetc expressons and also on a set of real complex expressons wth promsng results. References [1] D. Blosten, A. Grbavec. Recognton of mathematcal notaton. Handbook on Optcal Character Recognton and Document Image Analyss, Queen's unversty, World Scentfc Publshng Company: , [2] W. Martn. Computer nput/output of mathematcal expressons. 2 nd Symp. on Symbolc and Algebrac Manpulatons, New York: [3] E.G. Mller, P. A. Vola. Ambguty and Constrant n Mathematcal Expresson Recognton. 15 th Natonal Conference on Artfcal Intellgence: [4] K-F. Chan, D-Y. Yeung. An effcent syntactc approach to structural analyss of on-lne handwrtten mathematcal expressons. Pattern Recognton 33: [5] C. Faure, Z.X. Wang. Automatc percepton of the structure of handwrtten mathematcal expressons. Computer Processng of Handwrtng, World scentfc, Sngapore: , [6] J. Ha, R.M. Haralck, I.T. Phllps. Understandng mathematcal expressons from document mages. 3 rd ICDAR: , [7] T-H Rhee, J-H Km. Effcent search strategy n structural analyss for handwrtten mathematcal expresson recognton. Pattern Recognton 42(12): , [8] S. Smthes, K. Novns, J. Arvo. A Handwrtng-Based Equaton Edtor. the Graphcs Interface: 84-91, [9] R. Yamamoto, S. Sako, T. Nshmoto, S. Sagayama. On- Lne Recognton of Handwrtten Mathematcal Expressons Based on Stroke-Based Stochastc Context-Free Grammar. 10 th IWFHR, La Baule, France: , [10] U. Garan, B. Chaudhur. Recognton of Onlne Handwrtten Mathematcal Expressons. IEEE Transactons on Systems, Man and Cybernetcs 34: , [11] J.A. Ftzgerald, F. G., T. K. Structural Analyss of Handwrtten Mathematcal Expressons Through Fuzzy parsng. 4 th IASTED: , [12] M. Held, R.M.K., The constructon of dscrete dynamc programmng algorthms, IBM Syst. J. 4(2): , [13] M. Schenkel, I. Guyon, D. Henderson. Onlne cursve scrpt recognton usng tme delay neural networks and hdden Markov models. Mach. Vs. Appl., Specal Issue on Cursve Scrpt Recognton 8: , [14] T.V. Raman. Audo system for techncal readngs. Cornell Unversty, [15] A Kosmala, G Rgoll, S Lavrotte, L Potter. On-Lne Handwrtten Formula Recognton Usng Hdden Markov Models and Context Dependent Graph Grammars. 5 th ICDAR: , [16] A.M. Awal, R. Cousseau, C. Vard-Gaudn. Convertsseur d'équatons LATEX2Ink. 10 th CIFED, Rouen, France: , [17] A.M. Awal, H. Mouchère, C. Vard-Gaudn. Towards handwrtten mathematcal expresson recognton. 10 th ICDAR, Barcelona, Span: , 2009.