Skew Estimation in Document Images Based on an Energy Minimization Framework

Transcription

1 Skew Estmaton n Document Images Based on an Energy Mnmzaton Framework Youbao Tang 1, Xangqan u 1, e Bu 2, and Hongyang ang 3 1 School of Comuter Scence and Technology, Harbn Insttute of Technology, Harbn, Chna 2 Deartment of New Meda, Harbn Insttute of Technology, Harbn, Chna 3 Honors School, Harbn Insttute of Technology, Harbn, Chna Abstract - Skew estmaton s mortant for document analyss and alcaton. Most exstng methods are roosed to deal wth the document mages consstng of words. In most cases, a comlex document may nclude tables, rregular ctures and other non-text comonents. To address the challengng roblem, ths aer rooses a novel skew estmaton aroach based on an energy mnmzaton framework for skewed scannng document mages. In the roosed aroach, the foreground xel state nformaton s comuted at frst. Then a new cost functon that consders both background and foreground nformaton for skew estmaton s constructed by usng state nformaton. A coarse skew s yelded by emloyng lne fttng technque. Then the coarse skew s refned by teraton so that the cost functon gets mnmum. The ICDAR2013 DISEC dataset s used to evaluate the roosed aroach and the exermental results show ts effectveness. Keywords: Skew Estmaton, Energy Mnmzaton, Cost Functon, Lne Fttng, Foreground Pxel State Estmaton 1 Introducton th the ncreasng develoment of dgtal technology, the emergence of electronc documents s more and more oular n eole s routne lfe due to ther convenence and ersstence. Such as usng camera or scanner to record management and store hstorcal documents, and so on. hle acqurng electronc document mages, a lttle skew s unavodable. However, most of the document based systems (such as OCR, age layout analyss and character recognton, and so on) are senstve to skew n document mages. Thus, skew estmaton becomes an mortant ssue n the feld of document mage analyss and understandng [1]. To elmnate the skew, an algnment rerocessng s necessary durng dgtzaton rocedure. There are two dfferent knds of document mages, handwrtten document mages (HDIs) and machne-rnted document mages (MPDIs). For handwrtten document mages, t s a hard work to detect ther skew angles whle the texts are wrtten n an unconstraned condton. Comared wth handwrtten document mages, machne-rnted document mages are regularly algned. Accordng to the content of MPDIs, they are categorzed nto smle MPDIs whch only consst of ure texts and comlex MPDIs whch nclude tables, rregular ctures and other non-text comonents excet texts. In recent years, extensve researches have been carred out on skew estmaton for MPDIs. However, t s a stll challengng roblem to estmate the skew of comlex MPDIs. For comlex MPDIs, although the contents of them are uncertan, the global nformaton of them s obvous, such as: () The outermost foreground xels can be ftted nto lnes n four dfferent drectons of MPDI and there s at least one lne whose skew angle s close to the orgnal skew angle of the MPDI. () The length of ga regon changes wth rotatng the MPDI whle the sze of rotated mage s fxed and t can get maxmum when the rotaton angle s close to the orgnal skew angle of MPDI, here the ga regon s defned as the regon has no foreground xel n a certan drecton. () The varance between the numbers of foreground xels n a certan drecton can get maxmum as (). Ths aer rooses a novel aroach that consders these nformaton. To be recse, we convert the skew estmaton roblem nto an energy mnmzaton roblem. A new cost functon whch consders global background and foreground nformaton s constructed frstly. Then the mnmzaton of the cost functon yelds the skew angle of MPDI. The roosed aroach s summarzed as a flow dagram showed n Fgure 1. The rest of ths aer s organzed as follows: Secton 2 gves a bref ntroducton to some related work. Secton 3 resents a descrton of the roosed aroach n detal. Secton 4 reorts our exermental results. Fnally, the conclusons are resented n Secton 5. Outermost state nformaton Lne fttng Inut MPDI Foreground xel state comutaton Coarse skew All state nformaton Energy mnmzaton Fnal skew Stage 1 Stage 2 Fgure 1: Flow dagram of the roosed aroach.

2 2 Related work Rezae et al [2] surveys most exstng methods of skew estmaton. And they consder these methods can be roughly dvded nto eght categores: rojecton rofle analyss [3-5], Hough transform [6, 7], nearest neghbor clusterng [8], crosscorrelaton [9], ece-wse coverng by arallelogram [10, 11], ece-wse antng algorthm [12], transton counts [13], and morhology [14, 15]. In ths aer, we brefly summarze several methods ublshed n recently. Sngh et al [16] emloy the Hough transform for skew estmaton wth a rerocessng stage whch reduces the number of mage xels. Comared to tradtonal methods based on Hough transform, they address the seed and memory requrement roblem to some extent. Meng et al [17] exlot varous tyes of vsual cues of mage extracted by Radon transform for skew estmaton. A floatng cascade s used to reject the outlers of vsual cues by teratng. A baggng estmator s fnally emloyed to combne the rest of vsual cues on local mage blocks. Alae et al [12] horzontally and vertcally adot the ece-wse antng algorthm on document mage to obtan two anted mages rresectve to the flow of wrtng and content. Lnear regresson and a roosed majorty votng technque are utlzed to fnd the best fttng lne whose sloe s the skew angle of the document mage. Fan et al [18] roose a Rectangular Actve Contour Model (RAC Model) for content regon detecton and skew angle calculaton by mosng a rectangular shae constrant on the zero-level set n Chan-Vese Model (C-V Model) accordng to the rectangular feature of content regons n document mages. Guan et al [19] develo a blnear flterng model to extract the foreground regons and detect the edges n the document mages wthout consderng document layouts or contents. Then a domnant angle has been estmated as the skew angle of the document mage based on the detected edges. Most exstng methods only adot the foreground nformaton or background nformaton. However, the accuracy wll be decreased by usng one of them for skew estmaton. Ths aer consders both global foreground and background nformaton to mrove the accuracy and rooses a novel skew estmaton aroach based on an energy mnmzaton framework. 3 Methodology The roosed aroach conssts of two stages: foreground xel state comutaton and skew estmaton based on energy mnmzaton, as shown n Fgure 1. In ths secton, these rocesses wll be stated n detal. 3.1 Foreground xel state comutaton The roosed aroach begns wth the bnarzaton of nut document mages. Snce the background and foreground of MPDIs are easly slt, a smle thresholdng method s used for bnarzaton. After bnarzaton, we comute the state nformaton for each foreground xel wth the method descrbed as follows. Gven a bnary document mage I, a boundng box s defned as the boundary of I (seeng the yellow rectangle n Fgure 2 (a)). Let P denote the set of foreground xels and (, H) denote the sze of I. Then for P, assgn a state s = (x, y, w, h ) to as shown n Fgure 2 (a), where (x, y ) s the locaton of n I, w = x, and h = H y. These states S = {(x, y, w, h )} wll be used n P lne fttng and energy mnmzaton to estmate the fnal skew angle of document mages. 3.2 Skew estmaton usng energy mnmzaton Ths aer formulate the skew estmaton roblem as an energy mnmzaton roblem. The framework of energy mnmzaton roblem conssts of two bref stes: cost functon constructon and the mnmzaton of the cost functon Cost functon constructon 1) Proosed cost functon To construct an arorate cost functon s mortant n energy mnmzaton roblem, because the cost functon affects the otmal soluton drectly. Here, ths aer resents a new cost functon consstng of two terms. And ts mnmzaton consders global background and foreground nformaton of document mages. Secton 3.1 ntroduces the rocess of comutng foreground xel state nformaton. e can observe the state nformaton wll be dfferent whle the boundng box s fxed and the document mage s rotated around the center. Based on ths observaton, the skew estmaton roblem s formulated as Sˆ arg mn E S (1) where the cost functon ncludes two terms,.e., S E S E S (1 ) E S. (2) B where ω s a constant. The frst term E B (S) consders the global background nformaton whch s the length of ga regon (LGR) n horzontal and vertcal drecton of document mages. The second term E F (S) reflects the global foreground nformaton whch s the varance of the foreground xel number (VFPN) n every row and column of document mages. 2) Desgn of E B (S) and E F (S) The states S havng large LGR and small VFPN s desrable, so the frst term E B (S) s gven by and we set φ( ) and φ( ) as below ( S) ( S) F E ( ), B S e (3)

3 rato= angle= rato= angle= rato= angle= rato= angle= (a) (b) (c) Fgure 2: (a) gves an examle of MPDI and the llustraton of foreground xel state nformaton comutaton. The yellow rectangle s the boundng box of the MPDI. And the green dot lnes are the fttng lnes from four sde of the MPDI. (b) lots the dstances between the outermost states and fours sdes of the boundng box from (a). The black dot lnes show that these below onts can be ftted as lnes. (c) llustrates the lne fttng rocedure and gves the fnal fttng results from (b). 1 1 ( S) Sgn Y, ( ) H H S Sgn X j 1 j1 Y s s S y s y X s s S x s x j j where Sgn( ) s a sgn functon formulated as 1, A Sgn( A) 0, A. The second term E F (S) s gven by F ( ) ( ), and we set δ( ) and λ( ) as below (4) (5) E S S S (6) 1 1 f ( Y) S f ( Yk ) Y M Y Y f ( Y ) 1 f ( Y) 1 1 k 1 f ( Y) k 1 f ( X ) S f ( X k ) X M X X f ( X ) 1 f ( X) k 1 f ( X ) k 1 f ( Y ) k f ( X ), H where Y = =1 (Y Y ), X = (X j X j ), M Y k j=1 = max{f(y ) Y Y}, M X = max{f(x ) X X}, and f( ) s a functon to comute the number of elements n a set. There s one arameter ω n the roosed cost functon. And t s determned by exerments conducted on tranng dataset Energy mnmzaton The mnmzaton of the cost functon s a hard and tmeconsumng work whle the evaluaton of E(S) requres a number of oeraton (such as state rotaton, varance 2 2 (7) comutaton and so on). Hence, we address ths roblem by develong an otmzaton technque descrbed as follows: we use the outermost states to get a coarse soluton at frst, then teratvely refne ths soluton wth all state to estmate the fnal skew. 1) Lne fttng for coarse skew estmaton For the coarse soluton, we exlot the lne fttng technque stated as below. As shown n Fgure 2 (a), a boundng box has four sdes: to, bottom, left and rght. For each sde, such as to, we get a subset TS S by TS y s y mn y y s x s x. 1 e lot a fgure by usng x TS as x-coordnate and y TS as y-coordnate, as shown n the to-left of Fgure 2 (b). In the same way, we can get the rest of three subsets BS, LS and RS, corresondng to bottom, left and rght of the boundng box, resectvely. From Fgure 2 (b), we observe that the bottom onts of all fgures can be ftted as lnes, and the skew angles of some lnes are close to the orgnal skew angle of I. So n ths aer, we use lne fttng technque to obtan the coarse skew angles of document mages. Here, we take TS as an examle to descrbe the rocess of lne fttng n detal. To seed u the rocess of lne fttng and get more accurate coarse skew estmaton, we take samle from TS before lne fttng. The TS s dvded nto N non-overlang arts STS as below STS TS N 1 STS ( 1) STS s x s 1 x. N N s STS j In ths work, we set N = 32. Then a subset FTS s constructed (8) (9)

4 rato= angle= orgnal skew: coarse skew: fnal skew: (a) (b) (c) Fgure 3: (a) shows the best-fttng lne from Fgure 2 (c) accordng to our crteron. So the coarse skew angle of Fgure 2 (a) s , whle the orgnal skew angle s (b) gves the teraton rocess of coarse skew refnement. x-coordnate s the teraton tmes and y-coordnate s E(S) corresondng rato=0.0530to teraton. The fnal skew angle s whle E(S) gets mnmum. (c) shows the skew correcton result of Fgure 2 (a). by angle= N 1 FTS s s STS y s y mn y y s s STS. Fgure 2 (c) gves the samlng results of Fgure 2 (b). (10) After dong the samlng oeraton, although most of outlers of states whch s far away from desrable fttng lne are removed, we need to further elmnate the outlers to obtan the vald states VTS (Seeng the states whch below the red dot lne n Fgure 2 (c)) by H VTS s s FTS y s y 3 (11) Then we use the VTS to do lne fttng wth the behavor of exhaustve search. The urose of exhaustve search s to fnd two states so that there wll be more other states (Seeng the nk cross onts n Fgure 2 (c)) whose dstances to the lne that across these two states are less than a threshold D. After obtanng four fttng lnes (Seeng the green dot lnes n Fgure 2 (a) ftted by the nk cross onts n Fgure 2 (c)) from TS, BS, LS and RS, the next task s to fnd the bestfttng lne. Let {l t, l b, l l, l r } denote the fttng lnes, and {LS t, LS b, LS l, LS r } denote the lne states (Seeng the nk cross onts n Fgure 2 (c)) whch are close to the corresondng fttng lnes. The lnes whose lne states number s less than a threshold M wll not be consdered n the followng stes. For each lne l, we calculate the sum of dstances SD between all lne states of LS and l. The rato R s comuted by SD R (12). f LS 2 Then {R t, R b, R l, R r } s adoted to get the best-fttng lne whch has the smallest R (Seeng the to-rght of Fgure 2 (c) and Fgure 3 (a)). Fnally, the sloe of the best-fttng lne s calculated as the coarse skew of the orgnal document mage. Durng lne fttng, there are two arameters (D and M) whch decde the accuracy of coarse skew estmaton. They are determned by exerments conducted on tranng dataset. hle all sdes of a document mage are rregular, the error devaton between the coarse skew and the orgnal skew of the document mage wll be large. To handle ths roblem, we roose an algorthm whch refnes the coarse soluton through an energy mnmzaton rocedure. 2) Mnmzaton of the cost functon After lne fttng, a coarse skew angle s yelded. The next work s to estmate a more accurate skew angle by emloyng all state nformaton and the coarse skew angle. From a coarse skew angle θ of I, we teratvely refne the skew estmaton based on the state rotaton and cost E(S) comutaton. In order to mrove the comutatonal effcency, we drectly rotate all states around the document mage center, rather than rotate the mage frstly then comute all foreground xel state nformaton. The rotaton rocess s conducted as followng oeraton S rotate S, (13) where rotate( ) s a functon comutng the rotaton result s of each state s S by H x x cos y sn H H y x sn y cos w x, h H y. (14) Durng teraton, we fx the range of rotaton about [θ range, θ + range] and set the angle ste of rotaton ste = 0.05 and range = 0.5 through exerments conducted on tranng dataset. So the teraton tmes s T = 21. After fnshng all teraton, the fnal skew angle β s the rotaton angle whch gets E(S ) mnmum (Seeng Fgure 3 (b)). Then the skew document mage s corrected accordng to the fnal skew angle (Seeng Fgure 3 (c)). Algorthm 1 descrbes the whole refnement rocess.

5 Algorthm 1: Inut: coarse skew angle θ, all states S,teratons T Outut: fnal skew angle β ntalze t 0, S 0 S, θ 0 θ, E mn E(S 0 ), β θ 0 for t t + 1 θ t θ t (t 1) 0.5 S t rotate(s t 1, θ t ) f E(S t ) < E mn E mn E(S t ) β θ t end untl t = T return β end 4 Exerments 4.1 Performance evaluaton Dataset The ICDAR2013 DISEC dataset [20] s used to evaluate the erformance of the roosed aroach. The dataset conssts of 200 document mages, reresentatve of most realstc cases. The document mages contan fgures, tables, dagrams, block dagrams, archtectural lans, electrcal crcuts, whle they are obtaned from newsaers, scentfc journals, museum gudes, and so on. And the mage documents are wrtten n Englsh, Chnese, Greek, and so on, whle there are reresentatve cases of varous szes of mage documents, any knd of mxed content, vertcal and horzontal wrtng, multszed fonts and multle dfferent number of columns n the same document. e slt ths dataset nto two arts. One contanng 50 document mages randomly selected s used for tranng, called tranng dataset. The other one ncludng the rest 150 document mages s used for test, called test dataset Evaluaton crteron The erformance evaluaton wll be based on a well establshed technque for document skew estmaton descrbed as [20]. More secfcally, the skew angle average error devaton (AED), the number of correct estmatons (error devaton of less than 0.1º) (NCE) and the number of good estmatons (error devaton of less than 0.2º) (NGE) wll be taken nto consderaton. 4.2 Parameter otmzaton e frstly determne these two arameters mentoned n lne fttng: D and M wth free searchng method. Ths s done by samlng for the best erformance of skew estmaton on tranng dataset over 3 D 7 at ntervals of 0.5, 4 M 8 at ntervals of 1. The exermental results have best erformance when D=5, M=5. And the largest error devaton s about 0.35º. So we set range = 0.5 n the refnement rocess. Then the arameters ste and ω of energy mnmzaton wll be determned resectvely due to ther ndeendence. e AED w Fgure 4: The erformance of skew estmaton on tranng dataset for testng ω when ste=0.05. test ω over 0 ω 1 at ntervals of 0.05 on tranng dataset whle we fx ste = The best exermental results can be obtaned when ω = Then test ω over 0.9 ω 1 at ntervals of The AED gets mnmum when ω = 0.98, as shown n Fgure 4. Then we fx ω = 0.98 to test ste wth ste = {0.01,0.03,0.05,0.1}. Table 1 lsts the exermental results. From Table 1, we can see when ste < 0.05, the accuracy of skew estmaton s good and stable, but the comutatonal comlexty s hgh. However, when ste = 0.1, the results are contrary. Consderng the trade-off, we fnally set ste = Table 1: The erformance of skew estmaton on tranng dataset (50 document mages) for testng ste when ω s set as ω = ste AED(º) NCE NGE Tme (s) All exerments are conducted n Vsual Studo 2012 envronment on the PC wth M (2.4 GHz) CPU and 2GB RAM. The comutatonal tme gven n Table 1 and 2 s tested on a document mage wth sze Exermental results Usng the arameters gven n Secton 4.2, we test the roosed aroach on test dataset wth dfferent schemes. As mentoned above, there are two stes for skew estmaton: coarse skew estmaton (CS) and energy mnmzaton (EM). Actually, these two stes are ndeendent. Hence, they can be used to estmate skew alone, or CS can be relaced by other skew estmaton methods. In ths aer, we do CS before EM wth consderng the trade-off between seed and accuracy. To further mrove seed, we can do 2:1 downsamlng (DS) of the orgnal document mages. Table 2 resents the results of skew estmaton wth dfferent schemes combnaton. As shown n Table 2, the erformance of skew estmaton s the best by adotng the combnaton of CS and EM schemes, whle the comutatonal tme s largest. e observe that the

6 Table 2: The erformance of skew estmaton on test dataset (150 document mages) wth dfferent scheme combnaton. Schemes AED(º) NCE NGE Tme (s) CS CS+EM CS+DS CS+EM+DS erformance obvously decrease although much comutatonal tme s saved, when downsamlng the document mage before skew estmaton. Because the document mage wll lose some mortant foreground xel state nformaton for skew estmaton after downsamlng. So dfferent schemes can be adoted accordng to dfferent alcatons or requrements. Fgure 5 lsts some examles of skew estmaton wth our roosed aroach. 5 Conclusons In ths aer, we roose a novel and effcent skew estmaton aroach for document mages. e formulate the roblem as an energy mnmzaton roblem. A new cost functon whch consders global background and foreground nformaton s constructed and the lne fttng technque s exloted to get a coarse skew angle. Then the mnmzaton of the constructed cost functon yelds the fnal skew angle. The exermental results on ICDAR2013 DISEC dataset have shown that our aroach accurately estmates the skew angle of document mages wth average error devaton º. 6 Acknowledgements Ths work was suorted by the Natural Scence Foundaton of Chna (Grant No ), the Program for New Century Excellent Talents n Unversty (Grant No. NCET ), the Fok Yng Tong Educaton Foundaton (Grant No ), and the Fundamental Research Funds for the Central Unverstes (Grant No. HIT. NSRIF ). 7 References [1] G. Nagy, "Twenty years of document mage analyss n PAMI," TPAMI, vol. 22, , [2] S. B. Rezae, A. Sarrafzadeh, and J. Shanbehzadeh, "Skew Detecton of Scanned Document Images," IMCECS, [3] S. L, Q. Shen, and J. Sun, "Skew detecton usng wavelet decomoston and rojecton rofle analyss," PRL, vol. 28, , [4] J. Sadr and M. Cheret, "A new aroach for skew correcton of documents based on artcle swarm otmzaton," ICDAR, 2009, [5] A. Paandreou and B. Gatos, "A Novel Skew Detecton Technque Based on Vertcal Projectons," ICDAR, 2011, [6] B. Eshten, "Determnng Document Skew Usng Interlne Saces," ICDAR, 2011, [7] D. Kumar, "Modfed Aroach of Hough Transform for Skew Detecton and Correcton n Documented Images," IJRCS, vol. 2, , [8] I. Konya, S. Eckeler, and C. Sebert, "Fast seamless skew and orentaton detecton n document mages," ICPR, 2010, [9] M. Chen and X. Dng, "A robust skew detecton algorthm for grayscale document mage," ICDAR, 1999, [10] C.-H. Chou, S.-Y. Chu, and F. Chang, "Estmaton of skew angles for scanned documents based on ecewse coverng by arallelograms," PR, vol. 40, , [11] A. A. Mascaro, G. D. Cavalcant, and C. A. Mello, "Fast and robust skew estmaton of scanned documents through background area nformaton," PRL, vol. 31, , [12] A. Alae, U. Pal, P. Nagabhushan, and F. Kmura, "A Pantng Based Technque for Skew Estmaton of Scanned Documents," ICDAR, 2011, [13] Y.-K. Chen and J.-F. ang, "Skew detecton and reconstructon based on maxmzaton of varance of transton-counts," PR, vol. 33, , [14] A. Das and B. Chanda, "A fast algorthm for skew detecton of document mages usng morhology," IJDAR, vol. 4, , [15] B. Dhandra, V. Malemath, H. Mallkarjun, and R. Hegad, "Skew detecton n Bnary mage documents based on Image Dlaton and Regon labelng Aroach," ICPR, 2006, [16] C. Sngh, N. Bhata, and A. Kaur, "Hough transform based fast skew detecton and accurate skew correcton methods," PR, vol. 41, , [17] G. Meng, C. Pan, N. Zheng, and C. Sun, "Skew estmaton of document mages usng baggng," TIP, vol. 19, , [18] H. Fan, L. Zhu, and Y. Tang, "Skew detecton n document mages based on rectangular actve contour," IJDAR, vol. 13, , [19] Y.-P. Guan, "Fast and robust skew estmaton n document mages through blnear flterng model," IETIP, vol. 6, , [20] "ICDAR2013 DISEC Dataset," htt://users.t.demokrtos.gr/~alexa/disec13/resources.ht ml.

7 orgnal skew:7.11 coarse skew: fnal skew: orgnal skew: coarse skew: fnal skew: orgnal skew:11.59 coarse skew: fnal skew: orgnal skew:14.96 coarse skew: fnal skew: Fgure 5: Four examles of skew estmaton wth the roosed aroach. The frst column gves the orgnal document mages. The second column gves the skew estmaton results of the document mages. The last column gves ther corresondng skew correcton results.