ICDAR2005 Page Segmentation Competition

Transcription

1 ICDAR2005 Page Segmentaton Competton A. Antonacopoulos 1, B. Gatos 2 and D. Brdson 1 1 Pattern Recognton and Image Analyss (PRImA) Research Lab School of Computng, Scence and Engneerng, Unversty of Salford, Manchester, M5 4WT, Unted Kngdom 2 Computatonal Intellgence Laboratory, Insttute of Informatcs and Telecommuncatons, Natonal Center for Scentfc Research Demokrtos, GR Aga Paraskev, Athens, Greece Abstract There s an establshed need for obectve evaluaton of layout analyss methods, n realstc crcumstances. Ths paper descrbes the Page Segmentaton Competton (modus operand, dataset and evaluaton crtera) held n the context of ICDAR2005 and presents the results of the evaluaton of four canddate methods. The man obectve of the competton was to compare the performance of such methods usng scanned documents from commonlyoccurrng publcatons. The results ndcate that although methods seem to be maturng, there s stll a consderable need to develop robust methods that deal wth everyday documents. 1 Introducton Layout analyss methods page segmentaton n partcular contnue to be reported n the lterature on a frequent bass, despte ths beng one of the most mature sub felds of Document Image Analyss. It s not dffcult to see that the reason for ths s that the problem s far from beng solved. Successful methods have certanly been reported but, frequently, those are devsed wth a specfc applcaton n mnd and are fne-tuned to the test mage data set used by ts authors. The wder gamut of documents encountered n real-lfe stuatons s far wder than the target applcatons of most methods. There s no doubt that, for a gven applcaton, or for a generc selecton of real-lfe documents, t would be desrable to obtan an obectve evaluaton of the performance of dfferent layout analyss methods. Such a drect comparson between algorthms s not straghtforward as t requres both the creaton of sutable ground truth (a relatvely laborous and precse task) as well as the defnton of a set of obectve evaluaton crtera (and a method to analyse them). Ths competton focuses on the evaluaton of page segmentaton and regon classfcaton subsystems. To the best of the Authors knowledge, ths s only the second nstance of an nternatonal generc layout analyss competton (the frst beng the ICDAR2003 Page Segmentaton Competton [1]). It should be mentoned that a relatvely close prevous nstance, focussng on a specfc applcaton doman, was the Frst Internatonal Newspaper Page Segmentaton Contest [2] held by the Authors n the context of ICDAR2001. Pror to that, an evaluaton of page segmentaton (as part of OCR systems) was performed at UNLV [3], based on the results of OCR. That approach, however, cannot not be strctly consdered to evaluate layout analyss methods snce the OCR-based evaluaton does not gve suffcent nformaton on the performance of page segmentaton and regon classfcaton and s only applcable to regons of text (or text-only documents). The motvaton for ths competton was the evaluaton of page segmentaton and regon classfcaton methods n realstc crcumstances. By realstc t s meant that the partcpatng methods are appled to scanned documents from a varety of sources, occurrng n real lfe. Ths s n contrast to the maorty of datasets and reports of results usng mostly structured documents (e.g., techncal artcles). The competton and ts modus operand s descrbed next. In Secton 3, an overvew of the dataset and the ground-truthng process s gven. The performance evaluaton method and metrcs are descrbed n Secton 4, whle each of the partcpatng methods s summarsed n Secton 5. Fnally, the results of the competton are presented and the paper s concluded n Sectons 6 and 7, respectvely. 2 The competton The obectve of the competton was to evaluate layout analyss (page segmentaton and regon classfcaton) methods usng scanned documents from commonly-occurrng publcatons. Whle there s a comparatve assessment element nvolved, the real advantage s an ntal look n the performance of dfferent classes of methods (e.g., connected component analyss, morphologcal processng, analyss of background etc.) n dentfyng dfferent types of regons n a varety of documents. Proceedngs of the 2005 Eght Internatonal Conference on Document Analyss and Recognton (ICDAR 05)

2 for the realstc evaluaton of layout analyss methods, real scanned documents gve a better nsght. It should be noted that ground truth there s scarce avalablty of ground truth for the evaluaton of methods analysng complex layouts (e.g., havng non-rectangular regons). Such a dataset was created for the ICDAR2003 competton [1]. However, the current competton was based on a subset of a sgnfcantly updated dataset. Ths dataset, whch wll shortly be released by the PRImA research lab, contans rcher ground truth (n a correspondngly updated XML format) that provdes a very wde range of nformaton on regon attrbutes (physcal and logcal). Although the dataset contans nstances (mages and ground truth) of an exhaustve lst of document types t does focus, however, (for meanngful evaluaton purposes) on the most heavly used (n terms of nformaton content and need to analyse) types of documents, such as offce documents, magazne pages, advertsements and techncal artcles. For the competton, a subset of documents was selected that reflected both realsm n ther frequent occurrence and, at the same tme, the exstence of suffcently general nterest to analyse them. Fgure 1. Sample page mages from the tranng dataset. The competton run n an off-lne mode. The authors of canddate methods regstered ther nterest n the competton and downloaded the tranng dataset (document mages and assocated groundtruth). One week before the competton closng date, regstered authors of canddate methods were able to download the document mages of the evaluaton dataset. At the closng date, the organsers receved the results of the canddate methods, submtted by ther authors n a pre-defned format. The organsers then evaluated the submtted results. It should be noted that the off-lne mode s based on trust that the results submtted by the methods authors are genune. Ths trust s even more necessary f the evaluaton system s publcly avalable. In ths case, the evaluaton system was not publshed (only the prncples) and above all, the organsers have fath n the authors scentfc ntegrty. 3 The dataset For any performance evaluaton approach, the Achlles' heel s the avalablty of realstc and accurate ground truth. As ground-truthng cannot (by defnton) be fully automated, t remans a laborous and, therefore, expensve process. One approach would be to use synthetc data [4]. It s the authors opnon, however, that Fgure 2. Sample page mage from the tranng dataset showng supermposed descrpton of regon contours. Furthermore, a balance had to be acheved between logstcs (a manageable number of document mages) and tractablty for current methods. The decson was, therefore, made to focus on a cross secton of 26 page mages, comprsng 30% techncal artcles (not necessarly wth Manhattan layouts) and 70% magazne Proceedngs of the 2005 Eght Internatonal Conference on Document Analyss and Recognton (ICDAR 05)

3 pages. It should be noted that also for reasons of tractablty, the competton mages were blevel (n the general dataset the orgnal mages are n colour). A sample of page mages gven as part of the tranng dataset can be seen n Fg. 1. The ground-truth of each page mage s an XML fle (defned as part of the general dataset) that contans mage and layout-specfc nformaton as well as the descrpton of the regons n terms of sothetc (havng only horzontal and vertcal edges) polygons. The ground-truth for the competton was produced usng a sem-automated tool developed by the authors. An XML vewer was developed for examnng the mages and the correspondng ground-truth XML, and was dstrbuted to the competton partcpants. Another sample page mage wth the correspondng descrpton of regons supermposed as sothetc polygons can be seen n Fg. 2. The types of regons defned for the competton (smplfed from the total number of dfferent types n the general dataset) are: text, graphcs, lne-art, separator, and nose. 4 Performance evaluaton The performance evaluaton method used s based on countng the number of matches between the enttes detected by the algorthm and the enttes n the ground truth [5-7]. We use a global MatchScore table for all enttes whose values are calculated accordng to the ntersecton of the ON pxel sets of the result and the ground truth (a smlar technque s used at [8]). Let I be the set of all mage ponts, G the set of all ponts nsde the ground truth regon, R the set of all ponts nsde the result regon, g the entty of ground truth, r the entty of result, (s) a functon that counts the elements of set s. Table MatchScore(,) represents the matchng results of the ground truth regon and the result regon. Based on a pxel based approach of [5], and usng a global MatchScore table for all enttes, we can defne that: MatchScore (, ) T( G R I ), where a T( (G R ) I ) { 1, f g r 0, otherwse a (1) If N s the count of ground-truth elements belongng to entty, M s the count of result elements belongng to entty, and w 1, w 2, w 3, w 4, w 5, w 6 are pre-determned weghts, we can calculate the detecton rate and recognton accuracy for entty as follows: DetectRate RecognAccuracy one2one g_one2many g_many2one w1 w 2 w 3 N N N (2 one2one d_one2many d_many2one w 4 w 5 w 6 M M M (3) where the enttes one2one, g_one2many, g_many2one, d_one2many and d_many2one are calculated from MatchScore table (1) followng the steps of [5] for every entty. A performance metrc for detectng each entty can be extracted f we combne the values of the entty s detecton rate and recognton accuracy. We can defne the followng Entty Detecton Metrc (EDM ): EDM 2DetectRate RecognAccuracy (4) DetectRate RecognAccuracy A global performance metrc for detectng all enttes can be extracted f we combne all values of detecton rate and recognton accuracy. If I s the total number of enttes and N s the count of ground-truth elements belongng to entty, then by usng the weghted average for all EDM values we can defne the followng Segmentaton Metrc (SM): SM N I I EDM 5 Partcpatng methods (5) Bref descrptons of the methods whose results were submtted to the competton are gven next. Each account has been provded by the method s authors and edted (summarsed) by the competton organsers. The descrptons vary n length accordng to the level of detal n the source nformaton provded. 5.1 The BESUS method Ths method BESUS stands for Bengal Engneerng and Scence Unversty, Shbpur (Inda) was submtted by S.P. Chowdhury, S. Mandal and A.K. Das (of that unversty) n assocaton wth B. Chanda of the Indan Statstcal Insttute (ISI) n Calcutta. Smlarly to the method submtted by the authors to the ICDAR2003 Page Segmentaton Competton [1], ths s a system constructed usng a number of morphology-based modules. In a pre-processng step that nformaton s gathered and skew s corrected. Horzontal and vertcal separators are extracted next by openng the blevel mage wth a N ) Proceedngs of the 2005 Eght Internatonal Conference on Document Analyss and Recognton (ICDAR 05)

4 horzontal or vertcal (respectvely) structurng element and connected component analyss [9]. Text s segmented based on the spatal relatonshp between pars of textlnes (dentfed based on the smlarty and dstrbuton of connected components) [10]. Graphcs regons are extracted from a greyscale mage (created from the orgnal blevel one) based on the analyss of a cooccurrence matrx n relaton to the result of openng and closng operatons on the whole mage [11]. Lne art regons (components) are dentfed based on topologcal features and a densty rato. Remanng regons are classfed as nose. 5.2 The Océ method Ths method was submtted by M. Blderbeek, Z. Goey and R. Audenaerde of Océ Technologes B.V. n the Netherlands. It s a varant of the wnnng method of the ICDAR2003 Page Segmentaton Competton [1]. Its workng prncples are as follows. Connected components are dentfed n the mage (after removng a 25-pxel wde border) and classfed nto small character, normal character, large character, photograph, graphc, vertcal lne, horzontal lne or nose (n terms of the regon types used n the competton, photographs are graphcs, lnes are separators and graphcs are lne-art) usng a manually constructed decson tree based on features such as wdth, heght, number of pxels etc. Usng the result of ths classfcaton four mages are splt off: (a) an mage contanng photos and nose, (b) an mage contanng graphcs, (c) an mage contanng lnes, and (d) an mage contanng text. In the last case, those blocks, n whch the maorty of connected components are classfed as large characters are splt off to a separate mage. Thus, the mage contanng text s dvded nto two mages: (d1)an mage contanng normal/small text, and (d2)an mage contanng headers. Next, the components n the normal/small text mage (d1), n the photo/nose mage (a) and n the graphcs mage (b) are oned nto blocks usng a run length smearng procedure. The resultng blocks are then classfed by a votng algorthm that takes the connected component class statstcs as ts nput. In the lne mage (c), each lne s consdered as a separate block wth class label separator. The blocks n the header mage (d2) are dentfed by applyng a connected component groupng algorthm, whch also apples a post-classfcaton step to assure that the blocks really contan text. A boundary trackng algorthm [12] s used to trace the outer contours of all blocks (orgnally represented as rectangles) n the smeared mages and represent them as polygons. Fnally, a cleanng step removes all polygons that are contaned wthn others, (re)labels all very small polygons as nose and merges polygons that overlap to a certan extent. 5.3 The Tsnghua methods D Wen and Mng Chen, of Tsnghua Unversty (State Key Laboratory of Intellgent Technology and Systems), n Chna submtted two dfferent methods. The frst one (referred to as Tsnghua method 1 here) s a bottom-up approach that works by progressvely mergng prmtves at dfferent levels (startng from connected components and resultng n text paragraphs etc.) based on the calculaton of a quanttatve measure (the Mult-Level Confdence MLC value). Ths method has been reported n [13] and s adapted to Englsh layouts for ths competton. The output of ths method s boundng rectangles only (a regon may appear splt as a result or boundng rectangles may overlap for dfferent regons) The second method ( Tsnghua method 2 ) s devsed to deal better wth rregular regons. It starts wth the output of method 1 and text regons are separated from non-text ones. Text regons are dentfed as sothetc polygons based on a background analyss algorthm smlar to [14] but workng wth connected components. Other types of regons are output as rectangles exactly as n method 1. 6 Results We evaluated the performance of the 4 segmentaton algorthms usng equatons (1) (5) for all 26 test mages wth parameters w 1 = 1, w 2 = 0.75, w 3 = 0.75, w 4 = 1, w 5 = 0.75 and w 6 = All evaluaton results for all enttes are shown n Fg. 3 where the EDM values averaged over all mages are depcted. Fg. 4 presents the Segmentaton Metrc (SM) values for all segmentaton algorthms averaged over all mages. Fg. 4 shows that the second approach of Tsnghua has an overall advantage. Concernng text regon segmentaton, the second approach of Tsnghua acheved the hghest averaged EDM rate value (53.22%) whle the frst approach Tsnghua, the Océ method and the BESUS method acheved an averaged EDM rate value of 46,64%, 31,16% and 29,62% respectvely. For graphcs, the second approach of Tsnghua acheved the hghest averaged EDM rate value (42,38%). For lne-art and nose enttes, the BESUS method acheved the hghest averaged EDM rate values (80% and 20,24% respectvely) whle for Proceedngs of the 2005 Eght Internatonal Conference on Document Analyss and Recognton (ICDAR 05)

5 seperator detecton, the Océ method acheved the hghest averaged EDM rate value (51,13%). The Tsnghua methods acheved zero EDM rate values for lne-art, separator and nose entty segmentaton. Fgure 3. Evaluaton results for all enttes (EDM values averaged over all mages). Fgure 4. Averaged Segmentaton Metrc (SM) values. 7 Conclusons The motvaton of the ICDAR2005 Page Segmentaton Competton was to evaluate exstng approaches for page segmentaton and regon classfcaton usng a realstc dataset and an obectve performance analyss system. The mage dataset used comprsed scanned techncal artcles and (mostly) magazne pages. The performance evaluaton method used s based on countng the number of matches between the enttes detected by the algorthm and the enttes n the ground truth. The competton run n an off-lne mode and evaluated the performance of four segmentaton algorthms. The evaluaton results show that the second Tsnghua method has an overall advantage (and gves better results for text and graphcs). The Océ method s thrd overall wth good consstency (and the best performance on separators). The BESUS method acheved the hghest rates for lne-art and nose entty segmentaton. References [1] A. Antonacopoulos, B. Gatos and D. Karatzas, ICDAR2003 Page Segmentaton Competton, Proceedngs of the 7 th Internatonal Conference on Document Analyss and Recognton (ICDAR2003), Ednburgh, UK, August 2003, pp [2] B. Gatos, S.L. Mantzars and A. Antonacopoulos, Frst Internatonal Newspaper Contest, Proceedngs of the 6 th Internatonal Conference on Document Analyss and Recognton (ICDAR2001), Seattle, USA, September 2001, pp [3] J. Kana, S.V. Rce, T.A. Nartker and G. Nagy, Automated Evaluaton of OCR Zonng, IEEE Transactons on Pattern Recognton and Machne Intellgence, Vol. 17, No. 1, January, 1995, pp [4] I.T. Phlps, S. Chen and R.M. Haralck, CD-ROM Document Database Standard, Proceedngs of 2 nd Internatonal Conference on Document Analyss and Recognton (ICDAR 93), Tsukuba, Japan, 1993, pp [5] I. Phllps and A. Chhabra, "Emprcal Performance Evaluaton of Graphcs Recognton Systems," IEEE Transacton of Pattern Analyss and Machne Intellgence, Vol. 21, No. 9, pp , September [6] A. Chhabra and I. Phllps, "The Second Internatonal Graphcs Recognton Contest - Raster to Vector Converson: A Report," n Graphcs Recognton: Algorthms and Systems, Lecture Notes n Computer Scence, volume 1389, pp , Sprnger, [7] I. Phllps, J. Lang, A. Chhabra and R. Haralck, "A Performance Evaluaton Protocol for Graphcs Recognton Systems" n Graphcs Recognton: Algorthms and Systems, Lecture Notes n Computer Scence, volume 1389, pp , Sprnger, [8] B.A. Yankoglu, and L Vncent, "Pnk Panther: a complete envronment for ground-truthng and benchmarkng document page segmentaton", Pattern Recognton, volume 31, number 9, pp , [9] S. Mandal, S.P. Chowdhur, A.K. Das and B. Chanda, Automated Detecton and Segmentaton of Form Document, Proceedngs of the 5 th Internatonal Conference on Advances n Pattern Recognton (ICAPR2003), December 2003, Calcutta, Inda, pp [10] A.K. Das and B. Chanda, Segmentaton of Text and Graphcs n Document Image: A Morphologcal Approach, Proceedngs of the Internatonal Conference on Computatonal Lngustcs, Speech and Document Processng (ICCLSDP 98), Calcutta, Inda, December 1998, pp. A50 A56. [11] A.K. Das, S.P. Chowdhur and B. Chanda, A Complete System for Document Image Segmentaton, Proceedngs of natonal Workshop on Computer Vson, Graphcs and Image Processng (WVGIP2002), Madura, Inda, February 2002, pp [12] F. Chang, C.J. Chen, and C.J. Lu, A Lnear-Tme Component-Labellng Algorthm Usng Contour Tracng Technque, Computer Vson and Image Understandng, vol. 93, no. 2, 2004, pp [13] M. Chen, X. Dng, et al. Analyss, Understandng and Representaton of Chnese newspaper wth complex layout. Proceedngs of 7th IEEE Internatonal Conference on Image Processng, Sept. 2000, Vancouver, BC, Canada, IEEE.. [14] A. Antonacopoulos, Page Segmentaton Usng the Descrpton of the Background Computer Vson and Image Understandng, vol. 70, no. 3, 1998, pp Proceedngs of the 2005 Eght Internatonal Conference on Document Analyss and Recognton (ICDAR 05)