ICDAR 2003 Page Segmentation Competition

Transcription

1 ICDAR 2003 Page Segmentaton Competton A. Antonacopoulos 1, B. Gatos 2 and D. Karatzas 1 1 PRImA Group, Department of Computer Scence, Unversty of Lverpool, Peach Street, Lverpool L69 7ZF, Unted Kngdom 2 Computatonal Intellgence Laboratory, Insttute of Informatcs and Telecomuncatons, Natonal Center for Scentfc Research Demokrtos, GR Aga Paraskev, Athens, Greece Abstract There s a sgnfcant need to obectvely evaluate layout analyss (page segmentaton and regon classfcaton) methods. Ths paper descrbes the Page Segmentaton Competton (modus operand, dataset and evaluaton crtera) held n the context of ICDAR2003 and presents the results of the evaluaton of the canddate methods. The man obectve of the competton was to evaluate such methods usng scanned documents from commonly-occurrng publcatons. The results ndcate that although methods seem to be maturng, there s stll a consderable need to develop robust methods that deal wth everyday documents. 1 Introducton Over the last two decades, a plethora of layout analyss page segmentaton n partcular methods have been reported n the lterature. It can be argued that the feld s now begnnng to mature and yet new methods are beng proposed clamng to outperform exstng ones. Frequently, each algorthm s devsed wth a specfc applcaton n mnd and s fne-tuned to the test mage data set used by ts authors, thus makng a drect comparson wth other algorthms dffcult. The need for obectve performance evaluaton of Layout Analyss algorthms s evdent. Ths competton focuses on the evaluaton of page segmentaton and regon classfcaton subsystems. To the best of the Authors knowledge, there has not been any prevous nternatonal generc layout analyss competton. The closest nstance, focussng on a specfc applcaton doman, was the Frst Internatonal Newspaper Page Segmentaton Contest [1] held by the Authors n the context of ICDAR2001. Before that, an evaluaton of page segmentaton (as part of OCR systems) was performed at UNLV [2], based on the results of OCR. That approach, however, cannot not be strctly consdered to evaluate layout analyss methods snce the OCR-based evaluaton does not gve suffcent nformaton on the performance of page segmentaton and regon classfcaton and s only applcable to regons of text (or text-only documents). The motvaton for ths competton was the evaluaton of page segmentaton and regon classfcaton methods n realstc crcumstances. By realstc t s meant that the partcpatng methods are appled to scanned documents from a varety of sources, occurrng n real lfe. Ths s n contrast to the maorty of datasets and reports of results usng mostly structured documents (e.g., techncal artcles). The competton and ts modus operand s descrbed next. In Secton 3, an overvew of the dataset and the ground-truthng process s gven. The performance evaluaton method and metrcs are descrbed n Secton 4, whle each of the partcpatng methods s summarsed n Secton 5. Fnally, the results of the competton are presented and the paper s concluded n Sectons 6 and 7, respectvely. 2 The competton The man obectve of the competton was to evaluate layout analyss (page segmentaton and regon classfcaton) methods usng scanned documents from commonly-occurrng publcatons. A secondary obectve was to create a useful dataset not only consstng of the document pages selected for the competton but wth addtonal mages and groundtruth to make avalable to Layout Analyss researchers, well beyond ICDAR2003. The competton run n an off-lne mode. The authors of canddate methods regstered ther nterest n the competton and downloaded the tranng dataset (document mages and assocated groundtruth). One week before the competton closng date, regstered authors of canddate methods were able to download the document mages of the evaluaton dataset. At the closng date, the organsers receved the results of the canddate methods, submtted by ther authors n a pre-defned format. The organsers then evaluated the submtted results. It should be noted that the off-lne mode s based on trust that the results submtted by the methods authors Proceedngs of the Seventh Internatonal Conference on Document Analyss and Recognton (ICDAR 2003)

2 advertsements. It s the vew of the organsers that the above categores represent a subset of documents that are both realstc n ther frequent occurrence and, at the same tme, of general nterest to analyse. Fgure 1. Sample page mages from the tranng dataset. are genune. Ths can be more so f the evaluaton system s publcly avalable. In ths case, the evaluaton system was not publshed (only the prncples) and above all, the organsers have fath n the authors scentfc ntegrty. 3 The dataset For any performance evaluaton approach, the Achlles' heel s the avalablty of ground truth. As ground-truthng cannot (by defnton) be fully automated, t remans a laborous and, therefore, expensve process. One approach s to use synthetc data [3]. It s the authors opnon, however, that for the realstc evaluaton of layout analyss methods, real scanned documents gve a better nsght. Furthermore, t should be noted that there s currently no ground truth avalable for the evaluaton of methods analysng complex layouts havng nonrectangular regons. Therefore, a new dataset had to be created for ths competton and for later dstrbuton. Followng the ratonale of the competton (realsm), the followng types of documents were selected for ncluson n the dataset (n order of layout complexty): techncal artcles, memos, faxes, magazne pages, and Fgure 2. Sample page mage from the tranng dataset showng supermposed descrpton of regon contours. For the test dataset for the competton, a balance had to be acheved between logstcs (a manageable number of document mages) and tractablty for current methods. The decson was, therefore, made to focus on a cross secton of 32 page mages, comprsng 25% techncal artcles (not necessarly wth Manhattan layouts) and 75% magazne pages. It should be noted that also for reasons of tractablty, the competton dataset was bnarsed (the orgnals n the augmented dataset are n colour). A representatve sample of page mages gven as the tranng dataset can be seen n Fg. 1. The ground-truth of each page mage s an XML fle (defned specfcally for the competton) that contans mage and layout specfc nformaton as well as the descrpton of the regons n terms of sothetc polygons. The ground-truth for the competton was produced usng a sem-automated tool [4]. An XML vewer was developed for examnng the mages and the correspondng ground-truth XML, and was dstrbuted to the competton partcpants. Another sample page mage wth the correspondng descrpton of regons supermposed as sothetc polygons can be seen n Fg. 2. The types of regons defned for the competton (smplfed from the total number of dfferent types n the general dataset) are: text, graphcs, Proceedngs of the Seventh Internatonal Conference on Document Analyss and Recognton (ICDAR 2003)

3 lne-art, separator, and nose. 4 Performance evaluaton The performance evaluaton method used s based on countng the number of matches between the enttes detected by the algorthm and the enttes n the ground truth [5-7]. We use a global MatchScore table for all enttes whose values are calculated accordng to the ntersecton of the ON pxel sets of the result and the ground truth (a smlar technque s used at [8]). Let I be the set of all mage ponts, G the set of all ponts nsde the ground truth regon, R the set of all ponts nsde the result regon, g the entty of ground truth, r the entty of result, Τ(s) a functon that counts the elements of set s. Table MatchScore(,) represents the matchng results of the ground truth regon and the result regon. Based on a pxel based approach of [5], and usng a global MatchScore table for all enttes, we can defne that: T( G R I ) MatchScore (, ) =, where a = T( (G R ) I ) { 1, f g = r 0, otherwse a (1) If N s the count of ground-truth elements belongng to entty, M s the count of result elements belongng to entty, and w 1, w 2, w 3, w 4, w 5, w 6 are pre-determned weghts, we can calculate the detecton rate and recognton accuracy for entty as follows: one2one g_one2many DetectRate + g_many2one = w1 + w 2 w 3 N N N (2 one2one d_one2many RecognAccu racy + d_many2one = w 4 + w 5 w 6 M M M (3) where the enttes one2one, g_one2many, g_many2one, d_one2many and d_many2one are calculated from MatchScore table (1) followng the steps of [5] for every entty. A performance metrc for detectng each entty can be extracted f we combne the values of the entty s detecton rate and recognton accuracy. We can defne the followng Entty Detecton Metrc (EDM ): 2DetectRate RecognAccuracy EDM + = (4) DetectRate RecognAccuracy A global performance metrc for detectng all enttes can be extracted f we combne all values of detecton rate and recognton accuracy. If I s the total number of enttes and N s the count of ground-truth elements belongng to entty, then by usng the weghted average ) for all EDM values we can defne the followng Segmentaton Metrc (SM): SM = N I I EDM 5 Partcpatng methods (5) The followng were the methods whose results were submtted to the competton. 5.1 The DAN method Ths method was submtted by L. Cnque, S. Levald and A. Malza of the Unversty of Rome La Sapenza n Italy. In bref, the DAN system archtecture ncludes four man components: (1) the preprocessor, (2) the splt module, (3) the merge module, and (4) the classfcaton module. The preprocessng algorthm (1) component s appled n order to enhance the qualty of nput data, removng portons of the mage, whch could be consdered as nose. The Splt module (2) takes nput from the preprocessng phase and apples a partcular quad-tree technque n order to splt the document nto small blocks. The result of the Splt module s passed to the Merge module (3), whch apples a pre-classfcaton crteron, mergng smlar regons nto larger regons. Local operators are used wth varable thresholds n order to compute the pre-classfcaton phase. Fnally, usng global operators, the engne of the system s n the Classfcaton module (4) whch executes the classfcaton procedure accordng to the classfcaton logc. In fact, the bran of the system s ths classfcaton module, whch outputs segmented regons and ther attrbutes such as type and sze n an XML fle. A more detaled descrpton of the prncples and workng of the DAN system can be found n a recent paper [9]. 5.2 The ISI method Ths method was submtted by S.P. Chowdhury, A.K. Das, S. Mandal and B. Chanda of the Indan Statstcal Insttute (ISI) n Calcutta, Inda. The system was constructed usng selected tools from a larger morphologcal-approach based system [10]. As the datasets of the competton provdes blevel mages and the half-tone segmentaton algorthm works N Proceedngs of the Seventh Internatonal Conference on Document Analyss and Recognton (ICDAR 2003)

4 wth greyscale mages, the frst step taken by the system s to blur the blevel mage (usng a 3x3 mean flter), producng a grey-valued one. Usng openng and closng operatons, half-tone regons are extracted from the mage. Next, returnng to the orgnal bnary mage (mnus the half-tones), large areas of nose near the edges of the mage are removed usng connected-components analyss. A skew detecton and correcton method s then appled to the mage. Separators, f any, are detected next. Text regons (as defned n the competton rules) are detected ntally as ndvdual math zones, headngs and general text regons. The remanng regons n the mage are nose and lne-art. Fnally, lne-art regons are separated from nose usng connected-component analyss and morphologcal operatons. Indvdual methods are descrbed n a number of publcatons by the system s authors. 5.3 The Océ method Ths method was submtted by Zoé Goey of Océ Technologes B.V. n The Netherlands. It works as follows. Frst, connected components are dentfed and classfed nto small character, normal character, large character, photograph, graphc, vertcal lne, horzontal lne or nose (n terms of the regon types used n the competton, photographs are graphcs, lnes are separators and graphcs are lne-art) usng a manually constructed decson tree based on features such as wdth, heght, number of pxels etc. Usng the result of ths classfcaton three mages are splt off: (a) an mage contanng graphcs, photos and nose, (b) an mage contanng lnes, and (c) an mage contanng text. In the last case, those blocks, n whch the maorty of connected components are classfed as large characters are splt off to a separate mage. Thus, the mage contanng text s dvded nto two mages: (c1) an mage contanng normal/small text (c2) an mage contanng headers Next, the components n the normal/small text mage (c1) and the graphc/photo/nose mage (a) are oned nto blocks usng a run length smearng procedure. The resultng blocks are then classfed by a traned decson tree that takes the connected component class statstcs as ts nput. In the lne mage (b), each lne s consdered as a separate block wth class label separator. The blocks n the header mage (c2) are found by applyng a connected component groupng algorthm, whch also apples a postclassfcaton step to assure that the blocks really contan text. At ths stage, all blocks are only descrbed by ther boundng boxes, snce the above algorthms, currently, do not support arbtrarly polygonal output. To output polygons, a whte space coverng algorthm s used on the smeared text (c1) mage and the resultng polygons are ntersected wth orgnal boundng boxes, removng polygons fully contaned n other polygons. It should be noted that there s a lot of room for mprovement n the polygon generaton step as the desrable mplementaton (had the method s authors had more tme) would be usng a boundary trackng approach. 6 Results We evaluated the performance of the 3 segmentaton algorthms usng equatons (1) (5) for all 32 test mages wth parameters w 1 = 1, w 2 = 0.75, w 3 = 0.75, w 4 = 1, w 5 = 0.75 and w 6 = All evaluaton results for all enttes are shown n Fg. 3 where the EDM values averaged over all mages are depcted. Fg. 4 presents the Segmentaton Metrc (SM) values for all segmentaton algorthms averaged over all mages. Fg. 4 shows that the Océ method has an overall advantage. Concernng text regon segmentaton, the Océ method acheved the hghest averaged EDM rate value (58.96%) whle the DAN method and the ISI method acheved an averaged EDM rate value of about 41%. For graphcs, lne-art, separator and nose enttes the ISI method acheved the hghest averaged EDM rate values (38,46%, 75%, 23,37% and 6,74% respectvely) whle the Océ method acheved lower rates (12,49%, 55,88%, 14,28% and 2,78% respectvely). The DAN method attaned low or zero averaged EDM rate values for graphcs, lne-art, separator and nose entty segmentaton (6,29%, 0%, 0% and 0% respectvely). Fgure 3. Evaluaton results for all enttes (EDM values averaged over all mages). Proceedngs of the Seventh Internatonal Conference on Document Analyss and Recognton (ICDAR 2003)

5 Fgure 4. Averaged Segmentaton Metrc (SM) values. 7 Conclusons The motvaton of the ICDAR2003 Page Segmentaton Competton was to evaluate exstng approaches for page segmentaton and regon classfcaton usng a realstc dataset and an obectve performance analyss system. The mage dataset used comprsed scanned techncal artcles and (mostly) magazne pages. The performance evaluaton method used s based on countng the number of matches between the enttes detected by the algorthm and the enttes n the ground truth. The competton run n an off-lne mode and evaluated the performance of 3 segmentaton algorthms: the DAN algorthm that ncludes four man components (the preprocessor, the splt module, the merge module, and the classfcaton module), the ISI algorthm that s based on selected tools from a larger morphologcal-approach based system, and the Océ algorthm that s based on connected component analyss. The evaluaton results show that the Océ method has an overall advantage whle the ISI method acheved the hghest rates for graphcs, lne-art, separator and nose entty segmentaton. Acknowledgements The organsers would lke to express ther grattude to the man sponsor of the competton, the UK s Government Communcatons Headquarters (GCHQ). The support of ABBYY and Scansoft, n terms of software, s gratefully acknowledged. Last, but not least, thanks are due to the followng people who contrbuted ther effort n dfferent ways: Davd Brdson, Cela Casado-Castlla, Mark Ells, Zhes He, Dave Kennedy, Hong Meng, Stavros Perantons and John Spafford. Transactons on Pattern Recognton and Machne Intellgence, Vol. 17, No. 1, January, 1995, pp [3] I.T. Phlps, S. Chen and R.M. Haralck, CD-ROM Document Database Standard, Proceedngs of 2 nd Internatonal Conference on Document Analyss and Recognton (ICDAR 93), Tsukuba, Japan, 1993, pp [4] A. Antonacopoulos and H. Meng, A Ground-Truthng Tool for Layout Analyss Performance Evaluaton, n the book Document Analyss Systems V: Proceedngs of the Internatonal Assocaton for Pattern Recognton (IAPR) Workshop on Document Analyss Systems (DAS2002), D. Loprest, J. Hu and R. Kash (Eds.), Sprnger Lecture Notes n Computer Scence, LNCS 2423, pp [5] I. Phllps and A. Chhabra, "Emprcal Performance Evaluaton of Graphcs Recognton Systems," IEEE Transacton of Pattern Analyss and Machne Intellgence, Vol. 21, No. 9, pp , September [6] A. Chhabra and I. Phllps, "The Second Internatonal Graphcs Recognton Contest - Raster to Vector Converson: A Report," n Graphcs Recognton: Algorthms and Systems, Lecture Notes n Computer Scence, volume 1389, pp , Sprnger, [7] I. Phllps, J. Lang, A. Chhabra and R. Haralck, "A Performance Evaluaton Protocol for Graphcs Recognton Systems" n Graphcs Recognton: Algorthms and Systems, Lecture Notes n Computer Scence, volume 1389, pp , Sprnger, [8 ]B.A. Yankoglu, and L Vncent, "Pnk Panther: a complete envronment for ground-truthng and benchmarkng document page segmentaton", Pattern Recognton, volume 31, number 9, pp , [9] I L. Cnque, S. Levald, A. Malza, and F. De Rosa, DAN: an automatc segmentaton and classfcaton engne for paper documents, Proceedngs of the Ffth IAPR Internatonal Workshop on Document Analyss Systems (DAS 2002), LNCS 2423, p , August 2002, Prnceton, New Jersey, USA. [10] A.K. Das, S.P. Chowdhur and B. Chanda, A Complete System for Document Image Segmentaton, Proceedngs of natonal Workshop on Computer Vson, Graphcs and Image Processng (WVGIP2002), Madura, Inda, February 2002, pp References [1] B. Gatos, S.L. Mantzars and A. Antonacopoulos, Frst Internatonal Newspaper Contest, Proceedngs of the 6 th Internatonal Conference on Document Analyss and Recognton (ICDAR2001), Seattle, USA, September 2001, pp [2] J. Kana, S.V. Rce, T.A. Nartker and G. Nagy, Automated Evaluaton of OCR Zonng, IEEE Proceedngs of the Seventh Internatonal Conference on Document Analyss and Recognton (ICDAR 2003)