Character-Stroke Detection for Text-Localization and Extraction

Transcription

1 Chrcter-Stroke Detection for Text-Locliztion nd Extrction Krishn Subrmnin Prem Ntrjn Michel Decerbo Dvid Cstñòn Boston University Abstrct In this pper, we present new pproch for nlysis of imges for text-locliztion nd extrction. Our pproch puts very few constrints on the font, size nd color of text nd is cpble of hndling both scene text nd rtificil text well. In this pper, we exploit two well-known fetures of text: pproximtely constnt stroke width nd locl contrst, nd develop fst, simple, nd effective lgorithm to detect chrcter strokes. We lso show how these cn be used for ccurte extrction nd motivte some dvntges of using this pproch for text locliztion over other colorspce segmenttion bsed pproches. We nlyze the performnce of our stroke detection lgorithm on imges collected for the robust-reding competitions t ICDAR Introduction With the prolifertion of digitl medi, techniques for utomted storge nd retrievl hve gined specil significnce. For imge nd video medi, text provides useful cues for content-bsed medi retrievl. There re two populr methods tht re used for nlysis of text in these medi: region-bsed nd texture-bsed. See Jung, Kim nd Jin [1 for comprehensive survey of these methods. We will follow their terminology throughout this pper. Texture bsed methods locte text regions by first locting regions of high contrst nd sptil periodicity. These low-level properties re computed for subset of pixels [2, 3 nd used to trin clssifier. This clssifier is then used to distinguish between regions of text nd bckground. Region-bsed methods use the color properties of text nd their surroundings. They exploit the fct tht there is very little vrition of color within text nd this color is sufficiently distinct from text s immedite bckground to render text perceptible. Most systems using region-bsed method tke dvntge of text s color property by either explicitly thresholding the imge t n intensity level in between the text color nd tht of its immedite bckground [4, 5 or implicitly through color-spce reduction (using clustering nd/or quntiztion) [6, 7. This segregtes the chrcters of the text from their bckground. By mking n dditionl ssumption tht text is monochrome, i.e. ll chrcters of word hve the sme color, text-like pttern in some of the resulting connected components (CCs) cn be detected nd hypothesized s text. The key issues here re finding the right thresholds nd detecting pttern. For medi in which these issues cn be overcome, which includes most high resolution imges nd video, regionbsed methods cn potentilly give better results thn other techniques. They re lso comprtively fster thn texturebsed methods. Their performnce on ICDAR Text locting competitions hs been very good [8, 9. We will review some of the CC-bsed techniques in more detil in Sec. 3. In this pper, we will explore modifiction to the regionbsed prdigm tht ttempts to overcome the twin problems of finding the right thresholds nd detecting text-like ptterns. To solve these problems, we propose bottom-up pproch in which we first locte chrcter strokes nd then use it for locliztion. We will first briefly motivte our intended use of strokes for text-locliztion (Sec. 2), lthough this will not be the focus of this pper. This pper will be chiefly concerned with n lgorithm to locte strokes (Sec. 3) nd its performnce when used in imges with scene text on complex bckground (Sec. 4). We will conclude by commenting on some pitflls tht need to be overcome nd future directions tht we pln to pursue (Sec. 5). 2 From strokes to locliztion After chrcter-strokes hve been detected, they cn be used in powerful wys. Consider section of scene text on very complex bckground, Fig. 1(). Using the procedure detiled in Sec. 3, we detect strokes long the blue horizontl line. The intensity plots long this line, together with line bove nd below it re shown in Fig. 1(b). The detected stroke regions re indicted by red followed by blck. Knowing the stroke regions, we cn ccurtely determine threshold,, tht optimlly seprtes chrcter from its surroundings. In the bove exmple, we use threshold level of to threshold which is shown in Fig. 1(c). The chrcter is poorly extrcted, mking it hrd to detect text-like ptterns. In ddition to the loction of the strokes, the lgorithm lso gives us n estimte of the stroke width,. Assuming some vrince in this estimte cross the entire chrcter, let be the minimum stroke width of the chrcter. We then erode Fig. 1(c) using msk dependent on to get clener seprtion. Fig. 1(d) shows clener imge fter erosion Ninth Interntionl Conference on Document Anlysis nd Recognition (ICDAR 27)

2 () (c).8 (b).6 (d) τ.4.2 τ= Figure 1. "! is the originl imge. #$% is the intensity imge of. () Originl imge, &. (b) Intensity plots long the blue line (' ), ')(+*, nd ',-*. is the stroke width. (c) /. 1 (d) The thresholded imge fter morphologicl opertions nd CC nlysis. using circulr msk of dimeter, 23, followed by connected component nlysis. Only the connected components which include detected strokes re shown. The clen segmenttion simplifies the tsk of pttern detection. 3 Algorithm As mentioned in the introduction, the two issues with CC-bsed techniques re finding ccurte thresholds nd detecting ptterns. There hve been severl pproches tht ddress issues. Mny systems [6, 7, 1 ttempt to solve them s top-down process. In these systems, color-spce reduction, which includes clustering, quntiztion, nd other ssocited pre-processing, is used to extrct useful informtion from the imge. Since there is very little vrition in text-color, it is hoped tht text s colors hve significnt color-spce density, i.e. lrge number of points in color-spce re very close together, in comprison to tht of the bckground. This will led them to form distinct clusters from the bckground fcilitting esy segmenttion. In the next stge, vrious heuristics re used to sift through the segmented ptterns to find text-like chrcteristics. These pproches re usully very fst nd re cpble of delivering rel-time performnce. But since they re built on the ssumption tht the imge is not sprse [1, they re not suitble for mny complex imges. Bottom-up techniques hope to overcome this limittion by loclizing the clustering nd/or segmenttion process to trgeted regions of the imge. In these systems, potentil regions re detected by using properties of text regions other thn color. The most prominent of these is contrst. Lee nd Knknhlli [4 detect regions of high contrst using the difference between djcent pixels. The color distribution in these regions is then used to threshold the imge, followed by heuristics to find ptterns. Using contrst lone leds to mny flse positives in mny complex imges, mking it difficult to detect ptterns. In order to reduce the flse positives nd simplify pttern detection, Wong nd Chen [11 use sptil periodicity of horizontl text in ddition to contrst to locte text-like 4 Stge Fetures 1 Sufficient contrst between chrcter nd its immedite surroundings. 2 Tolernce for vrition in color (nd intensity) within chrcter increses proportionl to the chrcter s color-contrst with its surroundings. 3 Chrcter-strokes (of word) hve similr width nd re reltively close to ech other. Tble 1. Textul fetures exploited t ech stge of the lgorithm regions. Their system effects trde-off between the requirements on color-spce density with sptil density nd contrst. The system prmeters re quite sensitive to the properties of imge. For instnce, it would be difficult to find set of (-priori) thresholds tht work well for imges of the type shown in Figs. 3(), 3(c). Our system is bottom-up pproch bering mny similrities to [11, 4. It uses line-scn-bsed pproch to finding interesting res hving text-like properties. The pproch is gered towrd finding strokes of chrcters forming word. In this system, we exploit few of the properties of text regions tht re pplicble to mny font systems. The entire system cn be broken-up into stges: ) scn horizontl line of the intensity imge to locte flux points or regions of high contrst, b) filter these flux points using sptil nd color-spce constrints to obtin seeds, nd c) collectively filter the seeds bsed on stroke proximity nd similrity (for them to be strokes of word) to obtin strokes. This pproch works only if multiple seeds re detected on chrcters of word or t lest on chrcters of words in sentence hving sme size nd lying close to ech other. Ech stge exploits distinct feture of text regions s listed in Tble 1. Before we describe the detils of our lgorithm, we would like to highlight the novel fetures of our system. Although we require sufficient contrst between text nd its surroundings, we do not require it to be sudden, i.e. we do not mesure chnge between djcent pixels but the totl chnge (see Sec. 3.1). Hence, it should work well Ninth Interntionl Conference on Document Anlysis nd Recognition (ICDAR 27)

3 V ( u even when the edges of chrcter re blurred, s in Fig. 3(c). The system works well with sprse imges nd is not restricted to monochrome text, s seen in Figs. 3(), 3(f). It lso doesn t put restrictions on sptil density of text with the only requirement tht the chrcter strokes of text hve pproximtely equl width. We will use some terminology to better describe the system in concise mnner. Line-scn A set of pixels long the horizontl line of the intensity imge. Group-scn Intensity plot long three consecutive even or odd numbered lines, 57698;:<5=:<5<>?8. In Fig. 1(b), this is represented by the yellow, blue nd cyn colored lines respectively. Flux region Regions of line-scn showing significnt chnge of intensity. Depending on whether the chnge is positive or negtive, the regions re clssified s hving positive flux or negtive flux respectively. Flux points Adjcent points within flux region with mximum intensity chnge. In Fig. 1b, these points re indicted by pink nd green dots. Seed The region between negtive flux followed by positive flux. Stroke A seed tht is hypothesized s chrcter-stroke. In Fig. 1(b), this is indicted by (for negtive flux) followed by (for positive flux). All stges of the lgorithm work with the intensity imge. The lgorithm detects drk text (low intensity vlue) on light bckground (high intensity vlue). Hence, the entire stroke-detection lgorithm is repeted twice, once on the imge nd gin on the inverse imge. Given n imge, the set of lines to scn is decided -priori. It is hoped tht the density of scnned lines is sufficiently lrge to cover ll text regions. Since ll hypothesis genertion is bsed on informtion from group-scn lone, the number of lines scnned cn be ltered without ffecting the performnce of the rest of the system. ACBED FHGJILKMGJILNPOMQSRCTU In the first stge of the stroke-detection process, linescn is performed on ech line belonging to the group-scn. The resulting plot is nlyzed for criticl points (minim nd mxim) nd difference between djcent criticl points is computed. Only pirs which exhibit n bsolute difference of more thn re retined nd form the flux points for this line. Flux points from the lines in the group-scn re then jointly nlyzed. A window of width is used to nlyze the group-scn. The set of flux points lying within this window re ccepted if: () the set of flux points in the window re of only one type, i.e. re either positive or negtive, nd (b) ech line of the group contributes exctly one point to this set. Otherwise, the flux points re rejected. The region between n ccepted set of flux points represent flux region. Note tht the intensity plot in flux region is monotonic. ACBXW FHGJILKMGJILNYUZTT[K\U A seed chrcterizes region with smll vrition in color. The tolernce to this color vrition is proportionl Figure 2. Finding seeds. to the region s contrst from its immedite surroundings. We exmine the region between negtive flux followed by positive flux for seeds. In formlizing our pproch to detecting seeds, few importnt points nd regions re indicted in Fig. 2. More detils re provided in Sec In the figure, group-scn between negtive flux followed by positive flux is shown. The region between these two fluxes is exmined for seed-like chrcteristics. In vlidting seed, four constrints re imposed on this region. They cn be clssified into sptil constrints nd color-spce constrints Sptil constrints _ nd ` re theh highest nd lowest points of the negtive flux. Likewise, nd ` re the highest nd lowest points of the positive flux. The first condition we impose is Yb tht the positive flux follows the negtive flux, i.e.. Since we only intend to detect drk text on light bckground, this condition is vlid. Next, using -priori knowledge bout stroke s width, we constrin the width of the seed, i.e. dcfe gh..i/cfjlk, where Hcfe g nd Hcfjlk re the minimum nd mximum width of chrcter respectively. These two conditions impose sptil constrints on the seed Color-spce constrints The color-spce constrints ensure tht the intensity vrition within seed is smll frction of its contrst with surroundings. Let mnop q be the intensity mp of the imge, _ where is the set of ll pixels of the imge. Since belong to flux regions, for ech line- l` / nd l` scn r, ts vu w (S ts ` uxw b nd ts vu 1w (S ts ` yw b. We first impose loose constrint which ensures tht there is good intensity-seprtion between seed nd its surroundings. Let z denote the medin function nd { denote the union of two sets. Then, }~ ƒ, ƒˆ where ƒ Š zœ7 =ŽZ Ž h {ˆ, Œ 7`, is n intensity vlue which lies in between seed intensity nd its bckground intensity. This is indicted Ninth Interntionl Conference on Document Anlysis nd Recognition (ICDAR 27)

4 ` { { ( Expt. SP CR WR (s) Tble 2. Performnce comprison by the purple dshed line in Fig. 2. The conditions, Œl Hb nd Œ7 X ) š h` ensure tht the seed nd its bckground re seprted in the color-spce. ` is the set of ll points between ` nd `. The next condition imposes tighter constrint on the intensity vrition within the seed in reltion to its surroundings. The upper nd lower envelopes of the three lines of the group-scn between ` nd re given by œx ` nd œnž ` respectively. The verge intensity vrition within the seed is given by Ÿ } zœœ ` ( œ ž `. The verge contrst between the seed nd its bckground is given by. } z Œl ( ; ` { ` The condition, q x (šÿt b ensures tht Ÿ is low compred to. is n -priori similrity threshold which ensures color consistency within the seed. ACB A FHGJILKMGJILNYU;ªy«T[U For finding strokes from seeds, we compre the position long the group-scn, š nd length, `\ H z / _ of seed with other seeds detected on the group-scn. In compring seeds nd ±, we ensure tht they re sufficiently close, depending on their widths. Seed is sid to be neighbor of seed ± if: () they re sptilly close to ech other, i.e. š²m(³.µ _ º¹ 7d², nd (b) their widths re similr, i.e. `L²M( `.¼». is scling fctor, is the width of seed, nd» is the llowble vrition in seed-width. A seed is hypothesized s stroke if it hs t lest ½ neighbors. 4 Performnce Stroke detection forms the first stge of the textlocliztion nd extrction process. At this stge, quickly finding chrcter-strokes with high recll rte is of primry importnce. In subsequent stges, we will use locl informtion t stroke-sites to estimte ccurte thresholds nd lso detect ptterns. In Figs. 3()-3(b), n exmple of stroke detection nd subsequent segmenttion on n imge re shown. The segmenttion results show only connected components which contin stroke. As seen in Fig. 3(b), most flse-positives from the stroke detection process form connected components hving dissimilr shpes nd sizes. It is esy to eliminte these components using heuristics. The key concern in the stges subsequent to segmenttion is to void dropping components ssocited with chrcters which leds to reduced text-locliztion performnce. As. demonstrted in Sec. 2, our technique prtilly llevites this concern by using locl estimtes of thresholds coupled with stroke-width informtion. In the rest of this section, we will study the performnce of the stroke-detection lgorithm on the dtbse collected for the ICDAR 23 robust-reding competitions [8. The imge corpus contins 246 high-resolution imges of scene text. All text ppering in n imge is nnotted t chrcter nd word level. Imges contin scene text ppering in complex settings. The size of the imge vries from ¾Z SÀ 4y pixels to ¾ y SÀ Á* y pixels. In our experiments, we use four mtrices to mesure the performnce of the stroke detection lgorithm: ) stroke precision, b) chrcter recll, c) word recll, nd d) computing time. Although the dtbse is not nnotted for strokes, it is still possible to glen useful informtion from the chrcter level nnottions in order to mesure stroke precision. In our experiments, we define hypothesized stroke s correct if it lies entirely within chrcter s boundries. Hypothesized strokes which lie between two chrcters re ignored. Stroke-Precision (SP) is the number of correct strokes to totl detected strokes. A chrcter is considered detected if it hs t lest one hypothesized stroke on it. We define Chrcter-Recll (CR) s the rtio of the number of detected chrcters to the totl number of chrcters. A word is considered detected if one or more of its constituent chrcters hve been detected. We define Word-Recll (WR) s the rtio of the number of words tht re detected to the totl number of words. Throughout the development of this system, conscious effort ws mde to reduce the number of prmeters used. In fct, our system s performnce is determined, primrily, by 3 prmeters: ) number of lines scnned, b) minimum contrst,, nd c) similrity threshold,. The defult vlues for other prmeters re»3, ½Â, í ÿ, cfe g Ã, cfjlk Z V, nd. In Tble 2, we show the results of 5 experiments tht were conducted. The individul impct of, nd on system performnce is compred ginst bseline system (Expt 1) by vrying one prmeter t time (Expts. 2,3, nd 4). We lso consider the combined effect of these chnges (Expt. 5). The performnce mtrices, SP, CR nd WR, shown in the tble re indictive of the overll performnce of the system on the entire imge dtbse. is the verge time tken to process ech imge. It is informtive to nlyze the performnce of the system on ech imge individully. In Figs. 3(i)-(k), histogrms of the SP, CR, nd WR for ech imge in the dtbse re shown. ÄSBED Å#ILÆÈÇ=ÉCUyGJU The stroke-detection results re independent of colorspce chrcteristics of text nd depends only on the locl contrst, color consistency within ech chrcter, nd stroke width. So, the reported results re representtive of monochrome nd polychrome text. In defining the CR metric for chrcter recll, we ssume tht in order to Ninth Interntionl Conference on Document Anlysis nd Recognition (ICDAR 27)

5 () (b) (c) (e) (i) Stroke Precision (f) (j) Chr Recll (g) (k) Word Recll Figure 3. ()-(b) Stroke detection followed by segmenttion. (c)-(g) More exmples. (i)-(k) Histogrms of performnce mtrices for ech imge for Expt. 5. The red ptches in the imge denote detected strokes. ccurtely extrct chrcter, it is sufficient to detect constituent stroke. Accurte text-locliztion, i.e. drwing boxes round clusters of chrcters, should then be possible. Hence, the reported results for the CR metric should lso be indictive of recll rte for the text-locliztion tsk for polychrome text. Likewise, for monochrome text, given tht the text is horizontl, one or more strokes on ny of the word s constituent chrcters mkes it possible to loclize the word. Hence, the reported results for the WR metric should be indictive of the recll rte for monochrome text. The WR rte reported for the best performing system t the ICDAR 23 competitions [8 ws * %. The current system is prototype nd hs not been optimized for run-time performnce. Comprison of the run-time performnce of our system with those of others using the metric my not be very useful. We use the metric to evlute performnce improvements between experiments nd lso for nlyzing system performnce on ech imge. As cn be seen from Tble 2, incresing proportiontely increses s cn be expected. In ddition, it improves CR nd WR t the cost of SP. Both nd impose strict requirements on locl contrst nd color consistency. Incresing them improves SP t the cost of missing some vlid strokes, detrimentlly ffecting CR nd WR. Further nlysis of performnce on individul imges (see the histogrms in Fig. 3(i)-(k)) revels tht the performnce on most of the imges is very good. It cn be inferred from 1 1 the histogrms tht the medin vlues of SP, CR, nd WR re much higher thn the verge vlues reported in Tble 2. 5 Conclusion nd comments In this pper, we pproch the text-locliztion problem using CC-bsed pproch by first detecting chrcter strokes. A detected stroke is used to estimte threshold nd stroke-width which re used for chrcter segmenttion. We demonstrte our chrcter segmenttion pproch on n complex imge nd motivte dvntges of using this pproch. We describe three-stge lgorithm to detect strokes long horizontl scn of the intensity imge. The sensitivity of the detection lgorithm to three key prmeters is evluted ginst four mtrices: stroke precision, chrcter recll, word recll, nd computing time. The lgorithm ws shown to perform well on vriety of imges contining scene text. Although the chrcter detection lgorithm performs well on most font systems, it performs poorly on itlic fonts or when chrcters of word re smudged together. In these cses, the chrcters re either too inclined or their stroke widths re not esy to detect. Instnces of these cses re seen in Figs. 3(c),3(e) respectively. The system uses intensity imges to detect fluxes. Performnce cn be improved if we work directly on the color spce to detect chrcter strokes. We re ddressing some of these issues in our ongoing work. We re lso working on using the chrcter detection lgorithm detiled in this pper for text-locliztion nd extrction. References [1 K. Jung, K.I. Kim, A.K. Jin, Text informtion extrction in imges nd video: survey, Pttern Recognition 37 (24) [2 Christin Wolf, Detection de textes dns des imges issues d un flux video pour l indextion semntique. PhD thesis, Institut Ntionl de Sciences Appliquees de Lyon Frnce, 23. [3 S. Kumr, N. Khnn, et. l, Locting Text in Imges using Mtched Wvelets, ICDAR 25, [4 C.M. Lee, A. Knknhlli, Automtic extrction of chrcters in complex imges, Intl. J. of Pttern Recognition nd Artif. Intl. 9 (1) (1995) [5 J. Ohy, A. Shio, S. Akmtsu, Recognizing chrcters in scene imges, IEEE Trns. Pttern Anl. Mch. Intell. 16 (2) (1994) [6 A.K. Jin, B. Yu, Automtic text loction in imges nd video frmes, Pttern Recognition 31 (12) (1998) [7 Y. Zhong, K. Kru, A.K. Jin, Locting text in complex color imges, Pttern Recognition 28 (1) (1995) [8 S.M. Lucs, A. Pnretos, L. Sos, A. Tng, S. Wong, R. Young, ICDAR 23 Robust Reding Competitions, ICDAR 23, [9 S. M Lucs, ICDAR 25 Text Locting Competition Results, ICDAR 25, 8-84 [1 Hse, T. Shinokw, M. Yoned, C.Y. Suen, Chrcter string extrction from color documents, Pttern Recognition 34 (7) (21) [11 E. K. Wong, M. Chen, A new robust lgorithm for video text extrction, Pttern Recognition 36 (23) Ninth Interntionl Conference on Document Anlysis nd Recognition (ICDAR 27)