MULTIRESOLUTION TEXT DETECTION IN VIDEO FRAMES

Transcription

1 MULTIRESOLUTION TEXT DETECTION IN VIDEO FRAMES Maros Anthmopoulos, Basls Gatos and Ioanns Pratkaks Computatonal Intellgence Laboratory, Insttute of Informatcs and Telecommuncatons Natonal Center for Scentfc Research "Demokrtos", GR-53 0 Aga Paraskev, Athens, Greece anthmop@t.demokrtos.gr, bgat@t.demokrtos.gr, pratka@t.demokrtos.gr Keywords: Abstract: Text detecton, vdeo frames, artfcal text, edge-based. Ths paper proposes an algorthm for detectng artfcal text n vdeo frames usng edge nformaton. Frst, an edge map s created usng the Canny edge detector. Then, morphologcal dlaton and openng are used n order to connect the vertcal edges and elmnate false alarms. Boundng boxes are determned for every non-zero valued connected component, consstng the ntal canddate text areas. Fnally, an edge projecton analyss s appled, refnng the result and splttng text areas n text lnes. The whole algorthm s appled n dfferent resolutons to ensure text detecton wth sze varablty. Expermental results prove that the method s hghly effectve and effcent for artfcal text detecton. INTRODUCTION Nowadays the sze of the avalable dgtal vdeo content s ncreasng rapdly. Ths fact leads to an urgent need for fast and effectve algorthms for nformaton retreval from multmeda content for applcatons n vdeo ndexng, edtng or even vdeo compresson. Text n vdeo and mages proves to be a source of hgh-level semantcs closely related to the concept of the vdeo. Moreover, artfcal text can provde us wth even more powerful nformaton for televson captured vdeo ndexng snce ths knd of text s added n order to descrbe the content of the vdeo or gve addtonal nformaton related to t. The procedure of retrevng text from vdeo s usually called Vdeo OCR and conssts of 3 basc stages: text detecton, text segmentaton and recognton. Text detecton s a crucal step towards the completon of the recognton process. The am of ths paper s to gve an algorthm for the spatal detecton of artfcal text n vdeo frames. The algorthm ntends to produce one boundng box for every text lne of the frame. Artfcal text presents some features and characterstcs n order to be readable from humans, lke hgh ntensty vertcal edge strokes, colour homogenety, contrast between text and background, horzontal algnment, varous geometrcal constrants etc. The above features and constrants are usually used by the text detecton systems for dstngushng text areas from non-text areas. On the other hand, there are many challenges that have to be faced lke, text embedded n complex backgrounds, wth unknown color, sze, font or low resoluton text. Many researchers have proposed methods based on dfferent archtectures, feature sets, and classfers. These methods generally can be classfed nto two categores: Bottom-up methods and Top-down methods. Bottom-up methods segment mages nto character regons and group them nto words. Lenhart et al. (Lenhart, 995) regard text regons as connected components wth the same or smlar color and sze, and apply moton analyss to enhance the text extracton results for a vdeo sequence. The nput mage s segmented usng a splt-and-merge algorthm. Fnally, a geometrc analyss, ncludng the wdth, heght, and aspect rato, s used to flter out any non-text components. Sobottka et al. (Sobottka, 999) use a regon growng method n order to detect homogeneous regons. Begnnng wth a start regon, pxels are merged f they belong to the same cluster. Then the regons are grouped to form text lnes assumng that text lnes consst of more than three regons havng a small horzontal dstance and a large vertcal overlap to each other. Top-down methods frstly detect text regons n mages and then splt them n text lnes. These methods are also dvded nto two sub-categores: Heurstc methods and Machne learnng methods. Heurstc methods use heurstc flters n order to detect text. Malobabc et al.(malobabc, 004) and X (X, 00) et al. propose edge based methods for detectng text regons. An edge map s calculated followed by smoothng flters, morphologcal operatons and geometrcal constrants. However the 6

2 VISAPP Internatonal Conference on Computer Vson Theory and Applcatons use of Sobel operator cannot preserve successfully the contours of the characters. Zhong et al. (Zhong, 000) use the DCT coeffcents of compressed jpeg or mpeg fles n order to dstngush the texture of textual regons from non-textual regons. Sato et al. (Sato, 998) apply a 3x3 horzontal dfferental flter to the entre mage wth approprate bnary thresholdng. If a boundng regon whch s detected by the horzontal dfferental flterng technque satsfes sze, fll factor and horzontal-vertcal aspect rato constrants, t s selected for recognton as a text regon. Du et al. (Du, 003) propose a methodology that uses MPCM (Multstage Pulse Code Modulaton) to locate potental text regons n colour vdeo mages and then apples a sequence of spatal flters to remove nosy regons, merges text regons, produces boxes and fnally elmnates the text boxes that produce no OCR output. Crandall et al. (Crandall, 003) use the DCT coeffcents to detect text areas. Then connected component analyss s performed n them followed by an teratve greedy algorthm whch refnes the skew, poston and sze of the ntal boundng boxes. Machne learnng methods use traned, machne learnng technques n order to detect text. L et al. (L, 000) propose a method based on neural networks traned on wavelet features. The NN classfes the pxels of a sldng wndow of 6x6 pxels. Wolf et al. (Wolf, 004) use an SVM traned on dervatve and geometrcal features. Yan et al. (Yan, 003) use a Back Propagaton Artfcal Neural Network traned on Gabor edge features. Ye et al. (Ye, 005) use SVM and wavelets. Wu et al. (Wu, 005) propose a system of two co-traned SVM s on edge and color features. Lenhart et al. (Lenhart, 00) propose a method based on neural network classfcaton usng gradent features. Chen et al. (Chen, 00) use several heurstcs based on edges to detect text and then refne the results usng a Bayesan Classfer traned on features based on geometry and projecton analyss. Clark et al. (Clark, 000) presents fve statstcal measures for tranng a NN. Chen et al. (Chen, 003) use features lke Greyscale spatal dervatves, dstance maps, constant gradent varance and DCT coeffcents fed to an SVM classfer. The structure of the remanng of our paper s as follows: Secton descrbes the proposed algorthm and ts dfferent stages, secton 3 presents the evaluaton method and the expermental results and secton 4 provdes a concluson. TEXT DETECTION ALGORITHM The proposed algorthm (Fgure. ) explots the fact that text lnes produce strong vertcal edges horzontally algned and follow specfc shape restrctons. Usng edges as the promnent feature of our system gves us the opportunty to detect characters wth dfferent fonts and colours snce every character present strong edges, despte ts font or color, n order to be readable. An example of artfcal text n a vdeo frame s gven n Fgure. Fgure : Example of artfcal text. Fgure : Flowchart of the proposed text detecton algorthm.. Text Area Detecton As a frst step of our methodology we produce the edge map of the vdeo frame mage. Several methodologes are used n the lterature for computng the edge map of an mage (Gonzalez, 99). For our algorthm we use Canny (Canny, 986) edge detector appled n greyscale mages. Canny uses Sobel masks n order to fnd the edge magntude of the mage, n gray scale, and then uses non-maxma suppresson and hysteress 6

3 MULTIRESOLUTION TEXT DETECTION IN VIDEO FRAMES thresholdng. Wth these two post-processng operatons Canny edge detector manage to remove non-maxma pxels, preservng the connectvty of the contours. Ideally the created edge map s a bnarzed mage wth the pxels of contours set to one (whte) and the background equal to zero (black) (Fgure 3a). After computng the Canny edge map, a dlaton by an element 5x s performed to connect the character contours of every text lne (Fgure.3b). Experments showed that a cross-shaped element has better results. Then a morphologcal openng s used, removng the nose and smoothng the shape of the canddate text areas (Fgure.3c). The element used here s also cross-shaped wth sze x45. Every component created by the prevous dlaton wth heght less than or wdth less than 45 s suppressed. Ths means that every edge whch could not connect to a component larger than the element of the dlaton wll be lost. Unfortunately ths operaton may suppress the edges of text lnes wth heght less than pxels. However, ths s not so devastatng snce characters of ths sze are ether way not recognzed n the fnal stage of the Vdeo OCR system. Now every component represents a canddate text area. Fnally a connected component analyss helps us to compute the ntal boundng boxes of the canddate text areas (Fgure.3d). (a) (c) (b). Text Lne Detecton The prevous stage has a hgh detecton rate but relatvely low precson. Ths means that most of the text lnes are ncluded n the ntal text boxes whle at the same tme some text boxes may nclude more than one text lne as well as nose. Ths nose usually comes from objects wth hgh ntensty edges that connect to the text lnes durng the dlaton process. Ths low precson also comes from detected boundng boxes whch do not contan text but objects wth hgh vertcal edge densty. To ncrease the precson and reject the false alarms we use a method based on horzontal and vertcal projectons. Frstly, the horzontal edge projecton of every box s computed and lnes wth projecton values below a threshold are dscarded. In ths way boxes wth more than one text lne are dvded and some lnes wth nose are also dscarded (Fgure. 4). Besdes, boxes whch do not contan text are usually splt n a number of boxes wth very small heght and dscarded by a next stage due to geometrcal constrants. (d) Fgure. 3: Text area detecton. (a) Edge map, (b) Dlaton, (c) Openng, (d) CC analyss, Intal boundng boxes. A box s dscarded f: Heght s lower than a threshold (set to ), Heght s greater than a threshold (set to 48), Rato wdth/ heght s lower than a threshold (set to.5). Then, a smlar procedure wth vertcal projecton follows (Fgure. 5). Ths method would actually break every text lne n words or even n characters. However, ths s not an ntenton of the algorthm so fnally the vertcally dvded parts are reconnected f the dstance between them s less than a threshold whch depends on the heght of the canddate text lne (set to.5*heght). In ths way, a boundng box wll splt only f the dstance between two words s larger than the threshold whch means 63

4 VISAPP Internatonal Conference on Computer Vson Theory and Applcatons that actually belong to dfferent text lnes or f a part of the canddate text lne contan only nose. The whole procedure wth horzontal and vertcal projectons s repeated three tmes n order to segment even the most complcated text areas and results to the fnal boundng boxes (Fgure. 6). Fgure 4: Example of horzontal projecton. flter s appled so the edges of the background become weaker compared to the edges of the text whch stll reman strong. Ths flter s not appled to the frst resoluton because t would destroy the contrast of the small characters that already suffer the blurrng caused by vdeo compresson. Takng nto account that artfcal text n vdeos usually does not contan very large characters and from the experence of related experments we chose to use two resolutons for ths approach: the ntal, and the one wth a scale factor of 0.6. In ths way the system can detect characters wth heght up to 80 pxels whch was consdered to be satsfyng. Fgure 5: Example of vertcal projecton. (a) Fgure 6: Boundng boxes after projecton analyss..3 Multresoluton Analyss Usng edge features n order to detect text gves to the method ndependence from text color and dfferent fonts. However, ths method clearly depends on the sze of the characters. The sze of the elements for the morphologcal operatons and the geometrcal constrants gve to the algorthm the ablty to detect text n a specfc range of character szes. Wth the values descrbed above the algorthm s capable of detectng accurately characters wth heght from to 48 pxels. To overcome ths problem we adopt a multresoluton approach. The algorthm descrbed above s appled to the mage n dfferent resolutons and fnally the results are fused to the ntal resoluton. Ths fuson mght be qute dffcult f we consder that the same text mght be detected n dfferent resolutons so boundng boxes wll overlap. To avod that, the algorthm suppresses the edges of the already recognsed characters n a resoluton before the edge map s passed to the next resoluton. For every resoluton, except for the ntal a blurrng (c) (b) (d) Fgure. 7: Multresoluton analyss. (a) Fne resoluton, (b) Result at fne resoluton, (c) Coarse resoluton, (d) Result at coarse resoluton. 3 EVALUATION METHOD AND EXPERIMENTAL RESULTS Desgnng evaluaton methods for text detecton s an aspect that has not be studed extensvely. Very few related works have been publshed, moreover ths works propose evaluaton strateges wth very complcated mplementatons or demand great effort for the generaton of the ground truth (Hua, 003), (Manohar, 006). In ths work we used as evaluaton 64

5 MULTIRESOLUTION TEXT DETECTION IN VIDEO FRAMES ndcators the recall and precson rates on a pxel base. For the computaton of these rates we need to calculate the number of the pxels for the ground truth boundng boxes, for the boundng boxes of the detecton method and for ther ntersecton. As fnal measure we use the F-measure, whch s the harmonc mean of recall and precson. However ths method proved to have several drawbacks whch have to be faced. The frst s that there s not an optmal way to draw the ground truth boundng boxes. Ths means that two boxes may be accurate enough for boundng a text lne although they may not nclude exactly the same pxels. To overcome ths problem one can segment the text pxels from the background pxels and then demand the presence of text pxels n the output boundng box. However, ths would make the detecton evaluaton depend on the performance of text segmentaton whch s surely not desrable. In ths work, we follow a more smple strategy to solve ths problem. The ground truth boundng boxes are drawn n a way that the margns between the text pxels and the edge of the box are equal for all text lnes. Moreover, as last stage of the detecton algorthm all boundng boxes grow by 8 pxels n wdth and heght, provdng a satsfyng approxmaton of the ground truth. Another drawback s the fact that ths method actually measures the percentage of detected pxels. However the goal of the detecton algorthm s not to detect maxmal amount of pxels but the maxmal number of characters. In other words, a text lne must have nfluence to the fnal evaluaton measure proportonal to the number of contanng characters and not to the number of ts pxels. Unfortunately, the number of characters n a box cannot be defned by the algorthm but t can be approxmated by the rato wdth/heght of the boundng box, f we assume that ths rato s nvarable for every character and the spaces between dfferent words n a text lne s proportonal to ts heght. In ths way, the recall and precson rates are gven by the equatons (), (). Where hg s the heght of the th ground truth boundng box, EG s ts number of pxels, EGD s the number of pxels of the ntersecton that belong to th ground truth boundng box, hd s the heght of the th detecton boundng box, ED s ts number of pxels, EDG s the number of pxels of the ntersecton that belong to th detecton boundng box, N s the number of ground truth boundng boxes and M s the number of detected boundng boxes. M = EDG hd = hd Precson = M () ED N = EGD hg = hg Recall = N () EG Where hg s the heght of the th ground truth boundng box, EG s ts number of pxels, EGD s the number of pxels of the ntersecton that belong to th ground truth boundng box, hd s the heght of the th detecton boundng box, ED s ts number of pxels, EDG s the number of pxels of the ntersecton that belong to th detecton boundng box, N s the number of ground truth boundng boxes and M s the number of detected boundng boxes. As an overall measure we use the weghted harmonc mean of precson and recall also referred as the F-measure (3). * Precson * Recall F = Precson + Recall (3) By usng ths evaluaton method we try to approxmate the rates of character detecton through pxel detecton rates. For testng the algorthm s performance, 3 sets of vdeo frames (70x480) have been used, captured from TRECVID 005 and 006 ( For the results of Table, a Pentum 4, 3. GHz processor has been used. The results of set proved to be worse than the others. Ths s probably because ths set contans mages wth very large fonts and also some scene text. Table : Results of the algorthm. Set Set Set3 Number of mages Number of ground truth boxes Recall 90.89% 83.4% 9.34% Precson 83.5% 88.87% 9.06% F-measure 87.05% 86.04% 9.9% Tme (secs) 37 3 Expermental results showed that very large fonts cannot be detected usng only the edge nformaton of the mage because the edge texture of a large font 65

6 VISAPP Internatonal Conference on Computer Vson Theory and Applcatons has many smlartes wth the texture of background objects. Set3 contans artfcal text wth small fonts and Set contans text n many dfferent szes as well as some scene text. 4 CONCLUSION In ths paper, we present an edge-based algorthm for artfcal text detecton n vdeo frames. The proposed methodology explots the fact that text lnes produce strong vertcal edges horzontally algned and follow specfc shape restrctons. Although the algorthm s desgned to detect horzontal artfcal text, scene text can also be detected n some cases. Expermental results advocate very good performance n a varety of dfferent vdeo frames. The method s vulnerable n very complex backgrounds. In our future work, we plan to explot the color homogenety of text. ACKNOWLEDGEMENTS Ths research s carred out wthn the framework of the European project BOEMIE ( The research n ths work has receved research fundng from the EU-IST Sxth Framework Programme. REFERENCES Canny J., 986. A computatonal approach to edge detecton, IEEE Trans. Pattern Analyss and Machne Intellgence, 8, Chen D., H. Bourlard, 00. and J. -P. Thran, Text Identfcaton n Complex Background usng SVM, Proc. of IEEE Conference on Computer Vson and Pattern Recognton, Vol., pp Chen Datong, Km Shearer and Herve Bourlard, 003. Extracton of specal effects capton text events from dgtal vdeo, Internatonal Journal of Document Analyss and Recognton IJDAR(5), No. -3, pp Clark P. and M. Mrmehd, 000. Fndng Text Regons Usng Localsed Measures, Proceedngs of the th Brtsh Machne Vson Conference. Crandall Davd, Sameer Antan, Rangachar Kastur, 003. Extracton of specal effects capton text events from dgtal vdeo IJDAR(5), No. -3, pp Du, Yngz, Chang, Chen-I Thoun, Paul D., 003. Automated system for text detecton n ndvdual vdeo Images, Journal of Electronc Imagng, (3), Gonzalez R. and R. Woods, 99. Dgtal Image Processng Addson Wesley, pp Hua Xan-Sheng, Lu Wenyn, HongJang Zhang, 004. An automatc performance evaluaton protocol for vdeo text detecton algorthms. IEEE Trans. Crcuts Syst. Vdeo Techn. 4(4), L Hupng, Davd Doermann, 000. A Closed-Loop Tranng System for Vdeo Text Detecton, Cogntve and Neural Models for Word Recognton and Document Processng, World Scentfc Press. Lenhart Raner and Frank Stuber, 995. Automatc text recognton n dgtal vdeos, Techncal Report / Department for Mathematcs and Computer Scence, Unversty of Mannhem. Malobabc J, O'Connor N, Murphy N, and Marlow S., 004. Automatc Detecton and Extracton of Artfcal Text n Vdeo, WIAMIS 004-5th Internatonal Workshop on Image Analyss for Multmeda Interactve Servces, Lsbon, Portugal. Manohar Vasant, Padmanabhan Soundararajan, Matthew Boonstra, Harsh Raju, Dmtry B. Goldgof, Rangachar Kastur, John S. Garofolo, 006. Performance Evaluaton of Text Detecton and Trackng n Vdeo. Document Analyss Systems, Raner Lenhart and Axel Werncke, 00.Localzng and Segmentng Text n Images and Vdeos, IEEE Transactons on Crcuts and Systems for Vdeo Technology, vol., NO. 4 Sato T., Kanade T., E. Hughes, and M. Smth, 998. Vdeo OCR for Dgtal News Archves, IEEE Workshop on Content-Based Access of Image and Vdeo Databases(CAIVD'98), pp Sobottka K. and Bunke H., 999. Identfcaton of Text on Colored Book and Journal Covers, Internatonal Conference on Document Analyss and Recognton, Bangalore, Inda, pp Wolf Chrstan and Jean-Mchel Jolon, 004. Model based text detecton n mages and vdeos: a learnng approach. Techncal Report LIRIS-RR Laboratore d'informatque en Images et Systemes d'informaton, INSA de Lyon, France. March 9th. Wu W., D. Chen and J. Yang, Integratng, 005. Co- Tranng and Recognton for Text Detecton, Proceedngs of IEEE Internatonal Conference on Multmeda & Expo 005 (ICME 005), pp X Je, Xan-Sheng Hua, Xang-Rong Chen, 00. Lu Wenyn, HongJang Zhang, A Vdeo Text Detecton And Recognton System, IEEE Internatonal Conference on Multmeda and Expo ICME 00. Yan Hao, Y Zhang, Zengguang Hou, Mn Tan, 003. Automatc Text Detecton In Vdeo Frames Based on Bootstrap Artfcal Neural Network and CED. The -th Internatonal Conference n Central Europe on Computer Graphcs, Vsualzaton and Computer Vson 003. Ye Qxang, Qngmng Huang, Wen Gao, Debn Zhao, 005. Fast and robust text detecton n mages and vdeo frames. Image Vson Computng 3(6): Zhong Yu, HongJang Zhang, Anl K. Jan, 000. Automatc Capton Localzaton n Compressed Vdeo, IEEE Trans. Pattern Analyss Machne Intellgence, (4):