Edge-Based Method for Text Detection from Complex Document Images

Transcription

1 Edge-Based Method for Text Detection from Complex Document Images Matti Pietikäinen and Oleg Okun Machine Vision and Intelligent Systems Group Infotech Oulu and Department of Electrical Engineering, P.O.Box 4500 FIN University of Oulu, FINLAND Abstract Detection of text from documents in which text is embedded in complex colored and textured backgrounds is a very challenging problem. In this paper, we propose a simple texture-based approach based on edge information for this task. The performance of our method is compared to that obtained by a method based on the discrete cosine transform which was recently proposed by Zhong et al. [12] for text localization in compressed digital video. In our experiments, both methods performed about equally well for small-sized text, but our method was better in the case of large-sized text. The principal advantage of our approach is that in addition to the text detection problem, the same edge representation can also be used for other image interpretation tasks. 1 Introduction Documents in which text is embedded in complex colored and textured backgrounds are increasingly common today, for example, in magazines, advertisements and web pages. Robust detection of text from these documents is a challenging problem. The approaches developed for ordinary documents, such as binarization by adaptive thresholding, are not generally applicable, because it seems to be impossible to find an optimal threshold or thresholds to preserve meaningful information and to eliminate unnecessary one. Texture-based methods have been proposed as a solution to this problem, where text is assumed to represent a different texture than non-text (see a brief survey of these methods in [9]). With this approach, text is often considered as a periodic texture, because the characters on a text line form a more or less periodic structure in the horizontal direction, while the text lines, in turn, form a periodic structure in the vertical direction. Basically, there are two different approaches to texturebased text segmentation pixel-based and block-based. According to the first approach, pixels are segmented into text or non-text on the basis of texture information computed in the neighborhood of each pixel [2, 7]. With the second approach, the image is divided into non-overlapping or partly overlapping blocks and then each block is labeled based on texture information in it [1, 3, 5]. A problem with the most proposed approaches is that they are computationally too complex to be useful in practice. Especially, the pixel-based methods are very questionable in this respect. In this paper, we will demonstrate that simple texture measures based on edge information provide very useful information for text detection from complex document images. A method using edge information is developed and compared to an approach based on the discrete cosine transform [12], which was recently proposed for text localization from compressed video. 2 Texture information in document images The textures occurring in document images are quite different from the ordinary textures, because the edges of the characters are sharp and the background intensity is sometimes gradually changing. Because of this, it may not be reasonable to apply texture operators designed for grayscale images, but to use operators that measure the most relevant information from document textures instead. These operators should be rotation invariant, because the edges of the characters can have all orientations and the text lines can also be often in different directions. The methods to be chosen should also be invariant to gray level or color variations in the background. Recently, Ojala et al. [8] have shown that key information for texture discrimination is provided by two orthogonal features measuring local spatial pattern and contrast information, respectively. Their results also suggest that the densities of these local features computed over a re-

2 < O gion should be used for texture description. In contrast, the spatial frequency information often used in mainstream research does not appear to be so important. Following these lines, the edge-based texture measures appear to have many of the desired properties. The gradient magnitudes usually have high values in the edges of the characters, even when the text is embedded in pictures. The edges are also invariant with respect to the background variations and to image rotation. An additional advantage of the edge-based approach is that the same detected edges can also be used for other image interpretation tasks. 3 Our method Our method for text detection from complex page images consists of the following steps 1. Color-to-grayscale image conversion, if necessary. 2. Edge detection using a 3x3 Sobel operator, nonmaximum suppression, and thresholding. 3. Edge image partitioning into small non-overlapping blocks and computing an edge-based feature for each block. 4. Block classification either as text or as non-text based on the value of the edge-based feature. Since we are interested in edges, it is natural to detect them in a grayscale image. One of the simplest ways to convert a color image into grayscale is to pick the luminance component of a color model for which luminance and chrominance information are decorrelated (separated), while ignoring chrominance components. For example, the I-component of the HSI model or the Y-component of the YIQ model are proper choices. Edge detection with 3x3 Sobel masks is then performed, followed by elimination of non-maxima and thresholding of weak edges. A threshold for eliminating weak edges can be set by the user or it can be automatically computed based on the following formula "!$#&%'() "!$## +*-, #./ *-, # (1) where and / are the height and width of the image and "!$# and ( 0"!1# are x- and y-components of the gradient magnitude + "!$# 32 4!1#&%'( "!$# for a pixel at (i,j). A pixel is assumed to belong to an edge if 1) its gradient magnitude is larger than, and 2) it is a local maximum in a 3x3 neighborhood along either horizontal or vertical direction. After this, the edge image is divided into small nonoverlapping blocks of pixels, where 5 depends on the image resolution and size. In choosing 5 we suggest that a block on the paper should not occupy an area larger than 2x2mm. Hence, typical values for 5 can vary from 6 to 12. For each block, the feature 9 is computed according to one of the following formulas 9 9 < 9 ; D D > + 4!1# *? *B, #C (2) + "!$# + 4!1# *? *B, # + "!$# *E # F@A 4!1# *-, # + "!$# + 4!1# *? *B, # 5 (3) (4) 4!1# is the edge image (1 - edge, 0 - non-edge) and is the step function 6A*IH # KJ, if 6MLNH, O if 6MPNH. In Eq. 2, 9 means the number of edge pixels per block. In Eqs. 3 and 4, 9 is the average gradient magnitude per edge pixel and the average gradient magnitude per pixel, respectively. 9 defined above reflects the fact that the number of edge pixels and their gradient magnitudes are usually higher for text than for non-text blocks. Block classification is performed with a predefined threshold (blocks with 9QLR are assigned to text and others to non-text). Unfortunately, it is difficult to determine its value automatically. The histogram of 9 computed over the whole image is almost flat, except for a high peak corresponding to 9. We can only heuristically define a possible interval [9 S 9 ] for from this histogram (9 S and 9 correspond T to the first and last nonzero bins of the histogram, T respectively, excluding 9 0). According to our experiments, we prefer values close to the left boundary of this interval as values for (say, within [9 S 9 S %VUXW 9 ], where W 9 9 *-9 S and UMY [0.1,0.3]), because our goal is not to miss T any text. We assume that the large values of 9 are associated with text, whereas the small values may correspond either to text or to non-text. 4 Experiments We selected 25 color images of advertisements from the Oulu University database for testing [11]. The chosen images are titled P00055, P00197, P00209, P00214, P00222, P00229, P00237, P00247, P00250, P00251, P00283, (5)

3 < O 9 P00365, P00382, P00383, P00403, P02109, P02110, P02111, P02161, P02162, P02163, P02164, P02165, P02166, P All images are stored in JPEG format in the database. The method [12] based on the discrete cosine transform (DCT) features was used for comparison. This method basically performs a similar task as ours, but it also relies on text periodicity. The method [12] is indended for processing JPEGencoded images and does not need to fully decompress the image. The input consists of quantized DCT coefficients ( vary from 0 to 7) of 8x8 pixels blocks. The method begins with detecting blocks of high horizontal spatial intensity variation by computing the horizontal text energy for every block (6) where 2 and than a threshold, the block is a text candidate, where 6. for a given block is higher is 1.45 times the average energy for all blocks in the image, computed for 0 and.. Obtained text candidate blocks are then refined by applying morphological operations consisting of closing with a 1x3 structuring element, followed by opening with the same element. The number of text candidate blocks is further reduced in two additional steps, but their description in the paper is a little unclear and one threshold was not specified. Hence, we restricted ourselves to the implementation of the first two steps. In experiments, was set to 0 instead of 1.45 in order to get as good results of text detection as possible even at the cost of a higher rate of false positives. Other parameters remained unchanged. to 8 and 12, was automat- For our method, we set 5 ically determined according to Eq. 1, 9 Eq. 3, while varied from 9 DS to 9 steps of O, W 9 (9 was computed by with DS % W 9 DS and W 9 are specific for each image) so that we ran our method with 6 different thresholds for every image tested. It is worth to mention that our method allows us to set any 5, in contrast to [12], in which 5 is 8 or a multiple of 8. For each image and for each threshold value we computed 6> H$ (accuracy of text localization). It explicitly defines another important " H$ (misclassification of text as non-text) 100% - 6> H$. To count 68 H$, we employed a special tool GROTTO [10] to generate ground truths for the test images. Like classified images, the ground truths consist of blocks, each of which has a specific label. Block sizes for an image and for its ground truth are equal and there are only three different labels text, non-text, and mixture of text and nontext. Computing 6> H$ is thus a comparison of a particular image and its ground truth and counting the number of cases when a block in the image is text and a block in the ground truth is either text or mixture of text and non-text. Results obtained for all test images are collected in Fig. 1 showing 68 H$ under the best conditions vs. image number in a sequence of images. The words best conditions mean 0 for the comparative method and DS for our method. Under such thresholds, the maximum number of text blocks can be detected. In Fig. 1, Method 1 stands for the comparative method, while Method 2 and Method 3 correspond to our method with 5 equal to 8 or 12, respectively. Small black squares in the plots associate a particular image with its 6> H$. In the sequence of images the image P00055 corresponds to number 1 and the image P02167 to number 25, respectively. Text Rate under the Best Conditions Method 1 Method 2 Method Image Number Figure 1. Comparison of three tested methods under the best conditions. The results shown in Fig. 1 demonstrate that our method is much more accurate in text detection than the comparative method. For several tested images 68 H$ was low with all methods, but those images contained only largesized characters. Blocks classified as text were only detected on borders of such characters, while blocks in the interior were labeled as non-text. Since in a ground-truth all blocks inside bounding boxes of the large-sized characters are labeled as text, the number of errors is high. However, this is not the case for the small-sized characters, where detected text blocks do not form isolated blobs but large connected regions. One of the images converted to grayscale is shown in Fig. 2 (its name is P02167). It contains large-sized text embedded in a picture, white text on the black background, and a non-uniform background in many parts of the image. Results for different methods are presented in Figs The content of the text blocks detected is copied from the

4 Figure 2. Grayscale image. original image to the resulting images (another palette was however used for display). For our method, values of U are given in order to define, because 9 S %BUXW 9. Table 1 gives 6> H for the three methods under the best conditions. Table H$ for image P02167 under the best conditions. Threshold Text Rate Method % Method % Method % By looking at Figs. 3-7, one may see that both methods accurately found small-sized text but our method was more accurate in detecting large-sized text even when the block size was the same for both methods. Figs. 5-7 demonstrate the sensitivity of the results obtained with our method to the values of when 5 is fixed to H$ for chosen values of U varied from 73.39% (Fig. 5) to 72.84% (Fig. 7), that is, it dropped only a bit. By looking at Table 1, it is easy to see that our method still outperforms the method [12] even under different conditions than the best ones. The same is true for 5 8 as well. Since our test set is small, it is worth to give a theoretical estimate for the empirical accuracy attained. To this end, we Figure 3. Text detection results by the method [12]. adopted a well-known statistical approach whose usefulness for various tasks of document analysis and recognition was demonstrated in [6]. According to it, we need to find the confidence interval [ ] containing an empirical estimate S U of an unknown probability (U is the number of correctly classified text blocks in the image and is the total number of text blocks in the ground-truth) with a low error probability. Computing S can be considered as making times a Bernoulli trial and it therefore implies that we can use the following formula (see [4], p.829) for computing confidence interval bounds for in case of binomial distribution H * H % (7) with H S % 2 S, * S #, % % where is the error probability (1- is the probability with which the confidence interval about S includes ) and is the -quantile of the standard normal distribution. From Eq. 7 it is clear that H?* # and. We chose to be 0.05 and therefore H % #

5 Figure 4. Text detection results by our method (m8 and k0.15). Figure 6. Text detection results by our method (m12 and k0.15). Figure 5. Text detection results by our method (m12 and k0.1). Figure 7. Text detection results by our method (m12 and k0.2).

6 1.96. In our experiments, the confidence interval about S was quite narrow. For example, when we computed S for all tested images, it was (under the best conditions) from to for the comparative method, from to for our method with 5 8 and from to for our method with 5 12, respectively. Even though is not very high for any of the methods, it is still much higher for our method. Since we used a simple feature to extract text, this result can be considered as very promising. Moreover, even better results can be expected by applying contextual postprocessing further eliminating non-text data. 5 Conclusion A simple method for text extraction from complex documents using edge information was proposed. The method performed well in our experiments, demonstrating a good tolerance to color or gray level variations. The results indicate that the key textural information in text is provided by the average magnitude and density of the edges of characters in a region. The principal advantage of our approach is that in addition to text detection, the same edge representation can also be used for other image interpretation tasks. Acknowledgment This work was partly supported by the Academy of Finland. Mr. Yu Yan s help with programming and groundtruth generation and comments of Mr. Topi Mäenpää are gratefully acknowledged. References [1] L. Cinque, L. Lombardi, and G. Manzini. A multiresolution approach for page segmentation. Pattern Recognition Letters, 19(2) , [2] D.F. Dunn and N.E. Mathew. Extracting color halftones from printed documents using texture analysis. Pattern Recognition, 33(3) , [5] A.K. Jain and Y. Zhong. Page segmentation using texture analysis. Pattern Recognition, 29(5) , [6] M. Junker, R. Hoch, and D. Dengel. On the evaluation of document analysis components by recall, precision, and accuracy. In Proc. of the 5th Int. Conf. on Document Analysis and Recognition, Bangalore, India, pages , [7] M. Murguía. Document segmentation using texture variance and low resolution images. In Proc. of 1998 IEEE Southwest Symposium on Image Analysis and Interpretation, Tucson, Arizona, USA, pages , [8] T. Ojala, M. Pietikäinen, and T. Mäenpää. Gray scale and rotation invariant texture classification with local binary patterns. In Proc. of the 6th European Conference on Computer Vision, Dublin, Ireland, pages , [9] O. Okun and M. Pietikäinen. A survey of texturebased methods for document layout analysis. In Matti Pietikäinen, editor, Texture Analysis in Machine Vision, volume 40 of Machine Perception and Artificial Intelligence, pages World Scientific, [10] O. Okun, A. Vesanen, and M. Pietikäinen. Experimental tool for generating ground truths for skewed page images. In Paul B. Kantor, Daniel P. Lopresti, and Jiangying Zhou, editors, Proc. of SPIE on Document Recognition and Retrieval VIII, volume 4307, pages 22 33, [11] J. Sauvola and H. Kauniskangas. MediaTeam document database II. CD-ROM collection of document images. University of Oulu, Finland, [12] Y. Zhong, H. Zhang, and A.K. Jain. Automatic caption localization in compressed video. IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(4) , [3] K. Etemad, D. Doermann, and R. Chellappa. Multiscale segmentation of unstructured document pages using soft decision integration. IEEE Trans. on Pattern Analysis and Machine Intelligence, 19(1)92 96, [4] J.W. Harris and H. Stocker. Handbook of Mathematics and Computational Science. Springer-Verlag, New York, Inc., 1998.