Separation of Overlapping Text from Graphics

Transcription

1 Separation of Overlapping Text from Graphics Ruini Cao, Chew Lim Tan School of Computing, National University of Singapore 3 Science Drive 2, Singapore {caorn, tancl}@comp.nus.edu.sg Abstract The separation of overlapping text from graphics is a challenging problem in document image analysis. This paper proposes a specific method for detecting and extracting characters that are touching graphics. It is based on the observation that the constituent strokes of characters are usually short segments in comparison with those of graphics. It combines line continuation with the feature line width to decompose and reconstruct segments underlying the region of intersection. Experimental results showed that the proposed method improved the percentage of correctly detected text as well as the accuracy of character recognition significantly. 1. Introduction The separation of overlapping text from graphics is a challenging problem in document image analysis [1-2]. This problem is found in many applications, including forms processing, maps interpretation and engineering drawings interpretation, where text and graphics are processed in fundamentally different ways. Fletcher and Kasturi [3] developed an algorithm for text string separation from mixed text/graphics image. Taking account of curving labeled road names in the map, Tan and Ng [4] developed a system using the pyramid to extract text strings. Both methods, however, assume that the text does not touch or overlap with graphics. In forms processing, the problem of overlapping text has been dealt with reasonably well [5-8]. The reason is that forms contain only straight lines and the problem is much simplified. However, for engineer drawings [9-11], especially maps [12], the problem is much more complex since there are more types of data, various types of lines, possible curvature and even branching of graphics. The segmentation of overlapping text and graphics is in fact a chicken-and-egg problem. On the one hand, the purpose of segmentation is character recognition. On the other hand, correct segmentation may require the recognition of the characters. A recent paper [13] presented a method for cooperative processing of text and line-art in topographic maps and gave promising experimental results. Yet, the method assumed that all the text should be detected and grouped prior to segmentation. The method as well as some other methods in the literature searched for graphics-connected characters in the neighboring areas of detected isolated characters. In this paper, we present an improved method for detecting, extracting and grouping characters, including characters connected to graphics. The improvement is in that it can detect the strings whose constituent characters are all connected to graphics without the risk of introducing too many false alarms. The damages to the text image are also reduced by appropriate reconstruction of the strokes at the intersections. 2. The proposed method 2.1. Preprocessing and initial classification The preprocessing includes sub-layer separation, solid graphical components removal and dashed lines removal. We extract the black layer in the following way: if the r, g, b values of one pixel are all less than a predefined threshold, then the pixel is considered black. The solid graphical components are removed by a morphological method [9]. Dashed lines are removed by detecting repeated bars with similar orientation and size [18]. After the preprocessing, we perform initial classification as described in [4]. In summary, connected components are generated first. Then, we use a size filter to roughly classify the components into text and graphics. Large components are identified as graphics and small components as text. Then, we get the minimum size S min and the maximum size S max of the text components to be used as refined size filters in the subsequent steps Detection and reconstruction of touching characters In the initial classification, text connected to graphics will be incorrectly classified as graphics since the touching text and graphics form a whole large connected component. We need to further search the graphics image

2 for these touching characters. It is observed that constituent strokes of characters are usually small in comparison with those of graphics. Therefore, so long as we can decompose the graphics image with touching characters into appropriate segments in such a way that each segment is either a textual stroke or a graphical line, we can then use a size filter to separate them from each other. We perform the decomposition process after thinning [16] since intersection points can be easily detected in the thinned graphics image. Intersection points are found at pixel locations where there are 3 or more neighbors. We first break the graphics image into small segments at the intersection points and then merge those segments that connected to the same junction by line continuation based on similar slopes of the adjoining segments within a thresholded difference. The slope of each segment is estimated by line fitting [17]. After decomposition, we use the size filters obtained in the initial classification to separate textual strokes from graphical segments. We extract all the segments less than S max and together with the intersection for which at least one segment connected to it has been restored. Then, we generate new connected components from the resultant image and remove those components smaller than S min with some tolerance. Figure 1 gives one example. CENTRAL CAVENAGH (d) Figure 1. Detection of touching characters: the original image; the thinned graphics image with intersection points grayed; combination of isolated characters with touching characters; (d) OCR results Removal of large graphical segments may damage the touching characters in the overlapping area. The reason is that some parts of removed segments possibly belong to both graphics and text. The damages may affect the OCR results severely. Figure 2 gives such an example. We can see the damaged touching character E is mis-recognized as F, and L is mis-recognized as I. To achieve better recognition results, we should restore the damaged characters. KFNG IFF RD KG JAVA Figure 2. Detection of touching characters: the original binary image; detected constituent strokes of touching characters; part of OCR results It is observed that the line is usually thicker in the overlapping part (Figure 2a). To reconstruct the damaged characters, we can restore the thicker parts of the removed segments. We choose the average width of the longest line segment in the image as the threshold. First, we pick the longest line segment in the thinned graphics image and represent it as a series of pixels: {P 1 (x 1, y 1 ), P 2 (x 2, y 2 ),, P n (x n,y n )}, where P i-1, P i+1 are the two neighbors of P i, and P 1, P n are endpoints. The line normal direction, which is perpendicular to the line slope direction, at pixel P i (x i, y i ) can be estimated as the cross product of the left 1-slope and the right 1-slope of this pixel: Ni = ( pi 1 pi ) ( pi + 1 pi ) (1) The line width W i is roughly the distance between the two boundary points along the normal direction in the original image (Figure 3). For each pixel at the removed segments, if the line width at this pixel is larger than the average line width, then add it onto the text image. Figure 4 shows the reconstructed touching characters of Figure Characters clustering We assume that the gap between (denoted as T w ) is larger than the gaps within (denoted as T c ). We perform a dilation operation on the character images with a square structuring element larger than T c and smaller than T w. The characters of each word will form a connected component in this way. The coarse text-box can be drawn by finding the top, bottom, leftmost and rightmost side of each component in the resultant image. This clustering method is efficient in comparison with the Hough Transformation based method in [3]. It also has

3 the advantage of the method in [4], that is, it can group curving labeled road names. One disadvantage is that it may group two that are too close incorrectly if the assumption given at the beginning of this sub-section is violated. W i P i-1 - P i N i Figure 3. Estimation of line normal direction and line width at each pixel, here P i-1, P i, and P i+1 are on the thinned line segments, W i is calculated on the thick line images KENG LEE RD KG JAVA Figure 4. The reconstructed touching characters (thinned) with isolated characters (thick); part of OCR results 3. Experiments and discussions P i+1 - P i We scanned 24 large maps at 300dpi and saved them in raw TIFF format. We cut out the neighboring area of touching text and obtained about 137 small images for the experiments. Table 1 shows the statistic data of the testing images. We can see that the touching of and graphics was serious (touching : 39.7%; totally touching : 6.9%). Here totally touching means all the characters within the word are connected to graphics. Table 1. Statistic Data of the Testing Images Touching Totally touching Total # of # % # % Total # of Touching characters characters # % The well-known IR standard measures, precision and recall, were used to measure the performance of the proposed method. Precision and recall are defined as follows: Precision = # of correctly detected / # of detected, Recall = # of correctly detected / # of in the image. If all the constituent characters of a word are detected, then the word is considered as correctly detected. Table 2 shows the experimental results of the proposed method. The results from initial classification are also shown to give some comparison. As we can see in Table 2, the proposed method detected almost all of the text in the test images. It improved the recall greatly (60.3% to 99.4%). At the same time, it did not decrease the precision (87.5% to 87.1%). The increase of false alarms was mainly due to small bars connected to graphics, some of which were dashed lines and some road line segments isolated by occlusion. Figure 5a shows part of the detection results for one of the large map images. Detected are shown within their bounding boxes. Figure 5b shows the corresponding extracted text image. Table 2. Text Detection Evaluation # of # of correctly false Precision detected alarms Recall Before % 60.3% After % 99.4% The extracted were also tested using an OCR system. The OCR system we used was the commercial system bundled with the HP scanner. The system can only recognize upright characters. Therefore, we rotated the characters into upright position before we conducted OCR experiments. Table 3 shows the testing results. As we can see, the recognition accuracy of both isolated characters and touching characters improved significantly after text extraction (71.7% to 94.3%, 30.2% to 64.8% respectively). Though our segmentation method only restores the touching characters and has no effect on the isolated characters, the recognition of the restored touching characters may have provided context information to the OCR system to improve the recognition of the nearby isolated characters. There are several difficult cases for recognizing touching text. As we explained earlier, the proposed method merges line segments at the intersections by line continuation. However, the outlines of the buildings are usually polygon contours. Therefore, when text touches these outlines at the adjoining points, the line segments may not be merged correctly. As a result, there are still some short graphical segments remaining within the text boxes. These segments usually cause confusion. Besides, some strokes of touching characters and lines coincide perfectly. The touching parts have the same line width as

4 the free lines. In this case, the touching parts of the characters cannot be restored. This also results in failure of recognition of the touching characters. Figure 6 shows two of such unsuccessful examples. Table 3. Text Extraction Evaluation by OCR Accuracy of Accuracy of Average Isolated Touching Accuracy Characters Characters Before 71.7% 30.2% 54.7% After 94.3% 64.8% 82.2% 4. Conclusions and future work A method of detecting and extracting text connected to graphics has been proposed in this paper. The proposed method combines line continuation with line width to interpret connection of text and graphics. Experiments showed that the proposed method improved the percentage of correctly detected text as well as the accuracy of OCR significantly. The commercial OCR system we used can only recognize upright characters, whereas, the road names can be in any orientation. Further, the characters within one word can also be in different orientations. Therefore, when we did the experiments, we rotated the characters into upright positions manually. In the near future, we hope to develop a method to rotate the characters upright automatically. The accuracy of recognizing touching text is still far too low to be usable. The integration of segmentation and recognition may be one promising approach to separate the touching characters completely from graphics. 5. References [1] G. Nagy, Twenty years of document image analysis in PAMI, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 1, pp , January 2000 [2] D. S. Doermann, An introduction to vectorization and segmentation, in Graphics Recognition: Algorithms and Systems, K. Tombre and A. K. Chhabra (eds.), Lecture Notes in Computer Science 1389, Springer, pp. 1 8, 1998 [3] L. A. Fletcher and R. Kasturi, A robust algorithm for text string separation from mixed text/graphics images, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 10, No. 6, pp , November 1988 [4] C. L. Tan and P. O. Ng, Text extraction using pyramid, Pattern Recognition, Vol. 31, No. 1, pp , 1998 [5] D. Wang and S. N. Srihari, Analysis of form images, in Document Image Analysis, H. Bunke, P. S. P. Wang, H. Baird (eds.), World Scientific, pp , 1994 [6] S. Naoi, Y. Hotta, M. Yabuki, and A. Asakawa, Global interpolation in the segmentation of handwritten characters overlapping a border, Proceeding of 1st IEEE International Conference on Image Processing, pp , 1994 [7] J. Yoo, M. Kim, S. Y. Han, and Y. Kwon, Line removal and restoration of handwritten characters on the form documents, Proceeding of 4th International Conference on Document Analysis and Recognition, pp , 1997 [8] K. Lee, H. Byun, and Y. Lee, Robust reconstruction of damaged character images on the form documents. In Graphics Recognition: Algorithms and Systems, K. Tombre and A. K. Chhabra (eds.), Lecture Notes in Computer Science 1389, Springer, pp , 1998 [9] R. Kasturi, S. T. Bow, W. El-Masri, J. Shah, J. R. Gattiker, and U. B. Mokate, A system for interpretation of line drawings, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12, No. 10, pp , October 1990 [10] D. Dori and Liu W., Vector-based segmentation of text connected to graphics in engineering drawings, in Advances in Structural and Syntactical Pattern Recognition, P. Perner, P. Wang, A. Rosenfeld (eds.), Springer, pp , 1996 [11] Z. Lu, Detection of text regions from digital engineering drawings, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 4, pp , April 1998 [12] H. Luo, G. Agam, and I. Dinstein, Directional mathematical morphology approach for line thinning and extraction of character strings from maps and line drawings, Proceeding of 3rd International Conference on Document Analysis and Recognition, pp , 1995 [13] L. Li, G. Nagy, A. Samal, S. Seth, Y. Xu, Cooperative text and line-art extraction from a topographic map, Proceedings of 5th International Conference on Document Analysis and Recognition, pp , 1999 [14] D. Dori, Liu W. and M. Peleg, How to win a dashed line detection contest, in Graphics Recognition: methods and Applications, R. Kasturi and K. Tombre (eds.), Lecture Notes in Computer Science 1072, Springer, pp , 1996 [15] R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision, MIT Press and McGraw-Hill, 1995 [16] B. K. Jang and R. T. Chin, One-Pass Parallel Thinning: Analysis, Properties, and Quantitative Evaluation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 11, pp , November 1992 [17] T. Pavlidis, Algorithms for graphics and image processing, Computer Science Press, 1982 [18] R. Cao and C. L. Tan, Text/graphics separation in maps, Proceedings of 4th IAPR International Workshop on Graphics Recognition, Ontario, Canada, September 2001

5 Figure 5. Part of for one of the large map images: text detection results; text extraction results ITE BaLestier tcampus 1) f4nder iedevatopryentu AVFNIlF Figure 6. Some unsuccessful examples of OCR: the original images; the extracted text images; the OCR results