A Cylindrical Surface Model to Rectify the Bound Document Image

Transcription

1 A Cylindrical Surace Model to Rectiy the Bound Document Image Huaigu Cao, Xiaoqing Ding, Changsong Liu Department o Electronic Engineering, Tsinghua University State Key Laboratory o Intelligent Technology and Systems Beijing, 84, P. R. China caohg@ocrserv.ee.tsinghua.edu.cn Abstract This article proposes a novel approach on how to rectiy the photo image o the bound document. The surace o the document is modeled by a cylindrical surace. By the geometry o camera image ormation, the equations using the cue o directrixes to map the points on the surace in the 3-D scene to the points on the image plane are achieved. Baselines o the horizontal text line are extracted as projections o directrixes to estimate the bending extent o the surace, and then the images are rectiied. The proposed method needs no auxiliary device. Experimental results are presented to demonstrate the easibility and the application o the method.. Introduction This article has been motivated by the advantage o using digital camera or document image capture. Compared with the scanner, the digital camera is quite easier to use, being able to capture images rom any viewpoint. However, the conveniences o using digital camera are accompanied with some image quality problems. When one takes a photo o a document page, one may oten ind that the resulting image is warped. This is generally caused by the existence o bookbinding, which prevents the page rom being lattened. One may imagine that, with a scanner, one can make the documents lat simply by pressing them intentionally on the glass board. An example o warping caused by bookbinding is shown in Figure. In this case, the process o document layout analysis and OCR will become complicated and unreliable, and it is hard to restore the position or the size o the characters accurately. There have been a number o literatures concerning the rectiication o this kind o warping. In [], a laser projector is used to project a D light network on the surace o the document, and then two dimensional distortions o the surace are corrected with a two pass mesh warping proposed by []. This method needs additional device and calibration. The method introduced in [3] is to get the depth o each point in the image by some stereo vision method, hence to make a depth image, and then unwarp the image according to the depth image. Although it seems capable o unwarping any type o image distortions, how to map the points on the rough, noisy surace deined by the depth image to the points on the plane is still a problem. In [4], the scanned images o bound books are rectiied by means o character segmentation. Characters in the shadow (along the binding line where the surace is curled) are located, estimating their orientations and rotating them to make them horizontal. Since the extent o bending is unknown by this method, it is not so accurate. In the resulting images, characters within the shadowy region still look narrower than characters in the rest region. In [5], a model o a combination o cylinder and plane is adopted to describe the surace o books, but how to estimate the parameters and make use o the model is unresolved, and it can only be applied to images scanned by the scanner. In [6], an applicable model is introduced to solve the more general case o document image warping. However, excessive approximation made the model inaccurate mathematically. In this article, an analytically accurate cylindrical surace model (CSM) is proposed to represent and rectiy the bound document image warping. This model can rectiy a single image, in no need o image sequence or any stereographic device. In section, the principle on how to use CSM to rectiy the image is discussed. In section 3, the detailed process o the rectiication is described. Some experimental results indicating the perormance o rectiication and some application in real OCR system are presented in section 4. The page is ended by section 5 with a summary on the proposed approach.. The cylindrical surace model With a scanner, one may press the document on the glass so that the surace o the document is partially lat, whereas while taking photos, documents are oten opened naturally, making the surace o the page close to a cylinder surace. The model described in [] representing the surace by a combination o a part o circular cylinder and a semi-plane is no longer suit or the surace o the document. Thereore, a general cylinder surace o a non-circular directrix is adopted here to represent the surace o the document. Actually, the directrix may have /3 $7. 3 IEEE

2 y = y z = D( Figure. A warped image by bookbinding. much more diverse shape. According to [7], the image ormation process o a lens system, such as the camera, can be approximately described by the perspective projection. In the case that the generatrix doesn t parallel the image plane, the mapping rom the points o the generatrix to the points on the image plane can be considered as a nonlinear projective transormation, which is diicult to deal with. Thereore, in this article, the rectiication problem is subject to the constraint that the generatrix o the cylinder parallels the image plane. The geometry o the image ormation process o the surace is demonstrated in igure. Assumed the generatrix o the cylinder parallels the image plane. On the image plane, take the point where the plane intersects with the optical axis as the origin, and select an X axis perpendicular to the generatrix, then a -D Cartesian coordinates rame is speciied. By obey the same X and Y directions, and taking the optical axis as Z axis, one can get the 3-D coordinates rame shown in the igure. From the geometry o the image ormation shown in igure, let s see how the warping is caused. Generally, the warping is the combined eect o two actors: () distance dierences, which make arther pitch o the suraces seem compacted compared with nearer ones, () oreshortening dierences, which is related to the orientation o the surace. Foreshortening also makes the pitch o surace seem compacted. A skewer surace means a more signiicant eect o oreshortening. I the surace is perpendicular to the optical axis, there is no oreshortening. Both o these two eects exist in the CSM. Distance dierences locally compact or dilate the image in both horizontal and vertical directions. While oreshortening only compact or dilate the image in horizontal direction. Firstly, consider an arbitrarily selected directrix on the surace: y = y l :. z = D( Now try to get the equation o the projection o l on the image plane, denoted by L. For the convenience, select the point r (, y, ) rom l as the reerence point, Figure. The process o image ormation o the document surace (a cylindrical surace). and its projection is R ( X, Y ) = (, y), where is the ocal distance. For any other point P(x, y, D(), assume that its image is P (X, Y). From the geometry, one may get X = x (), D( and Y = y (). D( Minus y rom both sides o (), one can get y Y y = [ D( ] (3). D( Notice that the depth dierence in one directrix is oten less than ~ cm, while the average depth o the surace is mach larger, typically 5~cm. In other words, D( << D (), where << means is ar less than. Using this assumption, rom () and (3), one can get: y Y = y + [ D( ] {[ D( ] + } y y + [ D( ] [ ] y = y D( (4). [ ] y D( = y D[ X ] [ ] Let D ( X ) = D[ X ]. Clearly, D (x ) is the curve D ( zoomed by / times, so they have identical shape. Notice that D( y / Y Y = =, and is a y / D( Y Y unction only related to X (see equation ()). So Y denote ρ ( X ) =. Practically, there may be some Y method to locate directrixes, so i given the coordinates (X, Y), one may ind the corresponding Y by tracing the projected image o the directrix that crosses (X, Y), then /3 $7. 3 IEEE

3 estimate the value o ρ(x ) or each X. Then equation (4) can be convert to y (5). L : Y = [ D ( ρ( X ) X )] Using the ollowing coordinates transormation X ' = ρ( X ) X (6) Y ' = Y to transorm the image rom the X-Y coordinates rame to a new X -Y coordinates rame. So the resulting projected directrix L satisies y (7). L ': Y ' = [ D ( X ')] The transormation by equation (6) eliminates the warping caused by distance dierences in x-direction. So this process can be called by the X-distance normalizing (XDN). More exited is that the projected image o any directrix in the -D image has a orm similar to the unction D, only dierent in scale or translation. From equation (7), one can get the ollowing result about the point in L : z z h z Y z D ( X ') = Y' = Y = H Y = Hρ( X ) y h y h Y h (8), where h is the actual height o the scene captured in the image, H is the height o the image, and z = is the approximate distance o the surace. For the mapping X ' = ρ( X ) X, denote its inverse by X = X(X '). Then z D ( X ' ) = Hρ[ X( X ' )] (9). h One can calculate the waveorm o D ( X ') using equation (9). Note X ' = [ / ] x, where the constant scale actor is the same as that od to D. Then, by make two coordinate transormations in the X and Y directions, respectively, one can get the image rectiied. The transormations are: t dd ( X" = arg{ + [ ] dx = X '} () t dx and Y " = ρ[ X( X ')] Y (). t In equation (), the integration dd ( + [ ] dx is the dx length o the curve D ( when x t. So this transormation means lattening the directrix. This process can be called by X-oreshortening eliminating (XFE). Similar to the X-distance normalizing, the transormation by equation () eliminates the warping o y-direction caused by distance dierences. This process can be called Y-distance normalizing (YDN). YDN maps all the points in a directrix to the same vertical ordinates in the X -Y coordinates rame. 3. Rectiying the image using CSM With the CSM, the rectiication can be perormed. One can irst locate the horizontal text lines in the image, and extract their baselines. Then use these lines to decide the let or right boundary lines o the column. A column is usually a rectangle region; hence one can speciy the Y direction (and thereore the X direction) o igure according to any o the two boundary lines. These baselines are taken as projected images o directrix to estimate the curve ρ(x ) and D ( X '), and construct a lattened image by the approach in section. Figure shows a document image o h6, the height o the real scene that seen rom the image, h 8cm, and the distance rom the lens to the surace o the document, z 59cm. The height o the image, H = 6pixels. In the ollowing o this section, let s take it as an example to see the key algorithms and processing in each steps o rectiying a warped image. 3.. Locating baselines o the text In most o the documents, the words in one text line are aligned along the baseline. I these baselines are located, they can quite accurately represent the directrixes. Firstly, Niblack s algorithm [8] is adopted to threshold the image. Next the mathematical morphological method proposed by [9] is applied to get the baseline o the text. In the baseline image, a connected component searching operation is executed to get all o the base lines o each text line, hence to locate the directrixes. The searching strategy is that, select a start point rom one end o a directrix, and search oreground points horizontally towards another end. In our experiments, each time rom a oreground pixel, search or the next oreground pixel horizontally within 4 pixels and e. The last text line o a paragraph usually stops hal way. However, the searching process, i it is rom the let to the right, sometimes does not stop, but traverses to an adjacent text line. To avoid this, the searching process is designed to run rom the right to the let. Figure 3. The base lines o text lines extracted rom image in igure /3 $7. 3 IEEE

4 The baselines o the text in igure are shown in igure 5. The let and right column border-lines well indicate the Y-direction. One may achieve it by straight-line itting. 3.. Estimating the curve ρ (X ) By the processing in section 3., one can get several sets o pixels. Each set represents a projected image o a directrix., Use each set to predict a unction values ρ (X ) or every X, then average all these unctions to get a smooth curve o ρ (X ). With ρ (X ), the transormation o the coordinates rame in equation (6) can be achieved. There may be no pixels in some set at certain coordinate X. To solve this problem, beore the above process, a interpolation is made using the neighboring pixels. Figure 4 shows the resulting ρ(x ) or the image in Figure. Figure 5 shows the image in the transormed coordinates rame. Notice that in Figure 4, the waveorms at the two ends o horizontal axis are diverse. The reason is that the waveorms o these intervals are not estimated by real data, but by interpolated data. The valid X value o text region is between 3 and 3 pixels. Within this interval, the variance o all the waveorms is slight, hence to ensure an accurate estimation o ρ (X ) Estimating the curve D ( X ') Like calculating the curve ρ (X ), all directrixes are used to estimate the curve D ( X '). Figure 6 shows the resulting curve D ( X '). The valid X value o text region is between 44 and 8 pixels. First, a 5-point averaging operator is applied to smooth the curve D ( X '), and then the irst order dierence [ D ( X ' + )] [ D ( X ')] is taken to approximate the derivative d [ D ( X ')]. An example o smoothing dx is shown in igure 7. You will see the irst order dierence ρ(x)-x -D * (X')-X' Figure 4. The curve ρ (X ) o the image in igure. ρ(x ) calculated by all directrixes. The averaged ρ (X ) o all the curves rom. Figure 6. The curve D ( X ') o the image in Figure. D ( X ') calculated by all directrixes. Averaged D ( X ') o all the curves in. Figure 5. XND using ρ (X ). The original image. The resulting image, visually slightly wider than the original one. Figure 7. Curve smoothing. A raction o row curve. The smoothed curve (by 5-point averaging ) /3 $7. 3 IEEE

5 (c) Figure 8. The rectiied image o igure. 89pix long pix long (c) 86pix long (a ) pix long (b ) pix long (c ) 99pix long Figure 9. -(c) Image patches o the word distribution extracted rom the original image. (a )-(c ) Corresponding image patches rom the rectiied image. is quite close to the derivative in the smoothed curve. Next, it is time to use the transormation by equations () and () to get the rectiied image. 4. Analyses and discussions on experimental results 4.. The inal result The rectiied image o the one in Figure is shown in igure 8. Notice that the text lines are quite straight and the widths o characters are uniorm. To see this clearly, we select the image pitches o the word distribution in three dierent locations o the image, and compare them between the original image and the rectiied one, as shown in igure 9. Binarised images and lengths o each word (in pixels) are also given in igure 9. These words get almost the same orientations and lengths ater rectiying. We have tried our method on around orty sample images o dierent languages (including English and Chinese) and dierent resolution ranges rom about 9dpi to dpi. Some o the results are shown in igure. In our experiment, most o the images get rectiied, except a ew on which our program cannot ind the Y-direction accurately. In these images, too many indented lines or lines that are the end o a paragraph in a single page result in the error o line-itting. When we use manually estimated Y-directions instead, each time the rectiying algorithm works well. We are now investigating methods (a ) (b ) (c ) Figure. Some o the results: -(c) are the original images, and (a )-(c ) are the corresponding results. on a better estimation o Y-direction. 4.. Applied to the pre-processing o OCR To see how our method can be used in some applications, we compare OCR tests between the original and the rectiied images. We take arbitrarily some samples in English and some in Chinese, and use ScanSot Omnipage Pro. and TH-OCR to make the OCR test o English and Chinese samples, respectively. Beore OCR, the unrectiied and rectiied images are zoomed in two times in both the width and the height by bilinear interpolation to achieve the proper resolution ( to 3 dpi) required by the OCR system, then they are binarised using Niblack s algorithm o the same window size (39 by 39). Then use OCR sotware to recognize the document images in the automatic mode o OCR sotware, i.e., automatically analyzing the layout, segmenting words, and recognizing them without any human intererence. A group o results o automatic layout analysis are shown in igure, where the one in the let shows the layout o the original image, and the one in the right shows the layout o the rectiied version o the let one. Without the process o rectiication, the layout analysis can hardly produce a correct result (see igure ), and the character segmentation is not reliable, producing disagreeable segmentation errors. A typical segmentation error is shown in igure, where two adjacent text lines cannot be split apart. This is due to the existence o warp in the image. While in the rectiied image, the horizontal text lines are straight. The recognition perormances are listed in table &. As can be seen rom these two tables, the rectiication process beore OCR can eectively reduce recognition errors /3 $7. 3 IEEE

6 Table. The recognition perormances o the original images and the rectiied images (o English text, using OmniPage). Images Number Correctly o recognized English English words words Recognition Error rate (%) (%) Not rectiied Rectiied rate Figure. The layout analysis result o the original image and the rectiied image using OmniPage. : the layout o the original image; : the Layout o the rectiied image. Figure. An example o a typical character segmentation error due to the existence o skew in the original image. The shadow region is an incorrect word region acquired by the segmentation Cost o computing time The time cost is mainly in the processes o inding the base lines and producing the new image. The estimation o ρ (X ) and D ( X ') do not spent much, because relatively ewer eature points are involved in the estimations o curves, compared with the ormer processes. Generally, it takes around 3 seconds to rectiy a h6 image (CPU: Intel.7GHz). This is usually much aster than the OCR process. 5. Summary In this article, a cylindrical surace model is proposed to describe the relationship between the surace o the bound document and its image ormed on the image plane. Through some image analysis and processing operations, our experiment results showed that it is possible to rectiy the images using the model. This method is then applied to the pre-processing o OCR, making remarkable improvement in the recognition quality. The proposed approach required that the generatrixes o the cylinder parallel the image plane. As or the photos taken reely rom any angle o view, the process o rectiication may be much harder. We will seek the solution o this problem in urther investigations. Table. The recognition perormances o the original images and the rectiied images (o Chinese text, using TH-OCR). Images Number Correctly RecognitionError o recognized rate (%) rate (%) Chinese Chinese characters characters Not rectiied Rectiied Acknowledgments This work was supported by 863 Hi-tech Plan (project AA48) & National Natural Science Foundation o China (project 645). Reerences [] A. Doncescu, A. Bouju, V. Quillet, Former books digital processing: image warping, in Proc. Workshop o Document Image Analysis, 5-9, 997. [] D. B. Smythe, A Two-Pass Mesh Warping Algorithm or Object Transormation and Image Interpolation, ILM Technical Memo #3, Computer Graphics Department, Lucasilm Ltd., 99. [3] M. S. Brown, W. B. Seales, Document restoration using 3D shape: a general deskewing algorithm or arbitrarily warped documents, in Proc. International Conerence on Computer Vision, July. [4] Z. Zhang, C. L. Tan, Restoration o images scanned rom thick bound documents, in Proc. 6th International Conerence on Document Analysis and Recognition,. [5] T. Kanungo, R. Haralick, I. Philips, Global and local document degradation models, in Proc. nd International Conerence on Document Analysis and Recognition, 993. [6] H. Cao, X. Ding, C. Liu, Rectiying the Bound Document Image Captured by the Camera: A Model Based Approach in Proc. 7th International Conerence on Document Analysis and Recognition, 3. [7] S. Russell and P. Norvig, Artiicial Intelligence: A Modern Approach, st Edition, pp Prentice Hall, 995. [8] W. Niblack, An Introduction to Digital Image Processing, Prentice Hall, Englewood Clis, 986. [9] A.K. Das, B. Chanda, A ast algorithm or skew detection o document images using morphology, in IJDAR, vol. 4, /3 $7. 3 IEEE