Inpainting of Occluded Regions in Handwritings

Transcription

1 Inpainting of Occluded Regions in Handwritings Fabian Hollaus and Robert Sablatnig Institute of Computer Aided Automation Computer Vision Lab Vienna University of Technology Abstract This paper deals with the reconstruction of handwritings that are partially overlapped by other texts. The presented system utilizes a high-order Markov Random Field in order to learn image models, which capture the statistics of handwritten strokes. Different handwriting models are used for the retouching of missing stroke regions in English words and ancient Greek handwritings. The Greek writings are overwritten by younger texts. The system makes use of multi-spectral images in order to separate the overwritings from the underwritings and to restore the older texts. 1 Introduction The analysis of historic and degraded documents can be facilitated by multi-spectral imaging [12]. Multi-spectral imaging has also proven its usefulness for the examination of palimpsests 1 [12], [6]. Palimpsests are historic manuscript pages that were reused. The original texts in such palimpsest were scraped off and overwritten by younger writings. The palimpsest folios, utilized in this work, are provided by the Archimedes Palimpsest Project [6] and contain erased underwritings, which are almost invisible under red light. Easton et al. [6] discovered that the ancient writings are most visible under UltraViolet (UV) illumination and proposed a pseudo-color approach, which visualizes the over- and underwritings. Figure 1 illustrates that such an pseudo-color image is obtained from a red light image in Figure 1 (a) and an UV image, given in Figure 1 (b). The resulting pseudo-color image is provided in Figure 1 (c). The readability of those texts is limited, since the ancient texts are partially occluded by younger ones. The aim of this work is to detect the occluded regions and to restore them automatically. The system consists of two stages: Firstly, the overwritings are detected by applying a binarization algorithm on red light images. Secondly, the retouching of the detected regions is fulfilled by applying an image inpainting algorithm on UV images. The term image inpainting describes the automated filling of a specific image region. The region is called inpainting region, or domain, and is provided in the form of a binary mask. Similar to [2], the presented system utilizes a Markov Random Field (MRF) for the training of an image model that captures the statistics of handwritings. The trained model is afterwards used for the recovery of occluded characters. The learning and inpainting sequence are based on the Fields of Experts (FoE) framework, which is suggested by Roth and Black in [13]. The paper is structured as follows. In the following section the basic ideas of other inpainting techniques are explained. In Section 3 an overview on the FoE framework is provided. Subsequently the This work was supported by the Austrian Science Foundation (FWF) under grant P Greek: palimpsestos - scraped again

2 (a) (b) (c) Figure 1. Multi-spectral imaging reveals the underwritings in leaf 40r. (a) Only the overwriting is recognizable under red light. (b) The underwriting is most visible under ultraviolet illumination. (c) The pseudo-color image exhibits the older and younger text. [6] detection of the overwritings is detailed in Section 4. An evaluation of the presented method is given in Section 5. Finally, the last section contains concluding remarks and an outlook. 2 Related Work The term digital image inpainting was coined by Bertalmio et al. in [1]. The authors suggest fulfilling the inpainting task by solving Partial Differential Equations (PDE). The basic idea of the algorithm is to smoothly complete isophote lines, arriving at the border of the inpainting region, from the outside of the mask into the inner region. Another PDE based algorithm is proposed by Chan et al. [3]: They suggest solving the inpainting problem with the minimization of an energy function, namely Total Variation (TV). Recently, a further TV based inpainting algorithm has been proposed by Dahl et al. [5]. Common to the aforesaid algorithms is that anisotropic diffusion is applied during the inpainting sequence, in order to maintain edges. The methods are only able to recover the geometry of an image, whereas texture is not preserved. Therefore, those methods are named geometric inpainting techniques. Typical for algorithms, belonging to this group, is also the fact that they only work well on narrow inpainting domains, whereas large regions are blurred out [9]. Recently other inpainting methods have been proposed, which overcome the disadvantages of geometric inpainting techniques. Those algorithms are called textural inpainting methods. A pioneering work is suggested by Criminisi et al. [4]. In this work the inpainting task is performed, by iteratively copying image patches from the outside of the mask into the inside. Komodakis and Tziritas [9] stress out that the algorithm by Criminisi et al. is based on heuristics and propose instead to formulate the inpainting task as a global optimization problem. Therefore they suggest the use of an MRF, which poses the filling task as a labeling problem. Contrary to geometrical methods, textural inpainting techniques do not blur out large and textured regions. Nonetheless, we have chosen a geometric inpainting approach for the recovery of handwritings, because images containing text are far less textured than natural images. Cao and Govindaraju [2] suggest an approach for the restoration of handwritings. The proposed method utilizes an patch-based and pairwise MRF for the binarization of a handwritten text and for the subsequent retouching of occluded regions. The FoE framework utilizes in contrast a pixel-based and high-order MRF for the reconstruction of gray-value images. 3 Fields of Experts An MRF can be formulated as an undirected graphical model. Such a graph G = (V, E) consists of nodes v V and edges e E that connect the nodes. With this graph G it is possible to model a d-dimensional random vector x. Each random variable x v is represented by a node v. The edges

3 describe the relationships between the nodes. A clique c C is a subset of neighboring nodes v. For each clique c there is a potential function f c existing that assigns a positive value to the clique. The joint distribution is defined as: p(x) = 1 Z f c (x (c) ) (1) c C where Z is a normalizing function. Depending on the maximal clique size there are two different MRF types existing: Pairwise MRFs and high-order MRFs. In a pairwise MRF the largest clique size is two. If the clique size is larger than two, the graphical model is called a high-order MRF. Model Definition Roth and Black propose a high-order MRF in [13]. Their model outperforms simple pairwise graphical models, because it is able to model long-range correlations in images, while pairwise models are only able to model dependencies between direct neighboring pixels. Another advantage over earlier MRFs is the fact that the parameters for the potential function are learnt from a training database. The potential functions that are used in the FoE framework are called experts and were introduced by Hinton in [8]. The experts are defined as: f(x (k) ) = f PoE (x (k) ; Θ) = N φ(j T i x (k) ; α i ), (2) i=1 where x (k) is the k-th patch of an image x. The image patch x (k) is projected onto a linear filter J i. α i is a scalar value, which is named expert parameter. There are N filters and expert parameters existing. In this work 3 3 filters were utilized and N was set to 8, as it is suggested by Roth and Black for this particular filter size. The expert function φ is a heavy tailed Student t-distribution. This distribution models the heavy tailed marginal distributions of natural images more accurately than a Gaussian distribution. The utilized expert function is defined by: φ(j T i x; α i ) = ( (JT i x) 2 ) αi (3) Since the potential functions are now defined, the generic MRF prior in Equation 1 can be reformulated into the FoE prior: p FoE (x; Θ) = 1 Z(Θ) K k=1 i=1 N φ(j T i x (k) ; α i ), (4) where K is the number of nodes - or respectively pixels - of an image x, Z(Θ) is a normalizing function for a concrete parameter set Θ and the other variables are defined as above. Learning The filters and the corresponding expert parameters are learned from a dataset that consists of image patches. Since the training of an FoE model is a highly time-consuming task, the model is trained on patches instead of whole images. This particular patch size was suggested by Roth and Black, in order to reduce the computational burden. The training algorithm is based on the Maximum Likelihood approach. Thus, the likelihood is maximized w.r.t. the parameter set Θ. The interested reader is referred to [13] for a detailed description of the training algorithm.

4 Inference The inference in the FoE framework is based on the Maximum A Posteriori approach. Since the computation of the exact inference is NP hard in general [13], an approximate technique is applied instead: In an iterative procedure, a gradient ascent is performed on the log of the posterior. The posterior contains no likelihood term, since it is assumed that no observation is made inside the inpainting region. Thus the posterior depends just on the FoE prior and is defined by: [ N ] x (t+1) = x (t) + ηm J i ψ (J i x (t) ; α i ), (5) i=1 where η is a user defined step size. The filter J i is obtained by mirroring J i around its center and t is the actual iteration of the inpainting process. A relatively large number of iterations is necessary to ensure convergence. In our implementation 2500 iterations are carried out and the step size η is set to 100. The function ψ is the derivate of the log of the Student t-distribution, defined in Equation 3. Applying the FoE Model on Handwritings The learning sequence for handwritings is similar to the original FoE algorithm, despite the fact that another training database is utilized. A further difference to the training of natural images is the selection of the training patches: The patches that are used for the training of a generic prior are randomly extracted from input images. Another sampling strategy is chosen for the generation of the handwritings training data: Each patch, which is sampled from a handwriting database, must contain a certain number of pixels that belong to a character stroke. Thus, the prior learns the statistics of the characters, in place of the background. The prior that captures the statistics of ancient writings was trained on patches, which were extracted from 40 UV images. A sampling in overwritten regions was avoided in order to learn solely the statistics of the underwritings. The inpainting of handwritings is also very similar to the restoration of the natural images, except one simple modification: In the standard implementation the inpainting region is initialized with zero, while in this work the domain is initialized with the dominant background color. It was found that this simple measurement leads to a reduction of the convergence time and a better avoidance of smearing artifacts. 4 Palimpsest Reconstruction Before the retouching of a palimpsest text may take place, it is necessary to identify the younger text. This text is generally in a better condition than the underwriting, but the contrast between the overwriting and the background is strongly varying. In order to extract a reliable inpainting mask, a recently suggested binarization algorithm by Su et al. [14] was implemented. The method is especially designed for the binarization of historical and degraded documents. Extraction of the Inpainting Mask The binarization algorithm starts with the construction of a contrast image. Each pixel in this image encodes the difference between the maximum and the minimum intensity value in a local neighborhood. The contrast is especially high at the border of character strokes. In order to suppress the background variation, the absolute difference is normalized and the contrast image is defined by: D(x, y) = f max(x, y) f min (x, y) f max (x, y) + f min (x, y) + ɛ (6)

5 where f max (x, y) and f min (x, y) denote the maximum and minimum gray value in a local neighborhood. In this work a window size of 3 3 is used. ɛ > 0 is an infinitely small value that avoids a division by zero. The contrast image D is afterwards binarized in order to detect high contrast pixels that are located at stroke boundaries. A pixel is marked as a high contrast pixel if its intensity D(x, y) exceeds a global threshold. This threshold is determined with Otsu s [11] thresholding approach. At this point, pixels at the stroke boundaries are marked as high contrast pixels. A pixel that is in the near of a stroke boundary is subsequently classified as a foreground pixel, if two requirements are met: Firstly, the pixel must have a certain number of high contrast pixels in its local neighborhood. In our implementation a pixel is marked as a foreground pixel if it has at least 4 high contrast pixels in a neighborhood window with a size of The second condition for a foreground classification is met, if the pixel intensity f(x, y) is smaller or equal than the mean intensity of the high contrast pixels (in the original image) in its neighborhood. The detected inpainting mask is afterwards slightly dilated. This post processing step is necessary, because otherwise pixels belonging to the border of the overwriting might not be covered by the mask. If such vestiges of the overwriting are touching the mask border, it is likely that the corruption is propagated into the inpainting domain. 5 Experiments In the subsequent experiments the Peak-Signal-to-Noise Ratio (PSNR) is used to measure the similarity between inpainted images and corresponding ground truth images. The PSNR is defined by: ( ) 255 P SNR = 20log 10, (7) MSE where MSE denotes the Mean Square Error (MSE) of two images. The PSNR values are given in decibels (db). Synthetic Data The first experiment is conducted on words, which consist of Latin letters. The utilized words are taken from the IAM handwriting database [10]. The performance of three different FoE models is evaluated: One prior captures the statistics of natural images. The remaining two priors are trained on diverse handwritings. The first handwriting model is trained on images that are randomly extracted from the database, which contains mainly cursive written words. The second prior is instead trained on uppercase letters, which have a different main orientation than the characters in the cursive words. Two test sets have been generated. The first set contains 100 randomly chosen words, whereas the second set is comprised of 100 words that are written in uppercase. The masks are generated from words, which have similar sizes as the input images. Table 1 shows the average of the PSNR values that are gained inside the inpainting mask. It can be seen that the handwritings models gain a higher performance if the statistics of the test images are similar to the statistics of the patches, which have been used in the learning sequence. Hence, the weakest performance is achieved by the model that is trained on natural images. The last column of Table 1 shows numerical results that are gained by a publicly available inpainting algorithm [5], which is based on TV. It is obvious that the FoE algorithm is more suitable for the handwriting recovery than the TV based method, which is designed for natural images. This can also be seen in Figure 2, where outputs of both handwriting priors are shown, along with a result that is generated by the TV based algorithm.

6 Test set \Training set Mixed letters Capitals Natural Images TV based algorithm [5] Lower case letters db db db db Upper case letters db db db db Table 1. Restoration performance of three different image priors and a TV inpainting technique. (a) (b) (c) (d) Figure 2. Text restoration. (a) Input image. (b) Image restored using a prior, which has been trained mainly on cursive words. The retouched image looks more natural than the one in (c), which is obtained with a prior that captures the statistics of uppercase letters. (d) Output of a TV inpainting algorithm. Palimpsest Reconstruction Four test panels have been extracted from different parchments of the Archimedes palimpsest. It turned out that the inpainting algorithm is limited to relatively small inpainting domains: The extend of the occluded palimpsest regions (at the original resolution of 700 Dots per Inch) is too large for a proper restoration, since character strokes are only restored at the border of the mask, whereas the mask centers are blurred out. Therefore, the test panels had to be downsized from pixels to pixels. A ground truth set was generated manually. The PSNR values of the restored panels are given in Table 2. The similarity inside the computed inpainting region is evaluated, like in the previous experiment. Additionally, the similarity of the entire image is provided, since the mask is not always covering the overwritings. The last two columns in the table show the numerical results that are gained by the TV based algorithm [5]. The PSNR values that are achieved by this algorithm are significantly smaller than the PSNR values, which are gained by the FoE method. Panel 99 verso is in a better condition than the other panels, which explains the relatively high PSNR values that are achieved inside the inpainting domain and on the entire image. The similarity values inside the mask regions are higher than the PSNR values, which are gained on synthetic data. Nevertheless, it has to be mentioned that the contrast between the foreground and the background in the palimpsest images is considerably smaller than in the synthetic images. Thus, the MSE is more bounded in the case of the ancient writings. Test set Image - FoE Mask region - FoE Image - TV Mask region - TV 40 recto db db db db 48 verso db db db db 58 verso db db db db 99 verso db db db db Table 2. Palimpsest recovery performance. In Figure 3 a part of the 48 verso panel is presented, along with various interim results. The final inpainting result is shown in Figure 3 (f) and the manually created ground truth image is given in Figure 3 (e). It can be seen that some gaps are successfully restored, while others are not altered adequately. A few parts of the overwriting are not covered by the mask. Those remaining parts lead to the introduction of not existing structures. One example for the introduction of artificial structures is illustrated in Figure 4. The overwriting is not fully detected, which can be seen by comparing the inpainting problem - given in Figure 4 (a) - with the ground truth image - shown in Figure 4 (b). The FoE prior propagates the surrounding dark regions into the unknown image regions, as it can be seen in Figure 4 (c). The unknown regions are

7 (a) (b) (c) (d) (e) (f) Figure 3. Portion of folio 48 verso. (a) Photography taken under red light. (b) The mask is generated from the former image. (c) UV image and superimposed inpainting mask. The mask is slightly dilated to prevent a propagation of remaining overwritings. (d) UV image. (e) Ground truth image. (f) Inpainting result. filled more heterogeneously, compared to the TV based inpainting result, which is shown in Figure 4 (d). This explains the relative weak performance - in terms of PSNR - of the TV based algorithm. (a) (b) (c) (d) Figure 4. Portion of folio 99 verso, illustrating the sensitivity to neighboring dark regions. (a) Input image and superimposed mask, which is not entirely covering the overwritten text. (b) Ground truth image. (c) Inpainted result produced by an FoE prior. (d) Result of the TV based algorithm [5]. 6 Conclusion This paper introduced an approach for the automatic inpainting of occluded handwriting regions. The FoE framework has been utilized in order to learn image models that capture the statistics of handwritten text. It was shown that such an image model learns the main orientation of character strokes, which is a major characteristic of handwritten text. The statistical inpainting method was also used for the reconstruction of underwritings in palimpsests. It turned out that the presented system has several weaknesses: Firstly, noise is often reinforced during the inpainting task, which leads to an introduction of not existing structures. Secondly, the inpainting approach is incapable of retouching the utilized palimpsest at the original resolution, since strokes belonging together are not connected. Therefore the images had to be downsized, which is inappropriate for a subsequent text analysis. Due to the described limitations, the developed technique is currently inapplicable for the automated retouching of palimpsest writings. It has to be mentioned that the performance may be increased by using 5 5 filters, as it is noted

8 in [13]. Furthermore, Heess et al. [7] have shown that the inpainting quality can be improved by using bimodal potential functions. These issues will be studied in our future work. It will also be investigated, whether a textural inpainting method is capable of overcoming the aforementioned drawbacks. References [1] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. In SIGGraph-2000, pages , [2] H. Cao and V. Govindaraju. Preprocessing of Low-Quality Handwritten Documents Using Markov Random Fields. PAMI, 31(7): , [3] T.F. Chan and J.H. Shen. Mathematical models for local deterministic inpaintings. Cam report 00-11, UCLA, [4] A. Criminisi, P. Perez, and K. Toyama. Region Filling and Object Removal by Exemplar-Based Image Inpainting. ICIP, 13(9): , [5] Joachim Dahl, Per Christian Hansen, Søren Holdt Jensen, and Tobias Lindstrøm Jensen. Algorithms and software for total variation image reconstruction via first-order methods. Numerical Algorithms, 53(1):67 92, [6] R.L. Easton, K.T. Knox, and W.A. Christens-Barry. Multispectral imaging of the Archimedes palimpsest. In 32nd Applied Imagery Pattern Recognition Workshop, pages , [7] N. Heess, C.K.I. Williams, and G.E. Hinton. Learning generative texture models with extended fields-of-experts. In British Machine Vision Conference, [8] G. Hinton. Products of experts. In International Conference on Artificial Neural Networks, volume 1, pages 1 6, [9] N. Komodakis and G. Tziritas. Image Completion Using Efficient Belief Propagation Via Priority Scheduling and Dynamic Pruning. ICIP, 16(11): , [10] U.-V. Marti and H. Bunke. The IAM-database: an English sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition, 5:39 46, [11] N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man and Cybernetics, 9(1):62 66, [12] K. Rapantzikos and C. Balas. Hyperspectral imaging: potential in non-destructive analysis of palimpsests. In ICIP, volume 2, pages , [13] S. Roth and M. J. Black. Fields of Experts: A Framework for Learning Image Priors. In CVPR, pages , [14] B. Su, S. Lu, and C. L. Tan. Binarization of historical document images using the local maximum and minimum. In Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, pages , 2010.