A Sparse Coding based Approach for the Resolution Enhancement and Restoration of Printed and Handwritten Textual Images Rim Walha, Fadoua Drira and Adel M. Alimi University of Sfax, ENI-Sfax, REGIM Sfax, Tunisia Email: {rim.walha, fadoua.drira, adel.alimi}@ieee.org Frank Lebourgeois and Christophe Garcia University of Lyon, INSA-Lyon, LIRIS Lyon, France Email: {franck.lebourgeois, christophe.garcia}@insa-lyon.fr Abstract Sparse coding has shown to be an effective technique in solving various reconstruction tasks such as denoising, inpainting, and resolution enhancement of natural images. In this paper, we explore the use of this technique specifically to deal with low-resolution and degraded textual images. Firstly, we propose a sparse coding based resolution enhancement approach to recover a textual image with higher resolution than the input low-resolution one. It is based on the use of multiple coupled dictionaries which are learned from a clustered training low-resolution/high-resolution patch-pair database. A reconstruction scheme is then suggested in order to adaptively select the appropriate dictionaries that are useful for better recovering each local patch. This approach can be applied for the magnification of both printed and handwritten characters. Secondly, we propose to integrate the magnification in a restoration framework specifically to denoise and reconstruct at the same time degraded characters. The performances of these propositions are evaluated on various types of degraded printed and handwritten textual images where loss of details and background noise exist. Promising results are achieved when compared with results of other existing approaches. Keywords-Resolution enhancement; sparse coding; printed and handwritten textual image; image restoration. I. INTRODUCTION The spatial resolution that refers to the pixel density in a given surface defines the quality of an image. In fact, a High- Resolution (HR) image means that the image is represented over a fine spatial grid carrying more information and thus fine details within it. For a Low-Resolution (LR) image, the spatial grid is relatively coarse and which in turn results in a loss of details. So, the spatial resolution is an important characteristic that influences the representation of details. If it has a minor impact on natural images, it is absolutely important to document images for which the loss of details can make a text unreadable by human and unable to recognize it by Optical Character Recognition systems. The need for high-resolution images continues to enlarge in various computer vision applications in order to achieve better performance in pattern recognition and image analysis. However, high quality images are not always available. This is mainly due to the inherent limitations of scanners used in early digitization projects. Poorly resolved images could also be resulted from a low resolution scanning process that is already chosen to satisfy the high speed access, the low-bandwidth transmission and even the reduction of the storage cost. So, the application of a resolution enhancement process (named also as image zooming, interpolation, up-scaling, magnification and enlargement) is needed in order to recover a HR image from an observed LR image. Such a process could resolve some imperfections of hardware devices and offer a better visualization and interpretation of LR images. The resolution enhancement has become a hot research topic for which various papers have bloomed into publications. Nevertheless, most of them have been devoted to the natural images with a very limited application on the textual ones. Thus, this work aims to tackle this lack by investigating the magnification of poorly resolved textual images. Recent studies based on sparse coding have attracted increasing interest due to its effectiveness in various reconstruction tasks like the resolution enhancement [1] [3]. The underlying idea of this technique is to represent each input data as a sparse linear combination of elements from a suitably chosen dictionary. Motivated by the key role of the dictionary in such a technique, we propose in this paper a new magnification approach based on the use of multiple dictionaries and adapted to the specificities of writing. Results achieved by the proposed approach are evaluated visually and quantitatively on printed and handwritten LR textual images and interesting results have been achieved. In addition, we propose a magnification based restoration approach for denoising and reconstructing degraded textual images. This approach is also applied to printed and handwritten textual images showing promising results. The rest of this paper is organized as follows: section 2 presents a brief review of related works. Details of the proposed multiple dictionaries based resolution enhancement approach are presented in section 3. Then, section 4 describes the proposed magnification based restoration approach. After that, experiments and comparative studies with results generated by other existing approaches are provided in section 5. Finally, conclusions and some perspectives are given in Section 6.
II. RELATED WORKS Resolution enhancement of a given low-resolution image is a long-studied task. Various interpolation based resolution enhancement approaches have been proposed in the literature. Non-adaptive interpolation methods apply a convolution on the image by using kernel functions such as linear, cubic or higher order. These methods are simple and fast, but they introduce artifacts such as blur and blocking artifacts along edges. A number of adaptive interpolation methods have been suggested in order to reduce the degradations generated by the previous methods. For instance, Li and Orchard [4] developed a New Edge-Directed Interpolation (NEDI) that is based on the local covariance and the geometric duality to estimate the HR covariance from the LR counterpart. A rule-based method was proposed by Battiato et al. [5] to detect edges and then adapt the interpolation. Resolution enhancement can be considered as an ill-posed inverse problem. A number of methods, such as [6,7], have been developed to regularize this problem with different priors. For example, Lukin et al. [6] proposed to use a bilateral Total Variation regularization. This category of approach is still suffering from non-natural artifacts and its performance decreases when the scaling-up factor increases. Various learning based magnification approaches have been developed to model the relationship between the input LR image and the output HR image from a training database. For instance, Sun et al. [8] proposed the gradient profile prior to capture local edges structure from training database, which allow to maintain the sharpness of the resulted image. Freeman et al. [9] designed a Markov Random Field model to predict the relationship between LR and HR image patches. In [10], the authors developed an exemplar-based approach which mapped blocks of the LR image into predefined HR blocks. Staelin et al. [11] designed neural networks with a spatial error feedback as interpolators to recover HR pixels. Luong and Philips [12] proposed a magnification approach which relies on the presence of repeating characters in the input image to reconstruct each LR character. This approach started with character segmentation which is not usually evident in LR textual images. More recently, learning approaches based on sparse coding theory have attracted increasing interest due to its effectiveness in various reconstruction tasks like the magnification [3,13,14]. The underlying idea is to suggest that an image patch could be sparsely represented from an appropriately chosen dictionary. In this paper, we propose a sparse coding based approach for the magnification of textual images using multiple coupled dictionaries. The following section details this proposition. III. THE PROPOSED SPARSE CODING BASED RESOLUTION ENHANCEMENT APPROACH In this section, we propose a sparse coding based resolution enhancement approach which learns multiple coupled Figure 1. Overview of the proposed sparse coding based resolution enhancement approach. dictionaries and reconstructs adaptively each local patch. The key idea of the proposed approach is firstly to find more appropriate dictionaries adapted to the particular specificities of characters and secondly to guide the reconstruction in selecting the appropriate dictionary to better recovering each local patch. An overview of the proposed approach is depicted in figure 1 and details are presented in the following subsections. A. Learning of Multiple Coupled Dictionaries Phase Sparse coding is an unsupervised method that is based on learning a succinct high-level representation (called dictionary) from the training data given only unlabeled inputs. Two coupled HR and LR dictionaries are generally learned for sparse coding based resolution enhancement task [2,3,15]. In order to find more appropriate dictionaries describing the specificities of writing, we propose to learn multiple coupled HR and LR dictionaries from clustered LR/HR patch-pairs database. These patch-pairs are extracted from printed character images automatically generated with a large variety of sizes, styles (italic, non-italic) and fonts (serif, sans-serif) [3]. An intelligent version of the K- means algorithm, referred as ik-means [16], is used to partition the training database into C clusters in an unsu-
pervised fashion because we haven t prior knowledge about the number of clusters. IK-means determines automatically the number of clusters C and initial cluster centers for K- Means using the Anomalous Pattern (AP) algorithm. Given the LR/HR patch-pairs of each cluster i, we turn to learn two coupled LR/HR dictionaries {D i l, Di h, i = 1..C} by using joint sparse coding method [2] whose goal is to have the same sparse representation for each LR/HR patchpair. The generation of the multiple coupled dictionaries enables the reconstruction phase which is described in the following subsection. B. Reconstruction Phase The reconstruction phase aims to recover a HR image from the input LR image. It is performed using the multiple learned coupled dictionaries which are generated from the previous phase. First, the input LR image is up-sampled by the bicubic interpolation. The interpolated image is considered as the LR image to be processed patch by patch from the upper-left corner with overlapping between adjacent patches. Second, the mean pixel value m is subtracted from each LR patch y j. Considering the centers of the C training clusters described in the previous phase, we propose to seek for each patch y j the K nearest centers. Specifically, Euclidean distances are calculated between the input patch y j and the cluster centers p i. Only the K nearest cluster are then chosen according to : min y j p i 2 2, i = 1..C (1) Via this strategy, we guide the reconstruction of y j to choose the K coupled LR/HR dictionaries {Dl k, Dk h, k = 1..K} learned from the selected K nearest clusters. Based on the sparse coding theory, the LR patch y j can be coded as a sparse linear combination of atoms from each LR dictionary Dl k. This leads to the generation of K sparse representations {α k, k = 1..K} for the same patch y j. This can be mathematically formulated as: (P 1 ) : min α k α k 1 s.t. y j D k l α k 2 2 ρ (2) where α k is the sparse representation of y j over Dl k and ρ defines an allowable reconstruction error. Several algorithms have been proposed in the literature to solve (P 1 ) [17,18]. In our implementation, the feature-sign search algorithm [17] is selected because of its efficiency and significant speedup. After that, we propose to select only the representation α that minimizes the local reconstruction error according to the appropriate LR dictionary D l : α = min α k{ y j D k l α k 2 2, k = 1..K} (3) The key idea is to guide the reconstruction to adaptively select the appropriate dictionary in order to better recover each local patch. The optimal solution α is then applied to generate a local HR patch x j from the corresponding HR dictionary based on: x j = D h α+m. Subsequently, the initial HR image X 0 is obtained by simply averaging the values in the overlapped regions to enforce compatibility between adjacent patches. In order to ensure consistency with the input LR image, we use the iterative Back-Projection method to apply a global reconstruction constraint by solving (4). In fact, such constraint supposes that an observed LR image Y consists of a blurred and downsampled version of a HR image X of the same scene: AHX = Y, where A and H represents respectively a downsampling operator and a blurring filter. X = argmin X X X 0 s.t. AHX = Y (4) Finally, the bilateral and shock filters are applied on the recovered image to better preserve edges and hence to further enhance the global reconstruction. IV. TOWARD TEXTUAL IMAGE RESTORATION AND RECONSTRUCTION THROUGH MAGNIFICATION The spatial resolution has a significant impact on the representation of details within an image. Degradations, such as background noise, could be clearly visible within an image when the spatial grid is fine. If such a grid becomes relatively coarse, degradations will be alleviated and may be disappeared. Then, we propose to use the resolution enhancement in order to reconstruct relevant details and to recover an image with a higher quality. Thus, the proposed restoration approach simply consists of two steps: First, down-sampling of the input degraded image. Second, up-sampling using the proposed magnification described in the above section. We note that the spatial resolution of the restored output image is the same as that of the degraded input image. The experimental study presented in section 5 shows that the proposed approach is able to both denoise and reconstruct printed and handwritten degraded textual images. V. EXPERIMENTS AND EVALUATIONS In this section, resolution enhancement and restoration using the proposed approaches are carried out on a variety of textual images. Results are evaluated both visually and quantitatively on the images quality and compared with other existing approaches. A. Resolution Enhancement of Textual Images The training database contains 124,000 patch-pairs of size 7 7. These patches are extracted from a variety of printed character images. By applying the intelligent clustering ikmeans [16], this database is divided into 13 clusters. Each dictionary learned from the clustered database contains 128 atoms. In the reconstruction phase, a sliding window of size 7 7 with 5 pixels overlap is selected to scan the input LR image. We note that only the luminance channel of the input image is processed because the human eye is more
Figure 2. Resolution enhancement of handwritten digit images. (a) The input low-resolution image. (b) Result of the bilinear interpolation. (c) Result of the bicubic interpolation. (d) Result of NEDI [4]. (e) Result of the proposed approach. sensitive to it. Color channels are predicted using the bicubic interpolation. In a first experiment, the proposed sparse coding based approach is applied for the magnification of handwritten digit images. For instance, two low-resolution images scanned at 72 dpi are enlarged and illustrated in figure 2(a). These images are altered by severe degradations. As shown in figure 2, the performance of the proposed approach proved to be better than bilinear, bicubic and New Edge-Directed Interpolation (NEDI) methods [4]. Results in figure 2(e) demonstrate that we are able to reconstruct handwritten characters although the training database contains patches extracted only from printed characters. A second experiment concerns the magnification of a synthetic printed textual image whose resolution is 72 dpi and which was generated by blurring and down-sampling of the HR ground truth image. In figure 3, enlarged regions are extracted from these images and from other ones recovered by bilinear, bicubic and NEDI [4] interpolation methods. Compared to the original LR images in figure 3(a), we can clearly notice improvements in visual quality in the results achieved by each resolution enhancement method involved in this study. On the other hand, we can observe that image recovered by the proposed method (figure 3(e)) is clearer and sharper at character edges. Thus, our resulting image has better visual quality than those produced by the interpolation based methods which generate significant artifacts appearing near character edges. Results are also evaluated quantitatively based on the widely used image quality measures including the Root Mean Square Error (RMSE), the Peak SNR (PSNR) and the Structural SIMilarity index (SSIM) [19]. Table 1 compares the measurement values of images recovered by the proposed method, bilinear, bicubic and NEDI [4] interpolation methods. According to this table, the proposed approach achieves the best results in terms of all these measurements. Figure 3. Visual comparison between word images selected from textual images recovered by different resolution enhancement approaches. (a) The input low-resolution image. (b) Result of bilinear interpolation. (c) Result of bicubic interpolation. (d) Result of NEDI [4]. (e) Result of the proposed approach. (f) The high-resolution ground truth image. Table I RMSE, PSNR AND SSIM RESULTS OF PRINTED TEXTUAL IMAGE RECOVERED BY DIFFERENT MAGNIFICATION METHODS. Measures Bilinear interpolation Bicubic interpolation NEDI [4] Proposed approach RMSE 45.614 42.739 52.980 33.783 PNSR 14.948 15.514 13.648 17.556 SSIM 0.701 0.746 0.661 0.855 B. Restoration and Reconstruction of Degraded Textual Images through Magnification In this experimental study, we demonstrate the usefulness of magnification for the restoration and reconstruction of textual images. We evaluated our proposition on printed and handwritten textual images which include severe degradations such as noise, blur and cuts. For instance, we give for lack of space in figure 4 selected regions of two characters from a degraded printed textual image and the processed versions with different restoration methods. In this experi-
Figure 4. Enlarged regions extracted from results of different restoration methods. (a) The input degraded image. (b) Tschumperlé et al. [21]. (c) Drira et al. [22]. (d) The bilateral filter. (e) Deriche et al. [20]. (f) The proposed approach. ment, we include the bilateral filter, shock filter proposed by Deriche et al. [20] and PDE-based methods such as the filters proposed by Tschumperlé et al. [21] and Drira et al. [22]. Compared to the input image (figure 4(a)), results illustrate that the different restoration methods involved in this experiment remove the noise from the background, but the degree of efficiency to restore the degraded characters is not the same. In fact, we can clearly notice in figure 4(f) significant improvements in the quality of characters that are reconstructed by the proposed magnification based restoration approach. The use of shock filter in our approach enables the enhancement of characters. Nevertheless, the direct application of such a filter on the tested image deteriorates characters as shown in figure 4(e). Figure 5 proves also the usefulness of the proposed magnification based approach when applied for the restoration of a handwritten character image. Promising results are achieved when compared to those of other existing restoration methods. VI. CONCLUSION In summary, we mention the main contributions of this work. A first contribution concerns the proposition of a sparse coding based resolution enhancement approach using multiple coupled dictionaries. This approach was successfully applied to both printed and handwritten low-resolution textual images. In addition, it was evaluated visually and quantitatively and compared to other existing magnification methods. As a second contribution of this work, the proposed resolution enhancement approach was applied for the denoising and reconstruction of degraded textual images. Experimental results demonstrate the usefulness of this proposition. An alternative extension to our work is to apply the proposed resolution enhancement approach to low-resolution texts embedded in video. REFERENCES [1] S. Yang, M. Wang, Y. Chen, and Y. Sun, Single-image superresolution reconstruction via learned geometric dictionaries and clustered sparse coding. IEEE Transactions on Image Processing, vol. 21, no. 9, pp. 4016 4028, 2012. [2] J. Yang, J. Wright, T. S. Huang, and Y. Ma, Image super-resolution via sparse representation, Trans. Img. Proc., vol. 19, no. 11, pp. 2861 2873, Nov. 2010. [3] R. Walha, F. Drira, F. Lebourgeois, and A. M. Alimi, Superresolution of single text image by sparse representation, in Proceeding of the workshop on Document Analysis and Recognition, ser. DAR 12. New York, NY, USA: ACM, 2012, pp. 22 29. [4] X. Li and M. T. Orchard, New edge-directed interpolation, Trans. Img. Proc., vol. 10, no. 10, pp. 1521 1527, Oct. 2001. [5] S. Battiato, G. Gallo, and F. Stanco, Smart interpolation by anisotropic diffusion. in ICIAP. IEEE Computer Society, 2003, pp. 572 577. [6] A. Lukin, A. Krylov, and A. Nasonov, Image interpolation by super-resolution, in 16th International Conference Graphicon 2006, Novosibirsk Akademgorodok, Russia, July 2006, pp. 239 242. [7] M. Ben-Ezra, Z. Lin, and B. Wilburn, Penrose pixels superresolution in the detector layout domain. in ICCV. IEEE, 2007, pp. 1 8. [8] J. Sun, Z. Xu, and H.-Y. Shum, Image super-resolution using gradient profile prior, in CVPR 2008, 24-26 June 2008, Anchorage, Alaska, USA. IEEE Computer Society, 2008. [9] W. T. Freeman, T. R. Jones, and E. C. Pasztor, Examplebased super-resolution, IEEE Comput. Graph. Appl., vol. 22, no. 2, pp. 56 65, Mar. 2002. [10] G. Dalley, W. T. Freeman, and J. Marks, Single-frame text super-resolution: a bayesian approach. in ICIP. IEEE, 2004, pp. 3295 3298.
Figure 5. An extract of handwritten character image (a) from the Gazette of Leyde database before and after restoration with the methods of respectively bilateral filtering(b), Goldstein et al. [23] (c), Tschumperlé et al. [21](d), Drira et al. [22] (e), Weickert [24](f), Perona-Malik [25](g), proposed approach(h). [11] C. Staelin, D. Greig, M. Fischer, and R. Maurer, Neural network image scaling using spatial errors, in HP Laboratories, 2003. [12] H. Q. Luong and W. Philips, Robust reconstruction of lowresolution document images by exploiting repetitive character behaviour, IJDAR, vol. 11, no. 1, pp. 39 51, Sep. 2008. [13] R. Walha, F. Drira, F. Lebourgeois, C. Garcia, and A. M. Alimi, Multiple learned dictionaries based clustered sparse coding for the super-resolution of single text image, in ICDAR, 2013, pp. 484 488. [14] R. Walha, F. Drira, F. Lebourgeois, C. Garcia, and A. M. Alimi, Single textual image super-resolution using multiple learned dictionaries based sparse coding, in ICIAP (2), 2013, pp. 439 448. [15] J. Yang, J. Wright, Y. Ma, and T. Huang, Image superresolution as sparse representation of raw image patches, CVPR, 2008. [16] B. Mirkin, Clustering for Data Mining: A Data Recovery Approach (Chapman & Hall/CRC Computer Science & Data Analysis), 1st ed. Chapman and Hall/CRC, Apr. 2005. [17] H. Lee, A. Battle, R. Raina, and A. Y. Ng, Efficient sparse coding algorithms, in In NIPS. NIPS, 2007, pp. 801 808. [18] M. Aharon, M. Elad, and A. Bruckstein, K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation, Signal Processing, IEEE Transactions on, vol. 54, no. 11, pp. 4311 4322, Nov. 2006. [19] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600 612, 2004. [20] P. Kornprobst, R. Deriche, and G. Aubert, Nonlinear operators in image restoration, Puerto Rico, 1997, pp. 325 331. [21] D. Tschumperle and R. Deriche, Vector-valued image regularization with pdes: A common framework for different applications, IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 4, pp. 506 517, Apr. 2005. [22] F. Drira, F. Lebourgeois, and H. Emptoz, A new pde-based approach for singularity-preserving regularization: application to degraded characters restoration, IJDAR, vol. 15, no. 3, pp. 183 212, 2012. [23] T. Goldstein and S. Osher, The split bregman method for l1-regularized problems, SIAM J. Img. Sci., vol. 2, no. 2, pp. 323 343, Apr. 2009. [24] J. Weickert, A review of nonlinear diffusion filtering, in Proceedings of the First International Conference on Scale- Space Theory in Computer Vision, ser. SCALE-SPACE 97. London, UK, UK: Springer-Verlag, 1997, pp. 3 28. [25] P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 7, pp. 629 639, Jul. 1990.