A fast edge-oriented algorithm for image interpolation

Transcription

1 Image and Vision Computing 23 (2005) A fast edge-oriented algorithm for image interpolation Mei-Juan Chen a, *, Chin-Hui Huang a, Wen-Li Lee a,b a Department of Electrical Engineering, National Dong-Hwa University, 1, Sec. 2, Da-Hsueh Road, Shoufeng, Hualien, Taiwan b Department of Radiological Technology, Tzu Chi College of Technology, 880, Sec. 2, Chien-Kuo Road, Hualien, Taiwan Received 30 January 2004; received in revised form 28 April 2005; accepted 5 May 2005 Abstract This paper introduces a fast algorithm for image interpolation. By using this method, real-time enlargement of video images is accessible. The basic idea of the algorithm is to partition digital images into homogeneous and edge areas based on the analysis of the local structure on the images. In addition, in order to have better performance on interpolating images, specified algorithms are assigned to interpolate each classified areas, respectively. Experimental results show that the subjective quality of the interpolated images is substantially improved by using the proposed algorithm compared with that of using conventional interpolation algorithms. The computational complexity of the proposed algorithm is much lower than those of others mentioned in this paper. We also successfully implemented real-time enlargement of QCIF video sequences to CIF sequences with better image quality in low bit-rate environment. q 2005 Elsevier B.V. All rights reserved. Keywords: Interpolation; Zooming; Enlargement; Video image; Real-time; Edge 1. Introduction Multimedia communication plays an important role in modern networks. Image transfer is one of the main applications in multimedia communication. Efficient manipulation of large amount of image data in a system with limited bandwidth is considered as a key issue in this process. Digital images and video sequences essentially result in huge volume of image data. With huge volume of image data and limitation of network bandwidth, the quality of the transferred images is often unacceptable and poor performance, such as jitters and flickers can occur. The bandwidth for image transfer can be saved much more and image quality can be improved if the low-resolution video sequences is coded in encoder end and the low-resolution video sequences is afterward enlarged to high-resolution ones using interpolation techniques in decoder end. Image interpolation can be used in image enlargement and local image zooming. Several commonly used interpolation algorithms have been suggested, such as zero-order interpolation [1], linear interpolation [1] and * Tel.: C ; fax: C address: cmj@mail.ndhu.edu.tw (M.-J. Chen) /$ - see front matter q 2005 Elsevier B.V. All rights reserved. doi: /j.imavis cubic convolution interpolation [2]. However, image artifacts like blurring or zigzag on edge may occur when these interpolation schemes are used. In order to reduce the effect of image artifacts, other algorithms have been proposed [3 10]. The method proposed in [3] is based on directions in spatial domain. Interpolated values along different directions are calculated using directional weights. The weights are dependent on the variation in the directions. The algorithms in [4 5] are convolution-based interpolation methods. Pixels to be interpolated are classified into two decimations in which pixels are interpolated by different specified filters. A hybrid of convolution and median-based scheme splits the interpolation into two directional stages [6]. The soul of the algorithm in [7] is based on variation models with smoothing and orientation constraints. The algorithm in [8] is applied to an image after it was expanded using either bilinear or bicubic interpolation. Edges in the expanded image are obtained by use of canny edge detector. The values of pixels around the edges are modified to yield a crisper and less zigzagged picture. Another algorithm of adaptive re-sampling analyses the local structure of an image and applies a near optimal and least time-consuming re-sampling function that can preserve edge locations and their contrast [9]. Algorithms described above [3 9] have better performance than common interpolation algorithms introduced in [1 2]. Another robust algorithm, New Edge- Directed Interpolation (NEDI), performs good subjective

2 792 M.-J. Chen et al. / Image and Vision Computing 23 (2005) quality for image enlargement. The method initially estimates local covariance coefficients from a low-resolution image and then the estimated covariance are used to adapt the interpolation at a higher resolution based on the geometric duality between the low-resolution covariance and the high-resolution covariance. By comparing to other algorithms, the algorithm of covariance-based interpolation has higher computational complexity [10]. The basic concept of the algorithms mentioned above is to interpolate images using the feature of pixels. Determination of pixel feature by these methods needs higher computational complexity. Consequently, these algorithms are unable to achieve real-time (30 frames/sec) image enlargement in video sequences. To solve the problem, we propose a new interpolation algorithm, which provides better subjective quality and lower computational complexity for image enlargement. This paper is organised as follows. Section 1 describes the importance of interpolation to communication and gives a brief review of previous algorithms. Our proposed algorithm is introduced in Section 2. The experimental results are provided in Section 3. Conclusions of this paper are made in Section Proposed algorithm 2.1. General concept of the proposed algorithm The proposed algorithm is expected to achieve two major goals: lower computational complexity and better First step Low-resolution image Detection of homogenous area (a) Horizontal (b) Vertical (c) Diagonal Fig. 2. The differences of four directions (one for horizontal, one for vertical and two for diagonal). subjective quality. Hence, the new method can be practically applied to video sequences and videoconference. Our proposed method interpolates images based on analysing the local structure of the images. The original images are classified into two categories: homogenous areas and edge areas. The interpolation of pixels in the different classified areas is accomplished by using individual interpolation algorithm, respectively. The conceptual procedure of our proposed algorithm is illustrated as Fig. 1. Determination of pixels to be either homogenous pixels or edge pixels is based on a preset threshold value. First, the differences of pixels values along horizontal, vertical and diagonal directions are determined in a 3!3 window, respectively. Choice of the 3!3 window can significantly reduce computational loading and structural complexity of our proposed algorithm. We determine the differences of pixel values of the four directions one by one. Fig. 2 gives a schematic illustration to the four directions, one for horizontal, one for vertical and two for diagonal directions. If the difference is less than the preset threshold value, a non-interpolated pixel (white dot) will be classified as a homogenous pixel. Non-interpolated points in the homogenous areas are simply filled using bilinear interpolation. If the difference of pixel values is larger than the threshold value, the non-interpolated point is assigned as an edge pixel. After the first step, some non-interpolated pixels are remained as points in the edge areas. These pixels on edges will be left for further processing in the second step. An example of the result of the first step is demonstrated in Interpolate homogenous area Second step Interpolate edge pixels High-resolution image Fig. 1. The procedure of the proposed algorithm. Fig. 3. The result of the first step of the proposed algorithm for Lena image.

3 M.-J. Chen et al. / Image and Vision Computing 23 (2005) Original pixel Interpolated pixel Interpolated pixel/ Edge pixels Edge pixel (a) Case 1 (b) Case 2 (c) Case 3 Fig. 4. All neighbouring pixels around a non-interpolated pixel. Fig. 3. Some non-interpolated pixels are located on edges in the image. The second step of the proposed algorithm is to interpolate the edge pixels using all neighbouring pixels, which contain original pixels and interpolated pixels at the first step. Various types of neighbouring pixels are shown in Fig. 4, including black points, grey-points and spot points. For each pair of pixels, the smallest difference of pixel values implies the highest correlation between them. The edge pixel is interpolated along the direction of having minimum difference. If all of the spot points are interpolated pixels at the first step, the minimum difference is found on four directions across the white point (edge pixel). If a spot point is classified as an edge pixel, the minimum difference is obtained on the other directions excluding the edge pixel Bilinear interpolation algorithm for homogenous pixels We assume the X of low-resolution image with size H! W is enlarged to the Y of high-resolution image with size 2H!2W. The Y 2i,2j is zoomed from X i,j. We determine the homogenous pixels at Y 2iC1,2j, Y 2i,2jC1 and Y 2iC1,2jC1 using pixel difference criterion (Seen from Fig. 5). When these pixels are homogenous, we interpolate these pixels using bilinear interpolation algorithm. The procedure for determining pixel difference is described as follows: DY 1 Z jy 2i;2j KY 2iC2p;2jC2q j where p; q2fð0; 1Þ; ð1; 0Þg if DY 2!threshold and DY 3!threshold then DY min Z minfdy 2 ;DY 3 g if DY min Z DY 2 Y 2iC1;2jC1 ZðY 2iC2;2j CY 2i;2jC2 Þ=2 else Y 2iC1;2jC1 ZðY 2i;2j CY 2iC2;2jC2 Þ=2 else if DY 2! threshold then Y 2iC1;2jC1 Z ðy 2iC2;2j CY 2i;2jC2 Þ=2 else if DY 3!threshold then Y 2iC1;2jC1 ZðY 2i;2j CY 2iC2;2jC2 Þ=2 else DY 2 Z jy 2iC2;2j KY 2i;2jC2 j DY 3 Z jy 2i;2j KY 2iC2;2jC2 j Y 2iC1;2jC1 is edge pixel if DY 1! threshold then Y 2i,2j Y 2i+1,2j Y 2i+2,2j Y 2iCp;2jCq Z ðy 2i;2j CY 2iC2p;2jC2q Þ=2 Y 2i,2j+1 Y 2i+1,2j+1 else Y 2i,2j+2 Y 2i+2,2j+2 Y 2iCp;2jCq is edge pixel Fig. 5. Interpolation of homogenous pixels.

4 794 M.-J. Chen et al. / Image and Vision Computing 23 (2005) Y 2i 1 2j 1 Y 2i 2j 1 Y 2i+1 2j 1 diff 2 diff 3 diff 1 diff 4 Y 2i 1 2j Y 2i 2j Y 2i+1 2j Y 2i 1 2j+1 Y 2i 2j+1 Y 2i+1 2j+1 Fig. 6. Interpolation of edge pixels. Table 2 Comparison of computational complexity (Assume n pixels are interpolated) Add Sub Mul. Div. Shift Inverse Zero-order Bilinear 3n 3n 6n Bicubic 27n 45n 135n 9n NEDI Homogenous n n Edge 4n 1288n 4n Proposeogenous Hom- n n Edge n 4n n After determining the pixel differences, the noninterpolated pixels are classified as the edge pixels and then processed by the following procedures Edge-oriented adaptive interpolation for edge pixels Minimum difference of various directions can be found in either of the following cases. In case 1, all of the neighbouring pixels are already interpolated at the first step. The edge pixels are interpolated by the neighbouring pixels of four directions. The minimum difference among these four directions is evaluated first. The direction of the minimum difference indicates that the edge pixel is oriented to this direction as shown in Fig. 6. The procedure of the minimum difference algorithm is described as follows: diff 1 Z jy 2iÿ1;2j ÿ Y 2iC1;2j j diff min Z minfdiff k g; for 1%k%4 In case 2, some of the neighbouring pixels which are not interpolated at the first step are classified as the edge pixels. The minimum difference is found by using the remaining neighbouring pixels. In Fig. 6, we assume that two pixels Y 2iK1,2jC1 and Y 2i,2jC1 are not interpolated at the first step, and the minimum difference is found by skipping the two directions along with diff 3 and diff 4. The procedure is expressed by the following equations. diff 1 ZjY 2iÿ1;2j ÿ Y 2iC1;2j j diff 2 ZjY 2iÿ1;2jÿ1 ÿ Y 2iC1;2jC1 j diff min Z minfdiff k g; for 1%k%2 From the cases discussed above, the orientation of the diff min can be found. For example, if the diff min is found as diff 2 Z jy 2iÿ1;2jÿ1 ÿ Y 2iC1;2jC1 j diff 3 Z jy 2i;2jÿ1 ÿ Y 2i;2jC1 j diff 4 Z jy 2iC1;2jÿ1 ÿ Y 2iÿ1;2jC1 j (a) Original (b) Zero-order (c) Bilinear Table 1 Average PSNR of the homogenous areas using different algorithms for six images ZreoZorder (db) Bilinear (db) PSNR Bicubic (db) (d) Bicubic (e) NEDI (f) Proposed Fig. 7. The interpolation results by various algorithms for artificial image A (Interpolated from 64!64 to 256!256).

5 M.-J. Chen et al. / Image and Vision Computing 23 (2005) (a) Original (b) Zero-order (c) Bilinear (d) Bicubic (e) NEDI (f) Proposed Fig. 8. Subjective quality of a portion of Pepper image (Down-sampled from 512!512 to 128!128, then interpolated from 128!128 to 512!512). diff 1, Y 2iK1,2j correlating closely with Y 2iC1,2j,Y 2i,2j is therefore interpolated by using Y 2i;2j ZðY 2iÿ1;2j CY 2iC1;2j Þ=2 3. Experimental results In this section, we first compare subjective and objective qualities of Zero-order, Bilinear, Bicubic, NEDI and our proposed algorithms applied to interpolate still images. Six grey-level images (Pepper, Milkdrop, Tiffany, Comtal, Jet and Lena) and three colour images (Pepper, Jet and Lena) are tested. Zero-order, bilinear and bicubic interpolation algorithms are well-known linear interpolation methods. The Peak Signal to Noise Ratios (PSNRs) for these three algorithms performing in homogeneous areas are shown on table 1. The performance of using bilinear interpolation is Table 3 PSNR (db) comparison for grey level images Pepper Tiffany Comtal Jet Lena Milkdrop 256! !512 Zeroorder Bilinear Bicubic NEDI Proposed ! !512 Zeroorder Bilinear Bicubic NEDI Proposed similar with that of using bicubic interpolation. Moreover, a comparison of the computational complexity for the five methods shown in table 2 which indicates that the bicubic method has much greater computational complexity than the bilinear method. Therefore, the bilinear interpolation algorithm is heuristically employed in the first step of our proposed algorithm in order to reduce computational loading. Step edge testing of the five algorithms is performed as the enlarged A s in Fig. 7. Our proposed method has similar subjective quality compared to that of NEDI and performance superior to the rest of the three methods. Subjective qualities of colour images for these algorithms are demonstrated by portion of Pepper images in Fig. 8. The interpolated image of our proposed algorithm has smoother edges than the others. Table 3 shows the objective quality of grey-level images up-sampling from 256!256 to 512!512 and from 128!128 to 512!512. Table 4 shows the objective quality of up-sampled colour images from 256! 256 to 512!512 and 128!128 to 512!512. In general, our proposed method has better objective quality seconded to NEDI. Table 4 PSNR (db) comparison for colour images 256! ! ! !512 Pepper Jet Lena Pepper Jet Lena Zero-order Bilinear Bicubic NEDI Proposed

6 796 M.-J. Chen et al. / Image and Vision Computing 23 (2005) Table 5 Average CPU time (seconds/image) of various algorithms (Interpolated from QCIF to CIF) QCIF to CIF (sec) Zeroorder Bilinear Bicubic NEDI Proposed Table 6 Average PSNR (db) comparison for video interpolation Zero-order Bilinear Bicubic Proposed QCIF to CIF QCIF to 4CIF Table 5 shows average CPU time (seconds/image) of the five methods to interpolate images of QCIF to CIF. Table 2 depicts that NEDI has the highest computational complexity on edge areas compared to the other methods. Therefore, we conclude that NEDI is not suitable for interpolating images in real-time. By contrast, our proposed algorithm has relatively lower computational complexity and average CPU time, which is capable of being used for real-time interpolation of images. Preset threshold value for pixels grouping is one of factors that may influence quality performance and processing speed. The total number of edge pixels in an image is increased when the threshold is set to a value closing to zero. In the circumstance, the PSNR of the interpolated image is accordingly increased, but computational complexity is much higher due to the increased number of edge pixels. Table 2 depicts the fact that the computation loading for interpolating an edge pixel is heavier than a homogenous pixel. The threshold value is set to 25 for our experiments with concerning quality performance and processing speed. The experimental results of our tests show that the amount of PSNR dropping is less than 0.2 db compared to that threshold values are set to near zero. It is worthy to note that no decreasing of the subjective quality can be found in images interpolated by using threshold value set to 25. The major goals of the proposed algorithm are to achieve real-time interpolation and to have better subjective quality. We conduct an experiment to observe the effect of transmitting a sequence of down-sampled video with different bit-rates in H.263C and compare the visual quality of interpolated CIF video with that of CIF video. TMN8 is employed as a rate control of our scheme. The down-sampled CIF (to QCIF) and original CIF are transmitted with the same channel bit-rate for each time. The bit rates are set to 16, 24, 32, 64, 96, 128, 300 K and 1.5 M bits/s, respectively. Frame-rate is 30 FPS (Frame per (a) 35 Dancer PSNR(dB) CIF Proposed 16k 24k 32k 64k 96k 128k 300k 1500K Bit-rate (Bits/s) (b) Encoded Dancer 16k 24k 32k 64k 96k 128k 300k 1500k Bit-rate (Bits/s) CIF QCIF Fig. 9. Performance comparison of interpolated CIF and CIF at 30 FPS for Dancer sequence. (a) Comparison of PSNR for CIF and interpolated CIF. (b) Number of encoded frames.

7 M.-J. Chen et al. / Image and Vision Computing 23 (2005) (a) Original CIF (b) QCIF (c) Decoded QCIF (d) Decoded CIF (e) Interpolated CIF from decoded QCIF Fig. 10. Subjective quality of Dancer sequence Frame#54. Second). Quantisation parameter is tuned to 13. A total of 151 frames are processed. In decoder end, the downsampled CIF is interpolated to CIF using our proposed algorithm. PSNRs of interpolated CIF and CIF are compared in the experiment. The calculation of PSNR is based on original CIF. If there is a skipped frame in the interpolated CIF or CIF, we use repeated frame to calculate PSNR. Table 6 shows the average objective quality of 4 different video sequences being interpolated from QCIF to CIF and CIF to 4CIF in size. We can observe that our proposed algorithm has better objective quality than other methods. (a) Original CIF (b) QCIF (c) Decoded QCIF (d) Decoded CIF (e) Interpolated CIF from decoded QCIF Fig. 11. Subjective quality of Silent sequence Frame#25.

8 798 M.-J. Chen et al. / Image and Vision Computing 23 (2005) Two video sequences, Dancer and Silent, are tested. The Dancer sequence is regarded as a high-motion sequence. The PSNRs for different bit-rates are shown in Fig. 9(a). Fig. 9(b) illustrates the number of encoded frames with varying bit-rates. The interpolated CIF performs better objective quality in low bit-rates because the interpolated CIF has fewer skipped frames than CIF. Figs. 10 and 11 show the subjective quality of interpolated CIF and CIF. The background and skirt are apparently blocky in Fig. 10(d) and relatively smoother in Fig. 10(e). The hand and face are affected by blocky effect in Fig. 11(d) and the hand and face look much smoother in Fig. 11(e). According to the experimental results, we know that the interpolated CIF has better performance than CIF in limited bandwidth. The super performance especially appears in high-motion portions of images. 4. Conclusions The proposed algorithm successfully achieves the goals of real-time interpolation and good subjective quality. We suggest that the algorithm can be applied to videoconference or video images based on the nature of low computational complexity and good image quality. Image transforming from QCIF to CIF size becomes practically accessible by applying our method. The image quality will be substantially promoted in very low bit-rate video or bandwidth-limited videoconference. Acknowledgements This research was supported by Leadtek Research Inc. under the contract 91A0034. References [1] W.K. Pratt, Digital Image Processing, Wiley, Toronto, Ont., Canada, [2] R.G. Keys, Cubic convolution interpolation for digital image processing, IEEE Transactions on Acoustics, Speech, Signal Processing 29 (6) (1981) [3] S.D. Bayraker, R.M. Mersereau, A new method for directional image interpolation, IEEE International Conference on Acoustics, Speech, and Signal Processing 4 (1995) [4] L. Khriji, M. Gabbouj, Directional-vector rational filters for color image interpolation Proceedings of the Tenth International Conference on Microelectronics (1998), pp [5] C. Lee, B. Zeng, A novel interpolation scheme for rectangularly subsampled images, International Conference on Image Processing 3 (1999) [6] B. Zeng, M.S. Fu, C.C. Chuang, New interleaved hierarchical interpolation with median-based interpolators for progressive image transmission, Signal Processing 81 (2001) [7] H. Jiang, C. Moloney, A new direction adaptive scheme for image interpolation, International Conference on Image Processing 3 (2002) [8] H. Shi, R. Ward, Canny edge based image expansion, International Symposium on Circuits and Systems 1 (2002) [9] A.M. Darwish, M.S. Bedair, S.I. Shaheen, Adaptive resampling algorithm for image zooming, IEE Proceedings-Vision Image and Signal Processing 144 (4) (1997) [10] X. Li, M.T. Orchard, New edge-directed interpolation, IEEE Transactions on Image Processing 10 (10) (2001)