Tanner Graph Based Image Interpolation

Transcription

1 2010 Data Compression Conference Tanner Graph Based Image Interpolation Ruiqin Xiong and Wen Gao School of Electronic Engineering and Computer Science, Peking University, Beijing, China Abstract This paper interprets image interpolation as a channel decoding problem and proposes a tanner graph based interpolation framework, which regards each pixel in an image as a variable node and the local image structure around each pixel as a check node. The pixels available from low-resolution image are received whereas other missing pixels of highresolution image are erased, through an imaginary channel. Local image structures exhibited by the low-resolution image provide information on the joint distribution of pixels in a small neighborhood, and thus play the same role as parity symbols in the classic channel coding scenarios. We develop an efficient solution for the sum-product algorithm of belief propagation in this framework, based on a gaussian auto-regressive image model. Initial experiments show up to 3dB gain over other methods with the same image model. The proposed framework is flexible in message processing at each node and provides much room for incorporating more sophisticated image modelling techniques. I. INTRODUCTION Interpolation refers to the process of generating a faithful high-resolution signal from a low-resolution counterpart. The possibility of fulfilling this objective lies in the belief that missing data in the high-resolution signal is highly correlated with that in the low-resolution one. Conventional interpolation methods include linear, cubic and spline interpolations, etc., which regard ground truth signals to be continuous and smooth. These methods work well in scenarios where such assumption is valid. They are also commonly used for image magnification, mainly due to the simplicity in computation. However, magnifying images using such methods usually produces annoying artifacts, including blurred textures, zigzagging or ringing edges, etc. It is well recognized that most images contain structures that cannot be well represented by the continuous smooth signal model. The statistics in an image may vary dramatically from one region to another, or from one direction to others. For example, correlation of pixel values is high in smooth regions but is low in regions of complex textures; also, continuity in pixel values is preserved along edges but corrupted across object boundaries. Therefore, the key task of a good image interpolation algorithm is to adapt to the spatially varying statistics and fit missing pixels to the inherent local structures exhibited by the low-resolution image. With the ever-increasing processing capability of consumer devices, generating visually pleasing high-definition images from existing small images through advanced interpolation methods become desirable, and many adaptive algorithms were proposed. Edge-based approaches [1], [2], [3] employ source models emphasizing visual integrity of detected edges. Edge pixels are first detected and then missing pixels are fitted by mapping a small neighborhood of edge pixel to a best-fit ideal step edge. More general geometry-based methods [4], [5], [6] consider isophote (intensity level curve) contours, and treat interpolation as a variational problem. Constraints are introduced to minimize curvature of interpolated isophote through partial differential equations (PDE). These algorithms usually require explicit estimation of geometry /10 $ IEEE DOI /DCC

2 structures, such as direction or width of edges. However, this can be very difficult when irregular textures or noises are present. Instead of using geometry information, other methods exploit image structures through statistical models. Li and Orchard [7] estimate covariance matrix of a high-resolution image from its low-resolution counterpart and interpolate missing pixels using the estimated covariance. In [7], only four 8-connected neighbors along diagonal directions are utilized for each missing pixel. Zhang and Wu [8] further include four 4-connected neighbors and employ soft decision to jointly estimate missing pixels in an octagon sliding window. Inspired by the great success of codes on graphs in recent years, this paper interprets image interpolation as a channel decoding problem. Using the language of tanner graph [9], [10], we propose a tanner-graph based adaptive interpolation (TAI) framework, in which each pixel in the high-resolution image is regarded as a variable (or information) node and the local structure around each pixel is regarded as a check (or function) node. The pixels available in the low-resolution image are symbols received from an imaginary transmission channel whereas the missing pixels of the high-resolution image are symbols erased during the transmission. Although the low-resolution image provides channel observations only for a small fraction (e.g. 1/4 for 2x zooming) of all variable nodes, local structures in the lowresolution image provide extra information on the joint distribution of pixels in a small neighborhood, thus play the same role as parity symbols in the classic channel coding scenarios. To decode missing pixels, we develop a computation efficient algorithm for the message passing process, based on the sum-product algorithm on tanner graph and a gaussian autoregressive image model. Our approach is similar to the work [8] in jointly interpolating a group of pixels each time, but the proposed framework is general and flexible in the message processing of each node. This provides more room for incorporating more sophisticated message processing and image modelling techniques. The remainder of this paper is organized as follows. Section II describes the tannergraph based adaptive interpolation framework. Section III develops practical solution for the message passing algorithm based on gaussian autoregressive image model. Experimental results are reported in Section IV and Section V concludes the paper. II. TANNER GRAPH BASED ADAPTIVE IMAGE INTERPOLATION FRAMEWORK A. Tanner Graph for Image Interpolation For an image I of W H pixels, we define a tanner graph T (V, C, E, F) as follows. V { } are variable nodes (v-node), each corresponding to one pixel of I. The total number of variable nodes is V W H. C {c i } are check nodes (c-node), each representing the local image structure at a specific location of I. F {F i (V i )} are local functions formulating such image structures, with each F i associated with the c-node c i with the same subscript i and the argument V i representing a vector of v-nodes (i.e. pixels) involved in the local structure of c i. In later discussion, we may use the c-node notation c i to replace the subscript i in F i and V i without bring any confusion. E {e i (c,v)} are edges, each connecting a c-node c to a v-node f and only if s in the argument V c of F c. A c-node c and a v-node v are neighbors if (c,v) E. We use N(c) and N(v) to denote all the neighbors of c and v, respectively. Fig. 1 demonstrates the idea of tanner graph for image interpolation. Fig. 1(a) shows the graph on image pixel grid. Each gray circle is a v-node representing one 377

3 c va v H v G (a) vb v I vf c vc v D ve va vb vc vd ve Pr( va...) vf vg Pr( v...) E (b) vh vi Pr( vi...) Fig. 1: Tanner Graph for Image Interpolation. Left: variable and check nodes shown on image pixel grid. Right: variable and check nodes in the classic tanner graph representation. pixel and each square with round-corner and dashed line is a c-node describing the local image structure around a pixel location. As an example, the c-node c in Fig. 1(a) describes structure of the region formed by v-nodes v A through v I, which is centered at v I. Fig. 1(b) is the classic tanner graph representation, which shows the edges connecting node c with nodes v A through v I more clearly. In this example, the argument associated with node c is V c (v A,v B,...,v I ). In the tanner graph defined above, c-node c actually defines the joint probability distribution of V c. For the example in Fig. 1, F c (V c ) Pr(V c ) Pr(v A,v B,...,v I ). (1) The specific formulation of (1) is determined by 1) the image model adopted to describe the dependency characteristic of nearby pixels, and 2) the model parameters estimated by fitting the image model to the pixel values in a small neighborhood. There are a few facts we shall note. Firstly, as in tanner graph for channel codes such as LDPC codes, the number of c-nodes does not have to equal to that of v- nodes. Secondly, the neighborhood configuration of each c-node does not have to be the 3 3 block exemplified in Fig. 1. Instead, other shape and size can also be used and such configuration may vary from one c-node to another. This aspect is worth investigating, but is beyond the scope of this paper. B. Interpolation as Channel Decoding Problem Now we interpret image interpolation as a problem of channel decoding on tanner graph, as shown in 2. Suppose we received a low-resolution image I R, which was generated from a high-resolution image I through a resolution reduction process, and we want to recover the original image I with a fidelity as high as possible. The objective is to find for each pixel of I a value with maximum a posteriori probability (APP) given the observation I R. Obviously, this can be mapped to a channel decoding problem by regarding the resolution reduction process as a virtual channel. In the channel coding scenarios, recovering the original source symbol stream from an observed symbol stream possibly containing errors or erasures requires the 378

4 I Resolution Reduction Virtual Channel I R Channel Model Image Model Estimation Prv i I M ( R, ) C Fi( Vi) Tanner-Graph Based APP Decoding Î Fig. 2: Image interpolation as a channel decoding problem on tanner graph. following critical information: 1) channel model, i.e. knowledge on the statistical behavior of transmission channel, and 2) stream model, i.e. the structures or constraints in the transmitted symbol stream, usually presented by parity symbols generated through channel encoding. For the image interpolation scenario, the situation is slightly different in the two aspects. Firstly, the channel model is determined by how resolution reduction is performed. If the low-resolution image is obtained by direct subsampling, it forms a simple erasure channel with regular erasure pattern. In this case, the channel-derived probability of each variable node either has an impulse distribution or a uniform distribution. If anti-aliasing filtering is applied before subsampling, as done in many applications, it leads to a more complicated channel, in which output signal is convolution of input signal and the lowpass filter. Furthermore, noises introduced during image acquisition or compression can also be considered in this virtual channel model. For simplicity, however, we only consider direct subsampling in this paper, keeping in mind that the proposed framework can be applied to other more general situations by updating the virtual channel model. In all these cases, the conditional probability density of variable nodes given the channel observation can be regarded to be gaussian. Secondly, in the image interpolation scenario, we usually do not have any extra information such as parity symbols to help us recover the missing pixels. However, we do know that pixels in an image are not totally independent and random; instead, there always exists inherent structures, or say constraints, among pixels, which have clear physical meanings. These structures are implicitly contained in the lowresolution image but can be formulated and approximated by some well-established image models. Parameters of these models can be estimated from the observed I R. In other words, we can derive some extra constraints on nearby pixels, by matching the observation I R with some image models. These constraints play the same role as parity symbols in the channel coding scenarios. Fig. 2 is a diagram demonstrating the overall interpolation process. The lowresolution image I R is fed to two modules. The first module outputs an initial probability estimation Pr( I R, M C ) for each variable node, based on a virtual channel model M C and the observation I R. The second module estimates model parameters by fitting the observation I R to an image model M I and outputs the specific formulations of local functions F i (V i ) for each check node c i. Both are fed to the tanner-graph based APP decoder, which employ the message passing (also called belief propagation) algorithm to solve the pixel values with maximum a posterior probability Pr( I R, M C, M I ). 379

5 c v j i v c i j c j ca cb c j N ( vi )\{ cj } c c C D χ χ va vb vc vi (a) vd v N( c)\{ v} j i E Pr( vi IR,MC) Fig. 3: Message passing process on tanner graph. (a) Message from check node to variable node. (b) Messages from variable node to check node. (b) C. Message Passing Algorithm The message passing algorithm on tanner graph is briefly summarized here for the completeness of this paper. The v-nodes and c-nodes exchange messages iteratively, for a specified number of times or until a certain criterion for termination is reached. Let χ v (l) i c j ( ) and χ c (l) j ( ) denote the messages sent along the edge (c j, ), from node to node c j and from node c j to node at the l th iteration, respectively. Both messages represent the latest a posterior probability density of the message-sending node has learned from the graph so far. The initial messages from v-nodes are set to χ (0) c j ( ) Pr( I R, M C ). In subsequent message exchange iterations (l 1, 2,...), the messages from c-nodes to v-nodes are χ c (l) j ( ) F cj (V cj ) χ v (l 1) k c j (v k ), (2) { } v k N(c j )\{ } whereas the messages from v-nodes to c-nodes are χ v (l) i c j ( ) Pr( I R, M C ) χ (l) c k N( )\{c j } c k ( ). (3) This process is illustrated in Fig. 3. After the iterations are terminated, the ultimate estimated probability of each v-node is Pr( I R, M C, M I ) Pr( I R, M C ) χ (lmax) c k ( ) (4) c k N( ) (l max is l of the last iteration that (2) is calculated) and the recovered pixel value ˆ is ˆ argmax Pr( I R, M C, M I ). (5) III. GAUSSIAN AUTOREGRESSIVE MODEL BASED SOLUTION The message passing process on tanner graph involves sum and product of several probability density functions, which generally have very high computation complexity. In this section, we develop a practical algorithm for the message passing process, based on a piecewise gaussian autoregressive image model which has been widely used in literatures including [7] and [8]. 380

6 A. Gaussian Autoregressive Image Model Gaussian autoregressive (GAR) image model is commonly defined as I(i,j) α( i, j) I(i + i,j + j) + ε(i,j), (6) ( i, j) T i.e. each pixel is a linear combination of a few pixels in its neighborhood and a random perturbation. Here T is a spatial template defined the neighborhood that each pixel is correlated with, and ε(i,j) is a white noise process. The validity of this model hinges on a mechanism that adjusts the model parameters piecewisely to the local image structures [8]. At a specific location (i,j), the constraint formulated by (6) defines a c-node on the proposed tanner graph. For the convenience of later discussion, we call the v-node at the left side of (6) as head variable node while v-nodes at the right side of (6) as subordinate variable nodes, both of the corresponding c-node. For example, in Fig. 1(a), v I is the head v-node of c while v A,v B,...,v H are subordinate v-nodes of c. The image model describes the way to predict head v-node from subordinate v-nodes. As we would like to avoid two-dimensional location indices of nodes, we reformulate the autoregressive image model as follows. For any c-node c C, we enumerate the v-nodes in N(c) as v c,0,v c,1,v c,2,...,v c,nc, with v c,0 being the head v-node of c and n c N(c) 1. The image model (6) is redefined as v c,0 k1:n c α c,k v c,k + ε c. (7) Here {α c,k } n c k1 are model parameters and ε c is model error at the c-node c, respectively. For convenience of later development, we define α c,0 1. It is implied in (7) that model parameters can vary from one c-node to another. The noises ε c are independent and identically distributed: ε c N(0,σ 2 ε). With this model, the local function associated with c-node c can be formulated as V c (v c,0,v c,1,v c,2,...,v c,nc ) and ( 1 F c (V c ) exp 2πσ 2 ε v c,0 ) 2 α c,k v c,k /2σ 2 ε. (8) k1:n c B. Message Evolution Generally speaking, calculating sum and product of several probability density functions in (2) and (3) is computation intensive, which may make the message passing process infeasible. Fortunately, when we adopt the gaussian autoregressive image model, the computation in (2) and (3) can be greatly simplified without any loss, based on the following Lemmas. Lemma 1: Linear combination of independent gaussian random variables is still gaussian. If y i1:n a ix i and x i N(m i,σ 2 i),i 1, 2,...,n, then y N(m y,σ 2 y), with m y i1:n a im i and σ 2 y i1:n a2 iσ 2 i. Lemma 2: Product of gaussian density functions is still gaussian density function. If N(x,m,σ 2 ) denotes the density function of gaussian distribution N(m,σ 2 ), then N(x,m 1,σ 2 1)N(x,m 2,σ 2 2) N(x,m n,σ 2 n) N(x,m,σ 2 ) (9) 381

7 With notations η i 1/σ 2 i and η 1/σ 2, referred to as the reliability of distribution N(m i,σ 2 i) and N(m,σ 2 ) respectively, we have η η i, m ( ) ηi m i. (10) i1:n Note that the reliabilities from each density function are summed up and the means from each density function are linearly combined using a set of weights proportional to the corresponding reliabilities. Since we use gaussian-style channel model and gaussian image model, according to Lemma 1, we conclude that the a posterior probability of all v-nodes are always gaussian at all iterations of the message passing process. Therefore, at each step, we i1:n only need to record the mean and variance of each v-node. We use c j, σ 2,(l) η (l) c j and c j, σ 2,(l) c j, η (l) c j η c j, to represent the mean, variance and reliability of carried by the messages χ v (l) i c j ( ) and χ c (l) j ( ), respectively, during the l th iteration. We first consider the initialization step (l 0). For the erasure channel considered in this paper, the distribution of v-node available in the low-resolution image is initialized as IR,M C N(p i, 0), i.e. m (0) c j m (0) p i, σ 2,(0) c j σ 2,(0) 0 and +. Here p i denote the received pixel value. For other v-nodes η (0) c j η (0), we have IR,M C N(p any, + ) (actually a uniform distribution) i.e. σ 2,(0) σ 2,(0) +, η (0) c j η (0) 0 and m (0) c j m (0) c j p any. The value of p any is not important here. Now we consider the message processing at c-nodes. Each c-node c collects messages from all its neighboring v-nodes and then compute a message to send back for each of its neighboring v-node. The message from c to its head v-node v c,0 represents an estimation of v c,0 based on those subordinate v-nodes v c,k,k 1, 2,...,n c, according to the prediction structure at c formulated in the image model. According to Lemma 1, the calculation of (2) can be reduced to c v c,0 σ 2,(l) c v c,0 α c,k m v (l 1) c,k c, (11) k1:n c k1:n c α 2 c,kσ 2,(l 1) v c,k c + σ 2 ε, (12) On the other hand, the message from c to any of its subordinate v-nodes, v c,i,i > 0, represents an estimate of v c,i produced by inversing the prediction structure at c based on v c,0 and other subordinate v-nodes of c. Similarly, the calculation of (2) can be reduced to ( ) c v c,i σ 2,(l) c v c,i ( m (l 1) v c,0 c σ 2,(l 1) v c,0 c + k1:n c, k i k1:n c, k i α c,k m (l 1) v c,k c α 2 c,kσ 2,(l 1) v c,k c + σ 2 ε /α c,i, (13) ) /α 2 c,i, (14) Now we consider the message processing at v-nodes. Each v-node collects messages from all its neighboring c-nodes and also from the channel, and then compute 382

8 a message to send back for each of its neighboring c-node. The outgoing message records the most recent estimation of the a posterior probability of. According to Lemma 2, the calculation of (3) can be reduced to η (l) c j c j η (0) + η (0) m (0) + η c (l) k c k N( )\{c j } c k N( )\{c j }, (15) η (l) c k c k /η (l) c j, (16) After the iteration process is terminated, the final estimated a posterior probability for is determined by m (last) η (0) m (0) + η (l max) c k m (l max) /η (last), (17) η (last) η (0) + c k N( ) η (l max) c k c k N( ) c k. (18) The recovered pixel value ˆ is resolved by ˆ m (last). Now we briefly explain the above message evolution process. For a specific v-node to be interpolated, what we are ultimately interested in is the expectation of. The mean of the probability density function carried in each of those exchanged messages can be regarded as a prediction of a v-node, while the variance of the density function reflects the reliability of this prediction. In the current channel model of direct subsampling, channel does not provide any direct information for missing pixels. Instead, information of completely comes from the c-nodes connected with. Each c-node c k N( ) provides a prediction for in the form of m c (l) k. When is head v-node of c k, is estimated by using the prediction structure (7) at c k in a normal way, as formulated by (11). When is subordinate v-node of c k, is estimated by inversing the prediction structure (7) at c k, as formulated by (13). The essential issue is how to fuse all the predictions from various c-nodes. The equation (17) indicates that, if we define the inverse of variance of each prediction as its reliability, then the optimal estimation of can be obtained by linearly combining these predictions using a set of weights proportional to their reliabilities. By comparing (12) with (14), we can find that the prediction generated by inversely using the prediction structures usually has lower reliability, since α c,i is usually smaller than 1. C. Further Simplification To further reduce the computation complexity in the message propagation process, we only consider the white noise ε c in the GAR image model itself and ignore the noise possibly contained in intermediate estimation of each v-node, after all v-nodes are properly initialized by at least one iteration. To be specific, for the message processing step at c-nodes, we temporarily assume that the most-current prediction for each v- node obtained from previous iterations is accurate (unbiased and very reliable). This means σv c 2,(l) 0, (c,v) E,l > 1. Under this assumption, the calculation in (12) 383

9 and (14) can be simplified to σc v 2,(l) c,0 σ 2 ε and σc v 2,(l) c,i σ 2 ε/α 2 c,i,i > 0, or equivalently, η c v (l) c,i α 2 c,i, i 0. In this situation, the reliability of prediction passing from a c-node to a v-node is proportional to the square of parameter α on this directed edge. We rewrite α c,i as α c vc,i. Based on this, for pixels to be interpolated, the calculation in (16) is simplified to c j c k N( )\{c j } α 2 c k c k / c k N( )\{c j } α 2 c k. (19) In this simplified solution, only (11), (13) and (19) are evaluated iteratively. The final estimation is m (last) α 2 c k m (l max) /. (20) c k N( ) c k c k N( ) IV. EXPERIMENTAL RESULTS α 2 c k We have conducted some experiments to evaluate the performance of our proposed interpolation algorithm. As the first step, we adopt exactly the image model in [7] which only considers four 8-connected neighboring pixels. We test the TAI algorithm using two sets of model parameters. For the first set, all α parameters are equal (α c vc,i 1/4, c,i 1, 2, 3, 4). This is actually a bilinear model. For the second set, the α parameters are estimated by minimizing model errors as done in [7]. In both cases, we compare our method with the non-iterative interpolation method like [7] using the same image model parameters. Fig. 4: Test images used in this paper. These images are labeled as Monarch, Bike, Window, Parrot, Img1, Img2,..., Img 8. The test images used in this paper are shown in Fig. 4. For each test image, a half-resolution image is generated by direct-subsampling and then interpolated to the original resolution. To measure the quality of interpolated image, we calculate the PSNR values, which are reported in Table I. For bilinear image model, the TAI algorithm achieves dB gain (with an average of 1.39dB), while for GAR image model with parameters estimated in [7], the TAI algorithm achieves dB gain (with an average of 0.88dB). 384

10 TABLE I: PSNR of the interpolated images using different methods. Image Interpolation Methods Bilinear TAI+Bilinear Gain NEDI [7] TAI+NEDI Gain Monarch Bike Window Parrot Img Img Img Img Img Img Img Img Average V. CONCLUSION This paper interprets image interpolation as a channel coding problem. The contributions include 1) the proposal of a general image interpolation framework based on decoding on graphs, where many statistical image models can be incorporated, and 2) the development of a computation-efficient message passing algorithm, in which we derive a principle for combining the estimations from various check nodes. Just like that parity information is very critical for a transmitted stream to recover from channel errors, the validity of image model and the accuracy of model parameters play the key role in determining the performance of this scheme. This aspect will be investigated in our future work. ACKNOWLEDGMENT This work was supported in part by the National Basic Research Program of China (973 Program) under Grant 2009CB and the National Natural Science Foundation of China under Grant and REFERENCES [1] K. Jensen and D. Anastassiou, Subpixel edge localization and the interpolation of still images, IEEE Transactions on Image Processing, vol. 4, no. 3, pp , Mar [2] S. Carrato, G. Ramponi, and S. Marsi, A simple edge-sensitive image interpolation filter, International Conference on Image Processing 1996, vol. 3, pp , Sep [3] J. Allebach and P. W. Wong, Edge-directed interpolation, Proc. International Conference on Image Processing 1996, vol. 3, pp , Sep [4] B. Morse and D. Schwartzwald, Isophote-based interpolation, Proc. International Conference on Image Processing 1998, pp vol.3, Oct [5] H. Jiang and C. Moloney, A new direction adaptive scheme for image interpolation, Proc. International Conference on Image Processing 2002, vol. 3, pp , [6] Q. Wang and R. Ward, A new orientation-adaptive interpolation method, IEEE Transactions on Image Processing, vol. 16, no. 4, pp , April [7] X. Li and M. Orchard, New edge-directed interpolation, IEEE Transactions on Image Processing, vol. 10, no. 10, pp , Oct [8] X. Zhang and X. Wu, Image interpolation by adaptive 2-d autoregressive modeling and soft-decision estimation, IEEE Transactions on Image Processing, vol. 17, no. 6, pp , June [9] F. Kschischang, B. Frey, and H.-A. Loeliger, Factor graphs and the sum-product algorithm, IEEE Transactions on Information Theory, vol. 47, no. 2, pp , Feb [10] H.-A. Loeliger, An introduction to factor graphs, IEEE Signal Processing Magazine, vol. 21, no. 1, pp , Jan