Solving Optimization Problems That Employ Structural Similarity As The Fidelity Measure

Transcription

1 Solving Optimization Problems That Employ Structural Similarity As The Fidelity Measure Daniel Otero and Edward R. Vrscay Department of Applied Mathematics, University of Waterloo, Waterloo, Ontario, Canada Abstract Many tasks in image processing are carried out by solving appropriate optimization problems. As is well known, the square of the Euclidian distance is widely used as a fitting term, even though it has been shown not to be the best choice in terms of quantifying visual quality. To overcome this problem, a number of papers have eaed the use of the Structural Similarity Inde Measure (SSIM) as a fidelity term. In this paper, we propose a general framework for solving optimization problems in which the SSIM is employed as a fidelity measure. Within the contet of quasi-conve optimization, an algorithm is also introduced in order to solve such optimization problems. Keywords: Structural similarity, l 1 norm constrained optimization, total variation, quasi-conve optimization, deblurring, zoog 1. Introduction Over the years, in the field of Image Quality Assessment (IQA), many metrics have been proposed to model the Human Visual System (HVS), including the Mean Opinion Score (MOS), the Universal Quality Inde (UQI) [1], the Perception-based Measure (PBM) [2], among others. More recently, Wang et al. introduced in [3] the Structural Similarity Inde Measure (SSIM), a visual metric that has been the focus of considerable research [4], [5], [6], [7], [8]. As opposed to error-based metrics such as Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR), SSIM relies on the assumption that the HVS has evolved to perceive errors as changes in structural information, hence quantifying these changes should correlate better with our perception of what we consider good visually. Indeed, in [3] and other works, it has been shown that the SSIM outperforms other measures of visual quality, including MSE. Many tasks in image processing are carried out by solving appropriate optimization problems. In the majority of these cases, the squared Euclidean distance is employed as a fidelity term. In light of our earlier comments, it is appealing to consider the SSIM as a fidelity term in such optimization problems, in an effort to enhance the visual quality of the solutions. That being said, the SSIM is not a conve function and therefore more difficult to handle mathematically, if not algorithmically. Notwithstanding these obstacles, optimization problems that include the SSIM as a fitting term have already been addressed. For instance, in [4] the authors find the best approimation coefficients in the SSIM sense when an orthogonal transformation is used (e.g., Discrete Cosine Transform (DCT), Fourier, etc.). Finding the best SSIM approimation coefficients is equivalent to imizing the function T (Φ(), y) = 1 SSIM(Φ(), y), (1) where Φ( ) is an orthonormal matri, and y the signal being approimated. The function T (, y), which will be used in this paper, may be considered as a measure of the visual dissimilarity between and y. Based on the the results given in [4], in [6] Rehman et al. introduce the SSIM version of the the image restoration problem proposed by Elad et al. in [9]. Furthermore, in [6] the authors also introduce a super-resolution algorithm also based on the SSIM to recover from a given low resolution image its high resolution version. Another interesting application for reconstruction and denoising of images is introduced by Channappayya et al. in [5]. In this case, the authors define the statistical SSIM inde (statssim), which is an etension of the SSIM for wide sense stationary random processes. The non-conve nature of the statssim is overcome by reformulating its maimization as a quasi-conve optimization problem, which is solved using the bisection method [10], [5]. Nevertheless, it is not mentioned that the SSIM under certain conditions is a quasi-conve function. More imaging techniques based on the SSIM can also be found in [7], [11], [8]. In these works, optimization of rate distortion, video coding and image classification are eplored using the SSIM as a measure of performance. Given that maimizing SSIM(, y) is equivalent to imizing T (, y) in Eq. (1), it can be shown that all the applications mentioned above can be stated as the following optimization problem, {T (Φ(), y) + λh()}, (2) where Φ is usually a linear transformation, h() is a regularizing term, and λ its corresponding regularization parameter. The constrained version of this problem is given by T (Φ(), y) (3) subject to h() λ.

2 The methods used for solving (2) and (3) may converge to the same solution; if this is the case, it may be said that both optimization problems are equivalent. Moreover, since many SSIM-based imaging tasks can be carried out by solving problems (2) and (3), we consider these equations as the general framework of what we call SSIM-based optimization. For this reason, we think that it is a better approach to develop algorithms for solving (2) and (3) rather than developing methods for addressing particular applications which is the tendency found in the literature. In this paper, we focus our attention on the quasi-conve approach; that is, we show how problem (3) can be solved. To do so, we show under what conditions the SSIM is quasiconve. In addition, we consider the cases where h() is conve and not necessarily differentiable. Furthermore, we show that (3) can still be solved if T (Φ(), y) is subjected to a set of conve constraints [10]. Applications such as Total Variation and l 1 norm constrained optimization are discussed, as well as comparisons between the l 2 and SSIM approaches. 2. Structural Similarity (SSIM) 2.1 Definition SSIM provides a measure of visual closeness between an image and a distorted or corrupted version of it. Since it is assumed that the distortionless image is always available, the SSIM is considered a full-reference measure of IQA [3]. The definition of SSIM is based on two assumptions: (1) images are highly structured that is, piels tend to be correlated, specially if they are spatially close and (2), that the HVS is adapted to etract structural information. For these reasons, SSIM measures similarity by quantifying changes in perceived structural information. This measurement is done by comparing luance, contrast and structure of the two images being compared. Changes in luance are measured by quantifying relative changes in the means of the images; contrast comparison is carried out by measuring relative variance; finally, structure is compared simply by calculating the correlation coefficient between the two images. The SSIM is computed by multiplying these three factors together. In what follows, and for the remainder of the paper, we let and y denote two images or image patches with n components, i.e.,, y R n. The SSIM between and y is then defined as ( ) ( ) ( ) 2µ µ y 2σ σ y σy SSIM(, y) = µ 2 + µ 2 y σ 2 + σy 2, (4) σ σ y which is equivalent to ( ) ( ) 2µ µ y 2σy SSIM(, y) = µ 2 + µ 2 y σ 2 + σy 2. (5) Here µ and µ y denote the means of and y, respectively, σ y denotes the cross-correlation between and y and all other terms follow. In order to avoid division by zero, positive constants C 1 and C 2 are added for purposes of stability, leading to the formula, ( ) ( ) 2µ µ y + C 1 2σy + C 2 SSIM(, y) = µ 2 + µ 2 y + C 1 σ 2 + σy 2. (6) + C 2 Since the statistics of images vary greatly spatially, SSIM(, y) is computed using a sliding window of 8 8 piels. The final result, i.e., the so-called SSIM inde, is basically an average of the individual SSIM measures. 2.2 Quasi-conveity In this paper, we consider a special case of the SSIM defined in (6), in which both and y are zero-mean vectors, i.e., µ = µ y = 0. In this case, the luance component, i.e., the first term in Eq. (6), is equal to one, so that the SSIM between and y becomes SSIM(, y) = 2σ y + C 2 σ 2 + σ 2 y + C 2. (7) Since µ = µ y = 0, it follows that σ y = 1 n i y i and σ 2 = 1 n 1 n 1 i=1 n 2 i. (8) i=1 Using (8) and setting C 2 = 0, Eq. (7) reduces to SSIM(, y) = 2 T y 2 + y 2. (9) One might argue that removing C 2 is not good for the sake of stability. However, we shall assume that the observation y is always a non-zero vector. This ensures differentiability, stability and continuity of the SSIM for any R n. Also, since the luance component is not taken into account, no information is known about the non-zero mean optimal solution, which we shall designate as. Nevertheless, in some applications, e.g, denoising of a signal contaated with zero mean additive noise, the mean of y and coincide, so that the non-zero optimal can be recovered. In order to show under what conditions SSIM(, y) is quasi-conve, we use the definition of quasi-conveity for a function f : R n R: f is quasi-conve if its domain and all its sub-level sets S α = { dom f f() α}, for α R, are conve [10]. The domain of SSIM(, y) is R n, which is conve. As for its sub-level sets, we have that 2 T y SSIM(, y) = 2 + y 2 α α T y α y 2 0, (10) where the latter epression describes a conve set as long as α 0. This implies that quasi-conveity eists if T y 0.

3 Similarly, it can be shown that SSIM(, y) is quasi-concave if T y 0. Furthermore, due to the quasi-conveity and quasiconcavity of SSIM(, y), it follows that T (, y) in Eq. (1) is (i) quasi-conve if T y 0 and (ii) quasi-concave if and y are negatively correlated. In fact, from Eqs. (1) and (9), T (, y) may be epressed as follows, T (, y) = y y 2. (11) The range of this epression is the interval [0, 2]: T (, y) = 0 when = y and T (, y) = 2 when = y. As mentioned before, since SSIM(, y) is a measure of similarity, T (, y) can be considered a measure of dissimilarity between and y. In fact, T (, y) in Eq. (11) is an eample of a (squared) normalized metric [12]. Since in the majority of optimization problems one imizes a distance or an error subject to a set of constraints, we will define the SSIM-based optimization problems using the dissimilarity measure T (, y). 3. Optimizing the SSIM We shall define a constrained SSIM-based optimization problem as follows, T (Φ(), y) subject to h i () 0, i = 1,..., m (12) A = b, where Φ( ) is some linear transformation, A = b is an equality constraint, and the h i () are a set of conve inequality constraints. Assug that the optimal zero-mean solution is in the region where T (Φ(), y) is quasi-conve, i.e.,(φ( )) T y 0, the problem in (12) can be solved by solving a sequence of feasibility problems. For this, we require a family of conve inequalities that represent the sub-level sets of T (Φ(), y) and a conve feasibility problem that is to be solved at each step. The bisection method may be employed to detere the optimal value of (12) up to a certain accuracy [10]. The family of conve inequalities is a set of functions φ α () : R n R such that f() α φ α () 0. (13) Also, for every, φ β () φ α (), whenever α β. The following functions satisfy such conditions: φ α () = (1 α) Φ() y 2 2α(Φ()) T y. (14) The feasibility problems then assume the form Find subject to φ α () 0 (15) h i () 0, A = b. i = 1,..., m If (15) is feasible, then p α, else, p > α, where p is the optimal value of (12). Using the fact that 0 T (Φ(), y) 2, and defining 1 and 0 as vectors in R n whose entries are all equal to one and zero respectively, we propose the following algorithm for solving (12): Bisection method for constrained SSIM-based optimization initialize = 0, l = 0, u = 2, ɛ > 0; data preprocessing ȳ = 1 n 1T y, y = y ȳ1 T ; while u l > ɛ do α := (l + u)/2; Solve (15); if (15) is feasible, u := α; elseif α = 1, (12) can not be solved, break; else l := α; end return, y = y + ȳ1 T. Notice that this method will find a solution as long as (Φ()) T y 0; i.e., if the condition holds, the algorithm converges to an optimal value p. This condition may seem restrictive, however, one is normally interested in solutions that are positively correlated to the given observation y. It is worthwhile to mention that it is not always possible to recover the mean of the non-zero-mean optimal solution. This is because the luance component of the SSIM has not been taken into account. Nevertheless, in many circumstances (e.g., denoising of a signal corrupted by zeromean additive white Gaussian noise), the mean of y and Φ( ) coincide. In this case, = + ˆ, where is the zero-mean optimal solution and ˆ is a vector such that Dˆ = ȳ1. If Φ( ) is any m n matri D, it can be seen that ˆ is given by: ˆ = ȳ(d T D) 1 D T 1, (16) provided that the inverse of D T D eists. 4. Applications Clearly, different sets of constraints lead to different SSIM-based optimization problems. For instance, by disregarding the equality constraint in (12) and defining h() = A 2 2 λ, where A is either a real or comple p n matri, the result is an SSIM version of an optimization problem with a Tikhonov constraint. Nevertheless, we focus our attention on some very interesting applications that arise when the l 1 norm is used: (1) SSIM with l 1 constraint (2) total variation (TV) (3) deblurring and (4) zoog.

4 4.1 l 1 -constrained SSIM-based optimization Making the substitution Φ() = A, where A is a m n matri and R n (although may be comple), and by using the conve constraint h() = 1 λ, we obtain the following SSIM based optimization problem: T (A, y) (17) subject to 1 λ. This particular problem is appealing because it combines the concepts of similarity and sparseness. As with the classical LASSO method [13], [14], the solution of (17) is also sparse. To the best of our knowledge, this is the first reported optimization problem where the SSIM is optimized having the l 1 norm as a constraint. 4.2 SSIM and Total Variation By employing the constraint h() = D 1 λ, where D is a difference matri and Φ() =, we can define informally a SSIM-TV-denoising method for one dimensional discrete signals. Given a noisy signal y, its denoised version is the solution of the problem, T (, y) (18) subject to D 1 λ. Notice that instead of imizing the TV norm, we employ it as a constraint. This approach is not new it can also be found in [15], [16] Moreover, images can also be denoised by imizing the dissimilarity measure T (, y) subject to the following conve constraint: h() = D () 1 + D y () 1 λ, (19) where the difference matrices D ( ) and D y ( ) are linear operators used to compute the discrete spatial derivatives. Notice that the anisotropic TV norm is being used in this case. As far as we are concerned, the work reported in [17] and this application are the unique approaches in the literature that combine TV and the SSIM. 4.3 Deblurring The blurring of an image is usually modelled as the convolution of an undistorted image and a blur kernel τ. Nevertheless, in practice, the blurred observation y may have been degraded by either additive noise or errors in the acquisition process. For this reason, the following model is used to represent the degradation process [18], [19]: y = τ + η, (20) where η is usually white Gaussian noise. The problem of recovering can be addressed by the proposed approach by using the conve constraint h() = D () 1 + D y () 1 λ, and by defining Φ() = K, where K is a linear operator that performs the blurring process. That is, the unblurred image can be estimated by solving the following SSIM-based optimization problem: T (K, y) (21) subject to D () 1 + D y () 1 λ. 4.4 Zoog In this case, given an image y, assumed to be of lower resolution, we desire to find an approimation to a higher resolution version of y. This inverse problem can be solved in a manner very similar to the one described described in the previous section; that is, by defining Φ() = S, where S is a subsampling matri, and using the same conve constraint that is employed for the deblurring application. We claim that a good estimate of the high resolution image is the solution of the SSIM-based optimization problem given by T (S, y) (22) subject to D () 1 + D y () 1 λ. Observe that, in general, the matri S T S is not invertible, therefore, equation (16) can not be used to recover the optimal non-zero mean solution. Nevertheless, the mean of the low-resolution observation y can be used as a good estimate of the mean of the high resolution image that is being sought. The problem of zoog using the SSIM approach has also been addressed in [6], in which sparse representations of non-overlapping blocks of the image are used in the reconstruction process; however, the variational approach is not considered. Methods that employ the TV norm for estimating the high resolution image can be found in [20], [19], nevertheless, the fitting term is the commonly used square Euclidean distance. Problem (22) can be considered as a method that combines the SSIM and the variational approach for addressing this inverse problem. 5. Eperiments In a series of numerical eperiments, we have compared the performance of optimization methods employing (i) the usual squared Euclidean distance and (ii) Structural Similarity as fitting terms. For simplicity, we refer to these methods as (i) l 2 -based and (ii) SSIM-based methods, respectively. This is done by comparing the structural similarities between an undistorted given image and both l 2 and SSIM reconstructions. The structural similarities are calculated using the definition given by (9). By averaging the SSIM values of all non-overlapping 8 8 piel blocks, the total SSIM for each recovered image is obtained. The reconstructions are obtained by solving either a SSIM-based or an l 2 -based optimization problem over each piel block. Finally, for each application, the corresponding constraint h() being employed is the same for all non-overlapping blocks.

5 In all the applications that are presented below, the estimated mean from each block is removed prior to processing. Once the zero-mean optimal block is obtained, the optimal non-zero-mean block is recovered by means of Eq. (16), ecept in the case of zoog. In this case, the means of the high resolution blocks are approimated by the means of their corresponding low resolution counterparts. This is necessary since quasi-conveity of T (Φ(), y) is guaranteed for zero-mean vectors. This approach is also applied in the classical l 2 -based optimization method even though it is not required, for the sake of fair comparison of the two approaches. For the l 1 -constrained optimization problems of Section 4.1, both (17) and its l 2 version are solved over each nonoverlapping block. Here, Φ() = D, where D is a n n DCT matri and R n is the set of DCT coefficients that is to be recovered. As epected, the fitting term of the l 2 counterpart of (17) is substituted by D y 2 2. The result is that for a given λ, a sparse approimation problem is solved at each block. In this eperiment, a sub-image of the test image Mandrill was used. For the TV-based problems of Section 4.2, (18) is solved over each non-overlapping block. Since we are working with images, the constraint in (18) is replaced with the constraint D () 1 + D y () 1 λ. (23) The fidelity term y 2 2 is employed in the l 2 -based version of (18). In this eperiment, a noisy sub-image of the test image Mandrill, corrupted with additive zeromean Gaussian noise (AWGN) (σ = 1/32) was employed. For the deblurring problem of Section 4.3, a blurred and noisy piel subimage of the test image Lena was processed. The reconstructions were obtained by solving problem (21) and its l 2 version over each nonoverlapping block. The blurring kernel employed was a Gaussian with unit standard deviation. The blurred image was also contaated with AGWN with σ = Finally, with regard to the zoog problem of Section 4.4, the estimated high resolution images are obtained by solving both problem (22) and its l 2 version over each piel block. The fitting term employed in the l 2 -based method was S y 2 2. Some results of sparse reconstruction, deblurring and zoog are presented in Figures 1, 2 and 3, respectively. In each Figure are shown the original (uncorrupted) image, its corrupted version and SSIM- and l 2 -based reconstructions. For each set of eperiments, the SSIM maps between reconstructions and the original image are presented. The brightness of regions in the SSIM maps indicates the degree of similarity between corresponding image blocks the brighter a given point the greater the SSIM, hence visual similarity, at that location [3]. In Figure 1, where results of the sparse reconstruction problem are shown, the SSIM- and l 2 -based reconstructions are very similar. However, we notice that the SSIM reconstruction enhances the contrast at some locations (e.g., the wrinkles below the eye of Mandrill). With regard to Figure 2, where the deblurring results are shown, we see that edges in the reconstructions tend to be sharper than those in the blurred and noisy image. However, the reconstruction of tetures is not very good in both methods. We epect that improved results may be obtained by tuning the constraints of each piel block for optimal performance. With regard to Figure 3, where zoog results are shown, we observe that both SSIM- and l 2 -based reconstructions are quite good. Some vertical and horizontal artifacts can be seen in both reconstructions, located mainly near edges. This may be due to the fact that anisotropic total variation was employed as a constraint. A summary of quantitative results is presented in Table 1. The effectiveness of both SSIM- and l 2 -based approaches was quantified using the mean squared error (MSE) and the SSIM defined in Eq. (9). The best results with respect to each measure of performance are denoted in bold. We observe that the effectiveness of both methods is almost the same, although the proposed approach performs better respect to the SSIM measure SSIM(, y) = 1 T (, y), as epected. We also observe that a low MSE does not necessarily imply a high visual similarity (measured in terms of SSIM) between the reconstructions and the original images, as is well known in the literature [3]. 6. Concluding remarks We conclude by reding the reader that the primary purpose of this paper was to establish a general framework for constrained SSIM-based optimization based on the quasiconveity properties of the SSIM something that has not yet been done in the literature. The eperimental results presented above are preliary and can, in no way, be used to claim that SSIM-based optimization yields superior results in terms of visual quality. Further investigations will have to be performed in order to eae this conjecture. Our paper provides both the theoretical and computational backgrounds to continue such investigations. PROPOSED l 2 SSIM MSE SSIM MSE SPARSE RECONS TV DENOISING DEBLURRING ZOOMING Table 1: Numerical results for the different approaches and applications. Numbers in bold identify the best results with respect to each measure of performance.

6 References Fig. 1: Visual results for the sparse reconstructions. In this case, for each piel block, the maimum allowed value for the `1 norm of the coefficients that are to be recovered is 1. In the top row, SSIM maps are shown. Original and recovered images can be seen in the bottom row. Fig. 2: Visual results for the deblurring application. In the top row, the blurred and noisy image along with the SSIM maps are presented. As above, the recovered and original images are seen in the bottom row. Fig. 3: Visual results of the zoog eperiments. The low resolution image and the SSIM maps can be seen in the top row. Original image along with the SSIM and `2 reconstructions are shown in the bottom row. [1] Z. Wang and A. Bovik, A universal image quality inde, Signal Processing Letters, IEEE, vol. 9, no. 3, pp , [2] G. Albuquerque, M. Eisemann, and M. A. Magnor, Perception-based visual quality measures. in IEEE VAST. IEEE, 2011, pp [3] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, Image Processing, IEEE Transactions on, vol. 13, no. 4, pp , [4] D. Brunet, E. R. Vrscay, and Z. Wang, Structural similarity-based approimation of signals and images using orthogonal bases. in ICIAR (1), ser. Lecture Notes in Computer Science, vol Springer, 2010, pp [5] S. S. Channappayya, A. C. Bovik, C. Caramanis, and R. W. H. Jr., Design of linear equalizers optimized for the structural similarity inde. IEEE Transactions on Image Processing, vol. 17, no. 6, pp , [6] A. Rehman, M. Rostami, Z. Wang, D. Brunet, and E. R. Vrscay, Ssim-inspired image restoration using sparse representation. EURASIP J. Adv. Sig. Proc., vol. 2012, p. 16, [7] S. Wang, A. Rehman, Z. Wang, S. Ma, and W. Gao, Ssim-motivated rate-distortion optimization for video coding. IEEE Trans. Circuits Syst. Video Techn., vol. 22, no. 4, pp , [8] A. Rehman, Y. Gao, J. Wang, and Z. Wang, Image classification based on comple wavelet structural similarity. Sig. Proc.: Image Comm., vol. 28, no. 8, pp , [9] M. Elad and M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, vol. 15, no. 12, pp , [10] S. Boyd and L. Vandenberghe, Conve Optimization. Cambridge University Press, [11] S. Wang, A. Rehman, Z. Wang, S. Ma, and W. Gao, Perceptual video coding based on ssim-inspired divisive normalization. IEEE Transactions on Image Processing, vol. 22, no. 4, pp , [12] D. Brunet, A study of the structural similarity image quality measure with applications to image processing, Ph.D. dissertation, University of Waterloo, [13] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, Least angle regression, The Annals of Statistics, vol. 32, no. 2, pp , [14] J. M. F. Bach, R. Jenatton and G. Obozinski, Conve Optimization with Sparsity-Inducing Norms. Optimization for Machine Learning, MIT Press, [15] P. L. Combettes and J.-C. Pesquet, Image restoration subject to a total variation constraint, IEEE Transactions on Image Processing, vol. 13, no. 9, pp , [16] J. Fadili and G. Peyré, Total variation projection with first order schemes. IEEE Transactions on Image Processing, vol. 20, no. 3, pp , [17] Y. Shao, F. Sun, H. Li, and Y. Liu, Structural similarity-optimal total variation algorithm for image denoising, in Foundations and Practical Applications of Cognitive Systems and Information Processing, ser. Proceedings of the First International Conference on Cognitive Systems and Information Processing, Beijing, China, Dec 2012 (CSIP2012), vol Springer, 2012, pp [18] P. Getreuer, Total variation deconvolution using split bregman, Image Processing On Line, vol. 10, [19] B. Goldluecke, E. Strekalovskiy, and D. Cremers, The natural vectorial total variation which arises from geometric measure theory. SIAM J. Imaging Sciences, vol. 5, no. 2, pp , [20] A. Chambolle, An algorithm for total variation imization and applications. Journal of Mathematical Imaging and Vision, vol. 20, no. 1-2, pp , 2004.