QUANTITATIVE evaluation of an image s perceptual

Transcription

1 4270 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 10, OCTOBER 2014 VSI: A Visual Saliency-Induced Index for Perceptual Image Quality Assessment Lin Zhang, Member, IEEE, Ying Shen, Member, IEEE, and Hongyu Li Abstract Perceptual image quality assessment (IQA) aims to use computational models to measure the image quality in consistent with subjective evaluations. Visual saliency (VS) has been widely studied by psychologists, neurobiologists, and computer scientists during the last decade to investigate, which areas of an image will attract the most attention of the human visual system. Intuitively, VS is closely related to IQA in that suprathreshold distortions can largely affect VS maps of images. With this consideration, we propose a simple but very effective full reference IQA method using VS. In our proposed IQA model, the role of VS is twofold. First, VS is used as a feature when computing the local quality map of the distorted image. Second, when pooling the quality score, VS is employed as a weighting function to reflect the importance of a local region. The proposed IQA index is called visual saliency-based index (VSI). Several prominent computational VS models have been investigated in the context of IQA and the best one is chosen for VSI. Extensive experiments performed on four largescale benchmark databases demonstrate that the proposed IQA index VSI works better in terms of the prediction accuracy than all state-of-the-art IQA indices we can find while maintaining a moderate computational complexity. The MATLAB source code of VSI and the evaluation results are publicly available online at Index Terms Perceptual image quality assessment, visual saliency. I. INTRODUCTION QUANTITATIVE evaluation of an image s perceptual quality is one of the most fundamental yet challenging problems in image processing and vision research. Image quality assessment (IQA) methods fall into two categories: subjective assessment by humans and objective assessment by algorithms designed to mimic the subjective judgments. Though subjective assessment is the ultimate criterion of an image s quality, it is time-consuming, cumbersome, and cannot be implemented in systems where a real-time Manuscript received December 16, 2013; revised April 27, 2014 and July 8, 2014; accepted August 2, Date of publication August 7, 2014; date of current version August 29, This work was supported in part by the Natural Science Foundation of China under Grant and Grant , in part by the Shanghai Pujiang Program under Grant 13PJ , and in part by the Jiangsu Key Laboratory of Image and Video Understanding for Social Safety, Nanjing University of Science and Technology, Nanjing, China, under Grant The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Damon M. Chandler. L. Zhang is with the School of Software Engineering, Tongji University, Shanghai , China, and also with the Jiangsu Key Laboratory of Image and Video Understanding for Social Safety, Nanjing University of Science and Technology, Nanjing , China ( cslinzhang@tongji.edu.cn). Y. Shen and H. Li are with the School of Software Engineering, Tongji University, Shanghai , China ( yingshen@tongji.edu.cn; hyli@tongji.edu.cn). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TIP quality score for an image or video sequence is needed. Recently, there has been an increasing interest in developing objective IQA methods. According to the availability of a reference image, objective IQA indices can be classified as full reference (FR), no-reference (NR) and reduced-reference (RR) methods [1]. In this paper, the discussion is confined to FR methods, where the pristine distortion free image is known as the reference image. As a conventional fidelity metric, the peak-to-noise ratio (PSNR) or the mean squared error (MSE), works well for evaluating the quality of images sharing the same content and the same distortion type. However, quality scores predicted by PSNR or MSE do not correlate well with human beings subjective fidelity ratings when multiple images or multiple distortion types are involved [2]. In the past decade, several sophisticated IQA models have been proposed and some representative ones will be briefly reviewed here. The noise quality measure index (NQM) [3] and the visual signal-to-noise ratio index (VSNR) [4] emphasize the importance of human visual system (HVS) s sensitivity to different visual signals, such as the luminance, the contrast, the frequency content, and the interaction between them. The structural similarity index (SSIM) proposed by Wang et al. [5] can be considered as a milestone of the development of IQA models. SSIM is based on the hypothesis that HVS is highly adapted to extract the structural information from the visual scene and therefore a measurement of structural similarity can provide a good approximation of the perceived image quality. In their later work, Wang et al. proposed a multi-scale extension of SSIM, namely MS-SSIM [6] and it has been corroborated that MS-SSIM could produce better results than its single scale counterpart. In [7], Wang and Li improved the original MS-SSIM to the information content weighted SSIM index (IW-SSIM) by introducing a new information content weighting (IW)-based quality score pooling strategy. In [8], Sheikh et al. proposed the visual information fidelity index (VIF), which was an extension of its former version, i.e. the information fidelity criterion index (IFC) [9]. In VIF, Sheikh et al. treated the FR IQA problem as an information fidelity problem and the fidelity were quantified by the amount of information shared between the reference image and the distorted image. In [10], Zhang et al. proposed a Riesz transforms based feature similarity index (RFSIM). In RFSIM, 1 st -order and 2 nd -order Riesz transforms are used to characterize image s local structures and the Canny edge detector is employed to generate the mask for quality score pooling. Larson and Chandler argued that the HVS performs two distinct strategies when assessing the image quality for IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 ZHANG et al.: VSI FOR PERCEPTUAL IQA 4271 high-quality images and for low-quality images, and accordingly they proposed a most apparent distortion (MAD) based IQA index [11]. The feature similarity index (FSIM) proposed in [12] employs two features to compute the local similarity map, the phase congruency and the gradient magnitude. The authors claimed that the phase congruency and the gradient magnitude play complementary roles in characterizing the local image quality. At the quality score pooling stage of FSIM, phase congruency map is utilized again as a weighting function since it can roughly reflect how perceptually important a local patch is to the HVS. By considering that structural and contrast changes can be effectively captured by gradients, Liu et al. proposed a gradient similarity based metric (GSM) [13] for FR IQA. For a thorough survey of modern IQA development, please refer to [14] and [15]. On the other hand, in recent years how to build effective computational visual saliency (VS) models has been attracting tremendous attention [16] [18]. Given an image, its VS map computed by an appropriate VS model can reflect how salient a local region is to the HVS. Intuitively, VS and IQA are intrinsically related because both of them depend on how HVS perceives an image and suprathreshold distortions can be a strong attractor of visual attention [19]. Thus recently, researchers have been trying to incorporate VS information to IQA models to improve their performance. Meanwhile, there are also some other studies focusing on the relationship between visual attention and the perceptual quality of videos [30] [33]. The relationship between VS and IQA has been investigated by some researchers in previous studies and it is widely accepted that incorporating VS information appropriately can benefit IQA metrics. However, a practical VS-based computational IQA model that could achieve better prediction performance than the other state-of-the-art methods, such as IW-SSIM [7], FSIM C [12], GSM [13], has not come out yet. In this paper, we expect to fill this research gap to some extent. By analyzing the relationship between the changes of an image s VS map and its perceived quality degradation, we propose a simple yet very effective VS-based index (VSI) for the IQA task. We claim that the VS map cannot be only used as a weighting function at the score-pooling stage, but can also be used as a feature map to characterize the quality of local image regions. The underlying reason is that perceptible quality distortions can lead to measurable changes of images visual saliency maps. Consequently, in our proposed VSI metric, the role of an image s VS map is twofold: a feature map characterizing the image s local quality, as well as a weighting function indicating the importance of a local region to the HVS when pooling the final quality score. In our work, several eminent VS models have been explored in the context of IQA and the most suitable one is selected for VSI. VSI is thoroughly examined by extensive experiments conducted on four large scale databases. The results show that our proposed VSI works consistently better than all the other state-of-the-art IQA metrics. In addition, the computational complexity of VSI is quite low. The Matlab source code of VSI and the associated evaluation results have been made publicly available online at The remainder of this paper is organized as follows. Section II introduces the works relevant to this paper. Section III presents in detail the proposed VSI metric for IQA. Section IV presents the experimental results and associated discussions. Finally, Section V concludes the paper. II. RELATED WORKS This section presents works most related to our paper, which covers a brief review of modern VS models and their existing applications in IQA. A. Computational Visual Saliency Models As a consequence of evolution, most vertebrates, including humans, have a remarkable ability to automatically pay more attention to salient regions of the visual scene. Building effective computational models to simulate human visual attention has been studied by scholars in psychology, neurobiology, and computer vision for a long time, and some powerful models have been proposed. Although both bottomup (scene dependant) and top-down (task dependant) factors will affect the visual attention, most of the existing computational VS models are bottom-up since bottom-up attention mechanisms are more thoroughly studied than top-down mechanisms. In bottom-up VS models, it is supposed that visual attention is driven by low-level visual stimulus in the scene, such as intensity, color, orientation, etc. The first influential and best known VS model was proposed by Itti et al. [34] for still images. Itti et al. s model was based on the VS computational architecture introduced by Koch and Ullman [35]. Itti et al. s contribution mainly lies in two aspects. First, they introduced image pyramids for feature extraction, which makes the VS computation efficient. Second, they proposed the biologically inspired centersurround difference operation to compute feature dependant saliency maps across scales. In their later work, Itti and Baldi introduced a Bayesian model of surprise aiming to predict eye movements [36]. In [37], following Itti et al. s architecture, Harel et al. proposed the graph-based visual saliency (GBVS) model by introducing a novel graph-based normalization and combination strategy. In another work following Itti et al. s framework, Klein and Frintrop [38] modeled the centersurround contrast in an information-theoretic way, in which two distributions of visual feature occurrences are determined for a center and for a surround region, respectively. By analyzing the log-spectrum of the input image, Hou and Zhang [39] proposed a Fourier transform based method to extract the spectral residual (SR) of an image in the spectral domain and to construct the corresponding saliency map in the spatial domain; one prominent advantage of this method is its low computational complexity. In Hou s recent work [40], he developed a saliency algorithm based on the image signature (IS), which can approximate the foreground of an image and can be simply computed as the sign map of the image s DCT (discrete cosine transform) coefficients. In [41], Bruce and Tsotsos proposed the model of attention based on information maximization (AIM), in which an image s saliency is modeled as the maximum information that can be sampled

3 4272 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 10, OCTOBER 2014 from it. In [42], Seo and Milanfar used local regression kernels (LRK) as features and used a local self-resemblance measure, which indicates the likelihood of saliency, to build an image s saliency map. In [43], Achanta et al. proposed a conceptually simple approach for detecting saliency by combining image s responses to band-pass filters from three CIEL*a*b* channels. In [44], Judd et al. extracted various types of features and fed them into a SVM to train a model to predict the visual saliency of a given test image. In [45], Shen and Wu represent an image as a low-rank matrix plus sparse noises, where the low-rank matrix explains the non-salient regions while the sparse noises indicate the salient regions. By integrating prior knowledge from three aspects, frequency prior, color prior, and location prior, Zhang et al. [46] proposed an efficient saliency algorithm, namely SDSP (Saliency Detection by combining Simple Priors). For a complete recent survey of modern VS models, please refer to [16] [18]. B. Existing Investigations of Visual Saliency in IQA Recently, increased awareness to the close relationship between VS and quality perception has led to a number of approaches that try to integrate VS into IQA metrics to potentially improve their prediction performance. In [20], Vu et al. designed two experiments to examine visual fixation patterns when judging image quality. Their results revealed that regions where people fixate while judging image quality can be different from those obtained under taskfree condition. In their another work [21], five common fidelity metrics were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The results show that most metrics could be improved using fixation data and a greater improvement was found using fixations obtained in the task-free condition. Similar results have also been obtained by a recent study [22]. In [23], Larson et al. revealed that common metrics (such as SSIM, PSNR, VIF, etc.) could be improved by using spatially varying weights for pooling. In [24], a framework was introduced to extend existing quality metrics by segmenting an image into ROI (region of interest) and background regions. With such a method, the metrics are computed independently on ROI and background regions and then a pooling function is used to derive the final quality score. The abovementioned works demonstrate that if being incorporated appropriately, visual attention data can benefit the design of IQA metrics. However, it should be noted that the visual attention data or the ROI data used in these works are either obtained by eye tracking or are hand-labeled. Thus, these approaches cannot be used in applications where a fully automatic IQA metric is needed. Rather than using eye-tracking or subjective ROI data, some researchers attempted to incorporate VS information computed by using computational VS models into IQA models. Representative works belonging to this category include [25] [29] and they share some common characteristics. At first, these studies are based on the assumption that a distortion occurring in an area that attracts the viewer s attention is more annoying than in any other area, and they attempt to weight local distortions with a local saliency map. Consequently, pooling strategies adopted in these methods share a general form as: S = K i w i s i / Ki w i (1) where s i is the local quality value at the i-thlocationinthe local quality map, w i is the VS value at the location i, K is the number of points in the image, and S is the final quality score of the examined image. Secondly, for these methods, the motivation is actually not to design a new IQA index but to demonstrate that a VS-weighted pooling strategy could perform better than the simple mean scheme. Thus, for computing the local quality map, they all adopt some existing methods, such as PSNR, SSIM, and VIF, without discussing whether there could be more effective methods to characterize the local image quality. Thirdly, these works lack extensive evaluations to verify the effectiveness of the proposed IQA indices. Usually, the experiments were performed only on a specific dataset and only some classic IQA indices (e.g., SSIM, VIF, PSNR) were used for comparison. Some recently developed high performance IQA metrics, such as IW-SSIM [7], FSIM/FSIM C [12], and GSM [13], were not compared with, which makes the elicited conclusions less convincing. III. VS-BASED INDEX (VSI) FOR IQA As stated in Section II, VS has already been used as a weighting function for quality score pooling in some previous studies [25] [29]. In [12], Zhang et al. have shown that perceptible image quality degradation can lead to perceptible changes in image s low-level features. Since the bottom-up VS models are basically based on image s low-level features, the VS values themselves actually vary with the change of image quality. Therefore, why don t we use VS as a feature to compute the local similarity map between the reference image and the distorted image? We find that quality distortions could give rise to changes in images VS maps and the intensities of such measurable changes correlate well with the degrees of perceptible quality distortions. To support our claim, we have conducted a statistical analysis on VS maps of images in TID2013 [47], the most comprehensive dataset available for IQA research. In TID2013, there are 25 reference images, 24 distortion types and 5 distortion levels. Hence, for one distortion type at a particular distortion level, there are 25 samples. To perform such an analysis, we at first computed VS maps for all the images by using GBVS model [37] and then for each referencedistortion image pair, we computed the MSE between their VS maps. After that, we averaged MSEs belonging to the same distortion type and the same distortion level. The results are listed in Table I. 1 In each field of Table I, we also present the average subjective score for the corresponding distortion type and distortion level in a bracket. In TID2013, a higher 1 For better observation, MSE values listed here are the actual MSE values multiplied by 10 6.

4 ZHANG et al.: VSI FOR PERCEPTUAL IQA 4273 TABLE I AVERAGE MSESOFVS AND AVERAGE SUBJECTIVE SCORES FOR DISTORTIONS AT DIFFERENT LEVELS Fig. 1. (a) is a reference image while (c) and (e) are its two distorted versions; subjective scores for (c) and (e) are 5.05 and 2.40, respectively; (b), (d) and (f) are the VS maps of (a), (c) and (e), respectively; the MSE between (d) and (b) is 4.49 while the MSE between (f) and (b) is subjective score indicates a better image quality. Based on the results listed in Table I, we could have the following findings. At first, for nearly all types of distortions, a lower average subjective score corresponds to a severer average VS changes measured by MSE. Secondly, even for most cross-distortion cases, VS changes can also be a good indicator of perceptual image quality. For example, the average subjective score for the distortion SCN Level 5 (2.69) is much poorer than the one for the distortion AGN Level 1 (5.67); as expected, the average VS changes for the distortion SCN Level 5 (244.19) is much severer than the one for the distortion AGN Level 1 (2.68). The relationship between the quality distortions and VS changes is illustrated using an example in Fig. 1. Images used in Fig. 1 are from TID2013. Fig. 1(a) is a reference image and Figs. 1(c) and (e) are its two distorted versions. The subjective scores for Figs. 1(c) and (e) are 5.05 and 2.40, respectively. Figs. 1(b), 1(d) and 1(f) are the VS maps of Figs. 1(a), 1(c) and 1(e), respectively. The MSE between Figs. 1(d) and 1(b) is 4.49 while the MSE between Figs. 1(f) and 1(b) is Fig. 1(e) has a poorer quality than Fig. 1(c) and as expected, its VS map Fig. 1(f) alters much more than Fig. 1(c) s VS map Fig. 1(d) when being compared with the reference VS map Fig. 1(b). Based on the above analysis, it can be seen that in most cases, changes of VS maps can be a good indicator of distortion degrees and thus, in this paper we propose to use the VS map as a feature to characterize the image s local quality. However, from Table I, it can be seen that as a quality distortion indicator, VS map does not work quite well for the distortion type CTC (Contrast Change). The root reason is that due to the normalization operations involved in VS computational models, the VS value at a pixel is a measure to reflect its relative distinctiveness to its surroundings, which makes VS weak to characterize image s absolute contrast. Nonetheless, image s local contrast does affect much HVS perception of the image quality. We use an example in Fig. 2 to illustrate this fact. Fig. 2(a) is a reference image while 2(b) is a distorted version of it and the distortion type is contrast reduction. Fig. 2(c) and Fig. 2(d) are the VS maps of Fig. 2(a) and Fig. 2(b), respectively. It can be clearly seen that Fig. 2(b) has lower quality than Fig. 2(a). However, such a quality degradation caused by contrast reduction cannot be

5 4274 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 10, OCTOBER 2014 Fig. 2. (a) is a reference image while (b) is a distorted version of it (the distortion type is contrast reduction); (c) and (d) are the VS maps computed from (a) and (b) respectively using the GBVS model [37]; (e) and (f) are the GM maps computed from (a) and (b) respectively using the Scharr gradient operator. No significant difference can be observed between (c) and (d), which indicates that VS map behaves poorly in characterizing the contrast loss of the images. By contrast, apparent differences can be observed between (e) and (f), indicating that GM map has a good capability in reflecting the contrast loss of images. reflected in their VS maps, since no significant difference can be observed between Fig. 2(c) and Fig. 2(d). Fortunately, we can use an additional feature to compensate for the lack of contrast sensitivity of VS. The simplest feature of this kind may be the gradient modulus (GM). There are several different operators to compute the image gradient, such as the Prewitt operator, the Sobel operator, the Roberts operator [48] and the Scharr operator [49], and here we adopt the Scharr gradient operator, which has been proved very powerful in our previous work [12]. With Scharr gradient operator, the partial derivatives G x (x) andg y (x) ofanimage f (x) are calculated as: G x (x) = f (x) G y (x) = f (x) (2) The GM of f (x) is then computed as G(x) = G 2 x (x) + G2 y (x). GM maps of Figs. 2(a) and 2(b) are visualized in Figs. 2(e) and 2(f), respectively. Apparent differences can be observed between Figs. 2(e) and 2(f), indicating that GM map has a good potential capability in reflecting the local contrast loss of images. Therefore, VS and GM are complementary and they reflect different aspects of the HVS in assessing the local quality of the input image. From Table I, it can also be seen that as a quality distortion indicator, VS map does not work quite well for the distortion type CCS (Change of Color Saturation) either. Actually, color distortion cannot be well characterized by gradient either since usually gradient is computed from the luminance channel of images. Hence, to make the IQA metric possess the capability to deal with color distortions, chrominance information should be given special considerations. Consequently, for RGB color images, we first transform them into an opponent color space [50]: L R M = G (3) N B The weights in the above conversion are optimized for the HVS [51]. Then, the gradients are computed from L channels. M and N, two chrominance channels, will be used as features to characterize the quality degradation caused by color distortions. With the extracted VS, GM, and chrominance features, we can define a VS-based index (VSI) for IQA tasks. Suppose that we are going to calculate the similarity between images f 1 and f 2. Denote by VS 1 and VS 2 the two VS maps extracted from images f 1 and f 2 using a specific VS model; denote by G 1 and G 2 the two GM maps; denote by M 1 and M 2 the two M channels; and denote by N 1 and N 2 the two N channels. The computation of VSI consists of two stages. In the first stage, the local similarity map is computed, and in the second stage, we pool the similarity map into a single quality score. We separate the similarity measurement between f 1 (x) and f 2 (x) into three components, one for VS, one for GM, and the other for chrominance. First, the similarity between VS 1 (x)and VS 2 (x) isdefinedas: S VS (x) = 2VS 1(x) VS 2 (x) + C 1 VS1 2(x) + VS2 2 (x) + C (4) 1 where C 1 is a positive constant to increase the stability of S VS. Similarly, the GM values G 1 (x) andg 2 (x) are compared as: S G (x) = 2G 1(x) G 2 (x) + C 2 G 2 1 (x) + G2 2 (x) + C (5) 2 where C 2 is another positive constant. The similarity between the chrominance components is simply defined as: S C (x) = 2M 1(x) M 2 (x) + C 3 M1 2(x) + M2 2 (x) + C 2N 1(x) N 2 (x) + C 3 3 N1 2(x) + N2 2 (x) + C (6) 3 In our experiments, C 1, C 2 and C 3 are all fixed so that the proposed VSI can be conveniently applied to all datasets. Then, S VS (x), S G (x) ands C (x) are combined to get the local similarity S(x) of f 1 (x) and f 2 (x). We define S(x) as follows: S(x) = S VS (x) [S G (x)] α [S C (x)] β (7) where α and β are two parameters used to adjust the relative importance of VS, GM, and chrominance features. Having obtained the local similarity S(x) at each location x, the overall similarity between f 1 and f 2 can be calculated.

6 ZHANG et al.: VSI FOR PERCEPTUAL IQA 4275 Fig. 3. Illustration for the computational process of the proposed IQA index VSI. f 1 is a reference image and f 2 is a distorted version of f 1. It has been widely accepted that different locations can have different contributions to the HVS perception of the image quality and it would be better if the score pooling strategy could be correlated with human visual fixation. Consequently, in our VSI framework, it is natural to choose the VS map to characterize the visual importance of a local region. Intuitively, for a given position x, if anyone of f 1 (x) and f 2 (x) hasahigh VS value, it implies that this position x will have a high impact on HVS when it evaluates the similarity between f 1 and f 2. Therefore, we use VS m (x) = max(vs 1 (x), VS 2 (x)) to weight the importance of S(x) in the overall similarity. Actually, a similar form was used in [12]. Finally, the VSI metric between f 1 and f 2 is defined as: x VSI = S(x) VS m(x) x VS (8) m(x) where means the whole spatial domain. It can be easily verified that as a metric function VSI satisfies the symmetry property mentioned in [5]. The procedures to compute VSI are illustrated by an example in Fig. 3. TABLE II BENCHMARK DATASETS FOR EVALUATING IQA INDICES TABLE III SROCC VALUES OBTAINED BY USING VSI WITH DIFFERENT VS MODELS ON THE SUB-DATASET IV. EXPERIMENTAL RESULTS AND DISCUSSIONS A. Experimental Protocol Experiments were conducted on four large-scale image datasets constructed for evaluating IQA indices, including TID2013 [47], TID2008 [52], CSIQ [11] and LIVE [53]. The important information of these four datasets, in terms of the number of reference images, the number of distorted images, the number of quality distortion types, and the number of subjects performing the subjective evaluations, is summarized in Table II. Totally, there are 6345 distorted images contained in these datasets. Four commonly used performance metrics are employed to evaluate the IQA indices. The first two are the Spearman rankorder correlation coefficient (SROCC) and the Kendall rankorder correlation coefficient (KROCC), which can measure the prediction monotonicity of an IQA index. These two metrics operate only on the rank of the data points and ignore the relative distance between data points. To compute the other two metrics we need to apply a regression analysis to provide a nonlinear mapping between the objective scores and the subjective mean opinion scores (MOS). The third metric is the Pearson linear correlation coefficient (PLCC) between MOS and the objective scores after nonlinear regression. The fourth metric is the root mean squared error (RMSE) between MOS and the objective scores after nonlinear regression. For the nonlinear regression, we used the following mapping function as suggested by Sheikh et al. [53]: f (x) = β 1 ( e β 2(x β 3 ) ) + β 4 x + β 5 (9)

7 4276 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 10, OCTOBER 2014 TABLE IV PERFORMANCE COMPARISON OF 13 IQA INDICES ON FOUR BENCHMARK DATASETS TABLE V OVERALL PERFORMANCES OF IQA INDICES OVER 4DATASETS TABLE VI RANKING OF OVERALL PERFORMANCES OF IQA INDICES where β i, i = 1, 2,...,5, are parameters to be fitted. More details about the definitions and explanations of these four performance metrics can be found in [7]. VSI was compared with the other 12 state-of-the-art or representative IQA indices, including SSIM_I [29], SSIM [5], MS-SSIM [6], IFC [9], VIF [8], VSNR [4], MAD [11], GSM [13], IW-SSIM [7], RFSIM [10], FSIM [12], and FSIM C [12]. It needs to be pointed out that SSIM_I [29] is a representative IQA model which adopts a visual saliency map as a weighting function for score pooling. Specifically, SSIM_I uses SSIM [5] to compute the local quality map and uses Itti s model [34] to compute the visual saliency map. B. Evaluation of VS Models, Determination of Parameters, and Examination of Two Roles of VS In our VSI scheme, the VS map could be computed using various VS models. In order to find an appropriate candidate for VSI, eight eminent computational VS models, including Itti s model [34], GBVS [37], AIM [41], LRK [42], SR [39], FT [43], IS [40] and SDSP [46] were tested for VSI. In order to reduce the burden of parameter adjustment, in this experiment only a sub-dataset of TID2008 was used, which contained the first 8 reference images and the associated 544 distorted images. For each VS model evaluated, the related parameters were tuned experimentally and the tuning criterion was that the parameter value leading to a higher SROCC would be chosen. The SROCC values obtained by VSI with eight different VS models on the tuning dataset are listed in Table III, from which we can see that the SDSP model could achieve better results than the others. Thus, in all of the following experiments, SDSP was used to compute the VS map for VSI. Key parameters α and β are set as 0.40 and 0.02, respectively. In VSI, the role of the VS map is twofold: a feature map characterizing the image s local quality and a weighting function indicating the importance of a local region for quality score pooling. In this experiment, we will show the benefits brought by these two roles of VS map. The experiment was conducted on TID2013 dataset and we use SROCC as the performance measure. If VS map was used only as a weighting function for quality score pooling (in this case, the local quality map was computed based on the gradient modulus map and two chrominance channels), the SROCC obtained

8 ZHANG et al.: VSI FOR PERCEPTUAL IQA 4277 Fig. 4. Scatter plots of subjective MOS against scores obtained by model prediction on the TID2013 database. (a) SSIM_I, (b) SSIM, (c) MS-SSIM, (d) IFC, (e) VIF, (f) VSNR, (g) MAD, (h) GSM, (i) IWSSIM, (j) RFSIM, (k) FSIM, and (l) VSI. was If VS map was used only as a feature map and we used a simple averaging strategy for quality score pooling, the SROCC was If VS map was used both as a feature map and a weighting function, the SROCC was From this experiment, it can be seen that to better explore the power of VS map, it should be used both as a feature map and a weighting function for quality score pooling. C. Performance Evaluation In this section, the prediction performance measured by SROCC, KROCC, PLCC, and RMSE of each competing IQA

9 4278 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 10, OCTOBER 2014 TABLE VII SROCC VALUES OF IQA I NDICES FOR E ACH T YPE OF D ISTORTIONS index on each benchmark dataset is given. The results are listed in Table IV. For each performance measure, the two IQA indices producing the best results are highlighted in boldface. In addition, as suggested by Wang and Li [7], in order to provide an evaluation of the overall performance of the evaluated IQA indices, in Table V we present their weighted-average SROCC, KROCC and PLCC results over four datasets and the weight assigned to each dataset linearly depends on the number of distorted images contained in that dataset. The ranking of the weighted-average performances of the evaluated IQA indices based on three different performance metrics, SROCC, KROCC, and PLCC, is presented in Table VI.

10 ZHANG et al.: VSI FOR PERCEPTUAL IQA 4279 From Table IV, it can be seen that VSI performs consistently well on all the benchmark databases. Particularly, it performs greatly better than all the other competitors on the two largest databases, TID2013 and TID2008. On CSIQ and LIVE, even though it is not the best, VSI performs only slightly worse than the best results. By contrast, for the other methods, they may work well on some database but fail to provide good results on other databases. For example, though VIF and MAD can get pleasing results on LIVE, they perform quite poor on TID2013 and TID2008. In Tables V and VI, the statistical superiority of VSI to the other competing IQA indices is clearly exhibited since no matter which performance measure is used, VSI always achieves the best overall results. Hence, we can conclude that objective scores predicted by VSI correlate much more consistently with subjective evaluations than all the other IQA indices evaluated. In addition, in Fig. 4 we show the scatter plots of subjective scores against objective scores predicted by some representative IQA indices (SSIM_I, SSIM, MS-SSIM, IFC, VIF, VSNR, MAD, GSM, IW-SSIM, RFSIM, FSIM, and VSI) on TID2013, at present the largest benchmark database for evaluating IQA indices. The curves shown in Fig. 4 were obtained by a nonlinear fitting using Eq. (9). From Fig. 4, it can also be seen that objective scores predicted by VSI is more correlated with subjective ratings than the other competitors. D. Performance Comparison on Individual Distortion Types To more comprehensively evaluate an IQA index s ability to predict image quality degradations caused by specific types of distortions, in this experiment, we examined the performance of the competing methods on each type of distortions. We use SROCC as the performance measure. In fact, by using the other measures, such as KROCC, PLCC, and RMSE, similar conclusions could be drawn. The results are summarized in Table VII. There are a total of 52 groups of distorted images in the four databases. For each database and each distortion type, the first three IQA indices producing the highest SROCC values are highlighted in boldface. It can be seen that VSI is among the top 3 indices 34 times, followed by FSIM C (27 times) and GSM (25 times). Thus, we can have the following conclusions. In general, when the distortion is of a specific type, VSI performs the best, while FSIM C and GSM can have comparable performance. Moreover, in this case, VSI, FSIM C and GSM perform much better than the other IQA indices. E. Computational Cost The running speed of each selected IQA index was also evaluated. Experiments were performed on a standard HP Z620 workstation with a 3.2GHZ Intel Xeon E CPU and an 8G RAM. The software platform was Matlab R2012a. The time cost consumed by each IQA index for measuring the similarity of a pair of color images (taken from TID2013) is listed in Table VIII. From Table VIII we can see that VSI has a moderate computational complexity. Particularly, it runs much faster than the other modern IQA indices TABLE VIII TIME COST OF EACH IQA INDEX which could achieve state-of-the-art prediction performances, such as FSIM C, FSIM, IW-SSIM, and MAD. V. CONCLUSIONS In this paper, we proposed a novel metric for IQA, namely visual saliency based index (VSI). It is based on the assumption that an image s VS map has a close relationship with its perceptual quality. In VSI, the visual saliency (VS) map is explored at two stages. At the stage of local quality map computation, the VS map is taken as an image feature; while at the quality score pooling stage, it is used as a weighting function to characterize the importance of a local image region. Several representative VS models were examined under our framework of VSI for IQA tasks, and among them the SDSP model performs the best. The proposed VSI was thoroughly tested and compared with the other 12 state-of-the-art or widely cited IQA indices on four large-scale benchmark IQA databases. The results demonstrated that the proposed IQA index VSI could yield statistically much better results in terms of the prediction accuracy than all the other competing methods evaluated while still maintaining a moderate computational complexity. To some extent, VSI is actually an open framework; therefore, with the advent of even more powerful VS models, VSI of course can be improved accordingly. REFERENCES [1] Z. Wang and A. C. Bovik, Modern Image Quality Assessment. San Rafael, CA, USA: Morgan & Claypool, [2] Z. Wang and A. C. Bovik, Mean squared error: Love it or leave it? A new look at signal fidelity measures, IEEE Signal Process. Mag., vol. 26, no. 1, pp , Jan [3] N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik, Image quality assessment based on a degradation model, IEEE Trans. Image Process., vol. 9, no. 4, pp , Apr [4] D. M. Chandler and S. S. Hemami, VSNR: A wavelet-based visual signal-to-noise ratio for natural images, IEEE Trans. Image Process., vol. 16, no. 9, pp , Sep [5] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process., vol. 13, no. 4, pp , Apr [6] Z. Wang, E. P. Simoncelli, and A. C. Bovik, Multiscale structural similarity for image quality assessment, in Proc. 37th Asilomar Conf. Signals, Syst., Comput., Nov. 2003, pp [7] Z. Wang and Q. Li, Information content weighting for perceptual image quality assessment, IEEE Trans. Image Process., vol. 20, no. 5, pp , May 2011.

11 4280 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 23, NO. 10, OCTOBER 2014 [8] H. R. Sheikh and A. C. Bovik, Image information and visual quality, IEEE Trans. Image Process., vol. 15, no. 2, pp , Feb [9] H. R. Sheikh, A. C. Bovik, and G. de Veciana, An information fidelity criterion for image quality assessment using natural scene statistics, IEEE Trans. Image Process., vol. 14, no. 12, pp , Dec [10] L. Zhang, D. Zhang, and X. Mou, RFSIM: A feature based image quality assessment metric using Riesz transforms, in Proc. 17th IEEE Int. Conf. Image Process., Sep. 2010, pp [11] E. C. Larson and D. M. Chandler, Most apparent distortion: Full-reference image quality assessment and the role of strategy, J. Electron. Imag., vol. 19, no. 1, pp : :21, Jan [12] L. Zhang, D. Zhang, X. Mou, and D. Zhang, FSIM: A feature similarity index for image quality assessment, IEEE Trans. Image Process., vol. 20, no. 8, pp , Aug [13] A. Liu, W. Lin, and M. Narwaria, Image quality assessment based on gradient similarity, IEEE Trans. Image Process., vol. 21, no. 4, pp , Apr [14] W. Lin and C.-C. J. Kuo, Perceptual visual quality metrics: A survey, J. Vis. Commun. Image Represent., vol. 22, no. 4, pp , May [15] L. Zhang, L. Zhang, X. Mou, and D. Zhang, A comprehensive evaluation of full reference image quality assessment algorithms, in Proc. 19th IEEE Int. Conf. Image Process., Sep./Oct. 2012, pp [16] A. Toet, Computational versus psychophysical bottom-up image saliency: A comparative evaluation study, IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 11, pp , Nov [17] A. Borji and L. Itti, State-of-the-art in visual attention modeling, IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp , Jan [18] A. Borji, D. N. Sihite, and L. Itti, Quantitative analysis of humanmodel agreement in visual saliency modeling: A comparative study, IEEE Trans. Image Process., vol. 22, no. 1, pp , Jan [19] U. Engelke, H. Kaprykowsky, H. Zepernick, and P. Ndjiki-Nya, Visual attention in quality assessment, IEEE Signal Process. Mag., vol. 28, no. 6, pp , Nov [20] E. C. L. Vu and D. M. Chandler, Visual fixation patterns when judging image quality: Effects of distortion type, amount, and subject experience, in Proc. IEEE Southwest Symp. Image Anal. Interpretation, Mar. 2008, pp [21] E. C. Larson, C. Vu, and D. M. Chandler, Can visual fixation patterns improve image fidelity assessment? in Proc. 15th IEEE Int. Conf. Image Process., Oct. 2008, pp [22] H. Liu and I. Heynderickx, Visual attention in objective image quality assessment: Based on eye-tracking data, IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 7, pp , Jul [23] E. C. Larson and D. M. Chandler, Unveiling relationships between regions of interest and image fidelity metrics, Proc. SPIE, Vis. Commun. Image Process., vol. 6822, pp A: A:16, Jan [24] U. Engelke and H.-J. Zepernick, Framework for optimal region of interest-based quality assessment in wireless imaging, J. Electron. Imag., vol. 19, no. 1, pp , [25] A. K. Moorthy and A. C. Bovik, Visual importance pooling for image quality assessment, IEEE J. Sel. Topics Signal Process., vol. 3, no. 2, pp , Apr [26] I. Gkioulekas, G. Evangelopoulos, and P. Maragos, Spatial Bayesian surprise for image saliency and quality assessment, in Proc. 17th IEEE Int. Conf. Image Process., Sep. 2010, pp [27] Y. Tong, H. Konik, F. Cheikh, and A. Tremeau, Full reference image quality assessment based on saliency map analysis, J. Imag. Sci. Technol., vol. 54, no. 3, pp : :14, May [28] Q. Ma, L. Zhang, and B. Wang, New strategy for image and video quality assessment, J. Electron. Imag., vol. 19, no. 1, pp : :14, Jan [29] M. C. Q. Farias and W. Y. L. Akamine, On performance of image quality metrics enhanced with visual attention computational models, Electron. Lett., vol. 48, no. 11, pp , May [30] O. Le Meur, A. Ninassi, P. Le Callet, and D. Barba, Overt visual attention for free-viewing and quality assessment tasks: Impact of the regions of interest on a video quality metric, Signal Process., Image Commun., vol. 25, no. 7, pp , Aug [31] O. Le Meur, A. Ninassi, P. Le Callet, and D. Barba, Do video coding impairments disturb the visual attention deployment, Signal Process., Image Commun., vol. 25, no. 8, pp , Sep [32] J. You, J. Korhonen, A. Perkis, and T. Ebrahimi, Balancing attended and global stimuli in perceived video quality assessment, IEEE Trans. Multimedia, vol. 13, no. 6, pp , Dec [33] J. You, A. Perkis, M. M. Hannuksela, and M. Gabbouj, Perceptual quality assessment based on visual attention analysis, in Proc. 17th ACM Int. Conf. Multimedia, 2009, pp [34] L. Itti, C. Koch, and E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp , Nov [35] C. Koch and S. Ullman, Shifts in selective visual attention: Towards the underlying neural circuitry, Human Neurobiol., vol. 4, no. 4, pp , Apr [36] L. Itti and P. Baldi, Bayesian surprise attracts human attention, Vis. Res., vol. 49, no. 10, pp , Jun [37] J. Harel, C. Koch, and P. Perona, Graph-based visual saliency, in Advances in Neural Information Processing Systems, vol. 19. Cambridge, MA, USA: MIT Press, 2007, pp [38] D. A. Klein and S. Frintrop, Center-surround divergence of feature statistics for salient object detection, in Proc. IEEE Int. Conf. Comput. Vis., Nov. 2011, pp [39] X. Hou and L. Zhang, Saliency detection: A spectral residual approach, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp [40] X. Hou, J. Harel, and C. Koch, Image signature: Highlighting sparse salient regions, IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp , Jan [41] N. D. B. Bruce and J. K. Tsotsos, Saliency, attention, and visual search: An information theoretic approach, J. Vis., vol. 9, no. 3, pp. 1 24, Mar [42] H. J. Seo and P. Milanfar, Static and space-time visual saliency detection by self-resemblance, J. Vis., vol. 9, no. 12, pp , Nov [43] R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk, Frequency-tuned salient region detection, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp [44] T. Judd, K. Ehinger, F. Durand, and A. Torralba, Learning to predict where humans look, in Proc. IEEE 12th Int. Conf. Comput. Vis., Sep./Oct. 2009, pp [45] X. Shen and Y. Wu, A unified approach to salient object detection via low rank matrix recovery, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp [46] L. Zhang, Z. Gu, and H. Li, SDSP: A novel saliency detection method by combining simple priors, in Proc. IEEE Int. Conf. Image Process., Sep. 2013, pp [47] N. Ponomarenko et al., Color image database TID2013: Peculiarities and preliminary results, in Proc. 4th Eur. Workshop Vis. Inf. Process., Jun. 2013, pp [48] M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision. Stanford, CT, USA: Cengage Learning, [49] B. Jähne, H. Haubecker, and P. Geibler, Handbook of Computer Vision and Applications. New York, NY, USA: Academic, [50] J.-M. Geusebroek, R. Van den Boomgaard, A. W. M. Smeulders, and H. Geerts, Color invariance, IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 12, pp , Dec [51] J.-M. Geusebroek, R. Van den Boomgaard, A. W. M. Smeulders, and A. Dev, Color and scale: The spatial structure of color images, in Proc. Eur. Conf. Comput. Vis., 2000, pp [52] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli, and F. Battisti, TID2008 A database for evaluation of full-reference visual quality assessment metrics, Adv. Modern Radioelectron., vol. 10, pp , [53] H. R. Sheikh, M. F. Sabir, and A. C. Bovik, A statistical evaluation of recent full reference image quality assessment algorithms, IEEE Trans. Image Process., vol. 15, no. 11, pp , Nov Lin Zhang (M 11) received the B.S. and M.S. degrees from the Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, in 2003 and 2006, respectively, and the Ph.D. degree from the Department of Computing, Hong Kong Polytechnic University, Hong Kong, in In 2011, he was a Research Assistant with the Department of Computing, Hong Kong Polytechnic University. He is currently an Assistant Professor with the School of Software Engineering, Tongji University, Shanghai. His current research interests include biometrics, pattern recognition, computer vision, and perceptual image/video quality assessment.

12 ZHANG et al.: VSI FOR PERCEPTUAL IQA 4281 Ying Shen (M 13) received the B.S. and M.S. degrees from Software School, Shanghai Jiao Tong University, Shanghai, China, in 2006 and 2009, respectively, and the Ph.D. degree from the Department of Computer Science, City University of Hong Kong, Hong Kong, in In 2013, she joined the School of Software Engineering, Tongji University, Shanghai, where she is currently an Assistant Professor. Her research interests include bioinformatics and pattern recognition. Hongyu Li received the B.E. degree from Tongji University, Shanghai, China, in 2000, the Ph.D. degree from the Department of Computer Science, Fudan University, Shanghai, in 2008, and the Ph.D. degree from Department of Computer Science, University of Eastern Finland, Joensuu, Finland. In 2008, he joined the School of Software Engineering at Tongji University, where he is currently an Associate Professor. His research interests include computer vision, pattern recognition, and motion analysis.