Mean Deviation Similarity Index: Efficient and Reliable Full-Reference Image Quality Evaluator

Transcription

1 1 Mean Deviation Similarity Index: Effiient and Reliable Full-Referene Image Quality Evaluator Hossein Ziaei Nafhi, Atena Shahkolaei, Rahid Hedjam, and Mohamed Cheriet, Senior Member, IEEE two images of the same dynami range (usually low dynami range) [] or different dynami ranges [3], [4]. This paper is dediated to the IQA for low dynami range images. Among IQAs, the onventional metri mean squared error (MSE) and its variations are widely used beause of their simpliity. However, in many situations, MSE does not orrelate with the human pereption of image fidelity and quality [5]. Beause of this limitation, a number of IQAs have been proposed to provide better orrelation with the HVS [], [6] [4]. In general, these better performing metris measure strutural information, luminane information and ontrast information in the spatial and frequeny domains. The most suessful IQA models in the literature follow a top-down strategy [5]. They alulate a similarity map and use a pooling strategy that onverts the values of this similarity map into a single quality sore. For example, the luminane, ontrast and strutural information onstitute a similarity map for the popular SSIM index []. SSIM then uses average pooling to ompute the final similarity sore. Different feature maps are used in the literature for alulation of this similarity map. Feature similarity index (FSIM) uses phase ongrueny and gradient magnitude features. GS [17] uses a ombination of some designated gradient magnitudes and image ontrast for this end, while the GMSD [19] uses only the gradient magnitude. SR SIM [18] uses salieny features and gradient magnitude. VSI [1] likewise benefits from salieny-based features and gradient magnitude. SVD based features [6], features based on the Riesz transform [11], features in the wavelet domain [10], [7], [8] and sparse features [0] are used as well in the literature. Among these features, gradient magnitude is an effiient feature, as shown in [19]. In ontrast, phase ongrueny and visual salieny features in general are not fast enough features to be used. Therefore, the features being used play a signifiant role in the effiieny of IQAs. As we mentioned earlier, the omputation of the similarity map is followed by a pooling strategy. The state-of-the-art pooling strategies for pereptual image quality assessment (IQA) are based on the mean and the weighted mean [], [7], [11], [15] [17]. They are robust pooling strategies that usually provide a moderate to high performane for different IQAs. Minkowski pooling [9], loal distortion pooling [1], [9], [30], perentile pooling [31] and salieny-based pooling [18], [1] are other possibilities. Standard deviation (SD) pooling was also proposed and suessfully used by GMSD [19]. The image gradients are sensitive to image distortions. Different loal strutures in a distorted image suffer from different degrees of degradations. This is the motivation that the authors in [19] used to explore the standard variation of the gradientarxiv: v4 [s.cv] 19 Apr 017 Abstrat Appliations of pereptual image quality assessment (IQA) in image and video proessing, suh as image aquisition, image ompression, image restoration and multimedia ommuniation, have led to the development of many IQA metris. In this paper, a reliable full referene IQA model is proposed that utilize gradient similarity (GS), hromatiity similarity (CS), and deviation pooling (DP). By onsidering the shortomings of the ommonly used GS to model human visual system (HVS), a new GS is proposed through a fusion tehnique that is more likely to follow HVS. We propose an effiient and effetive formulation to alulate the joint similarity map of two hromati hannels for the purpose of measuring olor hanges. In omparison with a ommonly used formulation in the literature, the proposed CS map is shown to be more effiient and provide omparable or better quality preditions. Motivated by a reent work that utilizes the standard deviation pooling, a general formulation of the DP is presented in this paper and used to ompute a final sore from the proposed GS and CS maps. This proposed formulation of DP benefits from the Minkowski pooling and a proposed power pooling as well. The experimental results on six datasets of natural images, a syntheti dataset, and a digitally retouhed dataset show that the proposed index provides omparable or better quality preditions than the most reent and ompeting state-of-the-art IQA metris in the literature, it is reliable and has low omplexity. The MATLAB soure ode of the proposed metri is available at Index Terms Image quality assessment, gradient similarity, hromatiity similarity, deviation pooling, syntheti image, Human visual system. I. INTRODUCTION AUTOMATIC image quality assessment (IQA) plays a signifiant role in numerous image and video proessing appliations. IQA is ommonly used in image aquisition, image ompression, image restoration, multimedia streaming, et [1]. IQA models (IQAs) mimi the average quality preditions of human observers. Full referene IQAs (FR-IQAs), whih fall within the sope of this paper, evaluate the pereptual quality of a distorted image with respet to its referene image. This quality predition is an easy task for the human visual system (HVS) and the result of the evaluation is reliable. Automati quality assessment, e.g. objetive evaluation, is not an easy task beause images may suffer from various types and degrees of distortions. FR-IQAs an be employed to ompare H. Ziaei Nafhi, A. Shahkolaei and M. Cheriet are with the Synhromedia Laboratory for Multimedia Communiation in Telepresene, Éole de tehnologie supérieure, Montreal (QC), Canada H3C 1K3; Tel.: ; Fax: ; s: hossein.zi@synhromedia.a, atena.shahkolaei.1@ens.etsmtl.a, mohamed.heriet@etsmtl.a R. Hedjam is with the Department of Geography, MGill University, 805 Sherbrooke Street West, Montreal, QC H3A K6, Canada ( rahid.hedjam@mgill.a)

2 based loal similarity map for overall image quality predition. In general, features that onstitute the similarity map and the pooling strategy are very important fators in designing high performane IQA models. Here, we propose an IQA model alled the mean deviation similarity index (MDSI) that shows very good ompromise between predition auray and model omplexity. The proposed index is effiient, effetive and reliable at the same time. It also shows onsistent performane for both natural and syntheti images. The proposed metri follows a topdown strategy. It uses gradient magnitude to measure strutural distortions and use hrominane features to measure olor distortions. These two similarity maps are then ombined to form a gradient-hromatiity similarity map. We then propose a novel deviation pooling strategy and use it to ompute the final quality sore. Both image gradient [16], [17], [19], [1], [3], [33] and hrominane features [16], [1] have been already used in the literature. The proposed MDSI uses a new gradient similarity whih is more likely to follow HVS. Also, MDSI uses a new hromatiity similarity map whih is effiient and shows good performane when used with the proposed metri. The proposed index uses the summation over similarity maps to give independent weights to them. Also, less attention has been paid to the deviation pooling strategy, exept for a speial ase of this type of pooling, namely, standard deviation pooling [19]. We therefore provide a general formulation for the deviation pooling strategy and show its power in the ase of the proposed IQA model. In the following, the main ontributions of the paper as well as its differenes with respet to the previous works are briefly explained. Unlike previous researhes [16], [17], [19], [1], [3], [33] that use a similar gradient similarity map, a new gradient similarity map is proposed in this paper whih is more likely to follow the human visual system (HVS). This statement is supported by visual examples and experimental results. This paper proposes a new hormatiity similarity map with the following advantages over the previously used hromatiity similarity maps [16], [1]. Its omplexity is lower and it provides slightly better quality preditions when used with the proposed metri. Motivated by a previous study that proposed to use standard deviation pooling [19], we propose a systemati and general formulation of the deviation pooling whih has a omprehensive sope. The rest of the paper is organized as follows. The proposed mean deviation similarity index is presented in setion II. Extensive experimental results and disussion on six natural datasets, a syntheti dataset, and a digitally retouhed dataset are provided in setion III. Setion IV presents our onlusions. II. MEAN DEVIATION SIMILARITY INDEX The proposed IQA model uses two similarity maps. Image gradient, whih is sensitive to strutural distortions, is used as the main feature to alulate the first similarity map. Then, olor distortions are measured by a hromatiity similarity map. These similarity maps are ombined and pooled by a proposed deviation pooling strategy. In this paper, onversion to luminane is done through the following formula: L = 0.989R G B. In addition, two hromatiity hannels of a Gaussian olor model [34] are used: [ H = M] A. Gradient Similarity ( ) R G (1) B It is very ommon that gradient magnitude in the disrete domain is alulated on the basis of some operators that approximate derivatives of the image funtion using differenes. These operators approximate vertial G y (x) and horizontal G x (x) gradients of an image f(x) using onvolution: G x (x) = h x f(x) and G y (x) = h y f(x), where h x and h y are horizontal and vertial gradient operators and denotes the onvolution. The first derivative magnitude is defined as G(x) = G x(x) + G y(x). The Sobel operator [35], the Sharr operator, and the Prewitt operator are ommon gradient operators that approximate first derivatives. Within the proposed IQA model, these operators perform almost the same. Through this paper, Prewitt operator is used to ompute gradient magnitudes of luminane L hannels of referene and distorted images, R and D. From whih, gradient similarity (GS) is omputed by the following SSIM indued equation: GS(x) = G R(x)G D (x) + C 1 G R (x) + G D (x) + C 1 where, parameter C 1 is a onstant to ontrol numerial stability. The gradient similarity (GS) is widely used in the literature [16], [17], [19], [1], [3], [33] and its usefulness to measure image distortions was extensively investigated in [19]. In many senarios, human visual system (HVS) disagrees with the judgments provided by the GS for strutural distortions. In fat, in suh a formulation, there is no differene between an added edge to or a removed edge from the distorted image with respet to the referene image. An extra edge in D bring less attention of HVS if its olor is lose to the relative pixels of that edge in R. Likewise, HVS pays less attention to a removed edge from R that is replaed with pixels of the same or nearly the same olor. In another senario, suppose that edges are preserved in D but with different olors than in R. In this ase, GS is likely to fail at providing a good judgment on the edges. These shortomings of the GS motivated us to propose a new GS map. B. The Proposed Gradient Similarity The aforementioned shortomings of the onventional gradient similarity map (equation ) are mainly beause G R and G D are omputed independent of eah other. In the following, we propose a fusion tehnique to inlude the orrelation between R and D images into omputation of the gradient similarity map. ()

3 3 Fig. 1. R D (JPEG ompression) GS CS GCS (α = 0.7) R D (olor saturation) GS CS GCS (α = 0.7) maps. Complementary behavior of the gradient similarity (GS) and hromatiity similarity (CS) We fuse the luminane L hannels of the R and D by a simple averaging: F = 0.5 (R + D). Two extra GS maps are omputed as follows: GSRF (x) = GR (x)gf (x) + C GR (x) + GF (x) + C GD (x)gf (x) + C GSDF (x) = GD (x) + GF (x) + C (3) HR (x)hd (x) + MR (x)md (x) + C3 CS(x) = (4) where, GF is the gradient magnitude of the fused image F, and C is used for numerial stability. Note that GF 6= (GR + GD )/, and that GSRF (x) and GSDF (x) an or an not be is omputed by: equal. The proposed gradient similarity (GS) GS(x) = GS(x) + GSDF (x) GSRF (x). (5) The added term GSDF (x) GSRF (x), will put more emphasize on removed edges from R than added edges to the D. For weak added/removed edges, it is likely that weak edges smooth out in F. Therefore, GSDF (x) GSRF (x) always put less emphasize on weak edges. at this Comparing visually some outputs of the GS and GS step might not be fair beause they have different numerial might have sales. GS is bounded between 0 and 1, while GS negative values greater than -1, and/or positive values smaller than +. Therefore, this omparison is performed on the final similarity map and is presented in subsetion II-E as well as works. more explanation on how the proposed GS C. Chromatiity Similarity For the ase of olor hanges and espeially when the struture of the distorted image remains unhanged, the gradient may lead to inaurate similarity (GS) and the proposed GS quality preditions. Therefore, previous researhes suh as [16], [1] used a olor similarity map to measure olor differenes. Let H and M denote two hromatiity hannels regardless of the type of the olor spae. In [16], [1], for eah hannel a olor similarity is omputed and their result is ombined as: CS(x) = where C3 is a onstant to ontrol numerial stability. In this paper, we propose a new formulation to alulate olor similarity. The proposed formulation alulates a olor similarity map using both hromatiity hannels at one: HR (x)hd (x) + C3 MR (x)md (x) + C3 (6) HR (x) + HD (x) + C3 MR (x) + MD (x) + C3 (x) + C3 (x) + MD (x) + MR (x) + HD HR (7) Similar to the CS in equation (6), the above joint olor formulation gives equal weight to both hrosimilarity (CS) is more matiity hannels H and M. It is lear that CS effiient than CS. CS needs 7 multipliations, 6 summations, divisions, and shift operations (multipliations by ), needs 6 multipliations, 6 summations, 1 division, while CS and 1 shift operation. Note that CS an also be omputed through 8 multipliations, 6 summations, 1 division, and shift operations. In experimental results setion, an experiment along is onduted to ompare usefulness of the CS and CS with the proposed metri. an be ombined The gradient similarity maps (GS or GS) with the joint olor similarity map CS through the following summation (weighted average) sheme: GCS(x) = αgs(x) + (1 α)cs(x) (8) d GCS(x) = αgs(x) + (1 α)cs(x) (9) where the parameter 0 α 1 adjusts the relative importane of the gradient and hromatiity similarity maps. The proposed metri MDSI uses equation (9). Equation (8) is inluded to be ompared with equation (9). An alternative ombination sheme whih is very popular in state-of-the-art is through γ multipliation in the form of [GS(x)] [CS(x)]β, where the parameters γ and β are used to adjust the relative importane of the two similarity maps. For several reasons, the proposed index uses the summation sheme (refer to subsetion III-E). In Fig 1, two examples are provided to show that these are omplementary. In two similarity maps, e.g. GS and CS, the first example, there is a onsiderable differene between the gradient maps of the referene and the distorted images. Hene, the GS map is enough for a good judgment. However, this differene in the seond example (seond row) is trivial,

4 4 whih leads to a wrong predition by using GS as the only similarity map. The examples in Fig 1 show that the gradient similarity and hromatiity similarity are omplementary. D. Deviation Pooling The motivation of using the deviation pooling is that HVS is sensitive to both magnitude and the spread of the distortions aross the image. Other pooling strategies suh as Minkowski pooling and perentile pooling adjust the magnitude of distortions or disard the less/non distorted pixels. These pooling strategies and the mean pooling do not take into aount the spread of the distortions. It is shown in [19] by ase examples and experimental results that a ommon wrong predition by mean pooling is where it alulates the same quality sores for two distorted images of different type. In suh ases, deviation pooling is likely to provide good judgments over their quality through spread of the distortions. This is the reason why mean pooling have good inter-lass (one distortion type) quality predition but its performane might be degraded for intralass (whole dataset) quality predition. While this statement an be verified from the experimental results provided in [19], an example is also provided in subsetion III-D to support this statement. Human visual system penalizes severer distortions muh more than the distortion-free regions, and these pixels may onstitute different frations of distorted images. Mean pooling, however, depending on this fration, is likely to nullify the impat of the severer distortions by inlusion of distortion-free regions into the average omputation. Fig. shows overlapped histograms of two similarity maps orresponding to two distorted images. While mean pooling indiate that image #1 is of better quality than image # (µ 1 > µ ), deviation pooling provides an opposite assessment (σ 1 > σ ). Given that µ 1 > µ, and that image #1 has more severe distortions ompared to image # with their values farther from µ 1 than µ, there are larger deviations in similarity map of image #1 than that of image #. Therefore, deviation pooling is an alternative to the mean pooling that an also measure different levels of distortions. In the following, we propose the deviation pooling (DP) strategy and provide a general formulation of this pooling. DP for IQAs is rarely used in Fig.. Overlapped histograms of two similarity maps orresponding to two distorted images. Lower values of similarity maps indiate to more severe distortions, while higher values refer to less/non distorted pixels. the literature, exept the standard deviation used in GMSD [19], whih is a speial ase of DP. A deviation an be seen as the Minkowski distane of order ρ between vetor x and its MCT (Measure of Central Tendeny): ( 1 DP (ρ) = N N xi MCT ρ) 1/ρ. (10) i=1 where ρ 1 indiates the type of deviation. The only MCT that is used in this paper is mean. Though other MCTs suh as median and mode an be used, we found that these MCTs do not provide satisfatory quality preditions. Several researhes shown that more emphasis on the severer distortions an lead to more aurate preditions [9], [31]. The Minkowski pooling [9] and the perentile pooling [31] are two examples. As mentioned before, these pooling strategies follow a property of HVS that penalize severer distortions muh more than the less distorted ones even though they onstitute a small portion of total distortions. Hene, they try to moderate the weakness of the mean pooling through adjusting magnitudes of distortions [9] or disarding the less/non distorted regions [31]. The deviation pooling an be generalized to onsider the aforementioned property of HVS: ( 1 DP (ρ,q) = N N x q i MCT ρ) 1/ρ. (11) i=1 where, q adjusts the emphasis of the values in vetor x, and MCT is alulated through x q i values. Furthermore, we propose to use power pooling in onjuntion with the deviation pooling to ontrol numerial behavior of the final quality sores: [ ( 1 N DP (ρ,q,o) = x q i N MCT ρ) ] 1/ρ o. (1) i=1 where, o is the power pooling applied on the final value of the deviation. The power pooling an be used to make an IQA model more linear versus the subjetive sores or might be used for better visualization of the sores. Linearity might not be a signifiant advantage of an IQA, but it is pointed to be of interest in [19]. Also, aording to [36], linearity against subjetive data is one of the measures for validation of IQAs that should be examined 1. The power pooling an also have small impat on the values of Pearson linear Correlation Coeffiient (PCC) and Root Mean Square Error (RMSE). Note that the above deviation pooling is equal to the Minkowski pooling [9] when MCT = 0, ρ = 1 and o = 1. It is equal to the mean absolute deviation (MAD) to the power of o for ρ = 1, and equal to the standard deviation (SD) to the power of o for ρ =. The three parameters should be set aording to the IQA model. More analysis on these three parameters an be found in experimental results setion. For the proposed index MDSI, we set ρ = 1, q = 1 4 and o = 1 4. Therefore, the proposed IQA model an be written as: [ 1 N MDSI = ĜCS 1/4 ( 1 N i N N i=1 i=1 ] ĜCS 1/4 ) 1/4 i. (13) Note that possible interval for ĜCS is [0 δ 1 1+δ ], where δ 1 < 1 and δ < 1. It is worth to mention that values of ĜCS 1 Though linearity is measured after a nonlinear analysis.

5 5 R D L R L D L F G R G D G F GS RD (GS) GS DF GS RF GCS ĜCS GCS 1/4 ĜCS 1/4 Fig. 3. The differene between similarity maps GCS and ĜCS that use onventional gradient similarity and the proposed gradient similarity, respetively. mostly remain in [0 1]. Also, ĜCS < 0 are highly distorted pixels, while ĜCS > (1 ɛ) refer to less/non-distorted pixels, where ɛ < 1 is a very small number. The global variations of ĜCS 1/4 is omputed by mean absolute deviation, whih is followed by power pooling. Note that sine absolute of deviations is omputed, the quality sores are positive. Larger values of the quality preditions provided by the proposed index indiate to the more severe distorted images, while an image with perfet quality is assessed by a quality sore of zero sine there is no variation in its similarity map. The important point on the use of the Minkowski pooling on final similarity maps is that terms like more emphasize and less emphasize, regardless of the q values have been used, depends also on the pooling strategy and underlying similarity map. For example, plaing more emphasize on highly distorted regions by Minkowski pooling will derease the quality sore omputed by the mean pooling, but the quality sore provided by the deviation pooling might beome larger or smaller depending on the spread of the distortions whih is diretly related to the underlying similarity map. E. Analysis and Examples of GCS Maps In this setion, final similarity maps after applying the Minkowski pooling, e.g. GCS 1/4 and ĜCS1/4, are ompared along with suffiient explanations. The differene between these two similarity map is their use of gradient similarity. GCS uses onventional GS, while ĜCS uses the proposed gradient similarity ĜS. The best way to analyze the effet of the proposed gradient similarity is through step by step explanation and visualization of different examples. In subsetion II-A, several disadvantages of the traditional GS was mentioned. Here, eah of them are explained and examples are provided. a) Case 1 (Removed edge): Missing edges in distorted image with respet to its original image means that strutural information are removed, hene this disappearane bring attention of the HVS. These regions has to be strongly highlighted in the similarity map. b) Case (A weak added/removed edge): An extra edge in D or a removed edge from R bring less attention of HVS if its olor is lose to the relative pixels of that edge in R (D), or simply it is a weak edge. Fig. 3 shows how the proposed gradient similarity map ĜS performs for ase 1 and ase as a part of the ĜCS ompared to the GS for GCS. We an see that GS RD (GS) highlighted differenes with details. The edges orresponding to the loation of ropes in original image are mainly replaed with pixels of another olor (dark replaed with green), but many other edges with smaller strengths in R are replaed with pixels having the same olor (green). This latter holds for added edges to the distorted image. In fused image (L F ), some of these weaker edges are smoothed. This an be seen by omparing GS RD and GS DF. Both GS RD and GS DF indiate to high differenes at the loation of the ropes. GS RD + GS DF will also put high emphasize on this loation, but less emphasize on the weaker edges. The results is then subtrated by GS RF whih in turn again less emphasize is plaed on the weak edges (relevant to the darker pixels in GS RF ). Note that GCS and ĜCS have different numerial behavior, so it is fair to ompare them by looking at the GCS 1/4 and ĜCS1/4. Compared to the GCS 1/4, ĜCS 1/4 indiate to larger differenes at the loation of ropes, but smaller differenes elsewhere. ) Case 3 (Preserved edge but with different olor): Although a olor similarity map should measure olor differenes at the loation of the inverted edges, edges onstitute a small fration of total pixels in images, and it is ommon to give smaller weights to a olor similarity map than strutural sim-

6 6 5 syntheti images of video games and animated movies. It ontains 500 distorted images of 5 ategories. Fig. 5 shows an example of a referene and a distorted syntheti image. Finally, the digitally retouhed image quality (DRIQ) dataset [43] was used in the experiments. It ontains 6 referene images and 3 enhaned images for eah referene image. R D GSRD GSDF R GSRF (GSDF GSRF ) GCS1/4 d 1/4 GCS d 1/4 for Fig. 4. The differene between similarity maps GCS1/4 and GCS the ase of the inverted edges. Note that some intermediate outputs are not shown. ilarities suh as gradient similarity. While traditional gradient similarity does not work well in this situation, the proposed gradient similarity an partially solve this problem. Fig. 4 provides an example in whih most of the edges are inverted 1/4 d in the distorted image. We an see that GCS highlighted muh more differenes than GCS1/4 at these loations, thanks to the added term (GSDF GSRF ) to the traditional gradient similarity. In fat, GSDF is likely to be different than GSRF in this ase beause these edges in F are likely to beome loser to their surrounding pixels in either R or D images. III. E XPERIMENTAL RESULTS AND DISCUSSION In the experiments, eight datasets were used. The LIVE dataset [37] ontains 9 referene images and 779 distorted images of five ategories. The TID008 [38] dataset ontains 5 referene images and 1700 distorted images. For eah referene image, 17 types of distortions of 4 degrees are available. CSIQ [1] is another dataset that onsists of 30 referene images; eah is distorted using six different types of distortions at four to five levels of distortion. The large TID013 [39] dataset ontains 5 referene images and 3000 distorted images. For eah referene image, 4 types of distortions of 5 degrees are available. VCL@FER database [40] onsists of 3 referene images and 55 distorted images, with four degradation types and six degrees of degradation. In addition to these five datasets, ontrast distorted images of the CCID014 dataset [41] are used in the experiments. This dataset ontains 655 ontrast distorted images of five types. Gamma transfer, onvex and onave ars, ubi and logisti funtions, mean shifting, and a ompound funtion are used to generate these five types of distortions. We also used the ESPL syntheti image database [4] whih ontains D (Gaussian noise) Fig. 5. An example of referene R and distorted D image in the ESPL syntheti images database [4]. For objetive evaluation, four popular evaluation metris were used in the experiments: the Spearman Rank-order Correlation oeffiient (SRC), the Pearson linear Correlation Coeffiient (PCC) after a nonlinear regression analysis (equation 14), the Kendall Rank Correlation oeffiient (KRC) and the Root Mean Square Error (RMSE). The SRC, PCC, and RMSE metris measure predition monotoniity, predition linearity, and predition auray, respetively. The KRC was used to evaluate the degree of similarity between quality sores and MOS. In addition, Pearson linear Correlation Coeffiient without nonlinear analysis is used and denoted by LPCC. Twelve state-of-the-art IQA models were hosen for omparison [7], [9], [1], [15], [16], [18] [3], [44] inluding the most reent indies in literature [19] [3], [44]. It should be noted that the five indies SFF [0], GMSD [19], VSI [1], [], and SCQI [3] have shown superior performane over state-of-the-art indies. A. Performane omparison In Table I, the overall performane of thirteen IQA models on eight benhmark datasets, e.g. TID008, CSIQ, LIVE, TID013, VCL@FER, CCID014, ESPL, and DRIQ, is listed. For eah dataset and evaluation metri, the top three IQA models are highlighted. On eight datasets, MDSI is 31 times among the top indies, followed by ADD-GSIM (16 times), SCQI (1 times), SFF/FSIM /VIF (6 times), VSI (5 times), GMSD/SR SIM/MAD (4 times), DSCSI ( times), and MSSSIM/IWSSIM (0 times). To provide a onlusion on the overall Note the onflit between MAD [1] as an IQA model, and MAD as a pooling strategy.

7 7 TABLE I PERFORMANCE COMPARISON OF THE PROPOSED IQA MODEL, MDSI, AND TWELVE POPULAR/COMPETING INDICES ON EIGHT BENCHMARK DATASETS. NOTE THAT TOP THREE IQA MODELS ARE HIGHLIGHTED. MSSSIM VIF MAD IWSSIM SR SIM FSIM GMSD SFF VSI DSCSI ADD-GSIM SCQI MDSI SRC TID PCC KRC RMSE SRC CSIQ PCC KRC RMSE SRC LIVE PCC KRC RMSE SRC TID PCC KRC RMSE SRC VCL@ PCC FER KRC RMSE SRC CCID PCC KRC RMSE SRC ESPL PCC KRC RMSE SRC DRIQ PCC KRC RMSE SRC Diret PCC Avg. KRC Weighted Avg. SRC PCC KRC performane of these indies, diret and weighted 3 overall performanes on the eight datasets (8150 images) are also listed in Table I. It an be seen that MDSI has the best overall performane on the eight datasets, while metris VSI and SCQI are the seond, and third best, respetively. B. Visualization and statistial evaluation For the purpose of visualizing quality sores of the proposed index, the satter plots of the proposed IQA model MDSI with and without using power pooling are shown in Fig. 6. The logisti funtion suggested in [45] was used to fit a urve on eah plot: f(x) = β 1 ( e β(x β3) ) + β 4 x + β 5 (14) where β 1, β, β 3, β 4 and β 5 are fitting parameters omputed by minimizing the mean square error between quality preditions x and subjetive sores MOS. It should be noted that reported PCC and RMSE values in this paper are omputed after mapping quality sores to MOS based on above funtion. The reported results in Table I show the differene between different IQA models. As suggested in [45], [46], we use F-test to deide whether a metri is statistially superior to 3 The dataset size-weighted average is ommonly used in the literature [15], [19] [1]. LPCC = PCC = LPCC = PCC = Fig. 6. Satter plots of quality sores against the subjetive MOS on the LIVE dataset for the proposed model MDSI with and without using the power pooling. Comparison of LPCC and PCC values indiate that MDSI beomes more linear with respet to MOS (the right plot) by using the power pooling. another index. The F-test is based on the residuals between the quality sores given by an IQA model after applying nonlinear mapping of equation (14), and the mean subjetive sores MOS. The ratio of variane between residual errors of an IQA model to another model at 95% signifiane level is used by F-test. The result of the test is equal to 1 if we an rejet the null hypothesis and 0 otherwise. The results of F-test on eight datasets are listed in Table II. In this Table, +1/-1 indiate that orresponding index is statistially superior/inferior to the other index being ompared to. If the differene between two indies is not signifiant, the result is shown by 0. We note that

8 8 TABLE II THE RESULTS OF STATISTICAL SIGNIFICANCE TEST FOR TEN IQA MODELS ON EIGHT DATASETS. THE RESULT OF THE F-TEST IS EQUAL TO +1 IF A METRIC IS SIGNIFICANTLY BETTER THAN ANOTHER METRIC, IT IS EQUAL TO -1 IF THAT METRIC IS STATISTICALLY INFERIOR TO ANOTHER METRIC, AND THE RESULT IS EQUAL TO 0 IF TWO METRICS ARE STATISTICALLY INDISTINGUISHABLE. THE CUMULATIVE SUM OF INDIVIDUAL TESTS FOR EACH METRIC IS LISTED IN THE LAST COLUMN WITH TOP THREE IQA MODELS BEING HIGHLIGHTED IN THE SAME COLUMN. TID sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI LIVE sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI ESPL sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI CSIQ sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI TID sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI CCID sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI DRIQ sum 1 VIF MAD SR SIM FSIM GMSD SFF VSI ADD-GSIM SCQI MDSI type I error might be ourred, speially when quality sores of IQA models are not Gaussian. However, even existene of possible errors is very unlikely to result in another onlusion about the superiority of the proposed index beause there is a onsiderable gap between the proposed index and the other metris as disussed in the following. From the results of Table II, we an see that MDSI is signifiantly better than the other indies on TID008, TID013, ESPL, and DRIQ datasets. Therefore, its sum value in the last olumn is +9 for these four datasets. SCQI is statistially superior to the other indies on the TID013 dataset exept for MDSI. On the LIVE dataset, indies MAD, MDSI, and ADD-GSIM are signifiantly better than the other indies. On the CSIQ dataset, only SFF performs signifiantly better than MDSI. On the CCID014 dataset, ADD-GSIM is signifiantly better than the other indies, while the statistially indistinguishable indies VIF and MDSI show promising results. Considering all eight datasets used in this experiment, with a minimum sum value of +6, the proposed index MDSI performs very well in omparison with the other indies. We an simply add the eight umulative sum values of eah metri for the eight datasets to have an overall omparison based on the statistial signifiane test. This sore indiates how many times a metri is statistially superior to the other metris. The results show that MDSI is the best performing index by a sore of +6 (out of maximum +7), followed by ADD-GSIM (+5), VSI (+1), SCQI (-), FSIM (-4), GMSD (-5), SFF (-6), SR SIM (-19), MAD (-4), and VIF (-6). The results based on the statistial signifiane test verify that unlike other IQA models, the proposed metri MDSI is among the best performing indies on different datasets. C. Performane omparison on individual distortions A good IQA model should perform not only aurate quality preditions for a whole dataset; it should provide good judgments over individual distortion types. We list in Table III the average SRC, and PCC values of thirteen IQA models for 61 sets of distortions available in the six datasets of TID008, CSIQ, LIVE, TID013, VCL@FER, and ESPL. The minimum value for eah evaluation metri and standard deviation of these 61 values are also listed. These two evaluations indiate to the reliability of an IQA model. An IQA model should provide good predition auray for all of the distortion types. If a metri fails at assessing one or more types of distortions, that index an not be reliable. The proposed index MDSI, has the best SRC, and PCC average on distortion types. MDSI, SCQI and FSIM in the worst ase perform better than the other IQA models, as an be seen in the min olumn for eah evaluation metri. This shows the reliability of the proposed index. The negative min values and lose to zero min values in Table III indiate the unreliability of related models when dealing with some

9 9 distortion types. The standard deviation of 61 values for eah evaluation metri is another reliability fator. Aording to Table III, MDSI, SCQI and FSIM have the lowest variation. Therefore, we an onlude that indies MDSI, SCQI and FSIM are more reliable than the other indies. TABLE III OVERALL PERFORMANCE COMPARISON OF THE PROPOSED IQA MODEL MDSI AND TWELVE POPULAR/COMPETING INDICES ON INDIVIDUAL DISTORTION TYPES OF SIX DATASETS (TID008, CSIQ, LIVE, TID013, AND ESPL). THE SIX DATASETS CONTAIN 61 DISTORTION SET, THEREFORE RESULTS ON DISTORTION TYPES ARE REPORTED BASED ON AVERAGE OF 61 CORRELATION VALUES. TOP THREE IQA MODELS ARE HIGHLIGHTED. IQA model SRC (Distortions) PCC (Distortions) avg min std avg min std MSSSIM VIF MAD IWSSIM SR SIM FSIM GMSD SFF VSI DSCSI ADD-GSIM SCQI MDSI TABLE IV PERFORMANCE OF THE PROPOSED INDEX MDSI WITH DIFFERENT POOLING STRATEGIES AND VALUES OF PARAMETER q. Pooling Weighted avg. SRC (8 datasets) Avg. SRC (61 Distortions) Mean MAD SD Mean MAD SD q = 1/ q = 1/ q = q = q = D. Parameters of deviation pooling (ρ, q, o) Considering the formulation of deviation pooling in equation (1), we used the mean absolute deviation (MAD), e.g. ρ = 1, for the proposed metri. Standard deviation (SD), e.g. ρ =, is another option that an be used for deviation pooling. In addition, the Minkowski power (q) of the deviation pooling an have signifiant impat on the proposed index. In Table IV, the SRC performane of the proposed index is analyzed for different values of q and ρ = {1, }. Mean pooling is also used in this experiment. The results show that MAD pooling with q <= 1 is a better hoie for the proposed index. Also, the performane of the mean pooling on 61 distortion set onfirms our statement that mean pooling has a good performane for inter-lass quality predition. The impat of the proposed power pooling of the deviation pooling on the proposed metri was shown in Fig. 6. Power pooling an be also used to inrease linearity of other indies as well. For example, LPCC and PCC values of VSI [1] for TID013 dataset, by setting o = 18, an be inreased from to 0.898, and to , respetively. E. Summation vs. Multipliation Two options for ombination of the two similarity maps GS/ĜS and ĈS are summation and multipliation as explained in subsetion II-C. Deiding whether one approah is superior to another for an index depends on many fators. These fators might be the pooling strategy being used, overall performane, performane on individual distortions, reliability, effiieny, simpliity, et. In an experiment, the performane of the MDSI using the multipliation approah was examined. Based on the many set of parameters were tested, we found that γ = 0. and β = 0.1 are good parameters to ombine ĜS and ĈS via the multipliation sheme. The observation was that summation is a better hoie for TID008, TID013, VCL@FER, and DRIQ datasets, while multipliation is a better hoie for ESPL dataset, and that both approahes show almost the same performane on other datasets. Overall, the summation approah provides better performane on individual distortions. This experiment also shown that MDSI is more reliable through summation than multipliation based on the reliability measures introdued in this paper. Based on this experiment, the simpliity of the summation ombination approah and its effiieny over multipliation, the former was used along with MDSI. Table V justifies our hoie. TABLE V DIFFERENT CRITERIA USED TO CHOOSE THE COMBINATION SCHEME. Property Summation Multipliation Statistially superior over more onsidered datasets Better dataset-weighted average Better performane on individual distortions Reliability Simpliity Effiieny F. Parameters of model The proposed IQA model MDSI has four parameters to be set. The four parameters of MDSI are C 1, C, C 3 and α. To further simplify the MDSI, we set C 3 = 4C 1 = 10C. Therefore, MDSI has only two parameters to set, e.g. C 3 and α. For an example, we refer to the SSIM index [] that also uses suh a simplifiation. Note that gradient similarities and hromatiity similarity have different dynami ranges, therefore, these parameters should be set suh that the relation between these maps also be taken into aount. In Fig. 7, the impat of these two parameters on the performane of the MDSI is shown. Even though the parameters C 1, C and C 3 are set approximately, it an be seen that MDSI is very robust under different setup of parameters. MDSI has greater weighted average SRC than 0.90 for any α [ ] and C 3 [ ]. Note that many other possible setup of parameters are not inluded in this plot. In the experiments, we set α = 0.6, C 1 = 140, C = 55, and C 3 = 550. G. Effet of hromatiity similarity maps CS and ĈS In this setion, the impat of using CS [1] and proposed ĈS on the performane of the proposed index is studied through the following experiment. Contrast distorted images of the CCID014 dataset [41] were hosen. The reason of hoosing this dataset is to evaluate the ability of measuring olor hanges by CS and ĈS. We analyzed the SRC performane of the CS and ĈS as a part of the proposed index for wide

10 10 Weighted SRC α = 0.5 α = 0.6 α = C 3 Fig. 7. The weighted SRC performane of MDSI for different values of C 3 and α on eight datasets (TID008, CSIQ, LIVE, TID013, VCL@FER, CCID014, ESPL, and DRIQ). range of C 3 values. Three pooling strategies were used in this experiment, e.g. mean pooling, mean absolute deviation (MAD) pooling and the standard deviation (SD) pooling. Fig. 8 shows the SRC performane of the proposed index for different senarios. From the plot in Fig. 8, the following onlusions an be drawn. MAD pooling and both CS and ĈS are good hoies for MDSI. For almost every pooling strategy and parameter of C 3, the proposed ĈS performs better than CS. This advantage is at the same time that the proposed ĈS is more effiient than the existing CS fifteen IQA models when applied on images of size and The experiments were performed on a Corei7 3.40GHz CPU with 16 GB of RAM. The IQA models were implemented in MATLAB 013b running on Windows 7. It an be seen that MDSI is among top five fastest indies. The proposed index is less than times slower than the ompeting GMSD index. The reason for this is that GMSD only uses the luminane feature. Compared to the other ompeting indies, SCQI, VSI, ADD-GSIM, SFF, and FSIM, the proposed index MDSI is about 3 to 6 times, 3 to 9 times, 4 to 5 times, 4 to 5 times, and 4 to 11 times faster, respetively. Another observation from the Table VI is that the ranking of indies might not be the same when they are tested on images of different size. For example, SSIM performs slower than the proposed index on smaller images, but faster on larger images. TABLE VI RUN TIME COMPARISON OF IQA MODELS IN TERMS OF MILLISECONDS IQA model PSNR GMSD [19] MDSI SSIM [] SR SIM [18] MSSSIM [7] ADD-GSIM [] SFF [0] SCQI [3] VSI [1] FSIM [16] IWSSIM [15] DSCSI [44] VIF [9] MAD [1] SRC Mean_CS (pro) MAD_CS (pro) SD_CS (pro) 0.76 Mean_CS MAD_CS SD_CS C 3 Fig. 8. The SRC performane of the proposed index MDSI with two hromatiity similarity maps CS and ĈS (proposed) for different values of C 3 and three pooling strategies on CCID014 dataset [41]. IV. CONCLUSION We propose an effetive, effiient, and reliable full referene IQA model based on the new gradient and hromatiity similarities. The gradient similarity was used to measure loal strutural distortions. In a omplementary way, a hromatiity similarity was proposed to measure olor distortions. The proposed metri, alled MDSI, use a novel deviation pooling to ompute the quality sore from the two similarity maps. Extensive experimental results on natural and syntheti benhmark datasets prove that the proposed index is effetive and reliable, has low omplexity, and is fast enough to be used in real-time FR-IQA appliations. H. Implementation and effiieny Another very important fator of a good IQA model is its effiieny. The proposed index has a very low omplexity. It first applies average filtering of size M M on eah hannel of the R and D images, downsample them by a fator of M and onvert the results to a luminane and two hromatiity hannels [47]. The value of M is set to [min(h, w)/56] [48], where h and w are image height and width, and [.] is the round operator. Then, the proposed index alulates the gradient magnitudes of luminane hannel, the hromatiity similarity map, and apply deviation pooling. All these steps are omputationally effiient. Table VI lists the run times of ACKNOWLEDGMENTS The authors thank the NSERC of Canada for their finanial support under Grants RGPDD and RGPIN REFERENCES [1] Z. Wang, Appliations of objetive image quality assessment methods [appliations orner], IEEE Signal Proessing Magazine, vol. 8, no. 6, pp , Nov 011. [] Z. Wang, A. Bovik, H. Sheikh, and E. Simonelli, Image quality assessment: from error visibility to strutural similarity, IEEE Transations on Image Proessing, vol. 13, no. 4, pp , April 004. [3] H. Yeganeh and Z. Wang, Objetive quality assessment of tone-mapped images, IEEE Transations on Image Proessing, vol., no., pp , Feb 013.