AN IMAGE, before it is displayed to a human, is often

Transcription

1 IEEE SIGNAL PROCESSING LETTERS, VOL. 23, NO. 1, JANUARY Decision Fusion for Image Quality Assessment using an Optimization Approach Mariusz Oszust Abstract The proliferation of electronic means of communication entails distortion of visual information carried by processed images. Therefore, automatic evaluation of image perceptual quality in a way that is consistent with human perception is important. In this letter, an approach to full-reference image quality assessment (IQA) is proposed. The perceptual quality of the image is evaluated using an aggregated decision of several IQA measures. An optimization problem of designing a decision fusion of 18 IQA measures is defined and solved using a genetic algorithm. Obtained fusion strategies are evaluated on widely used large image benchmarks and compared with 32 state-of-the-art IQA approaches. Results of comparison reveal that the proposed approach outperforms other competing techniques. Index Terms Decision fusion, full-reference, genetic algorithm, image quality assessment, optimization. I. INTRODUCTION AN IMAGE, before it is displayed to a human, is often an output of many subsequent processing steps, such as acquisition, enhancement, compression, or transmission. Each step can introduce distortions to its content, and therefore it is important to be able to automatically assess the resulting image in terms of perceptual quality. Image quality assessment (IQA) measures (metrics or models) fall into three categories: (1) full-reference, (2) no-reference, and (3) reduced-reference techniques. The assignment of the measure depends on the usage of reference images along with their corresponding distorted equivalents. Over the last decade, many different full-reference IQA approaches have been introduced. They explore various image properties or model distortions, all aiming to reflect the human visual system. A thorough review of recently introduced measures can be found in, e.g., [1] or [2]. Another direction of research is to provide a better full-reference IQA measure through a fusion strategy of some approaches. Since the approach presented in this letter belongs to this category, similar and representative approaches will Manuscript received August 30, 2015; revised October 11, 2015; accepted November 12, Date of publication November 13, 2015; date of current version November 23, The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Alexandre X. Falcao. The author is with the Department of Computer and Control Engineering, Rzeszow University of Technology, Rzeszow, Poland ( marosz@kia.prz.edu.pl). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /LSP be described below. For example, authors of most apparent distortion algorithm (MAD) [3] adopted two strategies for IQA, i.e., local luminance and contrast masking assess high-quality images, and changes in the local statistics of spatial-frequency components are used for low-quality images. In an another technique, which is similar to MAD, distortions are considered as additive or detail loss based, and their measurement is combined [4]. Peng and Li [5], in turn, adopted the combination scheme introduced in MAD and improved structural similarity (SSIM) indices, combining them with an edge-quality. In the scheme proposed in [6], the image is divided into blocks and content of such blocks is classified to be either smooth, edge or texture type. A nonlinear combination of features extracted from several difference of Gaussian (DOG) frequency bands is applied in the approach proposed by Pei and Chen [7]. In other approaches, objective quality measures are combined using: adaptive weighting [8], internal generative mechanism [9], nonlinear combination [10], canonical correlation [11], support vector regression [12], regularised regression [13], or conditional Bayesian mixture of experts with support vector machines classifier [14]. In this letter, a fusion of IQA measures takes a form their weighted product. Here, weights are considered as decision variables in an optimization problem. The problem is solved using a genetic algorithm. The genetic algorithm selects appropriate IQA measures and assigns weights to them while minimizing an objective function that incorporates two evaluation indices, typically used for comparison and assessment of IQA measures. The main difference of this approach from other fusion techniques is usage of the genetic algorithm together with IQA model selection. In most approaches, the number of used models is predefined, e.g., in [10], and usually smaller than eight [13]. Since the number of developed IQA models is growing in recent years, it would be desirable if a fusion technique could select which models should combine their outputs. This can be seen as a promising advantage of the proposed approach. The rest of this letter is organised as follows. Section II covers the formulation of an optimization problem of fusion of IQA models. The development of the proposed approach is presented in Section III. Experimental results with related discussions are presented in Section IV. In this section, a set of proposed fusion strategies is evaluated on four large image benchmarks. The section also contains comparison of the approach with 32 stateof-the-art full-reference IQA measures. Finally, Section V concludes the letter and indicates possible directions of future research IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 66 IEEE SIGNAL PROCESSING LETTERS, VOL. 23, NO. 1, JANUARY 2016 II. OPTIMIZATION PROBLEM In order to reveal which IQA measures should take part in the fusion and to determine their contribution, the following optimization problem is formulated. Let be the output of the IQA model or an aggregated decision of several models. In the second case, the aggregation of IQA measures can be expressed as,where is an aggregation operator. In this letter, the weighted product is used for determination of joint decision of IQA measures. The usage of the product rule is motivated by its simplicity and sensitivity to the individual decisions that are aggregated [15], [16]. Thus, the product rule can be expressed as follows:,where denotes the vector of weights,. In this letter, represents decision variables in an optimization problem of finding a promising fusion of IQA measures. However, since many different can be proposed, an appropriate objective function should be defined. The most straightforward approach is to use one of indices typically applied for IQA measures comparison. Such comparison is performedusingspearmanrankorder Correlation Coefficient (SRCC), Kendall Rank order Correlation Coefficient (KRCC), Pearson linear Correlation Coefficient (PCC), and Root Mean Square Error (RMSE) [17]. Evaluation indices are calculated after a nonlinear mapping between a vector of objective scores,, and a vector of subjective mean opinion scores (MOS) or differential MOS (DMOS),, using the following mapping function for the nonlinear regression:,where are parameters of the regression model [17], and is the mapped equivalent of. Different values may lead to different PCC and RMSE values, therefore authors in some cases report different scores for evaluated metrics. It is worth noting that the nonlinear regression does not affect SRCC and KRCC. In this letter, the following values were used: [18]. Higher SRCC, KRCC and PCC values are considered better, in contrary to the values of RMSE. Some preliminary experiments revealed that maximization of SRCC often leads to the result with unacceptably high RMSE. Therefore, a multiobjective approach is considered with a simple objective function that incorporates SRCC and RMSE:. III. FUSION OF IQA MODELS A genetic algorithm (GA) was applied [19] in order to determine the vector of decision variables. The choice of this optimization algorithm is motivated by its ability to provide acceptable solutions for NP-hard problems. The GA works on a population of randomly generated solutions (called individuals). Solutions are improved from generation to generation, leading to better values of optimised objective function. Here, the optimised function requires the vectors of subjective opinion scores ( ) and objective scores ( ). Subjective scores obtained in tests with human subjects, together with evaluated images, are available in IQA benchmark databases. In experiments, images which belong to the following four large image benchmarks were used: TID2013 [20] (25 reference images, 3000 distorted images, 24 types of distortions), TID2008 [21] (25 reference images, 1700 distorted images, 17 types of distortions), CSIQ [3] (30 reference images, 886 distorted images, 6 types of distortions), and LIVE [22] (29 reference images, 779 distorted images, 5 types of distortions). Optimization based fusion was performed using the following 18 IQA metrics ( ): VSI [18], FSIM [23], FSIMc [23], GSM [24], IFC [25], IW-SSIM [26], MAD [3], MSSIM [27], NQM [28], PSNR [17], RFSIM [29], SR-SIM [30], SSIM [22], UQI [31], VIF [32], VSNR [33], SFF [34], and IFS [35]. In this letter, the fusion was performed using only several reference images and their corresponding distorted equivalents (20% of first images from a given database). Finally, four fusion strategies were obtained, each with the help of one database. It is worth noting that most of compared approaches used images and objective scores from benchmark databases described above for determination of their parameters, e.g., 20% in [5], 30% in [18], [36], 100% in [7], [10], [12], [14], or several datasets jointly [37]. Other works use smaller databases [9], [34] or images with distortions that are present in benchmark databases, e.g., [3], [23]. However, in order to ensure database independence, many of these techniques were evaluated on other databases, or only a small part of images from the benchmark was utilised. The fusion strategies presented in this letter were also cross-database tested from this reason. The GA was run for 100 generations, with a population of 100 individuals. After 100 runs, the best solution was selected. The size of the population and the number of generations were chosen experimentally, taking into account convergence of the mean value of individuals objective scores through the generations. In order to provide a better exploration of search space, each individual was able to perform a model selection, i.e., to decide which IQA models are aggregated. Finally, four fusions of IQA models, namely Evolutionary based Similarity Measures (ESIMs), were obtained: (1) ESIM1 on TID2013, (2) ESIM2 on TID2008, (3) ESIM3 on CSIQ, and (4) ESIM4 on LIVE. Due to the model selection approach, resulted aggregate models used only several IQA measures (see (1)). It can be seen that VSI contributed in each ESIM, and SR-SIM only for ESIMs trained on images from TID databases. The contributions of SFF, FSIMc, MSSIM and VIF are smaller, but noticeable. IV. EVALUATION Evaluation of the proposed approach is divided into two parts. In the first part, four obtained ESIMs are compared with IQA measures that were used in the optimization. Here, more thorough evaluation can be made, since objective scores of these approaches are publicly available or were obtained using their implementations. Then, in the second part, the state-of-the-art IQA measures, including fusion approaches, are compared with ESIMs based on published results. (1)

3 OSZUST: DECISION FUSION FOR IMAGE QUALITY ASSESSMENT 67 TABLE I COMPARISON OF PERFORMANCE OF THE PROPOSED APPROACH WITH MODELS THAT WERE USED IN OPTIMIZATION, THE TWO BEST MODELS FOR EACH CRITERION ARE SHOWN IN BOLDFACE TABLE II THE SUMMARY OF STATISTICAL SIGNIFICANCE TEST For evaluation and development of four developed ESIMs, 18 state-of-the-art IQA models were used. Due to space limitation, Table I presents evaluation results, in terms of SRCC, KRCC, PCC, and RMSE, only for the best eight models and PSNR, since PSNR is often used as the bottom baseline. The top two models for each criterion are shown in boldface. The table also contains direct and weighted averages of obtained values. For the weighted average, the number of images in the database is used as its weight. Results show that all ESIM measures, IFS, SFF, VSI and FSIMc are among top performing models. To evaluate the statistical significance of the performance difference between obtained IQA fusions and compared measures, hypothesis tests based on the prediction residuals of each measure after nonlinear mapping were conducted using left-tailed F-test. In the test, smaller residual variance denotes the better prediction [3]. The summary of significance tests is presented in Table II. The symbol 1, 0 or -1 denotes that the model in the row is statistically better with a confidence greater than 95%, indistinguishable, or worse than the model in the column. Tests on image benchmarks are separated by comma. It can be seen that ESIM2 significantly outperformed compared measuresontiddatabases.esim1wasonlyworsethanifsand SFF here. ESIM1 and ESIM2 also performed well on CSIQ and LIVE. ESIM3, in turn, yielded good results on LIVE and TID databases. ESIM4 is better than most compared non-esim metrics, it is better than ESIM3 on both TID benchmarks. Among other techniques, IFS, SFF, MAD and FSIM with its colour version (FSIMc) also performed well. Taking into account values of evaluation indices and significance tests, it can be said that ESIMs obtained very promising results, and future extension of the approach by adding other IQA measures to the optimization step may bring further improvements. In Table III, the approach is compared with recently introduced techniques and state-of-the-art fusion approaches. Results for VSI, IFS, SFF and MAD are presented here as well, since they also represent state-of-the-art performance. The following metrics are compared: VSI, SURF-SIM [36], GMSD [38],SFF,IFS,ESSIM[39],MAD,IGM[9],ADM[4],CQM [10], BMMF [6], RMSSIM [5], DOG-SSIM [7], MMF [12], and approaches introduced by Zhou et al. [37], Wu et al. [40], Peng and Li [14], Lahouhou et al. [13], and Barri et al. [8]. Comparison is made on the basis of SRCC values reported in referenced works, since values of PCC, KRCC and RMSE are often not available.

4 68 IEEE SIGNAL PROCESSING LETTERS, VOL. 23, NO. 1, JANUARY 2016 TABLE III COMPARISON OF THE APPROACH WITH STATE-OF-THE-ART MEASURES BASED ON SRCC The four resulting fusion models strongly rely on VSI metric (see (1)), therefore, additional experiments were run in order to show that the proposed framework is able to obtain good models without access to newly developed approaches, i.e., VSI,SFF, and IFS. This would also be a step towards fair comparison with fusion metrics that were proposed before these three measures were developed. The following four IQA models are obtained: New fusion models mostly rely on FSIM family, SR-SIM, MSSIM, and MAD. Table III contains SRCC values obtained with these models. Most approaches require training step or parameters tuning. Therefore, they cannot be fairly compared with other techniques on the dataset that was used for this purpose. For example, Wu et al. [40], Peng and Li [5], [14], or Liu et al. [12] developed different sets of parameters for different (2) image benchmarks. Zhou et al. [37], in turn, used images from several datasets, and Okarma in [10] trained and tested proposed metric on only one database. The non-independent approaches were excluded from the comparison, but remained in Table III as underlined, regardless whether they utilised the entire benchmark database or only its part. In the table, two best results are shown in boldface, two best fusion measures are written in italics, and denotes fusion approach. It can be seen that measures trained on TID2008 tend to perform poorly on TID2013, e.g., BMMF, IFS, ESIM2, or ESIM6. The main factor that affects the performance of evaluated techniques, is the number of distortions present in benchmark databases. Approaches trained on large benchmarks, which contain variety of distortions, usually perform better than techniques that are trained on smaller datasets (cf. ESIM1 and ESIM4, or ESIM5 and ESIM8). VSI and DOG-SSIM outperformed other measures on TID2013, with ESIM4 as the second best fusion approach. Many approaches were not evaluated on this dataset, therefore overall results take into consideration only TID2008, CSIQ and LIVE benchmarks. For TID2008, ESIM1 and ESIM3 are outperforming other approaches. SFF and the approach introduced by Barri et al. turned out to be the best on CSIQ dataset. Here, ESIM2 is also found among two best fusion measures. For LIVE, MAD and ESIM3 outperformed other measures. Overall results (weighted and direct averages) show that ESIM2 is the best performing metric, followed by ESIM3 and DOG-SSIM. Finally, ESIM2 seems to be the best performing IQA metric, based on the comparison presented above. Results for fusion measures that do not utilise VSI, SFF and IFS reveal that ESIM5 and ESIM6 are better than non-esim fusion models, and some single measures, such as VSI or SURF-SIM. These two models perform comparably to GMSD, IFS and SFF. The remaining two fusion models, ESIM7 and ESIM8, also obtained promising results. They are better than newly developed fusion approach proposed by Barri et al. [8] and close to ADM or popular MAD. V. CONCLUSION In this letter, an approach to decision fusion of 18 IQA fullreference models is presented. At first, the fusion strategy was formulated as an optimization problem incorporating weighted product of IQA models. The optimization problem was solved using GA that was also able to perform the IQA model selection. Finally, four fusion strategies were proposed. In general, two of them, i.e., ESIM2 and ESIM3, performed better than other competing 32 state-of-the-art IQA models in terms of four evaluation indices. Furthermore, the proposed technique is able to find well-performing models without access to newly introduced best IQA approaches. The obtained results are promising, and since the developed approach benefits from incorporated IQA measures, it can be assumed that adding new measures could possibly lead to further improvements. Therefore, Matlab source code of the approach, its evaluation and scripts that would allow running the optimization with any added measure with known objective scores, are available to download at:

5 OSZUST: DECISION FUSION FOR IMAGE QUALITY ASSESSMENT 69 REFERENCES [1] D. M. Chandler, Seven challenges in image quality assessment: Past, present, and future research, ISRN Signal Process., vol. 2013, p. 53, [2] H. Wu, Introduction: State of the play and challenges of visual quality assessment, in Vis. Signal Qual. Assess.,C.Deng,L.Ma,W.Lin,and K. N. Ngan, Eds. Berlin, Germany: Springer, 2015, pp [3] E. C. Larson and D. M. Chandler, Most apparent distortion: Full-reference image quality assessment and the role of strategy, J. Electron. Imag., vol. 19, no. 1, p , [4] S. Li, F. Zhang, L. Ma, and K. N. Ngan, Image quality assessment by separately evaluating detail losses and additive impairments, IEEE Trans. Multimedia, vol. 13, no. 5, pp , Oct [5] P. Peng and Z.-N. Li, Regularization of the structural similarity index based on preservation of edge direction, in Proc. IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC), Oct. 2012, pp [6] L. Jin, K. Egiazarian, and C.-C. Kuo, Perceptual image quality assessment using block-based multi-metric fusion (BMMF), in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP),Mar. 2012, pp [7] S.-C. Pei and L.-H. Chen, Image quality assessment using human visual dog model fused with random forest, IEEE Trans. Image Process, vol. 24, no. 11, pp , Nov [8] A.Barri,A.Dooms,B.Jansen,and P. Schelkens, A locally adaptive system for the fusion of objective quality measures, IEEE Trans. Image Process, vol. 23, no. 6, pp , Jun [9] J. Wu, W. Lin, G. Shi, and A. Liu, Perceptual quality metric with internal generative mechanism, IEEE Trans. Image Process., vol. 22, no. 1, pp , Jan [10] K. Okarma, Combined full-reference image quality metric linearly correlated with subjective assessment, in Artificial Intelligence and Soft Computing, 2010, pp , Springer. [11] M. Liu and X. Yang, A new image quality approach based on decision fusion, in Proc. Int. Conf. on Fuzzy Systems and Knowledge Discovery (FSKD), J. Ma, Y. Yin, J. Yu, and S. Zhou, Eds., Jan. 2009, pp , IEEE. [12] T.-J. Liu, W. Lin, and C.-C. Kuo, Image quality assessment using multi-method fusion, IEEE Trans. Image Process., vol. 22, no. 5, pp , May [13] A. Lahouhou, E. Viennet, and A. Beghdadi, Selecting low-level features for image quality assessment by statistical methods, CIT, vol. 18, no. 2, [14] P. Peng and Z.-N. Li, A mixture of experts approach to multi-strategy image quality assessment, in Image Analysis and Recognition, A. Campilho and M. Kamel, Eds., 2012, vol. 7324, pp , ser. Springer Lecture Notes in Computer Science. [15] J. Kittler, M. Hatef, R. P. Duin, and J. Matas, On combining classifiers, IEEE Trans. Patt. Anal. Mach. Intell., vol. 20, no. 3, pp , [16] J. Ghosh, Multiclassifier systems: Back to the future, in Multiple Classifier Systems, 2002, pp [17] H.R.Sheikh,M.F.Sabir,andA.C. Bovik, A statistical evaluation of recent full reference image quality assessment algorithms, IEEE Trans. Image Process., vol. 15, no. 11, pp , Nov [18] L. Zhang, Y. Shen, and H. Li, VSI: A visual saliency-induced index for perceptual image quality assessment, IEEE Trans. Image Process., vol. 23, no. 10, pp , Oct [19] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, 1st Ed. ed. Boston, MA, USA: Addison-Wesley/Longman, [20] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, and C.-C. J. Kuo, Image database TID2013: Peculiarities results and perspectives, Signal Process.-Image, vol. 30, pp , Jan [21] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli, and F. Battisti, TID a database for evaluation of full-reference visual quality assessment metrics, Adv. Mode. Radioelectron., vol. 10, pp , [22] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process., vol.13,no.4,pp ,Apr [23] L. Zhang, L. Zhang, X. Mou, and D. Zhang, FSIM: A feature similarity index for image quality assessment, IEEE Trans. Image Process., vol. 20, no. 8, pp , Aug [24] A. Liu, W. Lin, and M. Narwaria, Image quality assessment based on gradient similarity, IEEE Trans. Image Process., vol. 21, no. 4, pp , Apr [25] H. Sheikh, A. Bovik, and G. de Veciana, An information fidelity criterion for image quality assessment using natural scene statistics, IEEE Trans. Image Process., vol. 14, no. 12, pp , Dec [26] Z. Wang and Q. Li, Information content weighting for perceptual image quality assessment, IEEE Trans. Image Process., vol. 20, no. 5, pp , May [27] Z. Wang, E. P. Simoncelli, and A. C. Bovik, Multi-scale structural similarity for image quality assessment, in Proc.IEEEInt.Conf.on Signals, Systems, and Computers, (Asilomar), 2003, pp [28] N.Damera-Venkata,T.D.Kite,W.S.Geisler,B.L.Evans,andA. C. Bovik, Image quality assessment based on a degradation model, IEEE Trans. Image Process., vol. 9, no. 4, pp , Apr [29] L. Zhang, L. Zhang, and X. Mou, RFSIM: A feature based image quality assessment metric using riesz transforms, in Proc. IEEE Int. Conf. on Image Processing (ICIP), Sep. 2010, IEEE. [30] L. Zhang and H. Li, SR-SIM: A fast and high performance IQA index based on spectral residual, in Proc. IEEE Int. Conf. on Image Processing (ICIP), Sep.2012,IEEE. [31] Z. Wang and A. C. Bovik, A universal image quality index, IEEE Signal Process. Lett., vol. 9, no. 3, pp , Mar [32] H. R. Sheikh and A. C. Bovik, Image information and visual quality, IEEE Trans. Image Process., vol. 15, no. 2, pp , Feb [33] D. M. Chandler and S. S. Hemami, VSNR: A wavelet-based visual signal-to-noise ratio for natural images, IEEE Trans. Image Process., vol. 16, no. 9, pp , Sep [34] H.-W. Chang, H. Yang, Y. Gan, and M.-H. Wang, Sparse feature fidelity for perceptual image quality assessment, IEEE Trans. Image Process., vol. 22, no. 10, pp , Oct [35] H.-W. Chang, Q.-W. Zhang, Q.-Q. Wu, and Y. Gan, Perceptual image quality assessment by independent feature detector, Neurocomputing, vol. 151, pp , 2015, part 3. [36] F. Wang, X. Sun, Z. Guo, Y. Huang, and K. Fu, An object-distortion based image quality similarity, IEEE Signal Process. Lett., vol. 22, no. 10, pp , Oct [37]F.Zhou,Z.Lu,C.Wang,W.Sun,S.-T.Xia,andQ.Liao, Image quality assessment based on inter-patch and intra-patch similarity, PLoS ONE, vol. 10, no. 3, p. e , Mar [38] W. Xue, L. Zhang, X. Mou, and A. C. Bovik, Gradient magnitude similarity deviation: A highly efficient perceptual image quality index, IEEE Trans. Image Process., vol. 23, no. 2, pp , Feb [39] X. Zhang, X. Feng, W. Wang, and W. Xue, Edge strength similarity for image quality assessment, IEEE Signal Process. Lett., vol. 20, no. 4, pp , Apr [40] J. Wu, W. Lin, and G. Shi, Image quality assessment with degradation on spatial structure, IEEE Signal Process. Lett., vol. 21, no. 4, pp , Apr