IMAGE QUALITY ASSESSMENT BASED ON EDGE

Transcription

1 IMAGE QUALITY ASSESSMENT BASED ON EDGE Xuanqin Mou 1, Min Zhang 1, Wufeng Xue 1 and Lei Zhang 1 Institute of Image Processing and Pattern Recognition, Xi'an Jiaotong University, China Department of Computing, Hong Kong Polytechnic University, Hong Kong xqmou@mail.xjtu.edu.cn ABSTRACT The research on image quality assessment (IQA) has been become a hot topic in most area concerning image processing. Seeking for the efficient IQA model with the neurophysiology support is naturally the goal people put the efforts to pursue. In this paper, we argue that comparing the edges position of reference and distorted image can well measure the image structural distortion and become an efficient IQA metric, while the edge is detected from the primitive structures of image convolving with LOG filters. The proposed metric is called NSER that has been designed following a simple logic based on the cosine distance of the primitive structures and two accessible improvements. Validation is taken by comparison of the well-known state-of-the-art IQA metrics: VIF, MS-SSIM, VSNR over the six IQA databases: LIVE, TID008, MICT, IVC, A57, and CSIQ. Experiments show that NSER works stably across all the six databases and achieves the good performance. Keywords: Quality assessment (QA), Zero crossing, Laplacian of Gaussian; Cosine distance; Non-shift Edge. 1. INTRODUCTION The research on image quality assessment (IQA) has been become a hot topic not only limited in image coding and communication, but also in most area concerning image processing and analysis. The rapidly growing of development of IQA models is gradually changing the situation that MSE or its counterpart, PNSR, has been dominating the judgment metrics of image difference in the past time. Seeking for the efficient IQA model with the neurophysiology support is naturally the goal people have been put efforts to pursue because the essential fact is that discriminating image difference is indeed of a psychological and physiological process carried out by our sophisticated human neural system. Based on this bottom-up philosophy, many efforts have been made successfully to find and simulate the functional properties of early vision system to develop the visual error sensitivities IQA models [1]-[4]. Among them, the Sarnoff JNDmetrix visual discrimination model (VDM) [1] has been widely used. Other representative metrics include the noise quality measure (NQM) [4] and the wavelet based visual signal to noise ratio (VSNR) [3], which operates based on the visual property of perceived contrast and the mid-level visual property of global precedence. Exploring the human vision system (HVS) is a long-term effort and large numbers of mechanisms are still undiscovered, the knowledge people have obtained in this area so far cannot completely meet the requirement to design the ideal IQA model. Researchers turned to search a direct way to compute the image distortion. Wang et al. proposed firstly a framework that approximates the perceptive image distortion by measuring the structural information change, which is called SSIM [5]. Based upon this, the image quality is evaluated using the structural information rather than pixel intensities themselves. The structural information is obtained through three component including the mean, variance, and covariance of the reference and distorted image. In this way, several paradigms, such as UQI, SSIM, MS- SSIM, CW-SSIM [5]-[8], are proposed to achieve attractive effects. Another way that researchers tried to directly mimic the entire properties of HVS in IQA task was to explore the information fidelity criteria. This work was motivated by natural scene statistics (NSS) model and quantified the information shared between the distorted and reference images, to form the so called information fidelity criterion (IFC) [9]. Later, IFC was extended to the VIF (Visual Information Fidelity) by divisive normalization and considering the perceptual noise model [10]. Nevertheless a great diversity of methods were introduced, detecting the structural information change has been recognized as one of the most important way to contribute to IQA model. In SSIM framework, structure similarity is the Digital Photography VII, edited by Francisco H. Imai, Feng Xiao, Jeffrey M. DiCarlo, Nitin Sampat, Sebastiano Battiato, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 7876, 78760N 011 SPIE-IS&T CCC code: X/11/$18 doi: / Proc. of SPIE-IS&T/ Vol N-1

2 key factor in scaling the perceptual image distortion [5, 11]. The computation of structure similarity was proposed as the mean local correlation coefficients of intensities of two images [5]. Although there are many other ways to realize the computation of structural similarity, for example CW-SSIM [8], we are interested in exploring the way by which the structural distortion can be measured from the signals of human visual path way, especially of early visual process stage. In HVS, image structures are firstly extracted by the ganglion cells and lateral geniculate nucleus (LGN) neurons. The response of the ganglion and LGN cells is featured with classical receptive field (CRF) and can be modeled by Laplacian of Gaussian (LOG) function. Marr considered that the signals extracted by the earliest stage of HVS come into being the basic primitive structures of image and the LOG operator is indeed an edge filter [1]. It plays an important role in many pattern recognition and understanding applications. Here it is interesting that if it is possible to measure the structure distortion by those earliest neural signals. In this paper, we make the effort to explore the way to measure structural information change using those signals. The proposed method is based on the measurement of the basic primitive structures extracted by the earliest visual neurons. And it is validated over all the available image quality assessment databases in this paper.. METHODS Human early vision has been studied in both computational and biological ways for a long time. In the visual pathway of mammals, an external scene imaged on the retina is fist processed by the ganglion cells and the LGN neurons with the CRFs of several scales, and then the generated signals are transmitted to the primary visual cortex (V1) [1]. From a computational point of view, the response of CRF can be modeled by a series of hierarchical filters, i.e. center-surround LOG filters [1], which is defined as x + y 1 x + y σ Gxy (,, σ ) = 1 4 πσ σ e (1) where G is the Gaussian kernel and the parameter σ represents the scale of the receptive filed x + y σ 1 Gxy (,, σ ) = e. () πσ Marr thought that those signals extracted by the earliest stage of HVS come into being the basic primitive structures of image and the LOG operator is indeed an edge filter [1]. Those produced signals are primitives to construct perceptive structures in late human visual processing stages. Based on this, we here give the first formula to measure the structure distortion by directly computing the cosine distance of those primitives, which we call Cosine Distance of LOG space (CD-LOG) and transforming it into psychological intensity by the well-known Werber-Fecher s law, Ri( x, y), Di( x, y) CD LOG = log 10 (1 ). (3) i Ri( x, y) Di( x, y) R i (x,y)and D i (x,y) represent the scale images of decomposed reference image R i (x,y) and the distorted image D(x,y) with LOG filters at scale i, respectively. R xy Gxy Rxy, D( x, y) = G( x, y, σ ) D( x, y ) (4) i(, ) = (,, σ i) (, ) According to Marr s theory, the crucial information of the basic primitives is carried by the edge locations where zero-crossings occur. It was conjectured that an image could be completely represented by the zero-crossing data that are detected from its multi-scale LOG operator filtered images. Computational vision researchers have been studying the degree to which an image can be reconstructed from its edges or zero-crossings [13]. Although the current works show that zero-crossing locations alone cannot form a complete representation for image, locations of edge still carry most information of an image. They are the critical elements in constituting natural scene s structure in higher-level computations [13]. So the second formula to measure the structure distortion is designed by replacing the primitives in the first formula with their edge maps, which we call Edge based Structure Similarity (ESS). Given that edge maps of E Ri and E Di are generated from the scale images of R i (x,y)and D i (x,y) by zero-crossing detection, respectively, and they have only the binary value of 0 to represent background and 1 to represent edge point, the ESS is expressed as Sum( ER I E ) i Di ESS = log 10 (1 ). (5) i Sum( ER ) Sum( E ) i D i i i Proc. of SPIE-IS&T/ Vol N-

3 Here the operator represents the logical AND operation, and the Sum( ) operator represents that counts the number of edge points in an edge map. To obtain edge map, the method for zero-crossing detection used in this paper is as below, where f(x,y) is the filtered image with LOG operator, and the TH is the detection threshold. if f( x, y) < 0 and f( x+ 1, y) > 0, and f( x+ 1, y) f( x, y) > TH if f( x, y) < 0 and f( x 1, y) > 0, and f( x 1, y) f( x, y) > TH if f( x, y) < 0 and f( x, y+ 1) > 0, and f( x, y+ 1) f( x, y) > TH 1, Ef ( x, y) = if f( x, y) < 0 and f( x, y 1) > 0, and f( xy, 1) f( xy, ) > TH (6) if f ( x, y) = 0 and f( x+ 1, y) f( x 1, y) < 0, and f( x+ 1, y) f( x 1, y) > TH if f ( xy, ) = 0 and f( xy, + 1) f( xy, 1) < 0, and f( xy, + 1) f( xy, 1) > TH 0, else Obviously, E Ri E Di is indeed the edge points that stay in the same position when the image is distorted. We call it Non-Shift Edge (NSE) map. In the task of image quality assessment, the reference is normally considered as a perfect image. We can think that all the edge points detected from the reference image are significant to contribute to subsequent neurons system for perception. To some extent, people assess the distortion in the way to know/extract how much information of reference from the distorted image. It is to say that the reference and distortion image has the different role in the task of image quality assessment. The deteriorated process applied on an image will generate extra structures that do not provide perceptual information in accordance with that the original image provides. If we focus on developing a metric to measure the lost structural information from reference, not the structure difference between the two images, we can make a modification of Eq. (6) by excluding the factor of distorted image in the denominator of the correlation coefficient calculation, resulting a new edge based method that estimates how much information of the reference image is preserved in the distorted one by the ratio of the number of the NSE map to the number of the edge map of reference image. In this way, the third formula, which we call NSE based Ratio (NSER), is then given as Sum( ER I E ) i Di NSER = log 10 (1 ). (7) i Sum( ER ) i All above three designed metrics are based on measurement of the basic primitive structures extracted by the earliest visual neural cells between the reference and distortion image. Those metrics will be validated over all the available IQA databases in the following section. The demonstrated respective performance, especially the stability on different databases, as well as each arithmetical detail, will help to understand the possible way by which human vision system handle the perception of image distortion. At the same time, those metrics are also compared to the state-of-theart IQA models to evaluate their possibility to be an independent IQA method. 3.1 The Subjective Image Database 3. SUBJECTIVE EXPERIMENTS AND VALIDATIONS All the proposed methods are evaluated on the six subjective rated image quality databases. They are LIVE [14], IVC [15], MICT [16], TID008 [17], A57 [18] and CSIQ [19], respectively. Table 1 gives a brief description of the six databases. Table 1. Subjective image quality assessment databases Database Number of Number of images observers Image type Image size Distortion types IVC Color 51*51 4 LIVE Color Mainly 768*51 5 CSIQ Color 51*51 6 MICT Color 768*51 A Gray scale 51*51 6 TID Color 51* Proc. of SPIE-IS&T/ Vol N-3

4 3. Simulation Details and Calibration of the Objective Score Seven IQA methods are compared in this paper, including the three state-of-art methods VIF [10], VSNR[3], MS- SSIM[7], the three proposed metric in section : CD-LOG, ESS and NSER, and more especially the multi-scale structure correlation (MS-SC) that is the structural similarity component considered as the key factor of MS-SSIM [5, 7, 11]. In the following experiments, CD-LOG is applied with five LOG filters with the scale σ of [0.5, 1.3,.6, 5., 10.4]; ESS and NSER are implemented with the same scales while the threshold TH for zero-crossing detection of each scale is [0.6, 0.4, 0., 0.08, 0.0]. The used codes of VIF, VSNR and MS-SSIM are available at and MS-SC is implemented by using correlated part of above code of MS-SSIM. These seven IQA methods are all operated upon the luminance component only. According to the VQEG Phase-II testing and validation, a nonlinear mapping between the objective and the subjective scores was allowed [0]. A five-parameter nonlinearity (a logistic function with additive linear term constrained to be monotonic) is selected. It is shown as follows 1 1 Quality( x) = β 1( ) 4 5. x 1+ exp( β( x β3)) + β + β (8) Then the Pearson linear correlation coefficient (CC) and Spearman rank order correlation coefficient (SROCC) between the subjective score and the regressed objective results are computed to evaluate the consistency and monotonicity of the IQA methods. 4. RESULTS AND DISCUSSION Here we present experimental results of validation of compared IQA methods on the above databases. Tables and 3 summarize the results for the different IQA methods with the performance of CC and SROCC between the predicted objective quality and subjective rating, respectively. The best two metrics producing the greatest correlations for each database are marked in bold. Compared CD-LOG with MS-SC, we can see that these two metrics works all square over the six IQA databases: CD-LOG works better than MS-SC on IVC, MICT and CSIQ databases in terms of both CC and SROCC, works better on A57 database in terms of SROCC, and work worse on rest cases. For mean score over the six databases, CD-LOG exhibits a little better than MS-SC both of CC and SROCC. Those facts show that using the structural signal generated by the earliest stage of human visual process, which is the output of ganglion and LGN cells, and the computation strategy of cosine distance can also perform well for comparing structural difference between reference and distortion images. Compared to CD-LOG in terms of CC and SROCC, ESS plays similarly on LIVE and A57, slightly better on IVC and MICT, a little worse on TID008 and CSIQ. In general, ESS and CD-LOG works all square. The strategy of forming ESS is taken by replacing the linearly decomposed coefficients in CD-LOG with the nonlinearly detected edge map. In other words, the feature used by ESS is the edge elements extracted from the basic primitive structures that is the feature used by CD-LOG. The process of edge detection will lose some information existed in original image or their basic primitive structures. For the approximate performance of ESS and CD-LOG, we think there are two possible reasons. First, as we know, researchers from Marr have been tried to argue that an image can be reconstructed from their multiscale edge maps. Although their works cannot well support it, it is convinced that edges carry most information existed in an image. Secondly, although the primitive structures has removed second-order correlation as the LOG operator is an edge filter [1], high-order statistical dependencies still exist. On the contrary, edge map has the high independency property. So edges of image can be used well for IQA task. After the further improvement applied on ESS by using only the number of reference image edge to scale that of NSE map, the resulting NSER exhibits a dramatically improved performance on all the databases, which proves that the ratio of the number of edge points in NSE map to that in reference edge map can predict the image quality well. For better illustration the different performance of NSER and the state-of the-art methods, figure demonstrates, in the form of bar graph, the results of the VIF, VSNR, MS-SSIM and NSER. It shows that NSER achieves the best performance on IVC, MICT and CSIQ databases in terms of both CC and SROCC; on A57 database, NSER ranks second both for CC and SROCC; and on LIVE and TID008 databases, although NSER performs behind MS-SSIM and VIF, its performance approaches the second one very closely. For the contrastive IQA metrics, MS-SSIM performs stably across all six IQA databases, while VIF performs worse on A57 and VSNR does the same on IVC, TID008 and CSIQ Proc. of SPIE-IS&T/ Vol N-4

5 databases. When considered the average performance over all the databases, NSER performs best, followed by MS- SSIM in terms of CC, while MS-SSIM performs the best, followed by NSER in terms of SROCC. For further reference, Figs in Appendix show the scatter distribution of the subjective rating against the predicted scores of the IQA metrics of NSER, MS-SSIM, VIF and VSNR on the six IQA databases, respectively. From above experiments and discussion, we can see that the so called NSE map plays an important role in IQA task of NSER and the number of edge points of reference edge map can be considered as a normalized factor to compensate the complexities of different reference images. In physical meaning, NSE map contains the zero-crossing points that do not change their positions when image is deteriorated. How does NSE map act in IQA task? We first illustrate a sample to demonstrate it. Fig. a) is a reference image from the CSIQ database at Three types of distorted images of Fig. a) with similar objective quality evaluation indexed by DMOS are shown in Fig. b) (contrast decrement), Fig. c) (pink Gaussian noise) and Fig. d) (JPEG compression), respectively. The edge maps of the four images are shown in the nd column of Fig., while the NSE maps of the three distorted image are shown in the 3rd column of Fig.. We see that although the edge maps (nd column) are very different, interestingly the NSE maps (3rd column) of the three distorted images are very similar to each other, which is similar to the objective quality evaluation. This example clearly demonstrates that NSE can be used to evaluate the perceptual quality of distorted images. There are several reasons that why the proposed NSE based IQA method works. First, no matter what type of distortion is applied to the reference image, the edge point belonging to a strong structure has a low probability to change its position or disappear, while the edge point belonging to a weak structure feature can be easily changed. Since the stronger structure feature is more important than the weaker one for HVS perception, the number of non-shift edges can quantify the quality of images. Secondly, we can find evidence from Fig.. By observing the NSE maps of Figs. i), j) and k) from different distortion types, and comparing them with that of the reference image, we can see that the NSE maps preserve the edge points of strong structures. Thirdly, although NSE map loses some information during the binary edge detection process, NSER metric still works very well across all the six databases. The possible reason is that NSER actually selects the most significant features in the reference and distorted images, and it eliminates much information redundancy in the image. The information lost in the binary edge detection process is not so important for IQA. From the experimental results, we can see that both MS-SSIM and NSER perform very well across all the databases. But they are very different from each other. Specifically, MS-SSIM directly mimics the entire HVS to build the IQA metric [5, 7], while NSER uses only the early vision features (i.e. edges) in the IQA metric design. On the other hand, MS-SSIM includes three distortion components: luminance, contrast and the so called structural-similarity, among which the structural-similarity is the core factor, while NSER uses only the binary edge maps to measure the image quality in the form of NSE that can be considered as the structural-similarity in some sense. Interestingly, by using only the primitive zero-crossings, NSER still achieves comparable performance to MS-SSIM. This fact shows that zero-crossings can be very effective and efficient for IQA. A nature scene is constituted by spatially distributed structures. The pixels belonging to a structure are related to each other with a specific intensities modality and the information the structure carries is hid behind this modality. After being deteriorated, the structure varies as well as the modality does accordingly. So the structure based SSIM [4-8] and the information based IFC [9, 10] indexes can be used well for IQA task. In general, IQA metrics of the two types, as well as the others based on known HVS models, take into account the components of perceptible structure that are in form of intensities distributions of group pixels bounded by decomposition scale. While NSER uses the earlier feature, i.e. zero-crossing, that is in form of single point feature revealed by its computation model. The difference of visual process between NSER and other metrics is discussed as follow: First, the external stimuli imaged on retina are sensed by ganglion and LGN neurons in form of zero-crossings that are the primitives for producing the perceptible structure in later visual process [1]. Secondly, the information and pattern of later produced structure are hid in the spatial distribution of zero-crossings [1]. Thirdly, the structure varying and information losing during distortion arise as the same as change of the distribution. Moreover, the structural similarity and shared information can be expressed by NSE map, and more specifically, can be computed in statistics independently from those zero-crossing points as NSER model exposes. Overall, the proposed NSER metric is very simple but highly effective. The success of NSER reveals that image features produced by the earliest visual processing stage, such as edge locations, contain crucial information in form of NSE map that directly related to perceptual quality assessment. Proc. of SPIE-IS&T/ Vol N-5

6 Table. CC(correlation coefficient) Performance comparison of IQA models after nonlinear regression. CC VIF VSNR MS-SSIM MS-SC CD-LOG ESS NSER LIVE A IVC MICT TID CSIQ MEAN Table 3. SROCC(spearman rank-order correlation coefficient) performance comparison of IQA models. SROCC VIF VSNR MS-SSIM MS-SC CD-LOG ESS NSER LIVE A IVC MICT TID CSIQ MEAN Figure 1. Comparion of NSER with the three state-of-art IQA methods, VIF, VSNR and MS-SSM, for indexes of CC (left) and SROCC (right). 5. CONCLUSION In this paper, we proposed a novel full reference IQA metric that is called NSER. This metric measures image perceptual distortion by counting how many edge points stay in their original positions when image is deteriorated and normalizing the counted number by the edge points of reference edge map, while the edge points is extracted from the basic primitive structures of image produced by the earliest stage of human visual path way by zero-crossing detection. Experiments show that NSER can achieve a stable and efficient performance across all the IQA databases. The work in this paper also reveals the high correlation between visual perception of image quality and the location of image edges. 6. ACKNOWLEDGEMENT This work is partially supported by National Natural Science Foundation of China (NSFC) through No and Proc. of SPIE-IS&T/ Vol N-6

7 Figure. An example of NSE maps with different distortion types. All the images are from the CSIQ database and the rated subjective scores have similiar DMOS values. All the edge maps are detected by the LOG operator with scale.6 and threshold 0. a) lady liberty (reference) e) edge map of a) b) contrast decrement DMOS=0.641 f) edge map of b) i) NSE map of b) c) g) edge map of c) j) NSE map of c) h) edge map of d) k) NSE map of d) pink Gaussian noise DMOS=0.64 d) JPEG compression DMOS=0.655 Proc. of SPIE-IS&T/ Vol N-7

8 REFERENCES [1] Sarnoff Corporation, JNDmetrix Technology. [] S. Daly, "The visible difference predictor: an algorithm for the assessment of image fidelity," Digital Images and Human Vision,A.B.Watson, Ed. Cambridge, MA: MIT Press, pp ( 1993). [3] D. M. Chandler and S. S. Hemami. "VSNR: A wavelet-based visual signal-to-noise ratio for natural images," IEEE Trans. Image Process. 16, pp (007). [4] N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A.C. Bovik. "Image quality assessment based on a degradation model," IEEE Trans. Image Process., vol. 4, no. 4, pp (000). [5] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. "Image quality assessment: From error measurement to structural similarity," IEEE Trans. Image Processing, vol. 13, no. 4, pp (004). [6] Z. Wang and A. C. Bovik, "A universal image quality index," IEEE Signal Processing Letters, vol. 9, pp (00). [7] Z. Wang, E. P. Simoncelli, and A. C. Bovik. "Multi-scale structural similarity for image quality assessment," presented at the Asilomar Conf. Signals, Systems, and Computers (003). [8] Z. Wang and E. P. Simoncelli, "Translation insensitive image similarity in complex wavelet domain," in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Proc., Philadelphia, PA, (005). [9] H.R. Sheikh, A. C. Bovik, and G. de Veciana. "An information fidelity criterion for image quality assessment using natural scene statistics," IEEE Trans. Image Process, vol. 14, no. 1, pp (005). [10] H.R. Sheikh and A.C. Bovik. "Image information and visual quality," IEEE Trans. Image Process., vol.15, no.pp (006). [11] David M. Rouse and Sheila S. Hemami, "Understanding and simplifying the structural similarity metric," Proceedings of ICIP, San Diego, USA, pp (008). [1] D. Marr, "Vision," W. H. Freeman and Company, New York, N. Y., (1980). [13] J. H. ELDER, "Are Edges Incomplete?," International Journal of Computer Vision 34(/3), 97 1(1999). [14] H. R. Sheikh, Z. Wang, L. Cormack, and A. C. Bovik, Live Image Quality Assessment Database Release. [Online]. Available: quality (005). [15] P. Le Callet, Polytech Nantes University de Nantes, France, Subjective Quality Assessment-IVC Database. [Online]. Available: [16] Horita, K. Shibata, Y. Kawayoke, Z. M. Parvez Sazzad, MICT Image Quality Evaluation Database. [Online].Available: [17] N. Ponomarenko, M. Carli, V. Lukin, K. Egiazarian, J. Astola, F. Battisti, "Color Image Database for Evaluation of Image Quality Metrics," Proceedings of International Workshop on Multimedia Signal Processing, Australia,, pp (008). [18] Visual Communications Laboratory of Cornell University, USA. A57 Database. [Online]. Available: cornell.edu/dmc7/vsnr/vsnr.html. [19] Image Coding and Analysis Lab, The CSIQ database. [Online]. Available: [0] VQEG: The Video Quality Experts Group, [1] A J Bell and T J Sejnowski, "The independent components of natural scenes are edge filters," Vision Research, 37(3): (1997). Proc. of SPIE-IS&T/ Vol N-8

9 APPENDIX MS-SSIM NSER VIF VSNR A57 CSIQ IVC LIVE MICT TID008 Figure. The scatter plots of DMOS/MOS versus results of model prediction. Each dot represents an image in the database. The results of four IQA models, including MS-SSIM, NSER, VIF and VSNR, on the six image quality databases (A57, CSIQ, IVC, LIVE, MICT, TID008) are demonstrated here. Proc. of SPIE-IS&T/ Vol N-9