Performance of Quality Metrics for Compressed Medical Images Through Mean Opinion Score Prediction

RESEARCH ARTICLE Copyright 212 American Scientific Publishers All rights reserved Printed in the United States of America Journal of Medical Imaging and Health Informatics Vol. 2, 1 7, 212 Performance of Quality Metrics for Compressed Medical Images Through Mean Opinion Score Prediction Basant Kumar 1, S. P. Singh 2, Anand Mohan 2, and Animesh Anand 3 1 Department of Electronics and Communication Engineering, Motilal Nehru National Institute of Technology, Allahabad-2114, India 2 Department of Electronics Engineering, Institute of Technology, Banaras Hindu University, Varanasi-2215, India 3 Department of Applied Mathematics, Institute of Technology, Banaras Hindu University, Varanasi-2215, India This paper examines the performance of two objective quality assessment metrics; peak signal-to-noise ratio (PSNR) and Structural SIMilarity (SSIM) index for compressed medical images through subjective mean opinion score () prediction. prediction models have been developed by establishing mathematical relation between the theoretically computed objective (PSNR and SSIM) 6 and subjective Mean Opinion Score () quality parameters. Based on the developed prediction models, prediction values have been generated for varying PSNR and SSIM values for compressed MRI and ultrasound images. It is observed that for same value of PSNR/SSIM, values are different depending on the type of compression technique used. It is found that SPIHT scheme gives higher predicted values as compared to JPEG and JPEG2 schemes at lower PSNR ( 38 db) for considered MR and ultrasound images. SPIHT scheme also gives higher predicted values at lower SSIM (.75) values for MR images but for ultrasound images JPEG2 gives better predicted values at SSIM (.9). This paper also provides information about correlation coefficient (CC) between peak signal to-noise ratio (PSNR)/Structural SIMilarity (SSIM) index and experimental subjective quality metrics. It is observed that PSNR gives better correlation with values for all compression schemes. Keywords: Model, Experimental, Medical Image Compression, Teleradiology. 1. INTRODUCTION The quality evaluation of compressed medical image to ascertain the performance of a lossy compression algorithm requires defining quantitative or qualitative parameters. The objective image quality measures (distortion assessment approaches) like mean squared error (MSE) and peak signal-to-noise ratio (PSNR) are frequently used for quality evaluation of reconstructed images. These mathematically defined objective quality measures are attractive because of their low computational complexity as well as viewing conditions and observer independent characteristics. Another set of objective quality evaluation uses human visual system (HVS) characteristics incorporating perceptual quality measures 1 3 of the reconstructed image. For example, Structural SIMilarity (SSIM) index is a measure of structural similarity quality using contrast sensitive function approach of HVS to estimate the quality of the compressed image. 6 This evaluation method provides good approximation to perceived image Author to whom correspondence should be addressed. distortion because HVS is highly adaptive in extracting changes in structural information from the viewing field. However, even HVS based complex metrics does not provide any clear advantage over MSE and PSNR under strict testing conditions and different image distortion environments. 4 5 Different compression algorithms have different artifacts associated with them. Blockiness in DCT-based compressed images; blurring and ringing distortions in wavelet-based encoding standards are prominent artifacts that affect the quality of images. Moreover, because of different noise characteristics in different medical image modalities the performance of the objective quality assessment parameters may be different and they may have different degree of match with the subjective quality. In view of the above, present study investigates the performance of two widely used objective quality metrics namely PSNR and SSIM for compressed MRI and ultrasound images considering JPEG, JPEG2, and SPIHT compressions at varying rates in the range.5 to 2. bits/pixel. In spite of numerous studies on improving the computation methods for objective quality assessment (QA) of reconstructed images, 2 these schemes do not correlate well with quality J. Med. Imaging Health Inf. Vol. 2, No. 2, 212 2156-718/212/2/1/7 doi:1.1166/jmihi.212.183 1

RESEARCH ARTICLE J. Med. Imaging Health Inf. 2, 1 7, 212 index which provides realistic quality assessment based on visual image perception. prediction models have been developed by establishing a relation between objective quality assessment parameters (PSNR/SSIM) and experimental to provide observer independent realistic quality assessment. This relation has been established using scatter plots generated for variations in as functions of PSNR and SSIM for the three different compressions of MRI and US images and finding the best fitting relation, which in the present case is given by the Boltzmann logistic function. The developed mathematical models have been used to predict the observer independent values corresponding to particular values of PSNR/SSIM. Validation of the developed models was carried out by computing the correlation coefficient (CC) and MSE between the observer independent predicted values and the experimental values. Based on the developed prediction models, prediction values have been generated for varying PSNR and SSIM values for compressed MRI and US images. It is observed that for same value of PSNR/SSIM, values are different depending on the type of compression technique used. Based on the correlation coefficient (CC) between PSNR/ SSIM and experimental it is reported that the PSNR has higher effect on experimental value as compared to SSIM for compressed MRI and ultrasound images. Section 2 reviews the advances in quality assessment parameters and their utility in deciding compression thresholds for various compression algorithms. Section 3 describes the development of novel models for observer independent prediction of reconstructed MRI and ultrasound images considering JPEG, JPEG2, and SPIHT compressions at varying rates in the range from.5 to 2. bits/pixel. Performance of PSNR and SSIM are evaluated for various compression techniques considering MRI and ultrasound images in Section 4. This section also contains determination of CC between PSNR/SSIM and experimental quality metrics. The conclusion of the entire work is given in Section 5. 2. BACKGROUND 2.1. Types of Quality Assessment Parameters Several approaches for the evaluation of image quality are 5 6 8 11 presented and discussed in the literature. These can be grouped into two main classes: subjective and objective methods. The former estimates the visual quality by subjective tests where a group of persons evaluate several compressed or corrupted images, with appropriate criteria, methodologies and hardware. In subjective evaluations, the final users and the human visual system (HVS) are directly involved. Assessments done in subjective tests are exactly a measure of the perceived quality. Nevertheless, performing such tests is complex and time as well as money consuming. Consequently, subjective testing is rarely employed, e.g., for performance assessment of particular compression techniques, for evaluating different quality metrics or, more often, for the parametric optimization of compression techniques. 7 Objective methods try to estimate the amount of distortion within an image using mathematical operations in the spatial/frequency domain of image or video sequences. Standard objective distortion measures as MSE, SNR and PSNR, represent simple means to provide a measure of the differences between the two given images. Digital image compression techniques generally introduce well known distortion effects and artifacts that cannot be evaluated properly with these simple objective distortion measures. Even more importantly, the metrics do not accurately predict visual quality across a set of images with varying content such as edges, textured regions, and large luminance variations. In response to the failure of standard mathematical metrics, image quality metrics that incorporate perceptual factors to varying degrees have been proposed. Usually, perceptual metrics are reported to provide more consistent estimates of image quality than mathematically defined metrics when artifacts are near the visual threshold. However, the implementation of the metrics is often so complex, and psychophysical testing required to validate them is so time consuming that a comprehensive validation and direct comparison of performance between metrics is rarely performed. 2.2. Medical Image Compression and Quality Metrics Coding of medical images differs from that of standard natural images as it is crucial to preserve the integrity of the diagnostic information in medical images while providing a reduction in storage space and network transmission bandwidth requirements. Inevitably, the ultimate solution is reversible compression schemes. However, at present, the existing state-of-the-art reversible technologies cannot achieve a significant reduction in bit-rate required for the current practical applications in biomedical imaging. 12 Other approaches such as progressive image transmission 13 and irreversible image coding schemes 14 16 have been investigated and applied to alleviate this problem. Most wavelet compression methods are superior to DCT methods. 17 It has been proven that algorithms such as SPIHT and JPEG2 achieve good results for compressed digital images. Results have shown that SPIHT and JPEG2 perform well at higher compression rates compared to JPEG. 18 In image compression algorithms there has been a need for good image quality metric that incorporates properties of the HVS and of a good quantization matrix to provide better quality of image compression with higher compression ratios. Different compression algorithms have different artifacts associated with them. Blockiness in DCT-based compressed images, blurring and ringing distortions in wavelet-based encoding standards is prominent artifacts that affect the quality of images. It has been observed that wavelet based coding is better matched with HVS system and hence the performance of perceptual quality parameters are better for this type of compression algorithm as compared to DCT based algorithms at the given bit rate. 19 The statistics of medical images are quite different from those of natural images owing to imaging equipment characteristics, resulting into different types of noise contents in the image. Noise in MRI images is primarily due to the quantum noise inherent in photon detection and electronic noise in the projection of data. The presence of inherent characteristic texture called speckle and desire to preserve it are the issues that make the ultrasound image compression problem different from the compression of natural images [Gravel et al. (24)]. Because of different noise characteristics of different medical image modalities the performance of the objective quality assessment parameters may be different and they may have different degree of match with the subjective quality parameters. 2.3. Quality Assessment Metrics Although subjective and diagnostic evaluations are even more suitable for radiological applications, 2 these can t be 2

J. Med. Imaging Health Inf. 2, 1 7, 212 RESEARCH ARTICLE incorporated into automatic systems. Therefore, objective quality measures are often used since they are easy to compute and are applicable to all kinds of images regardless of the application. 2.3.1. Objective Measures The widely used objective quality measures of an image are MSE, PSNR and SSIM index. MSE is defined as the mean of the squares of the differences between the original and reconstructed pixels, x and x, of an image. It is expressed as MSE = 1 MN M N x i j x i j 2 (1) j=1 where x and x are the intensities of original and reconstructed pixels respectively and M N is the image size. PSNR in decibel (db) can be evaluated using Eq. (2). ( ) P 2 PSNR = 1 log 1 db (2) MSE where P is the maximum possible pixel value, e.g., 255 for an 8-bit grey-level image. The objective quality measure exploits HVS properties. An effective method for objective image quality assessment was suggested by Wang et al. 12 based on the determination of variation in SSIM by comparing local patterns of pixel intensities that have been normalized for luminance and contrast; and assuming that HVS is highly adaptive to extract structural information from the viewing field. Consider a and b as two non-negative image signals. If one of the signals is assumed to have perfect quality then the similarity measure can be used as a quantitative parameter for the quality of the second signal as compared to original signal. Therefore SSIM can be computed using the following equation: SSIM a b = 2 a b + C 1 2 ab + C 2 2 a + 2 b + C 1 2 a + 2 b + C 2 where a and b are the mean intensities of a and b respectively, a and b are the corresponding standard deviations and C 1 and C 2 are constants. In discrete form ab can be estimated as: ab = 1 N 1 (3) N a i a b i b (4) The image quality assessment is achieved by locally applying the SSIM index to compute local statistics within a local w w square window which moves pixel-by-pixel over the entire image. At each step, the local statistics and SSIM index are calculated and the overall quality measure of the entire image is determined using mean SSIM (MSSIM) index given by Eq. (5) MSSIM A B = 1 M M SSIM a i b i (5) where A and B are the original and reconstructed images respectively, a i and b i are the image contents at the ith local window; and M is the number of local windows of the image. 2.3.2. Subjective Measures Subjective evaluation by viewers with normal or corrected to normal eye sight is still commonly used in measuring image quality. In case of medical images, a radiologist can judge the quality of the medical image by inspecting the loss of diagnostic information in the image. The perceived image quality has been defined by CCIR (International Radio Consultative Committee) using 1 5 scale as bad, poor, fair, good and excellent. Finally, an average score is computed to obtain the mean opinion score () for a specific image using Eq. (6). j = 1 n n S i j (6) where n denotes the number of observers and S i j is the score given by the ith observer to image j. We have considered six observers to compute of MRI and US images under the present study. In a typical rating experiment an image from a set is displayed to an observer, who rates the image using 1 5 scale. This continues until all images have been rated a number of times by different observers. These scores can be used to validate an image quality metric in a number of ways. In this paper, raw scores for each subject were normalized using the mean and variance of scores for that subject (i.e., raw values were converted to z-scores). z-score z j for a score x j is defined as z j = x j x (7) where x is the mean, and is the standard deviation of scores provided by one particular subject. The Z-score thus tells how many standard deviations away from the mean is x j. Z-score of an image is obtained by averaging the z-scores given to that image by individual observers. The averaged Z-scores are rescaled to fill the range from 1 to 1 and the rescaled values are considered as mean opinion score () values. 3. DEVELOPMENT OF PREDICTION MODELS Experiments were performed considering 1 MRI and US images of size 512 512 each with 8-bit grey scale levels available in Ref. [22] and extended from the earlier used CT scan images. 23 The images were compressed using SPIHT, JPEG2 and JPEG compression algorithms. The MSE, PSNR, and SSIM index objective quality assessment parameters have been evaluated by varying bit rates (bit per pixel) in the range.5 to 2.. Bit rates were chosen non-uniformly such that the resulting distribution of subjective quality scores is approximately uniform over the entire range. Six observers with normal vision or corrected to normal vision were involved in subjective quality evaluation of the compressed images and perceived image quality was graded on a continuous linear scale 1 5 (bad, poor, fair, good and excellent). Raw scores for each subject were normalized by taking z-score. The averaged z-scores were rescaled to fill the range of values from 1 to 1 for each image. It is observed that for same value of PSNR/SSIM, values are different depending on the type of compression technique used. In the present study, models for observer independent mean opinion score () prediction of reconstructed MRI and 3

RESEARCH ARTICLE J. Med. Imaging Health Inf. 2, 1 7, 212 US images have been developed by establishing a relation between objective quality assessment parameters (PSNR/SSIM) and experimental mean opinion score () values. This relation has been established using scatter plots generated for variations in mean opinion score () as functions of PSNR and SSIM for medical images using three different compression schemes. The best fitting relation in the present case is given by the Boltzmann logistic function. The scatter plots with the best fitting curves for JPEG, JPEG2 and SPIHT compressions are shown in Figure 1 for MRI images and in Figure 2 for ultrasound images. The Boltzmann function is expressed in the following form: y = A 1 A 2 1 + e x A 3 /A 4 + A 2 (8) where A 1, A 2, A 3 and A 4 are constants. These non-linear regression functions have been used to transform the set of PSNR/SSIM outputs into a set of predicted values. Subsequently the correlation coefficient (CC) and percent mean square difference (PRD) values have been computed Fig. 1.2. Scatter plots of subjective mean opinion score () versus PSNR SSIM model prediction for MR images considering JPEG2 compression. between the predicted and experimental values which are presented in Table I. The formula used to calculate PRD is as follows: N PRD = M i ˆM i 2 N 1 (9) i 1 M i 2 where M i denotes the experimental, ˆM i denotes the predicted data and N is the number of samples in the data set. It can be observed from Table I that PSNR- prediction Table I. Performance of prediction models with respect to subjective for MR and US images. Fig. 1.1. Scatter plots of subjective mean opinion score () versus PSNR and SSIM model prediction for MR images considering JPEG compression. JPEG JPEG2 SPIHT Prediction model CC PRD CC PRD CC PRD MR images PSNR-.9615 9 8439.9224 11 7937.9427 11.22 SSIM-.9584 12 7645.9147 9 3298.8966 1.142 Ultrasound (US) images PSNR-.93 11 34.9294 11 721.982 11.9787 SSIM-.9177 1 7151.8731 12 23.9532 12.421 4

J. Med. Imaging Health Inf. 2, 1 7, 212 RESEARCH ARTICLE model gives better correlated values for all the three compression techniques considered in the present study. Analysis of computed values of CC and PRD indicates that CC and PRD values between the two values are CC.87, PRD 12.8% for the considered range of medical image compressions. This provides theoretical validation of the developed subjective quality prediction models and also indicates the close matching between the experimental and predicted values. 4. PERFORMANCE EVALUATION AND DISCUSSION Based on the developed prediction models, prediction values have been generated for varying PSNR and SSIM values of compressed MRI and ultrasound images as shown in Figures 3 and 4 respectively. It is found that the prediction values are different at the given value of PSNR/SSIM for different compression schemes. Figure 3.1 shows that SPIHT compressed MR image has higher values for PSNR values below 38 db. At higher PSNR ( db) JPEG has higher values compared to SPIHT and JPEG2 compressed images. Subsequently, it can be observed from Figure 3.1 that SPIHT has higher prediction values at lower SSIM (<.75) whereas for higher values of SSIM, JPEG provides slightly greater values compared to SPIHT and JPEG2 compressed MR images. Figure 3.2 shows that SPIHT compressed ultrasound image has higher values for PSNR values below 38 db, but at higher PSNR ( 38 db) JPEG has higher values compared to SPIHT and JPEG2 compressed images. This result is similar to the result obtained in case of MR images. It can be observed from Figure 3.2 that JPEG2 has higher prediction values at SSIM (<.9) whereas for higher values of SSIM, JPEG provides slightly greater values compared to SPIHT and JPEG2 compressed ultrasound images. In order to determine the degree of influence of PSNR and SSIM objective quality parameters on, two performance metrics have been used. Metric 1 is the standard correlation coefficient (Pearson) between objective/subjective scores obtained through non-linear regression analysis. This metric provides an evaluation of prediction accuracy. The second metric is the Fig. 1.3. Scatter plots of subjective mean opinion score () versus PSNR SSIM model prediction for MR images considering SPIHT compression. Fig. 2.1. Scatter plots of subjective mean opinion score () versus PSNR SSIM model prediction for ultrasound (US) images considering JPEG compression. 5

RESEARCH ARTICLE J. Med. Imaging Health Inf. 2, 1 7, 212 Fig. 2.2. Scatter plots of subjective mean opinion score () versus PSNR SSIM model prediction for ultrasound (US) images considering JPEG2 compression. Spearman rank-order correlation coefficient (SROCC) between the objective/subjective scores. It is considered as a measure of prediction monotonicity. The computed values of correlation coefficients between PSNR/SSIM and values for three different compression techniques mentioned earlier are given in Table II. It is observed that PSNR gives better correlation with values for JPEG, JPEG2 and SPIHT compressed MR and ultrasound images. Table II. Performance comparision of quality assessment parameters for MR and ultrasound images for different compression techniques. JPEG JPEG2 SPIHT Parameters CC SROCC CC SROCC CC SROCC MR images PSNR.8955.7915.9199.8893.9186.952 SSIM.7427.7617.8533.8651.813.827 Ultrasound (US) images PSNR.8352.8213.942.98.8969.98 SSIM.8356.8767.8367.9382.855.91 Fig. 2.3. Scatter plots of subjective mean opinion score () versus PSNR SSIM model prediction for ultrasound (US) images considering SPIHT compression. 1 8 2 1 8 2 JPEG 2 2 3 5 PSNR (db) J.4.6.8 1 MSSIM Fig. 3.1. Variation in prediction values with PSNR and MSSIM for different compression schemes for MR images. 6

J. Med. Imaging Health Inf. 2, 1 7, 212 RESEARCH ARTICLE 9 8 7 5 3 2 1 8 JPEG 2 25 3 35 45 5 7 5 3 2 1 JPEG PSNR (db).5.6.7.8.9 1 SSIM Fig. 3.2. Variation in prediction values with PSNR and MSSIM for different compression schemes for US images. 5. CONCLUSION This paper investigated the performance of two objective quality assessment metrics; peak signal-to-noise ratio (PSNR) and Structural SIMilarity (SSIM) index for compressed MR and US images through subjective mean opinion score () prediction. For this, prediction models were developed and validated to provide predicted values which closely matched with the experimental values. It was found that the prediction values are different at the given value of PSNR/SSIM for different compression schemes. It was also established that PSNR gives better correlation with values for JPEG, JPEG2 and SPIHT compressed MR and ultrasound images. References and Notes 1. T. N. Pappas and R. J. Safranek, Perceptual criteria for image quality evaluation, A. C. Bovik (ed.), Handbook of Image and Video Processing, 2nd edn., Elsevier Academic Press, New York (2), pp. 939 956. 2. M. P. Eckert and A. P. Bradley, Perceptual quality metrics applied to still image compression. Signal Processing 7, 177 (1998). 3. G. Ginesu, F. Massidda, and D. D. Giusto, A multi-factor approach for image quality assessment based on a human visual system model. Signal Processing: Image Communication 21, 316 (26). 4. J. B. Martens and L. Meesters, Image dissimilarity. Signal Processing 7, 155 (1998). 5. A. M. Eskicioglu and P. S. Fisher, Image quality measures and their performance. IEEE Transactions on Communications 43, 2959 (1995). 6. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error measurement to structural similarity. IEEE Transactions on Image Processing 13, (24). 7. M. P. Eckert and A. P. Bradley, Perceptual quality metrics applied to still image compression. Signal Processing 7, 177 (1998). 8. U. Engelke and H. J. Zepernick, Perceptual-based quality metrics for image and video services, Proceedings 3rd EURO-NGI Conference on Next Generation Internet Networks, Trondheim, Norway, May (27), pp. 19 197. 9. S. Daly, The visible differences predictor: An algorithm for the assessment of image fidelity, edited by A. B. Watson, Digital Images and Human Vision, MIT Press, Cambridge, MA (1993), p. 179. 1. Z. Wang and A. C. Bovik, A Universal image quality index. IEEE Signal Processing Letter 9, 81 (22). 11. T. J. Chen, K. S. Chuang, J. Wu, S. C. Chen, I. M. Hwang, and M. L. Jan, A novel image quality index using Moran I statistics. Physics in Medicine and Biology 48, N131 (23). 12. B. J. Erickson, Irreversible compression of medical images, Great Falls, Society for Computer Applications in Radiology, VA, November (2). 13. W. J. Hwang, C. F. Chine, and K. J. Li, Scalable medical data compression and transmission using wavelet transform for telemedicine applications. IEEE Transactions on Information Technology in Biomedicine 7 (23). 14. J. Bradley and M. D. Erickson, Irreversible compression of medical images. Journal of digital imaging 155 (22). 15. Y. G. Wu and S. C. Tai, Medical image compression by discrete cosine transform spectral similarity strategy. IEEE Transactions on Information Technology in Biomedicine l5, 236 (21). 16. D. Wu, D. Tan, M. Baird, J. DeCampo, C. White, and H. R. Wu, Perceptually lossless medical image coding. IEEE Transactions on Medical Imaging 25, 335 (26). 17. B. J. Erickson, A. Manduca, P. Palisson, K. R. Persons, F. Earnest, V. Savcenko, and N. J. Hagiandreou, Wavelet compression of medical images. Radiology 26, 599 (1998). 18. Y.-Y. Chen, Medical image compression using DCT-based subband decomposition and modified SPIHT data organization. Journal of Medical Informatics 76, 717 (27). 19. A. B. Watson, G. Y. Yang, J. A. Solomon, and J. Villasenor, Visibility of wavelet quantization noise. IEEE Transaction on Image Processing 6, 1164 (1997). 2. P. C. Cosman, R. M. Gray, and R. A. Olshen, Evaluating quality of compressed medical images: SNR, Subjective rating, and diagnostic accuracy. Proceedings of IEEE 82, 919 (1994). 21. H. Lee, D. Haynor, and Y. Kim, Subjective evaluation of compressed image quality, Proceedings of SPIE, Image Capture, Formatting and Display, 1653, 241 (1992). 22. MedPix Database TM : Department of Radiology and Radiological Sciences, Uniformed Services University of Health Sciences (US). Available from:http://rad.usuhs.mil/medpix/medpix.htm Performance of Quality Metrics for Compressed. 23. Basant Kumar, Surya Pal Singh, Anand Mohan, and Animesh Anand, Performance of quality metrics for compressed medical images through prediction, Proceedings of ICSIVP, January (212). Received: 25 February 212. Revised/Accepted: 2 May 212. 7