Subjective Image Quality Prediction based on Neural Network

Transcription

1 Subjective Image Qualit Prediction based on Neural Network Sertan Kaa a, Mariofanna Milanova a, John Talburt a, Brian Tsou b, Marina Altnova c a Universit of Arkansas at Little Rock, 80 S. Universit Av, Little Rock, AR, USA 704; b U.S. Air Force Research Laborator, Wright-Patterson Air Force Base, OH, USA 4543; c Qbase, 69 Commons Boulevard, Daton, OH, USA 4543 Abstract: This paper investigates the applicabilit of the Neural Network approach for image qualit assessment. The aim is to predict the subjective qualit score, namel the difference mean opinion score (DMOS) obtained from human observers, b incorporating a neural network algorithmic approach utilizing etracted statistical features from test and original images. To ease this approach, a MATLAB user interface is developed and presented here. To validate the proposed approach, an image database is selected consisting of various distortion tpes as test bed in which a DMOS value is provided for each distorted image. Eperimental results show that the obtained output of Neural Network correlates well with DMOS values and Neural Network can mimic human observers. Kewords: subjective image qualit assessment, neural network, difference mean opinion score. INTRODUCTION The growth of advance technolog allows images to be captured, stored and transmitted from one device to another with ease. During these processes, imaging sstems ma add some distortion, degradation or artifacts to images. Recent advances in multimedia technolog dramaticall reveal an increasing demand for qualit controlled images. The problem of Image Qualit Assessment has been given great attention in literature. Traditionall, Image Qualit Assessment methods can be classified into two major categories, namel objective and subjective qualit assessments []. On the one hand, the objective qualit assessment methods measure the qualit between original and distorted images based on mathematical analsis. The objective qualit methods are also divided into 3 subgroups based on availabilit of original signal as follows; Full Reference Methods (FR), Reduced Reference Methods (RR), and No-Reference Methods (NR). Some wellknown metrics are such as MSE (Mean Squared Error), PSNR (Peak Signal to Noise Ratio) [], SSIM (Structural Similarit) [], and VIF (Visual Information Fidelit) [3]. On the other hand, the subjective qualit assessment method [] essentiall measures the perceived qualit on images based on human observers opinion. In literature, most commonl used metric are MOS (Mean Opinion Score) and DMOS (Difference Mean Opinion Score) [4]. Gastaldo et all. [5] use Circular Back-Propagation (CBP) neural network, including one additional input to the neural network architecture, etracting first-order histogram, features drives from co-occurrence matri and discrete cosine transform features from an enhanced image to obtain objective image qualit assessment. Bouzerdoum et all. [6] uses Multi-Laer Perceptron (MLP) using modified Wang and Bovik features from JPEG/JPEG000 images in order to predict Mean Opinion Score (MOS) for images. Paolo Gastaldo and Rodolfo Zunino [7] improves his previous work without using reference image and also incorporates Principal Component Analsis (PCM) to etract best characterize features for JPEG/JPEG000 images. Here, our work is applicable for various distortion techniques while predicting Difference Mean Opinion Score with ease of using an interface. In subjective qualit assessment, even though MOS or DMOS establish the fundamental aspect for signal processing analzing purposes, the also bring several drawbacks including: -) subjective qualit assessment test is costl and time consuming; -) it is not automated or cannot be automated; 3-) it is not a real time evaluation. In objective qualit assessment, several limitations of this method can be addressed when: -) the

2 correlation of objective qualit assessment with human visual perception has been questioned; -) all qualit effecting parameters can not be considered at the same time; 3-) high computing power ma be required. In this stud, the ultimate goal is to design and develop a hbrid sstem based on a neural network algorithmic approach while taking into account subjective parameters. Besides, the unique framework is devised as automated and real-time. The MATLAB interface ehibits user-friendliness which is one of the major intended characteristic of the framework. The organization of the paper is addressed as follows: Section provides a brief overview of Image Qualit Assessment. The neural network is summarized and the approach is eplained with details in Section 3. The eperimental results of our approach are presented in Section 4. In Section 5, the conclusions of this paper are summarized.. IMAGE QUALITY ASSESMENT The basic approach for developing image qualit assessment models includes: ) The selection of a set of features which can be objectivel measured and ) the establishment of a model which can simulate the HVS for the processing and analzing of selected features. B weighing individual features, perceptual models can predict not onl specific attributes but also a global image qualit. The main challenges are: first the HVS is etremel comple and not full understood and second, comple models are time consuming for real-time applications. Different objective metrics simulate different features of Human Visual Sstem (HVS). Natural images signals are highl structured. It is assumed that the measure of structural information changes provides a good estimation of the perceptual image distortion [8].The fundamental principle of the structural approach is that the human visual sstem is highl adapted to etract structural information from the visual scene and therefore measurement of structural similarit (or distortion) could provide good approimation of subjective perceptual image qualit. i i =,,..., N = i i =,,..., N where N is the number of piels and Consider two images = { } and { } i and i are the i th piels of the images of and, respectivel. SSIM- SSIM (, ) combines three comparison components, namel luminance- l (, ), contrast- c (, ) and structure- s (, ) [8]: SSIM (, ) = f ( l(, ), c(, ), s(, )) () Luminance, contrast and structure comparisons are defined as follows: l c μ μ + C (, ) =, C = ( KL μ + μ + C + C (, ) =, C = ( K L + + C + C3 C s(, ) =, C3 = + C 3 ) ) () where μ, μ,, and are means of and, variances of and and correlation coefficient between and. K and K are scalar constants that K, K << and L is the dnamic range of the piel values. Finall, SSIM inde ields to:

3 In Bovik [8] C=6.50, C= 58.5 (μ μ + C)( + C ) SSIM(, ) = ( μ + μ + C )( + + C ) (3) The main drawback of SSIM algorithm in spatial domain is that it is highl sensitive to translation, rotation and scaling of images. Comple-wavelet SSIM algorithm was developed to work in transform domain and can capture non-structured image distortions that is topicall caused b movements of image acquisition devices. CW-SSIM methods works onl when the amount of translation, scaling and rotation is small compared to the wavelet filter size. The problem can be solved using multi scale SSIM. 3. NEURAL NETWORK The Neural Network model we adopt in this stud is multilaer neural network. The fundamental of multilaer network [9] is based on the backpropagation algorithm. The backpropagation is also known as gradient descent in error. The backpropagation is essentiall generalization of the LMS (Least-Mean-Squares) algorithm. The idea of the backpropagation is to increase training speed, improve performance and obtain desired output values b adjusting weights and scaling inputs. The single perceptron can onl be used for the classes that are linearl separable whereas a tpical multilaer network [0], which is learned b the backpropagation algorithm, has aptitude to classif the classes which are not necessaril linearl separable. 3.. Feedforward Operation A simple three-laer neural network [9], illustrated in Figure, has input, hidden, and output laers. There is also a single bias unit that has connection with all units ecept input units. Since these connected components treat like biological neurons, the are named as neurons. All input vectors are presented to the input laer, and the outputs of input neurons corresponds to component vectors. Net activation is computed in hidden neurons which take the weighted sum of inputs into account. Briefl, net activation is the sum of the inner products of weight and input vectors added b initial weight. The net activation is shown below. d t net = w + w = w w, j i ji j0 i ji i= i= 0 d (4) where i denotes ith input, and j denotes a node (neuron) in the hidden laers. The activation function f (.) produces output of the hidden neuron: j = f( net) (5)

4 Figure 3.. A simple three-laer neural network scheme Figure : A simple three-laer neural network scheme 3.. The Backpropagation Algorithm The Backpropagation algorithm [9] is one of the most common methods in multilaer neural networks for supervised learning. The major issue here is to obtain the desired outputs b adjusting weight vectors that are based on inputs provided b the training sets. The idea behind the backpropagation algorithm is to obtain the sum of the smallest squared difference between actual outputs and desired outputs. This learning rule is valid for not onl two-laer architecture, but also for three-laer architecture. The underling idea behind network learning is to present a training pattern to an untrained network s input laer and to let the signal pass through hidden laer to output laer. Output laer then generates output. The difference between generated output and desired output (target value) is called an error. The aim of the learning rule, which is presented here, is to minimize the error adjusting the weights and presenting learning rule is also pattern basis method. The training error on a pattern basis can be defined as the sum of the squared difference between desired output (target value) and the actual output. This can be epressed with the following equation. A description of the backpropagation algorithm can be found in [9]. c Y( w) ( tk zk) (6) k = In this work, we use neural network to model and evaluate in real-time how human subjects estimate image qualit when distorted b changes in qualit effecting parameters such as JPEG, JPEG000, Gaussian blur, fast fading, and white noise. Here, the neural network architecture we adopt is multilaer feedforward neural network. Besides, the backpropagation is selected as learning algorithm for proposed framework. In classification view point, the backpropagation is capable of mapping nonlinear relations between the qualit effecting parameters and DMOS.

5 4. EXPERIMENTAL RESULTS The LIVE database [] is selected as test bed to perform eperiments and validate our approach. The LIVE image database contains 9 high-resolution 4 bits/piel color reference images and theirs distorted images under five distortion tpes: JPEG000, JPEG, White noise, Gaussian blur, and bit errors. Here, we chose 50 distorted images from each distortion tpe. Each distorted image has a computed Difference Mean Opinion Score (DMOS) ranging from to 00. JPEG000 images were generated using various bit rates. White noise images were obtained using White Gaussian noise. Gaussian kernel was used to create Gaussian blurred images. Fast-fading Raleigh channel model was utilized to generate transmission errors in JPEG000 bit stream. The eperiment consists of two major steps, namel, training and testing. The training section encompasses three aspects as follows; creating feature vectors, obtaining target vectors and designing neural network architecture. To create feature vector, we divide an image into grids such that 8 b 8 sliding window can scan through entire image. In this 88 window, statistical features such as mean, standard deviation and covariance are computed for each original and distorted image pair for a batch process. This process is repeated for each 4 pairs of original and distorted images among 5 pairs. The reason is that 4 pairs of images (original and distorted) are used for training and pair is considered for testing purposes. B doing so, the feature vectors are generated for a tpe of images. This is for onl one batch training process. This process is repeated for 50 image pairs for each distortion tpe. Obtaining target vectors are essentiall based on subjective score, namel DMOS, carried out b human observers. The DMOS value is alread provided for each distorted image given in the LIVE database set. To be able obtain the DMOS value corresponding each 88 window, we use mean weighed technique as follows. We calculate mean of each window and entire image. The DMOS score corresponds to mean of the entire image, and computed DMOS is assigned each window based on theirs weighed mean. With this fashion, target vectors are generated corresponding feature vectors. As we mentioned in the previous section selected neural network architecture is multilaer feedforward neural network. The backpropagation is chosen as learning algorithm for proposed framework. The number of hidden laers is composed of 3 and, each hidden laer consists of 6 neurons. The logistic sigmoid activation function is used in the hidden laers and the linear activation function is emploed in the output laer. Using feature vectors and target vectors under determined neural network architecture, training process is achieved which gives a net which is saved for testing section. The MATLAB training interface screen shot is depicted in Figure for training section. In the testing section, the first pair image on the left, among 5 pairs for the same tpe of images is used to ield input vectors with the same method eplained above, as the MATLAB testing interface screen shot illustrated in Figure 3. After the input vectors are fed into the neural network sstem, the ultimate goal is obtain DMOS score. The output of neural network would be predicted DMOS score. This process is repeated for all 50 images under various distortion tpes of the original images. With this procedure, obtained data is depicted in the scatter plot in Figure 4. Non-linear regression analsis was performed to fit data. Each sample point in the scatter plots has corresponding DMOS and neural network output values. DMOS values are represented in the -ais, whereas predicted DMOS values are located in the -ais of scatter plot. As shown in Figure 4, as DMOS values increase, predicted DMOS values increase meaning that the are in proportion.

6 Figure : The MATLAB training interface Figure 3: The MATLAB testing interface

7 Figure 4: Scatter plot of DMOS vs. Predicted DMOS b NN 5. CONCLUSIONS In this research, the primar goal is to design a model based on multilaer neural network for predicting the subjective image qualit score, known as the difference mean opinion score carried out b participants, in image qualit assessment. Along with the application of this technique, statistical features are etracted from both test and original images using a hand tool. We validate our proposed method using the LIVE Image Dataset Release. Based on our proposed eperiment, the obtained output of Neural Network correlates well with the DMOS values. The trained Neural Network can obtain the abilit to imitate human observers.. REFERENCES [] Wang Z., Bovik A. C., Modern Image Qualit Assessment, Morgan & Clapool, 006 [] Wang Z., Bovik A. C., Mean Squared Error: Love It or Leave It?, IEEE Signal processing magazine, vol, no, Januar, 98-8, (009). [3] H. R. Sheikh, A. C. Bovik, and G. de Veciana, An information fidelit criterion for image qualit assessment using natural scene statistics, IEEE Trans. Image Process., vol. 4, no., pp. 7 8, Dec. 005 [4] H. R. Sheikh, M. F. Sabir, and A. C. Bovik, A statistical evaluation of recent full reference image qualit assessment algorithms, IEEE Trans. Image Process., vol. 5, no., pp , Nov. 006 [5] Gastaldo P., Zunino R., Vicario E., Henderick I., CPB neural network for objective assessment of image qualit, Proc. IJCNN, Portland USA, pp. 94-9, 003.

8 [6] Bouzerdoum A., Havstad A., and Beghdadi A., Image qualit assessment using a neural network approach, in Proc. 4th IEEE Int. Smp.Signal Process. Inf. Technol., pp ,004. [7] Gastaldo P., Zunino R., "Neural networks for the no-reference assessment of perceived qualit", J. Electron. Imaging 4, , Aug 7, 005. [8] Wang Z., Bovik A C., Image Qualit Assessment: From error Visibilit to structural Similarit IEEE Transaction on Image Processing, vol 5, no 4, 600-6, April (004). [9] R. Duta, P. Hart, and D. Stork, Pattern Classification, New York, NY: John Wile and Sons Inc., 00. [0] T. M. Mitchell, Machine Learning, McGraw-Hill International Editions, 997. [] H. R. Sheikh, Z. Wang, L. Cormack and A. C. Bovik, "LIVE Image Qualit Assessment Database Release ",