REDUCED-REFERENCE ASSESSMENT OF PERCEIVED QUALITY BY EXPLOITING COLOR INFORMATION

Transcription

1 REDUCED-REFERENCE ASSESSMENT OF PERCEIVED QUALITY BY EXPLOITING COLOR INFORMATION Jdith Redi, Paolo Gastaldo, Rodolfo Znino, and Ingrid Heyndericx (+) University of Genoa, DIBE, Via Opera Pia a Genova Italy (+) Philips Research Laboratories Prof. Holstlaan AA Eindhoven NL and Delft Technical University, Meelweg 4 68 CD Delft NL { dith.redi, paolo.gastaldo, ingrid.heyndericx@philips.com ABSTRACT Image qality assessment systems based on a redcedreference paradigm rely on the extraction of salient featres both from a target image and the corresponding original, ndistorted image. Ths, they can spport realtime modeling of perceived qality. This paper shows that the color correlogram and its featres can be effective descriptors for a redced-reference system, embedding relevant information on the change in color distribtion in the images. In the proposed framewor, the featre-based descriptions of the images feed a doble-layer system: first, the artifact affecting the image is identified by exploiting a classifier based on Spport Vector Machines (SVMs); secondly, the nmerical representation of the image is mapped onto a qality score by a dedicated predictor exploiting the MltiLayer Perceptron (MPL) paradigm, which is specifically trained to assess image qality for a given artifact. Experimental reslts based on sbective qality data confirm the general validity of the approach.. INTRODUCTION Modeling perceived qality is a tas of paramont interest in the field of digital commnication technologies. Accrate qality assessment sally relies on data derived from experiments with hman sbects, which are ased to evalate the overall qality of a representative set of stimli. Collecting data throgh sbective experiments can provide ratings reflecting coherently hman perception, bt is also time consming, and often difficlt to model in a deterministic way []. Hence, when aiming at bilding a real-time system that optimizes the qality of any inpt signal atomatically, qality assessment that does not reqire any hman evalation, is needed. To this prpose, obective methods can be exploited [, 3]. These methods rely on the extraction of salient featres from images and on the processing of these featres towards a consistent estimation of the images qality. Obective methods sally adopt a fll reference paradigm [, 3], reqiring the nowledge of both the target image and the corresponding original, ndistorted image. In addition, some fll-reference methods exploit analytical models of the hman visal system (HVS) to improve their accracy and reliability. When woring in real time, thogh, the fll information on the ndistorted image is sally not available. Redced reference methods circmvent this lac by sing in the processing phase only a limited nmber of featres extracted from the original signal [, 4]. This paper presents a qality assessment system based on a redced-reference obective method. Some specific characteristics are extracted from the original, ndistorted image, as well as from the corresponding distorted (e.g. compressed) image. The non-linear mapping between the extracted featres and the qality rating is determined sing comptational intelligent methods, sch as a neral networ. Actally, by training the neral networ to mimic perceived image qality, the design of an explicit model of the hman visal system is avoided. In this research, color information is sed to constrct a compact bt informative featre-based description of the image. Artifacts introdced in images by digital systems seriosly affect the original color distribtion. Moreover, often the changes cased by a particlar artifact are very pecliar, so that it is liely that both the distortion and its amont can be identified by comparing the statistics of the original and distorted image. Color statistics have been widely sed for image indexing [6] and textre recognition [5], bt not for their potential contribtion to obective qality assessment. Ths, based on previos wor [7, 8], this research adopts a set of featres derived from the color correlogram [6], a second-order histogram of color information in an image. In particlar, the model extracts significant statistics from the he layer of an image, both at a local and at a global level.

2 A doble-layer system based on connectionist paradigms is designed to map this featre-based description of the image into qality scores. First, a classification algorithm is exploited to determine the artifact affecting the image. Then, a regression machine trained on solving the problem of assessing qality for images affected by that particlar distortion is sed to obtain a reliable estimation of the qality gain/loss of the sample with respect to its original version. The second release of the LIVE database [9] is sed for the performance evalation of or proposed approach. In particlar, three inds of distortion are addressed: White noise, Gassian Blr and JPEG compression. The reslts confirm the general validity of the connectionist paradigm, as well as of the se of color statistics to efficiently predict image qality.. FRAMEWORK FOR REDUCED REFERENCE IMAGE QUALITY ESTIMATION In formal terms, a redced reference qality prediction problem, in which a few data extracted from the original, ndistorted image are sed to predict the qality of a target image, can be expressed as follows. Let I be a reference image, and I the image obtained by applying some artifact to I, being r the distortion level. Let also x and ( n x r denote a nmerical representation of I and I, respectively. Finally, let q and q be the sbective qality ratings for I and ( n I r, respectively. Then, the adopted paradigm aims at estimating the distance ( n) d S ( q, q ) between the sbective scores associated with the two images by comparing the obective descriptors, x and ( n x r. It shold be noticed that the information extraction from the original image can be performed before transmission and the reslting descriptor ( x ) can be sent along with the image throgh the broadcast channel as metadata. The descriptor x is then extracted at the otpt side of the channel, as visalized in Figre. Figre also illstrates that the proposed redcedreference framewor decoples the prediction of image qality into two tass. The rationale behind sch an approach is that every ind of distortion modifies the image in a different way, ths reqiring a specific obective metric to feed the qality predictor. So, having determined the set A={a, a,, a n } of artifacts of interest, implementing a dedicated qality predictor for each of them can be a sccessfl strategy. As no a-priori nowledge can be assmed on the natre of the distortion affecting the inpt signal, the framewor operates as follows. First, an artifact detector exploits the descriptors x and ( n x r to identify the distortion (a i A) applied to Figre - Overall scheme of the proposed system for obective qality assessment the image I. This modle then forwards the image descriptors to the qality predictor P(a i ), designed to qantify effects on perceived qality cased by the artifact a i. Finally, the specialized qality predictor provides a estimation for the difference in qality between the reference and distorted images, ( n) ( q, q ) d S. 3. COLOR INFORMATION EXTRACTION FOR QUALITY EVALUATION For the proposed architectre to be effective, an efficient nmerical representation of the image is needed. De to limitations in the transmission bandwidth, data representing the original image has to be small-sized, bt still very informative. The choice of a proper nmerical descriptor for an image is therefore crcial. The present stdy proposes to represent an image by the second order statistics of its color information. Recent wor [5, 6] showed how second order histograms can be sccessflly employed in different applications. Gastaldo et al. showed that co-occrrence matrices [7, 8] are sitable for a featre-based representation of an image able to convey information on the visal qality. The color correlogram [6] belongs to the family of the second order histograms, as well as the co-occrrence matrix. Aiming at maintaining in the final description both local and global information, the color correlogram is compted on eqally sized sb-regions of the image, and a featre representation is constrcted for each of them. Global information is then obtained by

3 assembling local information in a single descriptive vector by sing a statistical approach. 3.. Local information extraction A color correlogram Η b of a sb-region b (inclding W b H b pixels) for a predefined distance, is a threedimensional matrix that describes the spatial distribtion in lminance, color or any other image property that can be represented by qantized vales. More precisely, the color correlogram describes how the spatial correlation of pairs of colors changes with distance. Formally, the entry Η ( i, of the correlogram matrix is defined as: b ( m, n), m < Wb, n < Hb, s.t. Ηb ( i, = m, n] = Ci; p, q] = C ; dist( m, n], p, q]) = () Each matrix element Η b ( i, specifies the probability of finding a pixel of vale C at a distance from a pixel of vale C i. The dist() operator denotes the measre of distance between pixels selected to calclate the color correlogram. In or present wor, the L -norm is embedded for the operator dist(). In practical terms, choosing the L -norm means that only those pairs of pixels with a distance in horizontal and vertical direction are considered in the comptation. The proposed system preserves local information by compting the color correlogram on sb-regions of an image. To this end, images are split into non-overlapping sqare regions, each holding N a N a pixels. For each bloc, the set of featres Φ = {f 0, f, f 5 }, as defined in table I, is extracted from the correlogram. 3.. Global-level nmerical representation The local information extraction phase otpts as many vales for each featre of Φ as the nmber of blocs. This is far too mch data for a descriptor that shold be sent in real time throgh a commnication channel. Moreover, sbective scores sally express the qality of a whole image, and not of a set of blocs. So, from a modeling perspective, the featre-based image description shold consist of a single vector, to be associated with the single qality score. Toward this end, bloc-based information is combined sing statistical descriptors, namely percentiles, to represent the distribtion of a featre f over the image. Recalling the notations sed in the previos paragraphs, the following psedo-code can be applied to both the reference and the distorted image for constrcting the global descriptors. Inpts: a pictre I, a descriptive featre f Φ and a vale for distance. Bloc-level featre extraction. a. Split I into N b non-overlapping sqare blocs, and obtain the set Β = { b ; m =,..., } m N b Table I Featres calclated from the color correlogram for describing the images Featre name b. For each bloc b m B: compte the associate color correlogram: Η ( i,. c. For each matrix Η : compte the vale x b m, m of featre f, and obtain the set: Χ = b m { x ; m,.., N }, m =. Global level nmerical representation. a. Compte a percentile-based description of Χ ; let p α be the α th percentile: ϕ α, = α ( ) p Χ b. Assemble the obective descriptor vector, x, for the featre f on the image ( n, I Definition Energy f 0 = [ Η b ( i, ] Eventally, the extraction process reslts in a descriptor of an image, Ī, being a global pattern, x. This otpt of the extraction process is forwarded to the doble-layer assessment system for qality estimation. 4. COMPUTATIONAL INTELLIGENCE FOR OBJECTIVE QUALITY ASSESSMENT The system described in section consists of two steps. First, the artifact affecting the image has to be identified; secondly, the nmerical representation of the image has to be mapped onto a qality score by a dedicated predictor, which is specifically trained to assess image qality for a given artifact. To complete these two steps, two very different problems have to be solved. The first layer consists of a i, Diagonal Energy [ ] f = Η b ( i, i) b b ( i, i, Entropy f = Η ( i, log Η Contrast f 3 = z P( z) Homogeneity f 4 = Η b [ + ( i ] i, Energy Ratio f 5 = f0 f x P ( z) = Ηb ( i, i = z { ; [ 0,00] } ϕ α, = α z b ()

4 classification problem. Therefore, a system able to identify, given a set of classes, to which class an inpt pattern belongs to, needs to be designed. When aiming to detect the artifact a i affecting the sample I, the inpt pattern is the global descriptor (), and the m classes represent the artifacts of interest a, a, a m. The mapping fnction relates the inpt vector to a discrete vale, identifying the artifact. The second layer, instead, mst otpt a continos vale, being a qality score. Therefore, this modle has to be designed to solve a regression problem. This research adopts Spport Vector Machines (SVM) for the classification tas and a feed forward neral networ, namely the Circlar Bac-Propagation (CBP) networ for the regression tas. 4.. Spport Vector Machines In binary classification problems, the system is reqired to discriminate between only two classes. Spport vector machines are widely proven to be powerfl tools for solving these tass. Given a set of np patterns Z ={ (x l, y l ); l=,..,np; y l {-,+} }, a SVM relies on the soltion of the following Qadratic Programming problem, to find the optimal hyper-plane w separating the two classes: np np min αlα m yl ymk( x l, xm ) αl (3) α l, m= l= 0 α l C, l np sbect to: = ylα l 0 l= where α l are the SVM parameters setting the classseparating srface and the scalar qantity C is a fixed reglarization term that rles the trade-off between accracy and complexity. The ernel fnction Κ() allows inner prodcts of patterns in a higher dimensional, transformed space, yet disregarding the specific mapping of each single pattern. If Φ(x ) and Φ(x ) are the points in the featre space that are associated with x and x, respectively, then their dot prodct can be written as Φ ( x ), Φ( x ) = K( x, x). A common choice for the ernel Κ() that has been proven to be effective, is the Radial Basis Fnction (RBF) ernel, formlated as: K ( x x = e x /σ x Problem setting (3) has the crcial advantage of involving a qadratic-optimization problem with linear constraints, ensring that the soltion is niqe. The evental generalization performance of a SVM depends on the specific choice for the ernel parameters {C, σ}. Both theoretical [0] and empirical [] approaches can be adopted to determine the generalization (4) limits. In or present research, the parameters {C, σ} are tned following an empirical approach involving -fold cross validation []. 4.. Circlar Bac-Propagation In or proposed system feed-forward neral networs map featre-based image descriptions into the associated estimates of perceived qality, which, in the present formlation, are scalar vales. Theory proves that feed-forward networs embedding a sigmoidal nonlinearity can spport arbitrary mappings []. The MltiLayer Perceptron (MLP) model [] belongs to this class of networs, and has been proved to perform effectively for a wide variety of problems. The Circlar Bac Propagation (CBP) networ [3] extends the conventional MLP with an additional inpt, being the sm of the sqared vales of all the networ inpts. The qadratic term boosts the networ s representation ability withot affecting the fritfl properties of an MLP strctre. The CBP architectre can be formally described as follows. The inpt layer connects the n i inpt vales (featres) to each of the n h nerons of a hidden layer. The -th hidden neron performs a non-linear transformation on the weighted combination of the inpt vales, with coefficients w,i ( =,, n h ; i=,, n i ): ni ni a = sigm w,0 + w, i xi + w, n + x i i i= i= (5) where sigm(z)=(+e -z ) -, and a is the neron activation. Liewise, the otpt layer provides the actal networ responses, y, ( =,, n o ): n h y = sigm w,0 + w, a = (6) The strctral CBP enhancement still allows to adopt conventional bac-propagation algorithms [] for weight adstment, yielding an effective training. A qadratic cost fnction measres the distortion between the actal system otpt and the expected reference otpt for a sample of training patterns. The cost is expressed as: n p no ( l) ( l) ( t y ) E = non p l= = (7) where n p is the nmber of training patterns, and t is the desired reference otpt. In the present application, = and the expected otpt is given by the qality score obtained from sbective panel tests.

5 5. PRACTICAL IMPLEMENTATION The present section describes a possible implementation of the proposed redced-reference framewor, consisting of an SVM-based modle to solve the artifact identification problem, and a CBP architectre, mapping featres into qality scores. 5.. Nmerical description The first step in the neral-based qality estimator is the featre extraction, in which both the reference image I and the target image r I ) are processed to yield the vectors ( n x ) and x, compted according to the procedre reported in Section 3. The inpt image is divided into sqared sb-regions of 3x3 pixels, and for each bloc, all pixels are converted from the RGB color space to the HSV color space. Then, the color correlogram is compted on the he component (H-component) of the blocs. From these vales, the set of featres Φ defined in table xx is calclated. Finally, the global descriptor is assembled by compting 6 percentiles of the distribtion of each featre f, and combining them in the vector, r x n = ; α = 0,0,40,60,80,00; = 0,,...,5. ( ) { } ϕ α, For each featre, the obective representation of the stimls is obtained simply by combining the two vectors: [ ( n) ( ] n, z = x, x (8) The reslting -dimensional vectors r z ) = 0,...,5 feed the doble-layer qality estimator. It shold be noticed that not always the whole featre based information vector is needed, as will be shown later when discssing or experimental reslts. 5.. Neral-Based Qality Loss/Gain Qantification The role of the artifact identifier modle in the system is to solve a mlticlass problem, associating each z to the appropriate artifact a i A. To implement this mlticlass classifier, several binary predictors are connected in series. In this stdy three possible artifacts are considered: White noise, Gassian Blr and JPEG compression. The classification modle is implemented sing a first SVM to select images distorted by White Noise, and a second SVM to separate blrred images from JPEG compressed images. Both SVMs receive as inpt a vector based on one featre, namely Energy Ratio (see f 5 in table I). This featre seemed sfficiently effective for the artifact classification. The second layer, i.e. the qality level predictor is composed of as many sbsystems as the nmber of considered artifacts; in or case three. The three prediction modles have the same architectre; in particlar, to improve robstness, an ensemble strategy is applied [4]. To this end, a coordinate-partitioning approach is sed; in particlar, each single predictor involved in the ensemble fed with a single, distinct featre, satisfying the reqired hypothesis of disoint inpt sb-spaces. Being independent from one another, each artifact-dedicated sb-modle maes se of a particlar pair of featres, designed to be as mch informative as possible for the specific artifact: to qantify the effect of white noise on qality, the pair {Contrast, Total/Diagonal Energy} is sed; the pair {Diagonal Energy, Homogeneity} is sed to describe blrred images and the cople {Diagonal Energy, Entropy} is sed to characterize JPEG stimli. 6. EXPERIMENTAL RESULTS The second release of the LIVE database [9] was sed as a testbed for the performance evalation of or proposed model. In particlar, the three datasets inclding samples distorted with White noise, Gassian Blr and JPEG compression were considered. To evalate the overall performance of the doble-layer qality estimator, a - fold-lie test method was applied. From the LIVE database, which incldes several distorted versions of 9 original images, each artifact specific sbset was divided into 5 different folds, each containing all the distorted versions of a few original images, in sch a way that none of the 9 contents belonged to more than one grop. Both layers were trained independently, nonetheless for each of them the -fold strategy was adopted, performing 5 rns. In each rn 4 of the 5 folds were sed as training data, and the remaining one was sed as test data. In this way, the system was proven to be able to generalize independently of the specific image content sed for the training. For all experiments, the color correlogram was compted for a distance = sing the L norm. For the artifact classification system, it was necessary to tne the ernel parameters to improve the machine s generalization ability. To this prpose, the -fold crossvalidation techniqe was applied. First, the three datasets provided by the LIVE database were merged in a single one. The first SVM was trained to recognize noisy images. For this SVM, a linear ernel was employed, and the parameter C was finally set to 0 3. The second SVM was trained on a sbset of the first dataset, inclding only blrred and compressed images. In this case, the se of a RBF ernel was necessary, confirming that the problem of distingishing blr from compression artifacts (which is intitively more difficlt than the previos tas, since the visal effects prodced by the two distortions often overlap) is indeed of increased complexity. Based on the cross-validation otpt, the parameter C was set to 0 5 and σ =.

6 Table II - Performance of each SVM machine for artifact identification in terms of % of misclassified patterns White Noise recognition Table II reports the percentages of misclassified patterns for each rn and for both SVM classifiers. It clearly illstrates that the first SVM has a perfect performance, since none of the blrred or compressed images is misclassified as noisy image. The performance of the second SVM is worse, de to its more complex tas. Nonetheless, in the worst case still only abot 0% of the images is misclassified. For the qality estimation layer the three ensembles of two CBP neral networs were each trained on a different dataset. The datasets for white noise and blrred images contained 45 patterns, while the remaining testbed inclded 59 JPEG compressed images. For each image a qality score was provided in the LIVE database. The Difference Mean Opinion Score (DMOS), originally ranging between [0,00] was remapped for comptational reasons into the range [-, +]. De to the flexibility of the system, it was possible to design the networ architectre on prpose for each tas. For coherence, all the 6 neral networs shared the same inpt layer with the dimensionality of the global descriptor z. The networs for the qality estimation of noisy and blrred images were eqipped with a 5-nerons hidden layer, while, to maximize the generalization ability, the qality estimator for the JPEG compressed images conted 7 hidden nerons. For the three sb-systems, two different indicators were sed to evalate their accracy, defined as the discrepancy between the obectively predicted qality ˆ ( n) d q, q and the vale provided by the LIVE change S ( ) ( n) database d ( q, q ) S Blr/Compression recognition Rn # 0.00% 0.00% Rn # 0.00% 4.76% Rn #3 0.00% 0.00% Rn #4 0.00%.9% Rn #5 0.00% 8.0% Average 0.00% 4.80% : the mean vale of the absolte prediction error µ err between d s and dˆ S (in percentage), and the root mean sqare (RMS) error. Table III shows how the second layer in or system is able to prodce satisfactory qality predictions. Assessing the qality gain/loss of noisy images, the system s mean absolte prediction error, averaged over the five folds, is 5.8%. Dealing with compressed images, the mean absolte prediction error is 6.03%, which improves the accracy obtained in past stdies [8]. Finally, the qality estimator for the blrred images achieves an accracy of 93.5%, which represents an improvement of.90% with respect to previos reslts [8]. Table III - Performance of qality estimators in terms of percentage absolte error and RMS between predicted and sbective qality scores White Noise JPEG Compression 7. REFERENCES Gassian Blr %µ err RMS %µ err RMS %µ err RMS Rn # 4.87% % % 0.9 Rn # 5.06% % % 0.06 Rn # % % % 0.08 Rn #4 5.8% % % 0.09 Rn #5 5.64% % % 0.5 Average 5.77% % % 0.75 [] Wang, Z., Sheih, H.R., and Bovi, A.C.: Obective video qality assessment. In: Frth, B., and Marqes, O.: The Handboo of Video Databases: Design and Applications. CRC Press, Boca Raton, FL (003) [] International Telecommnication Union, Methodology for the sbective assessment of the qality of television pictres, ITU-R BT.500 (995). [3] Sheih, H.R., Sabir, M.F., and Bovi, A.C.: A statistical evalation of recent fll reference image qality assessment algorithm. IEEE Trans. Image Processing 5 (006) [4] Z. Wang and E. P. Simoncelli, Redced-reference image qality assessment sing a wavelet-domain natral image statistic model, in Proc. SPIE Hman Vision and Electronic Imaging X, vol (005). [5] Haralic, R.M., Shanmgam, K., and Dinstein, I.: Textral Featres for Image Classification. IEEE Trans. On Systems, Man and Cybernetics SMC-3 (973) 60- [6] 8. Hang, J., Ravi Kmar, S., Mitra, M., Zh, W.-J., and Zabih, R.: Image indexing sing color correlograms. In Proc. IEEE CVPR 97 (997) [7] Gastaldo P. and Znino Z.: Neral networs for the noreference assessment of perceived qality. Jornal of Electronic Imaging 4 (005) [8] Redi J., Gastaldo P., Znino R. and Heyndericx I.: Cooccrrence matrixes for the qality assessment of coded images. In Proc. ICANN 08 (008) [9] Sheih, H.R., Wang, Z., Cormac, L., and Bovi, A.C.: LIVE Image Qality Assessment Database at [0] V. Vapni, Statistical Learning Theory. New Yor: Wiley, 998. [] Bartlett P, Bocheron S, Lgosi G Model selection and error estimation. Machine Learning, 00, Volme 48, Nmber -3 pp.85-3 [] Rmelhart, D.E., and McClelland, J.L.: Parallel distribted processing. MIT Press, Cambridge, MA (986). [3] Ridella, S., Rovetta, S., and Znino, R.: Circlar bacpropagation networs for classification. IEEE Trans. on Neral Networs 8 (997) [4] Kittler, J., Hatef, M., Din, R. P. W., and Matas, J.: On combining classifiers. IEEE Trans. Pattern Analysis and Machine Intelligence. 0 (998) 6 39