IMAGE VISUAL QUALITY METRICS VERIFICATION BY TID2013: EXPLORING OF MEAN SQUARE ERROR DRAWBACKS

Transcription

1 IMAGE VISUAL QUALITY METRICS VERIFICATION BY TID: EXPLORING OF MEAN SQUARE ERROR DRAWBACKS Nikolay Poomareko ( ), Vladimir Luki( ), Oleg Ieremeiev ( ), Beoit Vozel( ), Kacem Chehdi( ), Kare Egiazaria ( ), Jaakko Astola ( ) ( ) Natioal Aerospace Uiversity, Kharkov, Ukraie ( ) Uiversity of Rees, Laio, Frace ( ) Tampere Uiversity of Techology, Tampere, Filad ABSTRACT Specialized image databases like TID, allow quatitative evaluatio of adequateess to huma perceptio characterized by mea opiio score () ad a verified full-referece visual quality metric usig certai criterios such as rak order correlatio coefficiets (ROCC). I this paper, we propose modificatios of kow Spearma ad Kedall ROCCs that take ito accout the fact that for existig databases has bee measured with a limited accuracy ad is most certaily erroeous (cotais oise compoet ). For the database TID we have also marked images with practically ivisible distortios. This has led to cosiderig criteria for detectig such images. Usig a mea square error we also aalyze metrics based o alterative orms l, l, ad l 4. Aalysis is performed for the database TID ad differet sesitivity of huma visio system (HVS) to distortios i color ad itesity is take ito accout. Oe approach to aalyze adequateess of metrics for differet types of distortios is described ad proved exploitable. Idex Terms Image quality, image color aalysis, detectio algorithms, correlatio coefficiet, mea square error methods. INTRODUCTION Full-referece image visual quality metrics [] are widely used i various applicatios of image ad video processig. They, i particular, iclude lossy compressio, image/video deoisig, digital watermarkig, etc. Efficiecy of solvig these tasks cosiderably depeds upo a adequateess of used HVS-metric to image/video perceptio by humas. For example, if a iadequate metric has bee exploited i a desig of image deoisig method, the it is hard to expect that a desiged method would occur efficiet i practice. It is difficult to preset a adequateess of a give metric to huma perceptio by some mathematical expressio. Because of this, researchers employ several stadard statistical approaches applicable for such large databases as TID8 [] or LIVE []. A large umber of voluteers (observers, participats) are attracted to subjective experimets iteded for obtaiig mea opiio score for each image i a database. The, all images of a give database are used to verify a give metric for the obtaied array by calculatig correlatio factors. Rak order correlatios such as Spearma (SROCC) ad Kedall (KROCC) [4] are used more ofte to avoid data overfittig. Their larger (approachig to uity) values show that there is a good correspodece betwee a aalyzed metric ad image perceptio (visual quality assessmet) by humas. Values of for ay image database are based o opiios of idividual observers that are subjective ad ca be iflueced by such errors as iattetio, radomly pressig a wrog butto (grade, image) ad so o. Besides, a umber of experimets ad comparisos for each particular image is limited i spite of huge efforts ad time spet o performig experimets. However, ote that alogside with array offered for each database, the database creators ofte preset the values of stadard deviatio (STD) or variace for each. This STD values characterize accuracy of experimetal data. However, the observed errors i are ot take ito cosideratio i calculatig SROCC ad/or KROCC ad i metric verificatio. To partly get aroud this shortcomig, below we itroduce modified versios of SROCC ad KROCC that are able to icorporate data o STD of for images. These modified SROCC ad KROCC are later used i our aalysis. A statistical estimate of metrics correspodece to a huma perceptio is better if a database is more represetative, i.e. cotais more distorted images. The largest opely available database is TID [] that cotais test images with 4 types of distortios ad levels of distortios. Therefore, i aggregate, the database icludes distorted color images. However, such a large umber of images forces to carry out experimets for

2 obtaiig with dividig distorted images ito subsets. For example, for TID8 ad TID, the subjective experimets have bee carried out separately for subsets of distorted images that relate to each of etalo images. To get aroud this problem, we modify methodology of ROCC correlatio that takes ito accout aforemetioed peculiarity to obtai. A typical image processig task dealig with image visual quality is to provide a ivisibility of itroduced distortios. Examples are lossy image compressio ad digital watermarkig. A good metric should be able to detect a visibility of itroduced distortios. It is desirable to aalyze this at the stage of metrics verificatio. We show how this appears possible due to markig images with ot oticeable (ivisible) distortios i TID. It has bee show i [] that a good HVS metrics should take ito accout color iformatio ad that for certai distortio types there are uderestimatio or overestimatio of visual quality. Usig the database TID, we aalyze these factors o the stadard metric Mea Square Error (MSE) as well as o the metrics based o other orms, l, l, ad l 4.. PROPOSED MODIFICATIONS TO A VERIFICATION PROCESS As it was metioed, useful iformatio o metric adequateess is provided by SROCC ad/or KROCC that characterize differece i positios of images i sorted array of a metric ad sorted array of values. However, due to aforemetioed errors i estimatig, there are situatios whe ot oly a cosidered metric is ot ideal but also is oisy ad, thus, its positio i the sorted array is displaced (with respect to a positio i case of umber of experimets approachig ifiity). For example, of the image I.bmp i TID is equal to. whilst for the image I.bmp oe has.. Both images are corrupted by additive white Gaussia oise where for the latter image the oise variace is larger. The, visual quality of the former image has to be higher ad its should be larger but this does ot happe. However, takig ito accout that STD values for these images are equal to. ad., respectively, the observed differece i values ca be expected. However, such a fact leads to decreasig ROCCs values. To decrease the ifluece of such errors i o calculated ROCC, let us itroduce the modified KROCC K R calculated as K R = δ(i, j) ( ), () i+ where is the umber of estimates of,, Mi σi M j δ( i, j) =, Mi is a value of i-th elemet, Mi σi > M j of the array arraged i ascedig order accordig to metric values, σ i deotes the correspodig STD. Expressio for the modified SROCC ca be writte as: R S = λ(i), () ( + )( ) i= where λ (i) deotes a differece betwee raks of i-th image i sorted (i ascedig order) arrays of a cosidered metric ad. Here, i a differece calculatio, the images for which their differs from of i-th image by o more tha σi are ot take ito accout. If σ=, expressios () ad () reduce to stadard KROCC ad SROCC. Sice values i TID have bee formed separately for each of test (etalo) images [], modified SROCC ad KROCC ca be separately determied for each subset of images ad the averaged it R it K = Kl, = l= R S Sl () l= where K R l ad S R l are modified KROCC ad SROCC calculated accordig to () ad () for distorted images relatig to l-th etalo image. Table cotais data for several metrics. We preset covetioal values of SROCC ad KROCC give i [] ad the modified values determied by (). Table. Values of covetioal SROCC ad KROCC ad the values of S R, K R, K it ad S it for several metrics Metric SROCC KROC S R K R C S it K it FSIMc [] PSNR-HA [].9. MSSIM [8] SSIM [9] As it follows from data i Table, the metric FSIMc [] remais to be the best accordig to the modified ROCCs ad SSIM [9] remais the worst. Meawhile, K it is always smaller tha the correspodig S it similarly as a covetioal KROCC is smaller tha the correspodig SROCC. Note that K it is larger tha the correspodig KROCC ad S it is larger tha the correspodig SROCC. This is because the ifluece of errors preset i the obtaied has bee reduced. Note that advatages of the metrics FSIMc ad PSNR-HA ca be attributed to the fact that they take ito accout color iformatio [].

3 Let us cosider ow a ivisibility of distortios ad detectio of such images by HVS-metrics. To aalyze this, all distorted images i TID have bee visualized ad viewed by a group of experts to determie ad mark those images for which distortios are practically (with high probability) ivisible. Totally 4 out of total images i TID have bee marked. As it ca be expected, visibility of distortios depeds upo image cotet ad distortio type. Fig. shows the umber of distorted images for each group relatig to each etalo image ( totally) ad Fig. illustrates a depedece o distortio type (4 totally). Number of distorted images where distortios are almost ivisible Referece image Fig.. Number of images with ivisible distortios for each referece (etalo) image Number of distorted images where distortios are almost ivisible Type of distortio Fig.. Number of images with ivisible distortios for each type of distortio As it ca be see i Fig., ivisible distortios more ofte take place for referece images which either cotai a lot of fie details or are highly textural (images # ad #, respectively). Cocerig distortio types (Fig. ), the most ivisible type i TID is a distortio type #4 (chromatic aberratios). Amog leaders i this sese, there are distortio types # (due to JPEG compressio), #4 (masked oise), # (additive oise i color compoets), # (comfort oise). These results oe more time support kow observatios o differet sesitivity of HVS to distortios i color ad itesity as well as a ifluece of maskig effects. Availability of the marked images with ivisible distortios allows aalyzig performace of differet metrics for detectig such images. Usually, detectio is 4 characterized by, at least, two probabilities. However, recetly a ew parameter called AUC [] that provides a opportuity to characterize a detectio by oly oe value i the limit from to (perfect detectio) has bee itroduced: A=(S-(+)/)//, (4) where ad are the umbers of positive ad egative examples, S is sum of raks of positive examples i the raked list. Table presets the values of AUC for detectig images with ivisible distortios i TID usig differet HVS-metrics. As it ca be see, FSIMc ad MSSIM are the best whilst PSNR-HA ad SSIM produce worse results. Table. Values of AUC for some HVS-metrics FSIM c PSNR-HA M SSIM SSIM ANALYSIS OF MEAN SQUARE ERROR MSE ad peak sigal to oise ratio (PSNR) which are covetioal metrics i image processig have bee may times reported as ot adequate for characterizig image visual quality. However, the operatio of MSE estimatio i several modified forms is still preset i may visual quality metrics []. Thus, it is worth startig to cosider the ways to improve a metric performace just from MSE ad its closest derivatives. Recall that MSE is calculated as W T MSE = (Aij Aij ), () WT where A is referece image, A is a image with distortios, i ad j are pixel idices, W ad T defie image size. Let us preset a extreme example showig drawbacks ad iadequateess of MSE. Image i Fig.,b is corrupted by AWGN with variace 9. The image i Fig..c is the homogeeous image that has the same mea as the test image Barbara i Fig.,a ad its MSE is also equal to 9. a) b) c) Fig.. Example of iadequateess of MSE to huma perceptio: a) referece image, b) image corrupted by AWGN with variace 9, c) homogeeous image that also has MSE=9 with respect to the referece image

4 As it is see, metrics that use pixel-by-pixel image compariso are ofte uable to adequately characterize image visual quality. Other examples are possible. However, our study is iteded o searchig ad aalyzig other tedecies usig the database TID. I particular, alogside with MSE that exploits l orm, it is possible to use other alteratives as mea absolute error (MAE) based o l as well as the metrics E l ad E l4 based o the l ad l 4 orms: W l E = Aij Aij, WT T W T l4 4 E = (Aij Aij ). WT Table presets the values of S it ad K it for these metrics calculated ad averaged for the three compoets of color images i RGB color space. MSE has the largest correlatio but the results for the metrics E l ad E l4 are quite close ad better tha those for MAE. Table. Values of S it ad K it for the cosidered metrics Criterio Metric MAE M SE E l E l4 S it K it Note that while calculatig S it ad K it, the values of these metrics have bee take with the opposite (egative) sig to provide icrease of the trasformed metric values for higher visual quality (ad ) of images i TID. Aggregate modified SROCC Aggregate modified KROCC D Fig.. Depedece of S it o D for the metric MSE D Fig.. Depedece of K it o D for the metric MSE () Although may metrics that do ot take ito accout color iformatio are calculated for R, G, ad B compoets separately ad the averaged, this way is ot the best []. The database TID cotais may images with specific color distortios ad this allows takig ito accout differet sesitivity to distortios i color ad itesity compoets usig YCbCr color space. Suppose that a metric is calculated as a weighted sum of this metric values for the itesity compoet Y (this weight is fixed ad equals to uity) ad color compoets Cb ad Cr (for both compoets we used the same weight D). Figures ad show the depedeces of the modified SROCC ad KROCC o D for the metric MSE. Table 4 presets approximately optimal D for the cosidered metrics as well as the correspodig values of S it, K it, ad AUC. As it is see, after weight optimizatio, the metric MSE remais to be the best accordig to all three aalyzed criteria. The results of aalysis show that it is worth desigig HVS-metrics based o the orm l. Thus, we cotiue our aalysis for oly MSE. Let us deote the weighted MSE as. Table 4. Results of optimizig D for the cosidered metrics Criterio Metric MAE M SE E l E l4 D S it K it AUC Figures ad 8 preset the values of the modified SROCC ad KROCC for each referece image of TID. Modified SROCC Referece image Fig.. Modified SROCC betwee ad for each referece image of TID Modified KROCC Referece image Fig. 8. Modified KROCC betwee ad for each referece image subset of TID

5 Diagrams i Figures ad 8 show that the referece images #, #4, ad # are the most complex for adequate characterizatio by. The images # ad #4 are textural ad does ot take maskig effects typical for these images. Complexity of the referece image # cosists i the fact that it is artificial ad cotais differet cotrast oise-like texture whilst homogeeous areas are abset. Fig. 9 presets the plots of first ad secod type errors i detectig images for which distortios are ot visible. Detectio has to be doe by settig a certai threshold of PSNRw (or, equivaletly ). Here we preset data for PSNRw coected with as PSNRw = log( /). Type I ad Type II errors 8 4 Type I error (umber of false positives) Type II error (umber of false egatives) PSNRw, db Fig. 9. Depedeces of the first ad secod type error umbers o PSNRw for detectig images with ivisible distortios i TID As it is see from a aalysis of the depedeces i Fig. 9, PSNR (ad, respectively, MSE) is uable to serve well for detectig images with ivisible distortios. Suppose that we eed 9% of such images to be detected (i.e., 9 out of 4 images with ivisible distortios) ad this is achieved for the threshold PSNRt=8 db. However, the umber of false positives (i.e., wrogly detected images) is over ad exceeds the umber of true positives by about times. To dimiish the umber of falsely detected images with ivisible distortios (e.g., to provide ot more tha % of false detectios), oe has to set PSNRt=. db. However, i this case the umber of false positives is more tha half ( false positives, true positives). Thus, a questio what threshold to set does ot have a perfect aswer ad the settig is problematic. Let us cosider ow what types of distortios are most ufavorable for. For this purpose, we have modified a expressio () to determie a cotributio β(j) of each distortio type #j i TID to the sum λ(i) : i= β( j) = % ϑ(i) λ(i) λ(i), () i= Type of distortio Fig.. Cotributio ito decrease of the modified SROCC of distortio type i TID As it ca be see, additive white Gaussia oise (distortio type #) has the smallest cotributio to reductio of SROCC ad, therefore, it is the simplest distortio type of (ad this feature of MSE is widely exploited). Let us use the data for this distortio factor as referece curves i our further aalysis. More i detail, we will aalyze depedeces of o for differet other types of distortios. If a curve for a give distortio type goes below the referece curve, the the metric decreases distortio level (overestimates the quality) ad vice versa [] Additive oise (#) Mea shift (#) Cotrast chage (#) Color saturatio (#8) Fig. vs for distortio types #,#,#,#8 Fig. presets depedeces for distortios due to mea shift chage (#) ad cotrast chage (#) which are the most complex for (see data i Fig. ) ad color saturatio distortios (#8). As it is see i Fig. ), cosiderably overestimates distortios for mea shift chage ad cotrast chage for the cases of ehaced cotrast (that correspod to about ad 4). Meawhile, for color saturatio the quality of the correspodig images is overestimated. Fig. presets the plots for distortio types that relate to image deoisig tasks. where ϑ (i) equals to, if a i-th image relates to distortio type #j ad to otherwise.

6 Additive oise (#) Multiplicative oise (#9) Impulse oise (#) Image deoisig (#9) Color oise (#) Correlated oise (#) Additive oise (#) High frequecy oise (#) Gaussia blur (#8) Patter oise (#4) Comfort oise (#) Quatizatio oise (#) Fig. vs for distortio types #, #, #, #, #9 ad #9 relatig to image deoisig task overstimates quality for impulse oise distortios as well as for residual distortios after deoisig or for spatially correlated oise. Meawhile, level of distortios of the type # (oise i color compoets) is still overestimated by. This drawback could be slightly compesated by usig D smaller tha recommeded (see Table 4). However, this leads to additioal overestimatio of image quality for the distortios of type #8 (see data i Fig. ). Fig. presets depedeces for distortio types #, #4, #, #, #, ad #4. overestimates quality for JPEG (#) for small compressio ratios (that correspod to small ) ad uderestimates quality for high CR. 4 Additive oise (#) Masked oise (#4) JPEG (#) JPEG (#) Chromatic aberratios (#) Sparse samplig (#4) Fig 4. MSE vs for distortio types #, #, #, #8, #4 ad # As it is see, uderestimates distortio level for quatizatio oise (#) ad high levels of blur (# 8). Meawhile, this metric cosiderably overestimates distortio level for high-frequecy oise (#). This is because this metric is uable to take ito accout differet sesitivity of HVS to distortios i differet spatial frequecies. Similarly, overestimates oise level for o eccetricity oise (#4) ad for comfort oise (#) due to iability to accout for maskig effects.. CONCLUSIONS The paper presets possibilities of verifyig quality metrics usig TID usig MSE ad its derivatives as examples. The images i TID that have ivisible distortios have bee marked ad this has allowed to evaluate detectability of such images usig the cosidered metrics. Modified versios of SROCC ad KROCC are proposed that take ito cosideratio errors of. A way to determie a cotributio of a give type of distortio to iadequateess of a give metric to huma perceptio of image quality is proposed. 4 4 Fig. vs for distortio types #, #4, #, #, # ad #4 Similar properties are observed for chromatic aberratios (#4) (for small distortio level they are masked ad these effects are ot take ito accout by ). For JPEG (#) ad sparse codig (#), image quality is overestimated for all distortio levels. For the masked oise (#4), overestimates distortio level sice it does ot take ito cosideratio maskig effects. Fig. 4 presets the results for distortio types #, #, #, #8, #4, ad #.. REFERENCES [] B.W. Keela, Hadbook of Image Quality. Marcel Dekker, Ic.: New York,. [] N. Poomareko, V. Luki, A. Zelesky, K. Egiazaria, M. Carli, ad F. Battisti, TID8 - A Database for Evaluatio of Full-Referece Visual Quality Assessmet Metrics, Advaces of Moder Radioelectroics, Vol., pp. -4, 9. [] Sheikh, H.R., Wag, Z., Cormack, L., Bovik, A.C.: LIVE Image Quality Assessmet Database Release, [4] M.G. Kedall, The advaced theory of statistics. Vol., Lodo, UK, Charles Griffi & Compay limited, 94. [] N. Poomareko, O. Ieremeiev, V. Luki, K. Egiazaria, L. Ji, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, C.-C. Jay

7 Kuo, A New Color Image Database TID: Iovatios ad Results, Proceedigs of ACIVS, Poza, Polad, pp [] L. Zhag, L. Zhag, X. Mou, ad D. Zhag, FSIM: a feature similarity idex for image quality assessmet, IEEE Trasactios o Image Processig, Vol., No, pp. 8-8,, [] N. Poomareko, O. Eremeev, V. Luki, K. Egiazaria, ad M. Carli, Modified image visual quality metrics for cotrast chage ad mea shift accoutig, Proceedigs of CADSM, Ukraie, pp.-,. [8] Z. Wag, E. P. Simocelli, ad A. C. Bovik, Multi-scale structural similarity for image quality assessmet, i Proc. IEEE Asilomar Cof. Sigals, Syst. Comput., pp. 98 4,. [9] Z. Wag, A. Bovik, H. Sheikh ad E. Simocelli, Image quality assessmet: from error visibility to structural similarity, IEEE Trasactios o Image Processig, Vol., Issue 4, pp. -, 4. [] C.X. Lig, J. Huag ad H. Zhag, AUC: a Statistically Cosistet ad more Discrimiatig Measure tha Accuracy, Proceedigs of 8th Iteratioal Coferece o Artificial Itelligece, pp. 9-,.