EXPERIMENTAL ANALYSIS AND MODELING OF DIGITAL VIDEO QUALITY Mylène C.Q. Farias, a Michael S. Moore, a John M. Foley, b and Sanjit K.

Transcription

1 EXPERIMENTAL ANALYSIS AND MODELING OF DIGITAL VIDEO QUALITY Mylène C.Q. Farias, a Michael S. Moore, a John M. Foley, b and Sanjit K. Mitra a a Department of Electrical and Computer Engineering, b Department of Psychology, University of California Santa Barbara, Santa Barbara, CA USA ABSTRACT In this paper we present a review of the work done at University of California Santa Barbara (UCSB) in the area of video quality. We performed a series of specially designed psychophysical experiments with the goal of studying the appearance, visibility, annoyance, and importance of different types of artifacts commonly found in digital videos. We used both real and synthetically generated artifacts. Full and reduced reference video quality metrics were developed based on a simple human visual system model and annoyance and detection models. For real-time applications, we have developed reduced and no-reference quality metrics based on measurements of individual artifacts. The proposed metrics have good performance and relatively low complexity. 1. INTRODUCTION A video impairment is any change in a video signal that if sufficiently strong will reduce the perceived quality. Video impairments can be introduced during capture, transmission, storage, and/or display, as well as by any image processing algorithm (e.g. compression) that may be applied along the way. Most impairments have more than one perceptual feature, but it is possible to produce impairments that are relatively pure. We will use the term artifacts to refer to the perceptual features of impairments and artifact signal to refer to the physical signal that produces the artifact. Examples of artifacts introduced by digital video systems are blurriness, noisiness, ringing, and blockiness [1]. There is an ongoing effort to develop video quality metrics that are able to detect impairments and estimate their annoyance as perceived by human viewers. Most of the quality metrics proposed in the literature are Full Reference (FR) metrics. These metrics estimate the quality of a video by comparing reference and impaired videos. Some examples include the works by Daly [2], Lubin [3], Watson [4], Wolf and Pinson [5], and Winkler [6]. A more complete survey of the available FR video quality metrics is presented in [7]. The major drawback of the FR approach is the fact that a large amount of reference information has to be provided at the final comparison point. Also, a very precise spatial and temporal alignment of reference and impaired videos is needed to guarantee the accuracy of the metric. Reduced Reference (RR) quality metrics are metrics that require only partial information about the reference video. In general, certain features or physical measures are extracted from the reference and transmitted to the receiver as side information to help evaluate the quality of the video. Metrics in this class may be less accurate than the FR metrics, but they are also less complex, and make real-time implementations more affordable. Some examples include the works of Webster et al. [8] and Brètillon et al. [9]. Requiring the reference video or even limited information about it becomes a serious impediment in many real-time transmission applications. In these cases, it becomes essential to develop ways of blindly estimating the quality of a video using a No-reference (NR) video quality metric. It turns out that, although human observers can usually assess the quality of a video without using the reference, creating a metric that will do this is a difficult task and, most frequently, results in a loss of performance in comparison to the FR approach. Most of the proposed NR metrics estimate annoyance by detecting and estimating the strength of commonly found artifact signals. For example, the metrics by Wu et al. and Wang et al. estimate quality based on blockiness measurements [10, 11], while the metric by Caviedes et al. takes into account measurements of 5 types of artifacts [12]. In this paper we present a review of two different projects completed during the last several of years [13, 14]. The main goal of the first project was to test some assumptions made in video quality metric research and to develop FR and RR metrics with the goal that the RR metric would have similar accuracy but reduced computational complexity. First, a set of experiments was performed that measured the detectability, annoyance, and importance of MPEG-2 compression impairments. Then, visibility and annoyance models were developed using a simplified model of the human visual system that took into account factors like defect size, duration, location, importance and video content. The goal of the second project was to investigate the possibility of designing a

2 quality metric based on blind artifact measurements. We first studied the visibility, annoyance, and strength of four of the more commonly found artifacts (blockiness, blurriness, noisiness, and ringing) by performing experiments using synthetically generated artifacts. Then, we created an annoyance model by developing a set of NR artifact metrics for estimating the strength of each of these artifacts and a rule for combining the outputs of these metrics to predict overall annoyance. 2. EXPERIMENTAL METHODOLOGY The normal approach to subjective quality testing is to degrade a video by a variable amount and ask the test subjects for a quality or annoyance rating. The degradation is introduced by compressing, transmitting, or processing an entire video in a way that will produce different degrees of impairments. The impairments introduced using this approach are rarely evenly spread. In this research, we have developed an experimental paradigm that introduces brief spatially and temporally limited impairments with varying strengths into video sequences [13]. We control the impairment strength by scaling the pixel-by-pixel difference between the corrupted and the original videos and inserting them into specific regions of the video for a short time interval. Different spatial regions and time intervals (defect regions) are used for each original to prevent the test subjects from learning the locations where the defects appear [13]. This method allows varying the strength of an impairment without affecting its waveform. Each experimental session includes five stages: instructions, training, practice trials, experimental trials, and interview. During the instructions, the experimenter orally describes the task to the subject. In the training stage, examples of the experimental sequences representing the impairment extremes are shown to the subjects to establish the score range. The practice trials are used to familiarize the subjects with the task and to stabilize their responses before the experimental trials. In the experimental trials subjects are asked to perform two or three of the following tasks: detection, annoyance, strength, appearance, and importance [13, 14]. The detection task consists of detecting a spatially and temporally localized impairment in a five-second video sequence. After each test sequence the subjects are asked Did you see a defect or an impairment?. The subject responds yes or no. The annoyance task consists of giving a numerical judgment of how annoying the detected impairment is. The subject is instructed to enter a positive numerical value indicating how annoying the impairment is. Any defect as annoying as the worst impairments in the training stage should be given 100, half as annoying 50, twice as annoying 200 and so forth. The description task consists of indicating the appearance of the detected impairment. For this task, the subject is given a set of classifiers and asked to choose the appropriate ones to describe the appearance of the impairment she/he has just seen. The strength task consists of estimating how strong each artifact is in the detected impairment. After the video is played, the subject is asked to enter a number in a scale with range from 0 to 10. Any artifact as strong as the strongest artifacts in the training stage is given 10, half as strong 5, and so forth. The content importance task consists of rating the importance of a region that is missing from the video. For this type of experiment, we generate a series of test sequences where the video content in the defect region is simply missing. The subjects are asked to give a response on a fixed scale ranging from 0 (insignificant) to 10 (vitally important). From the data gathered from the detection task, for each test sequence we compute the probability of detection (P) by dividing the number of subjects who detected the artifact by the total number of subjects. The PD as a function of the total squared error (logtse) (psychometric function) is fitted using the Weibull function [15], which is defined as: ( ) k ( xx ) P x = 1 2 T (1) where P(x) is the probability of detection, x is the logtse, x T is the 50% detection threshold, and k is the slope of the transition. From the data gathered from the annoyance, importance, and strength tasks we compute the mean annoyance values (MAV), mean strength values (MSV), and mean importance values (MIV), respectively, by averaging the subjective scores over all observers for each test video. The MAV as a function of the logtse (annoyance function), is fitted with the standard logistic function [15]: ( ( )) ( 100) 1 exp ( ) y = + x x β (2) where y is the predicted annoyance and x is the logtse. The parameter x (mid-annoyance logtse) translates the curve in the x-direction and β controls the steepness. 3. FR AND RR METRICS BASED ON DETECTION AND ANNOYANCE MODELS In our first project we performed psychophysical experiments where the appearance, strength, size, duration, and location of MPEG-2 impairments were varied [16]. The subjects performed detection, annoyance, and importance tasks. We found a strong correlation between the mid-annoyance defect strength ( x ) and the detection threshold ( x T ). In fact, x could be approximated using the following linear expression: x = cx T, where c was found to be equal to 1.14, 1.19, and

3 1.15 for three of the experiments. Figure 1 depicts the plot of x versus x T obtained from the data of one of the experiments [16]. We can conclude that the common assumption that annoyance can be predicted using a good detection model is correct. Analysis of variance of the parameters x and x T showed that the statistically most significant sources of variance were the original and the location of the defect region. They were more significant than variances due to size, duration or defect type. Indeed in most experiments type did not have a statistically significant effect. This means that, for errors of constant TSE, the context, and not the nature of the defect, is the primary determinant of its visibility and annoyance. The location and size of the defect region were the variables with the largest effect on the importance of a region. Surprisingly, the detection threshold did not significantly depend on importance, and annoyance was only weakly correlated to importance. This may be a consequence of our task, which required subjects to actively search for defects. Based on these results, separate models were developed for the dependence of threshold and annoyance values on defect strength. In Figure 2 a generic block diagram for both models is presented, where the output corresponds to an estimate of either the detection probability or mean annoyance values. For the detection model, the high level model corresponds to the psychometric function, while for the annoyance model it corresponds to a logistic function. Both models include a human visual system (HVS) model that takes into account human contrast sensitivity and spatial masking effects, as shown in Figure 3. The HVS model used in this work was developed starting with a simple HVS model and then adding complexity as necessary to improve prediction performance. A similar approach was also explored by Watson and Malo [17]. The HVS model implementation computed contrast using a single channel and Peli's definition of contrast [18]. For this study, the lowpass filter LLP was a simple FIR filter and the bandpass filter was its delay complement. The best filters to use for these calculations are not known. Many models use a multichannel approach, where a bank of filters implement a pyramid decomposition of the input [2]. Several early attempts to create perceptual quality metrics filtered the difference pattern using the human spatial contrast sensitivity function (CSF), in effect treating this part of the visual system as a linear system [2]. Although the visual system is known to be non-linear, a linear spatial filter can improve the performance of a fidelity metric by de-emphasizing high frequency components. The filter used for this work was derived from a CSF published in [19]. Figure 1 Plot of mid-annoyance versus detection threshold. Input Reference Register Videos Adjust HVS HVS + - Summarize High-level Fidelity Measure Figure 2. A generic fidelity measure overall block diagram. Reference Video Test Video R 0 T 0 Display Display R 1 T 1 Lowpass Contrast Contrast R 2 T 2 Spatial CSF Spatial CSF R 3 T 3 Masking Measure Variability Masking Figure 3. Schematic illustration of the HVS model. R 4 T 4 Sum of Squares There are many possible approaches to modeling masking, including contrast gain control models, noise masking, entropy masking, and activity masking. Our system used an activity masking model. Activity may be measured by summing deviations from a local mean or by summing the high frequency coefficients in the frequency domain [20]. Activity can be computed locally or globally [21]. Our system used an activity masking model. The implemented masking model used an estimate of the local contrast variation to weight the contrast values. Specifically, I ( q M IFC M0v 1) = +, where I FC are the CSF-filtered contrast values, M 0 is a scaling factor, v is the range of the values in a 12' local neighborhood around each pixel, and q is a compression factor. M0 and q were set to 40 and 0.7. The unit offset was added to ensure that the denominator never equaled zero. The local range is not a typical measure of variability, but it can be a good approximation to the local standard deviation and can be computed rapidly. After the model transformation has been applied separately to the impaired and the original (reference) videos, the difference between two signals is computed for each pixel. These differences are each raised to a TE

4 power greater than 1, weighted based on their position in space and time, and summed over the video. The common log of this Total Squared Perceived Error is related to detection probability by a Weibull function and to annoyance by a logistic function. The annoyance and detection models performed well for several experimental data sets using these fairly simple implementations for the contrast sensitivity and masking stages. The correlation of the FR metric was compared to of the Sarnoff model [22]. The execution time was much shorter than for the Sarnoff model. Figure 4 shows a plot of the measured annoyance versus the FR annoyance model output for the set containing data from four experiments. In the RR model there are two processing stages. In the first the reference video is processed to compute thresholds for faux patterns that are like MPEG-2 impairments. In the second stage both videos are highly downsampled and the difference computed. Then the thresholds for the faux patterns are used to adjust these crude difference measures. The RR model performed equivalently to the FR model. In some cases, the RR model actually improved on the FR performance. The amount of reference information required to process a test video was decreased 400 times and the execution time required by the comparison stage was decreased 10,000 times. These implementation gains were significant and did not decrease model accuracy. Nevertheless, it is important to emphasize that the RR model requires considerable time to compute the thresholds and allows for fast quality assessment only when there has been prior stage 1 analysis. 5. NR AND RR METRICS BASED ON ARTIFACT MEASUREMENTS When the reference is not available or insufficient processing time is available no-reference metrics are needed. Their design is often difficult because it is hard to differentiate the natural content of the video from the artifact signals. The approach taken in this project consists of designing individual artifact metrics and, then, combining them into an overall annoyance model. The assumption here is that, instead of trying to detect and estimate the strength of an unknown impairment signal, it is easier to detect individual artifact signals and estimate their strength because we know their appearance and the type of process which generates them. In Figure 5 a simplified block diagram of our NR model is shown. The development of an NR model requires a study of the visibility, annoyance, and strength of the most relevant artifacts, since we still do not have a good understanding of how each individual artifact depends on the physical properties of the video and how they combine to produce the overall annoyance. For this purpose we have used Figure 4. FR annoyance model. Figure 5. Block-diagram of the NR annoyance model. synthetic artifacts that look like real artifacts, yet are simpler, purer, and easier to describe. This approach offers a great degree of control with respect to the amplitude, distribution, and mixture of different types of artifacts and makes it possible, for example, to study the perceptual importance of each individual artifact. We have implemented algorithms for generating four types of synthetic artifacts commonly found in compressed video sequences blurriness, blockiness, ringing, and noisiness [23-26]. Using these synthetic artifacts, we have performed two sets of experiments. In the first set of experiments, subjects performed annoyance, detection, and description tasks. The results showed that the synthetic artifacts, besides being visually similar to real compression (MPEG-2) impairments, had similar psychometric and annoyance functions. Figure 6 shows the annoyance functions of four test videos containing only blockiness, blurriness, ringing, and noisiness for the original video Cheerleader. ANOVA tests performed on the data showed that the original had a significant effect on x and x T, while the artifact signal did not. It was also found that x and x T were highly correlated and related by x = a xt + b, where a and b varied according to the type of artifact. In the second set of experiments, half of the subjects performed strength tasks while the other half performed annoyance and detection tasks. When the individual synthetic artifact signals were presented by themselves with high TSE, the strength judgments indicated that they were correctly identified. At low TSE, on the other hand, other artifacts were also reported. Strength judgments for

5 combinations of artifact signals showed that noisy artifact signals seemed to decrease the perceived strength of the other artifacts, while blurry artifact signals seemed to increase them. Annoyance increased both with the number of artifact signals and their strengths. We examined both a linear model and a weighted Minkowski model for describing the relation between artifact strengths and overall Annoyance. The optimal value for the Minkowski exponent was 1.16 and the coefficients were 6.18, 9.35, and 7.74 for blockiness, blurriness, and noisiness. For the linear model, the coefficients were equal to 4.05, 6.36, and 5.31, for blockiness, blurriness, and noisiness. Both models produced a very good correlation (r = 0.87) with the data with no statistical difference in performance. Figure 8 shows a plot of the measured versus predicted annoyance using the linear model. The coefficients found for ringing were very low compared to the coefficients of the other three artifacts, so this artifact was excluded from the final model. This may be due to the fact that the other artifacts are more visible than ringing. We then attempted to predict the perceived artifact strengths from the physical artifact signals. We have designed a set of NR artifact metrics that are simple enough to be used in real-time applications. The metrics were tested using videos which contained only the desired artifact signal or combinations of artifact signals at several strengths. We obtained a model for overall annoyance based on a combination of the best artifact metrics using both a Minkowski metric and a linear model. Both models produced a good correlation (r = 0.86) with the data with no statistical difference in performance. The fit for the Minkowski metric returned an exponent equal to 0.66 and scaling coefficients equal to 0.91, 3.40, and 2.51, corresponding to blockiness, blurriness, and noisiness, while for the linear model the scaling coefficients were equal to 3.41, 7.40, and 5.39, corresponding to blockiness, blurriness, and noisiness. Figure 8 shows a plot of the Figure 6. Annoyance functions for 4 test videos containing blockiness, blurriness, ringing, and noisiness. Figure 7. Perceptual annoyance model (linear). Figure 8. NR Physical annoyance model (linear). measured versus predicted annoyance using linear metric. The annoyance models using the artifact metrics were somewhat similar to the annoyance models obtained using the artifact perceptual strengths, i.e., similar parameters were found for both perceptual and physical models. Nevertheless, the model using perceptual strengths produced a better fit than the one using physical strengths. This fact may be an indication that the perceptual and physical strengths of the artifacts are related by a non-linear function. A RR approach was also proposed that did not require any changes to the algorithms. It simply consisted of sending a single number per video that corresponded to the output value of the artifact metric for the original or reference video. The RR approach produced better fits with a correlation equal to 0.88, while the NR model produced a correlation of around SUMMARY AND CONCLUSIONS In this paper we present a review of two different projects completed during the last several of years. We performed a series of specially designed psychophysical experiments with the goal of studying the appearance, visibility, annoyance, strength, and importance of different types of artifacts commonly found in digital videos. We used both real and synthetically generated artifacts. The results

6 showed that the detection threshold and the midannoyance defect strength are correlated and related by a linear expression. Also, the original had a significant effect on mid-annoyance defect strength and detection threshold, while in most cases the artifact signal did not. FR and RR quality metrics using a simplified human visual system model produced good results. Nevertheless, these models are not fast enough to operate in real time. For real-time applications, we proposed a NR and RR approach based on simple artifact metrics. The outputs of the artifact metrics were combined in order to obtain an annoyance model. The model produced a slightly worse performance than the FR model, but it had the advantage of being simpler and not requiring the original. 5. ACKNOWLEDGEMENTS This work was supported in part by CAPES Brazil, in part by a National Science Foundation Grant, and in part by a University of California MICRO Grant with matching support from Philips Research Laboratories. 6. REFERENCES [1] M. Yuen and H. R. Wu, "A survey of hybrid MC/DPCM/DCT video coding distortions," Signal Processing, vol. 70, pp , [2] S. Daly, "The visible differences predictor: an algorithm for the assessment of image fidelity," in Digital images and human vision, Andrew B. Watson, Ed. Cambridge, Massachusetts: MIT Press, 1993, pp [3] J. Lubin, "A human vision system model for objective picture quality measurements," IEE International Broadcasting Conference, Amsterdam, The Netherlands, pp , [4] A. B. Watson, "Perceptual-components architecture for digital video," Journal of the Optical Society of America A- Optics & Image Science, vol. 7, pp , [5] S. Wolf and M. H. Pinson, "Spatial-temporal distortion metric for in-service quality monitoring of any digital video system," SPIE Multimedia Systems and Applications II, Boston, MA, USA, pp , [6] S. Winkler, "A perceptual distortion metric for digital color video," SPIE Human Vision and Electronic Imaging, San Jose, CA, USA, pp , [7] S. Winkler, "Issues in vision modeling for perceptual video quality assessment," Signal Processing, vol. 78, pp , [8] A. A. Webster, C. T. Jones, M. H. Pinson, S. D. Voran, and S. Wolf, "An objective video quality assessment system based on human perception," SPIE Human Vision, Visual Processing, and Digital Display IV, San Jose, CA, USA, pp , [9] P. Bretillon, N. Montard, J. Baina, and G. Goudezeune, "Quality meter and digital television applications," SPIE Visual Communications and Image Processing, Perth, WA, Australia, pp , [10] Zhou Wang, A.C. Bovik, and B.L. Evan, "Blind measurement of blocking artifacts in images," IEEE International Conference on Image Processing, pp , [11] H.R. Wu and M.; Yuen, "A generalized block-edge impairment metric for video coding," IEEE Signal Processing Letters, vol. 4, pp , [12] J. Caviedes and J. Jung, "No-Reference Metric for a Video Quality Control Loop," Int. Conf. on Information Systems, Analysis and Synthesis, [13] M. S. Moore, Psychophysical Measurement and Prediction of Digital Video Quality, Ph.D. Thesis, Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA, [14] Mylene C.Q. Farias, No-Reference and Reduced Reference Video Quality Metrics: New Contributions, Ph.D Dissertation, Dept. of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA, [15] ITU Recommendation BT.500-8, "Methodology for the subjective assessment of the quality of television pictures," [16] M.S. Moore, J.M. Foley, and S.K. Mitra, "Defect visibility and content importance: Effects on perceived impairment," Image Communication, [17] A. B. Watson and J. Malo, "Video quality measures based on the standard spatial observer," Proceedings International Conference on Image Processing, vol. 3, pp. 41-4, [18] S. Winkler and P. Vandergheynst, "Computing isotropic local contrast from oriented pyramid decompositions," IEEE International Conference on Image Processing (ICIP), Kobe, Japan, pp , [19] M. Nadenau, Integration of human color vision models into high quality image compression, Doctor of Philosophy, Signal Processing Laboratory, Ecole Polytechnique Federale de Lausanne, Lausanne, [20] R. Rosenholtz and A. B. Watson, "Perceptual adaptive JPEG coding," IEEE International Conference on Image Processing, Lausanne, Switzerland, pp , [21] Barry G. Haskell, Atul Puri, and Arun N. Netravali, Digital video: an introduction to MPEG-2. New York, NY, USA: Chapman & Hall: International Thomson Pub., [22] J. Lubin, "Sarnoff JND Vision," Sarnoff Corp. [23] M.C.Q. Farias, J.M. Foley, and S.K. Mitra, "Detectability and annoyance of synthetic blurring and ringing in video sequences," IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing, Montreal, Canada, [24] M.C.Q. Farias, J.M. Foley, and S.K. Mitra, "Perceptual Analysis of Video Impairments that Combine Blocky, Blurry, Noisy, and Ringing Synthetic Artifacts," SPIE Human Vision and Electronic Imaging, San Jose, CA, USA, [25] M.C.Q. Farias, J.M. Foley, and S.K. Mitra, "Some Properties of Synthetic Blocky and Blurry Artifacts," SPIE Human Vision and Electronic Imaging, Santa Clara, CA, USA, pp , [26] M.C.Q. Farias, J.M. Foley, and S.K. Mitra, "Perceptual Contributions of Blocky, Blurry and Noisy Artifacts to Overall Annoyance," IEEE Intl. Conf. on Multimedia & Expo, Baltimore, MD, USA, pp , 2003.