Instantaneous Video Quality Assessment for lightweight devices

Transcription

1 Instantaneous Video Quality Assessment for lightweight devices Antonio Liotta Eindhoven University of Technology, The Netherlands Decebal Constantin Mocanu Eindhoven University of Technology, The Netherlands Luciana Cagnetta Georgios Exarchakos University of Modena, Eindhoven University of Italy Technology,The Netherlands Vlado Menkovski Eindhoven University of Technology, The Netherlands ABSTRACT Monitoring and controlling the user s Quality of Experience (QoE) in modern video services is a challenging proposition, mainly due to the limitations of current video quality assessment algorithms. While subjective QoE methods would better reflect the nature of human perception, these are not suitable in real-time automation cases. On the other hand, the existing objective algorithms are either too complex or too inaccurate, particularly in the context of lightweight devices such as camera sensors or smart phones. This paper introduces a novel objective QoE algorithm, Instantaneous Video Quality Assessment (IVQA), that is comparably as accurate as the most heavyweight algorithm available in the literature but can also be run in real-time. This approach is tested against a selection of ten objective metrics and benchmarked with a subjective user dataset. Categories and Subject Descriptors H.5.. [Multimedia Information Systems]: Video; D.4.8. [Measurements]: Video quality metrics General Terms Algorithms, Measurement, Performance, Experimentation Keywords Quality of Experience, video quality assessment, objective metrics, subjective metrics. INTRODUCTION Digital videos are becoming more and more common in people s everyday life thanks to the proliferation of video Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MoMM 23, Vienna, Austria Copyright 23 ACM /3/2...$5.. architectures and applications such as digital television, digital cinema, internet videos, IPTV, video teleconferencing, mobile broadcasting, and the more popular video streaming and video sharing services. Network and content providers have as primary goal providing high quality video services accordingly to satisfaction of users. Monitoring and controlling the user s Quality of Experience (QoE) [2], [5], [9] in these modern video services is a challenging proposition, mainly due to the limitations of current video quality assessment algorithms. Nowadays, the best methods to assess the QoE for people are the subjectives ones [2], due to the fact that they are the most suitable to accurately reflect the nature of human perception. However, they are not suitable to be used in real world scenarios, where there is need for a fast automated decision process. Towards this direction a good substitution for the subjective methods are the objective ones [2], which rely just on the video sequence characteristics and not on the human factor. On the other hand, due to the widespread of lightweight devices (e.g. smart phones, cameras sensors, in general low-resource devices) and to the need of real-time automation applications, even the traditional objective metrics for video quality assessment have started to fail, particularly, for these low resources machines. This is the case because existing objective algorithms are either too complex or either too inaccurate. Aiming to solve the above problems, this paper proposes a novel algorithm, namely Instantaneous Video Quality Assessment (IVQA), that simultaneously offers: () assessment of video quality with a high accuracy, comparable with the most heavyweight algorithms available in the literature; and (2) a very fast computational time O(k) where k is a constant, more exactly it can be run in a fix amount of time, independently of the video characteristics (e.g. resolution, length, and so on). To achieve these goals the IVQA approach relies only on parameters which are calculated during the video encoding process. The performance of IVQA was tested by comparing it with a selection of ten objectives metrics on the Live video databases [2], while it was benchmarked with a subjective user dataset. The remainder of this paper is organized as follows. Section 2, presents background knowledge on QoE assessment

2 for the benefit of the non-specialist reader. Section 3, details the proposed method including all the mathematical details. Section 4, describes the experiments performed and reflects upon the attained results. Finally, Section 5 concludes and presents directions of future work. 2. BACKGROUND ON QOE ASSESSMENT In this section, different QoE algorithms for Video Quality Assessment are presented. Firstly, is discussed one subjective metric, namely Maximum Likelihood Difference Scaling (MLDS), used in Section 4 to benchmark the results of all the objective metrics. Secondly, a brief presentation of different objective metrics is given. 2. Subjective QoE benchmark The best methods to reflect the human perception are the subjectives QoE methods, due to the fact that they involve humans to evaluate human-perception factors. However, it s not always possible to employ subjective tests, particularly when the application requires real-time feedback for the purpose of service monitoring and control. More commonly, subjective studies are successfully used to benchmark the results of the objective QoE methods. The Maximum Likelihood Difference Scaling procedure [6] is a two-alternative-forced-choice (2AFC) method from the domain of psychophysics that asks people to discriminate between the intensity of two stimuli. This type of tests has less bias and variability because the mechanisms of direct comparison is more natural than rating. Moreover MLDS has the advantage of requiring less testing than traditional subjective approaches [7], while being reasonably accurate. The MLDS method in the literature has been adopted by Charrier et al. [2] in the estimation of quality differences for images compressed at different levels. The authors of [2] presented an observer with two pairs of images picked up randomly from the set of compressed images and asked him to judge which exhibited the larger perceptual difference. Moreover, they proved the user-friendliness of the MLDS procedures and the ease of collecting data with it. In [8], instead, MLDS is used to estimate the perceptual differences in the quality of a video based on the intensity of impairments introduced by ten different levels of H.264 video compression. 2.2 Objective Metrics for QoE In contrast with subjective metrics, the objective ones offer more advantages from an economical and practical perspective. But it has to be kept in mind that subjective techniques are still needed to benchmark the performance of objective methods, which are less accurate. Further, ten objectives metrics for video quality assessment are briefly presented, selecting the most state-of-the-art solutions. Mean Signal Error(). [7] can be calculated as a function of the original and distorted luminance signal, as shown in the following formula: N M i= j= [f(i, j) F (i, j)]2 = () M N where f(i, j) is the original signal and F (i, j) is the test signal at pixel (i, j), M and N represent the picture dimensions. When it is applied to video sequences, it is necessary to average the values calculated for each single frame over all frames of the sequence. Peak Signal to Noise Ratio(PSNR). PSNR [2] is obtained by putting the in relation to the maximum possible value of the luminance as shown below: ( ) MAXI P SNR = 2 log (2) where MAX I is the maximum possible pixel value of the image (e.g for a typical 8-bit value this means MAX I = 2 8 = 255). Structural Similarity(SSIM). SSIM index [9] [8] was created as a method for quality assessment of images. Afterwards, it was extended to video by applying it frame by frame on the luminance component of the video and taking as overall SSIM index for the video the average of the frame level quality scores. Multiscale SSIM (MS-SSIM). The MS-SSIM index [2] is an extension over SSIM index. It takes the reference and distorted image signals as inputs, iteratively applies a low-pass filter and down-samples the filtered image by a factor of 2. The original image is indexed as Scale, and the highest scale is Scale M. The MS-SSIM index can be extended to video by applying it frame by frame on the luminance component of the video and the overall MS-SSIM index for the video is computed as the average of the frame level quality scores. Visual Signal-to-Noise Ratio(VSNR). VSNR [] is a quality assessment algorithm proposed for still images. It is basically a full reference still image quality metric based on near-threshold and supra-threshold properties of human vision. The VSNR metric has also shown a good performance in assessing video quality when applied on a frame by frame basis and then averaged. Visual Information Fidelity(VIF). VIF [4] is a quality assessment algorithm for natural images that combines visual statistics and human visual system modeling. It is based on the extraction of the loss of image information after the distortion process (e.g compression) and explores the relationship between image information and visual quality. This metric was extended to video using temporal derivatives in [3]. Pixel Visual Information Fidelity(VIFP). VIFP [4] is an adaption of VIF in the pixel domain, done by the same authors. This method comes with an advantage of computational complexity. Universal Quality Index(UQI). UQI [6] is a quality measure for various image processing applications. Instead of using traditional error summation methods, UQI is designed by modeling any image distortion as a combination of three factors: loss of correlation, luminance distortion, and contrast distortion. It can be applied to video frame by frame on the luminance component and taking their averaged UQI scores.

3 Information Fidelity Criterion(IFC). IFC [5] approaches the problem of quality analysis by proposing an information fidelity criterion that is based on sophisticated models that capture the statistics of natural signals (i.e. pictures and videos) of the visual environment. The principle lying under this method, completely different from other methods, is that natural images can be considered as signals with particular statistical properties. Motion-based Video Integrity Evaluation(MOVIE). MOVIE [] is an objective, full reference video quality index recently developed at the Laboratory for Image & Video Engineering in Texas. It combines both spatial and temporal aspects of distortion assessment. In fact, it extracts motion information from the reference video and evaluates the quality of the distorted video along the motion trajectories of the reference video. In the literature, it achieved the best results for an objective video quality metrics, but it comes with the downside of a high computational complexity. 3. INSTANTANEOUS VIDEO QUALITY AS- SESSMENT This section introduces an Instantaneous Video Quality Assessment (IVQA) metric. Firstly, an intuition describing the metric is discussed. Secondly, the parameters on which the novel proposed metrics relies are detailed. Thirdly, we give mathematical details. 3. Intuition While subjective QoE methods would better reflect the nature of human perception of video quality, these are not suitable in real-time automation cases. On the other hand, the existing objective algorithms are either too complex or too inaccurate, particularly in the context of lightweight devices such as camera sensors or smart phones. It is well known that the content of a video plays an important role in the perceptual quality of a video. For these reasons, while trying not to reinvent the wheel, we investigated what parameters, which are already incorporated into video file, can be used to describe as much as possible the content of a video file, with the ultimate goal to use these parameters to measure the video quality. Thus, we came across the Scene Complexity and Level of Motion, which are described further. 3.2 Scene Complexity and Level of Motion This subsection details the parameters which can be used to characterize the content of a video. More exactly, these parameters are Scene Complexity (C) and Level of Motion (M), as they were introduced in a recent study by Hu and Wildfeuer [4]. Formally, C [, ] and M [, ] and they can be calculated as follows: Bits I C = QP I Bits P M = QP P where Bits I are bits of coded Intra (I ) frames, Bits P are bits of coded Inter (P ) frames, QP I represents the average quantization parameter of I-frames and QP P represents the average quantization parameter of P-frames. More than (3) (4) that, Bits I, Bits P, QP I and QP P are variables computed during the encoding process. Thus, the computational time needed to calculate C and M for a video on the client device is O(k), where k is a constant, and it is not dependent upon the size of the video (i.e. resolution, bit-rate, length). The interested reader should refer to [4] for a better understanding of these concepts. 3.3 Mathematical Details Relying on the Scene Complexity and Level of Motion parameters, in Equation 5 we propose the novel IVQA metric as an objective algorithm for video quality estimation. ) ( e b Br C c Md Q IV QA (Br, C, M) = a where C and M are the formerly presented complexity and motion parameters and Br is the bit-rate. To ensure the convergence of the maximum quality index to the a parameter is set to, and b, c, d parameters are optimized for a generic video type, as it is shown further. The starting point for the previous formula, was a prediction model depending only on the bit rate: Q(Br) = ( α e α Br) β (6) where and were calculated through a non linear regression between the MLDS data obtained from [] and their α β respective bit-rates, on the Live video databases [2] which is extensively used in videos studies. Afterwards, the parameters α were modeled using as responses the complexity and β motion factors, i.e. they were non linearly regressed using the following function: (5) α β = b (7) C c M d The parameters b, c and d were optimized to be used on every content type. The resulting values were: b =, 82357, c =, and d =, The main advantage of the IVQA metric is the running time. Due to the fact that it is relying just on variables computed during the encoding process, which already incorporate the characteristics of the movie, the time needed to calculate IV AQ Q is O(k), where k is a constant, and it is not dependent on the video resolution, length, size, etc. For this reason, it becomes a very good candidate to calculate the video quality on the lightweight devices. 4. EXPERIMENTS AND RESULTS To test the IVQA algorithm an extensive set of experiments was done using the Live video databases [2] in a step wise fashion. The database contains videos, described in Table, each of them encoded at 64, 28, 256, 384, 52, 64, 768, 24, 536, and 248 kbits. The description of each type of video can be obtained from the Live database. On each possible combination (movie,bit-rate) we applied all the objectives metrics presented in Subsection 2.2 and the IVQA metrics. For the implementation of all objectives metrics (i.e., PSNR, SSIM, MS-SSIM, VSNR, VIF, VIFP, UQI, IFC, and MOVIE) we used free software, downloadable from the Internet. Due to the fact that the performance of an objective video quality metric depends on its correlation with subjective results, which are the benchmark for any visual quality approach, all of the metrics used in this

4 paper were benchmarked with a subjective user dataset, using Maximum Likelihood Difference Scaling (MLDS). This dataset was created in one of our previous work at Eindhoven University of Technology []. 4. Analysis of Scene Complexity and Level of Motion The first step, was to analyze the Scene Complexity (C) and the Level of Motion (M) for each video and for each possible bit-rate. The averaged values over all ten available bit-rates are depicted in Figure. In the second step of the experiments, from the ten videos studied we picked up the three videos, which have extremes C and M values, for a detailed examination. More specifically these are rb, rh, and tr. Distance All metrics against MLDS on "rb" video PSNR SSIM MSSIM VSNR VIF VIFP UQI IFC MOVIE IVQA Motion Average Motion vs Complexity bs mc pa pr rb rh sf sh st tr Complexity Figure : Complexity vs Motion for each video, averaged over all possible bit-rates. 4.2 IVQA performance on rb, rh, and tr videos In the second step of the experiments, we analyzed the IVQA performance on the three most extreme videos available, rb, rh, and tr, in comparison with ten objectives metrics, using the MLDS benchmark. To be able to compare all the objective metrics their outputs were first scaled into the subjective domain interval [, ] Distance from MLDS Using the normalized values for each objectives metrics, we estimated how far away they were from the MLDS outputs. We achieved these by calculating the Euclidean distance between the quality estimated by each of the eleven metrics (including IVQA proposed herein) and the quality assessed by the subjective MLDS method. We considered a number of cases leading to Fig. 2, 3, and 4. Notably, IVQA performs well, comparably with the best objective metric, available in the literature(movie) Prediction accuracy, monotonicity and consistency Bitrate [kbps] Figure 2: Video Quality Metrics analyses by using the Euclidean distance between the predicted values of the objective metrics and their MLDS correspondent on rb video. After gaining the first insights into the results of IVQA on the three borderline videos, we evaluated its performance with respect to the following three main attributes : prediction accuracy, prediction monotonicity, prediction consistency [3]. Prediction accuracy is the ability of a metric to predict subjective ratings with minimum average error and can be evaluated by means of the Pearson linear correlation coefficient (LCC) and the root mean square error (R). In particular, Pearson LCC is calculated as the correlation coefficient between the subjective and the mapped objective scores, while the R is calculated as follows: R = N N i= (Q objective i Q MLDS i ) 2 where N represents the total number of data points. Prediction monotonicity measures if increases or decreases in one variable are associated respectively with increases or decreases in the other variable, independently of the magnitude of the increase or decrease. The degree of monotonicity is calculated by the Spearman rank-order correlation coefficient (SROCC). However, this calculation was not necessary, because the monotonicity was ensured by the equivalence of the ranks of the objective metrics values and of MLDS values for each video and each metric. Prediction consistency is the degree to which the model maintains prediction accuracy over the range of video test sequences. It can be evaluated by measuring the outliers ratio (OR), which is simply defined as OR = N outliers, where N N outliers is the number of outliers and N is the total number of data points. An outlier is a point i from the dataset for which Q objective i Q MLDS i > 2 σi MOS, where σi MOS is the standard deviation for the all the votes, given by the users, in the case of MLDS, for that specific point. For this specific experiment the number N of data points is, corresponding to the bite-rates options. From Table 2

5 Acronym Name Description bs Blue Sky Circular camera motion showing a blue sky and some trees. rb River Bed Still camera, shows a river bed containing some pebbles in the water. pa Pedestrian Area Still camera, shows some people walking about in a street intersection. tr Tractor Camera pan shows a tractor moving across some fields. sf Sunflower Still camera, shows a bee moving over a sun-flower in close-up. rh Rush hour Still camera, shows rush hour traffic on a street. st Station Still camera, shows railway track, a train and some people walking across the track. sh Shields Camera pans at first, then becomes still and zooms in; shows a person walking across. a display pointing at it. mc Mobile and Calendar Camera pan, tor train moving horizon tally with a calendar moving vertically in the background. pr Park run Camera pan, a person running across a park. Table : Description of the Live videos under scouting. Predictions on rb video Predictions on rh video Predictions on tr video Metric Accuracy Consistency Accuracy Consistency Accuracy Consistency Pearson LLC R OR Pearson LLC R OR Pearson LLC R OR PSNR SSIM MSSIM VSNR VIF VIFP UQI IFC MOVIE IVQA Table 2: Objective metrics performance with respect to MLDS results on 3 borderline movies, averaged over all bit-rates possibilities. it can be inferred that IVQA performs very well, in terms of prediction accuracy and consistency, comparably with the best results achieved by the other objective algorithms, on each of the extreme videos under scouting. 4.3 IVQA performance on all videos In the third step of the experiments, we analyzed the IVQA performance on all the videos available in the dataset following the same main attributes as in Subsection : prediction accuracy, prediction monotonicity and prediction consistency. For each objective metrics, firstly, the Pearson LLC, R, and OR values were calculated for each videos over all possible bit-rates options. Secondly, the previous obtained values were averaged over all videos. In Table 3 it can be seen that IVQA offers good performance also in this scenario, very close the the performance of the best objectives metrics for the assessment of video quality. All of these are achieved by IVQA, while its running time is a constant O(k), being much faster that any of its competitors for which their running time is at least O(n), where n is linearly dependent by the video characteristics (e.g. resolution, bit-rate, length and so on). 5. CONCLUSION This paper proposes a novel objective metric for video quality assessment, namely Instantaneous Video Quality Assessment (IVQA). The approach was benchmarked based on Metric Predictions Accuracy Consistency Pearson LLC R OR PSNR SSIM MSSIM VSNR VIF VIFP UQI IFC MOVIE.94 7 IVQA Table 3: Objective metrics performance with respect to MLDS results, averaged over all movies and all bit-rates possibilities. the Live video dataset [2], ten existing objective metrics (i.e. PSNR,, SSIM, MSSIM, VSNR, VIF, VIFP, UQI, IFC and MOVIE) and a subjective metrics (MLDS). The results show that IVQA performance is comparable with the best objective metric from the literature, while its running time is a constant, making it suitable for implementation in real-time applications in lightweight devices (i.e. mobile devices, video cameras and so on).

6 Distance All metrics against MLDS on "rh" video PSNR SSIM MSSIM VSNR VIF VIFP UQI IFC MOVIE IVQA Distance All metrics against MLDS on "tr" video PSNR SSIM MSSIM VSNR VIF VIFP UQI IFC MOVIE IVQA Bitrate [kbps] Bitrate [kbps] Figure 3: Video Quality Metrics analyses by using the Euclidean distance between the predicted values of the objective metrics and their MLDS correspondent on rh video. Figure 4: Video Quality Metrics analyses by using the Euclidean distance between the predicted values of the objective metrics and their MLDS correspondent on tr video. As further work, it would be interesting to study how IVQA could perform in a real world application, when the video signal is streamed from a server over a wireless network to different clients lightweight devices. 6. REFERENCES [] D. M. Chandler and S. S. Hemami. Vsnr: A wavelet-based visual signal-to-noise ratio for natural images. Trans. Img. Proc., 6(9): , Sept. 27. [2] C. Charrier, L. T. Maloney, H. Cherifi, and K. Knoblauch. Maximum likelihood difference scaling of image quality in compression-degraded images. J. Opt. Soc. Am. A, 24(): , Nov 27. [3] S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam. Objective video quality assessment methods: A classification, review, and performance comparison. TBC, 57(2):65 82, 2. [4] J. Hu and H. Wildfeuer. Use of content complexity factors in video over ip quality monitoring. In Quality of Multimedia Experience, 29. QoMEx 29. International Workshop on, pages 26 22, 29. [5] H. l. Kim and S. G. Choi. A study on a qos/qoe correlation model for qoe evaluation on iptv service. In Proceedings of the 2th international conference on Advanced communication technology, ICACT, pages , Piscataway, NJ, USA, 2. IEEE Press. [6] L. T. Maloney and J. N. Yang. Maximum likelihood difference scaling. Journal of Vision, 3(8), 23. [7] V. Menkovski, G. Exarchakos, and A. Liotta. Tackling the sheer scale of subjective qoe. In MobiMedia, volume 79 of Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pages 5. Springer, 2. [8] V. Menkovski, G. Exarchakos, and A. Liotta. The value of relative quality in video delivery. J. Mob. Multimed., 7(3):5 62, Sept. 2. [9] V. Menkovski, G. Exarchakos, A. Liotta, and A. C. Sanchez. Quality of experience models for multimedia streaming. IJMCMC, 2(4): 2, 2. [] V. Menkovski and A. Liotta. Adaptive psychometric scaling for video quality assessment. Image Commun., 27(8): , Sept. 22. [] K. Seshadrinathan and A. C. Bovik. In B. E. Rogowitz and T. N. Pappas, editors, Human Vision and Electronic Imaging, SPIE Proceedings, page 724. SPIE. [2] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack. Study of subjective and objective quality assessment of video. Trans. Img. Proc., 9(6):427 44, June 2. [3] H. R. Sheikh and A. C. Bovik. A visual information fidelity approach to video quality assessment. In in The First International Workshop on Video Processing and Quality Metrics for Consumer Electronics, pages 23 25, 25. [4] H. R. Sheikh and A. C. Bovik. Image information and visual quality. Image Processing, IEEE Transactions on, 5(2):43 444, Feb. 26. [5] H. R. Sheikh, A. C. Bovik, and G. de Veciana. An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Transactions on Image Processing, 4(2):27 228, 25. [6] Z. Wang and A. C. Bovik. A universal image quality index. IEEE Signal Processing Letters, 9(3):8 84, Mar. 22. [7] Z. Wang and A. C. Bovik. Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures. Signal Processing Magazine, IEEE, 26():98 7, Jan. 29. [8] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P.

7 Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 3(4):6 62, Apr. 24. [9] Z. Wang, L. Lu, and A. C. Bovik. Video quality assessment based on structural distortion measurement. Signal Processing: Image Communication, 9(2):2 32, Feb. 24. [2] Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In Proc 37th Asilomar Conf on Signals, Systems and Computers, volume 2, pages , Pacific Grove, CA, Nov IEEE Computer Society. [2] S. Winkler and P. Mohandas. The evolution of video quality measurement: From PSNR to hybrid metrics. IEEE Transactions on Broadcasting, 54(3):66 668, Sept. 28.