ON THE IMPROVEMENT OF NO-REFERENCE MEAN OPINION SCORE ESTIMATION ACCURACY BY FOLLOWING A FRAME-LEVEL REGRESSION APPROACH

Transcription

1 ON THE IMPROVEMENT OF NO-REFERENCE MEAN OPINION SCORE ESTIMATION ACCURACY BY FOLLOWING A FRAME-LEVEL REGRESSION APPROACH Katerina Pandremmenou, M. Shahid, Lisimachos P. Kondi Department of Computer Science and Engineering, University of Ioannina, GR-, Ioannina, Greece Blekinge Institute of Technology, Karlskrona, Sweden. ABSTRACT In order to estimate subjective video quality, it usually happens to deal with a large number of features and a small sample set. Applying regression on complex datasets may lead to imprecise solutions due to possible irrelevant or noisy features as well as the effect of overfitting. In an effort to establish a robust regression model that generalizes well to unknown data and increase Mean Opinion Score (MOS) estimation accuracy, we propose a frame-level MOS estimation approach, where average MOS for each sequence are obtained. Since it is impractical to obtain the per-frame MOS ground truth through subjective experiments, we propose an objective metric able to do this task, providing an intuition about the relative quality of each individual frame as well. Experiments on both sequence- and frame-level domains reveal that regression on frame level offers better results in terms of all examined measures of performance either feature selection is performed or not. Index Terms Estimation accuracy, frame-level, MOS, objective metric, overfitting, sequence-level.. INTRODUCTION The continuous technological advancements have enabled the proliferation of streaming video services and consequently, the matter of Video Quality Assessment (VQA) has become very popular. Since the goal of each video communication product is to satisfy user s experience, subjective quality assessment is the most reliable way for evaluating its quality. Nonetheless, the conduction of subjective tests faces many challenges. For example, a large number of viewers is required, and the viewers are not always available or willing to rate a large variety of differently distorted video sequences. In addition, through subjective experiments we are unable to get instantaneous measurements of video quality due to many practical limitations. Thus, a single quality value for the whole video does not provide any information about the individual quality of the video frames, being unable to know which parts of the video have the greatest influence in forming the viewer s judgement. An alternative approach to subjective VQA is to automatically get video quality scores, through the use of objective metrics [, ], where an ideal objective metric is able to provide quality scores that highly correlate with human ratings. In the literature, many works construct objective metrics through the use of various machine learning techniques such as Partial Least Squares Regression (PLSR) [], Neural Networks (NN) [], Support Vector Machines (SVM) [] and Support Vector Regression (SVR) [6]. A number of qualityrelevant features that account for different types of distortion are given as input in each considered regression model and significantly influence the accuracy of MOS estimations [7]. Theoretically, the larger the number of features the better the estimation power of the regression model. However, a large number of features along with a possibly small number of observations (sample set) involves the risk of fitting the model to the noise of the training data, being unable to generalize well to unseen data (testing data). For this reason, a feature selection procedure often takes place before video quality estimation [8, 9]. In this work, we propose a novel approach for estimating perceptual video quality separately for each frame, and in the following we average these values in order to obtain a Mean Opinion Score (MOS) estimate for each sequence. The objective of this paper is two-fold: i) to improve the persequence MOS estimation accuracy through the development of a model which is robust and has a good generalization capability, ii) to provide a reliable indicator for the quality of each frame, offering an intuition about their individual contribution to the overall video quality score. Driven by the work presented in [], we develop a metric that provides the ground truth of the frame MOS, which plays the role of the target variable in the regression procedure. The works presented in [, ] elaborated on a similar concept of considering frame quality measurements and measurements over small units of video sequences that guide the overall video quality rating. However, the goal of both [, ] is different from the one of the current paper. In [] a NR objective metric that provides video quality scores per second so as to align with the subjective results of a Single Stimulus Continuous Quality Evaluation (SSCQE) method [] was introduced, and in [] the authors applied a fusion mecha-

2 nism in order to integrate the scores from some video integrals into a final one, increasing in this way the correlation with the MOS. The rest of the article is organized as follows: Section presents the proposed Full Reference (FR) metric that offers the ground truth of frame video quality. Our approach for estimating MOS in frame-level is analyzed in Section along with experimental results and discussion that highlight its efficiency. Last, in Section conclusions are drawn.. PROPOSED VIDEO QUALITY METRIC In order to estimate MOS in frame level, EMOS fr, for Common Intermediate Format (CIF) resolution sequences, it is necessary to obtain the actual MOS for each frame, MOS fr, so as to be able to verify the model s performance. Due to practical limitations involved in collecting human votes for thousands of video frames, we develop an FR objective metric, which produces automatically quality scores for each frame. It is worth mentioning that since we intend to employ these results as the target values that need to be predicted in the regression model we describe later, it is important for the average values of per-frame MOS, MOS fr, in sequence-level to highly correlate with the actual MOS for each sequence, MOS s. The research conducted in [] proved that an exponential function can map Peak Signal to Noise Ratio (PSNR) to MOS with high accuracy, considering the temporal and spatial activity levels of the video sequence in question. However, the metric developed in [] is not universal since it was only tested on CIF resolution sequences with a frame rate of frames per second (fps), scourged mainly by channel distortion. Therefore, in this paper we extend the research of [] and examine if the exponential function given by: MOS = exp ( PSNR a b ) () fits also well to the case of CIF resolution sequences. The parameters a and b represent the vertical shift and the steepness of the curve, and the PSNR s corresponds to the PSNR of each sequence. For our experiments, we employ the database of [] for CIF resolution sequences (7x76 pixels). Our test material comprises of the Ice, Harbour, Soccer, CrowdRun, DucksTakeOff and ParkJoy sequences of 8, 98, 98,, and frames, respectively; the former three sequences are at fps and the latter three at fps. The Group Of Pictures (GOP) structure is IBBP with a GOP size of 6 frames. The sources are encoded using the JM. version of H.6/AVC reference software using the High profile, where a full row of MacroBlocks (MBs) is coded as a separate slice and each of the sequences is corrupted with a Packet Loss Rate (PLR) of.%,.%, %, %, % and %, for two channel realizations. Thus, our dataset consists of 7 distorted versions of the 6 original video sequences (6 original sequences x 6 PLRs x channel realizations). In addition, in [] subjective results from the Ecole Polytechnique Fédérale de Lausanne (EPFL) and Politecnico di Milano (PoliMi) are also included. For each of the 7 video sequences, we calculated the PSNR values and using the MOS values of [] from both EPFL and PoliMi, we plotted these values, fit the Eq. () and examined the coefficient of determination R, which is an indicator of how well the observed outcomes (our data points) are replicated by the model (Eq. ()). Figure presents indicative fitting results for the DucksTakeOff and ParkJoy video sequences, while similar fitting performance was also observed for the rest CIF sequences. However, for the sake of saving space we omit the presentation of their fitting curves from this paper. It is to be noted that the value of R was for all sequences above.9 and the Root Mean Squared Error (RMSE) less than.. MOS MOS DucksTakeOff PSNR ParkJoy 6 8 PSNR Fig. : MOS-PSNR fitting curves. Based on these results, it is clear that the exponential shape of Eq. () accurately describes the MOS-PSNR relationship for the CIF sequences as well. Having reached to this useful conclusion, we applied Ordinary Least-Squares (OLS) optimization in order to solve parameters α and β of Eq. (), using the pair values of (MOS s, P SNR s ) for each CIF sequence. Afterwards, we calculated the PSNR values for each frame of all considered CIF sequences, P SNR fr, and next we computed the MOS fr. In order to validate the quality of the obtained values, we go averaged the per-frame MOS values to obtain a representative value for each separate sequence, MOS fr, and checked for correlation with the corresponding measured values, MOS s. The estimated a and b parameters as well as the Pearson Correlation Coefficient (PCC) [] for each examined sequence are depicted in Table. PCC = indicates a perfect linear relationship between measured and estimated data (MOS in our case). As we can see from this table, if we ap-

3 ply to Eq. () the parameters resulting either from EPFL or PoliMi dataset, the subjective results can be estimated nearly perfectly. Table : Estimated a and b values. %. The horizontal line represents the actual MOS of the sequence (.777), while as we observe, there is a high discrepancy among the MOS values of each separate frame of the sequence, although their average value (.77) is virtually identical to the measured MOS. EPFL MOS CrowdRun DucksTakeOff Harbour Ice ParkJoy Soccer a b PCC PoliMi MOS a b PCC Moreover, Fig. is the scatter plot of MOS fr and MOS s, when both EPFL (on the left) and PoliMi (on the right) results are employed. MOS fr 6 Ice with PLR=%. data points linear fit EPFL Results. data points fit line PoliMi Results Frame M OSfr.. o line M OSfr.. o line Fig. : Per-frame MOS... MOS s Fig. : Correlation results... MOS s More light about the efficiency of the proposed FR metric is shed by the results presented on Table. A comparison of the proposed metric using results from both EPFL and PoliMi with the state-of-the-art Perceptual Evaluation of Video Quality (PEVQ) [6] and Video Quality Metric (VQM) [7] reveals that although all of these metrics correlate well with MOS, the metric proposed in this article is even more efficient in terms of PCC [], Spearman Rank Order Correlation Coefficient (SROCC) [8] and RMSE []. It presents a nearly perfect linear correlation with the measured MOS s values. Moreover, we signify that the performance of our proposed metric using either the EPFL or the PoliMi MOS results is equivalent. Hence, since accurate per-frame MOS were obtained, we recommend their use as ground truth in per-frame MOS estimation problems. Table : FR Objective Metrics Comparison Proposed (EPFL) Proposed (Polimi) PEVQ VQM PCC SROCC RMSE Continuing, an additional feature of our proposed metric is that it offers an intuition about the relative quality of each frame, and how much it contributes to the overall score of the sequence. Figure presents the MOS values for each frame of the Ice video sequence, when PLR is equal to Nonetheless, from Fig. it seems that some MOS fr are slightly above the upper bound of the -point rating scale []. It is expected as these results come from simulations and cannot mimic the weakness of subjective data, which are usually compressed at the ends of the rating scale. Therefore, it is reasonable that when we average these values in sequence-level, the obtained per-sequence MOS fall also outside the rating scale. In order to circumvent with this issue, we applied a non-linear mapping on the averaged per-frame MOS before computing any of the performance metrics. Specifically, we used the cubic polynomial function given by []: MOS fr = a MOS fr +a MOS fr +a MOS s +a () which is found to perform well empirically. The weights a, a, a and the constant a are calculated by fitting the function to the data (MOS fr, MOS s ) with the goal of maximizing their correlation.. MOS ESTIMATION IN FRAME-LEVEL After the collection of the per-frame MOS values we construct a regression model that can automatically generate perframe MOS estimates, which will be next averaged to provide an overall MOS value for each sequence. Our sequencelevel dataset consists of 7 video sequences and perceptual quality-related features, proposed in our previous work [7], and our frame-level dataset includes 98 frame observations and features. To be noted that for the rest of our experiments we adopted the PoliMi MOS. Before applying regression, we split the dataset into separate training and test sets such that each impaired sequence set

4 of each original video to be used for testing, and next we apply ridge regression [9, ] on both the sequence- and framelevel domains. Ridge is an extension of the OLS regression method and is able to improve the OLS estimates by allowing a little bias in order to reduce the variance of the estimated values. It solves the problem: min w ( ) n (y i w φ(x i )) + λ w i= where y includes the measured quality values for all n observations (n = 7 for the sequence domain and n = 98 for the frame-domain) and x i includes the values for all features. The parameter λ is a regularization parameter, which shrinks regression coefficient values towards zero. For λ = no regression is performed and the solution of the OLS is obtained, while for larger λ values, the closer to zero the regression coefficient estimates. In our experiments we set λ = in the regression models of both domains. Therefore, ridge tradeoffs the Sum of Squared Errors (SSE) (first term of Eq. ()) and the penalty (second term of Eq. ()). Table presents the performance statistics when regression is applied on both the sequence- and frame-level domains. At this point we should mention that since the goal of this work is not to deepen in the regression procedure, we omit the presentation of the regression coefficient estimates from this work. For the sequence-level case the actual MOS values are compared with the MOS values obtained from ridge. For the frame-level case the MOS estimates for each frame obtained from ridge are averaged to obtain a MOS value for each video sequence and next we compare these results with the actual sequence-level MOS values. Furthermore, in the same table we present the RMSE not only in the testing set but also in the training set. This is an interesting approach since the sequence-level dataset is fairly complex, that is it includes a large number of features as compared to the number of observations, posing the risk of overfitting. Table : Regression results on raw datasets. Sequence-level Frame-level PCC SROCC.8.96 RMSE (training)..9 RMSE (testing).8. From the results of Table it is clear that regression on the frame-level dataset guarantees better performance statistics as compared to the regression on the sequence-level dataset. After a thorough investigation of the training and testing errors in each dataset sub-partition of both the sequence- and framelevel cases, we confirmed that for the frame level case a similar amount of error in both the training and testing sets is observed. On the other hand, in the sequence-level case there is a great discrepancy between training and testing errors. This difference is more evident before applying Eq. () to account () for the quality rating compression at the extremes of the test range. Nonetheless, even after applying Eq. (), the overall testing error is three times greater than the corresponding error in the training set. This implies that our model may fit to the irregularities of the training set (a noteworthy low RMSE is observed in this case) and cannot effectively estimate the response variable when it deals with unknown data. In an effort to enhance the strength of the sequence-level regression model in making precise estimations when it is provided with new data, we applied stepwise regression [] to the sequence-level dataset so as to perform feature selection by iteratively adding features or their interactions (products of features in pairs) or trimming features from a linear model, based on their statistical significance in the regression process. The method begins with an initial dataset and then compares the explanatory power of incrementally larger and smaller datasets. In addition, stepwise regression was also performed on the frame-level dataset. After this operation, we ended up with a sequence-level set of dimension 7x6 and a frame-level set of dimension 98x9. For the sake of saving space, we do not mention in this paper the features as well as their interactions that have been selected by stepwise regression for each case. Based on our processed sequence-level and frame-level datasets, we applied again ridge regression to obtain new MOS estimates. The results presented on Table reinforce our claim that a more stable model is achieved when regression is applied on frame-level, which offers more precise MOS estimations compared to the model built in sequencelevel. For the frame-level model similar performance is achieved either the dataset is raw or free of useless features, or slightly improved in the latter case. Regarding the sequence-level case, the correlation measures are considerably improved and the difference between training and testing error is minor in this case. Table : Regression on processed datasets. Sequence-level Frame-level PCC SROCC RMSE (training).9.8 RMSE (testing)..8 Additional information about the improvement achieved on the sequence-level dataset after applying feature selection is provided on Table. From R value on the raw dataset we Table : Goodness-of-fit statistics. R Adjusted-R SSE Raw dataset Processed dataset can say that 96% of the total variance is explained by the regression model. While this value seems promising (the more variance accounted for by the regression model, the closer the

5 data points fall to the fitted regression line), this value may be misleadingly high because many features are included in the model. For this purpose, we also computed the Adjusted- R measure, which penalizes the R statistic when extra variables are added. Also, we assessed SSE to determine whether the coefficient estimates and predictions are biased. From the results of Table we confirm that all statistics are improved when stepwise regression is applied. Therefore, it is reasonable that we get better prediction results in this case as depicted in Table.. CONCLUSIONS In this work, we propose an alternative approach for estimating MOS in frame level, and then average the per-frame MOS estimations in order to obtain the overall MOS for each sequence, instead of performing regression directly in sequence-level. In order to obtain the per-frame ground truth of quality, we develop a FR metric, which is able to map per-frame PSNR to MOS. The accuracy of the results produced by this metric with regard to the actual MOS values highly recommends their use for model s verification. Ridge regression conducted on both the sequence- and frame-level domains verified the ability of higher-dimension datasets to estimate MOS more precisely overcoming the issue of overfitting. In addition, in both datasets we applied stepwise regression so as to keep the features and their interactions that are statistically significant. In this case, the performance of the sequence-level dataset was improved. However, the advantage of the frame-level dataset is clear since it not only gathers better performance statistics but also is more stable and robust, tackling the problem of MOS estimation with efficiency either feature selection is performed or not. Thus, the benefit of our proposed method of exploiting a large number of training data as a means of reducing the variance captured in the model is more important compared to feature selection or addition of extra (feature interactions) than can be favourable towards the same purpose.

6 . REFERENCES [] S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam, Objective video quality assessment methods: A classification, review, and performance comparison, IEEE Transactions on Broadcasting, vol. 7, no., pp. 6 8, Jun.. [] Zhou Wang, Hamid R. Sheikh, and Alan C. Bovik, Objective video quality assessment, in IN THE HAND- BOOK OF VIDEO DATABASES: DESIGN AND AP- PLICATIONS., pp. 78, CRC Press. [] C. Keimel, J. Habigt, M. Klimpke, and K. Diepold, Design of no-reference video quality metrics with multiway partial least squares regression, in Third International Workshop on Quality of Multimedia Experience (QoMEX), Sept., pp. 9. [] Jihwan Choe, Jonghwa Lee, and Chulhee Lee, Noreference video quality measurement using neural networks, in 6th International Conference on Digital Signal Processing, Jul. 9, pp.. [] Beibei Wang, Dekun Zou, and Ran Ding, Support vector regression based video quality prediction, in IEEE International Symposium on Multimedia (ISM), Dec., pp [6] M. Narwaria and Weisi Lin, SVD-based quality metric for image and video using machine learning, IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, vol., no., pp. 7 6, Apr.. [7] Katerina Pandremmenou, Muhammad Shahid, Lisimachos P. Kondi, and Benny Lövström, A no-reference bitstream-based perceptual model for video quality estimation of videos affected by coding artifacts and packet losses, in Proc. of IS&T/SPIE Electronic Imaging. Feb., SPIE, to appear. [8] Caihong Wang, Ting-Lan Lin, and P. C. Cosman, Network-based model for video packet importance considering both compression artifacts and packet losses, in IEEE Global Telecommunications Conference (GLOBECOM), Dec., pp.. [9] D. Culibrk, M. Mirkovic, V. Zlokolica, M. Pokric, V. Crnojevic, and D. Kukolj, Salient motion features for video quality assessment, IEEE Transactions on Image Processing, vol., no., pp , Apr.. [] Y. Kawayokeita and Y. Horita, NR objective continuous video quality assessment model based on frame quality measure, in th IEEE International Conference on Image Processing (ICIP), Oct. 8, pp [] Tsung-Jung Liu, Weisi Lin, and C. C. J. Kuo, A fusion approach to video quality assessment based on temporal decomposition, in Signal Information Processing Association Annual Summit and Conference (APSIPA ASC), Dec., pp.. [] Alpert Th. and Evain J-. P., Subjective quality evaluation the SSCQE and DSCQE methodologies, 997, EBU Technical Review. [] Francesca De Simone, Matteo Naccari, Marco Tagliasacchi, Frdric Dufaux, Stefano Tubaro, and Touradj Ebrahimi, Subjective quality assessment of H.6/AVC video streaming with packet losses, Eurasip Journal on Image and Video Processing, vol. Article ID 9,. [] Video Quality Experts Group, VQEG report on validation of video quality models for high definition video content,. [6] ITU-T Rec. J.7, Objective perceptual multimedia video quality measurement in the presence of a full reference, 8. [7],. [8] D. E. Hinkle, W. Wiersma, and S. G. Jurs, Applied statistics for the behavioral sciences., Boston, Mass: Houghton Mifflin,. [9] Ridge regression in practice, The American Statistician, vol. 9, no., pp., 97. [] A. E. Hoerl and R. W. Kennard, Ridge regression: Biased estimation for nonorthogonal problems, Technometrics, vol., pp. 67, 97. [] Norman Richard Draper and Harry Smith, Applied regression analysis, A Wiley-Interscience publication. Wiley, New York, NY [u.a.],. ed. edition, 998. [] J. Korhonen and J. You, Improving objective video quality assessment with content analysis, in Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM ), Scottsdale, AZ, USA, Jan..