QoE-estimation models for video streaming services

Transcription

1 QoE-estimation models for video streaming services Kazuhisa Yamagishi NTT Network Technology Laboratories, NTT Corporation, Tokyo, Japan Tel: Abstract As encoders and decoders (codecs), networks, and displays have become more technologically advanced, network and video-streaming-service providers have been able to provide video-streaming services over a network (e.g., fiber-to-the home and long-term evolution); therefore, the use of these services has been increasing drastically in the past decade. To maintain the high quality of experience (QoE) of these services, network and service providers need to invest in equipment (e.g., network devices, codecs, and servers). To increase return on investment, the QoE of these services needs to be appropriately designed with as little investment as possible, and its normality needs to be monitored while services are provided. In general, the QoE of these services degrades due to compression and network conditions (e.g., packet loss and delay). Therefore, it is necessary to develop a QoE-estimation model by taking into account the impact of compression and network on quality. This paper introduces subjective-quality-assessment methods and QoE-estimation models that assess user QoE in video-streaming services and standardization activities. I. INTRODUCTION Video-streaming-service providers can deliver highresolution (e.g., high definition (HD), ultra HD-1 (UHD-1) and UHD-2) audiovisual content over Internet Protocol (IP) networks because encoders and decoders (codecs) [1], [2], [3], streaming protocols [4], [5], and networks have become technologically advanced. As a result, the use of these services has been increasing drastically in the past decade. However, service providers need to maintain high-quality services by investing in equipment (e.g., network and server). This investment is a heavy burden for these service providers. To appropriately invest, it is important to design a service and monitor its normality on the basis of quality of experience (QoE). The QoE of video-streaming services is primarily affected by the coding, transport protocol, network, and client. Regarding coding, QoE is affected by the codec (e.g., AAC [1], H.264 [2], and H.265 [3]), audio bitrate, video resolution, framerate, and bitrate. The QoE impairments due to the network are different according to the transport protocol. In User Datagram Protocol (UDP)-based video-streaming services, such as Linear TV, freezing of the audio and video frame or audio and video frame loss occurs due to IP packet loss or discard. The playback of the audiovisual content is interrupted until sufficient data for playback is received in Transmission Control Protocol (TCP)-based video-streaming services such as adaptive-bitrate streaming services (e.g., HTTP-live streaming (HLS) [4] and MPEG-DASH [5]). This leads to initial loading and stalling events. Also in TCP-based streaming, when throughput reduction occurs, the quality level that is most suitable under the current network conditions can be selected (i.e., adaptation) because there are several files, i.e., chunks/segments, corresponding to representations of different bitrates on the server. Therefore, it is important to evaluate the QoE affected by these impairments. The QoE of video-streaming services should be evaluated on the basis of subjective assessment [6], [7]. However, subjective quality assessment, in which participants evaluate the quality of various processed video sequences (PVSs), is time-consuming and expensive. In addition, special assessment equipment such as a professional monitor and room is required. Moreover, it is difficult to ask users to evaluate the QoE while the service is provided. Therefore, it is desirable to develop an objective-quality-estimation model (hereafter, QoE-estimation model) that can be used for QoE assessment. QoE-estimation models [8] [46] have been developed on the basis of subjective scores and standardized. Such models can also be categorized into three types, i.e., media-based [8] [19], packet-based [20] [42], and planning [43] [46], on the basis of input. Media-based models [8] [19] take media signals as input to estimate QoE. These models can be used for, e.g., monitoring the normality of video-streaming services in the head-end. Packet-based models [20] [42] take application-layer, e.g., the bitrate, framerate, resolution, frame type (Iframe, B-frame, and P-frame), quantization parameter (QP), motion vector, frame loss, and stalling, as input to estimate QoE. Packet-based models can be used for monitoring the normality of video-streaming services at the client. Planning models [43] [46] take QoE-related planning parameters, such as audio bitrate, video resolution, framerate, and bitrate, and packet loss, as input to estimate QoE. Planning models can be used for designing an application and network. These subjective-quality-assessment methods and QoEestimation models should be standardized for network and video-streaming-service providers to design their services and monitor their normality on the basis of QoE. To do this, the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T) and ITU Radiocommunication Sector (ITU-R) are responsible for the standardization of audio and visual QoE-assessment methods and have been studying both subjective-quality-assessment methods and QoE-estimation models. The ITU-T Study Group 12 (SG12)

2 TABLE I SUBJECTIVE-QUALITY-ASSESSMENT METHODS Category rating Absolute rating ACR (P.910, P.911, P.920) Relative rating DCR (P.910, BT.500, P.911), CCR(BT.500) Continuous rating Absolute rating SSCQE (BT.500, P.911) Relative rating DSCQS (BT.500), SDSCE (BT.500) CCR: Comparison Category Rating SSCQE: Single Stimulus Continuous Quality Evaluation SDSCE: Simultaneous Double Stimulus for Continuous Evaluation a) Flow Fig. 1. Flow and rating scale of DSCQS b) Rating scale studies the performance, quality of service (QoS), and QoE of telecommunications services, is the lead SG for these study items, and coordinates its work with that of other standardization organizations. The responsibility of SG12 includes the standardization of video QoE-estimation models. The ITU-R SG6 is responsible for broadcasting services such as radio and television services. The QoE aspects are studied in working party 6C (WP6C). Another group is called the Video Quality Experts Group (VQEG), which technically investigates the validity of QoE-estimation models proposed to ITU and proposes the best one(s) to SG12. This paper gives an overview of subjective-qualityassessment methods and QoE-estimation models and their standardization activities. Subjective-quality-assessment methods are described in Section II and QoE-estimation models are described in Section III. Current standardization activities for QoE-estimation models are explained in Section IV Finally, a summary of this paper is given in Section V. II. SUBJECTIVE-QUALITY-ASSESSMENT METHODS Subjective quality assessment, in which participants judge video quality, has been extensively studied, and ITU has standardized various methods for different purposes [6], [7]. Subjective quality assessment is the most fundamental and reliable way to quantify user QoE. Subjective-quality-assessment methods are categorized based on absolute and relative ratings and can also be differentiated based on category and continuous ratings. ITU has standardized many subjective-quality-assessment methods such as the absolute category rating (ACR), ACR with hidden reference (ACR-HR), degradation category rating (DCR), and double stimulus continuous quality scale (DSCQS). These methods are summarized in Table I. Since these methods have different specific features, experimenters need to select a method in accordance with the aim of the planned subjectiveassessment experiment. In subjective quality assessment for video-streaming services, the DSCQS and ACR methods are often used. The DSCQS method is often used for determining the coding bitrate for a certain video codec because it has the ability of evaluating the small difference in quality between source and target videos. The ACR method is often used for evaluating the entire quality affected by the system among codec, network, transport, and client and is used for developing a QoEestimation model. The flows and rating scales of DSCQS and a) Flow Fig. 2. Flow and rating scale of ACR b) Rating scale ACR are explained as follows. With DSCQS, a pair of a reference video and a test video with artifacts, such as video coding, is presented, as shown in Fig. 1 a). These videos are presented twice, and assessment is carried out when the second videos are presented. The videos are presented in random order and participants are not told which one is the reference video. The participants assess the videos on a continuous quality-assessment scale based on five categories, as shown in Fig. 1 b). The assessment scale is normalized to the range 0-100, and the difference in videoassessment values for the reference and test videos in each pair is calculated. These video-quality differential values are averaged across all the participants to yield a DSCQS value. With ACR, test videos, including reference videos, are presented one at a time and are rated independently on the category scale shown in Fig. 2. Video quality is assessed by scoring a video on the basis of a five-category-discrete scale. Then, the video-quality values are averaged across all the participants to yield an ACR value as a mean opinion score (MOS). III. QOE-ESTIMATION MODELS QoE-estimation models can be grouped into three categories: media-based, packet-based, and planning, as shown in Table II. In this section, an overview of these standardized models is given. A. Media-based models There are three types of media-based models: full-reference (FR) [8] [13], which take source and degraded video signals as input to estimate the QoE, reduced-reference (RR) [14] [16], which takes degraded video signals and features calculated from source-video signals as input to estimate the QoE, and no-reference (NR) [17] [19], which takes degraded video signals as input to estimate the QoE.

3 TABLE II QOE-ESTIMATION MODELS Reference Degraded Media-based model J.144, J.247, J.341, J.246, J.342 Packet-based model P.1201, P.1202, P.1203 Planning model G ) Noise removal 1) Noise removal 2) Subsampling 2) Subsampling Coding/Streaming server QoE impairments Ch 1 3) Time alignment 4) Spatial alignment QoE impairments on Ch 1 QoE impairments Ch 1 Ch 2 5) Local similarity and difference feature 6) Global spatial degradation-blockiness 6) Global spatial degradation-blockiness Ch 2 Fig. 3. Use case of FR media-based model 7) Global temporal degradation-jerkiness 8) Analysis of feature distribution 7) Global temporal degradation-jerkiness Media-based models can be used as follows. If a codec is constantly or intermittently producing QoE impairments, such as coding artifacts and blackout in the head-end, all users who watch a certain video experience QoE degradation, as shown in Fig. 3. Therefore, it is important to promptly detect these impairments. To detect QoE impairments, human operators need to watch a certain video channel all the time. However, there are risks in overlooking QoE impairments. This also leads to companies baring high personnel expenses. To address these issues, it is preferable to incorporate an FR media-based model into a system because it can be used for detecting QoE degradation by comparing source and degraded videos. Three FR (ITU-T Recommendations J.144, J.247, and J.341) and two RR media-based models (ITU-T Recommendations J.246 and J.342) have been standardized. However, no NR media-based model has been standardized because the estimation accuracy has not reached a sufficient level. An overview of a standardized FR media-based model, i.e., ITU-T Recommendation J.341, is now given. ITU-T Recommendation J.341 is an FR-media-based model and can be used for QoE estimation of HDTV. A block diagram of the J.341 model is illustrated in Fig. 4. Each block processes as follows: 1) Each frame of the reference (source reference circuit (SRC)) and PVS is spatially lowpass filtered. 2) These SRC and PVS frames are subsampled to three different resolutions, R1 ( pixels), R2 ( pixels), and R3 ( pixels). 3) Time alignment is carried out using the SRC and PVS at low resolution, R3, because SRC and PVS frames are not always matched in terms of frame number. 4) Spatial frame alignment is carried out using the SRC and PVS at R1 because SRC and PVS pixels are not always matched. 5) The local-similarity and difference-feature calculates the similarity and difference features using the 9) Perceptually motivated aggregation Predicted score Fig. 4. Block diagram of J.341 model correlation and variance of the pixels at R2. 6) A blockiness feature is calculated using the frames at R1. This feature measures the visibility of block borders introduced by coding and/or transmission errors. 7) A jerkiness feature is calculated by averaging the product of relative display time, a non-linear transform of display time, and non-linear transform of motion intensity. 8) A local feature distribution is analyzed, and some features are calculated on the basis of the similarity and difference features. 9) Finally, a predicted score is estimated using the features described above. B. Packet-based model Three Recommendations, i.e., ITU-T Recommendations P.1201, P.1202, and P.1203, have been standardized for videostreaming services as packet-based models. Packet-based models can be used as follows. For example, video-streamingservice providers can incorporate a model into clients such as set-top boxes and smartphones, and QoE can be estimated at clients and reported to a QoE-management server to monitor the normality of services, as shown in Fig. 5 a). Another example is as follows. Network providers often measure the throughput to maintain QoS. However, excessive investment does not always lead to improvement in QoE. Therefore, network providers should invest in a network on the basis of the QoE level. To do this, the QoE for a certain area can be monitored and mapped to an area map using a packetbased model, as shown in Fig. 5 b). Then network providers can invest in the network for a low QoE area.

4 Video-streaming-service provider QoE-management server Packet-header P-E-V Parameter-calculation for video Video-qualityestimation Video MOS Codec/streaming server RTP header UDP header P-E-R Audiovisualqualityestimation Audiovisual MOS a) Case 1 Network provider IP header Side about codec, client behavior etc. P-E-A Available to all s Parameter-calculation for audio Key: P-E-V Defined Interface Parameter-extraction for video Audio-qualityestimation P-E-R P-E-A Audio MOS Parameter-extraction for rebuffering Parameter-extraction for audio Codec/streaming server Fig. 6. Block diagram of P.1201 model Low QoE b) Case 2 Fig. 5. Use cases of packet-based model High QoE The standardized packet-based models are now described. ITU-T Recommendations P.1201 and P.1202 can be used for QoE estimation of IPTV, and P.1203 can be used for that of adaptive-bitrate video-streaming services. 1) P.1201 models: The ITU-T SG12 has standardized the parametric non-intrusive assessment of audiovisual mediastreaming quality (P.NAMS) [27] lower-resolution (LR: i.e., quarter common intermediate format (QCIF, pixels), quarter video graphics array (QVGA, pixels), or half VGA (HVGA, pixels)) [28], higher-resolution (HR: i.e., standard definition (SD, pixels), and high-definition (HD, or pixels)) [30] application areas. The P.1201 models can be applied in the quality monitoring of UDP-based streaming, i.e., linear TV. The P and P models have a different QoEestimation algorithm for LR or HR, respectively. However, evaluating QoE impairments, such as coding artifacts, freezing frames, or frame losses due to packet loss, and stalling events, are the same, except for video resolution. Therefore, P is only explained in this paper. A block diagram of P is illustrated in Fig. 6. The parameter-extraction (P- E) s extract audio- and video-related parameters using real-time transport protocol (RTP) headers (i.e., RTP timestamp, sequence number, marker bit, and payload length) and stalling-related parameters such as the stalling start time and length. With these parameters, parameter-calculation s derive parameters that are used by quality-estimation s. Finally, the quality-estimation s output individual estimates of audio, video, and audiovisual quality. Audio quality is affected by the codec type, coding bit rate, packet loss, and rebuffering. Therefore, the relationship between the following QoE factors and subjective audio quality is modeled. Effect of audio codec (i.e., AMR-NB, AMR-WB+, AAC- LC, HE-AACv1, and HE-AACv2) on audio quality Effect of coding bit rate on audio quality Effect of lost audio-frame length due to packet loss on audio quality Effect of stalling on audio quality As with audio quality, video quality is affected by the codec type, coding bit rate, packet loss, and stalling. Video quality is also affected by the number of bits per video-frame type because it varies depending on the spatio-temporal of the video content. Therefore, the relationship between the following QoE factors and subjective video quality is modeled. Effect of video codec (i.e., MPEG-4 and H.264/AVC) on video quality Effect of video resolution (i.e., QCIF, QVGA, and HVGA) on video quality Effect of coding bit rate, framerate, and ratio between total bit count and intra-coded frame (I-frame) bit count on video quality Effect of the number of packet-loss events, number of damaged video frames, and video frame area damaged by the packet losses on video quality Effect of the number of stalling events, average stalling length, and effect of multiple stalling events (i.e. average interval between stalling events) on video quality Audiovisual quality is calculated based on audio and video quality. 2) P.1202 model: The ITU-T SG12 has standardized the parametric non-intrusive bitstream assessment of video-mediastreaming quality (P.NBAMS) [20]. The application area (i.e., LR and HR) of the P.1202 model is the same as that of a P.1201 model. However, this recommendation P.1202 only defines the video-quality-estimation model and cannot be used for the quality estimation of encrypted packets because the model requires access to the bitstream. In this recommendation, the relationship between the following factors and subjective video quality is modeled.

5 Effect of video codec (e.g., H.264/AVC) on video quality Effect of video resolution (e.g., QCIF, QVGA, and HVGA) on video quality Effect of framerate, QP, motion vector (MV), and macroblock on video quality Effect of the video frame damaged by packet losses on video quality by taking into account frame type as well as packet losses. Effect of the stalling duration and framerate on video quality Video quality is calculated on the basis of this relationship. 3) P.1203 model: The ITU-T SG12 has standardized the parametric bitstream-based quality assessment of progressive download and adaptive audiovisual-streaming services over reliable transport (P.NATS) [38]. The P.1203 model can be applied in the quality monitoring of TCP-based adaptive bitrate video streaming, i.e., HLS and MPEG-DASH. The P.1203 model can be used for the QoE estimation of audiovisual content encoded by Advanced Audio Coding Low Complexity (AAC-LC) (bitrate: kbits/s) and H.264/Advanced Video Coding (AVC) (bitrate: Mbits/s, resolution , framerate: fps) and various added types of stalling events. In addition, there are four modes for the video-quality-estimation : mode 0 takes metadata such as audio bitrate, video resolution, framerate, and bitrate as input, mode 1 takes meta-data and frame size/type as input, mode 2 takes meta-data and up-to 2% of the media stream as input, and mode 3 takes meta-data and any from the media stream as input in addition to a stalling event, i.e., a tuple of start time and duration, both measured in seconds. The output is defined as O.21: audiocoding quality per output-sampling interval, O.22: videocoding quality per output-sampling interval, O.23: perceptualbuffering indication, O.34: audiovisual-segment-coding quality per output-sampling interval, O.35: final audiovisual-codingquality score, which includes aspects of temporal integration, and O.46: final media-session-quality score. Outputs O.21, O.22, and O.34 are outputs per-one-second scores on a 1-5 quality scale. Outputs O.23, O.35, and O.46 are outputs a single aggregated score on a 1-5 quality scale for the session. A block diagram of the P.1203 model is illustrated in Fig. 7. Each is processed as follows: The P-E receives and acquires the about the audio and video bitrate, resolution, and framerate per chunk, measures the re-buffering timing and duration, and calculates the stalling-related parameters using re-buffering. The audio-quality-estimation (ITU-T Recommendation P ) estimates audio quality per one-secondsampling interval using audio bitrate. The video-quality-estimation (ITU-T Recommendation P ) estimates video quality per one-secondsampling interval. In mode 0, video resolution, framerate, and bitrate are used as input. Stream I.01 Media-parameter extraction Buffer-parameter extraction I.11 I.13 I.14 Input Pa: Audio-qualityestimation (ITU-T Rec. P ) Pv: Video-qualityestimation (ITU-T Rec. P ) I.GEN: Device info available to all modes Pq: Quality-integration (ITU-T Rec. P ) Pav: A/V integration/temporal Pb: Quality impact due to buffering Diagnostic Fig. 7. Block diagram of P.1203 model O.21 O.22 O.34 O.35 O.23 O.46 Integral MOS In mode 1, video frame size is used as input as well as mode 0 parameters. The average I-frame size (I) and average non-i-frame size (I n ) are calculated to derive the impact of the I-frame ratio (= I/I n ) on video quality. In mode 2, The QP derived from 2% of each videoframe bitstream and video-frame types are used as input as well as mode 0 parameters. In mode 3, The QP and video-frame types are used as input as well as mode 0 parameters. The quality-integration (ITU-T Recommendation P ) takes audio and video quality and stalling parameters as input to derive the impact of adaptivity and stalling event on the final media-session quality and estimates final media-session quality. C. Planning model ITU-T has standardized ITU-T Recommendation G.1071 for video-streaming services in a planning model. A planning model can be used by network or video-streaming-service providers. The following are several use cases. To optimize video quality, a video-streaming-service provider can design a coding bitrate for a given network condition. Like a videostreaming-service provider, to optimize video quality, a network provider can design a network condition (e.g., packet loss) for coding quality designed by a video-streaming-service provider. The G.1071 model is based on the P.1201 models and some intermediate parameters (e.g., freezing or lost video frame and video coding complexity) of a P.1201 model are calculated from the bitrate and packet loss. Therefore, ITU-T Recommendation defines the conversion rules of intermediate parameters of the P.1201 models. Application ranges of the G.1071 model is the same as those of the P.1201 models. A block diagram of the G.1071 model is illustrated in Fig. 8. IV. CURRENT STANDARDIZATION ACTIVITIES FOR QOE-ESTIMATION MODELS Recently, 4K-UHD video has been provided for both IPTV and adaptive-bitrate streaming services. Like HD video services, a model for estimating QoE of such services is preferred. Therefore, ITU-T SG12 has launched work items, APSIPA ASC 2017

6 Network planning assumptions Conversion rules Video Audio Audiovisual Fig. 8. Block diagram of G.1071 model ITU-T P.1201 MOSA MOSAV MOSV i.e., P.NAMS Phase 2 and P.NATS Phase 2, to extend from HD to 4K-resolution video. In addition, they are developing QoE-estimation models for 4K video encoded by H.265 as well as H.264 codec. They plan to develop these models and standardize them in Furthermore, VQEG have also studied media-based models for 4K-resolution video in conjunction with ITU-T SG12. These models will also be developed and standardized in V. CONCLUSIONS The paper gave an overview of subjective-qualityassessment methods, QoE-estimation models, and standardization activities. Regarding subjective-quality-assessment methods, the relationship between category and continuous ratings and that between absolute and relative ratings were summarized. In addition, use cases of these methods were described. In media-based, packet-based, and planning models, several standardized models were explained and their use cases summarized. Recently, 4K-UHD video have been provided for IPTV and adaptive-bitrate streaming services. Therefore, several types of models that can be used for QoE planning and monitoring have been studied and will be standardized under ITU-T SG12. REFERENCES [1] Information technology Generic coding of moving pictures and associated audio Part 7: Advanced Audio Coding (AAC), ISO/IEC :1997, Dec [2] Advanced Video Coding for Generic Audiovisual Services, ITU-T Recommendation H.264, Feb [3] High efficiency video coding, ITU-T Recommendation H.265, Apr [4] Http live streaming. [Online]. Available: [5] Information technology Dynamic adaptive streaming over HTTP (DASH) Part 1: Media presentation description and segment formats, ISO/IEC :2014, May [6] Subjective Video Quality Assessment Methods for Multimedia Applications, ITU-T Recommendation P.910, Apr [7] Methodology for the subjective assessment of the quality of television pictures, ITU-R Recommendation BT.500, Jan [8] Objective Perceptual Video Quality Measurement Techniques for Digital Cable Television in the Presence of a Full Reference, ITU-T Recommendation J.144, Mar [9] Objective Perceptual Multimedia Video Quality Measurement in the Presence of a Full Reference, ITU-T Recommendation J.247, Aug [10] Objective Perceptual Multimedia Video Quality Measurement of HDTV for Digital Cable Television in the Presence of a Full Reference, ITU-T Recommendation J.341, Jan [11] M. H. Pinson and S. Wolf, A new standardized method for objectively measuring video quality, IEEE Trans. Broadcast., vol. 50, no. 3, pp , Sept [12] P. L. Callet, C. Viard-Gaudin, S. Pechard, and E. Caillault, No reference and reduced reference video quality metrics for end to end QoS monitoring, IEICE Trans. Commun., vol. 89, no. 2, pp , Feb [13] Y. Xue, B. Erkin, and Y. Wang, A novel no-reference video quality metric for evaluating temporal jerkiness due to frame freezing, IEEE Trans. Multimedia, vol. 17, no. 1, pp , Jan [14] Perceptual Audiovisual Quality Measurement Techniques for Multimedia Services over Digital Cable Television Networks in the Presence of a Reduced Bandwidth Reference, ITU-T Recommendation J.246, Aug [15] Objective multimedia video quality measurement of HDTV for digital cable television in the presence of a reduced reference signal, ITU-T Recommendation J.342, Apr [16] T. Yamada, Y. Miyamoto, Y. Senda, and M. Serizawa, Videoquality estimation based on reduced-reference model employing activitydifference, IEICE Trans. Commun., vol. 92-A, no. 12, pp , Dec [17] P. L. Callet, C. Viard-Gaudin, S. Pechard, and E. Caillault, No Reference and Reduced Reference Video Quality Metrics for End to End QoS Monitoring, IEICE Trans. Commun., vol. 89, no. 2, pp , Feb [18] M. C. Q. Farias and S. K. Mitra, No-reference video quality metric based on artifact measurements, in IEEE International Conference on Image Processing 2005, vol. 3, Sept. 2005, pp. III [19] F. Yang, S. Wan, Y. Chang, and H. R. Wu, A novel objective noreference metric for digital video quality assessment, IEEE Signal Process. Lett., vol. 12, no. 10, pp , Oct [20] Parametric non-intrusive bitstream assessment of video media streaming quality, ITU-T Recommendation P.1202, Oct [21] Parametric non-intrusive bitstream assessment of video media streaming quality lower resolution application area, ITU-T Recommendation P , Oct [22] Parametric non-intrusive bitstream assessment of video media streaming quality higher resolution application area, ITU-T Recommendation P , May [23] O. Verscheure, X. Garcia, G. Karlsson, and J. P. Hubaux, User-oriented QoS in packet video delivery, IEEE Network, vol. 12, no. 6, pp , Nov [24] S. O. Lee, K. S. Jung, and D. G. Sim, Real-time objective quality assessment based on coding parameters extracted from H.264/AVC bitstream, IEEE Trans. Consum. Electron., vol. 56, no. 2, pp , May [25] A. P. C. da Silva, P. Rodriguez-Bocca, and G. Rubino, Optimal qualityof-experience design for a P2P multi-source video streaming, in 2008 IEEE International Conference on Communications, May 2008, pp [26] L. Anegekuh, L. Sun, E. Jammeh, I. H. Mkwawa, and E. Ifeachor, Content-based video quality prediction for hevc encoded videos streamed over packet networks, IEEE Trans. Multimedia, vol. 17, no. 8, pp , Aug [27] Parametric non-intrusive assessment of audiovisual media streaming quality, ITU-T Recommendation P.1201, Oct [28] Parametric non-intrusive assessment of audiovisual media streaming quality - lower resolution application area, ITU-T Recommendation P , Oct [29] K. Yamagishi and S. Gao, Light-weight audiovisual quality assessment of mobile video: ITU-T Rec. P , in Multimedia Signal Processing (MMSP), 2013 IEEE 15th International Workshop on, Sept. 2013, pp [30] Parametric non-intrusive assessment of audiovisual media streaming quality - higher resolution application area, ITU-T Recommendation P , Oct [31] M. N. Garcia, P. List, S. Argyropoulos, D. Lindegren, M. Pettersson, B. Feiten, J. Gustafsson, and A. Raake, Parametric model for audiovisual quality assessment in IPTV: ITU-T Rec. P , in Multimedia Signal Processing (MMSP), 2013 IEEE 15th International Workshop on, Sept. 2013, pp [32] K. Yamagishi and T. Hayashi, Non-intrusive packet-layer model for monitoring video quality of IPTV services, IEICE Trans. Fundamentals, vol. 92-A, no. 12, pp , Dec [33] K. Yamagishi and T. Hayashi, Parametric Packet-Layer Model for Monitoring Video Quality of IPTV Services, IEEE ICC 2008, CQ04-3, pp , May [34] J. Gustafsson, G. Heikkila, and M. Pettersson, Measuring multimedia quality in mobile networks with an objective parametric model, in th IEEE International Conference on Image Processing, Oct. 2008, pp

7 [35] A. Raake, M. N. Garcia, S. Moller, J. Berger, F. Kling, P. List, J. Johann, and C. Heidemann, T-V-model: Parameter-based prediction of IPTV quality, in 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Mar. 2008, pp [36] J. Joskowicz and J. C. L. Ardao, A general parametric model for perceptual video quality estimation, in 2010 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR 2010), June 2010, pp [37] Amendment 2: New Appendix III Use of ITU-T P.1201 for nonadaptive, progressive download type media streaming, ITU-T Recommendation P.1201 Amendment, Dec [38] Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport, ITU- T Recommendation P.1203, Nov [39] Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport Video quality estimation, ITU-T Recommendation P , Dec [40] Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport Audio quality estimation, ITU-T Recommendation P , Nov [41] Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport Quality integration, ITU-T Recommendation P , Dec [42] K. Yamagishi and T. Hayashi, Parametric Quality-Estimation Model for Adaptive-Bitrate Streaming Services, IEEE Trans. Multimedia, DOI: /TMM [43] Opinion model for video-telephony applications, ITU-T Recommendation G.1070, July [44] Opinion model for network planning of video and audio streaming applications, ITU-T Recommendation G.1071, Nov [45] K. Yamagishi and T. Hayashi, Opinion Model using Psychological Factors for Interactive Multimodal Services, IEICE Trans on Commun., vol. E89-B, no. 2, pp , Feb [46] K. Yamagishi and T. Hayashi, Video-Quality Planning Model for Videophone Services, ITE, vol. 62, no. 7, pp , July 2008.