Quality Evaluation of Coded Video

Transcription

1 r Qualty Evaluaton of Coded Vdeo Luís Mguel Malvero Perera Tomaz Roque Dssertaton submtted for obtanng the degree of Master n Eletral and Computer Engneerng Jury Supervsor: Professora Doutora Mara Paula dos Santos Queluz Co-Supervsor: Engenhero Tomás Gomes da Slva Serpa Brandão Presdent: Members: Professor Doutor José Manuel Bouas Das Professor Doutor Paulo Lus Serras Lobato Correa November 009

2

3 Aknowledgements Aknowledgements Frst of all, I would lke to thank to my supervsors Mara Paula Queluz and Tomás Gomes Brandão for the unque opportunty to perform ths thess and for the onstant knowledge and experene sharng. Ther orentaton, avalablty, gudelnes, advsng, opnon, and onstant support, were a key fator for the ompleton of ths work, and wll also be useful for my professonal future. To all my frends from Insttuto Superor Téno, for all the moments spent durng the aadem lfe, espeally Luís Gomes, João Nobre, Flpe Leonardo and Inês for all the help and enouragement durng the last sx years and to the Algarve and Trps group for makng my summers even better. To my brother João Pedro whose adve I ve always sought. To my grandparents, who always gave me all support and love to aheve ths goal. And, fnally, to my mother and father for ther unondtonal love and ther beleve that I an aomplsh anythng I purpose myself to do.

4

5 Abstrat Abstrat Wth the multmeda ommunatons emergene, there has been an nreasng need to develop qualty measurements tehnques that an predt pereved vdeo qualty automatally. In ths dssertaton two dfferent strateges for vdeo qualty measurement, n the presene of dstortons due to ompresson, are onsdered. These two dfferent vdeo qualty assessment metrs are ommonly known as Subjetve Qualty Metrs and Objetve Qualty Metrs. Wth regard to the frst one, a subjetve vdeo qualty assessment test sesson was onduted, n order to aheve, from a number of human observers, a subjetve qualty measurement, the Mean Opnon Sore (MOS), for a group of representatve vdeo sequenes. Ths frst method has been regarded for many years as the most relable for qualty measurement; however, ths assessment method s hghly tme onsumng and requres approprated vewng ondtons. In order to provde an automat evaluaton and montorng of vdeo data qualty, a Mean Opnon Sore predton model based n objetve qualty metrs s also proposed n ths dssertaton. The goal of ths type of vdeo qualty assessment measurement s to desgn an automated qualty assessment method that orrelates well wth subjetve qualty assessment and, as onsequene, wth human vsual perepton. The performane of ths seond vdeo qualty evaluaton method s valdated by onfrontng the resultng qualty measures wth the sores produed by human judgment (subjetve tests), and usng performane metrs proposed by VQEG (Vdeo Qualty Expert Group). Ths seond strategy provded good results beng able to predt vdeo qualty sores lose to those resultng from subjetve assessment. Keywords Subjetve Vdeo Qualty Metrs, Vdeo Qualty Assessment, Objetve Vdeo Qualty Metrs, Mean Opnon Sore.

6 v

7 Resumo Resumo Com o desenvolvmento das omunações multméda, tem havdo uma neessdade resente de desenvolver métodos que permtam avalar a qualdade de vídeo odfado de uma forma automáta. Neste projeto são onsderadas duas dferentes estratégas para medr a qualdade de vdeo na presença de dstorções devdas à ompressão. Essas duas dferentes abordagens de avalação de qualdade de vídeo são geralmente onhedas omo métras de qualdade subjetva e métras de qualdade objetva. Relatvamente à prmera, fo feta uma sessão de testes de avalação de qualdade de vídeo subjetva, om o objetvo de obter, a partr de um determnado número de observadores humanos, meddas de qualdade subjetva, o Mean Opnon Sore, para um grupo representatvo de sequênas de vídeo. Este prmero método tem sdo onsderado ao longo do tempo omo o mas onsstente para a medção de qualdade; ontudo, este método de avalação é demorado e requer ondções de vsualzação apropradas. Com o objetvo de forneer uma avalação automáta bem omo a montorzação da qualdade de vídeo, é proposto nesta dssertação um modelo de estmação do Mean Opnon Sore, baseado em métras de qualdade objetvas. A prnpal razão para desenvolver este sstema de avalação de qualdade prende-se om o fato de se pretender um método de avalaçao de qualdade autónomo que se orrelaone bem om a avalação de qualdade subjetva. O desempenho deste segundo método de avalação de qualdade de vídeo fo valdado onfrontando os valores resultantes deste modelo om a pontuação atrbuída pelos observadores durante os testes subjetvos. Para tal, utlzaram-se métras que permtem estabeleer uma relação quanttatva entre os métodos subjetvos e objetvos, propostas pelo VQEG (Vdeo Qualty Expert Group). Verfou-se que, de um modo geral, o método proposto produz bons resultados, sendo apaz de estmar uma pontuação de qualdade de vídeo próxma da pontuação resultante de uma avalação subjetva. Palavras-have Métras Subjetvas de Qualdade de Vídeo, Avalação de Qualdade Vídeo, Métras Objetvas de Qualdade de Vídeo, Mean Opnon Sore. v

8 v

9 Table of Contents Table of Contents Aknowledgements... Abstrat... Resumo... v Table of Contents...v Lst of Fgures... x Lst of Tables... x Lst of Aronyms...x Introduton... Vdeo Qualty Evaluaton Metrs...5. Introduton Subjetve qualty metrs Vewng ondtons Seleton of test materals Observers seleton Vdeo evaluaton sesson Useful nformaton for the assessment Vdeo qualty assessment methods Objetve qualty metrs Classfaton of objetve metrs Objetve assessment approahes... 3 Subjetve Qualty Evaluaton Introduton Subjetve assessment Vewng and test ondtons Charaterzaton of the test sequenes Vdeo sequenes seleton... 3 v

10 3.6 Vdeo ompresson Vdeo qualty evaluaton program nterfae Statstal analyss Calulaton of mean sores Confdene nterval Observer valdaton Subjetve qualty assessment results Objetve Qualty Evaluaton Introduton Proposed MOS Predton Algorthms Motvatons MOS predton models MOS evoluton wth eah feature Regresson model Prnpal Component Analyss (PCA) Metrs Performane Results and parameters analyss Low omplexty model Hgh omplexty model Features spae reduton wth PCA Comparson wth related work Conlusons and Future Dretons...8 Referenes...83 v

11 Lst of Fgures Lst of Fgures Fgure.: (a) Ishara Test plate; (b) Snellen Eye Chart...8 Fgure.: Test sesson struture...9 Fgure.3: Double Stmulus Imparment Test Tral Struture [ITU98]: a) DSIS I;b) DSIS II... Fgure.4: Fve pont Imparment Ratng Sale: (a) usng tehnologal means; (b) usng tradtonal ways... Fgure.5: Tral struture for Comparson Test for: a) Sngle presentaton; b) Double presentaton...3 Fgure.6: Comparson Ratng Sale....3 Fgure.7: Sngle Stmulus Tral Struture [WR06]...4 Fgure.8: Automat votng deve Slder [WP07]...6 Fgure.9: Double Stmulus Contnuous Qualty Sale Tral Struture...7 Fgure.0: Double Stmulus Contnuous Qualty Sale (parallelsm wth DSIS s qualty adjetves)...7 Fgure.: SDSCE prnple [ITU08]...8 Fgure.: SSIM s measurement system...3 Fgure 3.: Testng room...9 Fgure 3.: Sobel flters. (a) Sobel flter responsble for detetng horzontal pxel dfferenes; (b) Sobel flter responsble for detetng vertal pxel dfferenes [WP99]...30 Fgure 3.3: (a) and () Orgnal vdeo frames; (b) and (d) Correspondng gradent norm mages...3 Fgure 3.4: Temporal atvty measurement proess n a vdeo sequene...3 Fgure 3.5: Spatal-temporal atvty of a vdeo sequene set (CIF format)...3 Fgure 3.6: Table temporal atvty, frame by frame onsderng (a) all the vdeo sequenes; (b) the frames affeted by abrupt hange of amera perspetve...33 Fgure 3.7: Seleted vdeo sequenes for spatal-temporal atvty has been takng wth perentle 95%...34 Fgure 3.8: Vdeo s sequenes used n the subjetve tests...34 Fgure 3.9: Vdeo sequenes enoded wth dfferent values of btrate...35 Fgure 3.0: Artfats ntrodued by (a) H.64 ompresson (blur effet) and (b) MPEG- ompresson (blok effet)...37 Fgure 3.: MSU pereptual vdeo qualty player nterfae: a) vdeo label (referene/dstorted vdeo); b) vdeo wndow; ) play button to start the vdeo sequene; d) vdeo tme bar...38 Fgure 3.: Normal dstrbuton nterval...40 Fgure 3.3: MOS wth onfdene nterval of 95.5% for (a) H.64 and (b) MPEG Fgure 3.4: Webste sreenshots...45 Fgure 4.: MOS evoluton wth btrate of some vdeo sequenes...49 Fgure 4.: MOS evoluton wth the MSE of some vdeo sequenes...49 Fgure 4.3: MOS relaton wth spatal and temporal atvtes for a set of vdeo sequenes enoded at: (a) 8 kbt/s and (b) 04 kbt/s...50 Fgure 4.4: MOS evoluton wth the btrate for a) H.64; b) MPEG Fgure 4.5: Relaton between MOS and a) Global MSE for H.64; b) MSE Varane for H Fgure 4.6: Relaton between MOS and a) Global MSE for MPEG-; b) MSE Varane for x

12 MPEG Fgure 4.7: MOS evoluton wth: a) Global Temporal Atvty; b) Temporal Atvty Varane...56 Fgure 4.8: MOS evoluton wth: a) Global Spatal Atvty (5 kbt/s); b) Spatal Atvty Varane (5 kbt/s)...56 Fgure 4.9: MOS predton model desrpton...58 Fgure 4.0: MOS estmaton result for H.64: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...65 Fgure 4.: MOS estmaton result for MPEG-: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...66 Fgure 4.: MOS estmaton result for H.64 usng the true MSE: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...69 Fgure 4.3: MOS estmaton result for H.64 usng the estmated MSE: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...70 Fgure 4.4: MOS estmaton result for MPEG- usng the true MSE: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...7 Fgure 4.5: MOS estmaton result for MPEG- usng the estmated MSE: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...73 Fgure 4.6: MOS estmaton result for H.64 usng the estmated MSE after usng the PCA method: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...76 Fgure 4.7: MOS estmaton result for MPEG- usng the estmated MSE after applyng the PCA method: (a) tranng/0 test vdeo sequenes; (b) 5 tranng/7 test vdeo sequenes; () 8 tranng/4 test vdeo sequenes...77 x

13 Lst of Tables Lst of Tables Table.: General vewng ondtons [ITU99]...6 Table.: Fve grade sale...4 Table.3: Typal qualty assessment sale for DSCQS and SDSCE methods...7 Table.4: Vdeo qualty assessment methods man features...9 Table 3.: Dsplay and Room s ondtons...9 Table 3.: Compresson btrates used n H.64/AVC...36 Table 3.3: Compresson btrates used n MPEG Table 3.4: MOS usng vdeo ompresson standard H.64 and MPEG Table 4.: True PSNR and the estmated PSNR values...53 Table 4.: Regresson weghts for the low omplexty model: (a) for H.64 and (b) MPEG Table 4.3: Metrs performane for: (a) H.64 and (b) MPEG Table 4.4: Regresson weghts for the hgh omplexty model takng nto aount H.64 ompressed vdeo sequenes usng: (a) the true MSE; (b) the estmated MSE...68 Table 4.5: Model performane analyss for H.64 usng: (a) the true MSE; (b) the estmated MSE...7 Table 4.6: Regresson weghts for the three tranng/test onfguratons for MPEG- usng: (a) the true MSE; (b) the estmated MSE...7 Table 4.7: Model performane analyss for MPEG- usng: (a) the true MSE; (b) the estmated MSE...74 Table 4.8: Regresson weghts for the hgh omplexty usng the estmated MSE model after applyng PCA: (a) for H.64 and (b) MPEG Table 4.9: Metrs performane after applyng the PCA for: (a) H.64 and (b) MPEG x

14 x

15 Lst of Aronyms Lst of Aronyms ACR CIF DCR DCT DSCQS DSIS FR HDTV HVS IEEE ITU MOS MSE MSU NR PC PCA PSNR PVQM QCIF QoS RMS RR SDSCE SIF SS SSCQE SSIM VQEG VQM Absolute Category Ratng Common Intermedate Format Degradaton Category Ratng Dsrete Cosne Transform Double Stmulus Contnuous Qualty Sale Double Stmulus Imparment Sale Full Referene Hgh Defnton Televson Human Vsual System Insttute of Eletral and Eletrons Engneers Internatonal Teleommunatons Unon Mean Opnon Sore Mean Squared Error Mosow State Unversty No Referene Par Comparson Method Prnpal Component Analyss Peak Sgnal-to-Nose Rato Pereptual Vdeo Qualty Metr Quarter Common Intermedate Format Qualty of Serve Root Mean Square Error Redued Referene Smultaneous Double Stmulus for Contnuous Evaluaton Soure Input Format Sngle Stmulus Method Sngle Stmulus Contnuous Qualty Evaluaton Strutural Smlarty ndex Vdeo Qualty Experts Group Vdeo Qualty Metr x

16 xv

17 Chapter Introduton Introduton The assessment of mage qualty n vdeo and mage proessng systems plays an mportant role n dedng the qualty of serve n mage and vdeo ommunatons, network mantenane and even to ompare dfferent serve provders. Qualty assessment systems have a wde range of applatons from seurty serves to entertanment, whh nludes dgtal televson, nternet vdeo and n general the world of dgtal multmeda ommunatons. However, the automat evaluaton of dgtal magng systems qualty s a hallengng task sne t requres ether to math human perfeton or to overome human lmtatons. In order to gve an overvew of ths problem dffulty, t s neessary to understand the numerous fators that ontrbute to what a vewer pereves as vdeo qualty. Among these fators are the ndvdual nterests, qualty expetatons, vewng ondtons and dsplay type and propertes. The wde varety and subjetvty of some of these fators are ndators of the omplexty of the qualty measurement problem [Wnk07]. The man objetve of the present work s to approah the behavor of human vsual system n vdeo qualty evaluaton. In order to develop and standardze the requred tehnology for assessng the performane of dgtal vdeo proessng and ommunaton systems, some organzatons were formed. As example of

18 that, s the Insttute for Teleommunaton Senes (ITS) an Ameran organzaton whh s responsble for the promoton of advaned teleommunatons and nformaton nfrastruture development. The Insttute began n the 940s, however t was manly from 994 to 997, that ITS gave a large ontrbuton for the development of Ameran standards for gaugng the qualty of dgtal vdeo systems. These standards were named as ANSI T.80.0, ANSI T.80.0, ANSI T and ANSI T Generally, the ANSI T.80.0 provdes a standardzed set of vdeo test senes n dgtal format that an be used for subjetve and objetve testng of dgtal vdeo systems, whle the ANSI T.80.0 standard provdes a general desrpton of dgtal vdeo performane terms and mparments. Standard ANSI T defnes a whole new framework of objetve parameters that an be used to measure the qualty of dgtal vdeo systems, whle the ANSI T standard desrbes metrs for audo delay, vdeo delay, and audo-vsual synhronzaton, sne these parameters are mportant, partularly for nteratve teleommunatons serves. In Otober of 997, the Vdeo Qualty Experts Group (VQEG) was establshed n order to address vdeo qualty ssues. The VQEG prmary msson s to valdate objetve vdeo/multmeda qualty metrs and to report results to the ITU-T Study Groups 9 and and ITU-R Study Group 6. Ths organzaton s omposed of experts from varous bakgrounds and afflatons, nludng partpants from several nternatonally reognzed organzatons workng n the feld of vdeo qualty assessment. Currently, VQEG s ondutng an evaluaton of metrs n a multmeda senaro, whh s targeted at lower btrates and smaller frame szes as well as a wder range of odes and transmsson ondtons. Vdeo qualty evaluaton has thus beome a relevant subjet, whh s also evdened by the number of publatons, produts avalable (e.g., vdeo qualty evaluaton probes, known as Wtbe robots, for measurng the qualty of serve offered by multmeda ompanes suh as Portugal Teleom wth MEO) and reent nternatonal onferenes. Evaluaton of vdeo qualty an be aheved by two dfferent ways: through subjetve and objetve metrs. The subjetve vdeo qualty assessment s reognzed as the most relable mean of quantfyng user perepton sne human bengs are the ultmate reevers n most applatons. The Mean Opnon Sore (MOS), whh s a subjetve qualty measurement obtaned from a group of vewers, has been regarded for many years as the most onsstent form of qualty measurement. However, ths qualty measurement has some dsadvantages. These dsadvantages are related wth the fat that the MOS method s hghly tme onsumng for most applatons and annot be exeuted automatally. In order to provde an automat evaluaton and montorng of vdeo data qualty, relable and objetve metrs are requred. By ontrast to subjetve measurements, the objetve qualty metrs are based purely on mathematal methods, from qute smplst ones, lke Peak Sgnal-to-Nose Rato

19 (PSNR) and the Mean Squared Error (MSE), to sophstated ones that explot models of human vsual perepton and produe results far more onsstent wth the subjetve evaluaton [WSB03]. In other words, the objetve vdeo qualty measurement s done by a software whh proesses the vdeo sgnals n order to obtan a vdeo qualty sore. Thus, ths type of vdeo qualty metr s more advantageous as t ould provde real tme qualty montorng for vdeo applatons. Objetve vdeo qualty metrs an be lassfed aordng to the avalablty of the orgnal vdeo at the qualty assessment proess. Thus, objetve vdeo qualty metrs are lassfed n three lasses: Full Referene (FR), Redued Referene (RR) and No Referene (NR). If the orgnal vdeo s totally avalable as well as the dstorted vdeo, the objetve metrs are lassfed as FR. However, n many pratal vdeo serve applatons the referene vdeo sequenes are not aessble; n that ase, the metr s lassfed as NR f t s based only and exlusvely on the degraded vdeo. In some ases, to mprove the qualty estmaton, some haratersts of the orgnal vdeo are used, besdes the dstorted vdeo, thus the objetve metrs s ategorzed as RR metr. Comparatvely to FR, few approahes were proposed for RR vdeo qualty assessment and even less for NR qualty evaluaton. Relatvely to FR metrs, t should be hghlghted the great effort made by VQEG n order to develop them. VQEG developed FR metrs n two phases. From 997 to 000, VQEG arred out the frst phase named as Full Referene Televson - Phase I (FRTV), whle the seond phase was arred out from 000 to 004. Smlarly to the frst phase, ths seond phase was named as Full Referene Televson - Phase II (FRTV) [ITU04]. VQEG begun the RR and the NR televson tests n 000 and t was restarted n 005. The reently ompleted Multmeda Phase I test of VQEG assessed the performane of full-referene and redued-referene pereptual vdeo qualty measurement algorthms for QCIF, CIF and VGA formats. Based on t, two standards have been ssued by ITU-T [ITU08a] [ITU08b]. As future dretons, VQEG propose to study hybrd metrs, whh look not only at the deoded vdeo as n the other tests, but also at the enoded btstream 3. The work presented n ths thess has been organzed takng nto aount the two vdeo qualty assessment metrs mentoned prevously, the subjetve and the objetve metrs. The subjetve tests have been onduted n order to obtan the MOS of a number of representatves (n terms of spatal features, moton and odng artefats) ompressed vdeo sequenes. Two dfferent ompresson standards, the MPEG- and the H.64/AVC, have been onsdered. Beyond the fat that subjetve evaluaton s the most reognzed method for quantfaton of pereved qualty, the attanment of the MOS for a number of representatve ompressed vdeo sequenes ontrbuted to buld a database of great nterest for those workng on the vdeo qualty evaluaton feld. The man reason of that sgnfane s due to the fat that the majorty of subjetve results (e.g. those produed n MPEG groups) are only avalable for a restrt group of persons. Thus, the produton of a database of vdeo sequenes and assoated MOS, beomes a relevant subjet 3 3

20 sne the subjetve results as well as all type of nformaton related wth them, an be used n future works by people who has nterest n vdeo qualty evaluaton. The vdeo sequenes seleted to be presented n the subjetve tests sesson, were ompressed usng two broadly used ompresson standards, the MPEG- and the H.64/AVC. The MPEG- was hosen sne t s stll wdely used as the format of dgtal televson sgnals. However, H.64/AVC s experenng a wdespread adopton wthn several ountres and overng a wde number of applatons rangng from TV broadast to vdeo for moble deves and IPTV serves. After the subjetve tests havng been arred out, a new NR objetve qualty evaluaton method s proposed and evaluated, the man purpose of whh s to provde qualty sores well orrelated wth the ones resultng from the subjetve tests (MOS). The present thess s strutured n fve hapters. Chapter provdes a general overvew of the subjetve and objetve vdeo qualty evaluaton metrs. Chapter 3 presents an overall desrpton of the ondtons and hoes taken n order to perform the subjetve tests sessons, as well as ther results. In ths hapter, t s also explaned how the observer valdaton should be onduted wth the am of guaranteeng the relablty of the subjetve tests results. In Chapter 4, a new NR objetve vdeo qualty assessment method s proposed and evaluated. In Chapter 5, fnal remarks of the work arred out are presented and future researh dretons are ponted out. 4

21 Chapter Vdeo Qualty Evaluaton Metrs Vdeo Qualty Evaluaton Metrs. Introduton As mentoned n the prevous hapter, there are, n general, two lasses of methods avalable to measure vdeo qualty: the subjetve qualty metrs and the objetve qualty metrs. The subjetve qualty assessment ams to apture, through vdeo s presentatons, the user s perepton of qualty beng the most relable mean of quantfyng vdeo qualty. It s also the most effent method to test the performane of human vson models and objetve qualty assessment metrs. On the other hand, objetve qualty metrs are based purely on mathematal methods. The goal of ths knd of vdeo qualty assessment measurements s to desgn qualty metrs that an predt pereved vdeo qualty automatally. However, pereved vdeo qualty predton s a dffult task, due to the omplexty of the Human Vsual System (HVS). Ths hapter provdes a general overvew of the two lasses of methods mentoned above, namely subjetve and objetve qualty metrs, gvng a 5

22 partular emphass to the man haratersts of them.. Subjetve qualty metrs Ths seton presents an overvew about methodologes, ategores of subjets and rules for performng and desgnng subjetve tests, desrbed and standardzed n the Reommendaton ITU-R BT.500 and n the Reommendaton ITU-T P.90 by the Internatonal Teleommunaton Unon group. The Reommendaton ITU-R BT.500 ( Methodology for the subjetve assessment of the qualty of televson ptures ) [ITU98] s the referene for anyone who has to deal wth qualty of vdeo. In ths reommendaton, dfferent test methods are presented, overng all the possble ases n whh vsual qualty has to be measured. Wth regard to the Reommendaton ITU-T P.90 ( Subjetve vdeo qualty assessment methods for multmeda applatons ) [ITU99], ths reommendaton desrbes non-nteratve subjetve assessment methods for evaluatng the one-way overall vdeo for multmeda applatons suh as vdeoonferenng, storage and retreval applatons, tele-medal applatons, among others. The man dfferene between these two Reommendatons s the fat that the Reommendaton ITU-R BT.500 s foused on subjetve assessment of vdeo qualty for televson ptures,.e., for large vdeo formats; nstead, the Reommendaton ITU-T P.90 s foused on subjetve assessment of vdeo qualty for redued pture formats... Vewng ondtons Dfferent envronments wth dfferent vewng ondtons an affet the expermental results. Speally, there are three fators that must be onsdered when performng the subjetve tests: the lghtng, the ambene nose and the qualty and albraton of the dsplay. Aordng to [ITU99], the test should be arred out under the vewng ondtons presented n Table.. Table.: General vewng ondtons [ITU99] Vewng ondtons Parameters Settngs Vewng dstane -8H Bakground room llumnaton 0 lux Peak lumnane of the sreen Rato of lumnane of natve sreen to peak lumnane Rato of lumnane of bakground behnd the dsplay to peak of lumnane d/m 0,05 0, 6

23 In ths table, H s the mage heght... Seleton of test materals The subjetve qualty assessment results strongly depend on the vdeos sene or sequene ontent seleted to be vewed by the observers. In onsequene, the seleton of test materal must be done arefully. In order to get meanngful and realst tests results, t s mportant that a wde varety of vdeo materal s used durng the tests. Wth regard to test materal that should be nluded n the subjetve tests, t s mportant to norporate rtal materal (e.g., vdeos wth more detaled bakground nstead of only homogeneous bakgrounds). The man reason for that opton s beause t s not possble to extrapolate the test results from materal that s non-rtal, sne t s not possble to guess the observers behavour under other rumstanes. In partular, there are two relevant parameters whh should be taken nto aount when hoosng the test senes: the spatal and the temporal pereptual nformaton of the vdeos. In aordane wth [ITU99], n order to avod borng the observers and to aheve a mnmum relablty of the results, at least four dfferent types of senes n terms of spato-temporal ontent, should be hosen for the sequenes...3 Observers seleton The observers seleton s another mportant task n the subjetve qualty assessment. In order to produe relable and oherent results, n aordane wth [ITU98], at least 5 observers are needed, wth nreasng results auray and onssteny when ths number nreases. Before performng the subjetve tests, the observers should be submtted to ophthalmolog tests n whh they are sreened for auty, olor blndness and other vsual anomales. Experts or non-experts In order to answer to ths subjet, t should be referred that, n general, the publ whh onsume the vdeo materal are ommonly non-expert. In short, non-experts make part of the most representatve target group omparatvely wth the experts group. So, n ths way, t s obvous that the observers n the subjetve qualty assessment sesson should be non-experts. Other reason that supports ths hoe s dretly attahed wth the fat that the non-experts are not onerned wth televson pture as part of ther normal work. Therefore, the non-experts do not have a predetermned way of wathng a vdeo sequene as the experts have. In [ITU98] prelmnary fndngs suggest that non-experts observers may yeld more rtal results wth exposure to hgher qualty transmsson and dsplays tehnologes. Sreenng the observers In the subjetve qualty assessment, the human eye has a speal mportane sne s through 7

24 ths mean that the observers wll assess the vdeo qualty presented n subjetve tests sesson. In ths sense, t s neessary to guarantee that the observers are sreened n aordane wth two man fators before the subjetve tests: olour blndness and vsual auty. The olour blndness, or olour vson defeny, n humans, s the nablty to pereve dfferenes between some of the olours that the most ommon people an dstngush. The vsual auty s the auteness or learness of vson, whh s dependent on the sharpness of the retnal fous wthn the eye and s a quanttatve measure of the ablty to dentfy blak symbols on a whte bakground at a standardzed dstane as the sze of the symbols s vared. The most used standardzed tests to evaluate the olour blndness are the Ishara s test, whle the Snellen Eye Chart s used to assess vsual auty (Fgure.). (a) (b) Fgure.: (a) Ishara Test plate; (b) Snellen Eye Chart The Ishara s test onssts n showng to the observer a set of Ishara s plates, and n askng hm whh number he an see nsde of eah plate. On the other hand, the Snellen Eye Chart s based n the apaty that an observer has to dentfy a set of letters, at a pre-defned dstane from the Chart...4 Vdeo evaluaton sesson Aordng to [ITU98], the qualty assessment sessons should not exeed half an hour, sne f ths does not happen the observer gets tred and, as onsequene, the results wll not be oherent. These evaluaton sessons are dvded n two parts: warm-up sesson and the real test sesson. The warm-up sesson s presented to the observers ntally, before the real test sesson begns, as an be seen n Fgure.. The warm-up phase presents the observer wth some stablzaton presentatons. These vdeo sequenes are shown wth the ntenton to guarantee the observer s opnon stablzaton and to defne n hs/her mnd some vdeo qualty boundares. It s also mportant to add that the data ssued from these presentatons should not be taken nto onsderaton for further analyss. After ths ntal stage has been arred out, the real test sesson s ready to start and the results from ths seond phase are the major goal of all entre subjetve qualty evaluaton sessons. 8

25 Fgure.: Test sesson struture It s from ths seond stage, that the observers results wll be taken nto aount n order to alulate the Mean Opnon Sore (MOS). Durng ths phase t s presented to the observers a set of arefully seleted vdeo sequenes. The tests an be ether sngle or double presentaton: f the referene vdeo and test vdeo are presented only one ths presentaton s named as sngle; by ontrast, f the referene vdeo and test vdeo are presented twe, the presentaton s named as double. Ths opton wll be nfluened by the test method adopted to perform the subjetve tests. Wth regard to the tral struture, and dependng on the type of vdeo qualty assessment method used, the referene vdeo an be presented at frst plae and the test vdeo at seond plae (whh an be degraded or not relatvely to the referene vdeo) or, on ontrary, the test vdeo an be presented at frst plae and the referene vdeo at seond plae. Durng the presentaton, the vdeo sequenes should be n a random order. In ontrast, the test ondton order should be arranged so that any effets on the gradng of tredness or adaptaton are balaned out from sesson to sesson ([ITU98]). Wth the purpose of measurng the observer s oherene some manuals reommend to repeat some sequenes presentatons...5 Useful nformaton for the assessment Durng the phase that preedes the subjetve qualty evaluaton sesson, the observers should be arefully ntrodued to the method of assessment. Questons as what s the test about? and what s the gradng sale?, as well as the sequene tme, should be well explaned n order that the tests results are not nfluened by any msunderstandng. Also, as t was already mentoned n the prevous seton, before startng wth the real subjetve sesson, tranng sequenes representng the range and the knd of mparment to be seen durng the sesson should be shown to the observer...6 Vdeo qualty assessment methods Durng the last years a number of subjetve testng methodologes were proposed, some of them were standardzed n [ITU98] and n [ITU99], namely: The Double Stmulus Imparment Sale (DSIS) or Degradaton Category Ratng (DCR); The Comparson Sale Method or Par Comparson method (PC); 9

26 The Sngle Stmulus Method (SS) or Absolute Category Ratng (ACR); The Sngle Stmulus Contnuous Qualty Evaluaton (SSCQE); The Double Stmulus Contnuous Qualty Sale (DSCQS); The Smultaneous Double Stmulus for Contnuous Evaluaton (SDSCE). In order to gve an overvew about the vdeo qualty assessment standards depted above, n the next sub-setons these methods are desrbed gvng partular attenton to the methodologes and tral struture followed by them...6. Double Stmulus Imparment Sale (DSIS) or Degradaton Category Ratng (DCR) The Double Stmulus Imparment Sale (DSIS) [ITU98] s a very useful tool for evaluatng learly vsble mparments, suh as blokness, blurrng and rngng, whh are usually aused by the enodng proess. In the ontext of multmeda applatons, ths method s equvalent to the Degradaton Category Ratng (DCR) method desrbed n [ITU99]. Furthermore, the DCR method s a key method for the assessment of televson ptures whose typal qualty represents the hghest qualty levels found n vdeotelephony and vdeoonferenng serves [ITU99]. DSIS s not reommended for the qualty evaluaton of vdeo transmsson over paket networks lke the Internet. The reason for that, aordng to Mras et al. [Mra0], s beause of paket networks non-determnst behavour and the bursty nature of enoded vdeo. Ths means that, from the user s pont of vew, pereved qualty an vary sgnfantly over tme. In ths perspetve, Pearson et al. [Pear99] dsussed several hgher-order effets that nfluene users' qualty ratngs when assessng vdeo sequenes of extended duraton. In order to redue these types of effets on users' qualty assessment, what s needed s a method able to dynamally apture user's opnon as the underlyng network ondtons or vsual ontent omplexty hange. In short, the DSIS method should be used when t s mportant to hek the smlarty of the test ondton wth regard to the referene ondton; n addton, t should also be used for hgh qualty system evaluaton n the ontext of multmeda ommunatons. Methodology and Tral Struture The DSIS method s approprate for stuatons where the tests span the full range of mparments responsble for all vsble degradaton n the mage. The observer s presented wth vdeo sequenes organzed n pars: the frst to be dsplayed s alled the referene sequene whle the seond s alled the test or mpared sequene [GGC0]. The referene s the orgnal, undstorted soure sequene whle the mpared sequene s a dstorted verson of the referene (for nstane, the result of lossy enodng). 0

27 As for the number of presentatons of eah sequene par durng a test sesson, two varants are possble: varant I: eah par referene-test s presented a sngle tme, as s shown n Fgure.3.a). Ths means that the observer has only one opportunty to vew and to analyse the referene and test sequenes; varant II: eah par s presented two tmes, as s shown n fgure Fgure.3.b). In ontrast wth varant I, the observer has two hanes to wath and to analyse the referene and test sequenes, before dong hs judgement. When redued ptures formats are used n ths assessng method, suh as CIF, QCIF or SIF 4, t ould be useful to dsplay the referene and test ondtons smultaneously on the same montor. Fgure.3: Double Stmulus Imparment Test Tral Struture [ITU98]: a) DSIS I;b) DSIS II 4 CIF Common Intermedate Format, typally wth a vdeo spatal resoluton. QCIF Quarter Common Intermedate Format, typally wth a vdeo spatal resoluton. SIF Soure Input Format, typally wth a spatal vdeo resoluton of or

28 Based on those varants, the DSIS method s known as DSIS I or DSIS II, whenever the method orresponds to the varant I or varant II, respetvely. After the sequenes have been presented, the observer s asked to vote on the mpared sequene, but keepng n mnd the frst sequene as referene, n eah tral (Fgure.3). The DSIS s a method whh makes use of fve grade mparment sale. The qualty assessment grades on ths dsrete mparment sale are [ITU98]: Impereptble: n ths ase, the test sequene showed to the observer does not seem to be dfferent from the referene sequene; Pereptble, But Not Annoyng: f the observer hoose ths grade, t s probably beause he has noted some dfferenes between the test and referene sequenes, but those dfferenes dd not bother hm; Slghtly Annoyng: the observer sees some degradaton n the test sequene, and that degradaton bothers hm; Annoyng; n ths stuaton, the observer s hoe reflets a huge degradaton n the test sequene relatvely to the referene. Ths type of degradaton bothers so muh the observer, that he an stop usng ths materal; Very Annoyng: n ths ase, the observer s radal on hs opnon,.e., the observer would not wath ths knd of materal under no rumstanes. It s mportant to menton that ths type of evaluaton an be performed usng tehnologal means (suh as omputers) or tradtonal ways (lke paper and pen), as shown n Fgure.4.(a) and n Fgure.4.(b), respetvely. The mean opnon sores (MOS) are omputed at the end of the sesson, based n the mage qualty assessment results gven by all observers. (a) (b) Fgure.4: Fve pont Imparment Ratng Sale: (a) usng tehnologal means; (b) usng tradtonal ways

29 ..6. Comparson Sale Method or Par Comparson method (PC) The Comparson Sale Method performs a dret head-to-head omparson between two systems (A and B).The purpose of ths omparson s to know whh system s the best and how muh t s better than the other. In aordane wth the [ITU99], ths method s also addressed to as Par Comparson method n the ontext of multmeda applatons. Methodology and Tral Struture The tral struture of ths method has the partularty of beng blnd to the observer,.e., the referene and test sequenes that are shown to the observer are not dsplayed n a pre-defned order [WR06]. Smlarly to other methods, there s the opton of presentng eah sequene par one or twe, as depted n Fgure.5.(a) and n Fgure.5.(b). As for the DSIS method, when redued resolutons are used (e.g. CIF, QCIF or SIF), t ould be useful to dsplay eah par of sequenes smultaneously on the same montor [ITU99]. Fgure.5: Tral struture for Comparson Test for: a) Sngle presentaton; b) Double presentaton Wth respet to the evaluaton sale, n ths method the vewers are nstruted to assess the dfferene between the frst and seond presentatons usng a 0 m horzontal sale smlar to what s depted n Fgure.6. The omparson sale s a ontnuous sale that has three adjetve markers: A s muh better, A=B, B s muh better. Fgure.6: Comparson Ratng Sale. 3

30 ..6.3 Sngle Stmulus Method (SS) or Absolute Category Ratng (ACR) Aordng to [ITU98], the Sngle Stmulus s a method where the test sequenes are presented one at a tme and are rated ndependently on a ategory sale. Usng the termnology of [ITU99], the Sngle Stmulus method s also referred to as Absolute Category Ratng (ACR). Ths method allows nreasng the observers tme effeny, sne t s fast and easy to mplement. Methodology and Tral Struture The seres of assessment trals should be presented n a random order for eah observer. Smlarly to the DSIS method and n aordane wth [ITU98], t s possble to dstngush two varants based on the presentatons struture,.e., varant I and varant II. A typal Sngle Stmulus tral struture s represented n Fgure.7. Fgure.7: Sngle Stmulus Tral Struture The subjet usually knows the order n whh the referene and test vdeo sequenes appear n eah tral. If the order referene-test s also randomzed for eah tral, labels A and B an be used to dentfy the referene and the test vdeo sequenes. Wth regard to the evaluaton sale, the vdeo qualty assessment s performed usng one out of four possble sorng sales: a fve grade sale, a nne grade sale, an eleven grade sale or a ontnuous sale wth no numbers. The fve grade sale, represented n Table., s the most used one. Ths numeral sale allows the observer to assgn a number to eah dsplayed vdeo sequene that reflets ts judgement based on the mage qualty level. Table.: Fve grade sale Gradng Value Estmated Qualty 5 Exellent 4 Good 3 Far Poor Bad 4

31 However, f more dsrmnatng results are desred, a nne grade sale or even an eleven grade sale should be used. Both are varants of the fve grade sale, wth addtonal ponts for hgher dsrmnatve power. Fnally, the last sale whh an be used n the Sngle Stmulus method s the ontnuous sale. Ths sale enables a non-ategoral judgment, for the qualty of eah mage or vdeo sequene. In order to perform hs judgment, the observer wll mark a pont on a lne segment that represents the qualty sale n whh the lmts of t represent the worst and the best qualty. For referene, the sale usually nludes addtonal qualty labels at ntermedate ponts Sngle Stmulus Contnuous Qualty Evaluaton (SSCQE) One of the ontnuous evaluaton methods s the Sngle Stmulus Contnuous Qualty Evaluaton (SSCQE). The SSCQE s a method orented to the qualty assessment n dgtal televson systems. Basally, t onssts of measurng the qualty of a vdeo sequene along the tme, thus the observers are ontnuously provdng ther judgment of the vdeo qualty on a lnear sale. Typally, the assessment materal used on ths method onssts of vdeo sequenes that ontan senedependent and tme-varyng mparments. In the ontext of qualty montorng applatons, ths method yelds more representatve qualty estmates than the prevous ones. Methodology and Tral Struture The SSCQE methodology belongs to a lass of methods where a seres of vdeo sequenes are presented only one to the observer. Ths ontnuous assessment method s the best way to measure the qualty varaton of a sngle vdeo lp [Bst05]. In order to take nto aount the temporal varatons of qualty, eah vdeo sequene should be longer than 0 seonds. Smlarly to other methods that also use ontnuous ratng sales, ths method allows the observers to assess both audo and vdeo n vdeo-onferenng applatons. In order to keep a hgh level of onentraton and attenton from the observers and wth the am to redue fatgue, the SSCQE method adves the ntroduton of breaks durng eah test sesson. To mnmze the ontextual effets, the order of the test sequenes n the SSCQE s randomzed at the lp level, suh that every subjet wll vew the test lps n a dfferent order. Wth respet to the SSCQE assessment sale, n ths method eah vewer s opnon s regstered twe a seond by an eletron handset onneted to a omputer [Bst05]. The handset s basally a slder mehansm wth an assoated qualty sale, as an be observed n Fgure.8. Hene, the subjet an move the slder to any pont over the sale, refletng hs mpresson of qualty at eah tme nstant. The sldng sale s about 0 m long and s dvded n fve qualty levels.these deves are onneted to a omputer where the ontnuous ratng of the vdeo materal s reorded. It should be noted that eah qualty label represented n the ontnuous sale orresponds to a numer nterval. For nstane, Exellent (00-80), Good (79-60), Far (59-40), Poor (39-0) and Bad (9-0). Ths assoaton wll allow to ompute the Mean Opnon Sores (MOS) durng the analyss phase. 5

32 Fgure.8: Automat votng deve Slder [WP07]..6.5 Double Stmulus Contnuous Qualty Sale (DSCQS) The Double Stmulus Contnuous Qualty Sale (DSCQS) method has been used for performane evaluaton of the dgtal HDTV (Hgh Defnton Televson) Grand Allane System, whh was the bass for the North Ameran standards for dgtal TV broadastng. Ths method s espeally useful when t s not possble to span the full range of qualty stmulus [ITU98]. Aordng to [WP07], the DSCQS method s onsdered aurate and does not show sgnfant senstvty to ontext effets. Context effets our when subjetve sores gven by the observer are nfluened by the severty and orderng of mparments present n the test materal. The DSCQS methodology deals wth these by usng an alternate way of presentng the vdeo sequenes. Methodology and Tral Struture In the DSCQS method, the dsplayng order of the referene and the test sequenes s randomzed (the referene and test presentatons are blnd to the subjet). Thus, the subjet does not know whether the frst or the seond presentaton s the referene or the test sequene. The observer s then asked to evaluate both sequenes of mages. The DSCQS method an be dvded n two varants: Varant I: Eah observer, who s normally alone, s let to swth between the two sequenes, A and B, one of whh s always the referene and the other s the test. Varant II: In ths varant, t s shown two ondtons, A and B, onseutvely, to multple observers, one of whh s always the referene and the other s the test. The referene and test sequenes are shown twe to the observer ( double methodology). After havng fully wathed both sets of presentatons, he s nstruted to rate the sequenes, as represented n Fgure.9. 6

33 Fgure.9: Double Stmulus Contnuous Qualty Sale Tral Struture In the DSCQS method the vewers are nstruted to assess the qualty of both presentatons usng the double vertal sale shown n Fgure.0. The sale s dvded n fve equal ntervals representng the qualty levels. The paper verson of the sale should have 0 m heght. Fgure.0: Double Stmulus Contnuous Qualty Sale (parallelsm wth DSIS s qualty adjetves) After the assessment sesson, the pars of qualty sores (referene and test) are onverted to normalzed sores n the range 0 to 00. These sores are spread aordng to Table.3. Table.3: Typal qualty assessment sale for DSCQS and SDSCE methods Pereptual Qualty Equvalent Grade Qualty Exellent Good Far Poor 39-0 Bad 9-0 After normalzaton, the dfferenes between the sores gven to the referene and to the test sequenes are omputed for eah par. It s worth to menton that the use of a ontnuous sale has the advantage of redung the amount of quantzaton error n the observer s responses. In ths sense, the DSCQS method s preferred when the qualty of the referene and test sequenes s smlar. 7

34 ..6.6 The Smultaneous Double Stmulus for Contnuous Evaluaton (SDSCE) Besdes the other vdeo qualty evaluaton methods desrbed prevously, the Smultaneous Double Stmulus for Contnuous Evaluaton (SDCQE) method s also a standardzed and nternatonally aepted system for mage and vdeo qualty assessment tests. Ths vdeo qualty assessment method, proposed by MPEG, s sutable to evaluate the effet of sparse mparments, suh as transmsson errors, on the fdelty of vsual nformaton [ITU99]. The SDCQE method, whh has been derved from the SSCQE method, dffers from that one by makng slght devatons n what regards the way of presentng the mages to the observers and onernng the ratng sale used by them to perform the vdeo qualty assessment. Aordng to [ITU98], the SDSCE an be sutably appled to all those ases where fdelty of vsual nformaton, affeted by tme-varyng degradaton, has to be evaluated. Methodology and Tral Struture The SDSCE method onssts on assessng two lps smultaneously and ontnuously. The referene and mpared lps are dsplayed n parallel postons, as represented n Fgure., usng one or two dsplays (dependng on the vdeo resoluton). Sne the two lps are presented smultaneously, the observer wll have to shft hs attenton between the rght and the left presentatons, whh s a drawbak for ths method. Fgure.: SDSCE prnple [ITU08] The observers are asked to ontnuously judge the fdelty of the vdeo nformaton of the mpared sequene wth respet to the referene, by movng a slder on a handset-votng deve. The subjets are aware of whh sequene s the referene and whh sequene s the test ondton. The SDSCE, lke other ontnuous methods prevously desrbed, provdes to the subjets a ontnuous sale. The grade gven by them wll measure ndretly the level of the mparment n the test ondton omparatvely to the referene ondton qualty. One agan, the assessment sale assoated to ths type of method s dvded nto fve equal ntervals. These fve ntervals wll orrespond to fve dfferent qualtatve adjetves, presented n Table.3. 8

35 Smlarly to the SSCQE method, the SDSCE method provdes to the vewers a deve named slder. It wll be through ths automat votng deve that the observers wll gve ther pereptual qualty opnon Man Charatersts of Subjetve Methods Based on the desrpton of the dfferent subjetve methods, t s possble to summarze, as shown n Table.4, the man features of them. Table.4: Vdeo qualty assessment methods man features Parameter DSIS / DCR DSCQS SSCQE SDSCE Seleton of test methods - To measure the robustness of systems (DSIS); - When s testng the fdelty of transmsson (dstorted vdeo) wth respet to the referene sgnal (DCR); -To measure the qualty of systems relatve to a referene; -To measure the qualty of a stereosop mage odng; - To measure vdeo qualty n dgtal televson systems; - The best way to measure the qualty varaton of a sngle vdeo lp; - To measure the fdelty between two mpared vdeo sequenes; - To ompare dfferent error reslene tools; Explt referene Yes No No Yes Hdden referene No Yes No No Sale Very annoyng to mpereptble Bad to exellent Bad to exellent Bad to exellent Sequene length 0s 0s 5mn 0s Two smultaneous stmul Yes () Yes No Yes Varant I: one Presentaton of test Twe shown Varant II: twe shown materal onseutvely onseutvely One One Vdeos per tral Votng Contnuous qualty evaluaton along tme Dsplay Only test sequene No All (manly TV) Test sequene and referene No All (manly TV,DLP () ) Test sequenes Yes (movng the slder n a ontnuous way) All (manly TV) Dfferene between the test sequene and the referene smultaneously shown Yes (movng the slder n a ontnuous way) All (manly TV) () Aordng to ITU-T P90, t s possble to use a smultaneous presentaton when usng a redued pture format, lke CIF, QCIF,SIF (for DCR method); () Dgtal Lght Proessng whh represents a tehnology used n projetors and vdeo projetors. In aordane wth the methods desrpton provded n ths hapter, t s not possble to defntvely reommend one method over the others, sne all have strengths and weaknesses. Therefore, the expermenter who s leadng the test sesson should selet the method whh he thnks 9

36 that t s more adequate for the rumstanes..3 Objetve qualty metrs Ths seton overvews the man haratersts of objetve qualty metrs and presents some state-of-the-art metrs to perform ths type of vdeo evaluaton..3. Classfaton of objetve metrs The objetve qualty metr should allow to obtan a good predton of the vdeo qualty sores that human observers would gve to that vdeo sequene. These vdeo qualty metrs an provde qualty ontrol of the ompressed vdeo and more generally Qualty of Serve (QoS) n vdeo ommunatons. They an be ategorzed n three lasses, based on the amount of nformaton about the referene vdeo requred and avalable to estmate the vdeo qualty: Full Referene metrs (FR): These metrs requre the orgnal vdeo and the dstorted vdeo; Redued Referene metrs (RR): These metrs requre the desrpton of some parameters from the orgnal vdeo and the dstorted vdeo; No Referene metrs (NR): In ontrast to FR and RR, these knds of metrs only need the dstorted vdeo. The FR metrs are the most studed and developed objetve metrs. Ths type of metrs may have test mplementatons and, at the same tme, may provde good results respetvely to the fdelty of vdeo. Typally they are based n a frame-by-frame omparson between the referene and the dstorted vdeo sequenes, requrng an aurate spatal and also temporal algnment of the two vdeos, whh may be dffult to aheve n pratse. Ths spatal/temporal algnment requrement of the two vdeos s mportant sne every pxel n every frame of the dstorted vdeo must be mathed wth ts ounterpart n the referene vdeo, n order to allow a perfet frame-by-frame omparson among them. In a NR objetve metr senaro, the qualty sores predton s obtaned through the nformaton avalable n the reever sde only. Contrarly to FR metrs, n NR metrs there s no need to enfore a spatal and temporal algnment of the referene and dstorted vdeos sne no frame-byframe omparson of both vdeos s performed. The major drawbak of ths knd of metrs, n aordane wth [Wnk07], s related wth the fat that the NR metrs les n tellng dstortons apart from ontent, a dstnton humans are typally able to make from experene. In the ase of the NR metrs, t s neessary to begn by makng assumptons about two mportant tops, the vdeo ontent and/or the dstortons of nterest. As result 0

37 of these suppostons, the rse of the rsk of onfusng atual ontent wth dstortons s a realty. In lterature, a lmted number of NR metrs has been proposed. However, reently ths top has attrated a great deal of attenton. As example of that, s the fat that the VQEG onsders the standardzaton of NR vdeo qualty evaluaton methods as one of ts future workng dreton. Proposed NR algorthms falls typally n two ategores of methods: those that evaluate some spef odng artefats, suh as blok effet n blok-based DCT ompresson methods, edge dsontnutes, et; those that estmate pxels dstorton and weght those dstortons aordng to some human vsual model. In a RR metr senaro ertan features or physal measures are extrated from the orgnal vdeo and then transmtted to the reever as sde nformaton n order to help evaluatng the qualty of the vdeo. Thus, ths lass of metrs wll requre addtonal bandwdth (or addtonal hannel) to send the sde nformaton. Smlarly to the FR metrs, the RR metrs may also requre a spatal and temporal algnment between the sde nformaton and the dstorted vdeos; however, ths proess s normally less demandng than n the FR metrs, sne n ths ase only the extrated features from the referene vdeo need to be algned. The development of RR as well as NR metrs systems has beome a prorty matter to the vdeo qualty ommunty, sne n the ontext of vdeo dstrbuton senaro t s desrable to perform qualty evaluaton at the reevng sde wth low-level of nformaton or speally wthout aessng any nformaton from the orgnal meda data..3. Objetve assessment approahes Aordng to [Wnk07], the measurement of the vdeo dstortons n a vdeo ommunaton system an be performed n two ways: Data metrs: In order to measure the amount of dstorton ntrodued by the apture, ompresson and transmsson proesses, these metrs take nto aount only the sgnal relablty wthout onsderng the ontent of the vdeo under analyss. Pture metrs: Ths dstorton measurement s foused on the ontent of the vdeo under analyss,.e., ths approah allows quantfyng the effet of dstortons and ontent on pereved qualty. In ths ase, these metrs are loser to the human pereved qualty than the Data metrs method. The most relevant example of a smple data metr s the MSE, or ts equvalent PSNR. Although t does not orrelate well wth the subjetve evaluaton, t s wdely used n vdeo qualty evaluaton manly due to ts omputatonal smplty. A good example of the MSE lmtaton s the fat that two vdeos, quanttatvely wth the same MSE values, an n fat have dfferent subjetve sores. The MSE does not make any dstnton on the dfferent types of artefats,.e., MSE treats all errors n the same way, regardless of ts nfluene on the vdeo s qualty.

38 However, the MSE/PSNR has a good performane when omparng two ompressed versons of the same orgnal vdeo sequene, usng the same enoder. Ths phenomenon ours beause the ompressed vdeos are beng enoded wth the same dstorton haratersts. Other advantage of usng the MSE/PSNR, aordng to Brandão et al. [BQ08b], s the fat that ths vdeo qualty metr an be used as a NR qualty metr,.e., t s possble to produe aurate PSNR estmates wthout the need of the orgnal data. Formally, the MSE s gven by MSE = M N [ f (, j) F(, j) ] = j= M N (. ) where, f (, j ) s the orgnal vdeo omponent (lumnane or hromnane) at pxel (, ) F (, j ) s the dstorted vdeo omponent at pxel (, ) j ; j ; M N s the pture wdth; s the pture heght. The PSNR s derved by settng the MSE n relaton to the maxmum possble value of lumnane (for a typal 8-bt value ths s 8 = 55), and s usually expressed n logarthm unts through: PSNR = 0log 55 MSE 0, [db] (. ) Pture metrs are the result of muh effort made n order to develop better vsual qualty metrs that quantfy the effets of dstortons and ontent on pereved qualty. As a onsequene of ths effort, the Pture metrs an be lassfed n two groups [Wnk07]: a vson modellng approah; an engneerng approah. The vson modellng approah s based partularly on HVS,.e., these knds of metrs try to nlude human vson haratersts whh seem to be relevant to pture qualty, lke ontrast senstvty, olor perepton, applyng models and data from psyhophysal experments. The engneerng approah, nstead of fousng n the HVS as t s done n the vson modellng approah, reles on the extraton of ertan features or artfats n the vdeo under analyss. Ths type of approah, whh has ganed popularty n reent years, fouses on the strength of these extrated

39 features and then takes them nto aount n order to estmate the overall qualty of t. The extrated features are mage strutural elements or spef dstortons, ntrodued by a partular vdeo proessng step, ompresson tehnology, or transmsson lnk. An example of the engneerng approah, whh has ganed hgh popularty n reent years, s the Strutural Smlarty Index Method (SSIM) [WBSS04]. In ths metr, vdeo degradatons are onsdered as pereved strutural nformaton loss nstead of pereved errors. The SSIM onssts n omputng from the orgnal and from the dstorted vdeo, three measurements: lumnane, ontrast and strutural dstortons. These measurements are then separately ompared. At the end, the omparson results are ombned to yeld an overall smlarty measure. Fgure. desrbes brefly the SSIM s proess. The SSIM s measurement system has been onstruted on the assumpton that vdeo degradaton s often aused by the loss of underlyng strutured nformaton. One of the strengths of ths vdeo qualty assessment metr s the fat that ths metr has showed to perform well for some artfats whh are not dretly related to the ompresson step, suh as added nose. On the other hand, the SSIM an also adapt to artfats whh are dretly related to low btrate vdeo ompresson (suh as blokng effet) and provde as well pereptually onsstent qualty predtons. Fgure.: SSIM s measurement system Although the SSIM presents a relatve smplty, ths metr behaved qute well on the VQEG FR-TV Phase I database. Besdes the SSIM method, there are other popular strutural nformaton based metrs, suh as the metr developed by Wolf and Pnson, named as Vdeo Qualty Metr (VQM). The VQM metr, smlarly to the SSIM extrats, from the dstorted vdeo, a restrt set of features, whh are seleted emprally and arefully from a group of possble features. After that seleton phase, those features 3

40 are then ompared analogously wth the features from the referene vdeo. Aordng to [Wnk07], the VQM was among the best metrs n the VQEG FR-TV Phase II evaluaton. Another example of ths type of approah s the metr desgned by Hekstra et al. [HBL0], named Pereptual Vdeo Qualty Measure (PVQM). Ths vdeo qualty assessment uses a lnear ombnaton of three partular features, whh are the loss of edge sharpness, the olor error normalzed by the saturaton, as well as the temporal varablty from the referene vdeo. In aordane wth [Wnk07], the PVQM was also one of the best metrs n the VQEG FR-TV Phase I test. All the above mentoned metrs SSIM, VQM and PVQM belong to the lass of FR metrs. Re. ITU-T J.47 [ITU08b] provdes four FR vdeo qualty estmaton methods: NTT Full Referene Method (developed n Japan); OPTICOM s Vdeo Qualty Method (developed n Germany); Psytehns Full Referene Method (developed n Unted Kngdom); Yonse Unversty Full Referene Method (developed n Korea). The NTT full referene predton model, estmates subjetve vdeo qualty by an algnment proess and a vdeo qualty algorthm that reflets human vsual haratersts. The seond vdeo qualty model, developed n Germany, also known as Pereptual Evaluaton of Vdeo Qualty (PEVQ) model, s a very robust one whh was desgned to predt the effets of transmsson mparments on the vdeo qualty as pereved by a human subjet. The man targets of ths model are moble applatons as well as multmeda applatons. Regardng the Psytehns full-referene vdeo qualty assessment algorthm, t onssts on the dentfaton of pereptually relevant boundares and n the nluson of a model of the human vsual system. These two elements allow the model to dentfy and quantfy errors pereved by human vewers. As a result, the Psytehns vdeo qualty assessment model produes (objetve) qualty predtons that orrelate hghly wth human (subjetve) qualty judgment. It s observed that the human vsual system s senstve to degradaton around the edges. Furthermore t s observed that vdeo ompresson algorthms tend to produe more artefats around edge areas than on the remanng ones. Based on ths observaton, the Yonse Unversty full referene model provdes an objetve vdeo qualty measurement method that measures degradaton around the edges. In ths model, an edge deteton algorthm s frst appled to the soure vdeo sequene to loate the edge areas. Then, the degradaton of those edge areas s measured by omputng the mean squared error. From ths mean squared error, the Edge PSNR (EPSNR) s omputed. Furthermore, the model omputes two addtonal features whh are ombned wth the EPSNR to produe the fnal vdeo qualty metr. The full referene method s generally aepted as the model that provdes the best auray for pereptual pture qualty measurements. However t s also known that ths method s only sutable when the orgnal (referene) and the dstorted vdeo are totally avalable at the reever sde, and onsequently ths type of arhteture are not adequate for pratal meda dstrbuton senaros. 4

41 In the ontext of multmeda dstrbuton senaros, t s desrable to trak meda qualty at the reevers. Ths ould enable new serves, suh as users payng proportonally to the qualty they get, and new server possbltes, suh as adjustment of streamng parameters as a funton of the pereved qualty. Thus, n order to assess vdeo qualty wthout requrng the orgnal vdeo data, Redued Referene and No Referene methods (NR) are requred. As t was mentoned n sub-seton.3., the RR measurement method an be used when features extrated from the unmpared referene vdeo sgnal are readly avalable at the reever sde. Based on that, Re. ITU-T J.46 [ITU08a] proposes some RR models, based on the measurement of the edges degradaton. Aordng to these models, an edge deteton algorthm s frst appled to the soure vdeo sequene to loate the edge pxels and then, the degradaton of those edge pxels s measured by omputng the mean squared error (.),.e., the amount by whh the edge pxels from the soure vdeo sequene dffers from the ones loated on the dstorted vdeo s quantzed. After, the edge PSNR s omputed. Dependng on the nature of vdeos and ompresson algorthms, a dfferent edge deteton algorthm an be hosen [ITU08a]. In [OD07], another RR vdeo qualty metr for AVC/H.64 was proposed. In ths ase, the RR model evaluates a set of features suh as blur or blokng and ombnes these measurements wth few addtonal data (extrated from the orgnal vdeo and transmtted as sde nformaton) nto one qualty sore usng multvarate data analyss. In what onerns the NR metrs, few approahes were proposed n lterature and none has been standardzed yet. In fat, as t was mentoned before, as future work VQEG onsders the standardzaton of NR vdeo qualty evaluaton methods as a prorty matter. Reently, Brandão [BQ08a] presented an approah that an be used for mage qualty evaluaton wthout requrng any knowledge about the orgnal sgnal, thus belongng to the NR mage qualty metrs ategory. Qualty sores rely on statstal propertes of the orgnal, blok-based, DCT (dsrete osne transform) oeffent data that are estmated from the reeved (and quantzed) DCT oeffents, and on the pereptual haratersts of the human eye. The man goal was to estmate dstorton errors and orrespondng pereptual weghts, n suh a way that qualty sores gven to the dstorted mages resemble the pereptual metr proposed by Watson n [Wats93]. The method proposed n [BQ08a] for JPEG enoded mages was partally extended to H.64/AVC enoded vdeo n [BQ08b]. 5

42 6

43 Chapter 3 Subjetve Qualty Evaluaton 3 Subjetve Qualty Evaluaton 3. Introduton The subjetve qualty assessment s a human perepton based method that uses strutured expermental desgns as well as human partpants. The goal of these partpants s to assess the vdeo qualty presented durng the subjetve qualty evaluaton sessons. In ths hapter, the Mean Opnon Sore (MOS) of a number of vdeo sequenes s obtaned. The MOS s ntally omputed takng nto onsderaton all the observers present n the subjetve tests. In order to guarantee the oherene and the onssteny of the results provded by the subjetve tests, a statstal analyss s followed wth the am of valdate the observers opnons. After the observer s valdaton has been 7

44 performed the fnal MOS values are omputed. By ontrast wth MOS ntally obtaned, the new MOS s alulated takng nto aount only the oherent observers. At the end of ths hapter, the test results are presented as well as the graphs whh summarze these results. 3. Subjetve assessment Ths seton presents the test methodology used to ondut the subjetve tests and the man reasons to follow t. Takng nto aount sub-seton..6, where subjetve vdeo qualty evaluaton methods were desrbed, as well as [ITU99], n ths work the followed method was the Degradaton Category Ratng (DCR), also known as Double Stmulus Imparment Sale (DSIS). The man reason to hoose the DCR was the fat that t s reommended to assess redued vdeo formats, suh as CIF, QCIF or SIF. Furthermore, these redued vdeo formats are sutable for vdeo applatons n 3G wreless networks and for vdeo streamng haraterzed by low resolutons, and low btrates; for nstane, the CIF and SIF resolutons are ommonly used for data-ards and palmtops (PDA), whle the QCIF s generally used for ell phones. 3.3 Vewng and test ondtons As mentoned n hapter, there are two essental elements for ondutng the subjetve qualty evaluaton sessons properly: the envronmental vewng ondtons and the test ondtons. The man test ondtons are: Maxmum test duraton per sesson: mnutes Maxmum number of observers per sesson: Total number of observers n the subjetve tests sesson: Vewng dstane: 8 x of the pture heght shown n the sreen (H) In Fgure 3. the testng room used n ths dssertaton s shematally presented where the parameter H ndates the heght of the vdeo shown on the sreen. 8

45 Fgure 3.: Testng room Table 3. presents some aspets related to the dsplay and room haratersts used n ths dssertaton. Table 3.: Dsplay and Room s ondtons Dsplay and Room s ondtons Parameters Settngs Heght of the pture shown n the sreen (H) 8 m Vewng dstane 64 m Bakground room llumnaton 3,45 lux Peak lumnane of the sreen 95,8 lux Lumnane of natve sreen,3 lux Lumnane of bakground behnd the dsplay 0,5 lux Rato of lumnane of natve sreen to peak lumnane 0,03 Rato of lumnane of bakground behnd the dsplay to peak of lumnane 0,4 Based on Table 3., t s possble to onlude that the values aheved for our dsplay and room s ondtons are wthn the values reommended n [ITU99] (see Table.). 9

46 3.4 Charaterzaton of the test sequenes When seletng the vdeo sequenes to be used n the tests, t s mportant to take nto aount the fators that most nfluene the HVS. Aordng to [RNR07] the human vsual perepton of vdeo ontent s determned by the vdeo spatal nformaton, as well as by the type, dreton and speed of movement, or temporal atvty. Sne a small number of test sequenes wll be used n the test sessons, t s mportant to hoose a set of sequenes that span a large range of possble spatal and temporal nformaton. In other words, the hosen sequenes should be well representatve of the vdeo sequenes that an be enountered n the envsaged applaton. Hene, n order to hoose a set of vdeo sequenes, the spatal and temporal atvtes of eah vdeo sequene has to be omputed. The lterature provdes several dfferent methods of measurng these atvtes. In ths work, the methods reommended n [ITU99] have been used. Spatal atvty: The spatal atvty measurement uses two flters that work ndependently of eah other. One flter s responsble to ompute horzontal pxel dfferenes, or horzontal pture gradent, as shown n Fgure 3..(a), whle the other omputes vertal pxels dfferenes, or vertal pture gradent, as shown n Fgure 3..(b). These two flters are alled Sobel flters. Mathematally speakng, the Sobel flterng onssts n onvolvng the two 3x3 kernels presented n Fgure 3. wth eah frame of the vdeo sequene. In order to obtan for eah pxel a sngle measure, the gradent norm (the square root of the sum of the vertal and horzontal gradent squares) s omputed. Then, the standard devaton of t s obtaned n a frame bass. Ths proess s repeated for eah frame of the vdeo sequene and results n a tme seres of spatal nformaton of the sene. In order to aheve a global value for spatal atvty, the maxmum value n the tme seres s seleted wth the purpose of representng the spatal nformaton ontent of the sene. (a) (b) Fgure 3.: Sobel flters. (a) Sobel flter responsble for detetng horzontal pxel dfferenes; (b) Sobel flter responsble for detetng vertal pxel dfferenes [WP99] Fgure 3.3 shows the resultng gradent norm for two frames of the vdeo sequenes Stefan and Football. 30

47 (a) (b) () (d) Fgure 3.3: (a) and () Orgnal vdeo frames; (b) and (d) Correspondng gradent norm mages In Fgures 3.3.(b), and (d), hgher level of lumnane values orrespond to hgher values of gradent norm (spatal atvty). Temporal atvty: Aordng to [ITU99], a temporal atvty measure an be obtaned omputng the dfferene, pxel by pxel, between eah two suessve frames of the vdeo sequene. Ths proess s repeated for all vdeo frames. After ths proedure has been arred out, the standard devaton of the frames dfferenes s omputed. Smlarly to what happens n the spatal atvty, the global temporal atvty value s omputed as the maxmum of these standard devatons. Fgure 3.4 presents, n the rght sde, two onseutve frames of the orgnal vdeo and, n the left sde, the resultng dfferene between the two orgnal frames. t Fgure 3.4: Temporal atvty measurement proess n a vdeo sequene 3

48 Aordng to Fgure 3.4, two suessve frames are frst of all ompared pxel by pxel, n order to measure the absolute dfferene between them. After ths omparson proess has been arred out a lumnane frame s reated whh represents the temporal atvty exstng between the two ompared frames; the hgher the temporal atvty varaton between the two ompared frames, the hgher wll be the lumnane ontent of the frame dfferene. 3.5 Vdeo sequenes seleton Aordng to what was desrbed n the seton 3., the spatal-temporal nformaton was omputed for a set of vdeo sequenes, ommonly used by the vdeo odng ommunty, n CIF format. The results are presented n Fgure 3.5. Fgure 3.5: Spatal-temporal atvty of a vdeo sequene set (CIF format) However, there s one aspet that should be taken nto aount. Ths aspet s related to the fat that some vdeos present abrupt hanges of amera perspetve durng vdeo aquston whh wll onsequently ause an abrupt hange of senaro n two onseutve frames. Thus, when measurng the temporal atvty of a vdeo sequene, the resultng global value may not reveal the true value of the temporal atvty. Fgure 3.6 presents the temporal atvty of the vdeo sequene Table, frame by frame. 3

49 (a) Fgure 3.6: Table temporal atvty, frame by frame onsderng (a) all the vdeo sequenes; (b) the (b) frames affeted by abrupt hange of amera perspetve In Fgure 3.6.a), t s possble to observe that the sequene Table presents the symptoms desrbed prevously, relatvely to temporal atvty global value. Analysng the temporal atvty evoluton of the Table sequene, t s possble to see that ths vdeo sequene shows, n a great part of the tme, a regular temporal atvty. However t s also possble to see from Fgure 3.6.b) that between the frame number 30 and the frame number 3 there s a sudden peak n temporal atvty value. Thus, when measurng the global temporal nformaton of that sequene, there s a dsrepany between the real value of the temporal atvty and the omputed one. In order to mnmze and smooth ths effet, t was appled a mathematal proedure, named as perentle 95%, to the temporal and spatal atvtes of eah one of the vdeo sequenes seleted for the test sessons. Fgure 3.7 presents the global results of spatal and temporal atvtes, after applyng the perentle 95% to eah vdeo sequene. 33

50 Fgure 3.7: Seleted vdeo sequenes for spatal-temporal atvty has been takng wth perentle 95% In aordane wth Fgure 3.8 and takng nto aount that the vdeo sequenes must span a large porton of the spatal-temporal nformaton, eght vdeo sequenes were hosen. Fgure 3.8 presents a sample mage of eah vdeo sequene seleted for the test sessons. Conernng the vdeo sequenes Coastguard and Football, although presentng smlar spatal-temporal atvtes, n terms of subjetve evaluaton they an aheve dfferent sores, for the same ompresson bt-rate. Ths an be justfed by the fat that these vdeo sequenes present dstnt types of ontent. So, n order to ollet the MOS resultng from dfferent senaros, both vdeo sequenes were seleted for the test sessons. Fgure 3.8: Vdeo s sequenes used n the subjetve tests 34

51 3.6 Vdeo ompresson The orgnal vdeo sequenes seleted n seton 3.3 were enoded usng two standard vdeo ompresson tehnques: H.64/AVC 5 and MPEG- 6. Fgure 3.9 shows dfferent levels of vdeo qualty degradaton that an be found durng the subjetve qualty evaluaton tests, and resultng from the ompresson proess. In both standards eah sequene was enoded wth 4 dfferent btrates. The reasons for that were to test the HVS perepton to dfferent knds of vdeo qualtes and to fore the observers to use all ratng sale. Therefore, at the end of the sesson, the MOS would be more onsstent and relable. Fgure 3.9: Vdeo sequenes enoded wth dfferent values of btrate In what onerns the number of test presentatons showed to eah observer n the subjetve tests, and n aordane wth seton 3.5, they were: number of test sequenes: 8; number of test ondtons: 4 dfferent ompresson btrates for eah vdeo sequene; number of test presentatons: 3. 5 H.64/AVC Ths vdeo ompresson standard was developed by the ITU-T Vdeo Codng Experts Group (VCEC) together wth the ISO/IEC Movng Pture Experts Group (MPEG), and t was the produt of a partnershp effort known as the Jont Vdeo Team (JVT). 6 MPEG- MPEG- s a standard for vdeo ompresson whh was developed by the Movng Ptures Expert Group (MPEG). 35

52 In the frst subjetve qualty assessment sesson, the vdeo sequenes were enoded usng the H.64/AVC ompresson standard, wth the ompresson btrates shown n Table 3.. In the seond sesson, vdeo sequenes were enoded usng the ompresson standard MPEG-, wth the ompresson btrates shown n Table 3.3. The ompresson btrates presented n Table 3. and n Table 3.3, were seleted wth the goal of dsplayng to the observers dfferent types of vdeo qualty for eah vdeo sequene. All vdeo sequenes used durng the tests sesson had a 35 x 88 spatal resoluton, 0 s of tme duraton and, exept for Australa (whh has a frame rate of 5 Hz), a frame rate of 30 Hz. Aordng to Table 3. and Table 3.3, t s possble to observe that, n a general, the ompresson rates n H.64/AVC are larger than n MPEG- standard. In fat, to aheve smlar vdeo qualty degradaton n H.64 as n MPEG-, t s neessary to derease the btrate n H.64 relatvely to the btrate n MPEG-. Wth regard to the ompresson methods, there are several sub-proesses that take plae durng ompresson, hene dfferent artfats are ntrodued nto the meda by H.64/AVC and MPEG-. These ompresson tehnques take advantage of the HVS s haratersts n the sense that they elmnate, from the vdeo, ertan data that a ommon human observer s not sensble to. Table 3.: Compresson btrates used n H.64/AVC H.64 Tral Tral Tral 3 Tral 4 Stephan 8 kbt/s 56 kbt/s 5 kbt/s 04 kbt/s Table 64 kbt/s 8 kbt/s 56 kbt/s 5 kbt/s Moble 64 kbt/s 8 kbt/s 56 kbt/s 5 kbt/s Football 56 kbt/s 5 kbt/s 04 kbt/s 048 kbt/s Foreman 64 kbt/s 8 kbt/s 56 kbt/s 5 kbt/s Coastguard 64 kbt/s 8 kbt/s 56 kbt/s 5 kbt/s Contaner 64 kbt/s 8 kbt/s 56 kbt/s 5 kbt/s Australa 3 kbt/s 64 kbt/s 8 kbt/s 56 kbt/s 36

53 Table 3.3: Compresson btrates used n MPEG- MPEG- Tral Tral Tral 3 Tral 4 Stephan 5 kbt/s 04 kbt/s 048 kbt/s 4096 kbt/s Table 56 kbt/s 5 kbt/s 048 kbt/s 04 kbt/s Moble 56 kbt/s 5 kbt/s 04 kbt/s 4096 kbt/s Football 5 kbt/s 04 kbt/s 048 kbt/s 4096 kbt/s Foreman 56 kbt/s 5 kbt/s 04 kbt/s 048 kbt/s Coastguard 56 kbt/s 5 kbt/s 04 kbt/s 048 kbt/s Contaner 8 kbt/s 56 kbt/s 5 kbt/s 04 kbt/s Australa 8 kbt/s 56 kbt/s 5 kbt/s 04 kbt/s However, wth nreasng levels of ompresson rates, the dstorton ntrodued by the proess may overtake the pereptvty threshold, and onsequently ntrodue vsble artfats nto the vdeo. When a vdeo sequene s enoded wth H.64, the artfat ntrodued by ths ompresson s essentally the blur effet (Fgure 3.0.a)). Blurrness s aused by the removal or attenuaton of hgh-frequeny ontent due to quantzaton or low-pass flterng and s haraterzed manly by smudgng of edges and loss of detal throughout the mage. In ontrast, when the vdeo sequene s enoded wth MPEG-, the most vsble artfat s the blok effet (Fgure 3.0.b)). In ths ase, blokness s haraterzed by ntrodung several and vsble small bloks ondutng to an aentuated mage dstorton. (a) (b) Fgure 3.0: Artfats ntrodued by (a) H.64 ompresson (blur effet) and (b) MPEG- ompresson (blok effet) 37

54 3.7 Vdeo qualty evaluaton program nterfae The program used to arry out the subjetve tests was the MSU (Mosow State Unversty) pereptual vdeo qualty player whh was developed by Graphs&Meda Lab Vdeo Group. The program nterfae s shown n Fgure 3.. In order to begn the vdeo qualty evaluaton sesson, the observer has to press the start button (d), and then the vdeo sequene, whh an be the referene vdeo or the dstorted vdeo (b), s dsplayed on the sreen. Fgure 3.: MSU pereptual vdeo qualty player nterfae: a) vdeo label (referene/dstorted vdeo); b) vdeo wndow; ) play button to start the vdeo sequene; d) vdeo tme bar Addtonally, ths nterfae allows the observer to have a perepton of the vdeo tme duraton through the vdeo tme bar (e). After dsplayng the referene vdeo and the dstorted vdeo respetvely, a fve pont mparment ratng sale wndow (Fgure.4.a)) s shown on the sreen where eah observer an gve hs vdeo qualty opnon. 3.8 Statstal analyss After the subjetve qualty tests have been onluded, the vdeo qualty assessment method seleted to perform those tests (the DCR/DSIS method), and desrbed n Chapter, produed dstrbutons of nteger values between and 5. The dfferene between the observers opnons about the vdeo qualty wll result n varatons on the observers sores. Ths phenomenon s vulgar when workng wth a group of humans, sne the dfferenes n judgement between them, s a onstant. Wth referene to the statstal analyss of the results, the man steps followed were ([ITU98]): 38

55 Calulaton of mean sores; Calulaton of onfdene nterval; Observers valdaton; Calulaton of MOS fnal values Calulaton of mean sores The frst step of the analyss of the results s the alulaton of the mean opnon sore, for eah of the presentatons, whh s gven by: u jkr (or MOS), u jkr = N u jkr N = ( 3. ) where, u jkr s the sore gven by observer, for test ondton j, sequene k, repetton r ; N s the number of observers present n the assessment sessons Confdene nterval In order to verfy f the dstrbuton of sores from the test presentaton s normal or not, the test was appled. The test onssts n the alulaton of the kurtoss oeffent of a funton. Ths knd of test allows to know f the dstrbuton s symmetr or not. Therefore, f the dstrbuton guarantees the test ondtons, t s possble to approxmate that dstrbuton by a normal dstrbuton. The gven by: jkr oeffent, related wth a test ondton j, sequene k and repetton r, s jkr = m ( m ) 4 where, m x N ( ujkr ujkr ) x = = ( 3. ) N If s between two and four, the dstrbuton an be onsdered, aordng to [ITU98], as a normal dstrbuton (also known as a Gaussan dstrbuton). The mean sores omputed for eah one of the presentatons should have always assoated a onfdene nterval, sne t s based on ths nterval that the relablty of the test results an be 39

56 guaranteed. The onfdene nterval assoated to eah mean sore, s derved from the standard devaton and sze of eah sample. Aordng to [ITU98], the onfdene nterval that should be used n ths type of analyss s a onfdene nterval of 95.5%, typally. where, and σ jkr The 95.5% onfdene nterval s gven by: δ, u δ ] [ u jkr jkr s the standard devaton for eah presentaton, gven by jkr + ( 3.3 ) jkr δ jkr = σ ( 3.4 ) jkr N ( u jkr ujkr ) σ jkr = ( 3.5 ) = N Fgure 3. presents a normal dstrbuton where a 95.5% onfdene nterval s sgnalsed. nterval u Fgure 3.: Normal dstrbuton nterval Aordng to Fgure 3., the probablty that a random varable X assumes a value n the jkr δ < X u + δ s equal to 95.5%,.e., jkr jkr jkr P( u < X u + σ ) = 95.5% jkr σ ( 3.6 ) jkr jkr jkr Observer valdaton After havng arred out the test, the sores u jkr gven by eah vewer wll be ompared wth the assoated outler ondton 7, whh an be dfferent f the dstrbuton s normal or not. If the 7 Outler - In statst, an outler s an observaton that s numerally dstant from the rest of the data,.e., an outler s all observatons whh are outsde of the onfdene nterval. 40

57 dstrbuton of sores follows a normal dstrbuton ( 4 ) and the onfdene nterval s 95.5%, then an observaton u jkr s onsdered as an outler f, jkr u σ jkr u jkr + jkr or ujkr u jkr jkr σ ( 3.8 ) On the other hand, f the dstrbuton of sores s not normal, the observaton u jkr s onsdered as an outler f, u σ jkr u jkr + 0 jkr or ujkr u jkr 0 jkr σ ( 3.9 ) When the sore σ ujkr of a vewer s superor to u jkr jkr σ + (f normal) or u jkr + 0 jkr (f nonnormal), a ounter related wth that vewer, P, wll be nremented,.e., n ase of a normal dstrbuton and u jkr u jkr + σ jkr = P + P ( 3.0 ) n ase of a non-normal dstrbuton and u jkr u jkr + 0σ jkr On the other hand, f the sore σ ujkr of a vewer s nferor to u jkr jkr (f normal) or u jkr σ 0 jkr (f non-normal), a ounter assoated wth that observer, Q, wll be nremented,.e., n ase of a normal dstrbuton and u jkr u jkr σ jkr = Q + Q ( 3. ) n ase of a non-normal dstrbuton and u jkr u jkr 0σ jkr alulated,.e., Fnally, aordng to [ITU98], after obtanng the whole sesson, and observer s gven by [ITU98]: P and Q oeffents, two ratos wll be P + Q wll be dvded by the total number of sores gven by eah observer for the P Q dvded by Q P + as an absolute value. The rteron to elmnate the P + Q If > J K R P Q and < 0. 3, the observer should be rejeted ( 3. ) P + Q where, N J s the number of observers; s the number of test ondtons for eah vdeo sequene; 4

58 K R L s the number of vdeo test sequenes; s the number of vdeo sequenes repettons durng the sesson; s the number of test presentatons (n most ases the number of presentatons wll be equal to, however t s noted that some assessment may be onduted J K R wth unequal numbers of sequenes for eah test ondton). The observer elmnaton should not be appled more than one to the results of a gven sesson [ITU99]. After the observer s valdaton has been performed the MOS omputed prevously and usng (3.), must be re-alulated. The new MOS s omputed takng nto aount the number of observers N ' whh are n aordane wth the observers valdaton rteron explaned n (3.). Thus, smlarly to (3.), the new MOS s gven by, MOS' = ' N u jkr N = ( 3.3 ) where, N' = N number of rejeted observers. 3.9 Subjetve qualty assessment results In ths seton the results of the subjetve qualty assessment sessons are presented. Table 3.4 presents the MOS (omputed aordng to 3.3) based n the opnon gven by the observers n eah one of the sessons,.e., usng the vdeo ompresson standards H.64 and MPEG-. As t s possble to observe from Table 3.4, the values of the MOS show that the vdeo ompresson appled to eah vdeo sequene requred the observers to use the 5 grades of the sale. Fgure 3.3 shows the MOS assoated to eah task for H.64 and MPEG- wth the 95.5% onfdene nterval plotted as a vertal bar. The vdeo materal used to perform the subjetve tests, suh as the orgnal vdeo sequenes, the H.64 and MPEG- ompressed vdeos, as well as the tests results the opnon sores and the MOS are avalable for those who are nterested on the vdeo qualty evaluaton feld at Fgure 3.4 shows some sreenshots of the webste where the test materal used to perform the subjetve tests s avalable. 4

59 Table 3.4: MOS usng vdeo ompresson standard H.64 and MPEG- H.64 MPEG- Task Sequene Rate [kbt/s] MOS Sequene Rate [kbt/s] MOS Task Australa 3.9 Contaner 8.86 Task Table Stephan Task 3 Contaner Moble 56.4 Task 4 Football Foreman Task 5 Moble 8.57 Australa 8.43 Task 6 Coastguard Table Task 7 Foreman Coastguard Task 8 Stephan Football Task 9 Contaner Moble Task 0 Australa Contaner Task Table 8.95 Stephan Task Moble Australa Task 3 Coastguard Table Task 4 Football Foreman 5.57 Task 5 Stephan 8.05 Football Task 6 Foreman Coastguard Task 7 Australa Contaner Task 8 Foreman Coastguard Task 9 Coastguard Table Task 0 Moble Australa Task Contaner Moble Task Football Foreman Task 3 Table Stephan Task 4 Stephan Football Task 5 Contaner Moble 5.57 Task 6 Stephan Football 5.9 Task 7 Moble Australa Task 8 Coastguard Table Task 9 Table 64.9 Stephan 5.05 Task 30 Foreman Coastguard Task 3 Football Foreman Task 3 Australa Contaner

60 (a) (b) Fgure 3.3: MOS wth onfdene nterval of 95.5% for (a) H.64 and (b) MPEG- 44

61 Fgure 3.4: Webste sreenshots 45

62 46

63 Chapter 4 Objetve Qualty Evaluaton 4 Objetve Qualty Evaluaton 4. Introduton As mentoned n the prevous Chapter, subjetve tests are partularly mportant n vdeo qualty evaluaton sne they provde the means to quantfy qualty as t s pereved by the vewers. However, they are not sutable for montorng vdeo data qualty. Reently, the nreasng suess of dgtal TV has motvated the researh of objetve qualty evaluaton metrs. These metrs am to assess the qualty of a broadasted vdeo as t s pereved at the user-end, automatally and n a real tme bass. In order to valdate the performane of an objetve qualty metr, the human assessment must also be taken nto aount. Ths Chapter proposes two new objetve vdeo qualty assessment metrs that ombnes a small set of features extrated from vdeo sequenes avalable at the user sde. Regardng to the 47

64 objetve vdeo qualty assessment metrs proposed n ths hapter, a smlar strategy t was followed by Tobas n [KOD09] and n [OD07], n whh a NR and a RR vdeo qualty metrs were proposed, respetvely. However the well sueeded results were not aheved for both vdeo qualty models. In fat, Tobas aheved muh better results for the RR model proposed n [OD07] than for the NR model proposed n [KOD09]. In order to mprove the metrs performane results a statstal tehnque, named as Prnpal Component Analyss (PCA), s used. Ths method s generally used for extratng the relevant nformaton from a orrelated feature data set transformng t nto a smaller set of less orrelated varables (alled Prnpal Components). Ths hapter s organzed as follows. After the Introduton, n seton 4. the MOS predton model s proposed, based on a study of a set of vdeo features and ther effet on the MOS values. In the same seton, PCA s desrbed. In seton 4.3 a set of measurements, proposed by VQEG, are presented wth the ntenton of evaluatng the MOS predton model performane. In seton 4.4, the model results and a performane evaluaton of the proposed metrs for H.64 and MPEG- ompresson standards are presented. Fnally, n seton 4.5, the man onlusons resultng from the work reported n ths hapter, are drawn. 4. Proposed MOS Predton Algorthms 4.. Motvatons The man goal of the proposed MOS predton models s to estmate the qualty value that a human observer would gve to a vdeo sequene. From the analyss of the subjetve results t was possble to relate a set of vdeo features wth the orrespondent MOS values. In hoosng the vdeo features that should be nluded n the MOS predton model, a trade-off was set between the nfluene that eah feature has on the MOS values and the dffulty of obtanng eah one of them. Obvously not all the vdeo sequene features nfluene the MOS n the same way, there are features whh have a hgh mpat n MOS values and others that an be dsarded sne they have not muh nfluene on them. In order to study the effet that some features have on the MOS values, an analyss s performed takng nto aount the ndvdual effet of these features. Fgures 4. and 4. dept the MOS evoluton wth the btrate and the MSE, respetvely, for a set of vdeo sequenes dsplayed durng the subjetve tests. 48

65 Fgure 4.: MOS evoluton wth btrate of some vdeo sequenes Fgure 4. shows that, as expeted, MOS nreases wth the btrate of the enoded vdeo sequenes. Observng the fgure, t an be seen that the MOS evoluton wth the btrate s not lnear: for hgh btrates a large varaton on the btrate does not lead to a sgnfant varaton on the MOS; on the other hand, for low btrates a small btrate varaton an ondut to a large MOS varaton. The trend lnes n Fgure 4. an thus be desrbed by a logarthm funton appled to the btrate. Fgure 4.: MOS evoluton wth the MSE of some vdeo sequenes Fgure 4. shows that there s also a relaton between MOS and MSE. For ths ase, MOS values are roughly nversely proportonal to the MSE values. Thus, the hgher s the dfferene (MSE) between eah frame of the orgnal and the enoded vdeo, the lower wll be the grade gven by the observers (MOS). Other mportant observaton taken from the subjetve test results s the fat that the spatal atvty and the temporal atvty of the vdeo sequenes also nfluene the qualty grades gven by the observers. Fgure 4.3 presents the resultng MOS values evoluton wth ths spatal and the temporal atvtes for a set of vdeo sequenes enoded at two dfferent btrates. 49