Multimedia Quality in a Conversational Video-conferencing Environment

Transcription

1 ETSI workshop on Effects of transmission performance on Multimedia QoS June Prague, Czech Republic Multimedia Quality in a Conversational Video-conferencing Environment Quan Huynh-Thu, Psytechnics Ltd quan.huynh-thu@psytechnics.com

2 Presentation outline Introduction to video-conferencing Beyond video-conferencing HD video-conferencing subjective experiment Results Discussion 2

3 Video-conferencing: early hype What? Real-time audio-visual communication Why? Enhance collaboration Where? Desktops, laptops, mobile devices Integration in Unified Communications environment 3

4 Video-conferencing: challenges Real-time Delay Interactive Media Multimedia synchronization Bandwidth Coding artefacts limited Transmission errors and IP Application network congestion 4

5 Video-conferencing: issues Technology: Video compression (coding) IP-based transmission (errors) Real-time constraint (delay) Two or multi-party multimedia communication (handling of potential multiple end-points) Human factor: 'Un-natural' conversational feeling due to small resolution and camera set-up: I m looking at you... but can't see you Higher sensitivity to errors than for media streaming 5

6 Video-conferencing: user expectation Usage Requirements Public Look, Hi Cheap Simple usage Corporate/Enterprise Collaboration Increased productivity Face-to-face interaction Cost effective Reliable Quality 6

7 Beyond video-conferencing: High-definition Real-time audio-video communication Enhanced features for tele-collaboration Face-to-face feeling (tele-presence) 7

8 HD video-conferencing subjective experiment Real-time interactive/conversational experiment using high-definition video-conferencing system Trials: task-driven 2-min conversations 1. Task requiring focus on visual terminal 2. Task not especially requiring use of visual information Various bandwidth/network conditions 8

9 Subjective testing facilities State-of-the art subjective test rooms at Psytechnics headquarters 9

10 Experiment set-up: equipment 24 widescreen full HD display (1080p) Hardware codec box with associated HD camera Professional shotgun microphone Pair of speakers 10

11 Experiment: set-up speaker microphone camera speaker display Viewing distance: 4H (80cm) 11

12 Conversational task 1: Shape matching Purpose: maintain focus on visual terminal Description: work in partnership to build a 3D shape from construction blocks Subject A was given a bag of multicolour, interlocking construction blocks Subject B was given a completed figure made from an identical set of blocks Subject B had to instruct subject A on how to build the identical shape Subjects had to verify the correctness of the built shape 12

13 Conversational task 2: Who s in the bag? Purpose: free conversation not especially requiring focus on visual terminal Description: identify as many famous characters as possible. The Clue Giver takes a card from the bag and has to provide clues as to who is on the card without naming the character The Guesser must name the character and can ask questions Clue giver can skip the name he/she does not know the personality 13

14 Participants 20 naïve subjects: 8 females and 12 males 16 subjects recruited from public 4 subjects from Psytechnics Age: None had participated in subjective testing in the past None had experienced a HD video-conferencing system in the past 14

15 Experimental design Bandwidth: 3072 and 1472 kbps Packet loss ratio: 0, 0.5, 3 and 6% Task: 1 and 2 No delay or jitter Video codec: H.264 baseline profile Audio codec: AAC at 64kbps Network degradations generated using ITU-T G.1050 IP impairment model Full factorial design (3 variables): 16 test conditions 15

16 Experimental procedure Test conditions grouped by task: 1. 8 test conditions with task 1 2. Break 3. 8 conditions with task 2 Presentation order of test conditions was randomized Test condition was identical for both parties in a given trial Role of instruction giver and instruction follower was swapped between test conditions 16

17 Quality assessment ratings 1. How would you rate the audio quality of the connection? 2. How would you rate the video quality of the connection? 3. How would you rate the overall quality of the connection? 4. Did you have any difficulty in understanding the other party during the connection? 5. Was the overall quality of the connection acceptable for the task? 17

18 Quality assessment ratings 5-point discrete category rating scale for Qs 1-3: Excellent Good Fair Poor Bad Binary answer for Qs 4-5: Yes No 18

19 Example: Task 1 with 3072 kbps, PLR=0% 19

20 Example: Task 1 with 1472 kbps, PLR=6% 20

21 Example: Task 2 with 1472 kbps, PLR=0% 21

22 Example: Task 2 with 1472 kbps, PLR=6% 22

23 Results: distribution of quality ratings audio video multimedia Percentage e of votes Percentage e of votes Percentage e of votes Ratings Ratings Ratings 23

24 Results: audio quality Audio MOS Audio MOS task 1 task 2 Condition Condition PLR=0% PLR=0.5% PLR=3% PLR=6% 1.4Mbps Task2 3Mbps Task1 3Mbps Task2 1.4Mbps Task1 3Mbps,PLR=0 1.4Mbps,PLR=0 3Mbps,PLR= Mbps,PLR=0.5 3Mbps,PLR=3 1.4Mbps,PLR=3 3Mbps,PLR=6 1.4Mbps,PLR=6

25 Results: video quality Video MOS Video MOS task 1 task 2 Condition Condition PLR=0% PLR=0.5% PLR=3% PLR=6% 1.4Mbps Task2 3Mbps Task1 3Mbps Task2 1.4Mbps Task1 3Mbps,PLR=0 1.4Mbps,PLR=0 3Mbps,PLR= Mbps,PLR=0.5 3Mbps,PLR=3 1.4Mbps,PLR=3 3Mbps,PLR=6 1.4Mbps,PLR=6

26 1.4Mbps Task2 3Mbps Task1 3Mbps Task2 Results: multimedia quality Multimedia MOS Multimedia MOS task 1 task Mbps,PLR=0 1.4Mbps,PLR=0 3Mbps,PLR= Mbps,PLR=0.5 3Mbps,PLR=3 1.4Mbps,PLR=3 3Mbps,PLR=6 1.4Mbps,PLR=6 Condition Condition PLR=0% PLR=0.5% PLR=3% PLR=6% 1.4Mbps Task1

27 ANOVA: audio quality Source Sum Sq. d.f. Mean Sq. F Prob>F Nobs Effect size BitRate PLR Task BitRate*PLR BitRate*Task PLR*Task BitRate*PLR*Task

28 ANOVA: video quality Source Sum Sq. d.f. Mean Sq. F Prob>F Nobs Effect size BitRate PLR Task BitRate*PLR BitRate*Task PLR*Task BitRate*PLR*Task

29 ANOVA: multimedia quality Source Sum Sq. d.f. Mean Sq. F Prob>F Nobs Effect size BitRate PLR Task BitRate*PLR BitRate*Task PLR*Task BitRate*PLR*Task

30 Intelligibility Percentage of subjects (P1) who had no difficulty in understanding the other party during the connection MOS=2.95 Task 1 Task 2 Conditio n P1 (%) Conditio n P1 (%)

31 Acceptability Percentage of subjects (P2) who found the quality of the connection acceptable for the required task MOS=2.95 Task 1 Task 2 Conditio n P2 (%) Conditio n P2 (%)

32 Summary Conversational video-conferencing experiment using Full-factorial design based on 3 variables: bandwidth, packet loss ratio and task Random packet loss No delay or jitter Most participants unexpectedly provided high quality ratings and found the quality acceptable even if video was severely degraded (user expectation) The system in test produced similar video quality at both 3Mbps and 1.4Mbps (without packet loss) Packet loss ratio was found to be the most important factor influencing multimedia quality amongst the 3 variables considered. Statistical analysis showed an interaction effect between the visual impact of packet loss and task on multimedia quality 32

33 Discussion User expectation: naïve participants (public) might have lower quality expectation than target users (experienced with video / business users) Balance/distribution of errors/qualities on the audio and visual signals using real-world systems in subjective experiments Suitable tasks to exercise both audio and video components, create eye-contact Conversational tests represent heavy investment for relatively small amount of data: Full-factorial design with small number of variables to examine main/interaction effects Fractional design with high number of variables but no possibility to examine main/interaction effects 33