Subjective Consumer Evaluation of Multichannel

Size: px

Start display at page:

Download "Subjective Consumer Evaluation of Multichannel"

Laurel Tate
5 years ago
Views:

1 Audio Engineering Society Convention Paper 6558 Presented at the 119th Convention 2005 October 7 10 New York, New York USA This convention paper has been reproduced from the author's advance manuscript, without editing, corrections, or consideration by the Review Board. The AES takes no responsibility for the contents. Additional papers may be obtained by sending request and remittance to Audio Engineering Society, 60 East 42 nd Street, New York, New York , USA; also see All rights reserved. Reproduction of this paper, or any portion thereof, is not permitted without direct permission from the Journal of the Audio Engineering Society. of Multichannel Audio Codecs James L. Swinburne University of Technology, Melbourne, Victoria, Australia jbarbour@swin.edu.au ABSTRACT Are normal listeners able to identify any significant differences between multi-channel audio codecs when listening to commercial music releases on good quality, consumer audio equipment? Audio professionals have often suggested anecdotally that consumers are unable to hear the difference between high density, uncompressed multichannel formats and lower data-rate delivery formats. In this study, formal subjective listening tests were conducted according to the ITU-R BS.1534 (MUSHRA) [1] recommendation to evaluate consumer perception of popular 5.1 surround sound formats: Dolby AC-3, DTS, WMA and mp3surround. Experimental design, methodology and results will be presented and discussed. 1. INTRODUCTION Multi-channel surround sound is now understood and embraced by consumers through the rapid adoption of DVD video players and associated purchase of surround sound audio systems. Audio professionals are producing multi-channel music and soundtracks for computer games, film and video utilising high quality digital audio technology. Typical technical standards include 5.1 surround sound according to the ITU-R BS.775 recommendation [2], linear PCM at 24bit resolution and 96kHz sampling rate, and Direct Stream Digital for SACD [3]. Consumers are purchasing home theatre systems with universal DVD players and computer game players are purchasing multi-channel sound cards, and all are using surround amplifiers and loudspeakers to recreate exciting and immersive sound experiences. While there is a large and growing catalogue of multichannel audio material released on DVD-audio and SACD using high density, multi-channel coding formats, much listening by consumers is to multichannel audio encoded using lower data-rate delivery formats. Many audio professionals have lamented the use of these lower data-rate formats at AES workshop sessions and online in lists and forums, suggesting they are a weak link in the presentation and enjoyment of audio. Previous subjective testing of multi-channel codecs has focussed on listener evaluation using studio quality encoding and decoding technology and expert listeners. The principle aim of this formal study was to

2 establish whether consumers with normal hearing, using home theatre audio technology and listening to commercial music releases, were able to distinguish between different surround sound formats. 2. PREVIOUS SUBJECTIVE TESTING There have been many subjective listening tests conducted with results reported in journals and conference papers. These trials have mainly used professional encoders and decoders and trained, experienced listeners as subjects. Three trials were useful in the planning and execution of this trial. Wustenhagen et al [4] presented results of multi-channel codec testing at the 1998 AES 105 th Convention in San Francisco, when they tested Dolby Digital AC-3 and MPEG layer 2 at different bit-rates using professional encoders and decoders with subjects who were experienced and familiar with bit-reduced audio systems. Their audio samples included pitch pipe, applause, an orchestra and a rock concert, and their results indicated that the applause was the most critical item and the bit-rate was very significant in the perceived quality. Lorho and Zacharov [5] presented a trial of loudspeaker and headphone testing at the 2004 AES 116 th Convention in Berlin. They tested virtual home theatre systems where a 5.1 surround source is reproduced over only two loudspeakers or headphones, using software and hardware implementations of virtual surround. Samples included music, movie soundtracks and computer game sound, and the subjects were selected from an in-house permanent listening panel that was familiar with spatial audio. Results indicated that the original 5 channel source was preferred by all subjects and that the stereo downmix was preferred over two of the seven virtual systems over loudspeakers, and over headphones, the stereo downmix was preferred over five of the seven virtual systems. Soulodre and Lavoie [6] presented results of testing of Spectral Band Replication within the MPEG layer 2 codec at the 2004 AES 117 th Convention in San Francisco. Samples included rock music and audio from test CDs, and were coded at bit-rates of 128, 160 and 192 kbps, but no information was given qualifying the subjects of the trial. Testing was conducted using the ITU-R BS.1534 recommendation, a procedure which was considered to have been successful in their trial. 3. SELECTION OF CODECS AND BIT RATES The reference format for comparative testing was selected as 5.1 surround sound using linear PCM at 24 bit resolution and 48 khz sampling (24/48). This format has been suggested after subjective testing by Farina [7] as the lowest density LPCM which is virtually indistinguishable from higher density formats. The most common multi-channel audio format for DVD-video is Dolby Digital AC-3 at a bit rate of 448kbps [8]. Many videos also carry a DTS soundtrack at up to 1534kbps, while many multi-channel audio only releases are DTS encoded [9]. Many computer games and online sites streaming audio are using Windows Media Player [10], with the bit rate varying between 128kbps and 1Mbps. The Fraunhofer Institute has introduced mp3surround at 192kbps [11], a surround version of the popular mp3 stereo format, which is backwards compatible with stereo mp3 players. Table 1 below represents the formats selected for test: Format Bit rate Encoder Player Reference ~6.9Mbps 24/48 PCM RME PCI uncompressed and Anchor ~6.9Mbps reference 3.5 khz AC-3 448kbps Minnetonka Surcode Multiface RME PCI and Multiface Pioneer Universal player DV655A DTS 1284kbps Minnetonka Surcode Pioneer DV655A WMA 440kbps Nuendo 3 Adobe Audition / Audigy 2NX WMA 192kbps Nuendo 3 Audition / Audigy 2NX WMA 128kbps Nuendo 3 Audition / Audigy 2NX Mp3s 192kbps Fraunhofer encoder Fraunhofer decoder / Audigy 2NX Table 1 Selected formats and bit rates for test Page 2 of 9

3 4. CRITICAL AUDIO SAMPLES The principle aim of this study is to establish whether normal listeners with normal hearing, using readily available consumer audio technology and listening to commercial releases at home, are able to distinguish between different surround sound formats. Therefore, as an evaluation of consumer perception, all audio samples were derived from commercial consumer releases on either DVD-audio or SACD. No test disc audio material or synthesized audio samples were selected. This made selection particularly difficult, as much of the material considered did not produce significant artefacts even in the lower bit rate codecs compared to the reference stimulus, and thus was not suitable for test purposes. This point is considered further in the discussion of results at the end of the paper. After considerable time reviewing materials, 8 pieces were selected to represent a cross-section of musical styles, including rock and classical music. There was a balance between two principle production styles: discrete surround, where audio source locations include side and rear positions, referred to as F-F by Rumsey et al [12], and ambient surround, where all audio source locations are at the front and reverberation only comes from the rear loudspeakers, referred to as F-B by Rumsey et al.. Discrete audience applause was also used sourced from a commercial release. The test audio samples are listed in table 2: Description T Style Source Pink Floyd, 30s Discrete Dark Side of the Us and Them Moon, SACD, 2 32 Linkin Park, In The End 28s Discrete Reanimation, DVD-A track 2, 1 34 Applause 18s Discrete MDG , Hiroshima, Eternal Phoenix Monty Alexander, Moanin Karl Orff, O Fortuna from Carmina Burana Parade of the Charioteers from Ben Hur Richard Wagner, Tannhauser Act2, scene4, chorus Table 2 DVD-A, track 2 29s Discrete Jazzin Surround, SACD, Heads Up HUSA9005, s Discrete Telarc Sampler 1, SACD 60006, s Ambient Telarc Sampler 1, SACD 60006, s Ambient Telarc Sampler 2, SACD 60007, s Ambient Chesky Ultimate Surround Sampler CHDVD221, 3 40 Audio samples used in the evaluation tests 5. PREPARATION OF AUDIO SAMPLES All original DVD-A and SACD source materials were played from a Pioneer DV-655A universal player through 6 analogue outputs using on-board decoding. Samples were recorded into Nuendo 3 [13] as six mono wav files at 24/48 using an RME Multiface and PCI card [14], to become the original reference samples. The reference file for testing, a six channel interleaved wav file, and the 3.5 khz LPF anchor were created from the six original mono files. Using Minnetonka Audio s Surcode encoders [15], AC-3 and DTS files were created from each original reference sample and burnt to disc as a DVD-video soundtrack (AC-3) and as CD audio (DTS). The DVD-video and CD audio tracks were played from the Pioneer DV-655A and recorded through the RME Multiface as an interleaved wav file at 24/48. Nuendo 3 created Windows Media Pro files at 440kbps, 192kbps and 128kbps as six mono wav files, which were played out from an Audigy 2NX external Page 3 of 9

4 soundcard [16], with the analogue outputs recorded through the RME Multiface as interleaved wav files at 24/48. The Fraunhofer mp3s encoder was used to create the mp3s file, and the Fraunhofer mp3s decoder was used to play back through the Audigy 2NX soundcard, again with the analogue outputs recorded through the RME Multiface as interleaved wav files at 24/48. As a result, all test materials were six channel interleaved wav files recorded at 24/48 and stored on the test computer s hard drive. Playback for subject testing utilised the RME Multiface soundcard, Yamaha RV amplifier, Tannoy series 600 dual-concentric loudspeakers and a Nova Earthquake 40 subwoofer. The audio equipment chosen would realistically represent good quality home listening equipment. Listening tests took place in the audio research room at Swinburne University of Technology in Melbourne, Australia. While the room does not meet the specification of ITU- R BS.1116 [17], mainly because it is too small, it is highly damped and free from early reflections, and represents a good home listening environment. The reproduction level was adjusted using pink noise at - 18dBfs with an overall SPL of 85dBA, and was held constant for all listening tests. All tested audio materials were level matched and operated by the subjects using a custom interface built within Nuendo TEST PROCEDURE The testing conformed to the ITU-R BS.1534 (MUSHRA) recommendation, where subjects are able to compare multiple stimuli and establish a single score out of 100 representing their evaluation of the quality of each stimulus. The original reference was clearly identified and was always available, and there was a hidden reference and a hidden anchor as required in the recommendation. Subjects could control the playback and looping and could set start and finish times if required. All stimuli were time synchronised and there was a rapid inaudible cross-fade between each stimulus when selected. All subjects were given training using several audio samples also used in the tests, and completed the testing in a double-blind environment. Subjects were instructed to take into account the faithful reproduction of the spatial image, changes in tonal balance and distortion by introduced coding artefacts. All subjects were given sufficient time to compare and refine their judgements until satisfied with their scores. During testing, the order of the audio samples was randomised between subjects and the order of stimuli on the screen was also randomised. 7. SUBJECTS In total, a panel of 22 subjects completed the tests, consisting of two groups. There were 14 students in a radio training program aged between 20 and 32, 8 males and 6 females, who were experienced in focussed listening. The subjects answered a qualifying questionnaire which revealed the following characteristics: - none had a diagnosed hearing defect - none had participated in listening tests previously - all have experienced surround sound in a cinema - 64% have listened to DVD-videos with surround sound in a home environment - 21% have a surround sound system with a DVD-video player at home - none have recorded, edited or mixed surround sound The second group consisted of 8 audio professionals aged between 30 and 54, 6 males and 2 females, who are experienced in audio production and surround sound listening. In the results, it was possible to consider each group separately to identify variations in perception due to experience. 8. RESULTS The following charts show the results from the testing as mean and standard deviation, with normal listeners separated from the expert group. The hidden reference and the hidden anchor were identified by all listeners in all tests. Page 4 of 9

5 description of the changes for discrete applause for each codec is as follows: - DTS and WMA440: small timbral changes, but stable and unchanged localization with no warbling - Dolby AC-3: significant timbral change with reduced bass and changes to upper mid-range and treble, stable but slightly changed spatial location - WMA192 and 128: timbral changes with treble loss increasing at lower bit-rate, distinct warbling or chirping increasing with lower bit-rate, some localization shifts Figure 1: Normal listeners, Discrete applause This sample has clear, sharp handclaps all around the listening position with DTS and WMA440 being rated statistically indistinguishable from the reference. While sonic differences were identified between WMA 192 and mp3s, they were rated equally. - mp3surround: timbral change with treble loss and bass cut, localization image collapsed towards the median plane, stable image with no warbling, obvious phase cancellations with listener position movement. The three test samples with ambient surround are as follows: Figure 2: Expert group, Discrete applause Clear sonic differences were identified between all codecs, except WMA440, which was statistically indistinguishable from the reference. WMA128 was considered worse than the anchor. Judgement of all codecs apart from WMA440 was more severe than for the normal listener group. Figure 3: Normal listeners, Ambient surround: orchestra and chorus, with strong brass and percussion. Only mp3s and WMA128 were identified as being clearly different to the reference. Discrete applause with clearly identifiable sources of handclapping around the listener was the test sample which produced the greatest differences between codecs, noticeable to all subjects. A qualitative Page 5 of 9

6 Figure 4: Normal listeners, Ambient surround: orchestral performance with clear brass and percussion. As the bit-rate is reduced, coding artefacts become more obvious. Figure 6: All Ambient tests, Normal listeners: DTS, WMA440 and AC-3 were statistically indistinguishable from the reference, and WMA192 also performed well. Only WMA128 performed poorly, as mp3s was generally well regarded. Figure 5: Normal listeners, Ambient surround: orchestra and chorus, with brass, percussion and a clear xylophone. WMA192 and mp3s were identified as different but equivalent, with WMA128 preferred to the anchor. Figure 7: The Expert group rated WMA 192, 128 and mp3s more harshly than normal listeners, though 192 still performed well. There were four music samples with discrete surround and their results are as follows: Page 6 of 9

7 Figure 8: Normal listeners, Discrete surround: smooth rock mix with deep bass, vocals and solo sax. AC-3 was identified as lacking bass, while the lower bit-rate codecs produced distinct distortion. Figure 10: Normal listeners, Discrete surround: rock music. This track has heavy compression and limiting across all channels and is a very active surround mix with synthesizers, guitars and percussion swirling around, and many vocal effects also. AC-3 did not handle the compression well and spectral changes were significant. Spatial changes were clear with mp3s while artefacts with WMA192 were less obvious. Figure 9: Normal listeners, Discrete surround: acoustic jazz track featuring piano and lots of percussion including shakers. Spectral changes associated with AC- 3 were rated equivalent to the spatial changes and artefacts introduced by WMA192 and mp3s. Figure 11: Normal listeners, Discrete surround: electroacoustic jazz track, with synthesizers, guitars and lots of percussion all around. Changes were less obvious than for the acoustic jazz track, apart from WMA128. Page 7 of 9

8 indicate that for normal listeners, audible differences from the reference are not statistically significant for DTS or WMA440 for any of the music samples, either discrete or ambient surround, or for the applause sample. Dolby AC-3 at 448kbps was statistically indistinguishable from the reference for all ambient surround samples but changes were noticeable for all of the discrete surround samples. Figure 12: All Discrete tests, Normal listeners. AC-3 did not perform well for several of the samples and was rated equivalent to WMA192 overall. WMA128 was always clearly identified while mp3s performed better with some music styles than others. Figure 13: The Expert group rated WMA128 very poorly, mp3s only marginally better, while being critical of AC-3 and WMA192. Overall, only WMA440 performed well, with DTS being identified and marked down for several samples. 9. DISCUSSION Subjective assessment of the differences perceived by listeners is characterised as timbral changes, localization shifts and introduced artefacts referred to as warbling or chirping. For the discrete applause sample, these differences were clearly audible to all subjects, though judged more harshly by the expert group. Results For WMA192, significant audible differences were perceived for the discrete samples but not for the ambient samples. However, for the lower data-rate of WMA128, audible differences were perceived for all discrete and ambient samples, with all listener ratings in the fair to poor range. For the final tested codec, mp3surround at 192kbps, results varied substantially between ambient and discrete samples and between normal and expert listeners. For ambient samples, normal listeners rated mp3s good overall and for discrete samples it rated in the fair range. For expert listeners, the ratings fell to between poor and fair for all samples. This could in part be due to a perception by normal listeners that the spatial changes and phase problems introduced by mp3s coding are not as significant as the artefacts introduced by WMA192 and 128. With regard to the choice of samples, many of the music samples considered did not produce significant artefacts even in the lower bit rate codecs compared to the reference stimulus, and thus were not suitable for test purposes. This was particularly noticeable for vocal a Capella music and Mozart-style string orchestra, whether ambient or discrete recordings. It is noted in comparison that music with significant fast transient sounds and strong high frequency content, like percussion and brass, produced the most noticeable changes after coding. 10. CONCLUSION While it is essential that audio professionals use the highest quality uncompressed formats for recording and production of multi-channel audio, compressed multichannel formats are necessary to deliver a surround sound experience over bandwidth limited delivery systems. Popular multi-channel codecs include Dolby AC-3, DTS, Windows Media Audio and mp3surround, but it is inevitable that each compressed format will alter the original sound to some degree. Page 8 of 9

9 Formal testing has established that normal consumers listening to commercial releases on good quality audio equipment at home are able to perceive differences between some uncompressed and compressed digital audio delivery formats, dependent of the style of music, the production values of the surround mix and the datarate. Results indicate that DTS and WMA at 440kbps perform very well for all music styles and production techniques. Dolby AC-3 at 448kbps and WMA at 192kbps perform very well for ambient surround music but did not perform as well for discrete surround mixes. The mp3surround format provided satisfactory performance for ambient surround music but significant problems were identified for discrete surround mixes. At a data-rate of 128kbps, WMA performed poorly for all surround music. As manufacturers and broadcasters consider multichannel surround delivery, careful testing with critical audio samples should be performed lest the listening experience for consumers is compromised. The selection of music for testing is critical to the outcomes, and it is certainly possible to choose audio samples which will be unrealistically generous in their results. Producers and artists should be careful to fully explore all delivery formats for their particular music productions to be confident that the end results meet their expectations. It would be unfortunate if delivery data-rates are set so low that the end result for a consumer is uninspiring, hence undermining efforts to expand audience acceptance of multi-channel surround music. 11. REFERENCES [1] ITU-R BS.1534 (MUSHRA), , Methods for the Subjective Assessment of Small Impairments in Audio Systems including Multichannel Sound Systems, International Telecommunications Union, Switzerland. [2] ITU-R BS.775, 1993, Multi-channel stereophonic sound system with or without accompanying picture. International Telecommunications Union, Switzerland. [5] Lorho and Zacharov, Subjective Evaluation of Virtual Home Theatre Sound Systems for Loudspeakers and Headphones, AES 116 th Convention Paper 6141, Berlin 2004 [6] Soulodre and Lavoie: Subjective Evaluation of MPEG Layer 2 with Spectral Band Replication, AES 117 th Convention Paper 6185, San Francisco 2004 [7] correspondence from Angelo Farina 2/4/2005 [8] for information on Dolby Digital AC-3 and the DVD-video audio specification [9] DTS Coherent Acoustics: [10] Windows Media information: [11] mp3surround, Fraunhofer IIS: d/ [12] Rumsey, Zielinski, Kassie, Bech, Predicting Multichannel Audio Quality for Down-mix and other Resource-limited Conditions in Broadcasting, AES 116 th. Convention, Berlin, 2004 [13] Nuendo: [14] RME: [15] Minnetonka Audio software, AC-3 and DTS encoders: [16] Audigy 2NX: [17] ITU-R BS.1116 ITU-R BS , , Methods for the Subjective Assessment of Small Impairments in Audio Systems including Multichannel Sound Systems, International Telecommunications Union, Switzerland. [3] SACD information: [4] Wustenhagen et al, Subjective Listening Test of Multi-channel Audio Codecs, AES 105 th Convention Paper 4813, San Francisco 1998 Page 9 of 9

Proceedings of Meetings on Acoustics

Proceedings of Meetings on Acoustics Volume 19, 213 http://acousticalsociety.org/ ICA 213 Montreal Montreal, Canada 2-7 June 213 Engineering Acoustics Session 2pEAb: Controlling Sound Quality 2pEAb1. Subjective