Andrea Superbeam Array Microphone Speech Recognition Performance Using Microsoft Office XP

Transcription

1 Andrea Audio Test Labs White Paper Andrea Superbeam Array Microphone Speech Recognition Performance Using Microsoft Office XP July 12, 2002 Revision 3A Andrea Electronics Corporation 45 Melville Park Rd Melville NY, (800) Prepared by: Mark Schmidt VP Engineering Copyright 2002 Andrea Electronics Corporation All rights reserved.

2 Andrea Audio Test Labs Goals We compare the Andrea Superbeam Array Microphone s Speech Recognition performance to other microphones. Tests were performed under the same conditions to determine, on a relative basis, how the Superbeam microphone compares to a close-talking USB headset and a single element OEM microphone. The speech engine utilized in the tests was Microsoft Office XP s. The intention is not to determine absolute Speech Recognition accuracy. With this in mind, we are using only a single male voice for speech input. Minimum training is performed. Additional training will improve absolute accuracy. The accuracies herein are for comparison purposes only and do not reflect actual user experiences. A serious user will train more than the minimum, and the speech engine will improve even without training as a user corrects mistakes. The questions we answer are: How does the Andrea Superbeam Array Microphone compare to a Canceling USB Headset? How does the Andrea Superbeam Array Microphone with Andrea s DSDA Beam forming and PureAudio reduction software compare to other solutions? Is accuracy affected by the acoustics of the room or the area where the microphone is used? Is an analog, noise canceling, close-talking headset affected by ambient noise? Can PureAudio alone (as included with Analog Devices SoundMAX Cadenza TM ) improve speech recognition accuracy of an analog, noise canceling, close-talking headset? Can PureAudio alone (as included with Analog Devices SoundMAX Cadenza TM ) improve speech recognition accuracy of low cost PC OEM microphones? 2.0 Results Summary The Andrea Superbeam Array Microphone offers outstanding, consistent performance under a wide range of conditions. The results indicate the Superbeam Array Microphone performance is comparable to or better than a close-talking USB headset under a variety of conditions. Other solutions lack consistent performance under varying noise environments. Andrea s PureAudio Reduction Algorithm, as found in Analog Devices SoundMAX Cadenza improves the speech recognition accuracy of low cost OEM microphones, but does not improve performance enough in noisy and reverberant environments. In those situations, the best solution is the Andrea Superbeam Array Microphone with its combination of DSDA Beam Forming and PureAudio noise reduction. Even close-talking noise canceling headsets are affected by ambient noise. An analog, noise canceling headset can benefit from Andrea s PureAudio noise reduction software.

3 Andrea Audio Test Labs Detailed Results 3.1 Data Presentation and Analysis How does the Andrea Superbeam Array Microphone compare to a Canceling USB Headset? Superbeam vs. USB Headset Speech Recognition Accuracy With the MS Office XP Speech Engine the Superbeam Array outperforms the USB Headset in every test environment with and without noise. Chart 1 - Superbeam vs USB Headset - MS Office XP Speech Recognition Accuracy - % Superbeam USB Headset 70 Office Office Cube Cube Cube Voice Conf Rm Conf Rm Office Table 1 Superbeam vs. USB Headset - MS Office XP Accuracy % Small Open Open Cube Open Cube Office with Cube with Office with Voice Office Small Office Conf Rm Conf Rm with Office Superbeam USB Headset Note: The USB Headset is a Plantronics DSP-300

4 Andrea Audio Test Labs How does the Andrea Superbeam Array Microphone with Andrea s DSDA Beam-forming and PureAudio noise reduction software compare to other solutions? Open Cube Speech Recognition Accuracy The Superbeam Array outperforms both a USB Headset and an OEM PC microphone in the open cube environment. Office noise degrades the OEM PC Microphone performance making it unusable for speech recognition. Chart 2 - Open Cube - MS Office XP Speech Recognition Accuracy - % Superbeam USB Headset OEM PC Mic 50 Office Voice Table 2 - Open Cube - MS Office XP Accuracy % Office Voice Superbeam USB Headset OEM PC Mic Note: OEM PC Mic is a Telex M-30

5 Andrea Audio Test Labs Small Office Speech Recognition Accuracy With MS Office XP the Superbeam Array outperforms both a USB Headset and an OEM PC microphone. The OEM PC Mic has trouble even in quiet conditions because of room acoustics, and things get much worse with the addition of office noise. Again, the OEM PC Microphone performance is unusable for speech recognition. Chart 3 - Small Office - MS Office XP Speech Recognition Accuracy - % Superbeam USB Headset OEM PC Mic 50 Office Table 3 - Small Office - MS Office XP Accuracy % Office Superbeam USB Headset OEM PC Mic Note: OEM PC Mic is a Telex M-30

6 Andrea Audio Test Labs Conference Room Speech Recognition Accuracy With MS Office XP the Superbeam Array again outperforms both the OEM PC Mic and the USB Headset in the conference room environment. The room acoustics are a problem for the OEM PC Mic. Again, the OEM PC Microphone performance is unusable for speech recognition. Chart 4 - Conference Room - MS Office XP Speech Recognition Accuracy - % Office Superbeam USB Headset OEM PC Mic Table 4 Conf Room - MS Office XP Accuracy % Office Superbeam USB Headset OEM PC Mic Note: OEM PC Mic is a Telex M-30

7 Andrea Audio Test Labs Is accuracy affected by the acoustics of the room or the area where the microphone is used? Yes, room acoustics do affect speech recognition accuracy. Users of speech recognition software complain that sometimes it works well, other times it performs poorly. Room acoustics play a part in this. A major problem is reflections off walls, floors and ceilings. Also, room size determines when the reflections (echo) will return to the microphone. The Superbeam Array Microphone uses advanced de-reverberation techniques to reduce echo and minimize its effect on the speech engine Is a close-talking, noise canceling headset affected by ambient noise? The surprising answer is: Yes. Most high quality headset microphones utilize a Canceling microphone element. At low frequencies, (e.g. 100 Hz) the mic element will reduce ambient noise by about 25 db relative to the speaker. The noise cancellation effectiveness is reduced as frequency goes up until there is no attenuation of ambient noise at about 3 khz. Any noise above 3 khz will be picked up by the microphone and sent on to the speech recognition software. Figure 1 is a typical Canceling microphone element frequency response (bare element, not mounted in a headset). The close-talking response, given by the curve at L=1 cm shows a flat response from 100 Hz to 10 khz. The far-field response, given by the curve at L=50 cm shows the attenuation ( Canceling). In fact, Andrea s PureAudio can improve the accuracy of an analog close-talking headset under some noise conditions. PureAudio reduces stationary noise sources across the frequency range with db in addition to the reduction built in to the microphone element. Figure 1 Panasonic Canceling Electret Microphone Element Frequency Response The estimated frequency response needed by speech engines is 100 Hz to 6 KHz. PureAudio can further reduce noise in the 1 KHz and up range that is not canceled by the Canceling Microphone. Note: A USB Headset will not be aided by PureAudio as part of Analog Devices SoundMAX Cadenza. USB Headsets use external audio Codecs and utilize their own drivers and noise reduction software. SoundMAX Cadenza is available for the built in Codec of the computer only.

8 Andrea Audio Test Labs Can PureAudio alone (as included with Analog Devices SoundMAX Cadenza TM ) improve speech recognition accuracy of an analog, noise canceling, close-talking headset? Recognition accuracy of an analog close-talking headset was improved by adding Andrea s PureAudio as found in Analog Devices SoundMAX Cadenza. The improvement was evident in a Small Office environment with its difficult acoustics. There was no improvement in the Open Cube environment because it is acoustically less challenging. Chart 5 - Small Office - MS Office XP Speech Recognition Accuracy - % Andrea Headset with PureAudio Andrea Headset 70 Office Table 5 Small Office - MS Office XP Accuracy % Office Andrea Headset with SoundMAX PureAudio Andrea Headset Note: The Andrea Headset is an NC-61

9 Andrea Audio Test Labs Can PureAudio alone (as included with Analog Devices SoundMAX Cadenza TM ) improve speech recognition accuracy of low cost PC OEM microphones? Andrea s PureAudio will improve significantly the speech recognition accuracy of a low cost, single element PC OEM microphone. With the MS Office XP Speech Engine the PureAudio significantly improves recognition accuracy, BUT we can see noise reduces the accuracy to an unacceptable level. We cannot recommend this solution for all environments. We recommend the Superbeam array with its consistent, excellent performance that exceeds the accuracy of a close-talking headset microphone. Chart 6 - PureAudio improvement of OEM PC Mic - MS Office XP Speech Recognition Accuracy - % Office Office Cube Cube Cube Voice Conf Rm Conf Rm Office OEM PC Mic with Pure Audio OEM PC Mic OEM PC Mic with PureAudio Table 6 PureAudio improvement of OEM PC Mic - MS Office XP Accuracy % Small Office Small Office with Office Open Cube Open Cube with Office Open Cube with Voice Conf Rm Conf Rm with Office OEM PC Mic Note: The OEM PC Mic is a Telex M30

10 Andrea Audio Test Labs Test Methodology Notes 4.1 Enrollment Enrollment is the process of training the speech recognition software to your voice so that it can improve its accuracy. Since each microphone has its own tonal qualities, signal to noise, and sensitivity, each microphone requires separate enrollment. To minimize test time, and to accentuate any differences between microphones, the minimum enrollment is performed for each test microphone. Accuracy will most certainly improve if additional training is performed. 4.2 Test Script Our test script has a word count of 466. Punctuation is counted as words. A recording of a trained voice 45 yr old male is used. The text of the script is given in the Appendix. The script is direct dictation into Microsoft Word 2002 (Part of Office XP) using the Microsoft Speech Engine. 4.3 Scoring Scoring is done using an Andrea Audio Test Labs File Comparison Utility developed for speech test scoring. The utility compares the reference script to the speech-recognized script of the microphone under test. The scoring method counts substitutions, which are words, or commands that the computer misinterprets, and deletions, which are words or commands that the computer misses. The Speech Recognition Accuracy score is calculated as follows: Accuracy Score (%) = ((Total words Substitutions Deletions) / Total words) x 100 Each test is repeated at least three consecutive times, and the average score is calculated Consider relative scores only Because of the lack of an industry standard for speech recognition accuracy and the differing scoring approaches, all accuracy data should be made on a relative basis only Alternate scoring approach considering insertions There are different approaches on how to score continuous speech recognition accuracy. The Andrea scoring assumes that only substitutions or deletions are considered errors. Another approach is to consider insertions as relevant. Alternate Accuracy Score with insertions (%) = ((Total words Substitutions Deletions Insertions) / Total words) x 100 Example: Original Speech: The age of automation Speech program interpretation: The age of water nation The Andrea Accuracy Score here is: 4 (total words) 1 ( automation substituted by water nation ) / 4 = 75% The alternate scoring considering insertions is: 4 (total words) 1( automation substituted by water ) 1(insertion of nation ) /4 = 50%

11 Andrea Audio Test Labs 11 The accuracy scoring is usually lower when insertions are considered as errors. For accuracies of 80% and above the scoring will be about 1-5% lower when considering insertions. For accuracies below 80% the scoring can be 5-20% lower when considering insertions. 4.4 Results may vary Because of the many variables involved with speech recognition testing, recreation of our test environments may result in somewhat different results. The relative results however should be the same. We have made every attempt to minimize test variations, and after hundreds of tests our results have been very consistent.

12 Andrea Audio Test Labs 12 Appendix 1.0 Test Environments Testing was performed under the following controlled, but real life conditions: A small office An open cubicle A moderate sized conference room 1.1 Test Setup All setups used a computer (PC-1) as the source of the speech files that are played through a Brüel & Kjær artificial head and torso simulator. files, when needed, are played from PC-1 through an amplifier then through an amplified off axis speaker. To ensure repeatable results the Speech and noise files are combined into a single stereo file that is played left channel to the head and torso simulator and right channel to the off axis speaker. The Device Under Test (DUT) is connected to a second computer that is configured for speech recognition. The input for all tests (except the USB Headset) is via an Analog Devices Communication Network Riser (CNR) card with an 1885 Codec, Stereo Mic input and Mono Mic input. Audio Amp Amplified Speaker PC-1 Source Audio.wav files Artificial Head & Torso Microphone DUT PC-2 Win XP and Mic Input A source of variability is sound pressure level measurements. There are two main weightings for Sound Pressure Level (SPL): A weighting and C weighting. A weighting compensates for the non-linear response of the human ear, while C weighting is a flatter response. The response of the human ear to sound is worse at low and high frequencies. The two weightings will give different results depending on the frequency distribution of the sound. In an office environment, significant content can be missed by A weighting that we cannot hear, but the microphone will pick-up. The microphone and speech engine will be affected by this inaudible noise so we have considered it. Low frequency blower and air exchanger noise is of particular concern, as it is difficult to hear. All Andrea SPL readings use C weightings. We believe C weighting reflects more accurately the range of sounds the microphone picks up.

13 Andrea Audio Test Labs Small Office Testing in a 10 3 x 11 x 9 (high) closed office with sheetrock walls, dropped ceiling acoustic 2 x 4 tiles and carpeting is the Small Office data. conditions include noise from the system under test s electrical circuitry and acoustic noise from the fans in the computer enclosure. The test computer has three internal fans: one on the Pentium, one on the power supply and one on the rear of the case. This simulates what might be experienced with the door to an office closed. Office noise conditions include the noise and add a looped playback of Track 6 (Office) of The New CBS Audio-File Sound Effects Library CD 2. This office noise track includes voices, phones, doors and HVAC background as recorded in a typical office setting. The SPL of the sound track is adjusted for 65dBC SPL as measured at the microphone under test s location. The noise speaker is 90 offaxis, 57 from the microphone under test (when mounted on the computer monitor), and is in the left of the picture below. This simulates what might be experienced with the door to an office open.

14 Andrea Audio Test Labs Open Cubicle Testing in a modern office environment with sound deadening soft partitions, 9 suspended ceiling of acoustic 2 x 4 tiles and carpeting is the Open Cube data. The test cubicle is in a larger area with a total of 8 comparable sized cubicles. It is an interior partition with cubicles on three sides. conditions include noise from the system under test s electrical circuitry and acoustic noise from the fans and hard drive in the computer enclosure. The test computer has three internal fans: one on the Pentium, one on the power supply and one on the rear of the case. This simulates what might be experienced working alone in an open office after normal work hours. Office conditions include the noise and adds looped playback of Track 6 (Office) of The New CBS Audio-File Sound Effects Library CD 2. This office noise track includes voices, phones, doors and HVAC background as recorded in a typical office setting. The SPL of the sound track is adjusted for 65dBC SPL as measured at the microphone under test s location. The noise speaker is 60 offaxis, 5 feet from the microphone, and is in the left of the picture below. This simulates what might be expected in a normal office during peak work hours. Voice conditions include the noise and add an off-axis looped playback of The age of automation by a trained voice 30-year-old female. The SPL is adjusted for 94 1 from the speaker. This 60 off axis voice is 5 feet from the microphone, resulting in approximately 65 dbc at the microphone position. This simulates what might be experienced with someone talking close to your cubicle.

15 Andrea Audio Test Labs Moderate Sized Conference Room Testing in a 23 x 13.5 x 9 (high) Conference room with sheetrock walls, suspending ceiling of 2 x 4 acoustic ceiling tiles and carpeting is the Conf Rm data. During all tests both doors to the room were closed. conditions include noise from the system under test s electrical circuitry and acoustic noise from the fans and hard drive in the computer enclosure. The test computer has three internal fans: one on the Pentium, one on the power supply and one on the rear of the case. This simulates what might be experienced working alone in a conference room with the doors closed. Office conditions include the noise and adds looped playback of Track 6 (Office) of The New CBS Audio-File Sound Effects Library CD 2. This office noise track includes voices, phones, doors and HVAC background as recorded in a typical office setting. The SPL of the sound track is adjusted for 65dBC SPL as measured at the microphone under test s location. The noise speaker is 60 offaxis, 5 feet from the microphone, and is in the left of the picture below. This simulates what might be experienced working in a conference room with the doors open.

16 Andrea Audio Test Labs Test Microphones Summary of Characteristics Microphone Type Computer Interface Used Andrea Superbeam Array Microphone Far-field two-element array with unidirectional elements. Separate channels for each microphone. Analog stereo Mic level input Comments Requires Analog Device Codec with Stereo Mic input (and SoundMAX Cadenza software). Retail $24.95 Andrea NC-61 Close-talking, headset microphone with acoustic noise canceling element Analog Mic level mono input Gold Standard for speech recognition accuracy. Selected by IBM for inclusion with ViaVoice TM from 1998 to Retail $29.95 Plantronics DSP300 Close-talking, headset microphone with acoustic noise canceling element and DSP processing for noise reduction and signal enhancement USB Recommended by Microsoft for use with Office XP Provided with a version of IBM ViaVoice TM (Release 9 Professional). Est. Retail $79 to $ Telex M-30 OEM PC Mic Far-field single element with a unidirectional element AATL Lab testing showed KHz Analog Mic level mono input Supplied with some Dell computers. Est. Retail $ $14.99

17 Andrea Audio Test Labs Test Equipment & Test Software ITEM QTY Reference Type (or equivalent) Equipment Computer PC-1 1 Dell Dimension XPS T450, 450 MHz PIII 128MB RAM Creative SoundBlaster Live! Platinum soundcard. Computer PC-2 1 Intel motherboard D845WN with Analog Devices AD1885 Codec, Intel Pentium GHz with Intel cooling fan, 256 MB PC133 RAM, IBM 20.57GB 7200 RPM Hard Drive, #IC35L020AVER07; Mid Tower ATX case with Power Man ATX300 Watt Power Supply Model FSP BTV; Video card Matrox Millennium G450, 32 MB; Audio via Analog Devices CNR card with Stereo Mic input, Mic input using 1885 Codec. Three fans in PC: one rear exhaust, one Power Supply, one on Pentium heatsink Artificial Head & Torso 1 B &K (Brüel & Kjær) type 4128 Head and Torso Simulator Audio Amplifier 1 Radio Shack Model MPA-46 Cat No Amplified Speaker 1 Fostex Model 6301B3E (33W) Sound Level Meter (Digital) 1 Extech Model (±1.5 db accuracy) Sound Level Meter 1 Radio Shack Cat No (Analog) Speaker Stand 1 Fidek FS-102 Cables 1 Audio interconnect cables as required Software PC-1 Windows 98 SE 1 Microsoft PC-2 Windows XP 1 Microsoft Professional Office XP Professional 1 Microsoft Persono Version a 1 Plantronics software for DSP-300 Office 1 Track 6 (Office) of The New CBS Audio-File Sound Effects Library CD 2 Audio Setup wav file 1 Office XP Setup script recorded by 45-year-old male (RS) trained voice. Enrollment Script wav files 1 Office XP enrollment script recorded by 45-year-old male (RS) trained voice. Recognition Script Wav file 1 Andrea The Age Of Automation script recorded by 45-year-old male (RS) trained voice. Recognition Script Wav file 1 Andrea The Age Of Automation script recorded by 30-year-old female (CB) trained voice. Recognition Script.txt file 1 Reference text for the wav file Scoring Utility 1 Andrea Digital Scoring Utility

18 Andrea Audio Test Labs Test Script The text of the Age of Automation * test script follows. The age of automation, which was supposed to free American business from the burden of paperwork under which it had so long struggled, has instead only made things worse. Computers don't create less paperwork; what they do is make it possible to create more and better looking paperwork faster and more efficiently. Now that every office has fax capability in addition to computers, the mountain of paperwork is getting taller at a much faster rate. And what a mountain it is! A few years back one division of a large corporation we were working with removed 10 T of paper during a cleanup campaign. A California personnel company surveyed over 900 personnel directors about the portions of their jobs that was committed to routine paperwork. More than half of them said they spend between one and three hours each day, which works out to as much as roughly 94 work days a year. I can think of some fields in which the paperwork total would be far higher. Copies of reports and memoranda are circulated widely, with no one questioning whether they are needed. Top management is bombarded with documents from several levels down, and lower level managers find themselves submerged in cascades of paperwork from above, which seems to gather authority and urgency at each successive level. The possibility of "Computer crashes" dictates that files be backed up on disks as well as on "Hard" paper copies so now we have disc files to manage, but we have not gotten rid of our paper files. I can't count the number of people I know who have vowed to put all the vital paper information into their computers and then still kept all the paper files as well as back up disks. What do we do with all this paper? We push it around for a while, moving it from one stack to another, leave it in the in basket for a few days hoping it will disappear, maybe, and eventually file it - where it will probably never be looked at again. Less than 5 percent of filed documents are ever referred to after the first year, but we store them for years, "Just in case. " The solution? Devise ways to screen it more effectively, to handle faster the papers that do get through to you, and review skeptically the ones you are accustomed to creating. Be willing to try some unconventional techniques: Stop sending a report to see if anyone misses it! Setup system and stick to them. Purchase large wastebaskets and investigate possibilities for recycling. * From The Time Trap Third Edition, by Alec Mackenzie

19 Andrea Audio Test Labs 19 Superbeam Array Microphone Speech Recognition Performance 5.0 Test Data 5.1 Microsoft Office XP Speech Engine Complete Test Results Small Office Open Cube Conf Room Office Office Voice Office Microphone Setup Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy Accuracy (%) (%) (%) (%) (%) (%) (%) Close-talking 1" Andrea NC-61 Close-talking canceling headset NC-61 with PureAudio Plantronics DSP300 USB NC headset Far-field 18" Andrea Superbeam Array Microphone (Array2-S) Telex M Telex M-30 with PureAudio Notes to Office XP test data 1: All test percentages are the average of at least three consecutive runs. Tests performed 4/19/2002 to 4/29/2002 2: Enrollment is repeated for each setup and each location 3: SoundMAX Cadenza TM V3.2 Drivers composed of: AEAudio.sys driver version and Smwdm.sys version : Computer P4 1.6 GHz, running Windows XP