Speech Tuner. and Chief Scientist at EIG

Transcription

1 Speech Tuner LumenVox's Speech Tuner is a complete maintenance tool for end-users, valueadded resellers, and platform providers. It s designed to perform tuning and transcription, as well as parameter, grammar, and version upgrade testing of any speech application. With this GUI-based tool, companies developing speech applications on various ASR platforms (including Nuance and ScanSoft) can bring speech application tuning in-house and avoid professional service fees. LumenVox is on the cutting edge of speech technologies and customer satisfaction by supporting not only their own speech engine, but other leaders in the industry as well. Bruce Balentine Executive Vice President and Chief Scientist at EIG Why Do Companies Need the LumenVox Speech Tuner? No untuned speech application survives contact with actual customers. Tuning is an absolute requirement for every speech application deployment. Our Tuner allows you to quickly assess changes and upgrades. Tuner Capabilities: Evaluate and improve the speech recognition application Analyze each stage in the call process Transcribe audio data, make pinpoint adjustments, and immediately measure the effects on performance Test changes against actual calls immediately Analyze data collected using different ASR engines Test design and development decisions of new applications, using data from deployed applications 26 27

2 Tuner Overview The Speech Tuner is comprised of several key functions and windows. SRE Custom Tags displays log information supplied by the application s developers. The Call Log displays the list calls and controls the display of information in the rest of the Tuner. Each interaction (or turn) in a call is marked with an event type, such as: <Green Dot>: Beginning of the call <Green Waveform>: Speech event <Keypad>: Touch tone event <Red Waveform>: 'No input' event; the application did not detect any speech or touch tones <Yellow Question Mark>: 'Unknown' event; these events are typically ASR-specific <Red Dot>: End of the call The Transcription tool allows a transcriber to type the text of the caller's speech. Transcriptions are automatically evaluated and stored in the database for use in performance evaluations. The Statistics window displays performance statistics directly related to the calls in the Call Log window. The Answer window shows recognition results. Full n-best support is available, as well as the semantic interpretations, and actual words recognized. 28 Listen to the application prompt, and the caller's pre- and postprocessed speech. The recognized words are displayed where the ASR found them within the caller's speech. Vertical bars, which are useful for detecting problems, indicate the beginning and end of each word. 29

3 Tuning Processes LumenVox's Speech Tuner provides full support for LumenVox's Speech Recognition Engine, Nuance 8.5, ScanSoft OSR 2, and other ASRs. The Speech Tuner allows you to work with any supported ASR via a single interface. LumenVox is an active supporter of the Tools committee in the VXML Forum, and is working to help define standard logging information, to help ease the tuning process. The tuning process involves three easy steps: Import Data The basic process is simple. Users import call log data into the Speech Tuner database. All information stored by the call log is available in the Speech Tuner. In most cases, log fields between ASR engines are very similar; when the information differs, every effort is made to preserve the original data. Each special case is fully documented. Transcribe Speech. Transcribers can type the text of the caller's speech directly into the Speech Tuner. Once the audio is transcribed, the Tuner compares audio transcripts with the speech engine results to determine accuracy, greatly reducing errors associated with hand evaluations. If semantic interpretations are available, the transcriber can also mark whether the semantic interpretation was correct or incorrect. The transcripts are evaluated using the actual decode grammar, producing measurements such as word-error-rate, in- and out-of-grammar rates, and semantic error rates. Test Immediately. Selecting an interaction in the Call Log automatically loads the associated audio and grammar into the Tester. The grammar can be edited, speech engine parameters set, and individual recognition tests generated. The Speech Tuner natively supports industry standard SRGS grammars. Once a set of possible changes is identified, users can batch test audio to evaluate performance, using those changes. The Speech Tuner assumes the user possesses licensed versions of the relevant ASR, that the ASR platform is up and running, and that the platform is able to accept connections. LumenVox Speech Tuner Database The Speech Tuner communicates with an open-source, freeware database called SQLite ( The Speech Tuner manages call log importing, searching, and exporting so users can focus on the task of tuning, not log management. The database is contained in a single file, is easy to back up and transport, and can be queried using SQL-92 (see the SQLite website for full details) from a variety of exterior tools. Other speech engine vendors are free to convert their native logs to ones the engine understands. The format, content, and semantics of the LumenVox Speech Tuner database are published. The database maintains all the information contained in the original call log. The Speech Tuner includes not only the decode grammar and ASR results, but also the decode platform, parameter settings, alternative results, prompt audio, and pre- and post-processed audio. Depending on the platform logging capabilities, the database can provide more advanced information, such as ASR result alignments within the audio; the list of phonemes used in the decode; and word, utterance, and semantic interpretation confidence measurements. In addition, the Tuner stores all transcripts and evaluations within the call log. As transcripts are entered into the Speech Tuner, they are automatically evaluated against the decode grammar. These transcripts, and any notes or additional information, are stored directly into the database. Individual scores such as word error rate, semantic error rate, and in- and out-of-grammar measurements are stored along with their alignments, as well as information about how the scores were reached. Users can generate a variety of reports from these results, including error rate by grammar or dialog, confusion matrices, transcription progress, and confidence thresholds for confirmation or rejection settings. In the future, LumenVox's Speech Tuner will also support back-end database replacement, for use in enterprise level systems, where multiple users will be analyzing the same data simultaneously. Companies who use an ODBC-capable database can replace, with certain SQL changes, the diskbased SQLite system with an enterprise system such as MS SQL Server 2000, MySQL, PostgreSQL, and/or Oracle. LumenVox has created speech recognition products that are easy to code with and GUI-based tools, such as the new Speech Tuner that greatly simplifies post-deployment maintenance. Vern Baker President of engenic Corporation 30 31

4 Taking Out the Guesswork Make changes to grammars, parameters, or ASR engines, secure in the knowledge that those changes will make the application better, faster, and more accurate. The Speech Tuner uses historical information to validate your changes, ensuring your success. Grammar Tester Most 'tuning' tools are passive log viewers, requiring that changes be made in the live speech application and retested over a period of time with live callers. With LumenVox's Tuner, we send the changes to the Speech Engine, simulating the recognition process and evaluating changes instantly. Instead of slow, non-interactive, static tuning, the Speech Tuner enables on-the-fly, highly interactive, dynamic tuning. Make a change, do the test, get the results! The Grammar Tester is a dynamic testing component. You can switch ASR engines, grammars, and engine search parameters on-the-fly, and test changes in single or batch tests. Grammar Evaluation Evaluate speech and grammar sets against the speech engine, as they took place during the actual call. Adjust grammars and instantly re-test and re-score to evaluate improvements in performance. With LumenVox's Speech Tuner, you can instantly determine whether adding a new phrase to the grammar will improve your accuracy. Parameter Evaluation Setting parameters optimizes the speech engine performance, further improving the caller's experience. Traditionally, changing ASR parameters is a difficult and time-consuming task, often requiring long delays between changing a parameter, and evaluating its effects on performance. Our Speech Tuner can dramatically shorten the process. The dynamic test capability of the LumenVox Speech Tuner allows the user to quickly make and test parameter changes: now, ASR engine parameters such as search optimizations, speech end-pointing, and n-best result processing can be easily adjusted, and immediately re-tested and re-scored from within the Speech Tuner. Performance Measurements The Speech Tuner rates performance against commonly accepted measures like WER (Word Error Rate), Grammar Coverage, and Semantic Interpretation matching. This helps give an accurate picture of details such as average confidence scores, correct versus incorrect responses, and In-Grammar versus Out-of-Grammar performance. Assessing Upgrades Installing new versions of platforms and ASR engines entails a certain risk with each new upgrade. On occasion, new default settings, search routines, changes to acoustic models, and so on will actually worsen the caller's experience, until the application is re-tuned. But using the LumenVox Speech Tuner, you can perform baseline testing with the old version to establish the minimum acceptable performance. Then, using the upgraded version of the ASR engine, you can easily re-test all existing data and compare the results to the baseline. The new performance, judged against the baseline, gives you the information you need to make a decision, and deploy an upgrade with confidence

5 Tuner Reports The LumenVox Speech Tuner defines several pre-built queries for most common reports. The reports are generated using SQL queries against the Tuner database, with results produced in a pre-defined XML format. The format, content, and semantics of the LumenVox Speech Tuner database are published: if you need to extract data in logs that are not provided in the Tuner interface by default, you can easily produce custom reports by writing SQL commands. Tuner Tips When a problem occurs with a transaction, determine if the fault lies primarily with prompts or grammars. Never make a change in your call flow or grammar for just one failed call. Train acoustic models with environment noise in play. Train acoustic models to allow for caller dialects and regional pronunciations. Tune grammars, prompts, and all system parameters. Remember that accurate transcriptions need to account for noise. Speech Understood Speech Understood The Speech Tuner is an excellent fully integrated tool for improving speech applications. Bruce Balentine Executive Vice President and Chief Scientist at EIG 34 35