A Technical Overview: Voiyager Dynamic Application Discovery

Transcription

1 A Technical Overview: Voiyager Dynamic Application Discovery A brief look at the Voiyager architecture and how it provides the most comprehensive VoiceXML application testing and validation method available. Included Inside Abstract 3 Traditional Testing Limitations 5 Voiyager Testing Architecture 5 Dynamic Application Discovery 6 Issue Detection 7 Voiyager Browser Phone 8 System Requirements 10

2 Sales and Product Information \ Voiyager North Black Canyon Highway, Suite 100 Phoenix, AZ Syntellect, Inc. All rights reserved. Voiyager, Voiyager Dynamic Application Discovery, and the Voiyager logo, are trademarks of Syntellect, Inc. All other trademarks are or symbols are those of their respective owners.

3 Abstract Interactive Voice Response applications fill a critical operational role for many organizations. These systems have become the single largest touch point between businesses and their customers - one failed service interaction could result in lost business. It is this dependency on the reliability of IVR applications that drives the demand for these service-critical systems to be thoroughly evaluated and tested before and after they are placed into production. The development cycle of these applications is often pressured by the further speed of change within business. This leaves development teams faced with time constraints that threaten their ability to achieve careful, thorough application testing and validation. Notwithstanding, the pressure to maintain development cycles within budget constraints. The results of these pressures often translate to rushed testing cycles that ultimately only evaluate a small portion of the total application. Voiyager is introducing a new technology that will change the way VoiceXML applications are developed, deployed, and maintained. Until now there has not been a method that enabled the full evaluation and testing of a VoiceXML application. Only with Voiyager is it possible to completely validate the entire span of possible call transversals in a timely and cost effective manner. Voiyager automatically explores an entire IVR application from top to bottom. Now potential risks to customer satisfaction can be uncovered and resolved often and early during development and continually audited throughout the quality assurance and acceptance phases, and well into post deployment. Additionally, future production system updates can be applied with greater confidence using Voiyager to rapidly evaluate the potential effects of all system changes. This document will provide an outline on how the unique technology, made possible by Voiyager, is going to improve VoiceXML application development and empower development teams to maximize their testing efficiencies and deliver high quality self service applications in fractions of the time and cost. Voiyager Dynamic Application Discovery 3 v1.0

4 (This page intentionally left blank) Voiyager Dynamic Application Discovery 4 v1.0

5 Traditional Testing Methods Traditionally, Interactive Voice Response (IVRs) systems, also known as Automated Response Units (ARUs), have only been tested using a standard phone interface, simulating an actual caller. A test caller dials a number, follows the call flow procedure, listens to and interacts with the system responses, and attempts to validate the process against the specified business rules. For complex applications, this can be a very time consuming process and entails the efforts of several human resources (usually developers). Testing a system using large numbers of account variables while verifying all of the possible go right and go wrong paths is logistically impossible. With Voiyager this process is eliminated. Automated tools on the market are designed to perform load testing. Tests scripts have to be created that instruct the automated tool to call the application repeatedly, with the only variation potentially being a different account number or pin. These tools have some ability to detect deviations in response (length of pause between inputs, length of audio and whether or not the system under test hung up), but have no way to understand the actual behavior of the system under test. Additionally, the scripts created in order for these tools to function must be manually updated each time the application is changed or updated to accommodate changes in business needs. There are also some tools and development frameworks for testing web applications. However, all of those tools operate by statically analyzing the VoiceXML document returned by the application. A static review of the document will not tell the test software how the VoiceXML will interact with the caller. Current IVR Testing Challenges Telephony testing is time consuming Humans don t have time to test every error, every help, every path Currently available automated testing tools require extensive set up and script creation Exiting tools require that testing scripts are re-written when application changes are made Automated tools are designed for load testing and provide less coverage than humans Interpretations of the VoiceXML specification may vary from person to person Maintaining a consistent determination for problem areas is most difficult, especially if the testing phase is extended across several days or even weeks Errors detected and reported may not be repeatable The reporting system may be slow or imprecise Voiyager s Testing Architecture Voiyager performs the entire application analysis by interacting directly with the application and taking the place of the VoiceXML browser at the HTTP interface. At this interface, Voiyager has the ability to determine specifically what audio clips are played, what grammars are active, and therefore the valid choices that can be spoken or entered by the caller. Voiyager also bypasses the speech recognition engines that can complicate testing by failing to correctly identify input. As an added benefit, Voiyager can traverse an application much faster than a telephony based call processor as the typical input/response timing boundary is only limited by the HTTP request/response interface. Voiyager Dynamic Application Discovery 5 v1.0

6 Dynamic Application Discovery Process The Dynamic Application Discovery process starts with an exploration. It is through this process that Voiyager examines and scrutinizes a VoiceXML application, traversing each and every possible call flow path. Running an exploration is a simple process that can be accomplished in just a few steps: 1. A user provides Voiyager the application URL when creating a new project. 2. The user then creates an exploration task and starts it. 3. The discovery module makes simulated calls to your application exploring every go right and go wrong path in the application. 4. When complete, the user can review issues found during the discovery. Internally, Voiyager starts the process by making an initial page request to the VoiceXML application server. After the returned VoiceXML page(s) is parsed Voiyager fetches any dependent audio and grammar resources. Once all the necessary resources have been collected, Voiyager goes through the following analysis phases: State identification Grammar analysis Queuing of new calls State Identification Voiyager identifies a dialog state by the state s active grammars, output (audio and TTS) and its relation to other states within the call flow. As Voiyager explores a VoiceXML application it will continue to update its definition of a dialog state. State definitions are stored with the application project with the results of the exploration. This allows user s to tie their own display names to a Voiyager identified state. Grammar Analysis Voiyager parses SRGS or GSL grammars to find all paths that create unique semantic interpretation results. By default, Voiyager has been tuned to explore all grammars results that yield fewer than 100 unique choices. If a grammar has more than 100 choices (e.g. account number, pin number, phone number, postal code), Voiyager can be configured to stop or to try a user-specified number of random values. Queuing of New Calls After Voiyager analyzes the active grammars, it determines if the valid responses generate any new paths through the call flow. For each potential path, Voiyager queues a call to explore the path, leaving the one path to explore for the current call. Voiyager will also explore no input, no match and a hang up results Voiyager also allows application developers to specify a VoiceXML property to use as field names for state identification. User s can specify Voiyager s state identification method when creating a new Voiyager project. Voiyager Dynamic Application Discovery 6 v1.0

7 Issues Detected Through Dynamic Application Discovery After an exploration has been completed, a user can review all of the issues and potential issues that Voiyager has discovered. The user can review details regarding each issue as well as a complete transcript of each of the calls placed by Voiyager. As the user reviews suspect behavior, they have the option of identifying to Voiyager which behaviors are acceptable and which are not. Issues that can be detected by Voiyager: Critical Errors Critical errors are flaws within the VoiceXML application that should be addressed by developers. Browser was unable to retrieve data If Voiyager is unable to retrieve a document or application resource, the resource will be flagged. Default severity of the issue will change depending on the nature of a resource. For example, an audio with a specified TTS phrase, if missing, will be low severity whereas a missing audio clip without a TTS phrase will be considered medium severity. Grammar Compilation Error Indicates that Voiyager cannot parse or compile an application s grammar, but was able to fetch at least part of the document. Browser Error If Voiyager was unable to process a part of the VoiceXML document. This is often due to syntax error within the document. Suspect Behavior Suspect behaviors are application behaviors which may or may not be valid. User s can train Voiyager to suppress or escalate these issues. Application Hang Up Voiyager flags every occurrence of an application hang up. Application Transfer Voiyager flags every occurrence of an application transfer. Loop in Application Voiyager will exercise call flow loops to see if they have an exit point. For example, if help yields a message and leaves the user at the same state, Voiyager tries a configurable number of times to see if the application realizes the caller is trapped in a loop and diverts the call flow. If the call flow does not change, an issue is raised. Voiyager Dynamic Application Discovery 7 v1.0

8 Unexpected Behavior Change While Voiyager explores the application, data outside of Voiyager s control may change. For example, one call path may force a monetary transaction while another call path may play a balance. A shift in behavior, such as an account balance changing from one call to another, the user will be notified. Voiyager Requests Additional Information If Voiyager is unable to fully explore an application, the user will be notified of the problem and be given a method for resolving. input training. After providing input, the user can run the Voiyager exploration process again. Voiyager will explore the entire application again for each input provided. In most applications, dialog state training will be needed for account number and pin prompts, but it may be need for other complex input such as dates, phone numbers and postal codes. Testing data for user input, this is used in the current testing of an application, can me easily loaded into Voiyager. In this manner it becomes extremely simple to provide the training data for Voiyager. Dialog State Training Needed If Voiyager experiences a grammar that contains more than an acceptable number of unique inputs, the user can resolve the issue by providing caller Voiyager s Browser Phone Voiyager includes an interactive browser phone that allows users to rapidly perform individual call tests. Because the browser phone works with the same underlying logic as the Voiyager exploration process, grammar choices can be displayed in the input pull down. Voiyager Dynamic Application Discovery 8 v1.0

9 Resource Changed Voiyager creates a hash of all audio and grammar files fetched while processing a call. If a resource has changed, the user will be notified. State audio/tts changed If the order or selection of audio or TTS changes, the user will be notified. Voiyager Integration The product includes.net and Java libraries allowing developers to integrate Voiyager into their own Continuous Integration process. APIs include, but are not limited to, the following functions: Voiyager allows calls to be saved as bookmarks providing developers a method to rapidly recreate call problems until they are resolved. Bookmarks can also be exported and run by other users. This feature is most apparent during issue resolution where problematic calls can be exported and then sent to a developer where the problem can be recreated in a debugging environment. Application Comparison As previously indicated, Voiyager saves information regarding each of the calls placed during an exploration. At a later date, these calls can be replayed back to an application allowing Voiyager to discover how an application has changed. This feature can be used after making changes to an application to ensure the changes only exist in expected areas. The feature can also be used to insure the application in a production environment reacts the same way as it did in a development environment. Creating, editing and deleting projects Creating, editing and deleting exploration and comparison tasks Read task summaries Reading exploration and Comparison results, which include: Read Voiyager call model Read issues Read calls Place interactive browser phone calls Changes Detected During an Application Comparison: Call Flow Changed A Call Flow issue is raised whenever the call flow changes. For example, a choice on a menu now goes to a different or new state. This issue can also be raised when a state has significantly changed and is no longer identified as the previous state. Voiyager Dynamic Application Discovery 9 v1.0

10 Voiyager System Requirements Application Compatibility VoiceXML Voiyager has been designed to be VoiceXML 2.0 and 2.1 compliant. If an application has non-compliant components Voiyager will attempt to continue processing the document. Voiyager will ignore all tags that are not defined by VoiceXML 2.x. Voiyager uses Rhino for Java/ECMAScript processing. Grammars Voiyager supports inline or external grammars in the following formats: SRGS 1.0 with SISR 1.0 SRGS 1.0 with Nuance semantic interpretation Nuance GSL embedded within XML Skills and Training Requirements Software interfaces Basic VoiceXML application architecture concepts Product functionality (i.e. correct operation and purpose of product) How the application being tested should behave to different inputs Voiyager has been tested and certified with Apache Tomcat 5.5 and PostgresQL 8.2. Please contact Voiyager Product Management for support with other application servers and databases. Voiyager provides a Java and.net client interface to allow Voiyager to be added into an automated continuous integration development process. Memory constraints The Voiyager desktop client requires a minimum of 512 MB. The Voiyager server component requires a minimum of 2 GB of memory for a single user and application. Enterprise deployments with multiple applications and users of Voiyager will require a system with a minimum of 4 GB of memory. Operating system constraints The Server Components are not constrained by the operating system. The user interface components run on any version of Windows supporting.net 2.0. Data storage requirements The product will use a PostgresQL database for nearly all storage requirements. Storage use is less than 100MB per project. Voiyager Dynamic Application Discovery 10