Hands-off Use of Computer towards Universal Access through Voice Control Human-Computer Interface

Transcription

1 Hands-off Use of Computer towards Universal Access through Voice Control Human-Computer Interface Dalila Landestoy, Melvin Ayala, Malek Adjouadi, and Walter Tischer Center for Advanced Technology and Education Electrical & Computer Engineering Department Florida International University W. Flagler Street, Miami FL 33174, USA ; Abstract: This study presents a user friendly programming tool developed to assist persons with motor disabilities in communicating with computers via voice commands. The main purpose of the study was to develop a human computer interface (HCI) able to convert voice commands into computer actions. To this end, an interface was developed for facilitating the communication between MS Speech Recognition Engine and the computer operating system MS Windows such that the dictated commands are executed as if they were triggered by mouse or keyboard operations. Experiments were conducted to statistically establish a confusion matrix to assess the recognition accuracy of the voice-controlled HCI. Validation results from 15 subjects yielded a system performance with an 84% accuracy in the recognition of commands, and interestingly and unexpectedly, in most of the subjects, a high confusion degree between voice commands such as Mouse Right Click and Mouse Drag command was found. Such an outcome seems to be an issue of the MS Speech Recognition Engine which calls for an investigation as to why such a confusion arises in order to initiate further improvements of the voice interface. Key-Words: Speech Recognition, Motor Disability, Voice Commands, Confusion Degree. 1 Introduction The HCI design proposed in this study abides by the design principle less obvious but powerful, and the notion that people with disabilities should be able to use products and services like everyone else does, and if assistive or adaptive technologies are needed, they should be as free of constraints as possible. Therefore, this design approach is one that seeks intelligent software developments that will support our goal for universal access without burdening the users with any aspects of intrusion or overwhelming them with complex control mechanisms that are time consuming and difficult to integrate. The interface of choice is one that would be multimodal in order to address the different constraints imposed by the disability. The proposed voice-based interface is destined however for those individuals with severe motor disability but can still make use of their voice in order to initiate the proper commands. As designed this HCI is augmented with real-time software voice-induced commands for real-time applications, and to facilitate human-computer interaction in a userfriendly manner. For web browsing, , and editing. This is a major undertaking that will result in an alpha prototype that is reliable, accessible, cost effective, and certainly compatible with up to date assistive/adaptive technology based interfaces and with common computer-based platforms. Current speech recognition (SR) systems have evolved technologically to begin having an impact in HCI designs [1, 2, 3] that can be used by disabled individuals to interact with computers. Speech recognition in this case refers to a technology that runs on a personal computer and converts spoken words into text in a word processor [4, 5]. Words are entered into the computer via a microphone which is connected to a sound-card. Companies devoted to the commercialization of speech recognition systems have became more successful as the

2 technology became more viable. Currently, SR systems are being produced and used by a large number of companies [6]. Inherent problems to these systems continue to be related to the time needed to train the system in order to produce a voice profile that is unique to the user, and in minimizing training errors which result in unrecognized voice commands [7]. Other problems are in the scalability due to increasing vocabulary and in the adaptability of the engine to different users. Modalities such as mouse button operations by voice commands have unfortunately not been widely used to assist motor disabled individuals [8]. The main thrusts for this study are (1) to develop a user friendly speech recognition computer interface for motor disabled persons at low cost, and (2) to test and validate the system on different subjects for a performance evaluation. 1 General Design of the System The proposed application is a combination of several technical resources such as Microsoft Visual Studio.NET [9], User32 libraries, Microsoft Access database [10] and Microsoft Speech Recognition Technology [11]. The main functionality is enclosed in the code implementation using Visual Basic.NET as the programming language, which is the bridge to communicate with the different components used in the development of this application. Its basic structure consists of an on screenkeyboard that receives events from the voice recognition engine as illustrated in Fig. 1. The.NET application is the on-screen keyboard, whose input is a human voice. Once this input is received by the application, the speech recognition engine is initialized, and it loads its grammar rules from the XML file. During that process, the recognition event gets the results obtained from the engine and the result is passed on as an argument to a function call in order to execute the desired command. Finally, the execution of command is passed to the operation system, and the result can be physically seen, for example when the mouse moves to a certain direction. Fig.1 Block Diagram of the recognition process 2 Speech Recognition Engine The SR engine was instantiated and initialized with grammar rules from an XML file, which forms the basis of the SR engine vocabulary. In order to create the recognition engine object, a recognition context, which is the primary means by which an application interacts with a speech application programming interface (SAPI) for speech recognition, was created by using a two steps process. First, the context was declared and then, it was created by instantiating the specified class with a reference to a new object. By doing this, the SR engine was initialized. All applications on the machine using shared recognition contexts share a single audio input, the SR engine, and grammars. When the user speaks, the SR engine will do recognition, and SAPI decides which context to send the recognition result to, based on which grammar best matches the result. Furthermore, an IspeechRecoGrammar object was created, which enables applications to manage the words and phrases for the SR engine. After that, the grammar was explicitly created. In this specific application, a command and control grammar is used, which limits a word list by restricting the speaker to a small set of words. By doing this, users can speak a command, usually a single word, with greater chance of recognition. Moreover, it is necessary to initialize the grammar by loading it from a file. This was achieved by the CmdLoad

3 FromFile method, which loads a command and control grammar from the specified file, and provides two main parameters. The first parameter is the complete path where the XML file name is located in the machine, and the second one is the load option, which specifies whether the grammar is to be loaded for static or dynamic use. In this case, the grammar was loaded dynamically. Finally, the grammar was set to active by using the state property. All these previous steps are executed when the form is initially loaded. A SR application has a wide variety of events, of which most are by default active. Once the application receives a recognition event, the Recognition code is invoked automatically and immediately. Besides, each event is associated with a particular recognition context. There are two important possible results from a recognition attempt: Recognition event and FalseRecognition event. Therefore, in the development of this computer interface, these two relevant events were used. 3 XML File Description The voice commands that the application responds to are defined in an XML file that is dynamically loaded into the Grammar object at runtime (Table 1). Each command in the file is considered as a grammar rule that is defined with an ID name and value, which is unique and identifies the spoken word that is to be recognized by the SR engine as a voice command. In the tags that define a command, attributes can be set to specify precedence order of a particular word over another. This enables the use of phrases as commands rather than single words. For example, the engine would recognize the words Mouse Click as a command, but not the words Click Mouse, because the word Mouse has precedence over the word Click and it is recognized as the first part of a command. If the engine successfully recognizes a command from the XML file, it will include the associated ID in the event that passes to the application. The text grammar format is composed of XML tags, which can be structured to define the phrases that the speech recognition engine recognizes. The formal XML schema for the text grammar format is defined in a separate document, called XML Shema:Grammar. Grammar rules are elements that SR engines use to restrict the possible word or sentence choices during the SR process. SR engines employ grammar rules to control the elements of sentence construction using the predetermined list of recognized word or phrase choices. This list of recognized words or phrase choices contained in the grammar rules forms the basis of the SR engine vocabulary. Table 1: Commands defined in the XML file Voice Command Resulting Action Mouse Click A click is simulated at the present cursor position Mouse Double Click A double click is simulated at the present cursor position Mouse Right Click A right click is simulated at the present cursor position Mouse Move Left A move to the left is simulated Mouse Move Right A move to the right is simulated Mouse Move Down A move down is simulated Mouse Move Up A move up is simulated Mouse Drag A right down click is simulated Mouse Release A right up is simulated 4 Mouse Functionalities In this study, the problem of mouse operation was addressed by importing several API (application programming interface) from the User32 libraries in order to communicate directly with the operating system. Most of the functionalities of the mouse were implemented using a combination of API functions and SR technologies. In order to accomplish this, the mouse_event function was imported to synthesize mouse events and motion. Its parameters are extremely important since each one of them will provide significant information for implementing desired tasks. The dwflags specifies various aspects of mouse motion and button clicking. Parameter dx specifies the mouse's absolute position along the x-axis. Similarly, dy specifies the mouse's absolute position along the y-axis. Other important constants were imported in order to be passed as a parameter to the mouse_event function. To implement the mouse click event, a subroutine called MouseClick was created. A similar process was used when implementing the mouse double click action. To achieve this,

4 the MouseClick subroutine is called twice. Another subroutine was implemented to produce a right click action. In addition, a timer component, which allows the execution of a specific event at a defined interval, models a smooth transition of the cursor position in the specified direction at every 100 microseconds. Since the time interval is very small, the mouse easily navigates through the monitor as if manually executed. To achieve the desired movement in a certain direction, the coordinate parameters (x and y) are passed with different values. By passing the mouse's absolute position equal 1 or -1 in the x and y direction, the mouse will move one pixel at a time. To perform faster movements, a higher number is passed (for example 5 or - 5). In order to stop the mouse from moving the stop method of the timer component is invoked. 5 Training In order to use the software, the user needs to train the computer with the voice by using the included speech utility, to improve the recognition efficiency of the engine. This is needed because the engine itself is not user independent; it will only recognize someone s commands if it was previously trained to this subject s voice by building his or her own profile. The recognition profile can be updated for improvements. The profile could be trained by reading several sessions, which are short stories or fables that will help to interact with the training process. In addition, since the system is voice sensitive, noise should be kept to a minimum when speaking to a system. 6 Results A convenient tool for analyzing the results of this system is the confusion matrix, which is a matrix containing information about the actual and recognized commands. In this matrix, rows C i (i = 1,2,.,15) represent the actual commands, and columns C i * (i = 1,2,.,15) represents the commands recognized by the program. The commands to be executed as specified column wise (C i ) are recognized by the program as C i *. A perfect classification thus results in a matrix with 0's everywhere but on the diagonal. A cell which is not on the diagonal, but has a high count signifies that the command of the row is somewhat confused with the command of the column by the voice recognition system. In this study, 15 subjects were used to test the SR interface, a confusion matrix tool [12] was used to evaluate the performance of the SR engine. Table 2 shows the confusion matrix for this experiment. The symbols used for the commands are listed in Table 3. Table 2: Confusion matrix obtained after conducting the experiment, showing absolute values. It can be observed that the numbers of incorrect recognitions are not precisely low. Voice Commands Recognized Commands C*1 C*2 C*3 C*4 C*5 C*6 C*7 C*8 C*9 C*10 C*11 C*12 C*13 C*14 C*15 C C C C C C C C C C C C C C C

5 Table 3: Symbolic commands representation Symbol C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 Meaning Mouse Move Left Mouse Move Up Mouse Move Right Mouse Move Down Mouse Move Left Faster Mouse Stop Mouse Double Click Mouse Move Right Faster Mouse Right Click Mouse Move Up Faster Mouse Click Mouse Move Down Faster Mouse Drag Mouse Release Unrecognized Command The confusion matrix was used for finding pairs of commands that were most often confused. For example, C1's state of confusion with C2 is given by the amount of C1 s commands classified as C2 as a percentage of the amount of C1 s classified as either C1 or C2. Table 4 shows the confusion state matrix obtained in this study. From these results, it is observed that the highest confusion is between command C9 (Mouse Right Click) and C13 (Mouse Drag) with % confusion rate. A confusion matrix was obtained for every subject. Column 2 of Table 5 shows the correct classification rate per subject, while column 3 displays the highest rate of confusion and specifies the corresponding command pair. Fig. 2 shows the state of confusion of each command with respect to the others. The Correct Classification Rate (CCR) is the proportion of the total number of recognitions that were correct based on the total number of vocalized commands. It is determined using the equation: # of correct recognitions CCR = (1) # of given commands 6 Conclusions This study presented an integrated HCI approach to voice control commands as means to facilitate access and use of computers to persons with motor disabilities. Efforts were extended in the simplification of the user interface for real-time applications with viable performance. The developed application can be easily installed on a home computer, at no additional cost, and trained for later use without major difficulties. The system evaluation yielded what seems to be an internal problem in the MS Speech Recognition Engine, since a pair of commands was repeatedly incorrectly recognized across most of the subjects during the experiments. A logical continuation of this study will be the installation of the system in the homes of some motor disabled persons. This study is also intended to be extended to motor disabled patients from Miami Children s Hospital and will involve a patient follow up for future system enhancements. Table 4: Confusion state matrix obtained between voice commands and recognized commands. Voice Commands C Recognized Commands C*1 C*2 C*3 C*4 C*5 C*6 C*7 C*8 C*9 C*10 C*11 C*12 C*13 C*14 C*15 C C C C C C C C C C C C C C

6 Table 5: Confusion rate analysis based on individual subjects. Correct Classificati on Rate Highest Confusion Rate Value Subject (%) (%) Command C5 vs C C9 vs C C9 vs C C5 vs C C10 vs C C12 vs C C9 vs C C9 vs C C1 vs C C5 vs C C9 vs C C9 vs C C5 vs C C9 vs C C2 vs C15 Average C9 vs C13 State of Confusion C 9 C 5C1 C12 C 4C 6 C 2 Com ands C10 C 8C 3 C11 C 7 C14 C13 C15 Fig. 2: State of confusion of the voice commands based on the 15 subjects study. References [1] Becchetti, C., Speech Recognition Systems: Theory and C++ Implementation, Wiley, [2] Riverdeep Corp. Website. Just Speak Naturelly [Online]. Available: /04/front speech.jhtml [3] Hideki, H., Yoshifumi N., Yoichi, T., Speech Recognition Interface for General Purpose Workstation, IPSJ SIG Notes Human Interface, Abstract No , [4] Holmes, J., Holmes, J. N., Holmes, W., Speech Synthesis and Recognition, Taylor & Francis, Inc., January [5] Schroeder, M. R., Computer Speech: Recognition, Compression, Synthesis, Springer Series in Information Sciences, 35, Springer Verlag, July [6] Out-Loud. Website, Computing Out- Load [Online]. Available: [7] Ainsworth, W. A. ; and Pratt, S. R., Feedback Strategies for Error Correction in Speech Recognition Systems, International Journal of Man-Machine Studies 36(6), , [8] ScanSoft Inc. Website, QualiLife Selects ScanSoft Speech Technology to Help People with Disabilities Achieve Greater Independence [Online]. ses/2002/ _qualilife.asp [9] Beamer, G., "Upgrading Microsoft Visual Basic 6.0 to Microsoft Visual Basic.NET" By Microsoft Press, July 1, [10] Howard, T., The Smalltalk Developer's Guide to VisualWorks,SIGS Books, [11] Microsoft Corp. Website, Speech SDK 5.1 for Windows applications oad/sdk51/ [12] Adjouadi, M.; and Ayala M., NeuralStudio (A Neural Networks Simulator) [Online]. Available: Acknowledgments The authors gratefully acknowledge the support from the National Science Foundation Grants EIA and HRD , and the Office of Naval Research Grant N