ISCA Archive. Usability Engineering as approach to Multimodal Human Machine Interaction 1. INTRODUCTION. Robert Neuss

Transcription

1 ISCA Archive Usability Engineering as approach to Multimodal Human Machine Interaction Robert Neuss Usaneers GmbH Abstract: This paper describes a user-oriented approach to a multimodal onboardcomputer system for use in cars. The intention of a prototype design was to demonstrate multimodality under real conditions, so most of the user-tests took place in a driving-simulator environment. The system features speech, gesture and manual input as well as a GUI and speech output and is developed in five steps with systematic user-tests. All the subjects experiences were taken into account when redesigns were accomplished. The basic principle of the system is that the user can change the input modality at almost any time, but cannot use different channels simultaneously ( serial redundant multimodality ). Key words: multimodal man-machine-interaction, usability-engineering, speech recognition, Wizard-of-Oz 1. INTRODUCTION At present multi-modal human machine interaction is the focus of intensive scientific research. Apart from many technical issues like the influence of the environment on input devices like automatic speech and gesture recognition (ASR and AGR) important aspects of multi-modal applications are usability, suitability and consumer acceptance. This paper introduces an user-oriented approach to a multi-modal onboard computer system for cars based on previously performed studies. These experiments analysed the behaviour of subjects confronted with simulated systems based on various in- 1

2 2 Robert Neuss put and output channels or their combinations respectively. The knowledge gained through these tests enabled the design of a prototype which has been integrated in a driving simulation environment and that is serving as platform for studies and improvements. Over a one-year period the final prototype emerged through five development iterations that contained systematic usability-testing and redesign. This generated system provides about 30 selected functions that cover the domains communication, navigation and entertainment as well as scenarios like listbox-item selection with or without visualisation, selection and editing of proper names, numeric inputs and manipulation of graphical objects like maps. A major design emphasis for the multi-modal system was to meet the application specific demands (distraction, mental load, safety, total task times, robustness etc.) using the input channels control element (two push-turn-knobs for menu navigation and volume with two keys for speech and back ), speech and gesture as well as the output channels display, speech (prerecorded speech in CD quality) and general acoustic feedback. 2. PROTOTYPE DESIGN The main goal of the prototype design was to accomplish an user adequate interpretation of monitored user actions although it was obvious that recognition and input errors will occur. Therefore a very clear but unobtrusive user guidance, robust but powerful speech recognition, gesture recognition and a graphical user interface that shows input possibilities unambiguously had to be implemented. Moreover the realization of the prototype s first stage showed that there is a fundamental increase of the lines of code needed to provide a state machine handling multi-modal inputs because the user can put his plans into action using any sequence of provided input channels. 2.1 System Architecture One idea to reduce the effort for building a state machine was to treat comparable user inputs with the same event handler. When inserting a telephone number for example, the user can confirm a number by selecting a control element on the screen reading ok with the push-turn-knob, saying okay (after activating the ASR) or making a go gesture in the camera surveillance area (looks like pointing a gun to the screen ). To realize this behaviour a data fusion stage has to evaluate and pre-process incoming messages from the input devices. The state machine receives the ok or go message respectively containing a tag with the input source (speech, gesture

3 Usability Engineering as approach to Multimodal Human Machine Interaction 3 or control element). Now the system can respond to the event and can generate an input-source dependent feedback (i.e. if the user said ok the system will respond the number will be dialled, whereas when the user pressed a button there will be no speech output). Input Speech- Recognition Gesture- Recognition Preprocessing Statemachine Response Manual- Device(s) Figure 1. Datafusion with preprocessing stage. Function and usability tests not only confirmed this approach, but also showed that in some situations subjects preferred certain input channels or it didn t make sense to use other channels. This fact provokes further effort reductions, because in selected states it is not necessary to consider the full range of input channels. For example the usage of dynamic gestures for numerical input or virtual operation of control elements is not practicable (from a technological view this is very costly too). The state machine itself is embedded in a more complex architecture written in Visual C++. Input Channels Manual Input & GUI Wizard AGR ASR Control- Element GUI PreRecordedSpeech PRS Manager Log State Machine Command System Variables Figure 2. System architecture. The input devices are connected via sockets, so the system can be distributed on multiple machines. Because user input with the push-turn-knob is

4 4 Robert Neuss related to the GUI corresponding actions are sent to the GUI first, whereas other control elements cause messages sent directly to the data fusion stage. The method of having a separate logic for the GUI helps to keep the state machine compact and reduces the amount of information to be processed. According to Figure 2 data fusion and preprocessing is performed by the module named Manager. The Manager adds time stamps to the messages and decides in what order the messages are processed by the state machine. The Manager also handles barge-ins, for example when a speech output is running and the user presses the push-to-talk button the speech-output is discontinued. The state machine has a hierarchical structure with the aim of being as compact as possible with no unnecessary code. For monitoring reasons there is a log mechanism. A dedicated module named Command provides the technical infrastructure for accessing media, network, the GUI etc GUI The GUI is optimised for a small display with 4:3 ratio and a diagonal of 5.5. The control elements are placed in a round fashion with reference to the movement of the push-turn-knob, the number of menu items is limited to 5 or 6 entries. a) b) Figure 3. Screenshots of the main menu and a function menu. In the right picture a push-turnknob has been cut in to point out the correlation between layout and knob rotation. The top line shows the menu path whereas the status bar at the bottom displays information in condensed form. Here it shows the recognition states of AGR and ASR, a progress bar for the speech recognition window, the ambient temperature, date and time, a volume bargraph, an indicator for active phone calls and texts like radio stations or recognition results (commercial systems may show additional phone, clima, audio and/or navigation properties). Because there are space restrictions and minimum font sizes the prototype features a flexible status bar that switches contents.

5 Usability Engineering as approach to Multimodal Human Machin e Interaction 5 For example, when a phone call is active the audio device will be automatically switched off anyway, so the phone icon can be displayed instead. Otherwise it may not be necessary to show the volume indicator permanently, here it will be shown for some seconds when the driver uses the volume control. The recognition window is displayed in a similar fashion. Text information like recognition results or radio stations will be presented as popup in the middle of the status bar. a) b) c) d) Figure 4. a) Default view of status bar. b) Speech recognition in progress. c) Text message (here recognition result). d) Volume bargraph. Because some users (predominantly elder people) didn t notice displayed recognition results or the recognition window an alternative presentation as a coloured popup on the screen might be more adequate, but it is more intrusive (this might be a candidate for a configurable or even adaptive feature). There is always a discussion regarding distraction of the driver when popups are used. Here all popups are limited to the status bar and most of them are immediate feedback to user interaction, so their potential of distraction is relatively low (of course for a quantitative measure experiments with eye-trackers would be mandatory). Other text messages that are estimated more urgent like incoming phone calls or service indications ( check control ) are combined with sound events ( earcons ) to attract the user on purpose. One important role of the GUI is to present function names to the user he can use with the speech recognition ( speak what you can see ), so in the main area of the screen icons are only used as addendum for optical reasons. 2.3 Speech Dialog To realise speech dialog the utilized speech recogniser ASR1600V3 from Lernout & Hauspie has a vocabulary of about 500 commando words and names (for laptop presentations the number of names has been reduced to 2000). There is an effective garbage model, some function names

6 6 Robert Neuss have synonyms and there are some variations for numerical inputs (e.g. CD number 1, fist CD, CD 2, first track ). To enable a powerful speech input the most functions can be used all the time, the vocabulary is only reduced during dialog steps explicitly containing numbers and names. An important feature is the possibilty of shortcuts (functions with parameters). Using a multipass recognition on an utterance with dynamic vocabulary switching the system can process sentences like Bring me to X-Street in city Y without the necessity of having thousands of city-names active permanently (the multipass recognition procedure takes seconds according to CPU power). Even if the city isn t recognised correctly in the second pass the intention programming of new destination is detected in the first pass. This can be used to switch to the destination-editing menu and ask for the city name. Due to the static grammatical structure of the vocabulary these shortcuts only work with fixed phrases, but small variations and garbage modelling helps to provide a sufficient usability. Another unusual feature is keyword-initialising. As response to the poor acceptance of the push-to-talk button the user can start the speech dialog by speaking the isolated keyword Computer. This feature has been carefully developed and works well with only very few unintentional detections. An earcon is presented to signal the opening of the recognition window when the user initiates the speech dialog (there is also an earcon when the window closes). If the (speech) dialog has more than one step the system responses with speech output and starts the recogniser again shortly before the output text is finished (this is because users tend to speak too early ). So the system will not respond during text output, if the user wants to barge-in he has to press the push-to-talk button. The dialog is finished when a function is actually used or when the dialog is terminated by the user (or by the system when too many timeouts or rejections occur). Most of the speech commands result in an optical feedback when the menu is changed or a function is activated, but there are some commands that trigger none or only small graphical changes. Especially commands directly related to music produce an instant acoustical feedback, so there is no need for changing the menu (apart from the popup of the recognition result in the status bar). For example, if the user says Music off the music will turn off, additional speech output is superfluous. 2.4 Gesture Interaction One of the early prestudies showed that communication with gestures is an attractive enrichment for the interaction. When used as abstract commands that are not related to geometric points (deixis) or control elements (virtual operating) gestures have a good potential to be used as a comfortable

7 Usability Engineering as approach to Multimodal Human Machine Interaction 7 and nondistracting input media because under these conditions they can be performed without looking at the display. When the multimodal prototype was developed sufficient knowledge about gesture interaction already existed, but the gesture recognition itself was not robust enough yet. So instead of AGR a Wizard-of-Oz scenario was built up, featuring a human wizard who observed subjects during the tests and used a separate speech recogniser to create an input for the multimodal system. a) b) c) d) Figure 5. Some gestures: a) Ok ( Engage ), b) no or cancel ( something is wrong ), c) next/forward or previous/back ( left and right ), d) Zoom In or Zoom out (pull nearer, push away). Meanwhile the AGR developed at the Technical University of Munich makes progress and can be demonstrated in a standing car using cheap hardware components. 2.5 Multimodality The principle idea behind the prototype is that drivers can choose input channels at almost any time according to their personal preferences or to the modalities advantages (speech for selecting functions and using shortcuts, control element for adjusting values and safe operation, gesture for convenient application of single commands). There are obviously the above mentioned restrictions: you can t use two channels simultaneously and some modalities have more possibilities than other. This means, a user can start a dialog with clicking his fingers in the camera surveillance area to start the speech recognition. Then he can use a speech command like dial number to quickly access the corresponding menu, and maybe he doesn t like dictating numbers so he uses the push-turn-knob to enter the numbers and makes a hookoff-telphone-receiver-gesture to start the call. Further there are interesting details like the feedback strategy or the question if to start the speech recogniser when input modalities are changed. Here the first assumption is that in most cases 1 users expect a voice feedback when they make speech or 1 This applies not to actions that directly influence acoustical output of music like changing radio stations.

8 8 Robert Neuss gesture input 2 because these modalities are predominantly used blind while driving. Another assumption based on the author s experience is that once subjects overcome the barrier to use speech recognition they use this modality till they achieve their goal or they think another modality is significantly better to handle. So in this prototype speech input has a high priority, and once it is active it stays active, only the recognition window is prolonged when another modality is used in between. A simpler alternative would be to stop the speech recognition when another modality is used, so if the user wants to switch back to speech he has to start the ASR again. The second method has been used for another prototype and has the advantage that the user has a better control over the speech recogniser, but there is no comparison of the systems yet. 3. USABILITY TESTS Within the prototype development five test series were performed. Those tests can be divided into 3 groups. In the first group the number of tested functions was quite small (phone application with 4 functions). Test 1 served to find weaknesses of the system whereas test 2 focussed the effectiveness of the realized dialog system. The second group of tests covered the full function set built up with the methods that have been evaluated in the first step. Test 3 was only a function test to get the system error free, but test 4 had a more sophisticated design to get some quantitative results. Test Series # Number of Subjects Sessions Selection of Subjects Driving-simulation Demographic Data Interviews Scenarios Tasks Tutorial Free Exploration Group Comparison Videoevaluation Cognitive Walkthrough Statistic Auswertung Benchmark Σ Duration / min If the anticipated system reaction is delayed (like the calculation of routes for a navigation system) the user expects also speech feedback after manual input.

9 Usability Engineering as approach to Multimodal Human Machine Interaction 9 Figure 6. Features of the five test series. It can be seen that series #1 and #2 took less effort than the other series, especially #5. Selection of subjects means that the uses where carefully selected by the characteristics age, gender and technical background. Finally the system was refined with gesture input and served as a platform for the last test. This test was split up into two sessions within one week and was accurately analysed to get statistical data about multimodal interaction. The video analysis took about three hours per subject and all user actions were logged in a database where statistical correlations could be extracted afterwards. Besides technical testing the user tests helped to optimise the system in various aspects like function names or feedback texts. A very important roll of usability testing was to achieve a good discrimination of the ASRvocabulary and to gain training material. The tests also pointed out user needs that led to the development of advanced features like the speech shortcuts, keyword initialising and an adaptive tip-assistant. The tip-assistant presents short voice messages in very carefully selected situations. E.g. the dictation of telephone numbers works best if the numbers are spoken fluently. Because many people don t have experience with speech recognition they think they have to speak numbers one by one slowly and exaggerated, with the result that timing errors occur and the dialog takes a very long time while being not very satisfying. From the view of suitability it does not make sense to adapt to the user and use a special slow-speech model, because the user doesn t want to speak slowly, he just doesn t know better. Here the tip assistant decides that after three single spoken numbers it will generate a message like the machine works best if you speak numbers fluently like The tests proved that once subjects were presented the number-tip, they spoke numbers the optimal way and never use the old method again, even in the second test session some days afterwards (the number-tip had the best acceptance of all integrated tips).

10 10 Robert Neuss Figure 7. Statistical da ta from the 5 th test when the 8 subjects were asked to dial a specific number while driving in the simulator. The task was given five times (three times within the first session and two times in the second session). Left: Total task time in seconds. Right: Total number of dialog steps (starting the speech dialog and choosing a function in a menu manually is counted as one step). The last test provided statistical data on the full system. Among many results the data showed that even people with no technical background were able to use the system effectively and efficiently after a short learning period (see figure 7). The subjects made use of all modalities, where speech was the preferred input channel (speech 73%, gesture 15%, manual input 12%). The users made extensive use of modality switching while carrying out a plan, but there were clear interpersonal differences according to individual preferences. While 38% of tasks were solved only by speech, the combination speech and gesture occurred 30% (speech and manual input 12%, gesture 7%, speech + gesture + manual 7%, manual 5%, gesture + manual 0,3%) the modality was changed about 0,88 times per task. Benchmarks proved that under normal conditions (not to much background noise and easy driving conditions) multimodal input is faster than manual input 3. In conclusion the extensive tests with the final prototype proved quantitatively that users accept the multi-modal concept. This can be derived from the steep learning curve of the users, the fact that the subjects made context specific use of different input methods, the low error rates and the result of the benchmark tests. Another qualitative analysis indicates that in favour of robustness sometimes it is advantageous to reduce the number of provided input alternatives and apply a strong user-guidance. 3 Of course this is due to the good integration of the speech recognition and the relatively fast response of gesture interaction.

11 Usability Engineering as approach to Multimodal Human Machine Interaction REFERENCES Sharma R., Pavlovic C., Huag T.S.: Toward Multimodal Human-Computer Interaction. Proceedings of the IEEE, Vol 86, No 5, pp , IEEE, Neuss R., Usability-Engineering als Ansatz zum Multimodalen Mensch- Maschine-Dialog. Dissertation, Lehrstuhl für Mensch-Maschine- Kommunikation, Technische Universität München Zobl M., Geiger M., Bengler K., Lang M.: A Usability Study on Hand Gesture Controlled Operation of In-Car Devices. Poster Proceedings HCII New Orleans, USA, pp , Geiger M., Nieschulz R., Zobl M., Neuss R., Lang M.: Methods for Facilitation of Wizard -o f- Oz Studies and Data Acquisition. Proc. of the 9th Int. Conf. on Human-Computer Interaction (HCI International 2001), New Orleans, Louisiana, USA, Ed.: Lawrence Erlbaum Ass., New Jersey, Poster Sessions: Abridged Proceedings, pp