Getting rid of OK Google : Individual Multimodal Input Adaption in Real World Applications

Transcription

1 Getting rid of OK Google : Individual Multimodal Input Adaption in Real World Applications Felix Schüssel 1, Frank Honold 1, Nikola Bubalo 2, Anke Huckauf 2, and Michael Weber 1 1 Institute of Media Informatics, Ulm University, Germany 2 Department of General Psychology, Ulm University, Germany {felix.schuessel, frank.honold, nikola.bubalo, anke.huckauf, michael.weber}@uni-ulm.de Abstract. Multimodal Interaction has the potential to significantly increase the ease of use of human computer interaction (HCI). At the same time, due to error-prone recognition based inputs, it is merely used in real-world applications. While literature on multimodal input fusion describes modeling of different user behaviors as a key for increased robustness, it still failed to prove it s practical use. This article presents the design of a user study that applies previous theoretical work on individual user adaption in a smartwatch scenario. We describe the practical implementation of a process for error recognition and recovery based on the history of multimodal inputs and a concrete scenario suitable for evaluating its practical impact on the ease of use. This could prove the real-world use of individual multimodal input adaption and finally lead to multimodal systems less cumbersome than today. Keywords: multimodal interaction, input fusion, user behavior, interaction history, input error detection and recovery 1 Introduction Multimodal systems, that make use of parallel inputs from different modalities, are topic of extensive research for several years now, but still, their adoption in real-world applications is almost non existent. This could be due to the fact, that error rates of sensor inputs are still too high for practical use. Recent work has shown the theoretical possibility to detect and recover from sensor errors within the fusion of multimodal user inputs by making use of a so called interaction history (cf. [2]). While these results are promising, they are based on a rather abstract gaming task. The actual implementation and evaluation of such an approach in a realistic setting has not been conducted so far. To overcome this, we propose a setting that uses a popular interaction with smartwatches as realistic scenario: quickly displaying some kind of information like upcoming tasks, news headlines, or recent messages. The next section elucidates the idea behind using an individual interaction history within multimodal input fusion. It is a summary of the work in [2] that provides

2 a rather abstract process for error detection and recovery. This is followed by the proposed implementation of such a process in an existing input fusion component [1] and it s necessary enhancements. After that, the real-world scenario for an evaluation is described. Finally, these contributions are summarized and their relation to the state of the art in commercially available systems is discussed. 2 Individual User Adaption of Input Fusion Research from Oviatt et al. and others repeatedly showed the existence of different user behaviors in terms of temporal relations between multimodal constructions [12, 11, 13, 6, 7, 9, 3, 5]. These are called integration patterns and describe the dominant use of modalities in simultaneous or sequential temporal manner (cf. [10]). Two modalities are used sequentially, when their time intervals do not overlap, but instead show a temporal gap between each other. In the opposite case, i. e. when their time intervals overlap, they are used simultaneously. This is depicted in Fig. 1. An integration pattern is called dominant for a specific user, if more than 60% of inputs are uttered in the respective manner. Though this classification scheme has been applied by others [8, 4], it s potential to increase the robustness of a system, as suggested in [13], has not been proven yet. Inputs Time intervals Integra-on Simultaneous Sequen-al Fig. 1. Simultaneous and sequential integration of modalities. In order to investigate the occurrence and use of such integration patterns in relatively short lasting multimodal interactions, in [2] an experiment where users repeatedly had to perform a visual search task was conducted. Given a set of colored geometric objects, users had to spot the unique color/shape combination present in each trial (see Fig. 2). Once this combination was discovered, users had to specify the position and color of the object in a multimodal way. Results showed, that users did not exhibit such dominant integration patterns as found in the more lengthy tasks of Oviatt and others. It is concluded that for short lasting multimodal inputs, an individual user history of temporal distributions may be more appropriate than a simple classification into dominantly simultaneous or sequential users. Schüssel et al. then present a process for the use of such individual interaction history for error detection and prevention within the input fusion processing. This process can be used to detect and recover from three types of errors: false negatives, false positives, and conflicts. Applied to the recorded data set, a potential decrease of error rates from 4.9% to 1.2% is reported. But these values remain only theoretical and contain a large

3 number of unknowns, as parts of the suggested process could not be applied on the recorded data, as acknowledged by the authors. In addition, the task at hand is rather artificial and thus results may not necessarily be transferred to real world applications. To overcome these limitations, next we will describe the actual implementation of the theoretical process before we describe the scenario that we propose to finally evaluate the increase of robustness. matrix with number- labeled objects /me/money bar color- labeled bu,ons User: blue Fig. 2. A user indicating the individual item s location using touch, and its color by using speech. (taken from [2]) 3 Realisation As stated before, the realization of the error detection and recovery process extends the existing input fusion component of [1] and is build around it, as visualized in Fig. 3. Although the actual fusion methodology is of no importance for the interaction history module, still there are some requirements that must be met by the overall implementation. To begin with, the temporal onsets and offsets of all recognized events that lead to a fused input must be forwarded to the history data base, because they form the basis of the individual history for each user. In addition, the event times, i.e. the time an input event was raised by a sensor, must be tracked, too. This is needed in order to detect false negatives, i.e. when a sensor missed an input (cf. [2]). Of course, all those timestamps must be synchronized. The fusion itself must be able to respect those timestamps in the actual fusion process by fusing events based on their real-world onsets and offsets. This is of particular importance, when events are raised long after the input was performed by the user. This can happen, if a sensor takes some processing time to decide on an input (e.g. the speech recognition used in [2] takes about 642 milliseconds), or if a missed input is discovered by the error detection process and recovered much later than the user performed the input. The multimodal input fusion component from [1], that is based on evidential reasoning, has been extended to meet these requirements.

4 Input fusion extended by user individual interac5on history Input Sensors recogni'on false nega've? no Mul'modal Input Fusion result conflict? no false posi've? no: final result Applica'on yes yes yes determine 'me frame of missed event select most likely result discard event request reprocessing individual interac'on history of current user update history Fig. 3. Overview of the implemented process of error recognition and recovery extending an existing input fusion component. 4 A Real-World Scenario for Evaluation To actually proof the advantages of an individualized interaction history and to overcome the drawback of the rather abstract scenario chosen in [2], a realistic application is inevitable. We propose to use a smartwatch scenario for several reasons. First, smartwatches offer very limited interaction possibilities due to their small screen size that limits the possibilities and usefulness of touch interactions. Thus, they could benefit a lot from multimodal interactions. Second, as a very personal device, smartwatches seem the perfect choice when it comes to user individual adaption. Third, typical tasks performed on a smartwatch are relatively short and therefore transferability of the previous findings seems reasonable. The basic task chosen is that of quick information retrieval. In order to avoid boredom, nine different types of informations are offered: inbox, weather status, traffic information, news headlines, flight status, sports scores, current heart-rate, todays step counter and upcoming tasks. The most natural way of retrieving this information is to raise one s arm and tell the smartwatch which information should be displayed. So the multimodal interaction is modeled with two interactions: the lifting of the arm and a simple speech command specifying the kind of information requested. Both interactions can easily be gathered by common motion sensors and speech recognition engines found in all modern smartwatches. For the actual implementation in Google s Android Wear however, we chose to reuse an existing speech recognition module running on a desktop pc, as it already brings the necessary onset and offset information not accessible with the current speech API offered by Google. For the detection of the arm lifting movement, we implemented a simple finite state machine that is fed with live data from the smartwatch sensors. These data are then sent to the multimodal fusion system extended by the interaction history processing as explained in the previous section. Once a valid combination of arm lifting and voice command is detected, the display of the respective information is triggered on the smartwatch.

5 To keep participants interest on a high level for the necessary number of repetitions, a slight gamification will be applied. Participants will have to answer simple questions about each requested information on a separate touch screen. For a more realistic context of use, participants will have to move between three positions in the laboratory all equipped with a touch screen for answering the current question. This should also avoid a too monotonous, unrealistic behavior. In addition, we will establish a not completely silent environment by constantly playing some kind of radio station in the background. For the actual evaluation of the benefits of user individual interaction history, a between subject design will be applied. One group of participants will perform the experiment without any adaptions, while the other group will use the actual adaption system that should reduce error rates in the course of the experiment. 5 Summary and Discussion While previous work [2] has only theoretically shown the usefulness of a user individual interaction history to detect and recover from sensory errors that occur during multimodal input interaction, the actual implementation and final proof in a real world scenario is still missing. For the actual implementation of the detection and recovery process, we identified several requirements that must be met by an actual implementation not covered in previous work. With the implementation meeting these requirements, a follow up study can now be realized for final evaluation. For this, we propose to use a smartwatch scenario for several reasons, as described in the previous section. The proposed quick information retrieval task is one of the main use cases of a smartwatch. Applying a multimodal interaction consisting of an implicit arm lifting and a short explicit speech command makes information retrieval much more easier than using interactions offered by the state of the art in commercially available systems. For Android Wear, for example, the user must raise his arm, explicitly activate voice input by uttering OK Google, before he can finally state the wanted information. The same holds true for the Apple Watch, where one has to activate voice recognition with Hey Siri or via an explicit press on the crown of the watch. Both ways to retrieve information take much longer and seem less natural than the short interaction we propose. Of course, the rationale for an explicit command to activate voice recognition is the avoidance of input errors. With the use of a user individual interaction history, however, many errors may be avoided in the first place. That is what we feel confident to prove with the work presented here, independently of the scenario. However, if error rates are low enough, introductory commands found in todays smartwatches may become entirely needless. 6 Acknowledgments This work was supported by the Transregional Collaborative Research Center SFB/ TRR 62 Companion-Technology for Cognitive Technical Systems, which is funded by the German Research Foundation (DFG).

6 References 1. Anonymized authors: Anonymized title. In: Anonymized editors (ed.) Anonymized book, LNCS, vol. Anonymized volume, p. Anonymized pages. Springer Berlin Heidelberg (2013) 2. Anonymized authors: Anonymized title. In: Anonymized proceedings. p. Anonymized pages. Anonymized series, ACM, New York, NY, USA (2014) 3. Dey, P., Madhvanath, S., Ranjan, A., Das, S.: An exploration of gesture-speech multimodal patterns for touch interfaces. In: Proc. of the 3rd Int. Conference on Human Computer Interaction. pp IndiaHCI 11, ACM, New York (2011) 4. Dumas, B., Ingold, R., Lalanne, D.: Benchmarking fusion engines of multimodal interactive systems. In: ICMI-MLMI 09: Proceedings of the 2009 international conference on Multimodal interfaces. pp ACM, New York, NY, USA (2009) 5. Haas, E.C., Pillalamarri, K.S., Stachowiak, C.C., McCullough, G.: Temporal binding of multimodal controls for dynamic map displays: a systems approach. In: Proc. of the 13th international conference on multimodal interfaces. pp ICMI 11, ACM, New York (2011) 6. Huang, X., Oviatt, S.L.: Toward adaptive information fusion in multimodal systems. In: Renals, S., Bengio, S. (eds.) Machine Learning for Multimodal Interaction, MLMI 2005, Edinburgh, UK. Lecture Notes in Computer Science, vol. 3869/2006, pp Springer (2005) 7. Huang, X., Oviatt, S.L., Lunsford, R.: Combining user modeling and machine learning to predict users multimodal integration patterns. In: Renals, S., Bengio, S., Fiscus, J.G. (eds.) Machine Learning for Multimodal Interaction, MLMI 2006, Bethesda, USA. LNCS, vol. 4299/2006, pp Springer (2006) 8. Johnston, M., Cohen, P.R., McGee, D., Oviatt, S.L., Pittman, J.A., Smith, I.: Unification-based multimodal integration. In: Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics. pp EACL 97, Association for Computational Linguistics, Stroudsburg, PA, USA (1997) 9. Lee, M., Billinghurst, M.: A wizard of oz study for an ar multimodal interface. In: Proc. of the 10th international conference on Multimodal interfaces. pp ICMI 08, ACM, New York (2008) 10. Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions. Human-Computer Interaction 15(4), (2000) 11. Oviatt, S., Coulston, R., Tomko, S., Xiao, B., Lunsford, R., Wesson, M., Carmichael, L.: Toward a theory of organized multimodal integration patterns during human-computer interaction. In: ICMI 03: Proc. of the 5th international conference on Multimodal interfaces. pp ACM, New York (2003) 12. Oviatt, S., DeAngeli, A., Kuhn, K.: Integration and synchronization of input modes during multimodal human-computer interaction. In: Proceedings of the SIGCHI conference on Human factors in computing systems. pp CHI 97, ACM, New York, NY, USA (1997) 13. Oviatt, S., Lunsford, R., Coulston, R.: Individual differences in multimodal integration patterns: what are they and why do they exist? In: CHI 05: Proc. of the SIGCHI conference on Human factors in computing systems. pp ACM, New York (2005)