Middleware Based User Interface Migration: Implementation and Evaluation

Transcription

1 Middleware Based User Interface Migration: Implementation and Evaluation Timo Salminen, Simo Hosio & Jukka Riekki Department of Electrical and Information Engineering, University of Oulu, Finland Abstract User interface can be seen as one of the greatest challenges in mobile computing. One promising approach to make interaction with mobile devices more efficient and pleasant is utilizing UI devices in the user s environment, such as, public displays. However, it is not clear to what extent the user is able to benefit from using external UI devices. Clearly, it also depends on the user s current task. In this paper, we present a middleware based approach that enables migration of user interface from a mobile phone to a device with better UI capabilities, such as, to a desktop computer. We quantitatively and qualitatively evaluate the benefits of such an UI migration to the user with two use cases that are common to mobile phone users; SMS or writing and establishing a phone call. The evaluation bears evidence that the method greatly increases the performance, and especially the enjoyability, compared to the mobile device s own interface when typing messages. Users prefer our approach because it provides faster and more convenient way to e.g. write a long SMS message. On the other hand for simple tasks, such as selecting a receiver when establishing a phone call or typing a very simplified message, the UI migration was found to be highly undesirable. Categories and Subject Descriptors C2.4. [Distributed Systems]: Distributed applications; H.1.2 [User/Machine Systems]: Human factors General Terms Experimentation, Human Factors Keywords UI migration, ubiquitous middleware, user evaluation 1. Introduction Whilst performance of mobile devices has been continuously increasing, physical size of the devices has become very small. This trend sets strict limits to the user interfaces of mobile terminals which typically include a simple, often rather small, keypad for input and a small display and loudspeakers for output. Research has been done for finding solutions to improve user interaction in mobile devices. One promising approach, especially from the ubiquitous computing point of view, is utilization of public displays and other devices in the user s environment. This enables application to temporally transfer its user interface, for instance, to a larger and more suitable display as requested by the user. In the same manner, the user could utilize a standard keyboard as an input device instead of the small keypad of a mobile device. In both scenarios the application would run securely on the user s mobile device whilst the user interface of the application would physically reside somewhere else. The public display and keyboard approach may be highly useful especially if the user is willing to sacrifice his mobility for finishing his task more efficiently with better but stationary user interface. However, from the application developer s point of view, the described user interface migration is very complex and case specific. Instead of using standard UI libraries the developers must now take into account the challenges of distributed computing, such as, communication failures, UI event handling, security and more. Application developer s overload can be considerably relieved by providing developers with services and necessary APIs that facilitate development of applications with UI migration support. This can be achieved, for example, with a middleware that masks the complexity of networks and distributed systems and thereby allow developers to focus on application-specific issues [Bernstein 1996]. Middleware has been successfully utilized in many pervasive computing systems, including [Chetan et al. 2005], [Messer et al. 2006] and [Han et al. 2004]. In the solution described in this paper, the same approach has been taken to facilitate development of applications that support migrating user interfaces. In addition to providing support for the developers, objective of our work is to evaluate benefits of UI migration in a mobile phone from the user s point of view. We have built a prototype system along with applications that demonstrate UI migration. In our implementation the user can transfer user interface of the application in the mobile phone temporally to a desktop computer. For most of the people, mobile phone is a device for making phone calls. It has also been pointed out that 65 percent of the mobile phone users use SMS messaging daily [IDC]. These are the cases we

2 chose as basis, although the UI migration concept could easily be extended to various other application scenarios such as simplifying the complex menu structures when configuring phone s numerous settings or organizing contacts and s. We study how the user benefits from UI migration when making a phone call or writing an SMS message. Our research questions are how the user experiences such interaction and whether the migrating UI approach is beneficial in these applications from the user s point of view. Rest of the paper is structured as follows. Related work is presented in the Chapter 2. Chapter 3 presents our system framework and prototype implementation. Test environment and test scenarios are presented in the Chapter 4. Evaluation is presented in the Chapter 5. The paper is concluded in the Chapter 6 with discussion and future work considerations. 2. Related work Savidis et al [2005] introduced a Voyager development framework for supporting implementation of ambient dialogues. In their system, a small processing unit carried by the user, is continuously looking for surrounding wearable wireless gadgets that are capable of hosting interactions. When such devices are detected, they are employed for providing user interface to the processing unit. If some device that is currently in use goes out of range, the ambient dialogue is halted and resumed again when a corresponding device is detected. The Voyager system relies on a discovery mechanism, a query protocol and a control protocol. Discovery mechanism is used for detecting surrounding UI devices. Query protocol is used by the processing unit to chase up the capabilities of detected devices and to determine if the device can satisfy its needs. Finally, the control protocol is used for UI event delivery between the processing unit and UI device. Somewhat similar user interface solution can be found in [Bauer et al. 2001]. In their work, Bauer et al used a component-based software framework called DWARF for building augmented reality systems. In DWARF system, the user interface was separated from the application logic by a specialized component; user interface engine, which provided platform independence of user interfaces and multi-modal user interaction. If some application required user interface in DWARF system, it asked user interface engine to build it by providing an abstract description of desired UI. UIML-based user interface description language was utilized for describing user interfaces in the DWARF system. Bertini et al [2004] defined an interaction model for designing internet based applications that automatically adapt to different mobile devices. Building blocks of their system are abstract interaction units (AIUs) and UML activity diagram. AIUs define a single view in the user interface; data to be shown to the user as well as user interaction and information exchanged between the user and the system. Activity diagram connects AIUs together and defines how the user interface is composed to complete a certain task, such as, to book a hotel room. This kind of interaction model can also be applied for public displays since application logic is separated from the user interface. Instead, the application defines what kind of user interface it needs and individual UI device can build the required user interface. However, the proposal is intended to be used in only very limited set of internet based applications. Nylander et al [2005] provided the Ubiquitous Interactor (UBI) to address the problems of design and development that arise from services that need to be accessed from many different devices. The UBI offers a possibility to develop one device independent version of the service and then create device specific user interfaces for it. The UBI uses interaction acts to describe user-service interaction in a device independent way. This description is used by all devices to generate an appropriate user interface. Thus, the application developer does not need to stick into fixed set of UI elements provided by the underlying platform. Instead, the user interface is represented by abstract description which is given to the interaction engine. Interaction engine translates between description and the user interface. Presentation of user interfaces can moreover be controlled through customization forms, which contain service- and device-specific information on how user interfaces should be presented. We, on the other hand, present a middleware based solution that offers the application developers an easy way to implement UI migration enabled applications. By doing this we also enable the application users to make the decision themselves whether to initiate the UI migration or not, and moreover, where to migrate the UI. In this paper we present a survey of user experiences about utilizing such a system and study to what extent the user can benefit from switchable user interface method while using a mobile device in tasks that require keypad input. 3. System framework In our previous research, we have introduced a middleware for supporting the development of ubiquitous and contextaware applications [Riekki et al. 2006]. The middleware provides applications with services that are common to this application area including distributed communication, service discovery, context information delivery, and more. Middleware and its components are illustrated in Figure 1:. Figure 1: Applications are built on top of the middleware and using components distributed in the environment. Our middleware divides functionality into atomic entities called components. Each component is specialized in some task or domain area and provides corresponding functionality to the other entities in the system and to the applications. Furthermore, it places the commonly used functions in independent components, so they can be

3 shared across the platforms and software environments. The middleware and its components are distributed to the devices in user s environment. To the application layer, however, services that the middleware offers are represented as a single, coherent system. A component can represent a service or some resource in the user s environment. The components live inside a container called an engine. Each device in our system hosts one engine. The lowest layer of the engine, core, includes the basic services for the applications. An engine may also contain an unspecified number of value-adding components that extends functionality of an engine. Applications are developed on top of the middleware by utilizing services it offers. When an application is initiated, it can start using any component in the environment by asking it from a core component called component management. Component management is responsible for granting and giving access to the other components in an environment User interface component User Interface component is one of the most important services the middleware has to offer for the applications and developers. Driving force behind the UI component design was to keep application logic separate from the user interface itself. This supports also our goal by making user interface migration a relatively straightforward process. Several UI components may exist in the system and the application may select which one to use. The UI component provides the developer an abstract method to describe the user interface of an application in XUL [Mozilla XUL] based language. Example UI description is listed in Figure 2:. The description contains two labels, two textboxes and a button. The described UI can be used e.g. for querying and phone number from the user. When an application starts, it usually gives description of its user interface to the User Interface component running on the local device. The user interface is then built by the User Interface component using the GUI libraries of the underlying platform. Naturally, it is up to the currently employed UI component, how the user interface is rendered and how it actually looks like. <?xml version="1.0" encoding="utf-8"?> <box id = "querybox" align="center" orient="vertical"> <label id="label1" value=" "> </label> <textbox id=" " multiline="false" readonly="false"> </textbox> <label id="label2" value="phone number"> </label <textbox id="phone" multiline="false" readonly="false"> </textbox> <button disabled="false" id="button1" label="ok" association=" ,phone"> </button> </box> Figure 2: Description of UI with two labels, two textboxes and a button The user interface description is composed of UI elements and UI controls that the application needs as well as their related data. All elements and controls have unique identification code. Moreover, UI controls can be associated to the UI elements as necessary. The application can have for example a button control which is associated to a text field element. When the control is triggered, the data written to the text field element is sent to the application as a button control event. The UI event handling is taken completely care of by our middleware. This way the application is transparent to the UI component and conversely the UI component is transparent to the application. Only thing the UI component knows from its clients is application identification code and a network address where it is located. This information is used for delivering the events to the desired destination. 4. Test environment and scenarios Our prototype system is presented in Figure 3:. The system is composed of a Nokia 7610 mobile phone and a standard desktop computer (Windows XP, service pack 2). In the mobile phone side, we have developed an SMS application, which utilizes services provided by middleware. In addition to core functionality of the middleware, two user interface components exist in the environment. One is running on the mobile phone and another one on the desktop computer. In addition to standard configuration, the mobile phone is equipped with an integrated RFID reader. When touching an RFID tag, the reader software framework sends an event including the data stored in the tag to all its subscribers. We utilize RFID system to initiate the UI migration. When SMS application is started, it employs the UI component on mobile phone to build its user interface. The UI component running on the PC is represented by an RFID tag which contains identification data of the component. When the user touches the tag located on the screen, or perhaps on the keyboard, with a mobile phone, the application is notified with an RFID event and it starts the UI migration procedure. At the end, the User Interface component on the PC is employed by the application and its user interface is shown on the screen of the desktop computer. Communication between the mobile phone and the PC is done using Bluetooth (RFCOMM profile) as it is convenient and reliable for wireless personal area networking. All server side functionality was implemented using Java 5.0 SDK and serial communications API. Components on top of Symbian OS were built using C++ and Series 60 SDK v.2.1. Figure 3: Test environment architecture

4 We examined the system with two scenarios that demonstrated common mobile phone use. At first, the test user was asked to write SMS messages with lengths of 2, 10, 20 and 50 characters. Secondly, the user was asked to make a phone call by choosing a receiver from the list of 200 available contacts in the phone s contact list. We created two UI descriptions, one for each scenario. The description was used by UI components on mobile phone and on the desktop computer to render corresponding user interface for the application. A simple script with a label, textbox and an ok button was used for displaying the message writing application. The other one with a list box and an ok button was for choosing the receiver on PC side. At mobile phone side choosing the receiver was done using an authentic Series 60 contacts selection dialog provided by available native application engines. This was done in order to keep the selection as natural as possible to the test users, since most of the current Series 60 phones employ the same selection dialog. Figure 5: illustrate the user interfaces generated from the description on both mobile phone and PC. Migration of the UI from mobile phone to a PC was launched by the user by touching an appropriate RFID tag. This way the process for Bluetooth service and device discovery was avoided, and the connection establishment was highly faster, as testified in [Hidden for anonymity reasons]. At the beginning of the tests, the overall setup and usage of the devices and interfaces were shown and made sure to be familiar to the users. This way time variation caused by learning was eliminated. In the first case, the user typed the given text first with the mobile phone. After this, the same text was written by using UI migration, i.e. transferring the application user interface to the desktop and using a traditional keyboard. Total time to finish the task was measured with both methods. After finishing writing of all four texts, users started the second scenario. At first the users had to select a given receiver of the phone call from the list of 200 possible contacts using phone s native contact dialog. Total time for the transaction was measured. After this, the same operation was done with UI migration. User interface on mobile phone with all 200 contacts was transferred to a desktop computer. This time, the user made the selection with mouse and a scrollable list box. Figure 4: SMS application user interfaces. Left: text editor on mobile phone. Right: same UI migrated to a desktop computer Figure 5: Receiver selection user interfaces. Left: receiver selection dialog on mobile. Right: same UI migrated to a desktop computer 5. Evaluation In the first case we measured the absolute time from the moment the user started the writing procedure to the point that the text was ready to be submitted as an SMS. Without the UI migration, the timer started when user chose to start writing from menu and stopped when the ok key was pressed. With migration, timing was started when an appropriate RFID tag for the service was touched. In this case, also an intermediate time was taken when the Bluetooth connection was up and ready to be used. Final time was again marked when the written text had fully arrived via the same Bluetooth socket back to the mobile phone. Second case timed the interval between starting of the contacts selection dialog or touching the tag to move the UI to server and selecting the desired receiver. As in the first case, a checkpoint time was taken when the connection establishment was accomplished. The time required in sending the actual XUL or the text written at server via BT is negligibly small and thus left without further attention. Altogether 15 test users were used in evaluating the system. Of the group, fourteen (14) were male and one (1) female. The average age was 25,6 years with a standard deviation of 2,6 years. The group was very technologyoriented, and working among computing business. As easily noted, some of the factors of this kind of research are very hard to address: the varying typing skills of testers, customs with different keyboards, suitableness of the used mobile phone s keypad to different users etc. We aimed to familiarize the users carefully with the setup before tests and minimize the effect of environment. This all taken into consideration, the results can not be universally generalized to all mobile device users, but are rather giving a good hint on what might be the impact of such services in modern, computer-oriented society. After the performance tests, the users were also made a questionnaire about the usability and convenience of utilizing UI migration instead of finishing the given task by using just the mobile phone alone.

5 5.1. Quantitative evaluation When evaluating the efficiency and times of writing, no predictive input (e.g. T9) was used. All the texts were written using traditional mobile phone writing technique with a normal Nokia 7610 keypad. Users were all familiar with the keypad layout and functions offered by the mobile phone (e.g. where to find special or capitalized characters). The profit of using UI migration varied a lot according to the test in question. As one can easily predict, when typing a short, compact, message such as ok or similar, migrating the UI for more capable and better I/O is not worthwhile at all. The times consumed for each of the test cases can be seen in Figure 6. For a text of 2 characters the average time of writing at the device itself was 3,7 seconds with a standard deviation of 1,2s. The average time with the migration proved to be somewhat bigger, 5,8 seconds (stdev 1,1s). Thus, for short messages the migration brings rather harm than advantage, since the user is forced to take extra steps (initiate migration, switch the input/output device) and effort in writing. Figure 6: Total time for each writing task with both methods As we observed longer texts, the advantage gained from better input grew dramatically. Already with a text of 20 chars, the time consumed for writing with the mobile phone s keypad (30,5s, stdev 11s) was over three times as big as the corresponding time with the migration (10,1s, stdev 1,6s). At these lengths, still being relatively short, also the standard deviation also starts to take role in the total time when talking about writing with the mobile phone. If user types a full, 160 characters long, or even longer SMS message, the time saved might well be minutes in case of some users. When examining the selection of the receiver, no question remains which way is more efficient. Figure 7 shows that the time for selecting the receiver with UI migration (13,8s, stdev 3,7s) is over 2 times slower than with the traditional way (6,1s, stdev 1,7 s). Selection of the receiver was also already considered to be as simple and fast as desired in mobile phones, as the following chapter brings forward. Simple t-tests were performed to verify the significance of the results above. The biggest value gained for p was (the 10 character case), thus verifying that the results are also statistically significant User feedback evaluation When comparing the described methods also qualitative feedback of the users has to be taken into account. All test users filled in a short questionnaire after finishing the tests. When it comes to writing an SMS message (generally text), only one out of fifteen testers thought the traditional way to be more comfortable. This was because of the test person was very accustomed to the keypad of a mobile phone and saw no need for a better method. Anyhow, a number of good aspects in the traditional way were reported. Users liked it because it was easy, simple and no extra actions were needed to type the message, that is, the initiation of the UI migration and using an external device for writing (PC). Privacy was mentioned indeed in many questionnaires as a shortcoming. When using a small screen in a personal mobile phone, other people are not able to see what the user is typing. After UI migration to the public display, the situation is highly different. Major shortcomings mentioned in using mobile s own keypad was slow typing with clumsy interface compared to a bigger and more advanced keyboard. Also special characters and upper/lower case switching caused trouble. Ten of the test users liked UI migration because it was a lot easier to use compared to traditional method. Also typing and editing of the message was faster with migration. One test person preferred the migration because using mouse and keyboard was familiar and safe. Large screen was mentioned as a positive thing in two cases. Especially special characters were thought to be easier to find with a normal keyboard. Anyhow, 3 users were not ready to use such services if they existed in real world because of possible digital security flaws and especially with privacy issues when using large, public screens. Downsides of the UI migration were listed also. Privacy and security being the major ones, additional steps to start typing and immobility while typing were mentioned as well. Even fear for diseases spreading via public devices was brought out. In the second scenario, only one user preferred to select the receiver with UI migration. Reason for this was that the selection of a receiver for a message was reported to be already really fast and convenient. No improvement suggestions were made. Additional migration and changing the interface only took time and made the whole transaction more complicated. However, three of the respondents said they would use such services in real-life anyway. Figure 7: Times for selecting the call receiver with both methods

6 6. Discussion and conclusions We presented a middleware support for facilitating the development of applications that support migrating user interfaces. We developed a prototype application on top of the middleware components. Using the prototype, we evaluated benefits of the UI migration with two common tasks for mobile users and compared the actual efficiency between the migration and traditional method. A pure time based efficiency evaluation and a user survey about the experiments was provided. The plain times measured are supporting and providing background data rather than being universal averages, since there are many variables present in this case. From the technological point of view, utilization of UI migration and public displays introduces a number of challenges. Security and privacy are probably the most important, as notified also by some of our test persons. The main security threat in UI migration concerns communication between application and the UI device. The situation is clearly similar to using any web service where user s private data is accessed through web browser. The application, let s say, an online payment service is running on web server of the bank whilst user interface of the application is shown to the user locally on his/her PC. The attacker may use several techniques to access the ongoing transaction, such as, man-on-the-middle attack, eavesdropping, manipulation and more. Thus, a system where UI migration is supported has to deal with the common security threats related to internetworking. On the other hand, the security issues in question are quite well understood and the existing methodologies can be applied also to UI migration. In addition to security issues, solutions have been introduced for improving privacy for public screens. For instance, Sharp et al enhanced privacy in their system by hiding private data on public displays and filtering possible input events from the terminals [Sharp et al. 2006]. Various use cases exists where UI migration could be utilized in real life. This is due to fact that user interface of mobile terminals are inefficient for complex tasks. Technologically speaking the solution presented in this paper is already possible. Mobile phones have been equipped with various radio interfaces for communication for a long time. Furthermore, RFID is becoming a common technology and it can be utilized in session setup. Some mobile phone models today already offer RFID capabilities. As we pointed out in this paper, UI migration makes mobile phone use more efficient when user is writing text longer than just a few characters. Since smart phones today have capabilities, the UI migration could be useful for many users if deployed widely. It is easy to imagine an internet café, which provides public terminals to the customers with keyboard and screen. All personal data and the applications are located in the user s mobile terminal. However, during the usage of the device the external UI devices are utilized, which enhances usability and efficiency of the application user interface. User is able to watch movies or edit documents and use his device in a lot more convenient and enjoyable fashion. The whole concept of migrating UI and utilizing public devices is a fascinating one and we are going to continue work on it. As for user tests, we aim to deploy an updated system in public space, and go for larger surveys. For security, research is planned by developing the middleware onwards. This kind of systems might easily lead into unwanted situations of leaked personal information such as s or contact information. In general, user interface migration offers a great number of possibilities, but there is a mass of work to be done to make it an everyday luxury. 10. References Bauer, M., Bruegge, B., Klinker, G., MacWilliams, A., Recher, T., Riss, S. and San, M Design of a component-based augmented reality framework. In Int l Symp on Augmented Reality, New York, USA, Bernstein, P.A Middleware: a model for distributed system services. Communications of the ACM. Vol. 39, Num. 2. ACM Press, Bertini, E. and Santucci, G Modelling internet based applications for designing multi-device adaptive interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces. Chetan, S. Al-Muhtadi, J. Campbell, R. and Mickunas, M.D. (2005) Mobile Gaia: a Middleware for Ad-hoc Pervasive Computing. The Second IEEE Consumer Communications and Networking Conference, pp Han, S., Bong, Y. and Youn, H A New Middleware Architecture for Ubiquitous Computing Environment. In: Proceedings of the Second IEEE Workshop on Software Technologies for Future Embedded and Ubiquitous Systems (WSTFEUS 04), May 11-12, Vienna, Austria, pp IDC homepage, Survey on Mobile Data Service Usage. 3_ accessed September Messer, A. Kunjithapatham, A. Sheshagiri, M. Song, H. Kumar, P. Nguyen, P. and Kyoung Hoon Yi InterPlay: a middleware for seamless device integration and task orchestration in a networked home. Proc. of the Fourth International IEEE Conference on Pervasive Computing and Communications (PerCom 06), pp Nylander, S. Bylund, M. and Waern, A Ubiquitous Service Access Through Adapted User Interfaces on Multiple Devices. Journal on Personal and Ubiquitous Computing, Vol. 9, Issue 3, May 2005, Springer-Verlag, London, Riekki J., Salminen T., and Alakärppä I Requesting Pervasive Services by Touching RFID Tags. IEEE Pervasive Computing, vol 5, issue 1 Savidis, A., and Stephanidis, C Distributed interface bits: dynamic dialogue composition from ambient computing resources. In Journal on Personal and Ubiquitous Computing, Vol. 9, Issue 3, May 2005, Springer-Verlag, London, Sharp, R. Scott J. and Beresford A Secure Mobile Computing via Public Terminals. In Proceedings of International Conference on Pervasive Computing, Springer Berlin/Heidelberg, pp The homepage of Mozilla XUL documentation, accessed April 2007.