A Paradigm for Orientation-Based Universal Remote Control

Transcription

1 A Paradigm for Orientation-Based Universal Remote Control Clemens Holzmann, Stefan Resmerita, Michael H. Leitner, and Alois Ferscha Department of Pervasive Computing Johannes Kepler University Linz, Austria Abstract. This paper presents a novel paradigm for universal remote control of devices, where universality refers to the ability to control multiple devices with the same control artefact. Here, a device represents a physical appliance offering a set of services to the end user. Based on a tangible user interface approach, control actions are exerted by simple and intuitive physical manipulations of the control artefact, under the assumption that the physical device provides suitable feedback to the user. Rotation movements are employed for browsing devices and their services. Tilt movements are used for selecting/deselecting the device and the service to be controlled. Rotation is again employed to steer the input of the selected service. To apply these concepts, we consider the use of dedicated control interfaces, which are components that can be physically attached to different devices in order to present a uniform control interface with feedback capabilities between the user and these devices. The paper describes a fully functional application of the described paradigm in a domestic setting. 1 Introduction Most of today s electronic devices come with button-based remote control, which are often badly designed, unnecessarily complicated, bound to specific devices, they tend to get misplaces and their usage is not natural. These issues can be addressed by the tangible user interface (TUI) paradigm [1], where buttonbased control artefacts are replaced with physical objects whose manipulation allows for intuitive and natural expression of control. A fundamental issue is the mapping of artefacts with devices and the services they provide. This mapping can be done either a-priori or dynamically at runtime, thus giving the user the opportunity to control different devices with the same artefact. This paper presents an approach for dynamic artefact-device-mappings, where multiple devices and services can be selected and controlled with a single physical artefact. This approach is not only applicable for artefacts specifically designed for the purpose of remote control, but also for everyday objects like cups and pens with integrated inertial sensors in order to detect movements of the physical artefact. Considering progress in miniaturization of sensors and ad-hoc wireless communication, as well as upcoming passive RFID (radio frequency

2 2 identification)-based sensors which are powered by a reader and thus do not require a battery, this vision has already come within reach of current technology. However, the proposed paradigm is independent of used technologies. In order to control a device in our setting, the following sequence of operations should be applied: Browsing the controllable devices in the environment Selecting a device to control Browsing the services of the selected device Selecting a service to control Steering the input values of the selected service. We propose aggregating these operations and associating them with two basic manipulations of a physical control artefact: rotation and tilt. Thus, browsing and steering are exerted by rotation, while selecting (and deselecting) are executed by tilt movements. An add-on interface approach can be applied to translate from artefact movements to device-specific control actions on the one hand, and to provide feedback about the currently selected device and its state to the user on the other hand. Feedback to the user is of particular importance with regard to providing an awareness of available devices and services that can be controlled at a time. This can be achieved for example by visual and auditory means. There is a large palette of research addressing the issue of universal remote control [2] [3] [4]. Studies of the capabilities of physical objects as rich input devices are especially relevant to this paper [5] [6]. While flip, tilt, and rotation movements of artefacts have been employed for providing input to various appliances [6] [7] [8], we present here a systematic usage mode of these manipulations to address all the steps necessary for control, by a suitable aggregation and sequencing of manipulations. In particular, the usage of rotation for device browsing in combination with tilt for device selection is new, to the best of our knowledge. At the implementation level, this is combined with add-on control interfaces in order to provide a fully functional control system for today s devices. In the next section, the paradigm for universal remote control is discussed in more detail. Then, an application is presented, followed by conclusions and an outlook on future work. 2 A Paradigm for Universal Remote Control In general, the geometry of objects suggests manipulation affordances in a TUI. The geometry of a physical object defines a number of stable mechanical equilibria of the object when placed on a planar horizontal surface. In this paper, we consider control artefacts that have at least one stable equilibrium position. When the user intends to employ the object for control, he must hold the object in a stable equilibrium posture (without necessarily placing the object on a horizontal surface). Then, any rotation performed about a vertical axis will be interpreted as a control action. The operational steps mentioned above are executed as follows:

3 3 Device browsing: Upon grasping the control artefact and holding it in a stable equilibrium posture, a rotation about the vertical axis causes a browsing of the devices in the environment, according to a pre-specified order. This order can be determined by the placement of the devices in space around the user. Figure 1 illustrates this step, where a box-shaped object is used to browse among a radio set, an air conditioner, a TV set and a window s blinds. Appropriate feedback mechanisms must inform the user about the currently chosen device. Device selection: When the desired device is indicated, a tilt (or flip) movement selects the device. In other words, it indicates that the currently highlighted device is the one upon which subsequent control actions will be executed. At the end of this movement, the control object should be in a stable equilibrium posture. Service browsing: A rotation about a vertical axis will then browse the services on the selected device, as shown in Figure 2. As with device browsing, feedback mechanisms inform the user about the currently chosen service of the selected device. Service selection: Finally, a tilt movement as in the device selection step selects the currently chosen service for control. In the case of services with just two different parameter values, this movement for service selection causes a switch between the parameter values, and the artefact returns to service browsing mode afterwards. In the case of services with more than two parameters, the steering-mode is entered. Steering: The input values of the selected service are changed by rotating the control object about a vertical axis, as illustrated in Figure 3. The tilt-movement used for selecting and deselecting can be for example a down-up tilting, causing the artefact to end the movement in the same stable equilibrium where it started. Thus, the artefact always rotates about the same axis for both browsing devices or services and steering input values of services. A deselection of services and devices is achieved by tilting the control object in a direction opposite to the one used for selection (e.g. an up-down tilting ), causing the system to enter the respective browsing mode again. However, it is also possible that a simple down tilting movement brings the artefact to another equilibrium. In this case, the artefact rotates about different axis when browsing devices or services. For deselection, again a movement in the opposite direction is performed (e.g. up tilting ). Furthermore, if there is no change of orientation during a specified time interval, then the system is reset. That is, the next intentional control action will be device browsing. The advantages of the proposed solution are manifold. First, rotating an artefact in hand is a very intuitive movement for selecting devices that are scattered around the artefact, and it can be done in a simple and unobtrusive way just by moving the fingers. Second, this approach scales well to a large number of devices or services, as the total angle of rotation can be increased with the number of devices or services which are to be controlled. Third, it is only necessary to bring the controllable devices in a relative order defining the iteration sequence

4 4 Fig. 1. Browsing devices Fig. 2. Browsing services Fig. 3. Steering

5 5 of devices or services in browsing mode, and no additional location information is required as it would be the case with pointing postures. Fourth, no line-ofsight to the controllable device is needed, if wireless sensors are embedded in the artefacts and transmit their orientation information to a computer system in the background. Last but not least, the concept of add-on interfaces for translating artefact movements to device-specific control actions and providing feedback to the user allows to control arbitrary electronic devices with a suitable interface. 3 Application This section describes a fully-functional experimental application of the universal remote control paradigm, which has been implemented in a domestic setting. In the following, the used hardware and software are presented, and an application scenario is described afterwards. 3.1 Hardware and Software Setup As control artefact, we use a physical cube-shaped 3D model, which has been constructed with a rapid prototyping technology and can easily be grasped by hand (cf. Figure 4). The cube embeds a wireless InertiaCube3 orientation sensor from Intersense Inc. 1, which provides Euler angles (roll, pitch and yaw) as a full representation of the sensor s spatial orientation in three dimensions with respect to a fixed global reference frame. It has a size of about 31mm x 43mm x 15mm and is powered by a 9V battery for up to 8 hours of continuous operation. An image of the sensor and the respective block diagram can be seen in Figure 5. Fig. 4. Rapid prototype of a control artefact with embedded sensor As add-on control interface, we use Compaq IPAQ 3760 PDAs, which are equipped with wireless LAN adapters and have built-in IrDA-ports. They serve as interfaces between a central server (i.e. the control server) and the electronic devices that are to be controlled. The IPAQs run the Familiar Linux 2 operating

6 6 Fig. 5. Wireless InertiaCube3 orientation sensor system and LIRC (Linux Infrared Remote Control) 3 is used for accessing the IPAQs infrared ports. The software implemented for the demonstrator makes use of the Trolltech QT/Embedded Linux 4 for creating a graphical user interface. Finally, the control server runs a Java-based middleware for receiving and interpreting orientation data from the control artefact and causing the control interfaces to perform respective actions. The wireless InertiaCube3 sensors use proprietary wireless communication, wherefore a corresponding receiver from Intersense is required. Communication from the control server to the add-on control interfaces is done via socket connections, wherefore a dedicated message protocol for e.g. sending device-specific infrared commands or causing the PDAs to play sounds or display images has been implemented. 3.2 Scenario We have chosen three devices to be controlled: a TV set, an amplifier with integrated tuner, and lights which are plugged to a power switch that can be managed remotely via TCP/IP. While the first two can be controlled by their IrDA interfaces, the lights can be turned on and off via a socket connection to the power switch. Using not only devices with infrared ports shows the flexibility of the approach of using add-on control interfaces, which in principle allow to control arbitrary devices whose services can be accessed electronically. In order to make the user aware of the current state of control, the following two feedback modalities have been implemented: Auditory feedback: When browsing devices or services, the PDA plays a sound-file with the name of the device or service every time a new one is chosen. In addition, auditory feedback is given upon selecting/unselecting a device or service as well as steering input values. Visual feedback: In device browsing mode, the currently chosen device is highlighted with a red circle. Upon selecting a device, an icon of the currently chosen service is displayed in a red circle, as long as the system is in service browsing mode. If a service has been selected for steering its input values, the icon of this service or additional information about the parameter value are displayed in a green circle (cf. Figure 6)

7 7 Fig. 6. Control interface with exemplary visual feedbacks In the following, the application scenario is described on the basis of the five operations proposed in section 2. As soon as a person grasps the control artefact and starts to rotate it, the control server enters the device browsing mode. The orientation values of the wireless sensor embedded in the artefact (i.e. the blue cube) are sent to the server, which interprets them and sends respective commands via socket connection to the control interface (i.e. a PDA in this setting) of the currently chosen device. Starting with an initial device, the devices in the room around the user are chosen according to a specified order while the control artefact is being rotated. Each device indicates this by showing a red circle and playing a short sound file which says its name (e.g. TV-set ). By performing a tilt-movement with the control artefact (i.e. device selection), the currently chosen device is selected, which is again indicated by auditory means (e.g. saying TV-set selected ), and the service browsing mode is entered. By rotating the control artefact, services provided by this device can be browsed with rotary movements. The control interface of the selected device shows a red symbol of the currently chosen service, where the order of services is defined a-priori in our setting. Browsing services is again supported by a sound output with the name of the currently chosen service (e.g. channels ). Finally, a certain service can be selected with a tilt-movement, too (i.e. service selection). Upon selection, the colour of the displayed service symbol switches to green, and a respective sound informs the user that the chosen service can now be controlled (saying e.g. channel control ). In this steering mode, the input values of the selected service can be changed (switching the channels in this example). However, if a service has just two different input values (e.g. play and pause ), selecting the service itself causes a change. This allows the user to switch between the two states just with a tilt movement, and without entering the steering mode and turning the control artefact. 4 Conclusions and Future Work In this paper we have presented a novel paradigm for controlling the environment with physical artefacts that are aware of their spatial orientation. It allows to browse and select both devices and their services as well as to steer the input values of a selected service by manipulating an orientation-sensitive artefact in

8 8 a simple and intuitive way. The proposed solution scales well to large numbers of devices and services, and it has weak assumptions on the shape of the control artefact. Add-on components that can be physically attached to different devices are used as uniform control interfaces between the artefacts and the devices to be controlled. For showing the feasibility of our approach, an application for controlling multiple electronic devices in a domestic setting has been implemented. There are several issues for future work. First, we intend to extend the middleware to accommodate multiple artefacts for control. Second, a graphical tool for both associating the control interfaces with devices, and establishing the browsing order among them, is planned to be developed. A further issue is the development of a decentralized architecture, which would allow the control artefacts to communicate directly with the control interfaces, without the need for a central coordinator. Last but not least, user studies will be carried out in order to compare the proposed paradigm with traditional remote-controls. References 1. Ishii, H., Ullmer, B.: Tangible Bits: Towards Seamless Interface to Access Digital Information. In Extended Abstracts of Conference on Human Factors in Computing Systems (CHI 01), Seattle, Washington, USA, March 31 - April 5, 2001, ACM Press, pp Eustice, K.F., Lehman, T.J., Morales, A., Munson, M.C., Edlund, S., Guillen, M.: A universal information appliance. IBM Systems Journal, Vol. 38, Nr. 4, pp , Myers, B.A.: Using handhelds for wireless remote control of PCs and appliances. Interacting with Computers, Vol. 17, pp , Vlkkynen, P., Korhonen, I., Plomp, J., Tuomisto, T., Cluitmans, L., Ailisto, H., Sepp, H.: A user interaction paradigm for physical browsing and near-object control based on tags. Proceedings of the Physical Interaction Workshop at Mobile HCI Conference, Udine, Italy, Ferscha, A., Resmerita, S., Holzmann, C., Reichoer, M.: Orientation Sensing for Gesture-Based Interaction with Smart Artifacts. Computer Communications, Vol. 28, No. 13, Fitzmaurice, G.W.: Graspable user interfaces. Ph.D thesis, University of Toronto, Rekimoto, J.: Tilting operations for small screen interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 96), pp , Rekimoto, J., Sciammarella, E.: ToolStone: Effective Use of the Physical Manipulation Vocabularies of Input Devices. Proceedings of ACM User Interface Software and Technology (UIST), 2000, pp