Empowering the User to Build Smart Home Applications

Transcription

1 Empowering the User to Build Smart Home Applications Tao ZHANG, Bernd BRÜGGE Technische Universität München Institut für Informatik/ I1, Boltzmannstraße Garching b. München, Germany Abstract. Recent research has shown that end-user programming will form an important part of any general purpose context-aware computing system, since the system behavior of existing context-aware systems does not reflect the situation the end-user intended. This paper is concerned with end-user programming and modification of context-aware application behavior. We present a three-tier approach to build context-aware applications, which separates context collection and distribution from rule-based inference that controls application behavior. Prototypes were created to evaluate our approach. The primary focus was directed towards the rule conflict management and user interface issue, that are related to defining rules. We conclude this paper with a short usability case study based on these prototypes and an outlook into future work. 1. Introduction Beginning with Mark Weiser s [17] vision of ubiquitous computing, new technologies like tangible interfaces, wearable computing, mobile computing elevate the style of interaction beyond the traditional desktop environment and are becoming important in the everyday life. The foremost objective in building context-aware applications is to create a sensor rich environment that responds specifically to people, place and time. The user is not tied to one particular device but interacts with a network of intelligent devices embedded in the environment. To develop a context-aware application, developers currently have two options: Either they build a system that hypothesizes system behavior using an inference engine [11,15], or they predefine a specific system behavior [2]. Recent research has shown that both of these approaches have not proven long term compatibility with the end-users expectations [7]. The system behavior cannot be predicted by an inference engine, since there are many human or social aspects of context that cannot be sensed or even inferred by sensors or devices available today [1]. There is no ascertained way for a system to infer a correct contextual state just by collecting sensor information. At best, the system can provide an approximation or an educated guess of the real context. The behavior of the context-aware system and its applications cannot be recognized and hardcoded by the developer, since it is very difficult for the developer to foresee what information and sensors are necessary to infer a contextual state. Moreover he cannot foretell whether the results produced by the system are meaningful to the end-users [7].

2 Currently, a variety of systems have been built, that are hardcoded [2]. The behavior of these systems has become a routine matter of recognizing states and triggering corresponding activities. While this approach can work in situations with well-defined social expectations (e.g. turning off a mobile phone in a theater), it fails for situations in which end-users behavior is less predictable (e.g. at home). People feel like they are loosing control of the system and begin to fight the system, because the system view of context does not match what was actually happening. We believe that the only way a system can agree with the users expectations is to have the user program the system behavior. To accomplish this, we would like to empower users to build own applications in a most simple way. This paper is concerned with end-user programming and modification of context-aware environments, where an end-user is referred to as a person who has had experiences with handheld devices like mobile phones or PDAs, but cannot program in JAVA or C++. We present an architecture composed of three separate but interrelated layers a context layer for capturing and handling physical sensors, a rule layer providing high-level behavior specifications, and an user interface layer providing user interfaces supporting end-user control and modification. 2. Related work Most context-aware research has focused on building infrastructures to support developers in building applications and on the applications themselves [2], not enough emphasis has been placed on empowering end-users to build their own context-aware applications. Humble et al [8] presented the development of a user-oriented framework to support the user reconfiguration of ubiquitous domestic environments. They present a lightweight component model that allows users to interconnect a range of devices easily. The framework discovers available ubiquitous components and presents these to users as jigsaw pieces that can be dynamically recombined. However, the emphasis is on reconfiguration rather then programming and the approach does not seek the richness of programming expression. icap [16] is focused on helping developers rapidly prototype, test, and iterate their context-aware applications. It is based on previous work in ubiquitous computing applications, specifically those that involve the development of rule based conditions. Some of these include AgentSheets [14] and Stick-e notes [12]. In his most recent work, Dey presented a new prototype environment called a CAPpella [3] for end-users. In this environment, tools are available to end-users in order to demonstrate and change the system behavior without any programming skills. However, to build simple applications, end-users must first learn how a CAPpella works and then spend time extracting information from the sensor data manually. After that, an inference engine teaches a model based on annotated logs. For the system to work predictably users have to refine the system behavior many times. Both scenarios introduced in his paper (meeting scenario and medicine taking scenario) [3], discovered that the average users have to still spend a lot of time and effort to achieve an often very limited benefit.

3 3. A three-tier Architecture This paper is concerned with end-user programming and modification of context-aware application behavior using a rule-based approach. We present a three-tier approach to building context-aware applications, which separates context collection, distribution from rule-based inference that controls application behavior [18]. While the software developers are primarily responsible for the context layer and rule layer, end-users use a graphical user interface to modify and control the overall system behavior. Figure 1: Three-tier architecture for context-aware systems In the architecture shown in figure 1, we select the context-toolkit, a framework developed by Dey [2], which separates the acquisition and representation of context from the delivery and reaction to sensor context by a context-aware application, as our low-level communication infrastructure. The system behavior is specified through an ECA Rule Specification Language. Each rule is composed of an Event part, a Condition part, and an Action part. The ECA concept is based on past research in active databases, where application developers specify with the help of a powerful ECA Rule Specification Language sophisticated conditions over events, supplied by diverse event sources and associate to them actions [13]. In the past, ECA rules were applied to conventional databases, which can only carry out actions concerning database operations. A smart home may not only use this resource database in the ECA rule definition but can also trigger home services. Depending on incoming Sensor Events from the Context Layer, the Rule Monitor evaluates initially all the ECA-rules which Event parts that match the actual situation. These selected rules may conflict with each other if the Event parts of more than one rule are matched

4 concurrently or if the services to be executed are exclusive. The Rule Monitor delegates those conflicting rules to a component called Rule Conflict Manager, which solves the rule conflicts before calling the corresponding Services of the Context Layer. Rule Discoverer is a software component that discovers user behavior patterns using the frequent pattern discovery [9]. For example, if the user did the same action several times in a same situation, the system will remember it by observing the actions of the user and automatically generate new rules. Since it is quite challenging to provide good rules by just collecting sensor information, users are still expected to refine the rules by removing noise information from sensors. Using handheld computers like PDAs, end-users are then allowed to define high-level ECA rules. With these rules, the end-users can program different personal context-aware applications without special programming skills. They can not only read the rules, but also modify them, delete them, make each of them active or inactive (Rule Builder & Rule Inspector). Using this model, each end-user is now able to personalize the rules to suit his needs and notions for how the environment should react to his actions. Rules that are effectively executed are listed in chronological order. In case the user has difficulties to understand the system behavior, the user is able to track any executed rules in the past and figure out its rationale, he can then eventually undo, redo some actions, or overwrite some rules (Rule Debugger). 4. Scenarios Our first scenario deals with a remote antenna station for a high-power transceiver at a remote location like the top of a mountain, the middle of the desert or an oil rig. Exceptional situations such as transistor fail or temperature fail are automatically detected by sensors embedded in the environment. The antenna station is connected with a maintenance and installation information system, which gives the technician context-aware help, depending on the exceptional situation and the skill level of the technician. When an exceptional situation occurs and the technician is not on site, he is informed immediately and can monitor the situation in the antenna station with his UMTS mobile phone. A rule for this scenario can be defined as follows: Scenario: Event part: Condition part: Action part: Remote Maintenance When LED transistor fail OR temperature fail, (and) the technician is NOT in the antenna station, (and) the skill level of the technician is very low, (then) turn on the video recorder to capture the whole exceptional situation, (and) send the captured Video clip to the technician s UMTS mobile Phone, (and) send the SMS Please exchange transistor, and align quiescent current Our approach can also be used in a smart home to assist the elderly and individuals with disabilities. Users can specify personal activities they would like the house to automate (temperature control, control of hard-to-reach devices, etc.). For a grandmother sitting in a RFID-tagged [10] wheelchair, who usually has her breakfast between 7am and 8 am, the following rule could be defined by her caretaker or discovered by the system:

5 Scenario: Event part: Condition part: Action part: Having Breakfast When the wheelchair with RFID-tag is detected in the kitchen, (and) it is the first time between 7 am and 8 am, (then) turn on the kitchen light, (and) turn on the kitchen coffee maker, (and) displays the morning news on the kitchen video screen. 5. Status To evaluate our approach, different prototypes have been realized such as the component of the rule tier responsible for rule conflict management, as well as the user interface responsible for defining rules. 5.1 Using JESS for the Rule Layer We have implemented the Rule Monitor and Rule Conflict Manager using JESS [5] a rulebased expert system written in Java. Figure 2: Using JESS for the Rule Layer End-user defined rules and policies are stored in a relational database (mysql) and will be uploaded into the JESS internal rule base at runtime. Depending on the incoming events captured by sensors and context (e.g. state of system component), the facts in the JESS Working Memory will be updated. Pattern Matcher then searches automatically through vast amounts of combinations of facts to find those combinations that satisfy rules to figure out which rules should be fired, while Agenda is responsible for using the conflict strategy to

6 decide which of the rules, out of all those that apply, have the highest priority and should be fired first. This approach is suitable for our task, since the built-in inference engine implements the RETE algorithm [4], a very efficient mechanism to solve the difficult many-to-many matching problem. The Execution Engine component fires the rules. It calls the external services, which eventually change the context (see figure 2). The advantages of JESS are the following: (1) JESS consists of a rule specification language that can be used to define ECA rules. (2) JESS works well with the context-toolkit, since both are implemented in Java. (3) Low-level rule conflict resolution strategies are by default integrated in JESS. While the end-users define the priority of the rules, JESS is responsible for triggering rules in the right order at runtime. 5.2 User Interface Prototypes for defining personal rules The JESS syntax is not easy to understand. Therefore, prototypes of the Rule Builder and the Rule Inspector were implemented to evaluate different user interface concepts for defining rules. Figure 3: Rule Builder (version 1) Figure 4: Rule Builder (version 2) The first prototype (figure 3) provided a graphical representation of the JESS concept of rules. The second prototype was based on exemplar-based approach using puzzle-like pieces (figure 4). In this approach, end-users simply copy and refine existing rules - either predefined by the developers, or partly recommended by the Rule Discoverer (see 3). Since the behavior pattern is highly individual, end-users are expected to modify the system behavior by adding or removing conditions and actions with the drag and drop feature.

7 6. Evaluation and next steps We evaluated our prototypes in a laboratory-based intelligent environment with two smart boards, Java ibuttons, PDAs as well as Bluetooth-enable mobile phones. We have built context-aware applications as follows: Just after the user enters his office in the morning, his personal s will be showed automatically on the smart board, at the same time, the music program itunes starts playing his favorite classic music. As soon as the user get unexpected visit, s will disappear immediately and the music turns automatically down. The first graphical user interface prototype has been tested during winter semester 2003/04. Ten computer science students were asked to build their own applications using the Rule Builder (Version 1). Most of the students commented, that user control over the environment increases by using personal rules, because they can change the system behavior in run mode. Nevertheless, only three students were ready to spend time for building rules from scratch as the first user interface prototype is still too complicated to use. The second user interface prototype is now implemented and has been currently tested by fifteen computer science students. To define the same rules, students using the second prototype need about 75% less time, require substantially less help and make fewer mistakes than those using the first prototype. Therefore we believe an exemplar-based user interface can significantly improve the usability. In order to find out if non-computer scientists are capable of precisely specifying personal ECA rules, we plan to do more evaluation work with non-computer scientists in the future. 7. References 1. Bellotti, V. & Edwards, K, Intelligibility and Accountability: Human Considerations in Context Aware Systems. Human- Computer Interaction, Dey, A.K. et al. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. HCI Journal 16(2-4), 2001, Dey, A.K. et al. a CAPpella: Programming by Demonstration of Context-Aware Applications, CHI Forgy C.L. Rete: A Fast Algorithm for the Many Pattern/ Many Object Pattern Match Problem", Artificial Intelligence 19, pp , Friedman-hill, E., Jess in Action - Rule-based Systems in Java, Manning Publications Co., Gajos, K. et al. End user empowerment in human centered pervasive computing. Proc. Pervasive 2002, Greenberg, S. Context as a dynamic construct. Human-Computer Interaction, 16, Humble, J, et al, Playing with the Bits User-configuration of Ubiquitous Domestic Environments, Ubicomp Hand D.J, et al, Principles of Data Mining, The MIT Press, LARAN RFID, A Basic Introduction To RFID Technology and its Use in the Supply Chain, White Paper, Mozer, M.C. The neural network house: An environment that adapts to its inhabitants. Proc. AAAI Spring Symposium on Intelligent Environments 1988, Pascoe, J. The Stick-e Note Architecture: Extending the Interface Beyond the User. In Proceedings of Intelligent User Interfaces, pp Paton N.W. and Díaz O. Active Databases Survey, ACM Computing Surveys, Vol. 31 No.1, pp , March Repenning, A. Creating User Interfaces with Agentsheets. In Proceedings of Symposium on Applied Computing, pp Schmidt, A. and Van Laerhoven, K. How to build smart appliances. IEEE Personal Communications 8(4), 2001, Sohn, T. and Dey, A.K. icap: An Informal Tool for Interactive Prototyping of Context-Aware Applications. Extended Abstracts of CHI 2003, Weiser, M., Some computer science issues in ubiquitous computing, Communications of the ACM, Zhang, T., An Architecture for Building Customizable Context-Aware Applications by End-Users, Pervasive 2004 Doctoral colloquium.