utablesdk: Enabling Rapid Prototyping of Window-based Applications on Interactive Tabletop

Transcription

1 utablesdk: Enabling Rapid Prototyping of -based Applications on Interactive Tabletop Chenjun Wu, Yongqiang Qin Computer Science Department Tsinghua University Beijing, China p802 {anwingwu, qinyongqiang@gmail.com ABSTRACT Interactive tabletop has shown great potential in facilitating face-to-face collaboration in recent years. For many scenarios such meeting and brainstorming, the applications on tabletop should be window-based so that multiple independent or collaborative applications can be interacted with simultaneously by multiple users. However, this kind of application is currently hard to build since most current toolkits for tabletops only focus on full-screen, single application development. To address this challenge, we developed utablesdk, a software toolkit that allows researchers to rapidly create window-based applications that can be executed simultaneously on a multi-user windowing system specially designed for tabletops. In this paper, we describe utablesdk and report on our explorations of window-based programming techniques for tabletops. Several sample applications are also presented to demonstrate how the development of tabletop applications can be accelerated by utablesdk. Keywords UI Toolkits, Interactive Tabletop, -based Programming, Multi-Touch, Multi-user INTRODUCTION As a rising interactive device, tabletop system has shown great potential in facilitating multi-user collaboration in recent years. Yet, in spite of much promising research, there has been little investigation of tabletop interfaces that can support individuals frequently transitioning between shared tasks within a group [1]. Many activities on traditional tables, such as meeting, brainstorming, and planning, involve this kind of collaboration, and we believe the tabletop interfaces can bring significant benefits to them, especially when the large-scale tabletops are emerging [2]. To explore these valuable scenarios on tabletops, one essential requirement is that the applications on tabletops LEAVE BLANK THE LAST 2.5 cm (1 ) OF THE LEFT COLUMN ON THE FIRST PAGE FOR THE COPYRIGHT NOTICE. Yue Suo, Yuanchun Shi Computer Science Department Tsinghua University Beijing, China p802 {suoyue, shiyc@tsinghua.edu.cn should be window-based rather than full-screen so that multiple independent or collaborative applications can be interacted with simultaneously by multiple users. Besides, windows-based applications allow their size and position changed dynamically, which enable people to rearrange the table to suit different states of the collaboration. However, there arise several challenges for building window-based applications on interactive tabletop. In the first place, we still not fully understand how window should be designed on tabletop duo to three fundamental factors of tabletop: horizontal display, direct input and multi-user usage. In the second place, there is not an efficient tool that can help us explore the possibilities of window design and create window-based applications. Although recent tabletop research has been fuelled by software toolkits that allow researchers to rapidly create multi-touch applications, most these toolkits [3, 4, 5] only focus on full-screen, single application development. As a result, researchers have to develop their own specialized underlying plumbing such as window manager before they can concentrate on the window-based applications. In this paper, we present utablesdk, a software toolkit that addresses these challenges. It provides several novel ways to handle issues of input processing, window manipulation, window communication, window layout for interactive tabletops, and allows more possibilities to be easily explored. utablesdk allows researchers to rapidly create window-based applications that can be executed simultaneously on a customizable multi-user windowing system specially designed for tabletops. It is open-source, and freely available for academic research. RELATED WORK Tabletop Toolkits Various toolkits [3, 4, 5, 6] have been built specifically for the interactive tabletop. They all provide abstractions to support the core tabletop interaction functionality, allowing researchers to concentrate on their applications. For example, the DiamondSpin toolkit [3] implements around the table interaction techniques to address orientation challenges in tabletop use. The T3 toolkit [5] allows

2 multiple tabletops to be connected together to support mixed-presence collaboration. All these toolkits also target towards accelerating the development of applications. Typically, they provide users with additional functionality for dealing with the input, producing graphical output, and creating reusable components. PyMT [4], for instance, provides a growing collection of reusable widgets and interaction techniques. Unfortunately, all these toolkits assume that the tabletop system can only display a single application at one time, and the API they provide is based on the notion of a single rectangle display with no layout information. Applications created by these toolkits cannot be aware of the existence of other applications on the tabletop, neither. As a result, it s impossible to use these toolkits to set up scenarios such as meeting that need to execute multiple collaborative applications simultaneously, unless we modify the source code or create an ad-hoc layer that can host and manage other applications. However, both the approaches are a hard task, one that few HCI researchers are willing to do. Tabletop Environments These are also works focusing on modifying the current windowing systems to provide novel tabletop environments. Mettise [7] is a tool to facilitate the design of innovative window management techniques. It uses an image compositing approach that makes it possible to apply a number of visual effects and geometrical transformations on windows. As one of its examples, Mettise shows how it enables tabletop interface that features automatic window orientation and on-demand window duplication. Unfortunately, Mettise does not take input customization into consideration and only support single user interaction with traditional window applications. MPX [8], in contract, allows users to simultaneously interact with multiple traditional single-user window applications as well as novel multi-user applications on tabletop. It is a modified X System that can accommodate multiple input devices, and provides new APIs for developers to create collaborative applications based on these inputs. The shortcoming of MPX is that it does not consider the layout issues when windows are displayed on tabletop. For example, windows cannot be rotated, which is crucial for tabletop environment. The main problem of these new environments is that although their techniques have been shown to be effective, they implement only parts of the windowing system on tabletops. As a consequence, these environments are hardly usable to set up groupware scenarios and help researchers create multi-user window-based applications on tabletops. UTABLESDK ARCHITECTURE In this section, we provide an overview of utablesdk architecture, which is outlined in figure 1. utablesdk splits the user interface of tabletop into two parts: a windowing system and window-based applications. Each application contains one or several windows, and can be compiled independently into a DLL. This DLL should be placed into a local folder specified by the windowing system so that the system can see and load this application at runtime. Like most windowing systems on desktop, we use a tree structure to organize all the loaded windows into a hierarchy. The root of this tree is a full-screen application, which is called Tabletop and should be designated before the underlying windowing system starts to run. Other applications can then be started and hosted within the root window. A simple, well-documented C# API is provided so that developers of a certain application can be aware of the size, position, orientation and relationship of all windows on the tabletop, and change them dynamically. The main component of the windowing system is a window manger that has following three main responsibilities: Collecting Inputs. Various input devices such as App A Apps Root App (Tabletop Environment) Route Inputs Widget Tree Widget Root Widget App B Layout Policy Modify s Layout Inputs for a Certain System Layout Manager Inputs for s Layout Input Mapping <User ID, Input Event> Input Providers <User ID, Input Signal> Figure 1. utablesdk Architecture

3 TUIO, laser pen or keyboard can be added to the window manager by simply implementing an interface with only one event: InputReceived. The window manager can then gather input events rose by these input providers and send them to the input mappers. Mapping and Dispatching Inputs. After receiving the input events, they will be mapped to a certain window according to the layout of windows. If the mapped window is willing to receive the inputs, they will be directly sent to that window and routed from a certain leaf widget to the root widget unless they are consumed along the way. Otherwise, inputs will be sent to the layout manager for placing windows. Placing s. Each window holds an instance of layout policy that defines how its children windows should be placed. The layout manager uses the policy of the mapped window s parent window to modify windows layout according to user s inputs. Both the windowing system and applications are implemented with WPF and C# programming language. The detail of the windowing system is discussed in another paper. In the following sections, we will focus on windowbased design and programming techniques we explored and introduce how utablesdk enables rapid prototyping of window-based applications on interactive tabletop. WINDOW-BASED PROGRAMMING utablesdk provides APIs for creating window-based applications in the manner we ve been familiar with in desktop environments. Furthermore, it also considers the features of tabletops and makes several improvements. Creation In utablesdk, we call a window UObject, which implements IObject interface. It also inherits from WPF ContentControl class, which means we can edit the user interface of the window just like other WPF controls: edit the widget tree visually in Visual Studio. s are rectangular by default, but other shapes can be created by setting parts of the window with transparent pixels. Similar to desktop windows, utablesdk provides two ways to create a window and insert it to the windows tree: a normal window or a modal window. // Create and show a normal window, and put it on the tabletop UTableHelper.CreateObject(typeof(UObject1), UTableHelper.Tabletop); // Create a modal window, and put it on the window created it UTableHelper.ShowModalObject(typeof(UObject1)); Operations A number of operations are supported to manipulate windows, including rotate, reposition a window, maximize, restore a window, or change the parent of a window. These operations will be turned to layout commands and sent to the layout manager, which in turn will update the layout according the layout policy of the manipulated window s parent window. In fact, these operations are transparent to the developers of the window. However, there are several operations developers must be responsible to handle, such as input and window resize. Next we will discuss these operations and share our experience related with them. Input Handling Maybe the lowest level operations are multi-touch and other inputs sent to the window. As introduced, when a window receives inputs, they will be routed from a certain leaf widget to the root widget unless they are consumed along the way. Each widget along the path has the chance to explain these inputs to high-level window-specific commands. However, if none of the widgets handled the inputs, inputs will be sent to layout manager and explained to layout operations. Although seemed trivial at first glance, this input handling mechanism gives much more flexibility for differentiating layout inputs from window inputs. On desktop systems, this differentiation is done using the window frame. That is only the inputs hit on the frame will be used for layout. However, our experience tells this method is infeasible for the tabletop due to its direct input feature. Most of the time, the tile bar of the window is hard to reach and the border must be big enough so that fingers can manipulate it without mistake. The input handling mechanism of utablesdk helps escape this predicament by allowing any place that does not expect to receive inputs to be used to place window. Thus developers can design the interface so that the layout operation and window content operation can both become an easy task for the users. Resize vs. Scale When the size of a window changed, the SizeChanged function on that window will be called, and the developer is responsible for rearranging the user interfaces for the new size. For tabletop windows, the scale operation is also useful in reducing overlapping while preserving the layout of window contents. An interesting problem for tabletop windows is how to distinguish the resize operation and scale operation from a user perspective. For example, users can use two contacts zooming for resizing while four contacts zooming for scaling. The window layout policy can be used for exploring more possibilities, which will be discussed later. Menu In our experience, the fixed control buttons and menu is hard to use for tabletop windows since we usually can t reach them. To deal with this problem, we use pie menu instead. The pie menu can be placed anywhere on the window or popup automatically when a contact holds for a certain time. In addition, more than one menu is allowed to be shown concurrently, which is useful for multi-user usage. Inter-window Communication Inter-window communication is an important tool for facilitating the design of collaborative applications. utablesdk provides several mechanisms to help researchers deal with this issue.

4 Messaging At the lowest level, two pairs of methods are provided for sending message and querying data among windows: void SendMessage(object m) & OnMessageReceived(object m) object QueryData(object f) & object OnDataQuerying(object f) These methods are competent for any type of communication since they allow data of any type to be transferred synchronously or asynchronously, one way or two ways. Based on these two communication channels, several high level mechanisms can be established to facilitate the coordination of windows. Data Sharing utablesdk has a built-in share center that helps synchronize data among different windows. The window registered in the share center will be notified when the data has been modified by another window, and therefore it can update its local copy. This sharing feature has proved useful, for example, for updating a window s content dynamically when user is inputting data to a setting dialog. Duplication As discussed by Meredith [9], window duplication is an important coordination policy for co-located groupware. utablesdk supports three kinds of duplication: (1) creating a view linked to the original, (2) creating a readonly copy, and (3) creating a fully independent, read-write copy. These duplication types can be useful in different scenarios since they provide different authorization levels to the groupware coordination. Layout Policy Several particular features of tabletop make the layout of windows on tabletop environment much more complex compared with traditional desktop systems. For example, interactive tables provide a co-located collaborative environment for multiple users, which raise many new issues such as coordination and region division. utablesdk invents the notion of window layout policy to help researchers cope with these layout challenges on tabletop. A layout policy is an interface that defines how the layout of child windows is managed by their parent window. Every window can define its owner layout policy, as shown by the following code sample: class DemoLayoutPolicy: LayoutPolicy { override void OnInputReceived(IObject o, InputEventArgs a){ // handle layout inputs on a certain child window override void OnObjectInserted(IObject obj) { // define how to put an incoming window // other interfaces of LayoutPolicy class DemoObject: UObject { DemoObject(){ this.layoutpolicytype = typeof(demolayoutpolicy); this.layoutpolicyparameter = null; The layout policy is useful in several ways. First of all, it gives researchers much more flexibility to design the tabletop environment since they have the ability to define how the layout properties such as position, orientation, size and scale should be changed by user inputs on different regions. For example, we can apply a physical layout policy to the root tabletop so that each child window has inertia while for workspace we can use desktop s window layout policy to arrange windows. Besides the layout policy is also a reusable component which means it can be shared among different projects and help reduce the time of development. ENABLING RAPID PROTOTYPING utabledk provides several features to accelerate the development of multi-touch window-based applications. In this section, we will give a description for each feature. Replanting Traditional Single-Pointer s Replanting a traditional single-pointer, single-user WPF window to the utablesdk-powered application is supported by a very simple syntax: putting this window into an UControlWrapper control, as illustrated by figure 2. The UControlWrapper can handle all multi-touch inputs on that window, translate them into single-pointer mouse events, and inject them to the wrapped window. It means a large set of open-source WPF applications can be used immediately for the development of tabletop applications. <u:uobject> <u:ucontrolwrapper> <Chess /> </u:ucontrolwrapper> </u:uobject> Figure 2. Replant a desktop chess window to the tabletop Adding the multi-touch feature to the existing WPF controls is also an easy task. Researchers only need to create a new control, and let it inherit from the old control class and the IControl interface. This interface provides several functions and events that can be used to handle multi-touch events. Widgets and Templates for Multi-Touch s utablesdk takes advantage of WPF s great framework for reusing existing components, and provides a large set of widgets to facilitate the development of new multi-touch enabled windows. These widgets such as pie menu and grid viewer are all designed carefully and can work well under multi-touch inputs. Furthermore, several Visual Studio item templates are also provided for researchers to create reusable components within a project. For example, the utablecontrol template gives researchers a start point to create a multi-touch enabled control. Multi-Touch Handler for Handling Multi-Touch Input Handling multi-touch inputs is a much more complicated task compared with handling single pointer input. We must

5 care about the number of contacts, their tracks and changes of relative positions, which make the code of multi-touch handling still tedious to write and hard to maintain. However, the way to handle multi-touch inputs is usually very similar. For example, in many applications such as a map, we just need some standard gestures extracted from the inputs, such as tap and zoom. To help researchers more easily deal with these common input patterns, we invent the concept of Multi-Touch Handler. Multi-touch handler inherits from MultiTouchEventHandler class. It is attached to an utablesdk control and handles all multi-touch inputs for that control. Typically the handler will raise some events when it detects certain input patterns. Therefore, researchers can just listen to these special events instead of directly handle the raw multi-touch inputs. The following code sample shows how DetectContactsHandler can be used to decide whether a button is pressed. Additional handlers such as LocalGestureEngine and SingleFingerHandler are also included in utablesdk to help researchers detect gestures or only handle one contact input respectively. public UButton() { DetectContactsHandler h = new DetectContactsHandler(); handler.newcontact += new ContactEventHandler(hNewC); handler.nocontact += new ContactEventHandler(hNoC); MultiTouchProcessor.Handlers.Add(h); void hnewc(fingereventargs args) { this.ispressed = true; void hnoc(fingereventargs args) { this.ispressed = false; Development and Test Tools utablesdk integrates well into Visual Studio, and provides a comfortable development environment to researchers. For example, researchers can simply open a project template in Visual Studio in order to create a new utablesdk application. Besides, the application can be directly run under the Visual Studio, where utablesdk will create a simple window system and put this application into it. After the development finishes, the application can then be compiled into a DLL, and deployed into a real system. EXAMPLES In this section, we present several examples that demonstrate how utablesdk and its features discussed above facilitates the implementation of various non-trivial window-based applications on interactive tabletop. upaint upaint is a multi-touch drawing application designed for interactive tabletop. It includes many advanced features such as drawing smoothed strokes, editing the strokes, etc. Although seemed complex, this application is created within three hours since most of the components come from open-source projects or utablesdk s widget library, which are already ready for use. Most of the work thus is assembling these components and adding multi-touch functionality. For example, the fish eye panel located at the bottom of the window is a WPF control designed for mouse interaction. We simply put this control within an UControlWrapper, and it proved that this control also worked well under multi-touch environment. Another control we directly use is Microsoft s InkCanvas, which provides API for drawing and editing strokes. We add multi-touch support for this control by using a b c d e f Figure 3. Six utablesdk applications. (a) upaint, (b) umap, (c) ubook, (d) PhotoBrowser, (e) HouseDesigner, (f) umeeting

6 LocalGestureEngine to translate multi-touch inputs to commands such as scaling a stroke. umap This application lets the user browse the map using several standard multi-touch gestures such as one finger panning and two fingers zooming. The gesture detection is also achieved through LocalGestureEngine. Another feature of umap is its use of child windows. A modal dialog is used for inputting goto place; a normal dialog is used for map setting, which can update the map dynamically through data sharing. In addition, umap applies a simple layout policy to itself so that child windows can be moved within the region of the map. ubook ubook is an application for collaborative reading. This application is designed like a book and can be read page by page in a peeling manner. Since we do not apply a window frame to the application, users can switch the book between editing mode and viewing mode by toggling a menu item in order to read or reposition the book. Three kinds of window duplication are applied to a book in order to share this book with different authorization levels. For example, users can create several views linked to the original book so that when one is marked, it will be immediately seen by all of the people holding a view. ProtoBrowser This application allows multiple users to share their photos on the tabletop. These photos can be managed by means of album, which is created as a window. Therefore, albums can be easily copied for sharing. Each photo is also encapsulated into a window, which means photos can moved among different albums or be put directly on the tabletop. HouseDesigner This application is used for the design of home furnishing, and is launched as a full-screen application by default. Through this application, uses can select furniture from a toolbox and place it anywhere in the room. Various properties of furniture as well as the whole room can be set through popup menus. Besides, the whole furnishings can be reestablished in a 3D scene for more real browsing. This app demonstrates that utablesdk also provides good support for building traditional full-screen applications on interactive tabletops. umeeting umeeting is a windowing system designed for meeting scenario on large-scale interactive tabletop. This system shows how applications developed using utablesdk can be launched and interacted with concurrently by multiple users. When holding a finger on the tabletop for five seconds, a pie menu will pop up and allow the user to launch a special application called App Loader. This application scans the app folder of this system and lists all available applications for user to choose. Besides, umeeting also provides several auxiliary applications for users to organize the launched applications. For example, users can use workspaces to plot out the tabletop into several subregions. Actually, a workspace is a window that can host other applications as its sub-windows. It uses a customized layout policy that allows these sub-windows to be moved, maximized within the workspace. CONCLUSION In this paper, we have described utablesdk, a software toolkit developed to help researchers to rapidly create window-based applications on interactive tabletop. We have presented the architecture of this toolkit and describe several window-based design and programming techniques we explored. We also illustrate its utility through six sample applications. utablesdk is available at: REFERENCES 1. Tang, A., Tory, M., Po, B., Neumann, P., Carpendale, S. T.: Collaborative coupling over tabletop displays, In Proc. of CHI 06, pp ACM Press (2006). 2. Jens T., Marc H., Benjamin W, Lasse S., Sebastian F., Markus K., Advancing Large Interactive Surfaces for Use in the Real World. Advances in Human-Computer Interaction, Vol. 2010, pp (2010) 3. Chia Shen. DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction. In Proc. of CHI 04, pp ACM Press (2004) 4. Thomas, E.H., Juan, P.H.: PyMT: A Post-WIMP Multi- Touch User Interface Toolkit. In Proc. TABLETOP 09, pp ACM Press (2009) 5. 1hilip, T. and Peter, R.. T3: Rapid Proto-typing of High- Resolution and Mixed-Presence Tabletop Applications. In Proc. TABLETOP 07, pp (2007) 6. SurfaceSDK O. Chapuis and N. Roussel: Metisse is not a 3D desktop! In Proc. of UIST'05, pp , ACM (2005). 8. Hutterer, P. and Thomas, B.H.: Enabling Co-located Ad-hoc Collaboration on Shared Displays. In Proc. AUIC 08 vol. 76, pp Australian Computer Society Inc. (2008) 9. Ringel-Morris, M., Ryall, K., Shen, C., Forlines, C., Vernier, F.: "Beyond Social Protocols: Multi-User Coordination Policies for Co-located Groupware", In Proc. CSCW 04, pp , ACM Press(2004) BIOGRAPHY Chenjun Wu is a second-year master student in Department of Computer Science of Tsinghua University. He has three years research experience in software platform and applications of interactive tabletop, and has published three papers on this topic.