Drag and Drop by Laser Pointer: Seamless Interaction with Multiple Large Displays

Transcription

1 Drag and Drop by Laser Pointer: Seamless Interaction with Multiple Large Displays Liang Zhang 1, Yuanchun Shi 1, and Jichun Chen 2 1 Key Laboratory of Pervasive Computing, Department of Computer Science and Technology, Tsinghua University, Beijing , China blinkzhangpaul@gmail.com,shiyc@tsinghua.edu.cn 2 Institute of Software, Chinese Academy of Sciences,Beijing , China chen1937@sohu.com Abstract. This paper presents a novel interaction technique with multiple large displays in smart space. We call it D2LP, short for Drag and Drop by Laser Pointer, where a specially designed laser pointer upen is a handheld interactive device for users. By D2LP, large displays driven by different computers can be made as one seamless integrated uniform system. With a upen in hand users can directly point and draw on the surface of displays, drag and drop digital objects among multiple displays continuously and smoothly. Report on evaluation experiments shows that D2LP can effectively support the freely interacting with multiple wallsized displays and it is preferred over the conventional mouse-driven cursor based system for the application in smart spaces with rich information displays. 1 Introduction In smart spaces, especially those used in command centers or meeting rooms, large displays are needed for practical reasons. With the rapid development in large display manufacturing, multiple wall-sized displays are increasingly common in universities and corporate conference rooms. Large displays on walls can be easily seen by a group of users and multiple large displays can show more to the audience simultaneously. However, input devices, keyboards and mice, are designed for a single computer and a single user. When dealing with multiple large displays driven by different computers, users need to switch between various devices to manipulate different computers and they have to interrupt their presentations to physically move to the front of the computers. Furthermore, in traditional input devices based systems, digital objects transfer among multiple displays is rather difficult to carry out, but the transfer occurs quite often in smart spaces. To overcome this cumbersome discomfort, we develop a new interaction technique based on laser pointer for environment with multiple large displays: D2LP which is short for drag and drop by laser pointer. Common laser pointers can

2 2 Liang Zhang et al. not serve the purpose, so we also invent a special laser pointer named upen. A user can interact on multiple large displays more freely and simultaneously with a upen[1]. All of these large displays, driven by different computers, are treated as a coordinated system. Moreover, digital objects transfer-ring from one display to another can be done by drag and drop act smoothly and easily. 2 Related Works Studies on interaction with laser pointers on large displays at a distance have been conducted by many researchers. Kirstein and Muller [2] first presented a system that uses a laser pointer as a pointing device. Nielsen [3] proposed an inexpensive interaction technique by introducing a set of window events for the laser pointer such as laser-on/off, and laser-move/dwell. Winograd and Guimbretiere [4] proposed a new kind of interaction techniques for large displays, which are based on gesture and sweep paradigm instead of the usual point and click [5]. Chen and Davis [6] described a system that can provide multiple laser pointer inputs with multiple cameras. Ji-Yong [7] used different blink patterns to distinguish laser pointers. None of them focuses on the interaction between different large displays using laser pointers while some other researchers explore design issues for coordinated device activity, for example: Rekimoto s pick and drop [5], GeiBler s throwing [8], Chiu s ModSlideShow [9] and Khan s Frisbee [10]. But their ideas either need high cost touch boards or need some additional input devices such as tablets to deal with multiple large displays. Low cost laser pointer used as an input device is not much researched yet. The rest of this paper will discuss the principles and use study of D2LP system. 3 Interaction Principles In smart space, D2LP system augments multiple large displays to serve as extended information kiosks, where users can easily access, share and transfer digital objects. These processes are done by D2LP s functions of Distant Manipulation and Drag and Drop by Laser Pointer: 3.1 Distant Manipulation People often stand away to use large displays. For instance, in a conference hall, large title displays are always set at a high position where is out of users reach and even in a small meeting room, a speaker is not used to just sit in front a computer reading slides. He/She prefers to face the audience and displays and even walk around sometimes. Accordingly, there is a need to allow the user to interact with the displays at a distance with no cable connected. In systems supported with D2LP, this problem is well solved. One can use a upen [1], which

3 Lecture Notes in Computer Science 3 will be introduced in detail in Section 4.2, as an augmented mouse to manipulate the computer at a distance. The red laser spot emitted by the upen on the large screen is grabbed by a video camera, and its position is interpreted as the location of the cursor. The Left and Right buttons on the upen are collected by the receiver and emulated as a standard mouse s left and right button events respectively. Additionally, as the On/Off button simultaneously emits a laser beam and the upen s ID. The ID information can be added to the mouse event. As a result, the system has the ability of recognizing who is interacting currently, which is potential to support personalized service. 3.2 Drag and Drop by Laser Pointer Fig. 1. one D2LP scene in smart space Motivation: Multiple large displays are often adopted in smart space, and digital objects transferring occurs quite often, shown as a table in Rekimoto s paper in CHI 98 [5]. As mentioned above, there are several researches on the issue, but all focus on touch based displays or common computers. Interaction on normal wall sized large displays with distant input devices such as laser pointers is overlooked. We focus on D2LP interaction because there are several advances over the touch based display when transferring data using a laser pointer: All the manipulation is done at one position at a distance facing the displays, so it will make the physical movement as limited as possible, and it is quite user friendly while a user is making a speech or giving a presentation.

4 4 Liang Zhang et al. Some areas of the display are out of reach for the user, so touch based display system has to split the display and make a specified region for the manipulation area out of reach or adopt other devices such as tablets severed as additional input devices. But in D2LP system, this is done easily and smoothly: One just uses a upen to drag a digital object and drop at any position in the displays as one wishes. Implementation: D2LP is designed and implemented to facilitate interaction across multiple large displays driven by different computers, especially for the digital objects transferring between displays. D2LP allows users to seamlessly drag digital objects via a laser pointer from a display and across different displays then drop at a position on another display. It works as the following: A user selects the object on one large display by a upen (pointing the laser beam onto the digital object and pressing down the Left Button). The selected object will move along with the laser spot on the display. When the laser beam reaches the edge of the display, it continues its motion in the same direction, crosses over the gap between displays, and then moves on to the edge of the adjacent display. Following the track of the laser spot, the selected object is also transferred onto the machine which drives the adjacent display and continues its motion along with the laser spot until the user drops it by releasing the Left button. For a user, it seems that he/she is dragging the object on a large virtual display rather than manipulating on several separate screens driven by different computers, which makes the interaction more user-centered and facilitating. An actual D2LP scene set up in our laboratory is shown in Fig.1. time out click 1 drop 2 laser point shows in aonther display drag drop drag laser point out of one display 4 drop drop 7 time out/drop Fig. 2. the state transition diagram of D2LP D2LP system works according to the state transition shown in Fig.2. State 1 is the start state which is the time when the system has started up or finishes one object transfer already. The end states are State 1 and 7. If processes end at state 1, it means the digital object is still on the original display: the user either completes a drag and drop act on one display or fails when transferring the object to another display. If processes end at state 7, it indicates that the

5 Lecture Notes in Computer Science 5 drag and drop act transferring between different displays is successful. Transfers between different displays go along the route or while normal drag and drop acts take the transition loop The descriptions on the lines are the state transfer conditions. For example, Starting at state 1, only if the user clicks the Left button on a upen, the state will transfer to state 2, which means the upen s Left Button is pressed and the object is adhered to it. 4 System design 4.1 System configuration Normal Wall-size Display I Normal Wall-size Display II Laser Stroke PointerClient Agent PointerClient Agent Camera II upen in users hands Camera I The Receiver PointerServer Agent Fig. 3. a typical configuration of D2LP system Fig.3 shows the framework set up in smart classroom [11], which is a typical configuration of D2LP system. In this system, when standing or sitting in front of the multiple large displays at a distance, a user can use a upen, which is a handhold augmented laser pointer to complete all the jobs done by mice. Furthermore, he/she can interact with multiple large displays just as with one uniform large display screen. In order to make it clear, we give some detailed description on the devices shown in Fig.3: Pointer Server Agent is the key server computer of D2LP system. It collects the video data from the cameras, searches the laser point from every frame and simulates the mouse move events. At the same time, it also listens to the receiver collecting button click events and simulates the mouse click and release events. Then it sends out the simulated mouse events to the other agents through Smart Platform [12], which is a software infrastructure for smart space application in our smart classroom [11].

6 6 Liang Zhang et al. Camera I and Camera II take the charge of video data collecting. In order to decrease the search complexity in practice, we add on wave-filter screens in front of the camera lens. The Receiver is a device of the task to collect the button click events emitted by upen. Pointer Client Agents are computers which drive the displays by projects. Each display is driven by one pointer client agent, and it results in the difficulty in interacting between displays in traditional input devices and systems. Our solution is that every PCA receives the information from the smart platform and judges whether it is the target computer. If not, it ignores the information, or else it responds by stimulating the mouse act or filing transferring and so on. Wall-sized displays are normal displays and in our smart classroom, one of the displays is just part of one wall as shown in Fig upen: the interaction tool upen [1] is an augmented laser emitter, with three press buttons (On/Off Button, Left Button, Right Button), and a Contact-Activated Button and a wireless communication module. Fig.4 illustrates the structure of a upen. To receive the signal emitted by the upen, a receiver is connected to the computer, which is also illustrated in Fig.4. Fig. 4. (a) is the structure of upen and (b) is the inside of upen. (c) is the inside of the Receiver Functions of additional components on upen are detailed as follows: Laser emitter: Emitting a laser beam On/Off Button: Emitting the upen s ID Right Button : Emulating a mouse s right button Left Button : Emulating a mouse s left button

7 Lecture Notes in Computer Science 7 Contact-Activated Button: Emitting upen s ID when being pressed directly on the surface of the display. Note: this is designed for touch-base board in this paper it is not concerned. Wireless-communication module: Communicating with computer. 5 Experimental Evaluation 5.1 Goal In order to know better how D2LP performs in smart spaces, the experimental evaluation is undertaken. The two research questions are: Whether D2LP system is a potential substitute for the traditional input system in the manipulation in smart spaces with large displays driven by different computers. Whether D2LP system can facilitate users interactively while dealing with digital objects transferring. 5.2 Design Two experiments have been made and 6 subjects (5 male and 1 female), experienced computer users between the ages of 22 and 28 are recruited. Our configuration is shown in Fig.3. The subjects are asked to use the traditional input way: input with mice and keyboards and our D2LP system to complete two jobs: job 1: Open one PowerPoint file on the desktop of one display and make it full screen; then switch to another display and open another PowerPoint file with full screen model. job 2: Transfer a given digital object from one display to another. We request the subjects to do each job twice and three of them are asked to use mice and keyboards first while the others use D2LP first. The time each task cost is recorded and questionnaires are sent to subjects after they finish the experiments to get feedback. 5.3 Results Our data (shown in Fig.5) confirms that users perform much faster while interacting on multiple large displays with D2LP system compared with the traditional way of input. Fig.5a shows that the average time cost to finish job1 with D2LP is two seconds faster than that with a mouse. This is not notable and we should notice that when manipulating on one computer, the D2LP is a little slower than the mouse input system if the users are allowed to sit in front of the computer. But the reason why D2LP is still faster to finish job1 is that in smart space

8 8 Liang Zhang et al TIME COST IN TRADITIONAL INPUT TIME COST IN D2LP TIME/S 10 8 TIME/S TIME COST IN TRADITIONAL INPUT TIME COST IN D2LP USER NO. (a) USER NO. (b) Fig. 5. (a) is Job 1 time cost data and (b) is Job 2 time cost data users do not just interact with one display, they interact with multiple displays simultaneously. Although each task may be simple on one separate display, the real work consists of a series of them with different displays and switches between them. The switch time depresses the performance of the traditional input system. D2LP makes it easy: switch time is lowered to nearly none. From Fig.5b, the advantage of D2LP is obvious. In digital objects transferring between different large displays, D2LP facilitates interaction on one uniform screen and the time cost is quite brief, which is, on average, 20 seconds faster. Moreover, in traditional way users have to switch on different computers even with some physical move and they have to think of a way to transfer the object: three of the subjects use FTP to finish job2, and one uses floppy disk and the other two use share folder. But in D2LP system, this job is quite natural and simple. A user just needs to click, drag and drop with a laser pointer. From Fig.5b, it is evident that one subject costs much more time than the other five with some 4 seconds more (user no.5 in Fig.5b). It is because when he drags the objects through the gap between two large displays, he drops the object, which means he releases the Left button on upen heedlessly and he has to click on the digital object and drag to another display again. It costs additional time but, still, it is much faster than the traditional way. Some important information from the questionnaires indicate that most subjects enjoy the use of upen with D2LP in smart spaces. They admit that D2LP is nature and acceptable. They acclaim that it is amazing when dragging an object from one screen to another. They also propose issues which need to be further studied such as jittering and latency which hinder the manipulation; the confusion some users may have while the laser point and cursor are shown at the same time.

9 6 Conclusion and future work Lecture Notes in Computer Science 9 In this paper, we introduced D2LP - a new technique for interacting with multiple large title displays in smart spaces. D2LP is designed to support and facilitate the free interaction between multiple wall-sized displays. A user in smart space equipped with D2LP can use a upen to manipulate different large displays driven by different computers as on a large virtual display, including transferring the digital objects between displays in a nature and simple way. We also show the value of augmenting the traditional input system with D2LP by running an experiment and we report a significant performance improvement in the file transfer job. Future research work is needed in the following aspects. First, clicking is really a difficult act for interaction based on the laser pointer, because jitter is so distinct that users can not orientate the cursor at a distance at ease. We want to discover a gesture input function to substitute for clicking. Second, we plan to extend the D2LP to support multiple users with personalized service. upen is designed for multiple users to employ but the problem of simultaneity confusion remains to be solved now. Also we hope to find out whether other techniques can be added in the D2LP, replacing the laser pointing or augmenting it. ACKNOWLEDGMENT This work is supported by the National Development Project for the Next Generation Internet, CNGI A References 1. Xiaojun Bi, Yuanchun Shi, X.C.: upen: A smart pen-liked device for facilitating interaction on large displays. In: Proceedings of the First IEEE International Workshop on Horizontal Interactive Human Computer Systems. (2006) Carsten Kirstein, H.M.: Interaction with a projection screen using a camera-tracked laser pointer. In: Proceedings of Conference on Multimedia Modeling. (1998) Nielsen, D.R.T.N.: Laser pointer interaction. In: Proceedings of CHI01. (2001) Winograd, T., Guimbretiere, F.: Visual instruments for an interactive mural. In: CHI99 Abstracts. (1999) Rekimoto, J.: Pickcandcdrop: A direct manipulation technique for multiple computer environments. In: Proceedings of UIST97. (1997) Xing Chen, J.D.: Lumipoint: Multi-user laser-based interaction on large tiled displays. In: Stanford CS Technical Report. (2001) 7. Ji-Young Oh, W.S.: Laser pointers as collaborative pointing devices. In: Graphics Interfaces. (2002) GeiBler, J.: Shuffle, throw or take it! working efficiently with an interactive wall. In: CHI 98 LateCBreaking Results. (1998) Pederson, E., M.K.M.T.H.F.T.: An electronic whiteboard for informal workgroup meetings. In: Proceedings of CHI93. (1993) Khan, A., F.G.A.D.B.N.K.G.: A remote control interface for large displays. In: Proceedings of UIST04. (2004)

10 10 Liang Zhang et al. 11. Yuanchun Shi, Weikai Xie., G.X.R.S.E.C.Y.M., Liu, F.: The smart classroom: Merging technologies for seamless tele-education. In: IEEE Pervasive Computing. (2003) Xie, W.: Smart platform: A software infrastructure for smart space (siss). In: ICMI2002,IEEE CS Press. (2002)