Multi-Touch Gestures for 3D Objects

Transcription

1 Whitepaper Multi-Touch Gestures for 3D Objects

2 Multi-Touch Gestures for 3D Objects Concepts for intuitively moving 3D Objects using Gestures The widespread acceptance of smartphones and tablets has also led to an increased demand for industrial and medical solutions based on multi-touch technology. Together with some of their customers, infoteam identified this demand, evaluated and implemented gesture-based concepts for interacting with 3D objects. This white paper presents the usage concepts, the ideas they are based on and their implementation as well as the findings of a user survey. 1 Introduction and Motivation AutHor Daniel Fuchs infoteam Software AG Summary Using gestures to intuitively move 3D objects Keywords Multi-touch, 3D objects, gestures, usage concept, OpenGL, WPF, RST Version 1 # WP , infoteam Software AG Currently, the majority of multi-touch displays are used in mobile devices such as smartphones and tablets. Due to the devices prevalence, the usage concepts implemented for them have implicitly developed into standards. This includes zooming gestures for minimising or maximising content, or wiping gestures for scrolling. Touch-control applications are widely accepted by users, not least because of their ease of use. This is the reason why multi-touch displays are increasingly used in other areas of application, such as medical technology, to realise innovative and intuitive device usage concepts. Medical applications certainly differ in many ways from those for smartphones or games consoles. For example, users in sterile environments need to be able to quickly yet precisely manipulate faithful copies of 2D or 3D representations of human organs on their screens. For interacting with two-dimensional content the concept of RST (Rotate, Scale, Translate) gestures has been largely adopted. For handling three-dimensional content, however, there is no harmonised, or at least widespread usage concept yet. For this reason infoteam implemented different usage concepts for manipulating 3D objects using multi-touch displays. In addition, we cooperated with one of our clients and used a sample applica- 2

3 tion scenario with specific use cases to present in detail the advantages and disadvantages of the different concepts. 2 2D Gestures and Technologies 2.1 Gestures: Distinction and Terminology Basically, gestures can be distinguished as continuous or discrete gestures. The system recognises a discrete gesture only once the touch action has been completed; discrete gestures will trigger single events. Example: tapping the monitor to click a button (same as a single mouse click). The system recognises this movement as a tap gesture only after the finger has left the display at the end of the gesture because it realises that the contact was very brief and in a confined space. Now the system will trigger a single response. Contrary to this, a continuous gesture will trigger a response while still being performed on the monitor. This response continues throughout the whole duration of the touch contact and will be adjusted depending on the movement. Pinching two fingers, or moving them apart on the monitor, is an example for a continuous gesture ( pinch-to-zoom ). The movement scales an object or changes the zoom level of the representation. Gestures are also classified as directly or indirectly manipulative. If the gesture is directly manipulative, the reference point to an object on the display is created by directly tapping the object. Moving the finger over the display, it is the same pixel of the object that remains underneath the user s finger for the entire duration of the gesture. So if a finger is used to move an object along the screen the object remains directly underneath the user s finger. This is not the case with indirectly manipulative gestures. Here, the touch point of the touch input is irrelevant for the movement of the object. Only the movement of the gesture relative to the object is of relevance. Being able to use more than one finger or hand with multi-touch displays allows a significantly greater number of degrees of freedom than in traditional mouse-controlled user interfaces. Having a pointer on one screen plane restricts mouse-controlled user interfaces to two de- 3

4 grees of freedom. Using more than one pointer for entering positions on a multi-touch display provides two additional degrees of freedom for every additional pointer-cum-finger. Integrating these options smartly into the usage concept makes it possible to assimilate device usage more closely to the interaction with real objects. 2.2 Using Standard 2D Gestures Over the course of the last years, a few gestures have established themselves as cross-operating system standards, being supported by all the common systems (e. g. Windows 7/8, Mac OS X, ios). Table 1 shows the standard gestures we used as the basis for our 3D usage concepts, and to which we added several other gestures. Table 1: Overview of established RST gestures 4

5 3 Concepts for applying 2D Gestures to 3D Objects The challenge in using a multi-touch display to control three-dimensional objects lies in the two-dimensional nature of the input device, which is a flat monitor. Cleverly choosing suitable gestures enables you to map all the necessary actions. However, it requires more than just one finger since a single finger on the multi-touch display only has two degrees of freedom. Only the combination of several fingers, with each finger adding another degree of freedom, makes it possible to eliminate the need for additional controls and to realise system control exclusively by way of gestures. It seems natural to use already established RST gestures or slight variations of them for commonly used functions. It is desirable then for these gestures to have the expected result: for example, you would want a pinch-to-zoom or pinch-to-scale gesture to result in some sort of minimisation or maximisation of an object or a view. The sample implementation mentioned earlier is a 3D model viewer for displaying a three-dimensional object. To achieve this it is necessary to move and rotate the object along all of the axes. This application makes it possible to simplify and restrict the required set of gestures to some extent. For the sake of our example, we have made some simplifications: we only display a single 3D object, eliminating the need for selection gestures for representation and interaction. In addition, this single object will only rotate around its centre because during rendering, the correct centre is directly assigned to the object as its source. This use case leverages RST gestures as much as possible and for reasonable movements. For example, the scaling gesture pinch-to-zoom triggers a movement of the object along the Z axis (axis exiting the screen plane). This corresponds to moving an object closer to or further away from the camera, hence a zoom. A translation occurs when moving a single finger on the screen in the usual way. This movement is only performed in the X-Y plane. To change the depth you also need an additional scaling gesture. Furthermore, the standard rotate gesture will cause the object to rotate around the Z axis (see figure 1). Figure 1: Rotating an object around the object centre 5

6 3.2 Concept A: Rotation around fixed Axes The aim of the concept for rotating around fixed axes is to use the two-finger translate gesture for moving along a defined axis in a certain direction. This serves two purposes: you can rotate the 3D object around the X axis by moving your fingers in parallel to the Y axis direction (the screen s vertical direction). You can rotate the 3D object around the Y axis by moving your fingers in parallel to the X axis direction (the screen s horizontal direction). Figure 2: Representation of a 3D object and rotation around the object centre (a circle indicates a touch point on the display) The direction of the respective finger movement indicates the sense of rotation. Figure 2 serves to illustrate this concept. 3.3 Concept B: 3D Rotation according to the Sticky Fingers Principle This method uses a 3-finger gesture and both hands to rotate the object out of the plane. Two fingers of one hand define a rotational axis above the object. One finger of the second hand then moves vertically to this axis to rotate the object around the axis. The direction in which the free finger moves indicates the sense of rotation around the axis (see figure 3). Figure 3: Rotation according to the Sticky Fingers principle (a circle indicates a touch point on the display) 6

7 3.4 Concept C: 3D Rotation according to the Arcball Principle The object centre defines the rotational axis, which is vertical to the direction in which the two-finger translate gesture moves, while the angle of the movement direction to the rotational axis (either +90 or -90 ) defines the sense of rotation. Figure 4 shows the movements and illustrates the corresponding steps. Since we realised the movement as a continuous gesture you can change the direction and hence the rotational axis at any time during the rotation without the need of having to restart your gesture. So despite the fact that this concept uses the same gestures as concept A, it allows you to freely choose your rotational axis. Figure 4: Representation of rotating a 3D object according to the Arcball principle (a circle indicates a touch point on the display) 7

8 4 Sample Implementation to compare Usage Concepts You can only use multi-touch gestures to manipulate the sample 3D model. The implementation enables us to compare the three usage concepts presented in the previous chapters by way of application modes. The A, B and C modes correspond to the concepts introduced in Chapter 3. Three axes were added to the 3D model to represent the current orientation of the displayed object in space (green = Y axis, red = X axis, blue = Z axis). The program is a C# application capturing the touch events with the help of the embedded Windows Figure 5: Screenshot of the 3D model viewer Presentation Foundation (WPF) touch API of the Microsoft.NET Framework. We used a distinct class to realise gesture recognition. For this purpose, we implemented a unique model. Based on different movement vectors of the individual fingers we were able to decide which gesture the user was performing (see figure 5). Once the system has recognised the gesture it applies a number of scaling factors and then moves the 3D model. We have used a unique control to directly integrate the 3D object with WPF using OpenGL. 5 Evaluating the Usage Concepts We used the sample implementation to conduct user surveys. Our survey sample had to assess how intuitive, precise and responsive the different concepts were. We also asked them to perform certain rotations with the object. The time and the number of gestures required were logged. Throughout, users described concept A as a solid means for relatively easy handling of three-dimensional objects. The disadvantage of the solution is that complex rotations require you to perform several consecutive turn movements because you cannot choose the rotational axis. Concept B, on the other hand, is the most precise of all the three usage concepts. It delivers good results regardless of the task s complexity. However, users didn t find it intuitive. They voiced concerns over the fact that 8

9 they had to use both hands, making it difficult to integrate the concept into real-life applications. Concept C is very intuitive, yet offers only a satisfactory level of accuracy. This becomes very clear when comparing the results of simple rotations with those from the other two concepts presented here. On the other hand, concept C yields a result more quickly when rotations are more complex as the free choice of axis eliminates the need for a workaround using several rotations. To summarise, we have found that all of the three usage concepts presented here are suitable for handling 3D objects and that each concept brings its own set of advantages and disadvantages. Table 2 shows the major differences between the individual concepts. Table 2: Comparing the usage concepts 9

10 6 Conclusion and Outlook There are major differences in terms of user experience and accuracy between the examined usage concepts for manipulating 3D objects using multitouch input. To choose the most appropriate usage concept in an individual application you need to define the precise workflows within the planned application environment and examine them with a close view to the required level of accuracy. One key aspect is certainly to select an end-to-end concept for similar areas of application to increase the level of acceptance of multitouch input methods among their users and, most importantly, to avoid usage errors. 10

11 7 List of references [Table 1] [Fig. 1] [Fig. 2] Illustrations taken from Touch Gesture Icons. Online: fetched on 27 February 2013 (Hancock et al) Sticky Tools: Full 6DOF Force-Based Interaction for Multi-Touch Tables. ITS (pages ). Banff, Canada: ACM. (Shoemake, K. (1992). ARCBALL: A user interface for specifying three-dimensional orientation using a mouse. Proceedings of the conference on Graphics interface 92 (pages ). San Francisco: Morgan Kaufmann Publishers Inc. 8 Glossary RST WPF Rotate, Scale, Translate Windows Presentation Foundation 11

12 Contact infoteam Software AG Am Bauhof 9 D Bubenreuth Phone: +49 (0) 9131 / Fax: +49 (0) 9131 / info@infoteam.de infoteam Software AG Emil-Figge-Straße 80 D Dortmund Phone: +49 (0) 231 / Fax: +49 (0) 231 / dortmund@infoteam.de infoteam Software AG Laubisrütistrasse 44 CH-8712 Stäfa Phone: +44 (0) Fax: +44 (0) info@infoteam-software.ch infoteam Software (Beijing) Co., Ltd. Zhongguancun North Street 151 Yan Yuan Resource Tower, Room , Haidian District Beijing China Phone: +86 (0) Fax: +86 (0) info@infoteam.com.cn