International Journal of Human-Computer Studies

Transcription

1 Accepted Manuscript Evaluation of Edge-based Interaction on a Square Smartwatch Sunggeun Ahn, Jaeyeon Lee, Keunwoo Park, Geehyuk Lee PII: S (17) DOI: /j.ijhcs Reference: YIJHC 2145 To appear in: International Journal of Human-Computer Studies Received date: 4 October 2016 Revised date: 14 August 2017 Accepted date: 24 August 2017 Please cite this article as: Sunggeun Ahn, Jaeyeon Lee, Keunwoo Park, Geehyuk Lee, Evaluation of Edge-based Interaction on a Square Smartwatch, International Journal of Human-Computer Studies (2017), doi: /j.ijhcs This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

2 Highlights Exploration of edge-based interaction for a square smartwatch: single-edge, multiple-edge, and, edge x screen techniques. Evaluation of 1D and 2D pointing using square smartwatch edges in comparison with touchscreen-based pointing. Qualitative and quantitative empirical data for the design of edge x screen interaction. Discussion of design options of edge x screen interaction based on the empirical results. 1

3 Evaluation of Edge-based Interaction on a Square Smartwatch Sunggeun Ahn, Jaeyeon Lee, Keunwoo Park, Geehyuk Lee HCI Lab, KAIST 291 Daehakro, Yuseong, Daejeon 34141, South Korea Abstract Researchers are proposing many approaches to overcome the usability problem of a smartwatch owing to its small touchscreen. One of the promising approaches is to use touch-sensing edges to expand the control space of a smartwatch. We considered possible interaction techniques using touch-sensing edges in combination with the smartwatch touchscreen: single-edge, multi-edge, and edge screen (edge and touchscreen in combination). We call these techniques square watch interaction (SWI) techniques in this paper because they exploit the form factor of a square smartwatch. To explore the design space and evaluate the usability of the SWI techniques, we implemented a square smartwatch prototype with touch-sensitive edges, and conducted a series of user experiments. The experiment results showed that the SWI techniques enable precise 1D pointing and occlusion-free 2D pointing. The experiments also produced empirical data that reflect human manual skills for the edge screen techniques. The produced empirical data will provide a practical guideline for the application of the edge screen techniques. Keywords: 1. Introduction smartwatch user interface, edge-based interaction, edge screen interaction, square watch interaction The computing power of a smartwatch and its network environment are already good enough to make it a computing platform for diverse mobile applications. A more difficult challenge seems to be developing new user interfaces and new interaction techniques to overcome usability problems due to its small touchscreen. For example, the occlusion problem of a touchscreen interface is now more pronounced than before because its touchscreen size is comparable to the finger size. In addition, its small touchscreen may make useful multi-finger gestures, such as pinch-to-zoom, impractical. To overcome the small screen problem of a smartwatch, researchers have proposed many new user interface ideas. A common approach is to extend the control space beyond the touchscreen. The control space can be extended to the edges of a watch (Oakley and Lee, 2014), to the space above a watch (Harrison and Hudson, 2009), or even to the skin area around a watch (Laput et al., 2014). Among the various ways to extend the control space of a smartwatch, we are particularly interested in extending the control space to the edges of a watch, specifically the four side faces of a square watch, because of the following advantages: 1) Watch edges are physically contiguous with the touchscreen. Mapping from the edges to the GUI controls is almost direct. 2) Watch edges provide a physical Corresponding author. addresses: topmaze@kaist.ac.kr (Sunggeun Ahn), jaeyeonlee@kaist.ac.kr (Jaeyeon Lee), keunwoo@kaist.ac.kr (Keunwoo Park), geehyuk@gmail.com (Geehyuk Lee) guide for a stable finger movement. Moving a finger along an edge enables a precise operation. 3) The control space combining the touchscreen and the edges is small. Manipulating the touchscreen and the edges simultaneously with one hand is easy. The implication of the first two advantages seems clear: the edges will make precise 1D physical controls and do not occlude the screen. The third advantage is, in fact, more appealing to us and prompted us to imagine many possible operations with the simultaneous use of the touchscreen and the edges, which we call edge screen techniques in the following sections. For instance, touching an icon with a thumb on an edge may invoke a secondary function for the icon. In addition, sliding on the edge with another finger on a volume icon may change the music volume. We conducted a brainstorming session to collect possible scenarios using a touchscreen and edges for a smartwatch with a square form factor. While compiling the scenarios, which we call square watch interaction (SWI), we thought of the following questions. Will the edges really make a precise and occlusion-free physical control? Will the two finger operations covering the edges and the screen be really convenient? Among the various combinations of the edges and the screen areas, which one will be more usable? The goal of the current study was to answer the above questions. Toward this goal, we constructed a smartwatch prototype with touch-sensing edges, and, using the prototype, we implemented five main SWI operations that we selected from the SWI scenarios. Finally, we designed and conducted a series of user experiments to answer the above questions. The results of the experiments support Preprint submitted to International Journal of Human-Computer Studies August 30, 2017

4 the advantage of precise and occlusion-free edge-based operations in comparison with touchscreen-based operations. In addition, the results provide empirical data that can be used as a basis for the design of various SWI operations, which we believe is the main contribution of the current study. 2. Related Work In this section, we review the diverse approaches to overcome the limitations of the small touchscreen of a smartwatch. Researchers have proposed the use of space around a smartwatch, including the mid-air space and the skin surface, as an extended input channel for a smartwatch. Kim et al. (2007) proposed the Gesture Watch, a contact-less hand-gesture recognition system using infrared proximity sensors. Harrison and Hudson (2009) achieved a more precise gesture interaction by using magnetic sensors and a magnet attached to a fingertip. Laput et al. (2014) utilized the skin around a smartwatch as an input space by projecting small, touch-sensitive icons onto the skin. Sensing techniques using electric fields (Zhang et al., 2016c) and acoustic waves (Zhang et al., 2016a) were developed to capture touch inputs on the skin near a smartwatch. More recently, Sridhar et al. (2017) utilized a wearable depth sensor to capture multi-finger gestures on and above the skin on the back of a hand. Han et al. (2015) proposed an interaction strategy for a hover-tracking smartwatch that allows a user to continue a touch-screen operation into the mid-air space. Researchers also proposed the use of other parts of a smartwatch as an input space, including the bezel of the screen (Zhang et al., 2016b), the body of a smartwatch (Xiao et al., 2014; Ogata and Imai, 2015; Yeo et al., 2016; Seyed et al., 2016), and the band area (Lyons et al., 2012; Perrault et al., 2013; Funk et al., 2014; Ahn et al., 2015) of a smartwatch. Among the different parts of a smartwatch, the screen edge is the most popularly explored interaction area except for the touchscreen. Early researchers focused on the possibility that an edge provides tactile guidance for a precise and stable linear movement. Blasko and Feiner (2004) proposed the use of tactile landmarks to enhance and enrich stroke interactions on the edge of a smartwatch. Blaskó and Feiner (2006) also showed that tactile landmarks are helpful in eyes-free stroke interaction with a cursorless numeric entry system. Ashbrook et al. (2008) investigated the effectiveness of round watch interaction techniques wherein the watch bezel is used as a movement guidance. Kerber et al. (2016) investigated the usability of the digital crown of Apple Watch and the rotatable bezel of Samsung Gear S2. More recently, researchers have explored possible interaction techniques utilizing the touch-sensitive side edge of a watch. Oakley and Lee (2014) explored interaction techniques using the side edge of a round smartwatch. They proposed occlusion-free target selection and gesture input using a grasping pattern around the smartwatch. A patent by Yoo and Kim (2015) describes a comprehensive set of interaction techniques using the edges of a square smartwatch. The patent includes interaction techniques using one edge or two edges in combination for 1D pointing, 2D pointing, and other screen manipulation operations. Kubo et al. (2016) utilized the combination of sequential inputs on the touchscreen and bezel of a rectangular smartwatch as input gestures. They showed that the four sides of the screen bezel could be used as distinct input spaces. We could gain from these previous works many valuable insights for utilizing the edges of a square watch. First, movements along the edge may be more precise and stable because of tactile guidance. This also means that we may utilize a touch on the edge for a more precise input. Second, the four edges of a rectangular smartwatch may be used as distinct input spaces. Third, the touchscreen and the edges are close, and, therefore, we may manipulate easily the touchscreen and the edges in combination using a single hand. These insights suggest the potential of the input space using the edges of a square watch. We noted, however, that this input space has not been fully explored, and, in particular, we needed empirical data for designing the edge-based input space. Therefore, our main focus in the current study is to produce empirical data to highlight the strong and weak points of edge-based interaction and provide a basis for the optimal design of various edge-based interaction techniques for a square smartwatch. 3. Square Watch Interaction Below, we summarize SWI scenarios arranged in three categories: 1) using a single edge, 2) using multiple edges at the same time, and 3) using the touchscreen and an edge in combination Single Edge Interaction An edge can be mapped to a 1D control such as a list control and a scroll bar. As shown in Figure 1a, list items on the smartwatch screen may be small and direct pointing may be difficult owing to occlusion by the finger. In this case, an edge that is parallel to the list control may be used to select an item. The resulting interaction is occlusionfree and nearly direct, similar to the old technique of using an offset cursor (Potter et al., 1988). In Figure 1b, the right edge, mapped to a vertical scroll bar, is used to scroll the page. Similarly, the bottom edge may be mapped to a horizontal scroll bar. The mapping between the edges and the scroll bars in these cases is intuitive and the scroll operations are occlusion free Multiple Edge Interaction Selecting a small target in the smartwatch screen is difficult owing to occlusion by a finger. Figure 1c shows a technique using a horizontal edge and a vertical edge in combination to define a point on the screen. Selecting a

5 Figure 2: The SWI prototype with a linear multi-touch sensor around a watch: (a) top-down view and (b) side view showing the sensor. Figure 1: Single-edge (a and b), multi-edge (c and d), and edge screen (e h) interaction examples: (a) 1D pointing, (b) 1D scrolling, (c) 2D pointing, (d) zooming in and out, (e) alternative function, (f) alternative scrolling, (g) changing the music volume, and (h) changing a number. target in this way may be slower than touching a target on the touchscreen directly, but is occlusion free and may enable precise pointing. Two finger gestures for zooming or rotation on a small smartwatch screen may be difficult. An alternative technique using two edges in combination is shown in Figure 1d. Zooming can be done using the bottom edge and the right edge. Moving away from the bottom-right corner is reminiscent of an enlarging gesture, and moving the other way toward the same corner is reminiscent of a reducing gesture; these two operations can be mapped to zooming-in and zooming-out, respectively. Rotation may be done by using two opposite sides, such as the top edge and the bottom edge. Touching two opposite edges and sliding the fingers in opposite directions as if twisting the watch are reminiscent of a rotating gesture Edge Screen Interaction In this section, we present interaction scenarios wherein the touchscreen and the edges are used in combination. The edge has the advantage of precise and stable 1D input, whereas the touchscreen has the advantage of direct 2D input. Their complementary combination is expected to enable efficient novel input techniques An edge as a modifier Modifiers are used for expanding the input vocabulary. In touchscreen interfaces, a long touch or multiple touch points may be regarded as a modifier. However, using a long touch is often annoying and using multiple fingers on a small touchscreen may not be convenient. As an alternative, an edge may be used as a modifier for a touchscreen operation. Figure 1e shows an example. A user touches a name in a contact list to make a call. If a user touches a name while touching the bottom edge, however, the same touch may open a page showing the contact s details. Figure 1f shows another example. The user slides a finger on the screen to scroll the contact list. If the user does it while touching the bottom edge, however, the list may scroll faster, e.g., jumping to the beginnings of the alphabetical sections of the list. From these examples, we expect that some screen areas may be difficult to reach while a certain edge is being touched owing to the ergonomic constraints of the hand Selective Control Selecting an object on the smartwatch screen may be easy, but changing the properties of the selected object may not be easy owing to the limited screen space. Selective control is a possible solution for this situation. Figure 1g shows a selective control scenario. The user changes the music volume by sliding a finger on the right edge while touching a volume control on the touchscreen. Figure 1h shows another scenario, wherein the user changes the minute field of an alarm setting by sliding on the bottom edge while touching the minute field. From these examples, we may expect that there will be preferred combinations of a screen area and an edge owing to the ergonomic constraints of the hand. 4. SWI Prototype SWI requires a square smartwatch with touch-sensing edges. Each edge should be able to detect a finger touch as well as its position on the edge. Figure 2 shows the SWI prototype. We constructed a multi-touch sensor around a smartwatch (LG G Watch, resolution, 29.6 mm 29.6 mm screen). The multi-touch sensor is based on the TouchString concept introduced in Gu and Lee (2011). The sensor consists of a series of 32 infrared LEDs and 32 photodiodes (8 LEDs and 8 diodes on each side). The output of the sensor, a

6 Figure 3: Multi-touch sensing: (a) a proximity image from the sensor and (b) the corresponding touch situation. 32-pixel proximity image around the watch, is sent to a PC via an Arduino board for signal processing. The bar graphs in Figure 3a visualize the proximity image when two fingers are on the edges of the watch, as shown in Figure 3b. The position of the finger on the right edge, for instance, can be estimated by calculating the centroid of the finger image on the right edge. It is also required to know the touch state of a finger, i.e., whether a finger is on the sensor or over the sensor. The touch state can be determined from the amplitude of a finger image. The signal processing steps to determine the position of a finger are straightforward. However, some trial-and-error procedures were needed for the robust detection of touch events. A hysteresis mechanism was used to avoid chattering while a finger is detached from the edge. In addition, the derivative of the sensor output, as well as the output value itself, was used for more robust detection of touch events. The PC sends the signal processing results to the watch in the form of GUI events via a Bluetooth link. We share the implementation details of the SWI prototype on our web page 1 for those who may be interested in reproducing our study. 5. Experiments To examine SWI empirically, we first chose five unit SWI operations that underlie the diverse SWI scenarios reviewed in Section 3. 1D Select - to select a target in a 1D list. 2D Select - to select a target in a 2D space. Shift-Select - to select a screen area while touching an edge. Shift-Drag - to drag in the screen while touching an edge. Pivot-Slide - to slide along an edge while touching a screen area. 1 id= We then conducted a preliminary experiment and five main experiments to examine the unit SWI operations. The first goal of the experiments was to compare the SWI pointing operations with their touchscreen counterpart operations. The second but more important goal of the experiments was to produce empirical reference data for the design of the unit operations. For instance, there can be many possible edge and direction combinations in the case of Shift-Drag, and we need an empirical ground to determine the best combinations. In the first subsection, we describe the participants because some of the experiments were conducted sequentially with the same participants. In the subsequent subsections we describe the preliminary and the five main experiments Participants The preliminary experiment and Experiments 1 and 2 were conducted together in sequence with the same 12 participants recruited from our university (5 males and 7 females, mean age = (SD = 3.38)). They were all right-handed and smartphone users, but none of them was a smartwatch user. Experiment 3 was conducted with 16 right-handed participants recruited from our university (13 males and 3 females, average age = (SD = 2.91)). All of them were smartphone users, and four of them had an experience of using a smartwatch. None of them participated in the previous experiments. Experiments 4 and 5 were conducted together with 16 right-handed participants recruited from our university (13 males and 3 females, mean age = (SD = 2.14)). All of them were smartphone users, and two of them had an experience of using a smartwatch. None of them participated in the previous experiments. The order of Experiment 4 and 5 was counterbalanced across the participants. In all experiments, all participants wore a smartwatch on their left arm while performing the experiment tasks Preliminary Experiment This experiment was preliminary to Experiment 1; the goal of this experiment was to determine the edge to be used in Experiment 1. In a pilot study, we learned that the right edge and the bottom edge were clearly preferred to the left edge and the upper edge; however, we could not determine the preference between the right edge and the bottom edge. Therefore, the preliminary experiment was a within-subject experiment for the 1D Select task with two independent variables: Edge (Right and Bottom) and Size (5, 10, 20, 35, 70, and 140 pixels, where 1 pixel = mm). The measured metrics were the mean completion time (Completion Time) and the error rate (Error Rate). The whole experiment took approximately 20 min for each participant. The main conclusion was that Edge had no significant main effect on Completion Time or on Error Rate. In addition, there was no significant Edge Size interaction

7 Figure 4: 1D Select task: (a) selecting a target with an edge, (b) a target in a vertical bar, and (c) a target in a horizontal bar. effect on Completion Time or on Error Rate. The trends in Completion Time and Error Rate as Size increased were similar to those of Experiment 1 (the Edge case). However, 10 out of the 12 participants answered that they preferred the right edge to the bottom edge. On the basis of these results, we decided to use the right edge in Experiment Experiment 1: 1D Select The goal of this experiment was to evaluate the performance of the edge-based 1D pointing technique in comparison with its touchscreen-based counterpart. In particular, we were interested in the benefit of the occlusion-free aspect of the edge-based technique. Our hypothesis was that the performance of the edge-based technique would only be affected by Size, whereas that of the touchscreen-based technique would be affected by both Size and W idth Task As shown in Figure 4a, a vertical or horizontal bar with a target appears on the screen. The user is required to select the target. When the user touches an edge along the bar, a thin red cursor appears and moves with the finger. When the user moves the cursor over the target, the target turns magenta. When the user releases the touch, the target is finally selected. The watch provides a feedback about the success of the operation: a brief vibration for success and a black screen for failure. There are two task parameters as explained in Figure 4b and c: the size (Size) and the width (W idth) of the target. Note that Size refers to the length of the target along the direction of the finger movement in both cases Design and Procedure The experiment had a within-subject factorial design with three independent variables: T ech (Edges or Screen), W idth (Narrow (20 pixels) or Wide (80 pixels)), and Size (5, 10, 20, 35, 70, and 140 pixels where 1 pixel = mm). The participants performed one practice block and three main blocks. Each block consisted of 2(T ech) 2(W idth) factorial sessions in the W idth-major order. In other words, the participants repeated a session for each T ech for the first W idth and repeated a session for each T ech for the next W idth. The order of the factorial conditions was counterbalanced across participants by using a balanced Figure 5: Trends of (a) Completion Time (CT) and (b) Error Rate (ER) for the 2(T ech) 2(W idth) factorial cases as Size increased in Experiment 1. Error bars represent standard errors. Latin square. In each session, the participants performed ten 1D Select trials for each of the six sizes. The position of the target was random, and the order of Size was shuffled randomly. In total, each participant performed 720 (3 blocks 4 sessions 6 sizes 10 trials) pointing trials. We measured the mean completion time (Completion Time) and the error rate (Error Rate) for each session. The participants were not allowed to retry a selection in a single trial, and, therefore, Error Rate was simply the rate of failed trials. This was also true in all experiments in the current study Results Figure 5 shows the trends of Completion Time and Error Rate in the 2(T ech) 2(W idth) factorial cases as Size increased. In all cases, Completion Time tended to decrease and Error Rate tended to decrease as Size increased. Completion Time for the Screen case appears to be smaller than for the Edges case. Error Rate in the Screen and Narrow case appeared to be affected considerably by Size. 2(T ech) 2(W idth) 6(Size) RM-ANOVA (with a Greenhouse-Geisser adjustment) results on Completion Time and Error Rate are shown in Table 1. All independent variables had a significant main effect on Completion Time and Error Rate except that W idth did not have a significant main effect on Completion Time. Post hoc paired-sample t-tests with Bonferroni correction showed that the Edges case had a significantly lower Error Rate than the Screen case when Size was smaller than 20 pixels (2.12 mm) in the Narrow case (p < for all cases). To examine the effect of W idth and Size on Error Rate in each of the two T ech cases, we conducted a 2(W idth) 6(Size) RM-ANOVA on Error Rate for each of the two T ech cases (Edges and Screen), and the results are shown in Table 2. As expected, Size only had a significant main effect on Error Rate in the Edges case, whereas both 6

8 Table 1: Results of the 2(T ech) 2(W idth) 6(Size) RM-ANOVA on Completion Time (CT) and Error Rate (ER) in Experiment 1. Shaded p-values represent significant effects. DV Effect df df2 F-value p-value CT T ech(t ) <.001 W idth(w ) Size(S) <.001 T W T S <.050 W S T W S ER T ech <.001 W idth <.001 Size <.001 T W <.001 T S <.001 W S <.001 T W S <.001 Table 2: Results of the 2(W idth) 6(Size) RM-ANOVA on Error Rate for the two T ech cases (Edges and Screen) in Experiment 1. Shaded p-values represent significant effects. Condition Effect df df2 F-value p-value Edges W idth(w ) Size(S) <.001 W S Screen W idth <.001 Size <.001 W S <.001 W idth and Size had a significant main effect on Error Rate in the Screen case Discussion In terms of Error Rate, edge-based pointing had an advantage over touchscreen-based pointing for narrow targets as we expected because it does not have an occlusion problem. In terms of Completion Time, however, touchscreen-based pointing was better than edge-based pointing regardless of the target width. We speculate that this difference was due to the indirect nature of edge-based pointing. In fact, three participants said that they felt remoteness when using an edge owing to the separation between a touched position and a cursor position. The main conclusion of this experiment seems to be that edge-based pointing is more effective for accurate 1D pointing but is slower than touchscreen-based pointing Experiment 2: 2D Select The goal of this experiment was to evaluate the performance of the edge-based 2D pointing technique in comparison with its touchscreen-based counterpart. Owing to the occlusion-free nature of edge-based pointing, we expected that it would be particularly advantageous for 2D pointing. Therefore, our hypothesis was that edge-based pointing would be more accurate (i.e., having a lower er- 7 Figure 6: 2D Select task: (a) selecting a target with two edges, (b) selecting a target with the touchscreen, and (c) the size of a target. ror rate) than touchscreen-based pointing, in particular for small targets Task As shown in Figure 6a, a circular black target appears on the screen. The user is required to select the target. When the user touches the right and bottom edges with his/her fingers, a cross-hair (with a small blue circle at the center) appears on the screen and moves as the two fingers move. When the user moves the cross-hair over the target, the target turns cyan. When the user releases one of the fingers, the target is finally selected. The watch provides a feedback about the success of the operation in the same way as in the 1D Select case. Figure 6b shows the touchscreen counterpart; the target is selected when the finger is lifted from the screen. There is only one task parameter as shown in Figure 6c: the diameter (Size) of the target Design and Procedure The experiment had a within-subject factorial design with two independent variables: T ech (Edges or Screen) and Size (8, 16, 24, 32, 40, and 48 pixels where 1 pixel= mm). The participants performed one practice block and five main blocks. Each block consisted of two (T ech) sessions. The order of T ech was counterbalanced across participants. In each session, the participants performed ten 2D Select trials for each of the six sizes. The position of the target was random, and the order of Size was shuffled randomly among the 60 trials (6 sizes 10 trials). In total, each participant performed 600 (5 blocks 2 sessions 6 sizes 10 trials) trials. We measured the mean completion time (Completion Time) of the task and the error rate (Error Rate) for each session. After the experiment, we collected subjective feedback about the two techniques from the participants Results Figure 7 shows the trends of Completion Time and Error Rate for the two T ech cases as Size increased. In both cases, both Completion Time and Error Rate tended to decrease as Size increased. Completion Time for the Screen case appeared to be shoter than that for the Edges case for all sizes. Error Rate in the Screen case appeared to be affected more by Size than that in the Edges case.

9 Figure 7: Trends of (a) Completion Time (CT) and (b) Error Rate (ER) for the two T ech cases as Size increased in Experiment 2. Error bars represent standard errors. Table 3: Results of the 2(T ech) 6(Size) RM-ANOVA on Completion Time (CT) and Error Rate (ER) in Experiment 2. Shaded p-values represent significant effects. T ech Effect df df2 F-value p-value CT T ech(t ) <.001 Size(S) <.001 T S <.001 ER T ech <.001 Size <.001 T S <.001 The results of the 2(T ech) 6(Size) RM-ANOVA (with a Greenhouse-Geisser adjustment) on Completion Time and Error Rate are shown in Table 3. All independent variables had a significant main effect on both of the dependent variables. To examine the effect of T ech on Error Rate in each of the six Size cases, we conducted six post hoc pairedsample t-tests on Error Rate with Bonferroni corrections. The results indicated that touchscreen pointing had a significantly lower Error Rate than edge-based pointing when the target size was 48 pixels (p < 0.050). On the other hand, edge pointing has a significantly lower Error Rate than touchscreen pointing when the target size is 8 or 16 pixels (p < for both cases) Discussion Edge-based pointing took longer Completion Time than touchscreen-based pointing for all target sizes. On the other hand, edge-based pointing exhibited a reasonably lower Error Rate (mean = 19.8%, SD = 13.8) even when the target size was as small as 16 pixels (1.696 mm), whereas touchscreen-based pointing exhibited a considerably higher Error Rate (mean = 55.8%, SD = 16.8) when the target size was 16 pixels. Four participants felt discomfort while pointing using an edge, and they preferred using a touchscreen for selecting a large target. On the other 8 Figure 8: Shift-Select task: (a) an example, (b) the four edges to hold, and (c) the four quadrants to touch. hand, all participants said that edge-based pointing was good for accurate pointing. There were higher individual variations in Error Rate in the case of edge-based pointing; 6 out of 12 participants maintained an Error Rate below 30% even for the smallest target (8 pixels; mm), and one of them showed an 8% Error Rate for the smallest target. The main finding of this experiment seems to be that edge-based pointing is more effective for selecting a small target but is slower than touchscreen-based pointing Experiment 3: Shift-Select Shift-Select is a two-finger operation usually performed by using a thumb and an index finger. The usability of Shift-Select may depend on many design parameters such as the finger for holding an edge and the target position on the screen. The goal of this experiment was to identify good combinations of design parameters for Shift-Select. In other words, we did not have any hypothesis in advance but aimed to produce reference data showing the relative usability scores of different Shift-Select design possibilities Task As shown in Figure 8a, a blue mark indicating a target edge appears, and a black square mark indicating a target screen quadrant appears. The user is required to touch the indicated quadrant while touching the indicated edge. The watch provides a brief haptic feedback in response to the touch and release actions. The watch provides visual feedback about the touched points on the edge during the practice blocks, but does not during the main blocks. After each trial, the screen stays blue for 500 ms, enforcing the user to wait. The user is instructed to aim at the marks on the screen and the edge, but does not need to be precise to complete the task. There are two task parameters, as shown in Figure 8b and c: the edge to hold (Edge) and the quadrant to touch (Quadrant) Design and Procedure We conducted two sub-experiments for the Thumb-Shift and Index-Shift cases. Thumb-Shift means that the thumb is used for shifting, i.e., one holds an edge with the thumb and selects a target with the index finger. Likewise, Index- Shift means that the index finger is used for shifting, i.e., one holds an edge with the index finger and selects a target with the thumb. Except for the finger used for shifting, the

10 two sub-experiments were identical. Each sub-experiment had a within-subject factorial design with two independent variables: Edge (Right, Top, Left, and Bottom) and Quadrant (target screen area: Q1, Q2, Q3, and Q4. See Figure 8c). We decided to conduct two sub-experiments for the Thumb-Shift and Index-Shift cases separately, instead of conducting a single combined experiment where the shifting finger is an independent variable, because a strong interaction between the shifting finger and the two independent variables, Edge and Quadrant, was obvious, which, in turn, would make the interpretation of the statistical analysis results unnecessarily involved. Participants performed one practice block, five main blocks, and one evaluation block. Each block consisted of four sessions (one for each of the Edge cases). The order of Edge was counterbalanced across participants using a balanced Latin square. In each session, the participants performed five Shift-Select trials for each of the four Quadrant cases. The order of Quadrant was shuffled randomly. In total, each participant performed 400 (5 blocks 4 sessions 4 quadrants 5 trials) trials during the main sessions. In the evaluation block, the participants rated the comfort (Comfort) scores of the factorial conditions using a 7-point Likert scale (1 = very uncomfortable and 7 = very comfortable) while trying Shift-Select in the factorial conditions. Using the data from the five main blocks, we computed the mean completion time of the task (Completion Time) and the precision of the target selection (Precision) for each of the 4(Edge) 4(Quadrant) factorial conditions. Precision was the standard deviation of the selection errors of the five samples in each session Results Figure 9 shows the Comfort, Completion Time, and Precision data for the 4(Edge) 4(Quadrant) factorial conditions from the two sub-experiments. Each of the crossshaped tables in the figure summarizes 16 data values for the 16 factorial conditions. The four wings in the table correspond to the four Edge cases, and the four quadrants in each wing correspond to the four Quadrant cases. For example, the number 1.81 in the top wing of the first table in Figure 9a is the Comfort value for the Top and Q2 condition in the Thumb-Shift case. The trends of Comfort, Completion Time, and Precision seemed to be highly dependent on Edge, but did not seem to be dependent on Quadrant as much. Figure 10 shows the dependency of Comfort, Completion Time, and Precision on Quadrant and Edge. The curves for the Thumb-Shift case look inverted with respect to the curves for the Index-Shift cases, reflecting the opposing roles of the thumb and the index finger. We conducted 4(Edge) 4(Quadrant) RM-ANOVA tests on Completion Time, Precision, and Comfort for each sub-experiment. We used a Greenhouse-Geisser adjustment whenever the sphericity was violated. In the tests Figure 9: Results of Experiment 3: (a) Comfort, (b) Completion Time (CT), and (c) Precision (PR) data for the 4(Edge) 4(Quadrant) factorial conditions from the two sub-experiments. on Comfort, we used the aligned rank transform (Wobbrock et al., 2011) before performing the RM-ANOVA. The results are shown in Table 4. Significant effects are indicated by shaded p-values. In all cases, Edge had a significant main effect on all of the three metrics Discussion In the post-experiment interview, we could identify three types of ergonomic problems in the Shift-Select operation: 1) twisting of the wrist, 2) crossing between the thumb and the index finger, and 3) crowding of the thumb and the index finger. All participants mentioned the wrist twisting problem, 13 of them mentioned the finger crossing problem, and 7 of them mentioned the finger crowding problem. The Comfort data shown in Figure 9a reflect the wrist twisting problem clearly. When the top edge or the bottom edge was used for shifting, the severity of the twisting problem seemed to swing between two extremes depending on the finger used for shifting. When the left or the right edge was used for shifting, some twisting always occured regardless of the finger used for shifting, but the severity of the problem was mild. The finger crossing problem occured for some combinations of an edge and a quadrant, but they did not seem to 9

11 Table 4: Results of the 4(Edge) 4(Quadrant) RM-ANOVA on Completion Time (CT), Precision (PR), and Comfort (CF) for each subexperiment in Experiment 3. Shaded p-values represent significant effects. CF CT PR Effect df df2 F-value p-value df df2 F-value p-value df df2 F-value p-value Thumb-Shift Edge(E) < < <.005 Quadrant(Q) < < E Q < < <.010 Index-Shift Edge < < <.050 Quadrant < E Q Figure 10: Dependency of Comfort, Completion Time (CT) and Precision (PR) on Edge and Quadrant in Experiment 3. be as problematic as the wrist twisting problem, possibly because the screen size of a smartwatch is small. For the same reason, however, the finger crowding problem may be more problematic, and it suggests not to place a Shift- Select target near the edge used for shifting Experiment 4: Shift-Drag Shift-Drag is similar to Shift-Select except that it is used for dragging a target instead of selecting it. Therefore, we used the results from Experiment 3 to predetermine the finger for shifting: the index finger for the right and top edges and the thumb for the left and bottom edges. The main question in this experiment was which of the (EdgeandDirection) combinations is good for Shift-Drag. In this experiment again, we did not have any hypothesis in advance but aimed to produce reference data showing the relative usability scores of different Shift-Drag design possibilities. We describe Experiments 4 and 5 in separate sections, even though they were, in fact, conducted in sequence with the same participants. The order of the two experiments was counterbalanced across participants Task As shown in Figure 11a, a blue mark indicating a target edge appears and a headed line (5/7 of the screen 10 Figure 11: Shift-Drag task: (a) an example, (b) the four edges to hold, and (c) the four drag directions. size) appears on the screen. The user is required to drag along the line from the head while maintaining a touch on the indicated edge. The watch provides brief haptic feedback in response to the touch and release actions. The watch provides visual feedback about the touched points on the edge and drag paths on the screen during the practice blocks, but does not during the main blocks. After each trial, the screen stays blue for 500 ms, enforcing the user to wait. There are two task parameters as shown in Figure 8b and c: the edge to hold (Edge) and the drag direction (Direction) Design and Procedure The experiment had a within-subject factorial design with two independent variables: Edge (Right, Top, Left, and Bottom) and Direction (counter-clockwise (CCW), clockwise (CW), toward the edge (Near), and away from the edge (Far). See Figure 11c). The participants performed 1 practice block, 20 main blocks, and 1 evaluation block. In each block, the participants performed one Shift-Drag trial for each of the 16 (4(Edge) 4(Direction)) combinations. The order of the 16 combinations was shuffled randomly. In total, each participant performed 320 trials (20 blocks 16 combinations 1 trial) during the main blocks. In the evaluation block, the participants rated the comfort (Comfort) of each of the 16 combinations using a 7-point Likert scale (1 = very uncomfortable and 7 = very comfortable). Using the data from the main blocks, we computed the mean completion time of the task (Completion Time).

12 Figure 12: Results of Experiment 4: (a) Comfort (CF) and (b) Completion Time (CT) data for the 4(Edge) 4(Direction) factorial conditions. Table 5: Results of the 4(Edge) 4(Direction) RM-ANOVA on Comfort (CF) and Completion Time (CT) in Experiment 4. Shaded p-values represent significant effects. DV Effect df df2 F-value p-value CF Edge(E) <.050 Direction(D) <.001 E D <.001 CT Edge <.001 Direction <.005 E D < Results Figure 12 shows the Comfort and Completion Time data for the 4(Edge) 4(Direction) factorial conditions. Each of the square tables in the figure summarizes the data values for the four edges for one of the four directions. For example, the number 4.81 in the first square in Figure 12a is the Comfort value for the Top-CW combination. The trends of Comfort and Completion Time seemded to be highly dependent on Edge, but did not seem to be dependent on Direction as much. The left edge appeared to give bad results for any directions. We conducted 4(Edge) 4(Direction) RM-ANOVA tests on Completion Time with Greenhouse-Geisser adjustment whenever the sphericity was violated. In the tests on Comfort, we used the aligned rank transform before performing the RM-ANOVA. The results are shown in Table 5. Significant effects are indicated by shaded p-values. In all cases, both Edge and Direction had a significant main effect on all of the two metrics. We also conducted post hoc paired t-tests with Bonferroni corrections to compare Comfort and Completion Time between the Edge cases and between the Direction cases. Among the Edge cases, there was a significant difference between Left and each of the other cases (p < 0.010) in terms of Comfort and Completion Time. Among the Direction cases, there was a significant difference between the CCW-Near pair in terms of Comfort (p < 0.010), and between the CW-Far pair in terms of Completion Time (p < 0.010) Discussion Shift-Drag in the Near and Far directions is similar to the two-finger pinch gesture, and, therefore, the corresponding experiment results may be compared with those of Hoggan et al. (2013a), who analyzed the ergonomic problems of the pinch gesture. They reported that the pinch gesture takes more time when the two fingers move horizontally, which may correspond to the Left and Right edge cases in our case. The Completion Time of the Left- Near condition was relatively longer and seemed to support the correspondence, but the Completion Time of the Right-Near condition did not. Hoggan et al. (2013a) also reported that the pinch-in gesture is usually faster than the pinch-out gesture except for certain hand postures. This is in contrast to our results, wherein, in all edge cases, the Far direction exhibited a faster operation. These differences may be attributed to the constraint by the edge on the finger orientation in our case, as well as on its position. Shift-Drag in the CW and CCW directions is similar to the two-finger rotation gesture, and, therefore, the corresponding experiment results may be compared with those of Hoggan et al. (2013b), who analyzed the ergonomic problems of the rotation gesture. They reported that the rotation gesture in the CCW direction was faster than that in the CW direction when the two fingers started at 60 in the horizontal direction. However, when the two fingers started at 120 in the horizontal direction, the rotation gesture in the CCW direction was slower than that in the CW direction. In our case, an RM-ANOVA indicated that Direction had a significant main effect on Comfort and Completion Time, but the differences between the Direction cases seemed to be less outstanding than those found in Hoggan et al. (2013b). The differences may be due to the small rotation angle in our case compared with the usual two-finger rotation gestures. The effects of Edge on Comfort and Completion Time seemed to be clear in Figure 12 a and b. In particular, Left exhibited the worst scores among the four edges Experiment 5: Pivot-Slide Pivot-Slide is similar to Shift-Drag except that the finger on the edge is the one moving instead of the finger on the screen. Therefore, we used the results from Experiment 3 to predetermine the finger for sliding on an edge; the index finger is for the right edge and the top edge, whereas the thumb is for the left edge and the bottom edge. The main question in this experiment was which of the (Edge, Quadrant, and Direction) combinations was good for Pivot-Slide. In this experiment again, we did not have any hypothesis in advance, but aimed to produce reference data showing the relative usability scores of different Pivot-Slide design possibilities Task As shown in Figure 13a, a gray bar with a red head appears on one of the four sides of the screen and a gray

13 Figure 13: Pivot-Slide task: (a) an example, (b) the four quadrants and the four edges, and (c) the two directions on the four edges. mark indicating a target quadrant appears. As the user slides his/her finger along the edge from the red head while maintaining a fingertip in the indicated quadrant, the color of the quadrant changes every 40 pixel (4.24 mm) movement. The color order is red, orange, yellow, green, blue, and purple. The goal of the task is to change the color of the quadrant to match the color of the other quadrants. The watch provides brief haptic feedback in response to the touch and release actions. After each trial, the screen stays blue for 500 ms, enforcing the user to wait. There are three task parameters, as shown in Figure 13b and c: the quadrant to hold (Quadrant), the edge to slide on (Edge), and the slide direction (Direction). The user is instructed to complete the task without clutching if possible Design and Procedure The experiment had a within-subject factorial design with three independent variables: Edge (Right, Top, Left, or Bottom), Quadrant (Q1, Q2, Q3, or Q4), and Direction (counterclockwise (CCW) or clockwise (CW). See Figure 13c). The participants performed 1 practice block, 20 main blocks, and 1 evaluation block. In each block, the participants performed one Pivot-Slide trial for each of the 32 combinations (4(Edges) 4(Quadrant) 2(Direction)). The order of the combinations was shuffled randomly. In total, each participant performed 640 (20 blocks 32 combinations 1 trial) trials during the main blocks. In the evaluation block, the participants rated the comfort (Comfort) of each of the 32 combinations using a 7-point Likert scale (1 = very uncomfortable and 7 = very comfortable). Using the data from the main blocks, we computed the mean completion time of the task (Completion Time) Results Figure 14 shows the Comfort and Completion Time data for the 4(Edge) 4(Quadrant) 2(Direction) factorial conditions. Each of the cross-shaped tables in the figure summarizes the data in the same way as in Figure 9, and the tables on the left and on the right are for the CCW and CW cases, respectively. The trends of Comfort and Completion Time seemed to be highly dependent on Edge, but did not seem to be dependent on Quadrant as much. We conducted 4(Edge) 4(Quadrant) 2(Direction) RM-ANOVA tests on Comfort and Completion Time. 12 Figure 14: Results of Experiment 5: (a) Comfort (CF) and (b) Completion Time (CT) data for the 4(Edge) 4(Quadrant) 2(Direction) factorial conditions. Table 6: Results of the 4(Edge) 4(Quadrant) 2(Direction) RM- ANOVA on Comfort (CF) and Completion Time (CT) in Experiment 5. Shaded p-values represent significant effects. DV Effect df df2 F-value p-value CF Edge(E) <.001 Quadrant(Q) <.050 Direction(D) <.050 E Q <.001 E D <.001 Q D E Q D CT Edge <.001 Quadrant Direction <.010 E Q <.005 E D <.001 Q D E Q D We used a Greenhouse-Geisser adjustment whenever the sphericity was violated. In the case of Comfort, we used the aligned rank transform before performing the RM- ANOVA. The results are shown in Table 6. Significant effects are indicated by shaded p-values. In all cases, both Direction and Edge had a significant main effect on Comfort and Completion Time Discussion We mentioned earlier the following three ergonomic problems: wrist twisting, finger crossing, and finger crowding. The results of Experiment 5 shown in Figure 14 also reflect these problems. An outstanding trend was that the Top and Bottom cases were better than the other Edge cases in terms of Comfort and Completion Time. This may be explained by the wrist twisting problem; the hand in the Top and Bottom cases suffered less from the wrist