Michael S. Geary, Assistant Professor, Pensacola Christian College Mary Lind, Professor NCA&T State University

Transcription

1 DECISION SCIENCES INSTITUTE A Quasi-Experimental Study Comparing Michael S. Geary, Assistant Professor, Pensacola Christian College mgeary@faculty.pcci.edu Mary Lind, Professor NCA&T State University lindm@ncat.edu ABSTRACT The purpose of this research is the evaluation of the performance and usability for an alternative keyboard layout on a smart phone compared to the dominant QWERTY layout on a smart phone when presented to users gradually versus immediately. The QWERTY keyboard layout has existed since. Today s smart phones predominantly use soft keyboards with touch screens, which means many costs identified in past research related to keyboard hardware are eliminated. KEYWORDS: Alternative Keyboard Layout, QWERTY, Smart Phone, Soft Keyboard, TEMA, Total Error Rate, TER, Touch Screen, Usability INTRODUCTION The keyboard layout known as QWERTY has existed for almost 150 years, well before the Internet and World Wide Web, television, and automobile. And yet during these one and a half centuries, few changes have been applied to that original layout. The keyboard itself has been used on a variety of devices and platforms, including typewriters, computer keyboards, kiosk touch screens, gaming systems, tablets, and mobile phones. While technology has dramatically changed, including advances in voice recognition, QWERTY remains the standard where a full-sized keyboard is required. When mobile phones first arrived to the general consumer market at the end of the 20th century, the keyboard interface mimicked the 12-key, touch-tone phone layout as seen in Error! Reference source not found.. This layout made logical sense at that time. Especially on early mobile analog phones, the primary phone s usage was simply to make phone calls. And the 12- key layout matched closely the mental model held by people at that time, not to mention space for buttons was at a premium. When SMS (Short Message Service) messaging, commonly known as texting today, began growing in popularity, the 12-key layout proved to be inefficient and cumbersome for users. Therefore, a full-sized keyboard was needed, and the QWERTY keyboard was selected because of its familiarity to most users. When people use a hand-held mobile device to send text messages, however, they do not interact with the keyboard in the same way as they would on a desktop or laptop computer. Instead of using ten fingers, users normally type with only one or two thumbs. Some individuals may even use one index finger to type the individual characters (Oulasvirta et al., 2013). One solution to possibly overcome the time and frustration negatives could be to introduce the user to an alternative keyboard layout gradually over time. Assuming adaptation of the QWERTY keyboard layout still has potential (Findlater & Wobbrock, 2012), this study examines the viability of this gradual keyboard solution.

2 PROBLEM OPPORTUNITY STATEMENT When mobile devices changed from being only a device used to make cellular phone calls to a smart phone device useful for a variety of input tasks, the QWERTY keyboard layout was naturally applied. People were already familiar with this keyboard layout and that mental model worked well to complete this transition. But as with any activity or technology, it becomes the responsibility of individuals to test whether those decisions result in optimal solutions. In addition, the majority of studies have been performed before the onslaught of texting within the past five to seven years. The interaction mechanism (i.e., a user s fingers) used to type are completely different than those used on a standard, traditional keyboard. The user still uses fingers to type, but the dominant digits used are actually the thumbs for most users, which on QWERTY are normally reserved to only pressing the spacebar. And as to whether the QWERTY layout is better or worse than alternative keyboard layouts, some studies have shown that improvement is possible. Software and touch screens make altering a keyboard s layout minimal in respect to cost, which was a limiting factor with full-sized keyboard on typewriters and computers. Software can easily change what is displayed on a soft keyboard meaning that there are no hardware modifications necessary. Therefore, further study can be performed on whether QWERTY is the best choice for one or two finger typing on a smaller touch screen keyboard. Also the lack of easily available alternative keyboard layouts on smart phones may explain why people do not use those alternative layouts. If these layouts were available, individuals might be more inclined to use them especially since mobile devices (specifically phones) are more personal items that are typically used by only one individual. Therefore, individuals would not need to constantly switch between the QWERTY layout and an alternative keyboard layout when texting, which would have been a limiting factor with traditional keyboards. Therefore, the statement of the problem is that alternative soft keyboard layouts are not adopted because of the high switching costs (i.e., learning curve) for the user. Finding a way to lower this cost may more likely result in higher adoption of a different keyboard layout. Research Questions As mentioned previously, alternative keyboards have been rejected time and time again for typing with ten fingers. Various studies have attempted to explain reasons regarding this. But irrespective of those reasons, alternative keyboards have primarily remained an academic and theoretical discussion without much likelihood of an actual widespread implementation. Most alternative keyboard layouts are designed as a complete change from what the user would normally be accustomed to seeing (i.e., all keys are remapped). In addition, these new layouts rarely (if ever) include non-alphabetic keys. A slower or more gradual transition may be a solution to move from QWERTY to a more efficient layout. A radically different layout affects the mental processing required by the user. But if only one or two keys were changed at a time, the increase to the mental processing would mostly be insignificant, since the new keyboard layout would mostly match the same keyboard layout pattern already committed to memory by the user. Therefore, my research question involves two components. First, if the gradually changing keyboard concept is compared to the QWERTY layout and the final alternative keyboard layout, would the gradually changing keyboard allow the user to better maintain similar speed and accuracy to that of the QWERTY layout as compared to the final alternative keyboard layout? Second, will the user s perception of usability be significantly more favorable toward the gradually changing keyboard as compared to the final alternative layout indicating a proclivity toward adoption of the gradually changing keyboard?

3 HYPOTHESES It seems logical that typing speed and error rates will more closely resemble those for QWERTY when users are exposed to the gradually changing keyboard layout. It is expected that more noticeably typing speed will drop and error rates will rise for the completely new keyboard layout. The researcher expects to find a higher acceptance of the gradually changing keyboard as compared to the immediate new keyboard layout. Therefore, this research consists of the following hypotheses: - There is no significant difference in perceived usability by individuals exposed gradually to a new soft keyboard layout on a smart phone and individuals exposed immediately to a new soft keyboard layout on a smart phone. - Individuals exposed gradually to a new soft keyboard layout on a smart phone will perceive that keyboard to be more usable than those individuals exposed immediately to a new soft keyboard layout on a smart phone while learning the alternative keyboard layout. - There is no significant difference in typing speed between individuals exposed gradually to a new soft keyboard layout on a smart phone and individuals exposed immediately to a new soft keyboard layout on a smart phone. - Individuals exposed gradually to a new soft keyboard layout on a smart phone will maintain superior typing speed to those individuals exposed immediately to a new soft keyboard layout on a smart phone while learning the alternative keyboard layout. - There is no significant difference in total error rates between individuals exposed gradually to a new soft keyboard layout on a smart phone and individuals exposed immediately to a new soft keyboard layout on a smart phone. - Individuals exposed immediately to a new soft keyboard layout on a smart phone will maintain superior accuracy to those individuals exposed gradually to a new soft keyboard layout on a smart phone while learning the alternative keyboard layout. THEORETICAL PERSPECTIVES/CONCEPTUAL FRAMEWORK The statistics in Table 1 were obtained from two sources. The left-most column is based on MacKenzie and Soukoreff s (2003) article involving 500 phrases used to evaluate text entry. These phrases consisted only of alphabetic characters and the space. Capital and lower case letters were treated as identical. The actual phrase list was obtained directly from which is the web address provided in the original article. The second set was retrieved from NetLingo.com (n.d.), which specifically focused on acronyms, abbreviations, and other shortcuts used when texting. The list from that website was parsed such that only the actual acronyms, abbreviations, and shortcuts were used and not the descriptions that were also included on the website. This list consisted of 2,482 acronyms, abbreviations, and shortcuts and included numbers as well as special characters.

4 Table 1. Statistics of Frequently Used Symbols. MacKenzie and Soukoreff Phrases (space) e t o a i s r n h l d u c y m g p f w b v k j - 35 x - 34 q - 26 z - 13 NetLingo Acronyms, Abbreviations, and Texting Shortcuts t / - 39 a i & - 20 o s m b f l w ^ - 9 d y - 371? - 7 n - 367! - 5 c < - 5 r \ - 5 g p : - 4 u % - 3 h * - 3 e - 211, - 3 k j - 87 # $ - 2 (space) - 66 = v - 53 ~ - 2 z - 43 > ; - 1 x - 41 ' - 1 q - 39 The statistics in Table 1 were generated utilizing a custom-written Java application created specifically for the purpose of compiling these statistics using the data obtained from the sources given previously (MacKenzie & Soukoreff, 2003; Netlingo.com, n.d.). Characters (symbols) on a traditional keyboard that did not appear within the NetLingo.com data were not shown in the above table. It is interesting to compare the two lists, particularly the first ten characters that are most frequently used (see Figure 1 and Figure 2). The letters t, a, o, i, s, and l (see Figure 3) appear in both top-ten lists with the space character being ignored. The most noticeable difference is that the letter e barely made the top twenty from the NetLingo.com list. Another note is that the first non-alphabetic character in the NetLingo.com list is the digit 2, used as a shortcut for the words to and too ; however, it does not occur as frequently as might have been expected. Statistics of character frequency from an older study (Chubon & Hester, 1988) reported nearly identical frequencies to those given by MacKenzie and Soukoreff with only minor shifting of some characters.

5 Figure 1. Top Ten Characters from MacKenzie & Soukoreff Phrases. Figure 2. Top Ten Characters from NetLingo.com List. Figure 3. Common Characters Between Phrases and List. With text message language and grammar still evolving, multiple approaches could be taken here. But because the language and grammar of text messaging are evolving, the best direction would be to utilize past research. The phrase list produced by MacKenzie and Soukoreff (2003) has been cited by more than 350 works as of February 2015 according to Google Scholar. Additionally, a software application called TEMA (Text Entry speed and accuracy Metrics on Android devices), designed for collecting statistics via keyboard on touch screen interfaces (Castellucci & MacKenzie, 2013), will be used to administer the testing as well as record metrics to be used later for statistical analysis. The version of the software used in this study implements the same phrase list by MacKenzie and Soukoreff in testing text entry on a smart phone. The actual phrases and words are not as critical as is the letter swapping and the test subjects impression and reception of those shifting letters. Assumptions/Biases One assumption was the expected open-mindedness that users would have in respect to an alternative keyboard layout, even one that changes gradually. The QWERTY layout is so

6 ingrained in users mental models today that there would be a natural resistance to change, especially something that is as common to users as a smart phone s keyboard. While people believe they are willing to change, disruptions to normal routines and expectations are more often viewed negatively. Another assumption involved familiarity with a QWERTY keyboard layout. Regardless of whether test subjects had ever used a smart phone, users were familiar with the basic QWERTY layout today. This assumption was important since speeds and error rates would be compared between the QWERTY and alternative keyboards. A final assumption entailed the selection of test subjects. While this study did not select test subjects based on age, most test subjects selected were ages 18 to 23 based on the sampling process (i.e., convenience sample) and available population (i.e., college students). The selected test subjects still accurately represented users likely to use a soft keyboard on a smart phone. Significance of the Study Many people, both academic and non-academic, practically assume that QWERTY is the only keyboard layout that exists and will be used until voice recognition improves well enough to replace hand typing all together. However, studies have potentially missed a key gap in attempting to redesign the keyboard. This study has the potential to support that past researchers keyboard designs, such as Dvorak, OPTI, and Fitaly, were correct as more efficient layouts. The missing element to actually move from concept to implementation may have been gradual adaptation of the keyboard layout by making slow changes instead of the immediate, complete introduction of a new keyboard layout. Research Design The perspective for this research study is generally that of quantitative taking a quasiexperimental research approach to test gradual modification of the QWERTY keyboard layout to produce an alternative keyboard layout on a smart phone device. More specifically, this experiment compares results between two groups of test subjects, where one group was exposed to the alternative keyboard layout gradually and the other group was exposed to the alternative keyboard layout immediately. There were five primary steps to this experiment as seen in Figure 4. Step one asked all test subjects to complete a Keyboard Usability Survey (see Appendix C) asking objective, nonidentifiable questions as well as questions regarding keyboard usability for the final alternative keyboard layout. Step two had all test subjects establish a baseline typing speed and error rate on a QWERTY layout using one finger, one thumb, or two thumbs. The observer noted which fingers and/or thumbs the test subject used at the top of the survey (see top-right of Appendix C).

7 Figure 4. Visual Steps for the Keyboard Experiment. For step three, test subjects were divided into two groups. The first group was considered the baseline group. For this group, test subjects were given the final alternative keyboard layout (see Figure 5) for each testing session. There was no change to the keyboard layout between sessions for this test group. Figure 5. Final Alternative Keyboard.

8 The second group was the experimental group. This group of test subjects had the keyboard layout change slightly after each testing session by swapping two characters. The characters to be swapped were determined by the frequency of usage (based on Table 1 on page - 4 -) and current position within a QWERTY layout. Therefore, more frequently used symbols were swapped with less frequently used symbols that are closest to the center of the keyboard. The reasoning behind the more frequently used characters should be obvious, which is to increase efficiency by reducing movement between keys. The rationale behind moving those characters closest to the center of the keyboard is because whether a user makes use of one or two thumbs to type, the size of today s smart phones makes reaching the middle of the keyboard easy and reduces back-and-forth movements especially when using only one thumb. Table 2 sets out how the swapping was derived. Note that the table does not show formula results less than or equal to zero. Each character was given a ranking based on the frequency of usage as seen previously in Table 1 on page

9 Table 2. Mapping Letters to Swap Based on Frequencies.

10 The percentage of use for each character in both the MacKenzie and Soukoreff list and the NetLingo.com list were calculated separately, and the percentage for each character from both lists was averaged together. This was then sorted into descending order by that average percentage to obtain the ranking value. The position value was the number of keys on the keyboard from the center point as noted in Figure 6, with this center point residing between the g and h characters. The four keys touching that center point ( b, g, h, and y ) were given the position value of 5. Letters touching any of those keys to the left ( t, f, and v ) were given a position value of one less or 4, and the letters to the right ( u, j, and n ) touching any of those four keys in the middle were given a position value of one more or 6. This was continued for the remaining letters on the keyboard. Figure 6. Center Point on a QWERTY Layout. A value was calculated for each character mapped to every other character on the keyboard using the following formula: where rank i is the ranking value for the character (that is the character desiring to be moved toward the center) in the left-most column in Table 2, and rank j is the ranking value for the character in the heading row (that is the character that the other might benefit being swapped with). The value of 5 is the position number for the keys closest to the center point, and pos j (refers to Position in column 3 of Table 2) is the number of keys from the center point for the character whose position is being considered. Making this a little simpler, if a character with a higher frequency of use is further away from the center point than a character with a lower frequency of use, then the result of the calculation will be a larger result. Beginning with the first character in Table 2 of t, the largest value for that row matches the character in the heading of g that should be swapped with t. The shaded rows are the six characters that would be swapped based on frequency of usage and commonality between the lists in Table 1 on page The bolded values with a border are in the column for the heading character that should be swapped. Appendices A and B show these steps with the swapped keys highlighted in the diagrams after each step, although this highlighting was not shown to test subjects during the testing sessions. The physical keys themselves (i.e., the squares ) were not moved or shifted, but instead the characters themselves were swapped with other characters. The testing sessions for both groups are represented by step four. After the final testing session for each test subject, step five involved administering a second survey (see Appendix D), which was identical to the first survey in way of the ten usability questions (labeled as Keyboard-specific questions). The two groups were statistically analyzed using the usability (MacKenzie & Soukoreff, 2002; Nakagawa & Uwano, 2011; O'Brien, et al., 2008; Sirisena, 2002; Zhai, et al., 2002) data both from the surveys as well as the typing (1)

11 speed and total error rate results (MacKenzie & Soukoreff, 2002; MacKenzie, et al., 1999; R. William Soukoreff & Mackenzie, 1995, 2004; Zhai, et al., 2000, 2002) from the testing sessions. Corrected and uncorrected error rates are also presented, since the combination of these two make up the total error rate, but they were not used in the analysis portion of this study. Population and Sample While this study often deals with elements related to text messaging, this study has not limited its application to only text messaging. The study s primary goal involved increasing data entry efficiency using a touch screen keyboard on a smart phone device of which text messaging is a major part today. Potential test subjects did not need to be experts or even moderately proficient in the use of touch screen keyboards or smart phone devices. In fact, beginners who cautiously peck out each letter slowly and deliberately were also legitimate, potential test subjects. While one limiting factor could have been test subjects needing a knowledge of the QWERTY keyboard layout, prior studies have observed and noted that finding potential test subjects without knowledge of the QWERTY layout has become much more difficult in today s information society. As with any study, a sample was taken of the population who became the test subjects used in this study. Even limiting the population to individuals within the United States, if only 25% of that population uses a small keyboard for data entry on mobile devices, a conservative number of potential test subjects would be 75 million individuals. This would be practically impossible to test every individual; therefore, a representative sample would need to be used. Demographics played a part in selection of test subjects, with age being one limiting factor in the selection. Other characteristics, such as gender, ethnicity, religion, citizenship, and income level, had no bearing on the choice. Test subjects needed to be 18 years of age or older, since individuals under 18 are considered a protected group. Dominant hand as well as age and gender were used when reporting the findings, although dominant hand and gender were not used during the selection process. The sampling procedures contain characteristics of a few sampling types. Foremost, convenience sampling was initially used to obtain a pool of potential test subjects. Pensacola Christian College is a medium-sized private institution of higher learning located in Pensacola, Florida. The campus of Pensacola Christian College is home to approximately 4,500 undergraduate and graduate students, the majority of which reside in on-campus housing. The current diversity at the college includes students from all fifty states in the United States as well as more than sixty foreign countries (Pensacola Christian College). In addition to the student population, faculty and staff number nearly 1,000 total individuals. This combination of students, faculty, and staff provided a rich subset from which to acquire test subjects for this study. Members of this group were next given the opportunity to volunteer to be considered for this study. Simple random sampling was used to obtain the actual test subjects for this study from the collection of responding volunteers. Further details are outlined in the following section. Sampling Procedure Experimental studies related to keyboards have ranged from single-digit sized groups (Oulasvirta, et al., 2013), to groups of twelve test subjects (Findlater & Wobbrock, 2012; Himberg, Häkkilä, Kangas, & Mäntyjärvi, 2003), to groups of more than thirty test subjects (Azenkot & Zhai, 2012; Tossell, et al., 2012). A study by Perry (2008) related to using only one thumb for text entry involved forty test subjects, which were divided into two groups with each assigned to complete opposing tasks; for example, one group was asked to text while sitting, and the other group was asked to text while walking. Because this study s experimental design follows closely that of Perry, forty test subjects, divided into two groups of twenty test subjects each, were used for this study.

12 With the Business Department being one of the largest departments on the campus of Pensacola Christian College, this group reflected an accurate representation of the student body as a whole, which in turn represents the average users of smart phone devices. Two classes being offered during the fall 2014 semester from each subject area within the Business Department was randomly selected excluding classes that were being taught by the researcher. Subject areas included Accounting, Criminal Justice, Computer Science, Office Administration, Management, Marketing, and Sport Management. The researcher visited individually with each instructor from the randomly selected classes to provide an overview of the study and the procedure for announcing this study within their classes. Appendix F contains the script that instructors were asked to read to their respective classes. A sufficient number of handouts for interested individuals were provided for the maximum number of potential test subjects within each class. This handout included the study s general description, timeframe, compensation, duration, and website URL for test subject registration (see Appendix G). Individuals willing to participate were asked to take a handout and register on the website, with a screen shot seen in Appendix H. This website was made available to potential test subjects via the on-campus intranet. Each registered individual s information was stored within a SQL Server database. After excluding any potential test subject enrolled in one of the researcher s classes being taught during that semester, forty test subjects were randomly selected from the pool of potential test subjects to participate in this study. An enumerated, detailed list of sampling steps used for this process is provided in Appendix I. Potential test subjects not selected for this study were stored until the end of the entire experiment. If a selected test subject became unwilling or unable to participate after the selection process, an additional test subject was drawn from the pool of volunteers to ensure a total of forty test subjects are used in this study. This actually occurred two times. The first test subject, who actually was the first scheduled test subject, missed the initial scheduled time on October 2, We rescheduled for October 30, 2014, but the test subject missed that scheduled time as well. At that point, the next potential test subject was contacted via and scheduled later that same day. The second test subject unable to participate discovered a scheduling conflict. After initially establishing a testing time for October 6, 2014, this test subject contacted the researcher via requesting to be removed from the study. The next potential test subject from the stored list of respondents was contacted via and scheduled. After testing was completed, data on the potential test subjects has been maintained; however, at the conclusion of this dissertation process, all digital and physical data about the specific test subjects, whether they were used for this study or not, will be destroyed. Instrumentation Two primary instruments were used for data collection throughout this study. A brief usability survey, seen in Appendix C and based on the original System Usability Scale in Appendix E (Brooke, 1996), was administered prior to testing and immediately after testing (see Appendix D) to gauge user perception of the keyboard changes and layout. This instrument s purpose is to revisit the arguments made by David (1985) in respect to QWERTY s lock-in and other researchers findings (Hall, et al., 2011; Hossain & Morgan, 2009; Liebowitz & Margolis, 1990) regarding QWERTY s survival. The second instrument was the smart phone application used to administer the actual tests. I. Scott MacKenzie has been involved with keyboard research since the 1990s. Therefore, it made logical sense to utilize a tool which he helped create to collect metrics when entering text via a soft keyboard on a touch screen. The resulting application runs on an Android-based device and is called TEMA ( The TEMA version used

13 within this study was version 1.3. The application works for thumb typing (one or two thumbs) on a QWERTY keyboard layout. It also functions for handwriting and swiping of letters, but those functions are not within the scope of this particular study (Castellucci & MacKenzie, 2013). The TEMA study also made use of the phrase set list created by MacKenzie and Soukoreff (2003), which was the same phrase set used for this particular study. One important modification that needed to be made involved the keyboard on the smart phone itself. This change did not directly involve the TEMA software, since it uses whichever keyboard layout the Android device is set to use. In order to have had the keyboard layout change throughout the testing sessions, each step of the keyboard layout changing (see Appendices A and B) was created as a separate data file. The Android operating system allows alternative keyboard layouts to be created using an XML formatted file, with an example seen in Appendix J. Once these XML files were created, they were installed on the smart phone device. The researcher was responsible to manually change the keyboard layout between each testing session for those test subjects exposed to the gradually changing keyboard. For security reasons, the Android operating system does not allow direct manipulation or changing of the keyboard layout from within software applications. Also, the authors of the TEMA software would not release the source code to permit modifications to the TEMA software itself. Validity Two instruments were utilized during the experiment. The first was a survey (see Appendix C) based on the System Usability Scale as shown in Appendix E, which was used prior to the experiment as well as after the experiment to measure any change in perceived usability by the test subject regarding the alternative keyboard layout. This particular scale was developed by John Brooke (1996) and uses a Likert scale, 10-question set of questions. These questions are designed to elicit usability information regarding practically any type of system, whether that involves software or hardware, or whether that deals with technological or non-technological systems. This System Usability Scale has been applied many times since its introduction to test usability and has become the de facto standard when evaluating system usability. The System Usability Scale has consistently demonstrated the capability to differentiate between systems that are usable and other systems which are less usable (J. R. Lewis & Sauro, 2009; Sauro, 2011). The second instrument used was the TEMA application as presented by Castellucci and MacKenzie (2013). Their motivation behind TEMA was based on a similar application called TextTest, which accomplished the same purpose on a personal computer (Wobbrock & Myers, 2006). After initially developing their software, they performed an initial pilot study with six test subjects (Castellucci & MacKenzie, 2011) before completing a more complete study with sixteen test subjects testing the application s design and operation. The more complete study compared the entry speed (i.e., words per minute) and accuracy (i.e., error rate), both key metrics, for different input methods (MacKenzie & Soukoreff, 2002; MacKenzie, et al., 1999; R. William Soukoreff & Mackenzie, 1995, 2004; Zhai, et al., 2000, 2002). This software continues to be maintained, updated, and tested by its authors, with the latest version available being version 2.1 as of February Reliability The meaning behind reliability for a testing instrument is the key point of stability or consistency. Consistency within a testing instrument ensures that one researcher s findings can be reproduced by another researcher using that same testing instrument. The System Usability Scale s reliability has been demonstrated by its repeated usage since its creation. Normally, a questionnaire needs to achieve a reliability rating, or coefficient alpha, of 0.70 to be considered reliable (Helander, Landauer, & Prabhu, 1997). The SUS s reliability has been evaluated by various studies. Kirakowski (1994) cited an unpublished

14 dissertation that calculated the SUS s coefficient alpha at 0.85 (Lucey, 1991), and in a more robust study of more than 200 studies the Cronbach s alpha was computed at 0.91 (Bangor, Kortum, & Miller, 2008). Both of these studies easily exceed the minimum 0.70 reliability score supporting the use of the System Usability Scale questionnaire to evaluate usability in this study. The TEMA application calculates common metrics used in the comparison and evaluation of text-based entry methods. The first metric is entry speed, with Castellucci and MacKenzie s calculation for entry speed based on Yamada s (1980) calculation for words per minute (see Formula 2 on page under the Data Analysis section). The second metric includes the error rate, which is presented in three different forms. Uncorrected error rate (UER) includes mistakes that the typist did not correct, corrected error rate (CER) involves mistakes the typist did correct (e.g., backspacing and then retyping a character), and total error rate (TER) is a summation of the uncorrected and corrected error rates (Castellucci & MacKenzie, 2013; R William Soukoreff & MacKenzie, 2003). Formulas 3 through 6 beginning on page under the Data Analysis section show the calculation for each of these error rates. The TEMA software even incorporates an interruption timer that will pause the time if the test subject switches from the application (e.g., text message or phone call) to ensure an accurate calculation for both words per minute and error rates. Because the smart phone was provided to the test subjects, this feature did not need to be utilized during any of the testing sessions. Data Collection The researcher met with each instructor for the classes chosen to participate in the study. Those instructors were given a pre-written script to read (see Appendix F) describing the details of the study to their respective classes. Potential test subjects were initially asked by the instructors whether or not they wished to be considered to participate in this study. If an individual was willing to participate, then that individual was asked to take a handout paper with further instructions including the website URL to visit for registering for this study (see Appendix G). Included on the handout given to potential test subjects was an end date by which interested individuals needed to register for the study. The date given on that handout was September 19, After that end date, test subjects were selected and sent an informing them of their selection to participate, details about the testing location, and a request to select preferred times to complete the study (see Appendix K). Those individuals not selected to participate in this study were also ed thanking them for their interest (see Appendix L). After invited test subject responses were received, a Google Calendar invitation was scheduled and sent to each test subject. When the test subject arrived at the testing session, a manila folder was selected based on the order in which they arrived in relation to other test subjects. In other words, the first test subject tested received the manila folder labeled 1, the second test subject tested received the manila folder labeled 2, and so on. Folders were labeled from 1 to 40. An additional five folders were labeled 41 to 45 accounting for any test subjects who might choose to withdraw during the experiment without completing the entire experiment; however, this did not occur for any test subjects. After completion of the Informed Consent Form and the presentation of details regarding the study (see Appendix M) by the researcher, each test subject was asked to view a picture of the final alternative keyboard layout prior to completing the printed, one-page Keyboard Usability Survey (see Appendix C). Once the initial survey was completed, the researcher examined the initial survey to ensure that all items were completed by the test subject. Then the researcher returned the survey paper to the specific numbered manila folder selected for that test subject earlier. At this point, the test sessions using the soft keyboard on the smart phone occurred. Initially, all test subjects were asked to establish a baseline with the standard QWERTY layout via one testing session that involved the test subject typing ten random phrases. As testing

15 continued, test subjects who selected a manila folder with an odd number were given the gradually changing keyboard that swapped characters after each testing session, while test subjects who selected a manila folder with an even number were given the final alternative keyboard for each testing session. Test subjects used an HTC Thunderbolt smart phone running Android OS version with a resolution of 400x800 pixels. The physical phone size was 4.75 high x 2.44 wide x 0.56 deep (120.7mm x 62.0mm x 14.2mm) with a weight of 6.2 ounces (176.6g). The actual diagonal screen size was 4.3 (109.2 mm) with the keyboard dimensions being approximately 1.4 high x 2.0 wide (35.6mm x 50.8mm). Figure 7. HTC Thunderbolt. Test subjects were seated during each testing session in a typical classroom chair for each testing session, with each test subject being given seven testing sessions. Test subjects were asked to keep the smart phone in upright or portrait mode instead of using landscape mode. Test subjects were informed that they could use whichever finger or thumb combination they felt most comfortable with using. After each testing session, test subjects were invited to stand up, get a drink of water, use the restroom, and so on, although no test subject accepted. They were not permitted to use the smart phone during the break. A script used to introduce test subjects to the study when they arrived along with details they needed to know during the study can be seen in Appendix M. The TEMA software maintained a log of the results, including typing speed and error rates, from each testing session. The data was stored into a comma-separated file, with the file name including both the test subject number as well as the test session number. For example, a test subject number of 99 completing test session number 7 would have a file named 99_7_A_stats.csv. After the final testing session by one or more test subjects, the researcher removed the micro SD card from the smart phone and inserted the micro SD card into a computer with a card reader slot. The data was then accessed and downloaded from the micro SD card to the desktop computer. Since each test subject completed seven testing sessions, seven separate files were generated for each test subject. This data was consolidated into one

16 spreadsheet document in Microsoft Excel before finally being imported into SPSS to enable the data analysis phase. Once the test subject completed the final test session, the test subject was asked to complete the same Keyboard Usability Survey (see Appendix D) completed prior to the testing sessions, except for the objective demographic questions. The test subject was shown the same picture of the final alternative keyboard layout viewed prior to beginning the testing sessions, but the test subject was not permitted to view the previously completed survey form. After the test subject had completed the second survey form, the researcher examined this survey to ensure that all items were completed. The researcher then returned the survey paper to the specific numbered manila folder selected for that test subject earlier and in which the initial survey form had previously been placed. No uniquely identifiable information was stored in connection to the manila folder only the integer number labeling the folder was used to match the survey responses with the data stored from the testing sessions. Finally, the test subject was given a choice of a $10 gift card to a local business: The Drowsy Poet, a local coffee shop; Dunkin Donuts ; or Chick-fil-A. The test subject originally made a choice when registering to participate, although the test subject was able to change his or her mind at that time. In addition, test subjects were thanked for their participation in this study. Data Analysis The first quantitative measure recorded during testing involved typing or tapping speed, which is commonly measured in words per minute (MacKenzie & Tanaka-Ishii, 2010; R William Soukoreff & MacKenzie, 2003): (2) where T is the length of the typed phrase (i.e., number of characters) including spaces and S is the number of seconds needed to type the phrase. The second quantitative measure collected during testing was that of error rate. Soukoreff and MacKenzie (2003) described three error rate measurements, which included total error rate (TER), corrected error rate (CER), and uncorrected error rate (UER). Total error rate is calculated as: (3) where INF (incorrect not fixed) represents uncorrected errors, IF (incorrect fixed) represents corrected errors, and C (characters) represents the total number of correct characters actually typed. The above formula for TER combines both corrected and uncorrected errors. The corrected error rate is calculated as follows: (4) and the uncorrected error rate is calculated as: where again INF represents uncorrected errors, IF represents corrected errors, and C represents the total number of correct characters actually typed. Given these two formulas, the earlier formula for TER could be simplified to be the following calculation: (6) Based off of the example described by Soukoreff and MacKenzie (2003), assume a typist intends to enter the following phrase: lock the house (5)

17 yet the actual keys tapped by the typist, where represents a backspace key press, are: lok ck te hoise resulting in the final text being: lock te hoise The typist had one corrected error and two uncorrected errors. The corrected error was the backspace used to remove the first k that was entered too early in the first word; therefore, IF = 1. The uncorrected errors are the missing h and the incorrect i instead of u ; therefore, INF = 2. The total correct characters, including spaces, entered by the typist equals 12. Using the above values with the formula for CER: yields a corrected error rate of 6.7%, which would be stored as Using the above values with the formula for UER: results in an uncorrected error rate of 13.3%, which would be stored as Finally, using the formula for TER (3) above: or the shorter TER formula (6) above: yields a total error rate of 20% for this example, which would be stored as The usability score from the SUS ranges from 0 to 100, representing a combined usability score. This combined usability score was calculated from the ten items within the survey by summing each item s calculated value (ranging from 0 to 4) and multiplying that total by 2.5. The scale for each item is labeled as 1 through 5; however, the odd-numbered items scores are calculated differently from the even-numbered items. For the odd-numbered items, each item s value is calculated as s - 1, where s refers to the scale position (i.e., the label of 1 through 5). For the even-numbered items, each item s value is calculated as 5 - s. The overall usability score is then: (7) Several statistical comparisons were performed with the collected data from the tests as well as the surveys, which included a simple comparison of means as well as an independent samples test including both Levene s test of F value and t-test. As described previously, the typing speed and error rate were obtained directly from the testing sessions, which were stored automatically by the TEMA software. Other data collected was gender, dominant hand usage, age, and soft keyboard use frequency, with these being obtained from the initial survey prior to initiation of any testing. Primary digit usage (i.e., one thumb, two thumbs, or one finger) was recorded during the testing session by the researcher, and usability was measured before and after testing via the surveys. Data collected from surveys and testing sessions was imported into IBM s software application SPSS (Statistical Package for the Social Sciences). The usability scores (prior to testing and after testing) were compared between the groups. First, a basic comparison of the mean usability value was performed to determine what perceived usability occurred within each group and any change between the before and after measurements for both groups. Next, an independent samples test was used to determine if there was a significant difference in test subjects perceived usability of the new layout and whether the gradual presentation of the alternative keyboard layout was seen as more usable than the immediate presentation of the alternative keyboard layout. Whether test subjects were regular soft keyboard users or not, it was expected (although not required) that each test subject would likely have a clear mental model of the QWERTY layout. One study, while not directly

18 related to keyboards, found that personalization of devices (specifically apps on Apple iphones) improved users perception of their device s usability (Tossell, et al., 2012). This study also utilized the SUS, which is the same measuring device used by this study. Typing speeds were analyzed by comparing the baseline speed on QWERTY for all test subjects, the typing speed from those on the gradually changing keyboard, and the typing speed from those on the final alternative keyboard layout. The means for those typing speeds were compared to see what effect each method of keyboard introduction had on typing speeds. In addition, an independent samples test was performed to determine if a significant difference exists between the gradual and immediate keyboard presentations. Error rate, specifically TER, was also analyzed in the same way, comparing the results for the baseline from QWERTY, the gradually changing keyboard, and the final alternative keyboard. It was expected that typing speed and error rates would change little from that of QWERTY for those test subjects exposed to the gradually changing keyboard. However, at the beginning for those test subjects exposed immediately to the final keyboard layout, it was expected that the typing speed would be dramatically lower and the error rate noticeably higher. As these test subjects became accustomed to the new layout, speed would naturally increase and error rate should also decrease (Isokoski, 2004). Basic categorical analyses were performed utilizing the objective data collected from the initial survey. This objective data included gender, dominant hand usage, age, soft keyboard use frequency, and primary digits used during testing. After the experiments for this study were performed with test subjects, the data was imported into the SPSS application to enable the analysis that follows within this chapter. Initially from the data, overall demographics were compiled and are presented, some of which includes gender, dominant hand usage, age, and fingers used during the test sessions. Following those demographics, the data collected during the experiments are also presented including usability, speed, and accuracy. This data is analyzed by examining means and data normality along with F values and t-tests. Finally a presentation and discussion of the findings from this study is presented. Participant Demographics As supported in the previous chapter, forty test subjects were used in this study. Following are several tables and figures showing various characteristics of the test subjects used in this study. While the number of test subjects within each group does not lead to conclusions based on these demographics, they are provided to demonstrate the data sampling. Table 3 shows a breakdown of male and female test subjects, with a visual comparison seen in Figure 8. While it was expected that the breakdown would be about 22 female (55%) and 18 male (45%), the random selection yielded approximately that expectation although gender would unlikely have any significant effect on the data collected during testing. Between the two test groups, the breakdown was nearly identical. Table 3. Gender Breakdown. Gender Total Gradual Immediate Count Percent Count Percent Count Percent Female % % % Male % % % Total % % %

19 Figure 8. Gender Breakdown Chart. Test subjects were right-handed a sizeable 77.5% of the time or 31 of the 40 test subjects as seen in Table 4 and Figure 9, which was not an unexpected statistic. Yet even though the test subjects were randomly selected, 7 out of the 9 left-handed test subjects were female. Table 4. Dominant Hand Usage. Dominant Total Gradual Immediate Hand Count Percent Count Percent Count Percent Gender Left % % % Male: 2 Female: 7 Right % % % Male: 13 Female: 18 Total % % % Figure 9. Dominant Hand Usage by Gender Chart. Seventy-five percent of the test subjects were in the range of 18 to21 years old, which was expected given that the convenience sample of college students was predominantly ages 18 to 23. For the most part, the breakdown was similar for both test groups. As specified by the

20 IRB, no test subjects under 18 years old were used in this study. The one test subject who was 29 years old is a graduate student enrolled in an undergraduate class because of prerequisites. The entire breakdown can be seen in Table 5 and Figure 10. Table 5. Age Breakdown. Total Gradual Immediate Age Count Percent Count Percent Count Percent % 1 5.0% % % % % % % % % % % % 1 5.0% % % % 0 0.0% % 1 5.0% 0 0.0% % 0 0.0% 1 5.0% Total % % % Figure 10. Age Breakdown Chart. As expected especially based on the ages of the test subjects, test subjects regularly used a keyboard on a smart phone. Of the 40 test subjects in this study, 34 of them (75%) stated that they used a smart phone keyboard often, which was defined as more than 10 times per day. Table 6. Smart Phone Keyboard Usage. Usage Total Gradual Immediate (daily) Count Percent Count Percent Count Percent None 2 5.0% 0 0.0% % 1-5 times % 1 5.0% % 6-10 times 0 0.0% 0 0.0% 0 0.0% Often % % % Total % % %

21 Figure 11. Smart Phone Keyboard Usage Chart. During the testing sessions, an overwhelming majority equaling 90% of the test subjects utilized two thumbs to complete the typing on the soft keyboard. Of the two test subjects who reported that they did not use a soft keyboard nor used a smart phone, one of those test subjects used one finger while the other used two thumbs during the test sessions. Table 7. Finger Use During Test Sessions. Finger Total Gradual Immediate Use Count Percent Count Percent Count Percent 1 Finger 3 7.5% 1 5.0% % 1 Thumb 1 2.5% 0 0.0% 1 5.0% 2 Thumbs % % % Total % % % Figure 12. Finger Use During Test Sessions Chart.

22 The final descriptive statistic was not originally planned to be asked of the test subjects. And while the type of smart phone device would be unlikely to have any significant effect on the findings, it seemed that it could be interesting nonetheless to ask test subjects what type of mobile phone they used. The expectation was that most test subjects would have a smart phone running Android or iphone operating systems, with this being confirmed by 82.5% of test subjects. It was slightly surprising to have 2 test subjects reporting that they did not use a mobile phone at all. The breakdown between test groups was nearly identical. Table 8. Test Subject Mobile Phone Usage. Phone Total Gradual Immediate Type Count Percent Count Percent Count Percent Android % % % iphone % % % Other % % % None 2 5.0% 0 0.0% % Total % % % Figure 13. Test Subject Mobile Phone Usage Chart. These statistics presented in this section show the diversity and variation of the test subjects used in this study. Because of the smaller number of test subjects within each group resulting in an even smaller number of test subjects with particular characteristics, strong correlations have not been performed with this demographic data for this study. Presentation of the Data Data was collected from test subjects during October Each test lasted an average of approximately 50 minutes, with the shortest test lasting approximately 35 minutes, and the longest test lasting nearly 90 minutes. Test sessions were held in a classroom on the campus of Pensacola Christian College, specifically room AC 215. A few tests were conducted in an adjacent room, AC 214, because of another officially scheduled class being held in AC 215 during those test session times. The room layout was mostly similar without any noticeable effect to the testing sessions being held in that room.

23 Figure 14 contains a picture of the chair layout used for the test sessions in AC 215. The test subject sat in the left-most chair (with the smart phone resting on it). The chair in the middle and closest to the wall (in the picture, this is the empty chair) was used to hold folders, papers, and the smart phone when it was not in use. The chair on the right (actually has the folders on it in the picture) was where the researcher sat to administer and observe the testing. Figure 14. Testing Room Layout. After initial greeting and test presentation, the test subject was asked to complete an initial survey, which included two main sections. The first section asked the test subject for demographic data, including gender, dominant hand usage, age, and daily average use of a touch screen keyboard. The second section included ten usability questions that the test subject answered after viewing an image of an alternative keyboard layout. During the testing, the researcher observed which fingers the test subject used to interact with the keyboard. And between test sessions, the researcher inquired what kind of mobile phone the test subject personally used. The TEMA application used on the Android mobile device recorded typing speed as WPM and errors rates, including TER, CER, and UER. At the conclusion of the testing with the smart phone, the test subjects were asked to complete the ten usability questions again. The ten usability questions were combined into one metric, with a maximum value possible being 100 resulting in a usability value before testing began and another usability value after testing concluded for each test subject. Comparison of various means revealed some interesting findings, with a complete table being given in Appendix N. First, the usability values of before and after (again, possible usability values ranged from 0 to 100), as seen in Figure 15, showed an increase for both the gradual and the immediate keyboard. The mean increase for the gradual keyboard was 5.25, which is a respectable 9.98% increase in perceived usability by test subjects; however, the

24 mean increase for the immediate keyboard was 11.75, which is a larger 19.30% increase. While the usability mean is different by 8.25 between the two test subject groups prior to testing, the usability mean difference increased to after the testing was completed. Figure 15. Usability Before and After Means. Data entry speed by test subjects is also a critical metric in this study, since the two test groups were exposed to the new keyboard layout in two different ways. The words per minute (WPM) metric for the first test with both groups were expected to be the same, and with only a 0.59 WPM (or 2.0%) difference between the groups this held true. And also expected was that the immediate keyboard group s speed would fall dramatically as soon as the keyboard layout changed in test #2, while the gradual keyboard group s speed would be only minimally affected. As seen in the line chart of Figure 16, the immediate keyboard group s speed dropped by WPM (55.44%), while the gradual keyboard group s speed dropped by only 3.33 WPM (11.95%); therefore, this expectation also held true.