A Comparative Usability Test Orbitz.com vs. Hipmunk.com 1
Table of Contents Introduction... 3 Participants... 5 Procedure... 6 Results... 8 Implications... 12 Nuisance variables... 14 Future studies... 15 Appendix... 16 2
Introduction Currently, there are many other travel agent websites in the market. Among them, Priceline.com is famous for the cheap price that it can provide; KAYAK.com stands out with a very brief and very clear home page; and Hipmunk.com is known for its cute and friendly design and very graphical-oriented searching results presentation, and has been reviewed as the "world's best travel site" (http://www.forbes.com/sites/bruceupbin/2012/06/29/why-hipmunk-is-the-worldsbest-travel-site/). We compared Orbitz.com and hipmunk.com via an experimental study using direct observation in a controlled environment and two questionnaires to identify usability differences between the two websites. Different inquiry methods and different questionnaires were used to achieve a methodological triangulation. Hipmunk.com homepage 3
Presentation of flight searching results 4
Participants Total 8 participants were recruited. They were conveniently sampled to represent four populations: single full-time students, single non-students, married non-students without kids, and married students with kids. All participants had never used Orbitz.com or hipmunk.com before the time this study was conducted. Participants demographical information is briefly summarized by the following table (number of persons) Age range 21-25 26-30 31-35 1 4 3 Gender Male Female 4 4 Travel frequency Once Once a Once 6 Once a Less two month months year frequent years 1 4 1 1 1 Did use online travel agent before Yes No 8 0 5
Procedure All participants were tested individually. The whole procedure consisted of a pre-test questionnaire and a usability task session: Pre-test questionnaire Usability task session Web #1 Web #2 Book a trip Usability/user experience questionnaires Book a trip Usability/user experience questionnaires 1. Pre-test questionnaire The experiment started with a pre-test questionnaire for asking for basic demographic information (e.g. gender, age) and details of relevant experience (e.g. travel frequency). The questionnaire has been attached in the Appendix. 2. Usability task session In this session, participants were asked to book a trip on both Orbitz.com on hipmunk.com. The order of presentation of the two websites was counterbalanced across subjects for each population group. Book-a-trip task Participants were asked to plan a five-day vacation in Oahu island Hawaii for the family (the two family groups) or self (the two single groups) during the winter break (December to January). The task included booking a round-trip flight from San Francisco to the destination, and booking a hotel that was closed located. Instructions "Please pretend you (single) or your family (married) are going to have a vacation in Oahu island Hawaii during the winter break. You will take this trip in December or January. It will be five days long including the travel. You are going to book a round-trip flight and a hotel through this website. There is no time limit. There is no correct or wrong answer, either. Feel free to do it in your own way and make your own decision. 6
Please try to do this as if in your real life. If you don't have any questions, please get started." What was measured: Dwelling time Number of clicks on links (or steps) Obvious errors (error message; get stuck and ask for help) 3. Usability/User experience questionnaires Immediately after the participants completed the book-a-trip task, they were asked to finish two Usability and User experience questionnaires. Questionnaire #1 (see Appendix) Likert scale (5 points from strongly agree to strongly disagree) Overall User Experience was measured by Question 8-11 Questionnaire #2 (see Appendix) semantic differential scale (7 points) Overall usability and four specific usability: Legibility (screen), Visibility (information), Learnability (learning), and Effectiveness (capability) 7
Results Average dwelling time (seconds) Total Male Female Orbitz 368 304 432 Hipmunk 412 330 494 6 of 8 participants took longer time on Hipmunk than on Orbitz Females seems to linger longer on Hipmunk. Average number of clicks on links (or steps) Total Male Female Orbitz 22 22 22 Hipmunk 26 23 29 6 of 8 participants took more steps on Hipmunk than on Orbitz Number of steps was significantly correlated with dwelling duration (Spearman's r = 0.74 and 0.83, respectively) Errors/Total number of errors Type one Type two Type three Type four Orbitz 1 0 1 0 Hipmunk 1 4 0 2 8
Type one: Type two: Type three: Type four: Errors on the first Transition error difficulty in locating Others errors page information (e.g. selecting depart (e.g. don't know how (e.g. don't know Basically every other and return date; fill in to book hotel after where to see the map, error would be the start and booked the flight) hotel description, etc.) categorized in this destination) one. Total number of errors Total Male Female Orbitz 2 2 0 Hipmunk 7 2 5 More errors (especially Type two errors) occurred when using Hipmunk Females made more errors on Hipmunk. Questionnaire 1 1. 5 of 8 participants thought Orbitz was easier to navigate than Hipmunk 2. 5 of 8 participants thought Orbitz was slower than Hipmunk 3. 3 participants preferred using Orbitz, 4 participants preferred using Hipmunk, and 1 participant had equal preference 4. There was no correlation between Overall User Experience with task performance (either dwelling duration or number of steps). Questionnaire 2 9
1. Overall Usability in Questionnaire 2 was significantly correlated with the Overall User Experience in Questionnaire 1 (Spearman's r = 0.86 and 0.73, respectively). 2. 4 participants viewed that Orbitz had better usability than Hipmunk, and 4 participants viewed that Hipmunk had better usability than Orbitz. 3. Generally, female participants more preferred Orbitz and male participants more preferred Hipmunk. The following figure shows this gender-website interaction effect on Overall usability: Gender and Website interaction effect on Overall usability 10
Result Summary Most participants had better performances using Orbitz than Hipmunk: shorter dwelling time, less steps, and less errors Most participants thought that Orbitz was easier to navigate but much slower. Usability and User Experience results from Questionnaire 1 and Questionnaire 2 were consistent. Half of the participants preferred using Orbitz and half of the participants preferred using Hipmunk. There was a gender-website interaction: female participants more preferred Orbitz while male participants more preferred Hipmunk. 11
Implications All participants had used some kind of online travel agents before the test, such as Priceline, Expedia, etc. Those websites are much more similar to Orbitz than Hipmunk. Some participants preferred using Orbitz possibly because it is familiar and therefore easy to learn and use. Although Orbitz had a big advantage in familiarity, there are still half of the participants did not prefer using it. The possibilities are listed below. 1) Long waiting time One of the biggest issues of Orbitz is the loading time and the blank waiting time: it is slow. UI designer may not be able to change the algorithms for loading data, but it is possible to make waiting time seem shorter. Hipmunk does a much better job at this: User's are able to receive text messages about the information they are searching for. This gives users time and a choice in how they should best utilize the time they have before the information is ready. Hipmunk's loading window 2) Lack of transparency Results revealed that it is easier to book both flight and hotel through Orbitz compared to Hipmunk. Orbitz provides an integrated design with a ready option of choosing both flight 12
and hotel. However, in choosing this option, transparency becomes poor as user's are unable to differentiate the price for the flight or the flight of the hotel. This may cause user's to leave the site to double check prices and book with another website instead. In addition the total amount of money may intimidate user's and give them a perception that they are spending more when they are not. One participant claimed that the price that Orbitz provided was more expensive than Hipmunk: this was not true. Only the total prices (flight + hotel) are shown by Orbitz 3) Relevant information is not directly presented Location is an important factor travelers take into consideration when booking hotels. One participant had a hard time figuring out where a hotel was located, and opened up google maps to look for the information instead. Orbitz does provide a map, but the link is very small and hard to see. In contrast, Hipmunk provides a list of hotel options and a map on the same screen; when the map is dragged and moved, the hotel list will change correspondingly. The next page you will see an example. 13
The map link in Orbitz and the map in Hipmunk Nuisance Variables 1. The two variables, Married/single and male/female, were not well counterbalanced: 3 of 4 married individuals were female, which made the results hard to explain. We did not expect gender effects before the test. 2. Artificial condition: participants' performances could be totally different from those in reality. 3. Experimenter and participants relationship could affect results. Female participants and married individuals spent more time and were more careful. This is perhaps due to their concerns about their social relationship with the experimenter. 4. Study was conducted in three different locations. The speed of website was much affected by the local WiFi network. 14
Future Studies There are some other studies that can be conducted to better understand this project. They are briefly described below. Interview with experienced users It is to (a) understand how the users behave and think about things, (b) identify user experiences of the website in experienced users, and (c) detect problems in the current design that require change or improvement. About 3-5 experienced users will be interviewed, and they should have used this web at least 3 times and have used it at least once during the past 3 months. The interview will be conducted via a face-to-face manner and in a quiet and comfortable place. Information interested in learning from users includes: o The context of how the website fits into their lives or work: when, why, and how it is or will be used; o The kind of activity this website is required for (or supports the most) and the kind of activity this website does not support; o Goals and motivations for using this website; o What expectations users have about this website; o Problems and frustrations with the current design of the website. Online survey This study would serve to reach out to as many people as possible. The Pre-test questionnaire and Usability and User experience Questionnaire 1 and 2 that have been used in this study would be posted online. When participants are finished and submitted their responses, data will be automatically collected so that proper analysis can be performed. 15
Appendix -- pre-test questionnaire Please check the most appropriate selection: 1. Age range: 20 and younger 21-25; 26-30; 31-35; 36-40; 41 and older 2. Gender: Male Female 3. Travel Frequency: more frequently than once a month once a month once every two months once every six months once a year once two years less frequently than once two years 4. Did you use any online travel agents to book your trip: Yes No If the answer is "Yes," Which are they? What functions did you use on their websites? Flight only Hotel only Flight + Hotel Flight + Hotel + Car Others Which one do you prefer the most and why? 16
Appendix -- usability questionnaire 1 Please consider each statement and select your agreement with each one. Strongly agree Agree Neutral Disagree Strongly disagree 1. It is easy to navigate through this web site. 2. It is easy to find what I want on this web site. 3. This web site loads too slowly. 4. The graphics on this web site are pleasing. 5. It is easy to use this site upon my first visit. 6. Clicking on links takes me to what I expect. 7. The organization of information on the system screens is clear. 8. I am satisfied with the results, including flight, hotel, and price 9. I enjoy using this web to book my trip 10. I will use it again for my next travel 11. I will recommend this web to my friends 17
Appendix -- usability questionnaire 2 Please select one number to indicate your agreement with the following words. For example, 1 indicates strongly agree with the words on the left, 7 indicates strongly agree with the words on the right, and 4 indicates neutral. Overall reaction to the website 1 2 3 4 5 6 7 terrible difficult frustrating dull rigid wonderful easy satisfying stimulating flexible Screen 1 2 3 4 5 6 7 Reading letters on the screen hard easy Appropriately highlighting desirable links/information not at all very much Organization of information confusing very clear Sequence of screens confusing very clear Information 1 2 3 4 5 6 7 Position of messages on the screen inconsistent consistent Prompts for input confusing clear System informs about its progress never always Error messages unhelpful helpful Learning 1 2 3 4 5 6 7 Learning to operate the website difficult easy Exploring new features by trial and error difficult easy Remembering names and use of commands difficult easy Performing tasks is clear/direct never always Help messages on the screen unhelpful helpful Supplemental reference materials confusing clear Capabilities 1 2 3 4 5 6 7 Speed of the website too slow fast enough Web reliability unreliable reliable Correcting your mistakes difficult easy Designed for all levels of users never always 18