Statistical Analyses of E-commerce Websites: Can a Site Be Usable and Beautiful?

Size: px

Start display at page:

Download "Statistical Analyses of E-commerce Websites: Can a Site Be Usable and Beautiful?"

Georgiana Lyons
5 years ago
Views:

1 Statistical Analyses of E-commerce Websites: Can a Site Be Usable and Beautiful? Thomas S. Tullis and Cheryl M. Tullis Fidelity Investments; Bridgewater State College tom.tullis@fidelity.com; ctullis@bridgew.edu Homepages of 42 retail e-commerce websites were analyzed to determine if measurable characteristics could be defined which would correlate with user ratings of visual appeal and with scores from a standard customer satisfaction index. The visual appeal ratings correlated with the size of the area of the largest image on the homepage and the total number of characters on the homepage. Visual appeal ratings improved as the size of the largest image increased, and they worsened as the total number of characters increased. Those two variables accounted for 60% of the variance in the visual appeal ratings. Customer satisfaction ratings for the sites were from published data. Those ratings correlated with whether the sites used a variable-width or fixedwidth page design, the link density on the homepage, the presence of a Flash element on the homepage, and whether the site supported font scaling. Sites with a variable-width design tended to have higher satisfaction ratings, as did sites that support font scaling. Sites with a Flash element on the homepage tended to have lower satisfaction ratings, as did sites with higher density of links on the homepage. These four variables accounted for 40% of the variance in the customer satisfaction ratings. 1 Introduction Forty-two e-commerce websites were included in the American Customer Satisfaction Index (ACSI) Holiday 2006 report on retail sites, including Amazon, Barnes & Noble, L.L. Bean, Circuit City, Neiman-Marcus, and others (Freed, 2006). For each site, the report included the site s rating on the ACSI customer satisfaction index, which is based on responses to a battery of standard rating scales from thousands of users. The scores ranged from a low of 68 to a high of 86. The current study focused on two main goals: (1) To have users assess the visual appeal of the homepages of those websites, and (2) To determine if objectively measurable characteristics of the websites could be defined which would tend to correlate with either the customer satisfaction ratings or the visual appeal ratings. In essence, what characteristics of websites tend to make them more usable, and what characteristics tend to make them more beautiful? This research was inspired by several earlier studies, including Tullis (1984) and Ivory & Hearst (2002). The Tullis (1984) series of studies, which pre-dated the web and dealt with alphanumeric displays, found that measurable characteristics of screen

and the subjective ratings that users assigned to them.

pages that would predict ratings by experts.

2 Visual Appeal Ratings To assess the visual appeal of the websites, their homepages were printed, in color, one per sheet.

Twenty-five web users ranging in age from 16 to 57 were given the printouts of the webpages, in random order, and asked to

2 2 Thomas S. Tullis and Cheryl M. Tullis displays could reasonably predict the time it would take a user to find a given piece of information on those screens and the subjective ratings that users assigned to them. Ivory and Hearst (2002) conducted analyses of large sets of webpages to determine if they could define characteristics of the pages that would predict ratings by experts. Some of the characteristics studied were number of links on a page, number of images, how the page style is controlled, and download speed. While their focus was on predicting expert ratings of the pages, the present study focuses on predicting actual user ratings. 2 Visual Appeal Ratings To assess the visual appeal of the websites, their homepages were printed, in color, one per sheet. If the webpage was too long to fit on one printed page, it was cropped. Twenty-five web users ranging in age from 16 to 57 were given the printouts of the webpages, in random order, and asked to sort them into four categories from Most Visually Appealing (rating=4) to Least Visually Appealing (rating=1). They were asked to try to ignore what the site was actually featuring or selling. On this 1-4 scale of visual appeal, the mean ratings ranged from a high of 3.5 to a low of 1.3. The pages that received the highest and lowest visual appeal ratings are shown in Figures 1 and 2. Fig. 1. The three webpages that received the highest (best) visual appeal ratings. Fig. 2. The three webpages that received the lowest (worst) visual appeal ratings.

Statistical Analyses of E-commerce Websites: Can a Site Be Usable and Beautiful?

percentages, with a theoretical range of 0 to 100, with higher scores indicating greater satisfaction.

ratings of their ability to accomplish what they wanted on the site, the ease of reading information, the clarity

For the Holiday 2006 report, the homepages of the sites that received the highest and lowest customer satisfaction

Fig. 4. The three websites that received the lowest (worst) customer satisfaction ratings.

3 Statistical Analyses of E-commerce Websites: Can a Site Be Usable and Beautiful? 3 3 Customer Satisfaction Scores Customer satisfaction scores, as computed by the ACSI, can be thought of as percentages, with a theoretical range of 0 to 100, with higher scores indicating greater satisfaction. These are based on responses by thousands of online users of the sites to a variety of rating scales, including ratings of their ability to accomplish what they wanted on the site, the ease of reading information, the clarity of the site organization, and their overall satisfaction with the site. For the Holiday 2006 report, the homepages of the sites that received the highest and lowest customer satisfaction scores are shown in Figures 3 and 4. Fig. 3. The three websites that received the highest (best) customer satisfaction ratings. Fig. 4. The three websites that received the lowest (worst) customer satisfaction ratings. Overall, there was not a significant correlation between the visual appeal ratings and the customer satisfaction scores (r =.21). This is as expected, since the customer satisfaction scores tap far more aspects of a website than simply visual appeal. 4 Characteristics of the Webpages A total of 17 objectively defined characteristics of webpages were identified. Some are continuous variables, such as number of links on the page, number of images on

4 4 Thomas S. Tullis and Cheryl M. Tullis the page, total file size of the page and its inline elements, total length of the page in pixels, area (in pixels) of the largest non-text image on the page, number of text characters on the page, the RGB values of the average color of the page, and the average grey value of the page. Some are binary variables, such as whether the page supports font scaling via the browser, whether it uses a variable-width design, whether the text links on the page visually highlight on mouseover, whether it contains a site search feature, a contact link or information, a site map link, and a link for help or customer service, and whether the page contains a Flash element or other animation. In addition to these primary characteristics, a variety of secondary characteristics were calculated from them, such as number of links per unit of page length ( link density ). Correlations of each of these characteristics with both the visual appeal ratings and the customer satisfaction scores were then calculated for the 42 pages, followed by multiple regressions. 5 Predicting Visual Appeal and Customer Satisfaction Scores A number of the individual correlations between these characteristics and the visual appeal ratings were significant (p <.05): Size of largest image area (r =.71), Total page size per link (r =.61), Number of characters (r = -.51), Page length per link (r =.50), Sum of average RGB values (r = -.49), Number of links per unit of page length (r = -.40), Page length (r = -.36), and Number of links (r = -.36). Two of the strongest correlations are illustrated in Figure R 2 = Visual Appeal Visual Appeal R 2 = Largest Non-text Image Area (k) Number of Characters Fig. 5. Two of the strongest correlations with visual appeal: Size of largest non-text image had a positive correlation (left) and total number of characters had a negative correlation (right). Multiple regression analyses revealed that the most effective model for predicting visual appeal was a two-predictor model using the area of the largest non-text image and the number of characters. Visual appeal increased with the size of the image area and decreased with the number of characters. This model accounted for 60% of the variance in the visual appeal ratings (p <.001). Only two of the simple correlations between the webpage characteristics and customer satisfaction scores were significant (p <.05): Use of variable-width page

5 Statistical Analyses of E-commerce Websites: Can a Site Be Usable and Beautiful? 5 design or not (r =.43) and Number of links per unit of page length (r = -.30). However, multiple regression analyses revealed that the most effective model for predicting the customer satisfaction score was a four-predictor model with the following predictor variables: Use of variable-width design or not, Number of links per unit length, Use of Flash or not, and Support for font scaling or not. Higher customer satisfaction ratings were associated with the use of variable-width design and font scaling. Lower customer satisfaction scores were associated with the use of a Flash element on the homepage and higher link density. This model accounted for 40% of the variance in the customer satisfaction scores (p <.001). 6 Conclusions Overall, we found that different webpage characteristics tend to be associated with visual appeal than with customer satisfaction. Pages that users find visually appealing tend to have a relatively large non-text image and fewer characters (as in all three of the sites shown in Figure 1). The pages that received higher customer satisfaction ratings tend to have a variable-width design, support font scaling, do not include a Flash element on the homepage, and have lower link density. None of these features inherently conflict with each other, so all can be implemented in a single site. There are three important points to keep in mind about this study: All 42 of these sites are reasonably successful, high-profile commercial websites that were probably professionally designed. As a result, none of the designs are what might be considered obviously bad. Therefore, this study was really trying to identify design features that tend to distinguish the outstanding sites from the merely good sites. The correlations found in this study are simply indications of relationships, and not explanations of cause. The underlying cause for one site performing better than another could be some other factor not measured in this study that one of these measures happens to correlate with. While there is no direct evidence of causation, it appears possible to implement features that improve both visual appeal and usability without detracting from either. 7 References Freed, L. (2006). The 2006 Online Holiday Shopping Season: Winners and Losers. Available from Ivory, M. & Hearst, M. (2002). Statistical profiles of highly-rated web sites. Proceedings of CHI 2002 Conference on Human Factors in Computing Systems, April 20-25, 2002, Minneapolis, MN, USA, pp Tullis, T. S. (1984). Predicting the Usability of Alphanumeric Displays. Ph.D. Dissertation, Rice University, Houston, TX, USA. Also summarized in Tullis, T. S. (1988), A system for evaluating screen formats: Research and application, In Hartson, R. and Hix, D., Advances in Human-Computer Interaction: Volume 2, Ablex Publishing Corp., Norwood, NJ, pp

UPA 2004 Presentation Page 1

UPA 2004 Presentation Page 1 Thomas S. Tullis and Jacqueline N. Stetson Human Interface Design Department, Fidelity Center for Applied Technology Fidelity Investments 82 Devonshire St., V4A Boston, MA