Test Like a Hawk. To maximize results, base A/B Testing on hard facts, not on chance or intuition. A Clicktale e-book. Light up the digital world

Size: px

Start display at page:

Download "Test Like a Hawk. To maximize results, base A/B Testing on hard facts, not on chance or intuition. A Clicktale e-book. Light up the digital world"

Brianna Holt
5 years ago
Views:

1 Test Like a Hawk. To maximize results, base A/B Testing on hard facts, not on chance or intuition A Clicktale e-book Light up the digital world

2 3 A/B Testing: The Workhorse of Experience Optimization 5 Before Testing 10 During Testing 11 After Testing 13 Data-Driven A/B Testing in Action 15 How Clicktale Can Help

3 A/B Testing: The Workhorse of Experience Optimization A/B Testing is an important tool for virtually all website stakeholders, and a well-established, essential part of the user experience toolbox. A/B Testing allows us to zoom in on a given page element, and micro-optimize it for performance. It excels at providing answers to which questions, including: 1. Which of several elements works best in this spot? 2. Which page element do customers interact with more consistently? 3. Which version of the page converts more? The whys that explain A/B Testing results, however, are considerably more difficult to pick out: Why should I test this specific element as opposed to another? Why do more customers convert from this image than from another? Why are results as they are? Why are users dropping off more frequently than before? 03 Test Like a Hawk

4 While serial testing might reveal some of the answers to some of these questions, such an approach is costly and inefficient. Each testing cycle entails creative development, putting the creative into production, and once the test is launched, waiting until statistical significance is achieved. A/B testing can also be a highly political process. Permission to test promotions or content that were championed by powerful individuals in an organization may be hard to secure, leaving potentially problematic elements unexamined. Process of A/B Testing Process of A/B Testing with Clicktale Making A/B Testing Focused, Faster and Way More Informative Improving the selection of A/B Testing targets, expediting the testing process, and bringing deep customer experience-based understanding to results can further enhance the value of A/B Testing. 04 Test Like a Hawk

5 03 Test Like a Hawk Data Driven A/B Testing Objective Informative Impactful Data-driven issue identification to focus testing on what really matters, not on issues selected via gut-driven hunches Accelerated testing cycles leveraging quick, accurate discovery of unforeseen issues based on small-sample observations Actionable conclusions and enhanced buy-in based on insight, analysis and visualization of customer behavior, needs and intent.

6 Discovering What Makes Test Results Tick Data-Driven A/B Testing increases ROI from site optimization efforts by shining the bright light of data from actual customer behavior and in-page actions on every step of the testing process. In doing so, it adds a new layer of in-page insights to the testing toolbox. By leveraging existing user experience data and solutions, testers can visualize previously-hidden factors that contribute to the success of one element over another. It reveals insights that facilitate identification of productive test subjects and faster test iterations. Before tests, heatmaps and form analytics can help you pinpoint webpage elements that are most engaging, most unexpected, or most significantly affect customer behavior - and thus are more worthy of testing. After the test, session playbacks can reveal the impact of the changes you made on actual visitor interactions. But equally importantly, visualization of customer interactions earns you the trust and buy-in you need to make changes based on the findings of your tests. Data-Driven A/B Testing shines the bright light of data from actual customer behavior and in-page actions on every step of the testing process. 06 Test Like a Hawk

7 Before Testing Experienced A/B testers know that the most effective test is a well-researched one. One of the most challenging parts of the testing process is defining what to test, why it s important, and how the various potential results can impact page performance. In the pre-testing stage, the Data-Driven A/B Testing approach helps you focus test investments where they ll have the greatest impact, enabling you to better assess your hypotheses based on hard data. On-site activity If few visitors scroll past the fold, test elements that signal that there's more to see below Desktop and mobile heatmaps illustrate visitor mouse moves, clicks or taps, attention, and scroll reach or exposure. This gives you an aggregate-level visualization of what people are doing on your page. For instance, do very few visitors scroll down past the fold? Design and test elements or layouts that alert users that there is more they should see. Actionable Aggregated Insights Insights gained from closely analyzing in-page visitor behavior can help you create a more focused test. Visually examining aggregated data from in-page customer experience metrics can help you discover which page areas require improvement. For example: 07 Test Like a Hawk

8 Cross-segment and crossversion comparison The idea of comparing different versions of the same page is certainly not foreign to A/B testers. Side-by-side heatmap comparisons are a great preliminary tool, before you move into structured A/B Testing. For example, do men and women respond differently to the same page or page element? Comparing page interactions side-by-side can help you better design your A/B Test to micro-optimize for each target audience. Visitor interactions with links Link analytics drill down shows % of visitor clicks Reviewing a graphical representation of every link on the page - including visitor hovers, clicks, hesitation time, and visit order can provide key pre-testing insights into the decision-making processes. 26% 14% 12% Version A Version B 32% Design better A/B tests to optimize for segments 0.5% 0.2% 9% 0.5% 0.5% 0.5% 0.5% 0.5% 08 Test Like a Hawk

Make Every Visitor Count From the aggregate level, drill down to the individual level. Understand actual customer intent by watching real customer journeys.

9 Make Every Visitor Count From the aggregate level, drill down to the individual level. Understand actual customer intent by watching real customer journeys. Then judge how your user experience affects their ability to realize their goals using: Session Replays Event- Triggered Recorder Watch replays of user browsing sessions to visualize exactly what they re seeing and doing on your desktop or mobile site. Capture user sessions for key segments to synchronously analyze all analytics and marketing data. Examine visitor behavior around the specific issues that have the greatest impact on business results to gain better understanding of what changes to test in order to improve the user experience. Don t Forget Forms A smooth experience filling out and submitting forms can make or break the visitor experience and your business as a whole. Small wonder that forms are among testing s most important targets. Data-Driven A/B Testing takes the guesswork out of form testing. Leveraging advanced form analytics and session replays, testers can overcome the limitations of traditional form metrics like completion rates. By analyzing insights into real-world form usability issues, they can better focus testing on micro-optimizing layout, fields, and other form-specific issues that can dramatically impact conversions. Test entry formats and instructions for troublesome fields like phone numbers and CVV Select a Payment Type Saved Credit Card New Credit Card PayPal Card Number:* Name on Card:* Expiration Date:* Mary Roe January 2017 Enter the credit card security ID number or CVV Card Security ID (CVV):* Continue 09 Test Like a Hawk

10 During Testing Leverage visual evidence to iterate tests more rapidly and see results more quickly While your tests are running, the Data Driven A/B Testing approach helps speed time to results and increase return on your investment of time and resources in the test planning stage. For example, while tests are running validate that test variations are working as intended by watching session replays immediately after the first visitors experience your tests. This enables quick corrective action if needed, reducing potential lost testing time and expense. By leveraging visual evidence of what experiences are working best, you can iterate tests more rapidly and see results more quickly. Step 3 Change Step 1 Test Step 2 Watch 10 Test Like a Hawk

After Testing Version A Version B Prioritizing Changes and Future Tests In post-test assessments, Data-Driven A/B Testing empowers testers to better understand why certain experiences win and others

11 After Testing Version A Version B Prioritizing Changes and Future Tests In post-test assessments, Data-Driven A/B Testing empowers testers to better understand why certain experiences win and others lose. For example, by comparing test variations side-by-side using heatmaps, it s possible to uncover exactly how changes impacted visitor interactions. Removing the promotional header from Option A increased conversions by 4.3% for Option B By delving deeper into the why of A/B Testing results, you gain significant insights into the optimization cycle. Data-Driven A/B Testing helps you quickly and conclusively determine which versions were the true winners of the test cycle and how to best prioritize implementation of necessary changes. Accurate identification of future testing targets is another important benefit of data-driven A/B testing. In the words of a marketing analyst at major e-commerce company, we can engineer even better experiences and tests by knowing what were the elements in previous tests that led to success. 11 Test Like a Hawk

Visual presentation makes results clear and improves buy-in Getting Buy-In for Change Although A/B Testing is anything but subjective, it is often challenging to get organizational buy-in for changes

12 Visual presentation makes results clear and improves buy-in Getting Buy-In for Change Although A/B Testing is anything but subjective, it is often challenging to get organizational buy-in for changes indicated by testing results. Presentations of findings can easily degenerate into unnecessary and distracting questions about methodologies, confidence factors, variants, and other technical data. Data-Driven A/B Testing helps testers recast the dialog from technical issues to user experience. By integrating customer intent with A/B Testing data, you can markedly change the way your organization looks at web analytics as a whole. A/B Testing platforms show test result data and indicate which version of a given page or page element worked best. Add the why to this which by revealing the reasons one version performed better than another. It is precisely these whys that decision-makers and website stakeholders need to hear when evaluating testing findings and suggested changes. Finally, the Data-Driven A/B Testing approach provides testers with the unique ability to quantify the impact of changes. Intelligent use of user experience data enables accurate prediction of how revenues will be impacted when recommended changes are implemented and how much your company may lose if they re not. 12 Test Like a Hawk

13 Data-Driven A/B Testing in Action Growing Growing The The North North Face Face Page Page Conversions Conversions by by Over over 60% The business The challenge The solution A division of VF Corporation (NYSE: VFC), a global leader in lifestyle apparel, footwear and accessories, The North Face is one of the world s most recognized brands and one of the web s most heavily trafficked fashion and lifestyle destinations. Looking to increase conversions on their shopping cart page, The North Face evaluated page performance with their traditional web analytics solution. They found that many visitors were looking at the page, but not making it to checkout. Heatmaps and session playbacks revealed that a large number of visitors were not noticing the Checkout button on the Shopping Cart page. The North Face theorized that visitors were being distracted by a promotional banner above the button. To test this theory, The North Face ran an A/B test on the position of the banner and Checkout button. Version A was the control with the Checkout button positioned below the banner, whereas Version B reversed it with the Checkout button above the banner. 13 Test Like a Hawk

Version A Version B The results The North Face used Monetate to conduct the A/B test. The test showed that in the original version, around 16% of visitors to the Shopping Cart clicked the banner.

At the same time, there was an increase of 21% in the click-through rate of the Checkout button when it was placed above the banner.

14 Version A Version B The results The North Face used Monetate to conduct the A/B test. The test showed that in the original version, around 16% of visitors to the Shopping Cart clicked the banner. In the new version, by moving the Checkout button up alongside the Order Summary (pricing and shipping information), only 0.2% clicked the promotional banner. At the same time, there was an increase of 21% in the click-through rate of the Checkout button when it was placed above the banner. The conversion rate of Version A (checkout button below the banner) was 40%, whereas the conversion rate of Version B (checkout button above the banner) was 65% - effectively increasing the page s conversion rate by 62%! What they re saying "We had not realized that the position of the banner would cause confusion among potential buyers. Clicktale s in-page analysis steered us towards running the A/B test, and we are very happy with the outcome. After a short period of testing, we modified the page and the metrics could not be better. Bei He, Business Analyst, The North Face 14 Test Like a Hawk

15 How Clicktale Can Help Clicktale complements and extends traditional A/B Testing solutions by offering valuable insights that: Focus tests on what users actually experience Raise testing efficacy and relevance Neutralize political considerations in test target selection Enhance overall business impact Together, Clicktale and leading A/B Testing solutions create a powerful and highly-actionable user experience ecosystem, ushering in a new era of Data-Driven A/B Testing. Provide accessible and visually compelling validation of testing results Help move decision makers to action Light up the digital world Global Offices US: UK: About Clicktale: Clicktale lights up the digital world, revealing customer behavior, needs and intent across all key touchpoints: web, mobile and apps. The world s #1 enterprise-class experience management platform, Clicktale s deep drill-downs get to the heart of the how and why behind customer behavior. Via session replays, high-fidelity heatmaps, conversion analytics, powerful integrations and in-depth expert analysis, Clicktale provides critical insights needed to remove friction, delight every customer and achieve digital success. With a global customer base that includes Walmart, UBS, The North Face, MetLife, Lenovo, Adobe and CNN, Clicktale is the most advanced, robust enterprise-grade solution on the market, analyzing over 100 billion in-screen behaviors per month.

$ Turning errors into opportunities. How marketers can turn inevitable website errors into invaluable insights

x $ Turning errors into opportunities How marketers can turn inevitable website errors into invaluable insights 4 Executive Summary 5 Errors: They Happen and They Hurt 6 Why Traditional Tools Fall Short