A/B Tests in Google Analytics To Make UI Decisions

Size: px

Start display at page:

Download "A/B Tests in Google Analytics To Make UI Decisions"

Augustine Tucker
6 years ago
Views:

1 A/B Tests in Google Analytics To Make UI Decisions

2 Who do you think you are!? David Schlotfeldt Co-owner of Plaudit Design

3 What s this about? A/B Tests in Google Analytics to Make UI Decisions First, a couple comments on the title... Applicable beyond UI - making any decisions you can test on your website (e.g. price of a product) A/B but beyond A and B to allow for many variations

4 Covered in This Presentation What AB Tests Are Running an Experiment Howto Setup an Experiment in GA Reading Results of an Experiment During and After

5 What AB Tests Are

6 Explaining, by explaining why Modified questions on contact form

7 Why an increase? Because my form change frickin rocked!!..or We are in a seasonal industry Sales department picked up involvement in expos Started a large social marketing campaign Many reasons! If we could see what would have happened if we didn t make the change...

8 Same blue line... Modified questions on contact form

9 AB Tests Allow you to compare with to without Split users into the control group (they see the original) and those who will see changes (variations) Isolate the effects of the change Everything beyond the change we care to measure modifies the results of each segment equally.

10 Types of Tests A/B - Testing two variations of a page. Multivariate - Testing various combinations of components on a single page e.g. 5 different images x 4 different headlines = 20 variations A/B/N - Testing two or more variations of a page. == A/B/N model == Experiments

11 Running an Experiment

12 Example Will be using a real example throughout this presentation AB test selected because it is: Recent Small Perfect for after presentation group discussion

13 Example Goal: Increase sales leads Metric: Online contact form submissions Identified: A low number of users who view the first step of the contact form move on to the second step

16 ... Continued We believe the first step is overwhelming users. It contains a large number of options and each has a paragraph explanation. Hypothesis: If we simplify and decrease the number of options a user chooses between at any given time then form submissions will increase.

17 Steps to running an experiment: Plan Create Variations Configure Experiment Let it Run Analyze Results

18 1) Plan

19 What variations will be tested? Original always one variation When testing multiple changes at once attempt to have a variation per combination How many variations? Consider how long you want the experiment to run for and the traffic available when selecting number of variations. Up to 10 variations per experiment in Google Analytics (According to docs but you can configure 35 + the original.)

20 How to evaluate? We ll expand on the following as we walk through configuring an experiment. Need to decide: Metric that will be used to evaluate the results Page viewed, session duration, bounce, etc. Percentage of traffic to include in the experiment Distribute traffic evenly across variants (yes or no) Control length of experiment: Specify minimum time the experiment will run Minimum confidence threshold to be achieved before GA declares a winner

21 Our Example Evaluate: Increase contact form submissions Variations Variation 1 - Present options in multiple steps and remove descriptions. Doing so we believe users won t become overwhelmed and their level of commitment will increase with each selection they make. NOTE: Ideally we would have ran multiple variations to test multiple changes, but as noted earlier multiple variations increases timeline and best to minimize changes per variation.

24 2) Create Variations

25 Develop the Variations HTML, CSS, blah, tech stuff, blah Each variation will either be Their own page Differences contained on one page Which approach you use depends on how you integrate (More on this later)

26 3) Configure Experiment

27 Create Experiment Login Reports (Top) Behavior > Experiments (Left side) Click Create experiment (Right side)

30 Percentage of users to include in experiment. Those not included will see the original. When deciding consider: Volume of traffic to page Number of variation Riskiness of the changes

32 Multi-armed bandit experiments Distribute traffic evenly across all variants off by default on GA does not adjust traffic dynamically based on variation performance GA uses multi-armed bandit for experiments which has the goal of finding best performing and adjust distribution of users as experiment progresses Twice per day adjusts the fraction of traffic that each variation will receive going forward. Adjustments made using a statistical formula that considers sample size and performance metrics together. You benefit by traffic being moved towards winning variations gradually, instead of having to wait for a "final answer"

33 Multi-armed bandit experiments

35 Two Options Options presented are: 1. Manually insert the code 2. Send the code to webmaster These options are: Easy to implement Simply copy and paste Javascript Code will select a variation for each user and redirect them to the URL of the variation.

36 Oh wait! There are two more options! Additional options not presented in wizard Both require programming Both do not redirect to a separate page URL fields for variations can be fake. E.g. 1.com, 2. com. Validation errors can be safely ignored Both are more involved so I m only introducing them here.

37 Option 3: Javascript API Client Side Advanced Javascript Implementation  <script src="// <script> // Ask Google Analytics which variation to show the user. var chosenvariation = cxapi.choosevariation(); </script> More Info: com/analytics/solutions/experiments-client-side

38 Option 4 : Server Side Requires backend coding - PHP, Java,. net, etc. (Google provides great documentation on exactly how to code this.) No delay/flash in page load/initialization waiting for JS Able to test more - e.g. change price of a product

39 Option 4 : Server Side 1. Store and Refresh - Periodically update local data about experiments 2. Choose Variation a. For the current user interaction (e.g. pageview), is there an experiment running? b. Has the user been previously exposed to this experiment? c. Should the user be included in the experiment? d. Choose a variation for the user 3. Send experiment and select variation to Google Analytics (Normally injecting JavaScript into the page) More Info: experiments-server-side

40 4) Let it Run

41 Let it run! Last step of wizard: Click Start Experiment Automatically stops when clear winner or hits 3 months max List of Running Experiments: View of an Experiment:...

42 View of an experiment

43 Once start you can modify Name of experiment and variation pages Percentage of users in the experiment pages If notifications are sent Variations can be disabled. e.g. If you feel one of your variations is clearly under performing. Will shorten experiment since more users split between less variations. Stop an experiment. Keep in mind it cannot be restarted

44 5) Analyze Results

45 It finished! (Or you got tired of waiting ) Now what? Original page only - Users will no longer be presented variations Best variations - If a variation won, modify the site If variations existed at different URLs, set up permanent redirects to new page for those who bookmarked a variation Run another test! With more insight from that experiment consider running a new one for additional improvement

46 Analyzing Beyond Who Won Create segment to analyze behavior of the users who were served a specific variation Can use segment to analyze and compare any report in Google Analytics How they used the site: Behavior reports (e.g. Users Flow, Behavior Flow, etc) How they found the site: Acquisition reports (e.g. Channels, AdWords, etc) More Information: com/analytics/answer/ ?hl=en&ref_topic=

47 Wrapping Up The Preso

48 Summary Experiments allow you to isolate the effect of changes Google Analytics uses A/B/N model The process to running an experiment: 1. Plan a. What variations will be tested? b. How to evaluate? 2. Create Variations a. Development 3. Configure Experiment a. Setup in Google Analytics b. Configure on site with four potential approaches i. Manually insert the code ii. Send the code to webmaster iii. Javascript API Client Side iv. Server Side <-- My preference 4. Let it Run a. Watch reports. b. Modify if needed including disabling variations. 5. Analyze Results a. Put winner in place. b. Use segments to further analyze

49 Critique - Room for improvement! What changes could be tested in the next experiment to further improve conversions? Process 1. In small groups brainstorm ideas 2. Share ideas with the larger group 3. As a group select top ideas 4. Define variations

50 The end! Until next time

A/B Testing Overview

A/B Testing Overview A/B Testing (also known as Split testing) defines a way to compare two versions of an application or a web page that enables you to determine which one performs better. It is one of