A - B Test Planning How To Build A Process That Works
A - B Test Planning How To Build A Process That Works
A strong A/B testing plan will allow you to increase your revenue. You’ll also learn valuable insights about your customers
because you’ll k now their preferences instead of guessing what they want.
A/B testing produces concrete evidence of what actually works in your marketing. Continuously testing your hypotheses
will not only increase conversion rates, but it will also give you a better understanding of your customers. Having a clear
idea of what your customers actually like and prefer can do wonders for your branding and marketing in other channels as
well.
At its core, A/B testing belongs to a category of scientific optimization techniques that use statistics to increase the odds
that your site visitors will see the best-performing version.
By levels of sophistication, scientific optimization can be broken down into three categories:
1. A/B split testing. Simple testing of one element of a page against another to see which one results in better
performance.
2. Multivariate Testing. Testing several elements at a time. The goal is to get an idea of which elements work together
on a page and play the biggest role in achieving the objective.
3. Experimental Design. Developing your own research method for an in-depth analysis of a specific element.
This post is about the A/B testing process. A good A/B testing plan produces the fastest gains and has a lower chance of
error through misuse.
A structured process to improve your conversion rates needs to be continuous. It is a cycle of Measurement,
Prioritization, and Testing (then repeat). Each stage has a goal and a purpose, leading to the next stage.
For your plan to be successful, follow this four-step A/B testing template.
Step 1: Measure your website’s performance.
To continually improve your conversion rates, start by properly measuring your website’s performance. We want to know
what is happening and w
hy it’s happening.
This is the answer to the question: “Why does your website exist?” Make your objectives DUMB—Doable, Understandable,
Manageable, Beneficial. Companies often fail in web analytics because their objectives are not simple to understand or
measure.
Example: A business objective for an online flower store is to “increase sales by receiving online orders for our bouquets.”
Define your website goals.
Goals come from your business objectives and are mostly strategic in nature. So, if we were to continue with the business
objective example of increasing our bouquet sales, we have to:
Goals come from your business objectives and are mostly strategic in nature. So, if we were to continue with the business
objective example of increasing our bouquet sales, we have to:
Goals are your priorities, expressed as simply as possible. Before you start working with your data, make sure you have
them defined and properly set up in Google Analytics.
A metric becomes a KPI only when it measures something connected to your objectives.
Example: Our flower store’s business objective is to sell bouquets. Our KPI could be the number of bouquets sold online.
This is the reason why you need to define your business objectives clearly—without them, you’re unable to identify your
KPIs. If you have p
roper KPIsand look at them periodically, you’ll keep your strategy on track.
Our flower store sold 57 bouquets last month. Is that good? Or devastatingly bad? For your KPIs to mean something, they
need target metrics. Define a target for every KPI. For our imaginary flower store, we can define a monthly target of 175
bouquets sold.
Now you have a framework that ensures that the work you do is relevant to your business goals.
Why it’s happening: Talk to your visitors
Getting real feedback from your visitors is invaluable. Use surveys to discover your visitors’ objectives. Set up entry
surveys to find out why they’re visiting your site and exit surveys to find out if their goals were fulfilled.
You can find what is happening on your site by using Google Analytics—which features they use, where they exit, and who
is profitable. But Analytics doesn’t tell you the w
hy behind this. This is where qualitative data comes in. Qualitative data is
perfect for finding out why problems occur.
The best way to get qualitative data from your visitors is through surveys. Your goal is to find out why visitors buy or why
they leave without purchasing.
Ideas for gathering qualitative data
● Add an exit survey on your site, asking why your visitors didn’t complete the goal of the site.
● Add an exit survey on your thank-you pages to find out why your visitors converted.
● Perform u
sability testing with members of your target group.
● Send out feedback surveys to your clients to find out more about them and their motives.
If a large amount of people click on your ebook ad but only a few people actually buy after seeing the price, you’ll want to
dig deeper into the problem.
In this example, you could put up a survey, asking people if they have any questions that the page doesn’t answer. You can
also survey people who have already bought your book to see what made them buy.
You’ll start noticing trends after you’ve collected 50+ responses. Often, you’ll find that the site hasn’t addressed an
important objection of your client. The main takeaway is that qualitative data will help you understand which elements will
have the highest impact when running an A/B test.
Think about how you could spot emerging trends with your customer feedback. When something pops up, you can dive
deeper to find out exactly what’s going on.
The problem with using site averages in your testing is that you’re missing out on what’s going on inside the average (the
segments).
An experiment that seemed to be performing poorly might actually have been successful f or a certain segment. For
example, our experiment may have shown that a variation of a mobile landing page isn’t performing well. When looking
into the segments though, you may see that it’s performing exceptionally well for Android users but badly for iPhone
users. When not looking at segments, you can miss this detail.
Never report a metric (even the most hyped or your favorite KPI) without segmenting it a few levels deep. That is
the only way to get deep insights into what that metric is really hiding or to see what valuable information you can
use.” -Avinash Kaushik
To understand segments, we need to understand dimensions. A dimension is any attribute of a visitor to your website. A
dimension can be a source where they came from (a country, URL, etc.), technical information like their browser, or their
activity on the site (pages they looked at, images they opened). A segment is made up of a group of rows from one or
more dimensions.
By default, a lot of the data you get is useless. The number of visits to your site doesn’t really give you actionable
information. To get actionable data for testing, you need to segment the data you have, using dimensions. You can also
split test for single segments of your traffic, which is a part of behavioral targeting.
Avinash Kaushik has outlined a simple strategy for segmenting your traffic. T
he best ideas for taking action come from
segmentation.
Segment by source. Separate people who arrive on your website from email campaigns, Google, Twitter, YouTube, etc.
Find answers to questions like:
Segment by behavior. Different groups of people behave differently on a website because they have different needs and
goals. For example, an ecommerce store can separate people who visit more than 10 times a month from those who visit
only twice. Do these people look for products in different price ranges? Are they from different regions?
Segment by outcome. Separate people by the products they purchase, by order size, by people who have signed up, etc.
Focus on groups of people who have delivered similar outcomes and ask questions like the ones above.
Keep your most profitable segments in mind when building your split-testing plan. It’s good to know who your most
profitable visitors are b
efore you start split testing.
Step 2: Prioritize your testing opportunities.
Once you have your metrics in place and know your goals, the next step is to prioritize what to test. You could test
anything, but everyone needs a place to start from. Google Analytics gives you a lot of data, but it makes sense to start by
split testing opportunities that promise the biggest gains.
Your homepage may not be the most important area of your site. Look at your “top landing pages” report from Google
Analytics. You’ll likely see many different pages with entrances, some even more than your homepage.
Look at data on a page template level.When you add together the traffic from all of your pages that use the same
template, you may see that they get a lot more traffic than your homepage. Do this when you need to determine
opportunities for testing site-wide template layouts.
Prioritize pages with high potential for improvement.
Look for pages that aren’t performing well. Your Analytics data can show some clearly problematic pages, like landing
pages with high bounce rates, but other areas may not be so obvious.
If you optimize only shopping cart v iews, you may not fix the problem. You also need to look at your product and category
pages.
No single information source will perfectly identify split-testing opportunities. You need to look at several pages together.
Top exit pages. This is the last page that someone sees before leaving your site. Labeled “ % Exit” in Google Analytics, it
will show you the percentage of visitors who leave your site immediately after viewing the page. Top exit-rate pages can
identify problem areas within your u
ser flow.
A funnel in Google Analytics focuses on the bottom end of the funnel. If you have your funnels correctly set up, you can
gain valuable split-testing information from it.
Look for sudden drop-off rates in the funnel. For example, if only 18% of the traffic proceeds from Step 2 to Step 3 in the
checkout area, you have a problem in Step 2.
If you’ve identified a drop-off step in your transactional funnel, you should ask yourself why the problem is happening:
Answers to these questions should give you ideas for how to plan your split tests.
Prioritize tests based on value and cost.
Start with high-value, low-cost testing ideas. An example is testing variations in a checkout process step that shows
significant abandoment rates compared to previous steps.
Pages that have the highest traffic volume are the most important for testing. You have probably identified many pages
that perform worse than you would like them to, but if they don’t have a high volume of (expensive) traffic, don’t count
them as priorities.
You need pages with high traffic for completing your experiments within a reasonable timeframe. Pages with more than
30,000 monthly, unique visitors can reach statistical significance in a few weeks.
With a lower level of traffic, you need to run tests over a longer period of time. Because tests on high-traffic pages finish
sooner, you can move on to the next tests faster, which will speed up your optimization process.
Look for:
1. Most-visited pages. Look for information on unique visitors. When looking at the number of overall pageviews,
your conversion data will get skewed.
2. Top landing pages. When looking at the most-visited pages, you will see the most popular landing pages on your
site. You also need to look at the most-visited pages that people see when they first arrive on your site.
3. Pages with expensive visits. When choosing between two pages with similar traffic and conversion rates, pick the
one with a higher cost of traffic for a better split-testing ROI.
CXL created their own prioritization model that weeds out as much subjectivity as possible. It’s predicated on the
necessity of bringing data to the table. It’s called PXL and looks like this:
Grab your own copy of this spreadsheet template here. Just click File > Make a Copy to have your own customizable
spreadsheet.
Instead of guessing what the impact might be, this framework asks you a set of questions about it:
● Is the change above the fold? Changes above the fold are noticed by more people, thus increasing the likelihood of
the test having an impact.
● Is the change noticeable in under 5 seconds? Show a group of people the control and then a variation(s). Can they
tell the difference after seeing it for 5 seconds? If not, it’s likely to have less impact.
● Does it add or remove anything? Bigger changes like removing distractions or adding key information tend to have
more impact.
● Does the test run on high-traffic pages? Relative improvement on a high-traffic page results in more absolute
dollars.
Many of the variables specifically require you to bring data to the table to prioritize your hypotheses.
Having weekly team discussions on tests with these four questions will quickly make people stop relying on opinions.
There are also bounds based on the estimated time to implement (“ease of implementation”). Ideally, you’d have a test
developer as part of prioritization discussions.
Grading PXL
This is a binary scale. You have to choose one or the other. So, for most variables (unless otherwise noted), you choose
either a 0 or a 1.
Certain variables are also weighted because of their importance—how noticeable the change is, if something is
added/removed, ease of implementation.
Step 3: Test.
The reason for the in-depth analysis techniques described above is to get a clear understanding about what is happening
on your website. That process enables you to find areas where A/B testing can bring the best results and to plan
accordingly.
Now let’s set up your test.
A hypothesis defines w
hy you believe a problem occurs. If your problem is high abandonment rate in your checkout
process, your hypothesis may be that “People start questioning our worth after seeing grammar mistakes.”
After you define your problem and articulate your hypothesis, you can come up with specific split-testing variations.
Clearly defining your hypothesis will help you create test variations that give meaningful results.
In this case, we would summarize the benefits that the newsletter member would get from joining the newsletter. Even if
the original version works better in your A/B test, you learned something about your visitors. You clearly defined why you
did the test and can draw conclusions based on the outcome.
A good hypothesis…
Ignoring statistical significance when running an A/B test is worse than not running a test at all. The results will give you
confidence that you know what works for your site when, in reality, you don’t know more than before running the test.
If we were to toss a coin 1,000 times, we would reduce the influence of chance but still get slightly different results with
each trial.
The statistical significance of your experiment should be over 95%. But statistical significance alone is not a guarantee. It
is a measure of confidence.
At the end of the day, revenue is what matters. An increase in conversion rates may sometimes mean a decrease in
revenue if you’re tracking the wrong indicators.
Let’s imagine that you’re selling watches online and suddenly increased your prices by 25%. Your conversion rates will
probably drop, but your overall revenue may increase if the demand for your product is high enough.
You should always use analytics and customer feedback to plan your tests. But for a general guideline, here are elements
that historically have given good results:
“Never stop testing, and your advertising will never stop improving.” -David Ogilvy
This is the one rule of marketing that surpasses all others. No matter how well your landing pages or emails may be doing,
they can always be doing better. By not having a goal for constant improvement, you’re leaving money on the table and
letting your competition get ahead of you.
Not all split tests will be successful. You are doing well if 20% of split tests improve your conversion rates. Split-testing
will simply give you a new baseline to improve upon. You will start seeing considerable website performance
improvements after a number of different split tests.
Considering how competitive today’s online market is, if you aren’t constantly tracking and optimizing, you’re going to get
left behind—and outsold—by people who are.
A/B testing plan considerations
Most A/B testing is done one variable at a time. You test headlines, button text, images etc. These variables are simple to
test; your results will be clear; and your next steps will be obvious. By isolating one variable, you can be more confident in
your results. There’s a downside, though.
The argument against testing single variables is that, if you continue to do it for a long period of time, it will be impossible
for you to arrive at a much better design.
Instead, you’ll be improving your existing design in small increments and get stuck at the local maximum. You’ll hit a glass
ceiling in your design and, without a big change, be unable to earn larger gains.
Protect SEO.
Google openly endorses split testing because the goal is to improve a website and make users happy. But there are a few
things you should consider.
Don’t run your experiments longer than necessary. If your experiment has been running longer than what Google would
normally expect from a split test, you may confuse the search engine about which version of the content is the “real”
version.
The general rule is to update your site with the winning variation as soon as you are sure of its statistical validity. Google
wants to prevent people from deceiving search engines.
Use rel=canonical. Instead of using a noindex meta tag on your variation page, use rel=canonical. If you are testing two
variations of your homepage, you don’t want search engines to deindex both.
You need search engines to understand that your variations are what they are—variations of the original URL. Using
noindex in a situation like this may create problems later.
Use 302 instead of 301. When you need to redirect a visitor to one of you variations, use a 302 temporary redirection
instead of the permanent 301 version. You want Google to keep the original URL in its index.
A/A testing validates your test setup—if your variations display correctly and your software reports the right numbers. The
biggest problem with A/A testing is that setting it up and running it takes time (and traffic) that could be used for more
split testing.
:The volume of tests you start is important, but even more so is how many you finish every month.” - Craig Sullivan
It’s quicker for you to test your experiments before going live. To make sure your variations are displaying as they should
in different browsers, use cross-browser testing. For checking the numbers, use your split-testing and analytics packages
together on every test.
If, for whatever reason, you still need to do an A/A test, consider an A/B/A test, or 25/50/25 split, instead. There’s an even
better way: segmentation.
Besides significance, the other common problem A/B tests have is quoting percentages. Since conversions are measured
in percentages, there are two ways to report them:
Normally, Version 2 is used when reporting conversion rate improvements, but make sure you know this before deciding
anything based on the numbers.
Your A/B testing efforts will bring better returns when you execute them within a conversion optimization framework.
Different agencies have come up with different versions for this. Here are some of them:
The Conversion Rate Experts Methodology
Conversion Rate Experts have developed a systematic process for guiding a business through a series of steps toward
better conversion rates.
Building To The Ultimate Yes by MEClabs
MEClabs has a funnel that represents a series of decisions taken by the prospective customer. The idea revolves around
the customer taking small decisions step by step, “micro-yeses,” which lead to the Ultimate Yes, or the sale.
The idea is to prioritize conversion testing opportunities and implement the right experiments, which will drive the most
impact-filled results. This will help you maximize your conversion rate improvement.
ResearchXL by CXL
CXL has developed their own framework that’s used by dozens of agencies. It includes six parts of research and also has
a prioritization component, solving the oh-so-common problem of what to test.
1. Heuristic Analysis;
2. Technical Analysis;
3. Web Analytics Analysis;
4. Mouse Tracking Analysis;
5. Qualitative Surveys;
6. User Testing.
Mobile considerations
Visitors interact with their mobile devices through tapping, so their ability to quickly locate and hit these tapping areas is
key to mobile conversion rates.
When testing on mobile screens, begin testing elements related to the user experience and design of the
site—navigational elements or m
obile forms.
Mobile analytics generally fall into two categories: tracking how people interact with websites in a mobile browser and
in-app analytics. Let’s briefly touch on both.
● Call-to-action buttons. On a desktop screen, the CTA button won’t take up more than 2% of the screen real estate.
On a mobile device, you have the opportunity to make your CTA button take 25–50% of the screen real estate. You
can test any standard property on your button size, copy, and visual design.
● Navigation. You need to change the site navigation for mobile viewers. This is a good opportunity to direct visitors
to the most important page content. Think of mobile navigation as a tool for helping your visitors achieve their goal,
and remove anything that can get in the way.
● Copywriting. Since the screen real estate of mobile devices is small, focus your visitor’s full attention on your value
proposition. Test the wording carefully.
● Forms. Experiment with the design and number of fields, dropdowns, and error messaging.
Google Analytics has a mobile section, but it includes both mobile phones and tablets. Accessing a website on a tablet is
a different experience from a mobile phone, so make sure you use extra filters for cleaner data.
Compare your site performance (visit duration, bounce rate, conversion rate, etc.) for mobile visitors vs. desktop visitors
for insights into areas that may cause trouble for your visitors.
Note: Google Analytics tracks visitors only from browsers that support JavaScript. There are additional analytics solutions
available for tracking traffic from non-supporting browsers.
Use a previously set advanced segment (where you’ve filtered out non-mobile traffic) and apply that to your All Pages
report. You’ll get a good overview of which pages your mobile visitors use, which may be vastly different than those
viewed most often by desktop users.
This information can be helpful to test mobile menus. You can bring out the content your mobile visitors want to see the
most, creating a different navigation menu from your desktop version.
2. Testing in applications
Google Analytics has a separate tool for measuring and testing variations in mobile applications. The main difference
from testing in a browser is that, without putting in some thought, you’ll have a slower reiteration cycle.
You need to upload every test to the app store and wait for your users to update their apps. This means you should
carefully think through what you want to track and test in your application before you release it. It will be more difficult to
change this after your app has been downloaded.
When you perform a multivariate test, you’re not testing a different version of a web page like you are with an A/B test.
You’re performing a far more subtle test of the elements inside one web page.
A/B testing usually involves fewer combinations with more extreme changes. Multivariate tests have a large number of
variations that usually have subtle differences.
A/B testing is usually a better choice if you need meaningful results fast.Because the changes between pages are starker,
it’s easier to tell which page is more effective. A/B testing is also better if you don’t have a lot of traffic to your site.
Because multiple variables are tested together, multivariate testing needs a site with a lot traffic.
The goal of multivariate testing is to let you know which elements on your site play the biggest role in achieving your
objectives. Multivariate testing is more complicated and is better suited for advanced testers. It’s more prone to reporting
errors and common to have more than 50 combinations.
Value proposition:
Citycliq increased their conversion rates by 90% after experimenting with Value Propositions
Citycliq was testing their value propositions to see what converts the best. Eventually, they concluded that the value
proposition with the “Purest, most direct representation of their product” was the winner.
Call to action:
During the election period, the Obama team ran several A/B tests on the campaign’s landing page. The goal was to get
people to sign up with their email addresses. A/B testing generated an additional 2.9 million email addresses, which
translated to an extra $60 million in donations.
Images:
The challenger page increased the size of the contact form. They also replaced a general logistics image with the image
of an actual DHL emplyee.
A/B testing resources
Testing tools:
Instead of listing all of the various testing tools out there (and by now, there are many), take a look at our g
igantic list of
CRO tools. Each tool is reviewed by an actual practitioner, so you can choose which one is right for you.
Supporting tools
A widely used tool for calculating the significance of your A/B testing results.
The calculator allows you to calculate the maximum duration for which your test should run.
Heatmap software for tracking visitor behavior on your site. You can get good data for hypotheses generation.
Surveys
Qualaroo
This is for surveying visitors that are currently surfing your site. We use this in our agency. Allows you to set up
behavior-triggered surveys to find out why your customers are behaving the way they do.
SurveyMonkey
SurveyMonkey has been around for a long time and offers a stable experience. It has good analytical tools for analyzing
your responses.
Google Docs
Free and easy to use. Data is collected into spreadsheets which makes the data easy to analyze.
Typeform
You should always integrate A/B testing into a larger conversion optimization framework for good results. In the end, the
testing is all about knowing if your hypotheses are right and if your conversion plan is on the right course.
The main reason for doing split testing is to maximize the conversion potential of your website. It makes sense to invest
in conversion rate optimization before spending money on large-scale advertising campaigns.
After a while, your conversion funnel will be effective enough that you can transfer your winning campaigns to other
media. Because your core is optimized so well, by the time you go to offline media, your campaigns will convert well
enough to pay off.
1. Make sure you get actionable data from Google Analytics. To make use of data, you need to have goals first.
Define your target metrics.
2. Use qualitative surveys. Besides knowing what is going on, you also need to know w
hy.
3. Segment. It’s the only way to get valuable information.
4. Prioritize. Test pages with higher potential first.
5. Form a hypothesis. A/B testing has to start with a clearly defined hypothesis.
6. Statistical relevance. Never stop split testing before reaching a significance percentage of at least 95%.
7. Never stop testing. No matter how well your landing pages or emails may be doing, they can always do better.
8. A/B testing is part of a conversion rate optimization framework. Pick a framework and work on it constantly to
maximize your site’s potential.
Enjoyed this article? Join 95k+ analysts, optimizers, digital marketers, and UX
practitioners on our list and get more of this straight to your inbox.