The Ultimate Guide

to A/B Testing

UNBOUNCE
www.unbounce.com
What’s in the ebook? What is Unbounce?
This guide will teach you everything you need to Unbounce is the DIY Landing Page platform for
know about A/B tesing, including the all important Marketers. Build high converting landing pages
“What to Test”. for your PPC, email, banner and social media
campaigns.
• Introduction to A/B Testing
Find out why you should be testing. • Build & Publish Landing Pages In Minutes
• Getting Started with A/B Testing Use our powerful editor to re-create your design
5 things all pages should have & a case study. from scratch, or use one of our templates for a
• What to Test head start.
Learn what and how to test page elements.. • 1-Click A/B Testing For Optimization
• Workflow Need to solve an argument with your boss?
Learn now to manage your testing workflow. Stop relying on assumptions and set up a test
• Dealing with Data experiment.
When to stop, and how to use your data. • Simple Analytics To Track Campaigns
Our stats are powerful yet simple. It’s all about
clicks, conversions and how well your campaign
Feel free to share this document. is performing.

START YOUR FREE 30 DAY TRIAL

Table of Contents
1. Intro 4. An A/B Testing Workflow

a. An introduction to A/B Testing 1. Brainstorming

b. What about Multivariate Testing? 2. Creating page variants
c. What is the Multi-Armed Bandit? 3. Running your test
4. Promoting your winner
5. Start a new test
2. Getting Started with A/B Testing

a. Building a control page

5. Dealing with the Data
b. Creating variants
c. What makes a champion a. How to deal with a failed test
d. A case study b. When to stop testing
c. What’s next?

3. What to Test 6. A&B Testing Glossary:

Terminology & Tips
a. Developing a hypothesis of what to test
b. Page elements to test
I. Headline VI. Colors
II. Call to Action VII. Forms
III. Images VIII. Social proof
IV. Video IX. Confirmation pages
V. Copy
The Language of A/B Testing & Optimization
To make sure that we’re all on the same page, let’s set some basic vocabulary:

• Element - a discrete unit on the page: a block of text, a form, a button, an image, etc.
• Page - a web page or landing page that is considered the control page for your test.
• Variation - a version of a page that has some changes made to page elements. Also referred to as a variant.
• Test - a hypothesis that one version of an element will change the conversion rate in a significant, hopefully,
beneficial way.
• Conversion - when a visitor takes a desired action on the page.

The most common means of testing to improve conversions online are A/B Testing (aka Split-testing ) and Multivariate
Testing (aka MVT), which we’ll talk about in more detail below.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 4

What is an A/B test?

When you run an A/B test, you compare one page

against one or more variations that contain one
major difference in an element of the control page.
After a set amount of time, or visits, you compare
the results to how the change affected your results.

A/B testing is actually a nickname for a variety of

testing that involves single element changes across VISITORS RANDOMLY DISTRIBUTED
multiple variations, so it might actually be A/B/C/D VERSION VERSION
testing, but it is still called A/B testing and if you
see A/B/n or Split Test that is just a more accurate
expression of one control with many variations.

Every visitor will see one version of the page or

another, and you’ll measure conversions from
each set of visitors. A/B tests allow you to test
one version of copy, images, forms etc. against
another. Layout choices such as where you place the
elements can also be tested, or functionality choices
like when and how you show error messages. VERSION VERSION

VERSION B IS BETTER THAN VERSION A

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 5

How Much Traffic do You Need for a Valid Test? and customers in a controlled environment. Once
you have seen the results of a test you can take the
A common question is how much traffic you need new knowledge to your site wide design.
when running a test. This will depend on a few
factors. If you can run 100-200 visitors to your One of the best things about A/B testing is that it can
pages, you can sometimes make gut decisions be very fast and dramatic. It is also less complicated
about pages that are obviously underperforming and to set-up and reach significant results in order to
initate a new test. But generally you’ll want to wait make informed decisions.
for about 1,000 total visitors and make sure you
run the test for a week to cover daily variances. Advantages of A/B Testing

Use A/B Test Results for Major Site-Wide Decisions • They are fast
• Advanced analytics can be installed and
Split tests can also be used to temper drastic evaluated for each variation (e.g. click
changes like redesigning a homepage by only tracking, heatmaps, etc.)
serving it to a portion of your visitors. Companies • Can achieve more dramatic conversion rate
like Twitter and Facebook use this strategy to test lift results
major interface changes by only rolling out the new • Requires less traffic
version to a segment of their visitors and measuring
how that group reacts. Disadvantages of A/B Testing

A/B test results are influential in making good • More dramatic failures
decisions in site redesign because they help you see • Less specific understanding of element effects
what elements are important for your audience

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 6

What is a Multivariate test?

When you run a multivariate test you use one Advantages of MVT
page and dynamically supply multiple versions of
multiple elements. For instance, testing 3 versions • Better measurement of element interaction
of your headline, 2 versions of your call to action, • Failures are less dramatic
and 3 button colors simultaneously. That would give • More comprehensive testing
you 18 versions (3-headlines x 2-CTA x 3-button-
color) of the page that you are testing. Disadvantages of MVT

As you can see, with each successive element and • Can require many more variable combinations
variation on an element you multiply the number of to be run than A/B
versions that you are testing. This means that the • Requires vastly more traffic to reach statistical
amount of traffic required to reach confidence significance than A/B
increases. If you are testing 2 versions of 3 elements • Major layout changes are not feasible
you have 8 variations (2 x 2 x 2) and if you have 3 • Less layout change is possible
versions of 3 elements you have 27 variations (3 x 3 x • The restrictions of the test setup constrain
3), adding extra tests adds up very fast. marketing creativity
• Dynamic elements decrease the effectiveness
The purpose of multivariate testing is to refine of 3rd-party tracking tools
pages that already perform well by looking for • Less layout change is possible
compounding effects between elements. • The restrictions of the test setup constrain
marketing creativity
Multivariate testing is something that you should • Dynamic elements decrease the effectiveness
only be looking at if you have large amounts of of 3rd-party tracking tools
traffic.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 7

What is Multi-Armed Bandit Testing?

Recently a new type of testing has appeared on the There are varying opinions on whether this type of
scene. It’s called the multi-armed bandit. This type testing is actually successful. Some people like Visual
of testing is essentially the same as an A/B split test Website Optimizer say that in the long run multi-
and works very much the way that Google Adwords armed bandit requires a large conversion difference
ad delivery works. It starts with an experimental to be successful.
phase where it gives an even split of traffic across
all versions of the page and then it goes into an Why A/B testing is the right answer
exploitation phase where it give more traffic to most of the time
the version that is currently performing the best.
Part of why we use A/B testing is because it is
Over time if the winner starts to decline in less complicated for marketers that use it and is
conversion the system automatically gives generally more fruitful as a testing method. Chris
traffic to another version. The drawback to Goward at Wider Funnel says that they do 8-10
automated systems like this is that they can A/B tests for ever 1 multivariate test that they do
make counterproductive decisions, especially if for clients, and they only do MVT on pages that
they have a short experimental phase or choose get a very large number of unique visitors. Most of
a conversion event that is not on target for your the time A/B split-testing is the best answer for
bottom line needs. improving conversions and testing hypotheses, so
we want focus on that.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 8

Getting started with A/B Testing
This is one of those things that is hard for a lot of people. How do you start? The simple answer is to make one page
detailing the simple facts about your product or service. These simple facts are the features of your product. These
features have consequences. Those consequences are benefits. Secondly, ask yourself why are you better than the other
options (the competition) for solving the problem? Is it a lower price, better shipping, better return policy, etc?

Online you have two primary ways to communicate information: image and copy.

All good pages are going to cover the following 5 things:

1. A headline with a Unique Value Proposition

2. A demonstration of the product or the benefit - preferably in the context of how it would be used
3. A clear value for the viewer
4. Support of value
5. One action

Your action and your headline are the only things that need to be copy driven, everything else is wide open. When you
create your control page, make sure all 5 of these elements are represented. If you have done this you will have a solid
foundation for testing.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 9

Building a Control Page

Control pages are your first step in creating a

generation of tests. Your control is the parent and
variations are the children. The winning variation
becomes the next generation’s control for further
variation.

A good place to start your control page is a

wireframe. Take a look at KissMetrics’ Landing Page
Blueprint (seen on the right).

But, don’t let that stop you from using your own
vision in how you layout your elements. Just
because a certain structure is popular doesn’t mean
it’s right for you, also, standing out can be a very
effective strategy for good results.

Once your basic structure and content is in place

you will have a starting point for tests. Good
structure will make a world of difference in your
testing success, because it will make each change
less time consuming.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 10

Building Page Variations

Each variation should have one hypothesis of into designing your test, if you are communicating
change: color of background, different major image, to the wrong audience, you will acquire the wrong
different CTA in the headline, different CTA in the customers or leads. Keep an eye on what your new
action button, etc. It doesn’t matter how many customers or leads do, and if they’re not the right
hypotheses you have, but you will always get the ones, run a new test with different messaging to
best data by only having one hypothesis per be more inline with your most important business
variation. goals.

Once you get used to how your target market A good technique for tracking your test
behaves you may be able to get away with a multi- performance is to keep a record of your
hypothesis test, but when you start out stick to a hypotheses and your results, so you know where
framework that maximizes test cleanliness. you’re going right and where you went wrong. That
will make your next test better, and serve as a record
for stakeholders, as well as yourself.
What Makes a Test Champion
Note: You should use descriptive experiments
In simple terms your champion is the page variation names like Form Length 2012 Catalog or Call To
that converts the best. Once you’ve crowned your Action: Registration, something that makes the test
champion though you’ll want to dig a little deeper parameters apparent, not Steve’s Test or PPC Test.
and gauge the quality of your propects too; The
reality is that no matter how much time you put

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 11

A Case Study for A/B Testing

After the launch of the Noob Guide to Online Marketing we made a landing page that used PayWithATweet.com to help
expand the distribution of the guide, which averaged a 13% conversion rate. We then decided to see if we could get
better results from the landing page.

We had gathered some data by inserting a KISSinsights (now Qualaroo) widget on the page that suggested many people
(about 45%) would rather provide their email to get the PDF download, so we decided to see which would convert
better, “pay with an email” or “pay with a tweet”. In addition, some feedback indicated that it was fairly common for
people to pay with a tweet and then immediately delete it from their stream.

Our Test Hypothesis

Will allowing visitors to download the PDF by providing their email address perform better than receiving
it in exchange for a tweet? Considering that not everyone has a Twitter account, or is willing to share such
information with their followers.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 12

 Pay with a Tweet Averaged Pay with e-mail averaged
an 18% conversion rate. a 22% conversion rate!

Moving on from this test we’ve decided to discuss a possible hybrid option that might produce the best of both
worlds; A single version that lets the user decide whether they want to pay with a tweet or provide their email.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 13

What To Test
Developing a hypothesis of what to test

The first step in any A/B test is to create a hypothesis for what will make the page perform better. There are two very
common approaches: one revolves around data, the other involves people.

The data approach uses statistical data like bounce rate, exits, conversion starts, and funnel analysis. Data driven
hypothesis use tools like Clicktale, Google Analytics, Omniture, and CrazyEgg.

The people approach employs tools like Qualaroo (formerly KISSinsights) and SilverBack to gather user feedback. These
are all about addressing the quality of the experience on your site or specific pages.

Your most effective hypotheses will come from combining both the data and people approach.

On the data side you may see something like a high bounce rate and know that there is something wrong. You might
look at time on page too, since this can be a good indicator of how people are interacting with your page. If you have
a high bounce rate and a high time on page you probably have a problem near the bottom of the page, maybe the call
to action button. If you have a high bounce rate and a short time on page it means that you have something repelling at
the beginning of the page, maybe your headline.

If you cannot determine from the data approach what is it specifically that is hurting your conversion rate, you should
then use the people approach to gather direct user feedback to gain insight.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 14

So What Should You Test?

The Headline
Your headline is your first impression. Its success is dictated by how closely it matches what your viewer expected when
they made the decision to visit your page - whether from an ad, banner or email link etc.

You can try testing positive versus negative language in the way you express your value in the headline, for example:
Save Time By Downloading Now vs. Stop Wasting Time, Download Now.

People are impatient and will read your headline very quickly, so communicates your core value proposition in a
way that makes it really obvious what your offer is.

To assess problems with your headline try a 5-second test: where you flash the page in front of a person unfamiliar with
your brand for 5 seconds and ask ”What is this page about?” If they don’t know, it’s not clear enough and time to revisit
the messaging.

Headline Test Ideas:

• Long vs. short
• Positive vs. negative
• Benefit oriented vs. feature oriented
• Single headline vs. primary and secondary supporting copy
• Enhance the contrast to make your headline more visually powerful, to focus people’s attention on it

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 15

Images
Since images can convey complex ideas and emotions quickly they are a phenomenal source for testing.

This is one of the few areas that has clear best practices. Make sure you use high quality images ie. no pixelation or
blurriness. If you use people, have their gaze directed to your call to action, and choose images that enhance the
understanding of your offering.

It’s also been shown that image annotations are one of the first things people read on a page, so make sure you write
something compelling right below your primary image.

The most basic image tests will be:

• Color vs. black and white

• People vs. product
• One image vs. many images
• Masculine vs. feminine

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 16

Video
Using a video on your pages can have a dramatic 4. Directional cues within the video: Point your
effect on engagement and your conversion rate. But visitors at your landing page’s conversion goal.
what do you test? This can be a physical action (a person pointing)
or a verbal instruction.
1. Autoplay vs. press play: Usability guidelines
advise against autoplay as it is an interruption 5. Image vs. Video: Try a static image in one page
technique that annoys people and makes them variant and a video in the other.
click the back button. However, it can also
increase conversions. You’ll have to weigh up 6. Spokesperson: Not everyone is a natural speaker,
the potential negative impact on your brand’s and maybe your audience doesn’t want to listen
perception versus the potential for increased to the CEO pitching them. Find someone who
conversions. comes across as natural and likeable.

2. Calls to action (CTA): This can be permanently 7. Number of videos: If you are using them for
visible, or be shown at strategic times throughout testimonials, you might want to consider having
the video, or be presented at the end. more than one to increase the trust factor.

3. Length: Try short and long versions to see what 8. Lo-fi vs. Hi-fi: Professional videos are good
your customers need. Some products may need at communicating your focus on quality, and
a detailed study, while others might be better are usually the best way to go. But with a tight
suited to a 30 second “commercial” style. budget and even a camera phone, you can create
a convincing and authentic experience.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 17

Some Example Tests - Video

In this case study by VidYard they show how they got a conversion lift of 100% by using video on their landing page.

In this second test, adding a video made version B increase conversions by 216%!

VERSION A vs. VERSION B

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 18

Copy
The typical rule of thumb is to keep your pages short and concise, but don’t dismiss the long form either - it’s been
proven to convert - but typically only for the sales letter type of page.

Whether you have 5 paragraphs or just a set of bullets it is important to focus first on benefits and then support
with facts and function.

A good way to judge the effectivness of your copy (if you can’t tie your tests success or failure to it), try reading it
aloud to see how it sounds and flows. Even better, get a coworker to read it to you and see how much you cringe.

If you find that you are writing a lot of copy on your landing pages you should utilize the famous advice of Steve Krug,
“Remove 50% of your copy, then delete half of what’s left.” Being overly verbose can hurt your conversions if your
message can be described adequately in the short form. Again, test both versions to see which resonates best.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 19

Forms
The majority of landing pages include some kind of form to capture data from your visitors. To make them effective you
need to be offering something in exchange for the data. Examples would be an ebook, whitepaper or webinar registration.
But what do you test on the form? What you’re trying to do is balance your need for data with the “size of the prize”
(what you’re giving away) to find the optimal point of conversion.

Some things you can test on your forms:

• The number of fields

• The relevance of the fields to what you’re giving away

• The design of the containing element for the form (e.g. a box that encapsulates it)

• Required vs. non-required fields (the test below examines this)

• The position of the form (on the right or left hand side of the page)

• Adding directional cues that point to your form

• The form header (a description of what you’re asking the visitor to do that matches closely what you say in your
CTA)

• Placing a privacy policy link next to an email address field to improve trust
(make it pop up in a lightbox to prevent people leaving your page)

• Splitting your form over more than one page

• The button copy (discussed next)

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 20

The Call to Action (CTA)
Your call to action IS your conversion. It’s what you An Example Test - Call to Action
want your visitors to do (and nothing else). If people This example from our blog shows a series of case
aren’t converting your call to action may be in need studies on how to write CTA’s that convert including
of updating. a simple one (shown below), where they achieved a
38% lift in conversions by changing a single word.
CTA’s should be very descriptive, describing exactly
what will happen when clicked. A good tip when
writing your CTA is to finish the phrase “I want to...”,
for example:

I want to...
• Read More
• Subscribe to the Newsletter
• Download the ebook
• Get Free Shipping
• Upgrade Now

Strong CTAs should reinforce the line of thinking

introduced in your headline, content and imagery,
effectively ending the story of your page with a solid
“what to do now”.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 21

Colors
This is a contentious one. Some people insist that An Example Test - Color
the color of your buttons is highly influential in Going with the first idea, that button color does
creating desire to act; Green for go, blue as the have an impact, read this case study over at Hubspot
standard color of an interactive web element, or that shows how the red button outperformed the
orange/red to illicit an emotional reaction. green button by 21%.

Others say it’s not relevant and the more important VERSION A vs. VERSION B
aspects to consider are the contrast of the button so
that it stands out clearly from the rest of the page.
This is why A/B testing is so useful, because you can
remove the conjecture and find out if it will make a
difference with your audience.

Also remember that many people are color blind,

especially men, so if you have a male demographic
you will want to focus more on contrast rather than
color.

Just remember, that more often it’s the text on the

button that is going to have the most dramatic
effect on conversions, as described above.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 22

Social Proof & Trust
People are like sheep in many ways. They follow Other types include follower counts from Twitter or
the herd and like to join in with what others are Facebook. You can also show social sharing buttons
doing. To illustrate this, imagine walking down a to illustrate how popular the page is - but it’s often
street and seeing one person staring up at the sky. better to place them on your confirmation pages
You probably wouldn’t give it a second thought. so that your main page retains it’s ability to have a
But if there was a crowd of 30 people all staring at single CTA (we’ll get into more detail in a moment).
the same thing, you’d quickly divert your attention
to see what was going on. That’s social proof and it Below, you’ll find some things you can test on
builds trust in your visitors, helping conversions. your page, and the ways to test them would be:

So how do you add social proof to your landing • How many you use - one main
pages? testimonial vs. five
• Position on the page - is it more important
There are many types of social proof you can to establish trust right away or can it be an
include, such as: afterthought at the bottom of the page?

• Testimonials
• Endorsements: Logos of important press
placements, and customers with well known
brands
• Social influencers: A great example is to
show the number of registrants attending a
webinar.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 23

An Example Test - Social Proof & Trust
This example, run on VisualWebsiteOptimizer, the use of testimonials in version B resulted in a conversion lift of 34% for all
organic search traffic arriving at the page.

VERSION A vs. VERSION B

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 24

Post-Conversion Confirmation Pages
This is my favorite and probably the most overlooked area for testing. Confirmation pages appear after the completion
of a lead gen form or after completing a shopping cart checkout on an ecommerce site.

The reason it’s such a hot spot is that it’s the first page a customer sees after converting, meaning that they have invested
trust, time, personal data or money with you. So you need to take advantage of this short window of opportunity to try
and get a little more out of them.

Things you can test placing on your confirmation pages are:

• Social sharing widgets: As mentioned earlier, by placing them here you are removing the clutter from the main
page.
• Newsletter subscription: Ask them to subscribe to your newsletter (and explain the benefits of doing so, i.e. what
they’ll get: tips, advice, discounts etc.)
• Suggested products: Follow the Amazon model, for ecommerce confirmation pages try adding the classic “If you
liked that, you might like this.”
• A bonus prize: If you are trying to establish yourself as a thought leader by giving away content, consider adding an
extra free PDF download, it gives them extra incentive to spread the goodwill via word of mouth recommendations.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 25

An Example Test - Post-Conversion Confirmation Page
WiderFunnel did a test for TourismBC on their confirmation pages and increased engagement by 44% with the interactive
elements on the page.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 26

A/B Testing Workflow
Whether you are an individual marketer wearing A/B Testing Workflow
many hats or a fully fledged team, you’re going to
have a similar workflow. Step 1 - Brainstorming
All members participate in brainstorming and
Next is a sample workflow you can use to create, discussion of potential elements. The designer
optimize and test your pages, and who is responsible creates major element assets and builds a control
for each step. page layout with filler copy and every page element
you think might be needed to covey your message.
If you do have a team, it would look something like
this: Step 2 - Creating page variants
The optimizer and marketer then each take an
• The Marketer (for social, email, PPC etc) independent stab at coming up with page variations
• The Copywriter they think will convert best, removing elements and
• The Optimizer working with the designer and copywriter to help
• The Designer with messaging and layout changes.
• The Campaign Manager

Step 3 - Running your test

Each has a different role to play in the process of The test is launched and traffic is driven to the page
running a successful marketing campaign that by the channels chosen by the marketer. Meanwhile,
involves A/B testing. the campaign manager will be coordinating the
team and any external contacts if there are co-
marketing opportunities involved.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 27

Step 4 - Promoting your winner
Once you have identified the winning page variation,
it will be promoted to be the new control page (and
receive all of the inbound traffic) and any side bets
placed about who was going to make the best page
will be settled (makes it more fun).

Step 5 - Start a new test

Add, rinse, repeat. It’s now time to document your
test results and begin work on a new hypothesis
for your next test. Remember testing should be an
ongoing series of experiments.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 28

Dealing With Data
The purpose of A/B testing is twofold: improve the results of specific pages or campaigns, and derive insights to
improve your future campaigns.

Your process of dealing with the data you glean from your tests will be something like this:

I understand the results I don’t understand the results

Conversion Woo Hoo! You are a genius start Really? Ok, do a test to
Improvement your next test. figure out the next step.
(Pass) (Pass, kinda)

Conversion Good save. Stop the test and try Seriously, a kitten just died
Decline a new hypothesis. because of you.
(Pass) (Fail)

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 29

How to Deal With a Failed Test between the variants, make a gut-check call and
remove the poorer performing pages. If you can’t
The bottom line with data is that tests don’t fail, see an obvious winner, keep the test running until
analysts and test designers fail. The test can only it reaches around a 90% confidence level - this is
report data that is as useful as the question that when you’ll be able to make a decision based on
is being asked. There will be times when the data data.
is surprising, but if you ran a clean test then you
should be able to figure out what needs to be tested What next?
next to keep on track.
Just keep testing. Continue to make hypotheses
When Should You Stop Testing? and new experiments until you aren’t getting
acceptable improvement. Once you reach the point
Your end parameters should be defined when you of diminished returns you should start looking for
start your test. That means setting a minimum new sources of inspiration; A perfect example is your
sample size, and a length for your test. Generally customer support team, who are in direct contact
speaking (as mentioned earlier), you’ll want to push with your customers daily, and likely understand
at least 100-200 unique visitors through each page their needs better than anyone.
variant and ensure the test runs for 1-2 weeks to
account for day and time variations in visitor and Finally, remember to have fun. A/B testing can be a
demographic behavior. great way to involve other members of your team
and to share in the success of their ideas.
When you’re nearing the end of a test, if your
statistical confidence level is still low, but you’re
seeing a decent difference in conversion rate

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 30

 Start A/B Testing Your Pages Now

http://try.unbounce.com/for-a-b-testing

UNBOUNCE Feel Free To Share This Document

www.unbounce.com
A A&B Testing Glossary: Terminology & Tips

A/B Testing

Let’s start with the big picture. A/B testing is a research technique that marketers
have leveraged for decades (yes, it’s older than the Internet). Start with one page
element like your headline or call-to-action, and split your inbound traffic equally
so the at each group sees a different version and you can discover which version
resonates most strongly with your target market.

A/B Testing Tip

Clearly define what you’re testing up-front as you’ll need numbers to
compare between your groups. Do you care about open rates, click-
through rates, conversions, none of the above, or all of the above? Hint:
there’s more than one right answer.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 32

A A&B Testing Glossary: Terminology & Tips

Anxiety

Fence-sitters are the hardest type of customer. They can’t make up their minds,
and unless you use user feedback tools you’ll never learn what’s preventing them
from converting. Visitor anxiety is created by barriers on your page, for example:
unclear messaging, pricing, long forms or a lack of trust.

Anxiety Tip
Adding social proof, testimonials and other trust inducing elements to
your page will reduce this anxiety. As an example, consider a lead gen
page that’s asking a lot of questions in exchange for an ebook. Providing a
preview lets the visitor see how good you and your content really are, and
will be more inclined to break through those anxiety barriers.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 33

B A&B Testing Glossary: Terminology & Tips

Big Idea Testing

If your brand is going through a major change (like if you’re changing your
website’s user interface), it doesn’t make sense to adjust one small variable at
a time. Instead, you’ll want to test everything at once to see the effects of all
elements bundled together.

Big Idea Testing Tip

When you’re dealing with big idea testing, you’re running an aggressive
experiment with probabilistic successes and failures. Make sure to weigh
the risk vs. reward of your test when running such an aggressive test.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 34

B A&B Testing Glossary: Terminology & Tips

Bounce Rate

Your bounce rate is the percentage of people who visit your page and leave
immediately, in A/B testing this is also often referred to as abandonment rate. This
is a great metric to use when justifying your time spent on A/B Testing. If your
bounce rate is high, no matter what you do to improve your product or service
offering, your bounce rate will never decrease until you start making changes (and
testing) to your page elements.

Bounce Rate Tip

If you have a high bounce rate and a high time on page you probably
have a problem near the bottom of the page, maybe the call to action
button. If you have a high bounce rate and a short time on page it means
that you have something repelling at the beginning of the page, maybe
your headline.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 35

C A&B Testing Glossary: Terminology & Tips

Click-through Rate

Click-throughs are the first steps to conversion. Measured as outgoing clicks/

unique visits, you’ll want this number to be as high as possible.

Click-through Rate Tip

Keep in mind that there are some nuances to this number. For one, you’re
not directly attributing clicks per person, as a visitor may click more than
once. You can also have multiple CTRs per page (hint: for each and every
link).

Look for rough numeric ranges to tell the story that you want. It’s unlikely
that one user is going to sit there clicking 100 times. Unless it’s you trying
to cheat on your test, which would be very, very bizarre.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 36

C A&B Testing Glossary: Terminology & Tips

“ The best design isn’t Call to Action (CTA)

the one that makes your Your CTA is what you want your visitors to do on your
page, whether completing a form or clicking through
company look cool and to an ecommerce shopping cart page. It’s the big
edgy and sophisticated. It’s shiny button that you want to focus all the attention
on.
the design that supports
conversion, has room for
CTA Tip
good copy and powerful Use the tips of conversion centered design
calls to action that make to focus people’s attention on your CTA (or
in the case of a form, the entire conversion
people click the big area).

orange button.”
Brian Massey
www.conversionscientist.com

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 37

C A&B Testing Glossary: Terminology & Tips

Champion

Champion refers to the control page in your test. The one you are testing against.

Champion Tip
An important thing to note here is that even though the Champion is your
main page in the test, as soon as it gets out performed (with statistical
significance – or personal gut-based confidence), you need to promote
the current leader to take it’s place. It’s just like any championship fight.
The winner needs to get all the attention (in this case, the traffic).

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 38

C A&B Testing Glossary: Terminology & Tips

Click-through Page

A click-through page is most often used middle man between your ad an

ecommerce cart page. It’s purpose is to expand on the intention of the ad and
warm up the prospect with enough information that they’ll be ready to click your
“Buy Now” button.

Click-through Page Tip

There’s no point in breaking your sales funnel by pushing traffic directly
from an ad to a cart without at least a little love. Remember the adage,
about kissing before sex. The same principle applies here.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 39

C A&B Testing Glossary: Terminology & Tips

Confidence Level

The confidence level is the probability that the measured conversion rate differs
from the champion page conversion rate for reasons other than chance alone.

Confidence Level Tip

If you’re a stickler for data, wait until the confidence level reaches 90-
95% – then you will have good data to back up your choice of the winner
of the test. Other times you’ll just know in your gut that a page just isn’t
working, in which case, just dump that variant and try again.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 40

C A&B Testing Glossary: Terminology & Tips

Conversion Rate

The percentage of unique visitors that come to your page and complete your
conversion goal.

Conversion Rate Tip

Never be happy with your conversion rate. Just remember that every
page can be better. That’s the whole point of A/B testing.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 41

C A&B Testing Glossary: Terminology & Tips

Control Page

Your control page is your (A) page. The one that existed first that you are going to
start running tests against. If you run a test and discover a better performing page,
it would be promoted to be the control or “champion” page.

Control Page Tip

It can be a good idea to puke a ton of content onto your control page,
so that you can make variants that remove and simplify the content. This
will give you an idea of the level of content your prospects need to be
convinced to convert.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 42

D A&B Testing Glossary: Terminology & Tips

Distraction

The first thing a distracted visitor will do is hit the back button, so don’t let them
escape from your page. Make it feel like a nicely contained unit.

Click-through Rate Tip

Often you’ll want to link to other pages or assets, such as a photo gallery
or a privacy policy page – or just a deeper explanation of a certain point.
At this time, you absolutely do not want them to leave the page. So
instead, try to launch lightbox windows that contain your supplementary
content. As an example, click the “Privacy Policy” link next to the email
address on the form on this page. This way you keep them on your page
and remove the risk of them wandering off.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 43

E A&B Testing Glossary: Terminology & Tips

Element

A discrete unit on the page: a block of text, a form, a button, an image, etc.

Element Tip
Page elements will typically be the basis for your A/B tests. Start with the
most important ones which typically follow this type of order:

1. Headline
2. Call to action
3. Secondary benefit statement
4. Hero shot image or video

Test one at a time for a clean test.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 44

E A&B Testing Glossary: Terminology & Tips

External Factors

Your A/B test can never be a fully controlled experiment. There are way too many
external factors that could skew results. Screaming babies. Lightening storms.
Internet outages. Or more realistically, inbound traffic that comes from an origin
you didn’t expect.

External Factors Tip

If you are doing PPC, you only want traffic to arrive at your page from
your paid ads to prevent contamination of your data. To help this, ensure
the page isn’t getting indexed by Google etc. so that it doesn’t appear in
organic results.

Extra Tip: If you want to send traffic from multiple sources: email, PPC,
social – create a different page for each so that you can figure out which
channel is performing best. You can also change the content on each
page to suit the origin.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 45

E A&B Testing Glossary: Terminology & Tips

Experiment

An experiment is just another way of saying “Running an A/B test”.

Experiment Tip
Don’t make changes to your pages during an experiment. If you come
up with a new hypothesis for something that might work, either add it as
a new variant to the current test, or choose the best performing variant
from the current test, make it the champion and run your new idea
against that in a new experiment. Remember to document the reasons
(hypothesis) behind all of your experiments.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 46

E A&B Testing Glossary: Terminology & Tips

Eye Flow

This is the concept that by presenting any imagery of people (or even animals)
looking in the direction of your conversion goal area will lead you to look there
too. It works – we’re like sheep.

Eye Flow Tip

If you are using a video on your page, engage the viewer by looking
them in the eye, then when the time is right, look and point towards your
form or button from within the video, and ASK them to complete your
conversion goal.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 47

F A&B Testing Glossary: Terminology & Tips

The Fold

The fold is a term from ye olden days, referring to what you could see on the top
half of a newspaper when it was folded. The assumption that it’s the area that gets
the most attention.

The Fold Tips

This became a big deal in the web world too with pages being longer
than screen heights. Lately though, due to several factors (mouse scroll
wheels and larger monitors and screen resolutions) the fold isn’t quite
as important as once thought. Here’s an interesting post debunking the
myth of the fold. One of the key takeaways from the post is that you
should design pages that imply there is more to be explored further down
the page.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 48

F A&B Testing Glossary: Terminology & Tips

Form Testing

Forms are the life-blood of any marketer in the business of collecting leads, and how you design and
layout your forms (and what you ask for in exchange is critical to your success.

Form Testing Tips

Some things you can test on your forms:

• The number of fields

• The relevance of the fields to what you’re giving away
• The design of the containing element for the form (e.g. a box that encapsulates it)
• Required vs. non-required fields
• The position of the form (on the right or left hand side of the page)
• Adding directional cues that point to your form
• The form header (a description of what you’re asking the visitor to do that matches closely
what you say in your CTA)
• Placing a privacy policy link next to an email address field to improve trust (make it pop up
in a lightbox to prevent people leaving your page)
• Splitting your form over more than one page
• The button copy (your CTA)

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 49

F A&B Testing Glossary: Terminology & Tips

Funnel

The sales or conversion funnel is the pathway your prospects follow from entry into
your marketing sphere to your end conversion goal. The obvious idea is to pour as many
people through this funnel as possible, getting them all to Conversion Land vs. Back
Button Avenue.

Funnel Tip
A great way to visualize what could be happening in your conversion process
(and develop ideas for testing) is to tell the story from start to finish of what
a potential custom goes through on their journey. A Pain Point Analysis is an
interactive story plotted on a chart that walks through the experience of the
user, assigning positive and negative points along their traversal of the funnel. As
soon as you hit a downturn in the experience graph you know you’ve hit a pain
point and can use this to focus your next testing hypothesis on improvements.

Usage: Click the “Continue” link in the top left of the page to see experience
graph tell the story. It ends with a green “success” hypothesis that is an idea of
how to fix the broken experience.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 50

G A&B Testing Glossary: Terminology & Tips

Garbage in, Garbage Out (GIGO)

A common example of GIGO in testing would be someone who changes more

than one element in one test. In essence you “might” be making smart decisions
for each element you test, but you’ll never know which change resulted a the
conversion improvement or decline. So you might actually want to call it “Ideas In,
No Information Out” (IINIO).

GIGO Tips
Avoid GIGO by ensuring that you’re running a clean test.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 51

G A&B Testing Glossary: Terminology & Tips

Goals

Goals can be categorized between business goals and those in analytics software,
which we’ll get to later. What are the goals of your business? Do you even know
(beyond making money). If you don’t, you’re likely not marketing correctly and to
the wrong people.

Know exactly what you’re trying to achieve, and design your pages with a
singular mindset to do just that.

You want your customers to complete your conversion goals, not you!

Goal Tip
Use congruence: This is the ideas that every element of your page is
geared to steering the visitor towards your intended goal. It’s all about
focus. Check over your page, and if you’re doing/sayig anything that isn’t
“in congruence” with what the visitor should be doing, either change it or
remove it.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 52

G A&B Testing Glossary: Terminology & Tips

Gut Instinct

Sometimes you just have to go with your gut.

Gut Instinct Tip

If you quickly see one of your test pages tanking soooo badly that it’s
obviously a total failure, get it out of there right away and stop wasting
valuable traffic.

Caveat: It’s sometimes worth keeping a little traffic going to that page
(maybe 20%) because you never know when you might get some surprise
results. I’ve seen it happen.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 53

H A&B Testing Glossary: Terminology & Tips

Headline Testing

Your headline is the most critical element on your landing page to test.

Headline Testing Tip

Drayton Bird, perhaps the most experienced direct marketer in the
world, suggests spending 80% of your time writing your headline.
(Source: KISSmetrics). Why is it so important? Message match, if it doesn’t
match what the “clicker” was expecting to see when they made that all
important click, then you’ve failed them. Shame on you.

A good exercise is to stick your ads and page on the wall and compare
the messaging, then refine it until it’s very clear that they are in concert
with one another.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 54

H A&B Testing Glossary: Terminology & Tips

Heatmaps

This is another tool that you can use to summarize user behavior and click
patterns on your site. See where people are scrolling and concentrating their
visual attention.

Heatmap Tip
See where people are spending their mouse time on your page, if it’s
around your CTA, bravo. if they spend an awful lot of time wandering
or focusing on a less important area, you may have written something
confusing.

Tools you can use for these types of studies include: Crazy Egg or
EyeQuant.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 55

H A&B Testing Glossary: Terminology & Tips

Hypothesis

A hypothesis is a statement aimed at solving an identified problem with your

page. It should be written in simple language with a problem and a goal. A good
example would be:

Our Test Hypothesis

Will allowing visitors to download the PDF by providing their email address
perform better than receiving it in exchange for a tweet? Considering that not
everyone has a Twitter account, or is willing to share such information with their
followers

Hypothesis Tips
Read this blog post to understand the concept in more depth.

Bonus Tip: Brainstorm with people from different departments to get

a fresh perspective: customer support (the front lines), a designer,
copywriter and an information architect. You’ll be surprised how much
untapped wealth exists in your company.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 56

I A&B Testing Glossary: Terminology & Tips

Iterative Testing

A/B tests aren’t a one time deal. You should continuously test your landing pages
so that you can make incremental improvements as you continue to move
forward.

Iterative Testing Tip

Document the reasons and results for every test, so when your boss asks
you why you did something 6-months later you can say exactly why and
what happened.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 57

J A&B Testing Glossary: Terminology & Tips

“ We listen to our gut, Jumping to Conclusions

then test what it says. Don’t do it. Give your tests enough time to reach a
statistically relevant level. The exception being the
We gather marketing gut-check mentioned earlier.
research, then test it.
We create best practices, Jumping Tip
then test them. We listen Patience is definitely a virtue here. It’s really
easy to make a quick assumption that your
to opinions, then test test is failing and ditch a variant that may

them. We hear the advice make a comeback and surprise you. It’s far to
easy to do and you’ll often be mistaken, this
of experts, then test it.” is why we test. #fairwarning

Brian Massey
www.conversionscientist.com

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 58

J A&B Testing Glossary: Terminology & Tips

Justifying A/B Testing

Justyfing testing to your boss, CEO or client can be a tough business.

Justification Tips
Use these two posts to learn how to convince anyone of the value of A/B
testing and conversion rate optimization.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 59

K A&B Testing Glossary: Terminology & Tips

Key Performance Indicators (KPI)

Valuable business data is more complex than just a click-through rate or bounce
rate. Some of the important KPI’s include:

• Visitor to sign-up ration (acquisition)

• Sign-up to paying ratio (adoption)
• Churn % (retention)
• Average value per customer

KPI Tip
Read this Onboardly post on vanity metrics for a better understanding of
data points that matter most.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 60

K A&B Testing Glossary: Terminology & Tips

Keywords

If you’re running PPC campaigns, one of your primary concerns will be message
match. This means you need to include important keywords or phrases in your
ads that match closely or exactly with those in the important areas of your landing
page (primarily the headline).

Keyword Tip
Don’t build your page entirely out of images. Use text so that the search
engines can parse their meaning and give you a decent quality score.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 61

L A&B Testing Glossary: Terminology & Tips

Landing Page

In the purest sense, a landing page is any web page that a visitor can arrive
at or “land” on. However, when discussing landing pages within the realm of
marketing and advertising, it’s more common to refer to a landing page as being
a standalone web page distinct from your main website that has been designed
for a single focused objective.

Landing Page Tip

Two tips:

1. Your landing page should have no global navigation to tie it to your

primary website. The main reason for this is to limit the options
available to your visitors, helping to guide them toward your
intended conversion goal.

2. Subscribe to the Unbounce blog.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 62

L A&B Testing Glossary: Terminology & Tips

Lift

The % difference between your control page and a successful test variant. Or a
British elevator.

Lead Generation

Lead gen is all about fairness. In simple terms, if you want $100 worth of data from
someone, give them $100 worth of useful business information. This is known as
balancing the “barrier to access” with the “size of the prize”.

Lead Generation Tip

Write great content with original and timely data or instructional
information and package it up in a PDF. Then offer it up to people in
different ways. In the most recent version we did here at Unbounce, we
let people choose to receive the ebook for their email address or by
tweeting about it.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 63

M A&B Testing Glossary: Terminology & Tips

Message Match

We’ve touched on this a couple of times already, but in case you’re not the type
to read a phone book from cover to cover, we’d better start at the beginning.
Message Match is a measure of how closely your ad text correlates with the
primary test of your landing page. If your ad says “Laptop repairs from $100″
and your landing page says “Computer fixed starting at $50″ you’re getting too
generic.

Landing Page Tip

If you’re running any kind of marketing campaign, start by creating the
ultimate landing page to sell your product or service, then (and only then)
write an ad to lead people to it using language that matches from start to
finish. Your landing page should look like a bigger extension of your ad.
And if you’re running PPC, you’ll get a higher Quality Score and your ads
will be cheaper.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 64

M A&B Testing Glossary: Terminology & Tips

Multivariate Testing

With traditional A/B testing, marketers typically focus on one clearly defined element. Multivariate testing refers to the
discipline of testing many variables simultaneously.

When you run a multivariate test you use one page and dynamically supply multiple versions of multiple elements.
For instance, testing 3 versions of your headline, 2 versions of your call to action, and 3 button colors simultaneously.

That would give you 18 versions (3-headlines x 2-CTA x 3-button-color) of the page that you are testing. As you can
see, with each successive element and variation on an element you multiply the number of versions that you are
testing. This means that the amount of traffic required to reach confidence increases.

If you are testing 2 versions of 3 elements you have 8 variations (2 x 2 x 2) and if you have 3 versions of 3 elements
you have 27 variations (3 x 3 x 3), adding extra tests adds up very fast.

Multivariate Testing Tip

Don’t do it unless you are driving a lot of traffic, otherwise you’ll be waiting a long time for
meaningful results.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 65

N A&B Testing Glossary: Terminology & Tips

Non-Conclusive Results

Most tests fail. That’s just an unfortunate reality, unless you’re a conversion god(ess).

Non-Conclusive Results Tip

Don’t try to force success when it’s not there. Ambiguity is totally okay.
Just begin the process of devising a new hypothesis again and keep on
testing.

And don’t panic. Here’s what to do when your test fails.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 66

O A&B Testing Glossary: Terminology & Tips

Open Rate

Many email service providers let you A/B test your open rate based on things like
subject line & delivery date/time.

Open Rate Tip

Services like MailChimp do this really well, where you get to A/B test your
headline etc. for a small percentage of your list, then the winner is what
gets sent to the remaining bulk of your recipients.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 67

O A&B Testing Glossary: Terminology & Tips

Oli Gardner

Hmm, what do I say about myself without sounding like a self-congratulatory

egomaniac? (FTR It wasn’t my idea to add me in here).

Oli Gardner Tip

Follow @OliGardner on Twitter :) I know stuff about A/B testing and
conversion.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 68

P A&B Testing Glossary: Terminology & Tips

Post-Conversion Strategy

This is my favorite and probably the most overlooked area for testing. Confirmation pages appear after the completion
of a lead gen form or after completing a shopping cart checkout on an ecommerce site. The reason it’s such a hot spot
is that it’s the first page a customer sees after converting, meaning that they have invested trust, time, personal data or
money with you. So you need to take advantage of this short window of opportunity to try and get a little more out of
them.

Post-Conversion Strategy Tips

Things you can test placing on your confirmation pages are:

• Social sharing widgets: As mentioned earlier, by placing them here you are removing the clutter from
the main page.
• Newsletter subscription: Ask them to subscribe to your newsletter (and explain the benefits of doing
so, i.e. what they’ll get: tips, advice, discounts etc.)
• Suggested products: Follow the Amazon model, for ecommerce confirmation pages try adding the
classic “If you liked that, you might like this.”
• A bonus prize: If you are trying to establish yourself as a thought leader by giving away content,
consider adding an extra free PDF download, it gives them extra incentive to spread the goodwill via
word of mouth recommendations.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 69

P A&B Testing Glossary: Terminology & Tips

Privacy Policy

Having a privacy policy is essential for landing pages that have a lead gen form. It
builds trust and for PPC it keeps Google from banning you.

Privacy Policy Testing Tip

Test the position. Many people put it in the footer, but I find it’s more
successful in providing that sense of trust when placed directly beside the
email field i your forms.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 70

Q A&B Testing Glossary: Terminology & Tips

Qualitative Research

This is a technique used to gather useful insights from your target market and/or
existing customers. It’s based on opinion and needs to be considered carefully (e.g.
don’t listen to one person, listen to 20 and develop your insight from the bigger
picture.

Tip
When developing ideas for your next A/B test, use tools like live chat
(Olark), or surveys (Qualaroo), so you can figure out where you’re going
wrong.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 71

Q A&B Testing Glossary: Terminology & Tips

Quantitative Research

This is where you dig through your analytics to find extra insight as to where, when,
and why people are moving through your site/page and converting (or not), using
purely statistical analysis.

Quantitative Method Tip

Hire a Google Analytics expert for a day session and get all of your goals,
funnels and reports set up, so that you’re ready report to the higher-ups
at the drop of a hat.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 72

R A&B Testing Glossary: Terminology & Tips

Return on Investment (ROI)

This falls in line with the earlier discussion about justifying testing to your boss or
clients, and also the last point about analytics.

ROI Tip
If you need to understand the effect testing and optimization can have
on your bottom line, read this post on Finding Your Customer Acquisition
Sweet Spot.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 73

S A&B Testing Glossary: Terminology & Tips

Segmentation

Marketers are using more sources for driving traffic than ever
and it’s causing funnel gridlock. Sending up to 7 different
traffic sources into the same page (hopefully you’re using
landing pages and not your homepage) makes it impossible
to know what’s working and what isn’t.

So how do you figure out which traffic sources are

performing and where best to focus your marketing efforts?

Segmentation Tip
By using separate landing pages for each funnel,
you are free to test and optimize to your heart’s
content. This lets you empower your different
inbound marketing team members (the email guy
can run his own show – as can the social media girl)
to improve their own area without interfering with
or interference from the other channels. It can also
build a fun and competitive “which funnel is best”
type environment.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 74

S A&B Testing Glossary: Terminology & Tips

Social Proof

People are like sheep in many ways. They follow the herd and like to join in with
what others are doing. To illustrate this, imagine walking down a street and seeing
one person staring up at the sky. You probably wouldn’t give it a second thought.
But if there was a crowd of 30 people all staring at the same thing, you’d quickly
divert your attention to see what was going on. That’s social proof and it builds trust
in your visitors, helping conversions.

ROI Tip
You can add social proof to your tests in many ways: Testimonials,
endorsement or customer logos, what social influencers have to say
about your product. Other types of social proof are follower counts or
even adding social sharing buttons to illustrate how popular a page is.
More often than not though social follow and share buttons should go on
the confirmation page to reduce distraction from your CTA.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 75

T A&B Testing Glossary: Terminology & Tips

Traffic

Where is your traffic coming from? How does your paid traffic behave differently
from your organic traffic? What about direct visits and social media? And referral
visits? Testing different landing pages for each traffic source will answer these
questions and improve your marketing campaigns moving forward.

Traffic Tip
A common question that’s asked is “how much traffic you need when
running a test?”. If you can run 100-200 visitors to each of your pages,
you can often make gut decisions about pages that are obviously
underperforming and initate a new test. But generally you’ll want to wait
for about 1,000 total visitors and make sure you run the test for a week to
cover daily variances.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 76

U A&B Testing Glossary: Terminology & Tips

Unbounce

Start shameless plug: Unbounce is the DIY landing page platform that lets you
build, publish and test your landing pages without any IT or dev.

Unbounce Tip
Try the 30 day free trial, it makes A/B testing simple and even fun. If you’re
already a customer, did you know you can add viewers to your tests for
free? They can see stats and previews but won’t be able to mess with your
tests.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 77

U A&B Testing Glossary: Terminology & Tips

“Show your landing User Feedback

page to someone that A/B testing data can answer the question of “what,” but

doesn’t know anything the answer to “why” can be less than straightforward.
Complement your numbers with qualitative findings. This

about your business. means taking a step beyond your A/B test to ask more
complex questions.
After 10 seconds you
will have a long list of
User Feedback Tip
things to fix.” Check out tools like Qualaroo, UserTesting and
SilverBack for quick, reliable, and low-cost usability
testing.
Carlos del Rio
www.unbounce.com
Bonus tip: Read these two posts about user
feedback.

• 29 Ways to A/B Test Pages Based on *Real*

User Insight
• The Top 10 User Feedback Tools for
Improving Conversion

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 78

V A&B Testing Glossary: Terminology & Tips

Video Testing

Using a video on your pages can have a dramatic effect on engagement and your
conversion rate.

Video Testing Tip

Test things like autoplay vs. user-controlled play, adding CTAs, the length
of your video, how many you use etc. Take a look at this Case Study:
Using Video to Lift Landing Page Conversion Rate by 100%

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 79

V A&B Testing Glossary: Terminology & Tips

Visitors

A visitor is a unique individual coming to your website or landing page. We often

talk in terms of “Unique Visitor” which means each individual, no matter how many
times they visit your page, only gets counted once.

Visitors Tip
When you’re building your landing pages take a few minutes to really
understand your visitors. If you know they are coming from an email
campaign, particular ad group or a guest blog post, try addressing them
personally to improve conversions. ie. From your blog -”Hi Blog Reader!”,
from a guest post “Special offer for Hubspot Readers”

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 80

W A&B Testing Glossary: Terminology & Tips

Web Analytics

Even marketing rockstars sing the praises of Google Analytics, a free program that
makes it possible to track various features of website performance. While there are
other, proprietary programs that offer more advanced analytics features, I find GA
(as it’s known) to be adequate – even if it can be overly complicated at times.

Visitors Tip
Learn some tricks:

• Using Analytics to Measure Your Landing Page Performance

• Measuring Social Media ROI & Goal Conversions with Google
Analytics 5

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 81

X A&B Testing Glossary: Terminology & Tips

“X”

E[X]periment is about the best I can do here. After all, that’s what this whole post is
about, so it’s kinda perfect.

Tip
Read the Experiment section again.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 82

Y A&B Testing Glossary: Terminology & Tips

“Y”

Y should you be doing A/B testing?

Why You Should A/B Test Tip

Read back over this.

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 83

Z A&B Testing Glossary: Terminology & Tips

Zero Conversions

Wow, nobody likes your product.

Zero Conversions Tip

Before you panic, check your testing tool to see that there is a conversion
goal set, otherwise you may be missing a bunch of conversions that are
actually happening.

How’s that for a happy ending?

The Ultimate Guide to A/B Testing http://try.unbounce.com/for-a-b-testing 84

 Start A/B Testing Your Pages Now

http://try.unbounce.com/for-a-b-testing

UNBOUNCE Feel Free To Share This Document

www.unbounce.com