A/B Testing

Marketers often spend hours to come up with techniques to

promote their products. To make sure that their ideas are
successful, A/B tests can be run. This way they can gain insights
into what works, and what needs work. Whether creating a
new landing page to redirect website visitors or sending a
promotional email newsletter to contacts, A/B testing is one of the
most popular methods of discovery for online marketing strategies.
 Table of content

Before you start................................................................................................................01

What is A/B testing?............................................................................................................01

Why A/B testing?.......................................................................................................................02

How A/B testing works............................................................................................................03

Understanding A/B testing.....................................................................................................03

Benefits of A/B testing.............................................................................................................04

What to test?..............................................................................................................................05

Creating your first A/B test campaign...............................................................................07

Understanding A/B testing reports.....................................................................................12

A/B testing - best practices..........................................................................................................13

Frequently asked questions......................................................................................................14

 A/B Testing

Before you start

In this handbook, we’ll take you through different facets of A/B testing.
But before we get into the details, there are a few basic terms that you
should know:
Control - The original version of the campaign.
Variation - The modified version of the campaign.
Element - A test parameter. It can either be the email subject line, sender
details, CTA button, or any other content component.
Test - The process of comparing the control and the variation to come up
with a clear winner.
Conversion Rate - The rate of performance at which a version (A or B)
converts the test group recipients to opens or clicks.

What is A/B testing?

A/B testing (also known as split testing) is a marketing experiment that
compares two versions of an email campaign to determine which one performs
better. By conducting an A/B test, marketers can statistically analyze which
version performs better and works best for their contacts. The version with
a higher conversion rate is declared the winner and is sent to the rest of the
contacts.

1
 A/B Testing

Why A/B testing?

A well-planned A/B test can make a huge difference in the reach of your email
campaign. With A/B testing, you can discover the most effective elements of
your campaign and come up with a message that best suits the interests and
needs of your contacts.

Here are a few reasons to A/B test:

1) Through A/B testing, you can figure out the exact marketing strategy
that works best for your company and product.

2) It is an effective way to determine exactly what your contacts want, rather

than guessing their needs.

3) With access to all of this information, it is easier to make decisions

about conducting your campaigns.

2
 A/B Testing

How A/B testing works

In an A/B test, an element of your email campaign is selected and modified
as a variant to compare with the original. This change can be anything from
a slight tweak in the sender details or subject line, to a complete redesign of
the email content. The control version is then sent to one half of the selected
test group, while the variant is sent to the other half.

Once an A/B test campaign has been sent, the test group of contacts is
monitored for their opens and clicks over time. You can view the open and
click rates for each version under A/B Test Reports. The winning version will
be sent to the remaining contacts.

Understanding A/B testing

John runs a fine dining restaurant. He is a big fan of email marketing and uses
it regularly to connect with his customers. He wants to run a promotional
offer on Valentine’s Day, and decides to let his customers know through an
email campaign. So, he comes up with some campaign content but isn’t sure
what the best subject line would be.

He has two options:

A : We’re making Valentine’s Day sweeter

B : You can’t get a better gift for your valentine

3
 A/B Testing

To zero in on the perfect subject line, he decides to run an A/B test. 40%
of his customers receive the test; 20% get Option A (the control), and the
other 20% are sent Option B (the variant).

John had a feeling that Option A would perform better than Option B, but
once he saw the results, he was quite surprised. Option B beat out option A
with a conversion rate of 21%, over Option A’s mere 12%.

The winning campaign was then sent to the remaining 60% of his customers.
John’s restaurant saw an unusually huge turn out on Valentine’s Day, and
thanks to A/B testing, his profit rates shot up!

Benefits of A/B testing

Let’s take a look at some other ways in which A/B testing can be beneficial
to your business.

1) A/B testing blends quantitative and qualitative methods to check content

quality.

4
 A/B Testing

2) It incorporates your contacts to decide the message your email campaigns

will convey.

3) It provides you with a platform to experiment and understand the market

trends that influence your contacts.

What to test?
The first step in setting up a successful A/B test is to decide which element
to test. You may want to test every element of your campaign, but it is always
best to test one element at a time to avoid confusion and get accurate results.

The three variables to test are:

1) Subject line
2) Sender details
3) Content

Subject line

No matter how engaging the content is, your subject line is the element
that decides if your email will be opened or not. Most of us wait until we’re
about to click send to optimize our subject line, but ultimately it’s the first thing
that catches your contact’s eye. It is certainly worth investing some time to
write and test your subject lines. The core idea of a subject should remain the
same, but small variations in phrases can be made between the two versions
of your email campaign.

5
 A/B Testing

Sender details

By testing the sender names, marketers can easily match the perfect sender
name with their target audience. For your campaign to succeed, you should
use an authentic sender name that your contacts can relate to. A/B testing
helps you to determine whether to use your brand name, company name, or
even your own name. Sometimes you might even want to forget the name
and stick to email addresses.

Email content
Quality content attracts quality contacts, but how do you make sure that you
have quality content in your campaigns? You can experiment with headlines,
images, opt-in forms, calls to action, or literally any other element of your
content. The campaign which receives the best response will give you an
insight into the type of content you should have in the future campaigns.

Here’s a brief list of elements that can be tested in your content:

Headline - The headline is the first thing that grabs your contact’s attention
and gets them to read further. Experiment with different words, phrases and
text sizes to get the best results.

Content Length and Format - The length and format of your content can
make a huge difference in the reception of your message. Test out different
formats and lengths to see what works best.

Images - The images you use in your content play a vital role in your conversion
rate. Test the size and type of images and the optimum number of images.

6
 A/B Testing

Call to Action Text and Position- The text and position of your call to action
can have a surprising impact on your conversion rate. Make sure to test a few
different words and positions for better results.

Call to Action Style - The style of your call to action depends on whether
its a link or a button, it’s size and the color used. Test these elements
individually for accurate results.

Creating your first A/B test campaign

In an A/B test, you’ll select and revise elements of your email campaign.This
change can be anything from a slight tweak in your sender details and subject
line to a complete redesign of the email content. Send the control (original
version) to one half of the test group and the variant to the other half.

7
 A/B Testing

To set up an A/B test:

1. Select Campaigns from the navigation bar and choose Advanced Campaigns.

2. Click Create Campaign in the top-right corner of the screen.

3. Hover over the A/B Testing category and click Create New.
4. Fill out the following information on the A/B Test Setup page:

• Name

• Subject Line

• Sender Details

• Campaign Content

5. Select which aspect of your email to experiment with. You have three
options:

• Subject line

• Sender details

• Campaign content

8
 A/B Testing

6. Choose the proportion of recipients to receive each campaign.

7. Decide how the winner will be selected from the following three
options:

Based on Open Rate - The version with the highest open rate will be the
winner.
Based on Click Rate - The version with the highest click-through rate will be
the winner.
Manually – Manually select the winning version from A/B test reports.

8. Select the test duration. You can also opt to manually select the
duration while the test is still in progress by visiting the detail view page of an
A/B test campaign.

9. Click Next. The next steps will vary, depending on your previous selections.

Variants

Using subject or sender details as the test element:

1. Enter the subject line or sender details for both versions, depending on
the test element you selected in the setup page.

9
 A/B Testing

2. Select a sender email address from the dropdown list.

3. Select a reply-to address for the contact to respond to.

4. Click Next to proceed.

Using content as the test element

1. Enter two different content versions in the text areas.

10
 A/B Testing

2. Click Next to proceed.

Content Configuration

1. From the Content Configuration page, select either HTML & Plain Text
or HTML only as your email type.

2. Choose a content option from the following selections:

Templates - Design an HTML email using our template layouts.
Import/Upload - Import an existing HTML email and images as a .zip file.
HTML Editor - Design an HTML email using our HTML editor.
Import from cloud - Import content from various cloud storage services.

3. Click Next.

Choose Recipients

Finally, select a mailing list or segment to receive your campaign, and click
Proceed.

The A/B test campaign will now be sent for review. Once the review is
complete, you can send the campaign.

11
 A/B Testing

A/B test reports

Once an A/B test campaign has been sent, the test group will be monitored
for their opens and clicks. You can view the metrics for each version under
A/B Test Reports. The winning version will be sent to the remaining contacts.

Note: If you chose Manual in the A/B test setup page, go to

A/B test reports after the test is sent, and choose a winner.

To access A/B test reports:

1. Select Reports from the navigation bar and choose Sent Campaigns.
2. Select the A/B test campaign you wish to view.
3. Click A/B test Reports.

12
 A/B Testing

A/B testing - best practices

Test everything - Is your email on trend? Will it work for your contacts? Will it
convert or flop? Testing every element of your campaign will give you insights
into your audience and will let you improve your campaigns. What works for
your competitor might not work for you, so test away!

Test in advance - An A/B test should be your first step when considering
a change to your email campaign. There is no point in going too far with a new
strategy until you find out whether it’s likely to improve your results.

Trust the results - If the test results are not what you expected, resist the
urge to listen to your instincts. A controlled A/B test is always more accurate
than an opinion.
Learn from results - When an A/B test gives you a clear winner, your first step
is to send the winning campaign to the remaining contacts in the list, but
don’t stop there. Now that you know what works best for your contacts, you
can apply the changes to other campaigns as well.

Keep testing - One test is not sufficient to achieve maximum results. You
need to keep experimenting to keep up with your contacts. There’s no such
thing as a perfect email campaign; every campaign you send can be better
than the previous one.

13
 A/B Testing

Frequently asked questions

1. How do I create a chain campaign for all my contacts
who’ve left the A/B campaign unopened?
It is not possible to create chain campaigns for an A/B test. However, you can
export the unopened contacts, import them to a new mailing list, and send
the campaign to those recipients.

To export unopened contacts:

1. Click Reports and select A/B campaign.

2. Click Recipient Activity, select Unopened and choose Export.

2. How do I clone my A/B campaign?

You can clone your A/B campaign by following these instructions:
1. From the Email Campaigns page, click the Filter icon.
2. Hover over the A/B campaign you’ve chosen and click Clone.

3. Can I select A/B test campaign recipients based on numbers

rather than a percentage?
It is not possible to select the recipients by the number of contacts. Recipients
may only be selected by percentage.

4. Is it possible to pause an A/B test?

Yes, it is possible to pause an A/B test when it is in the ‘In Progress’ stage. To
pause an A/B test, open your campaign and select the Detailed View.From
there, click on the Pause button in the top-right corner. You can resume the
test whenever you want by clicking Restart.

14
 A/B Testing

5. Is it possible to stop or cancel an A/B test?

No, it is not possible to stop or cancel an A/B Campaign.

6. After sending a campaign for A/B testing, can I decide the

winner manually before the campaign is sent to the remaining
contacts?
Yes, you can manually select the winner and send the email campaign to
your remaining recipients.

7. If both A & B score equally - who will be the winner for

the resulting campaign?
If both the versions A and B score equally, then it is a ‘Tie’. In this case,
the test remains inconclusive and you will have to choose between one of
these options
Version A - Send version A to the remaining contacts.
Version B - Send version B to the remaining contacts.
Manual - Choose the winning campaign to be sent.

15
 Contact us:

Zoho Corporation
4141 Hacienda Drive
Pleasanton,
California 94588, USA

+1 (888) 900 9646 (USA)

+44 (20) 35647890 (UK)
[email protected]

Please feel free to share this document

You can find more details and step by step instruction on our Online Help
Visit https://www.zoho.com/campaigns/ to sign up for a free account!

© 2019 Zoho Corp. All Rights Reserved.