STEP 01

You are just starting out with

the testing process.

STEP 02
Your have limit number of
website visitors.
A/B Testing Overview
STEP 03
A/B testing is a user experience research
You have a limited number of
methodology. A/B tests consist of a randomized
conversions.
experiment with two variants, A and B. It includes
application of statistical hypothesis testing or “two-
sample hypothesis testing” as used in the field of
STEP 04
statistics.
You are looking for a radical departure
from existing designs.

STEP 05
You are just starting with a
new webpage.
 Product Roadmap via A/B Testing

Market Needs + Product Roadmap Deconstruct Roadmap Product Implementation Growth Conversion
Customer Product Managers or into Testable Chunks via A/B Testing CRO Data Scientist or
Expectations + business Analysts Analysts
Business Goals Business Analyst or UX Developers or CRO
Designers Analysts

01 02 03 04 05

Program Management
(Dedicated CRO program Manager)
 A/B Testing Process

01 Define Goal (conversion)

Before your create a test, your need 01
to know what, exactly, you’re
hoping to accomplish. 04 Crown the New Champion
Once you’ve launched your test, you
need to let it run for a long enough
02 05 period of time.
02 Brainstorm/Generate Idea
Once you’ve determined a goal you CONTINUOUS
want to accomplish, you’ll need to IMPROVEMENT
generate ideas.
05 Test Until Statistically Significant
By this point, you’ve set a goal, select
a page and determine what you want
03 Implement Changes 03 04 to test.

You will be need to implement

changes with highest potential,
intuition and best practices.
 A/B Testing Process

BEFORE OPTIMIZATION WITH OPTIMIZATION AFTER OPTIMIZATION

BUY NOW BUY NOW BUY NOW BUY NOW

Original -1% Variation 1-4.5%

$1,000 in Sales $4,500 in Sales

THANKS
 A/B Testing Process

VARIATION A

30%
50% visitors Conversion rate
see variation A

VARIATION B

16%
50% visitors Conversion rate
see variation B
A/B Testing Campaign

Campaign with a single objective. For

CAMPAIGN
example, website conversations.

Four separate Ad sets targeting two

Audience 01 Audience 02 Audience 03 Audience 04
different audience.

One Ad per Ad set. Notice there are only 2

Ad 01 Ad 01 Ad 02 Ad 03 different ads. Each Ad is displayed in its own Ad
set to each audience.
 A/B Testing Decision Tree

158 {n=38, 100%}

Pageviews < 8460

117 {n=7, 18%} 167 {n=31, 82%}

Pageviews < 7720 Clicks >= 775

17 {n=4, 11%} 149 {n=5, 39%} 184 {n=16 42%}

Experiment >= 0.5 Pageviews < 9907 Pageviews < 9418

134 {n=7 18%} 162 {n=8, 21%} 176 {n=16, 34%}

Experiment >= 0.5 Experiment >= 0.5 DOW = Sun, Tue, Fri, Sat

130 {n=5 13%} 171 {n=5 13%} 171 {n=11 29%}

Pageviews < 9765 DOW = Mon, Wed Clicks >= 730

121 132 103 124 139 142 164 196 147 154 178 198 222
n=2, 5% n=2, 5% n=2, 8% n=3, 8% n=2, 5% n=2, 5% n=4, 11% n=1, 3% n=2, 8% n=3, 8% n=8, 21% n=2, 5% n=3, 8%
A/B Test Example
A/B Testing To Statistically Significant
Improvements Confidence

Button Color Test +1.5 6

New Headline Test +0.8 8

Landing Page

Totally New Landing Page -4.0% 4.5

Another All-new Landing Page +6.0% 2

 Landing Page A/B Test
Sum of Newsletter Sign Ups Sum of Amount

160
$150.00

$140.00
140

120
SUM OF NEWSLETTER SIGN

120

SUM OF AMOUNT
$100.00 $100.00
100 $96.00
93
UPS

80 $80.00
80

65 66
$60.00 60
60
46

40

$20.00
20
12

Landing Page 01 Landing Page 02 Show Product No Product Video Remove Add Longer Form Page Shorter Form Page
Video Navigation Navigation
 A/B Test Dashboard
WEIGHTED SATISFACTION SCORE WEIGHTED SATISFACTION SCORE

120.9 27% 153.8 147.3 5.5% 154.4

No Feature Y Satisfaction Improvement Feature Y Feature X Off Satisfaction Improvement Feature X On

POSITIVE SATISFACTION RATING POSITIVE SATISFACTION RATING

78% 6.60% 85% 85% 2.3% 87%

No Feature Y Off vs On Feature Y Feature X Off Off vs On Feature X On

No Feature Y Feature Y Feature X Off Feature X On

40 40

20 20

0 0
 A/B Test Dashboard

EDIT WIDGET - SAVE WIDGET EDIT WIDGET - SAVE WIDGET

61.4% 65.7% 62.3% 60.0%

No Feature Y Feature Y Feature X Off Feature X On

AB Test Feature X Funnels

AB Test Feature Y Funnels

EDIT WIDGET - 6 MONTH LTV EDIT WIDGET - 6 MONTH LTV

44.0% 42.2% 43.2% 40.3%

No Feature Y Feature Y Feature X Off Feature X On

EDIT WIDGET IN TRIAL - EDIT WIDGET IN TRIAL -

PURCHASE PURCHASE

3.54% 6.32% 7.99% 5.055

No Feature Y Feature Y Feature X Off Feature X On
 A/B Test Analytics
Average session Subscribing occurrence
length
1% per day
24%
Control 181.8 Enable New Content

100

90

80
Views Content Time in subscribing
occurrences/session
28% per daily user
24%
70

60

50

40
Sessions per Sign In occurrences per
daily user
7% daily user
32%
30

20

10

0
Lifetime value per Subscribed occurrences per
daily user
60% Mar-02 Mar-04 Mar-06 Mar-08 Mar-10 Mar-12 Mar-14 Mar-16
daily user
9%
Daily Active Users
 A/B Test Results

VARIATION CONVERSION RATE IMPROVEMENT CONFIDENCE CONV./VISITORS STATUS

ORIGINAL 0.15% (+0.10%) - - 5/5,134 Stopped

VARIATION 01 0.25% (+0.20%) +150.00% 97.01% 13/5,152 Winner

A/B Testing Line Graph

Total visitors 100

90

80

Average conversion rate 70

60

50

Original 40

30

20

10
Variation 01
0
Mar-02 Mar-04 Mar-06 Mar-08 Mar-10 Mar-12 Mar-14 Mar-16
 A/B Test Results

VARIATION USERS CONVERSIONS CR UPLIFT CHANGE OF BEING BEST

A Default 30.000 1.200 4.00% - -

B Variation 30.000 1.260 4.20% 5.00% 89.1%

Sum of Newsletter Sign Ups Chance of B Outperforming A

1200.00%
1200.00%

1000.00%
1000.00%

800.00% 800.00%

600.00% 600.00%

400.00% 400.00%

200.00% 200.00%

0.00% 0.00%
-10% -8% -6% -4% -2% 0% 1% 3% 5% 7% 9% 11% 13% 15% 17% 19% A B
 A/B Test Tracking

OWNER
LANDING PAGE (LIST NAME STATISTICALLY
STATUS TEST START END PREDICTION RESULT NOTES NEXT STEPS
NAME OF DRI FOR SIGNIFICANT ?
TEST HERE)

Variation B performed 8% better than

Version B will perform
How to Create the original version and was
LP Header swapped better because the original Chose Variation B as winner, setting
Visuals for John Smith Completed 5/17/17 5/20/17 Success ✓ statistically significant at 95%
with subheader sub header is more up a/b on CTA placement next
Marketing confidence. Declared version B the
actionable
winner

Version B (image with Variation B performed 4% better than Chose Variation B. setting up a/b test
Demo Landing Amanda Different image on B
Not Started 5/27/17 6/20/17 background) will perform Inconclusive X A - not statistically significant but testing two different types of images
Page Morgan version
better declared B as winner. next

Version A will perform Version A performed 3% better, not

Call Sales Different Headline. A is Chose Variation A as winner, running
John Smith Not Started 6/23/17 7/6/17 better because it matches Inconclusive X statistically significant. Test ran for 3
Landing Page the title, B is descriptive template-level A/B test next
promo copy weeks.

Test "A" converted 33%better than

2018
Amanda Copy bullets vs long version B will perform Test "B." We are100% certain that the Chose Variation A as winner, setting
Predictions In Progress 6/17/17 7/12/17 Success ✓
Morgan form paragraphs better changes in Test "A" will improve your up a/b on CTA placement next
Landing Page
conversion rate.