Would you bet your job on your A/B test results?

Would you bet your job on
your A/B test results?

@QubitGroup #QubitABtest
Friday, 28 February 14

Agenda
• 5 digital experience technology trends that will guide
your technology investment
— Anjali Yakkundi, Forrester Research, Inc.
• 5 fundamental A/B testing mistakes and how to
avoid them
—Will Browne, Qubit
• How to run a successful A/B testing program
—Melanie Kyrklund, Staples
• Q&A

5 Digital Experience Technology Trends
To Guide Your Investment
Anjali Yakkundi, Analyst
February 27, 2014


Burberry brings digital into the store
© 2013 Forrester Research, Inc. Reproduction Prohibited
4


Room
Number

Address &
directions

Starwood’s Preferred Guest Streamlines The
Check-In Process
5


Agenda

› Why digital experience technology matters
› Key digital customer experience
technologies
› Five trends that will drive your digital
experience technology investment
› Recommendations

6


Agenda

technologies
› Recommendations

7


Age of the customer

Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report
8


Your customers are always connected…

Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report
9


…often on multiple devices

Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, August 2012
10


You need to sustain your customer’s attention
with engaging experiences.


Engaging experiences requires strong
technology management

12


Agenda

technologies
› Recommendations

13


Digital experience reference architecture


Many solutions make up the digital
experience ecosystem
Testing

Site search

Commerce platform

Product content
management

Optimization

Portals
Web content
management

Digital asset
management

Analytics
(mobile, social, Web)

Recommendation
engine
Online video
platform

Campaign
management

Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World”
Forrester report
15


Testing and optimization solutions are key to
engagement and measurement
Testing

Site search

Commerce platform

Product content
management

Optimization

Portals
Web content
management

Digital asset
management

Analytics
(mobile, social, Web)

Recommendation
engine
Online video
platform

Campaign
management

Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World”
Forrester report
16


Agenda

technologies
› Recommendations

17


#1- This is the beginning of the end of the allin-one-suite

9%

Only
prefer a
suite approach

Base: 233 Digital Experience Professionals Source: March 2013, “Digital Experience Delivery Online Survey”
18


#2- Tools must create business value out of
data

Solutions must effectively harness customer data,
product data, social media data, and other
information to create and optimize experiences.

19


#3- Tools must support personalized
experiences
•
•
•
•

• Likely next steps
Device / Channel • Historical past
• Immediate past
Location
• Current session
Time of day
Weather

Situation

Real-time
optimization

Prediction

20


#4- Look for more than software: look for
solutions

› Look for solutions that help marketers and business
users quickly adapt to changing customer needs.
This can be achieved by:
•Keeping IT out of critical publishing paths
•Prioritizing user interface needs

21


#5- Solutions must prioritize multichannel

22


Agenda

technologies
› Recommendations

23


Recommendations

› You can’t do it all at once — prioritize key customer
touchpoints
› Assess current technology gaps– most
organizations tend to have gaps around:
•Optimization
•Personalization
•Multichannel delivery

› When sourcing new solutions, look to vendors that
can help you connect the dots.

24


Thank you
Anjali Yakkundi
ayakkundi@forrester.com
@Ayakkundi


5 fundamental
A/B testing mistakes
and how to avoid them
Will Browne, Qubit
will.browne@qubitdigital.com

Why do I have to test?


A/B testing is an essential tool

@TheGrahamCooke


• It is the best, simplest way to prove the business value of
your digital technology

@QubitGroup
@TheGrahamCooke

#QubitABtest


• It is the best, simplest way to prove the business value of
your digital technology

• It is the only way to get close to proving cause and eﬀect
in what you do

@QubitGroup
@TheGrahamCooke

#QubitABtest

But there are costs associated with A/B testing



• The monthly cost of your testing platform




• The salaries and time of your developers and team




• The salaries and time of your developers and team

• The lost revenue from implementing changes that do
nothing for your site (or actively harm it)


We’re here to show you how to
reduce these costs, and increase ROI



• Find the ideas that are likely to have a real
positive effect on user behavior (true
winners)



winners)

• Detect as many true winners as possible



winners)

• Detect as many true winners as possible
• Minimize the chance of finding tests that

look like winners, but have no true effect


Find the ideas that are likely to have a real positive effect
on user behavior

39



Size does matter


Detect as many true winners as possible

We’ve found that tests based on data are

7.7x more
effective
than tests based on intuition alone
(Qubit, 2013)


Imagine testing for weighted coins

We’vemany coin ﬂips would be on datato prove that one
found that tests based enough are
• How
coin was weighted?

7.7x more
effective
(Qubit, 2013)



• How
coin was weighted?

7.7x more
effective

• It is impossible to be 100% certain, but the more data you
have, the closer you can get

(Qubit, 2013)



• How
coin was weighted?

7.7x more
effective

• It is impossible to be 100% certain, but the more data you
have, the closer you can get

• We follow scientiﬁc standards and wait until there is an
80% chance of detecting a diﬀerence
(Qubit, 2013)


Minimize the chance of finding tests that look like winners,
but have no true effect

We’ve found that tests based on data are

7.7x more
effective
(Qubit, 2013)



•We’ve found that tests based on data are there is a 5%
We generally assume that with good testing
false positive rate

7.7x more
effective

(Qubit, 2013)



false positive rate

7.7x more
effective

• We can test this using A/A tests (showing both groups
the exact thing)

(Qubit, 2013)



false positive rate

7.7x more
effective

• We can test this using A/A tests (showing both groups
the exact thing)

• So out of 100 A/A tests, 5 would be winners, despite
than tests based nothing
doing absolutely on intuition alone
(Qubit, 2013)


Splitting tests too much can lead to false positives


Get a second opinion


Validate your results

We recommend a

95/5 split
to confirm your winning variation
is still winning

“Most winning A/B test results are illusory”, Qubit 2014


A worked example: Imagine Mr Bean’s jelly beans shop


@QubitGroup


A worked example: Imagine Mr Bean’s jelly beans shop

• Mr Bean has a 1% conversion rate
• He runs 20 diﬀerent tests in one year
• There are 2 good tests with uplifts of 5%
• There are also 2 bad tests with uplifts of -5%


@QubitGroup


Truth


Correct
method

Naive
method

Truth

Correct
method

Naive
method

No. of tests

20

20

20

Reported
winners

2

3

9


Truth

Correct
method

Naive
method

No. of tests

20

20

20

Reported
winners

2

3

9

After
validation

2

2

9


Truth

Correct
method

Naive
method

No. of tests

20

20

20

Reported
winners

2

3

9

After
validation

2

2

9

2 x ~5%

9x
(5-40)%

Reported
uplift

2 x 5%


Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)


Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

None

7 unnecessary
tests

Wasted resource


Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

None

7 unnecessary
tests

Improved

Degraded

Improved

Confused

Wasted resource
Customer
experience
Your
understanding


Correct method

Naive method

Reported uplift

2 x 5%

9 x (5-40)%

Delivered uplift

2 x 5%

2 x 5% + (? x -5%)

None

7 unnecessary
tests

Improved

Degraded

Improved

Confused

Happy

Angry

Wasted resource
Customer
experience
Your
understanding
Your manager

So, to conclude, those
5 mistakes are:
• Mistake 1: Not A/B testing
• Mistake 2: Not prioritizing tests properly
• Mistake 3: Not waiting for significance and
sample size
• Mistake 4: Testing too many things at once
• Mistake 5: Not validating the tests after
implementation

Questions you should ask your
testing provider
• How do you prioritize tests?
• How do you calculate the sample
size required?
• How do you account for the multiple
testing problem?
• How do you validate winning tests?


Thank you
If you have any detailed questions please contact
research@qubitdigital.com

Will Browne, Qubit
will.browne@qubitdigital.com

How to run a successful A/B testing program

Melanie Kyrklund
Senior Ecommerce Analyst


Testing programs have the potential to drive the
online customer experience
 Understand your customers
and respond to their needs
 Optimize the user journey

Customer experience

 Create personalized
experiences
 Quickly respond to market
pressures

Personalization

Persuasion

Targeted
offers

User
experience,
functionality &
content

 Rapidly align marketing &
onsite experience
 Validate development items
 Every visit represents an
opportunity to establish a
relationship with a customer

66


The reality of running a testing program
For many organizations, testing represents a cultural change with
many political and operational barriers to be overcome:
 Resource & operational factors
 Buy-in / Multiple stakeholders
 Alignment with other business priorities and development plans
 Instilling a data-driven culture

67

A lot of testing is required to deliver results

 1/5 tests or 1/10 tests will deliver statistically signiﬁcant wins
 A rapid pace of testing is required to build a program that is
delivering consistent results month over month

68

Two factors are critical in building a successful
testing program

Approach

Speed

Methodology
Idea Generation

Operational Process
Prioritization

69

The right approach to optimization
Be inspired – every visit is an opportunity to establish a relationship
with a customer
Use data to understand how visitors are using your website, what is
working and what isn’t

Customer feedback to understand why visitors are not purchasing on

your website

If working over multiple websites & countries, look for insight at
local level ﬁrst and then identify commonalities at a later stage
Persuasion research, understand the psychology of your customers
Supplement with quick UX testing
Thoroughly research & quantify each opportunity (visitors reached,
expected uplift and incremental revenue)

UX optimization

Personalization

Persuasion
70


Idea Generation

71

Speed – Operational Processes
Testing Tool selection : Map out the most efficient
implementation process
 Operational efficiency
 Pick managed services (analysis, implementation) as required
Prioritization
Build a test pipeline logging:
 Idea & hypothesis
 Expected uplift & incremental revenue
 Implementation estimates
 Test duration
 Prioritize tests weekly, to keep up momentum of testing and
focus on biggest wins

72

Conclusion: A rapid, data-driven approach is required
 Testing programs have the potential to drive the online
customer experience, however a rapid data-driven approach
is required to deliver results
 Plan the required resources upfront
 Consider ease of use, test time to market and managed services
in tool selection
 Map out the most efficient end to end process to get tests live
 Process supported by data-driven idea generation &
prioritization

73

Thank you
Any questions?


More A/B testing material from Qubit

bit.ly/qABresearch

75

@QubitGroup
#QubitABtests


bit.ly/qABinfographic

Would you bet your job on your A/B test results?

More Related Content

Viewers also liked (20)

Similar to Would you bet your job on your A/B test results? (20)

More from Qubit (19)

Recently uploaded (20)

Would you bet your job on your A/B test results?