Scribd
Upload
7 views

A B Testing Summary

Uploaded by

Esraa El-Sherbiny
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views

A B Testing Summary

Uploaded by

Esraa El-Sherbiny
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

A/B TESTING

FUNDAMENTALS OF A/B TESTING

Introduction to A/B Testing


A/B test (a.k.a. split test): The experimental method that
marketers use to compare different versions of a homepage,
ad, or email subject line.
Steps in an A/B test:
• Do research and decide what to test
• Formulate a hypothesis
• Design the test and select the right sample
• Collect data
• Analyze test results

Conversion rates: Turning site visitors into customers.

Variable: Any element that is altered in an A/B test. Don’t change multiple variables
in an A/B test.
Alternative hypothesis: Posits that there is a significant
difference between the two groups being compared.

Null hypothesis: Says there’s no significant difference


between the two groups being compared.

Designing and Conducting an A/B Test


Control: The group that receives the original version.
Make sure test traffic is
Treatment: The group that receives the modified version. distributed roughly evenly
between the control and
Representative sample: A group of individuals that have treatment groups.
similar characteristics to a tester’s target population.

Sampling bias: Occurs when a sample is not representative.

The sample size should be at least 500 members per test


group. The length of the test depends on the sample size
and the type of test.
Only stop an A/B test when you reach a predetermined
sample size.

©2022 QUANTIC SCHOOL OF BUSINESS AND TECHNOLOGY


A/B TESTING

Validity Threats

Validity threats: Forces that distort the validity of your


results.
Regression to the mean: As you run your test for a longer
period of time, the results will move closer to the average.
Novelty effect: The idea that site visitors react unnaturally to Only repeat visitors are subject
changes that are new. to novelty effects.
Instrumentation effect: An issue with the implementation of
your testing tools or code, usually due to user error.
History effect: The impact that external real-world factors
have on your marketing efforts.
Selection effect: The sample you’re using isn’t
representative of the typical population.

©2022 QUANTIC SCHOOL OF BUSINESS AND TECHNOLOGY

You might also like