THE ULTIMATE GUIDE TO

MOBILE APP
A/B TESTING
Contents

Introduction: The Rise of App Demand and The Need for Analytics ........01

How A/B Testing Can Supercharge Your App Analytics .......................................04

What Can You A/B Test? .....................................................................................................................07

Tips and Tricks for Top-Notch App A/B Testing ..............................................................12

Real-Life Case Study: RunKeeper ..............................................................................................15

Winning in the Mobile Market .......................................................................................................18

1
INTRODUCTION
THE RISE OF APP DEMAND AND
THE NEED FOR ANALYTICS
 INTRODUCTION
1 THE RISE OF APP DEMAND AND THE NEED FOR ANALYTICS

Mobile app demand shows no signs of slowing down. In fact, the App Store and Google Play
alone now boast nearly 1 million apps apiece, and over 56 million apps were downloaded
last year. The old pay-per-download model has shifted to in-app efforts such as advertising,
purchases, freemium and lead generation, while non-monetization goals like customer loyalty
and driving market share have risen. These days, apps need to differentiate from the crowd,
and monetize while doing it.

According to a recent Duke University study, the majority of marketers feel the pressure to
prove ROI in every channel, including mobile. So to stand out in a competitive space in which
millions of options beckon, marketers must know with certainty what is working and not
working within their brand’s mobile app, and use that knowledge to create more successful,
engaging interactions.

02 1. Introduction: The Rise of App Demand and The Need for Analytics
Testing and identifying the right analytics is key to eliminating app guesswork and realizing
success. Marketing is no longer a go-by-your-gut business, and “we think” has been replaced
with “we know.” But testing mobile apps has historically been more complicated than testing
a website or an email, both in scale and in execution. And when it comes to development
alone, you don’t want to waste precious time needlessly altering code and re-uploading to
the App Store.

In this eBook, you will

learn how automated
app A/B testing gives
you the freedom to trial
ideas and hypotheses
within your app,
on-the-fly and without
making drastic
development changes.

03 1. Introduction: The Rise of App Demand and The Need for Analytics
2 HOW A/B TESTING CAN
SUPERCHARGE YOUR APP
 HOW A/B TESTING CAN SUPERCHARGE YOUR
2 APP ANALYTICS

Mobile apps may be booming, but they also are, and always have been, an entirely different
platform than websites, particularly when it comes to measurement. Traditional online
marketing metrics, such as pageviews and clicks, simply don’t apply here. Marketers need to
track app-specific analytics that focus on user behavior, engagement and conversion rate in
order to create an experience that users love and want to return to. Simple but scientific
A/B tests can take your app analytics to the next level by proving what works or doesn’t
work within your app.

Simple but scientific A/B tests

can take your app analytics
to the next level by proving
what works or doesn’t work
within your app.

Long used by marketers across other channels, an A/B test presents two different versions
of a message, feature or piece of content to the same population segment. Because
of complex code changes and re-submission requirements, A/B testing of mobile apps
once was a slow, arduous process requiring data analysts and developers. But thanks to
automated platforms, mobile app marketers now have the opportunity to run A/B tests
quickly and easily and get immediate, actionable insights.

05 2. How A/B Testing Can Supercharge Your App Analytics

A/B tests provide the perfect opportunity to bring mobile analytics full-circle. For example,
a Fortune 500 eCommerce brand created and evaluated two segments within their app:
those who performed a brand name search versus those who did generic searches, and then
compared their respective lifetime value (LTV). The results were clear: users who made
brand-specific searches resulted in 154% greater average LTV. The company was then able to
take immediate action to improve the app’s design and search functionality to make brands as
prominent as possible.

With further A/B testing, the company was able to try new features and UI changes in
controlled tests. Without the need for code alterations or app store re-submissions, they were
able to perform these tests, measure the results and iterate product design before launching
to all users. Using automated testing and actionable analytics resulted in their mobile apps
contributing to increasing total company revenue by nearly 20% and net income by 25%.

When your end goal is to increase the usage and profitability of your app, ready-to-run
A/B testing is crucial to driving impactful marketing campaigns for your user segments and
optimizing lifetime value.

06 2. How A/B Testing Can Supercharge Your App Analytics

3 WHAT CAN YOU A/B TEST?
3 WHAT CAN YOU A/B TEST?

App marketers always encounter the question “What do I measure?”

As we’ve covered, traditional web analytics metrics don’t apply in the app world. While every
business model is different and companies may focus on different goals, there are certain
app metrics that most marketers will want to address, including usage patterns, time spent in
different parts of the app, session length and conversion rates.

Three different types of A/B testing opportunities within mobile apps will help you hone in on
the metrics that count.

In-App Messaging

In-app messages are contextually relevant messages sent

to users while they are in your app, and are a great way to
communicate with users as part of their app experience to
improve engagement and retention.

For instance, you can target super fans (your most engaged set
of fans) with a message encouraging them to rate your app, or
you could send out a creative “thank you” message. A/B tests
can help determine what makes a successful in-app messaging campaign using data and
user feedback, instead of making assumptions.

Take this example: as a news organization, you want to prompt free users to upgrade to a
paid subscription. You can start by creating a custom segment of users in which
everyone has:

Read the US edition of the publication

Started a session at least 6 times in the last 3 days

Read at least 8 articles in the Politics or World News sections in the last 3 days

08 3. What Can You A/B Test?

 How to Use In-App Messaging and Push Messaging Together

Once the A/B test is launched, you can monitor closely which version is most effective at
pushing fence-sitters to upgrade and then refine your in-app messages according to the
results, or conduct further testing around the content, context and timing of the
winning version.

Push Messaging

Push messages are notifications sent to users on their home

screen when they are not in the app. This is a great option for
re-engagement efforts -- that is, to drive users back to your app
if they are not actively engaged with it or have fallen out of a
desired funnel.

But content, frequency and timing have to be done right when it

comes to push messaging, or it can alienate and frustrate users.
A/B testing can help you determine which push messages perform best -- with the right copy,
engagement points and message structure.

09 3. What Can You A/B Test?

Here’s a push messaging example for a retail app:

Let’s say a department store launches a push messaging campaign within its iOS app to
promote a highly sought-after new line of handbags. For this promotion, the retailer sends an
offer to those who have previously shown interest in handbags for 20% off their first in-app
purchase. The goal is to pique their interest and drive in-app handbag purchases.

The retailer could run an A/B test with two versions of the push messaging campaign to
significant samples of its target audience to see which message yields more conversions.

The results? 20% of those customers who received Message A took the first step and opened
the app while 15% of those who received Message B took the same action.

At first glance, it seems that Message A wins the contest, because of its higher open rate.
But the contest is not over yet; the end goal is for users to make an in-app purchase. If you
were only tracking app opens from this campaign, you would continue to invest in
Message A – which may actually have a lower conversion rate to in-app purchase than
Message B. Because of this unknown, it’s critical to look at the full picture and purpose of
the A/B test.

10 3. What Can You A/B Test?

 App features and content

Different app types have various feature and content elements

that lend themselves to different triggered “events” marketers
can track. An event could be viewing an article on a news app,
or adding an item to a cart on an mCommerce app. In a social
app, it might be users sharing content.

It’s in this case that A/B testing an app is most similar to testing
a website landing page, and identifying which visual features
content changes improve conversion rate. However in apps you have the ability to test
around events, or actions taken, making it more reactive to visitor flow and interaction than
something like a landing page A/B test.

A/B tests allow you to offer different versions of features and content -- buttons, icons, colors,
screen flow and home screen layout, to name a few -- to different segments of your audience
so they all experience the same app but in different ways. The results of various tests can
show you which tested feature moves a user closer to or more quickly towards the ultimate
conversion event -- a purchase, an ad click, or some other measure of engagement.

11 3. What Can You A/B Test?

4 TIPS AND TRICKS FOR
TOP-NOTCH APP A/B TESTING
4 TIPS AND TRICKS FOR TOP-NOTCH APP A/B TESTING

A/B testing offers a tremendous opportunity to use real data from real users to prove what
works in your app. But doing the most successful A/B testing possible - that is, moving the
needle on the metrics that matter most -- is easier said than done, especially if you’re just
getting started.

Here are six essential elements to keep in mind as you dive into the mobile A/B testing waters:

1 Set clear goals that can be proven analytically

A/B testing itself is just an orchestration of an experiment. What’s important, though, are
the implications and analysis of the test. So, consider what you want to analyze -- what
repercussions will the test have for the app experience as a whole? What are you trying to
learn? Having a set hypothesis that you know can be proven or disproven by the data is the
only way to identify clear results and move forward.

2 Look for quick wins

Until you can prove the power of A/B testing to your company’s decision-makers, keep your
tests fast and simple, focusing on quick wins. Look at easily achievable elements you can test,
like push messaging content, to run simple but effective campaigns. This way, you can help
your organization see the value of A/B testing as early as possible before taking a deeper
data dive.

3 Test frequently and iteratively

A/B testing of your mobile app is never “one and done.” Instead, it’s an ongoing process
of continuous learning, which means testing, testing and then more testing. The power of
automated A/B testing is that you can have short test cycles where engineering doesn’t
have to be involved -- so you can take advantage of your testing platform to test regularly
and repeatedly. And when you measure against control groups, you can dynamically change
messaging and content in response, allowing for easier, smarter testing.

13 4. Tips and Tricks for Top-Notch App A/B Testing

4 Test by user segment

Instead of just testing across all audiences broadly, you can use segmentation to create
highly unique user groups, resulting in richer end results. Running multiple A/B tests across
your different segments ensures that you’ll be gaining the most targeted test results possible
from users who naturally behave similarly. Plus, it allows you to conduct more complex
testing across campaigns for better behavioral insight.

5 Test visitor flows

Testing and tracking different visitor screen flows is unique to app analytics, and is key to
determining fall-off points and conversions within A/B campaigns. For example, group A may
be given only a one-screen conversion process, but group B has a two-screen process; one
is faster but conveys less information, the other takes more time but provides added context.
You can test which screen experiences drive the greatest impact on retention and revenue,
in addition to testing messaging, context and visuals, for the most in-depth results.

6 Try multivariate testing

There are many small features you can address with a simple A/B test. The next step to
assessing deeper interactions is multivariate testing -- where you test two different elements
at the same time, such as the order of a feature set as well as the color, particularly by
segment for enhanced results. This needs to be built on a foundation of preliminary A/B tests
that have shown clear and actionable insight. Moving to the multivariate level requires a
baseline set of analytics that prove why you’re testing the next round of features.

14 4. Tips and Tricks for Top-Notch App A/B Testing

5 A REAL-LIFE A/B TEST EXAMPLE:
RUNKEEPER
5 REAL-LIFE CASE STUDY: RUNKEEPER

Known as the “personal trainer in your

pocket,” Localytics customer RunKeeper is
the leading health and fitness app for iOS
and Android with more than 25,000,000
organically acquired users. As your digital
trainer, RunKeeper uses the sensors built into
“Localytics saves me
phones to track your physical activity.
time and makes it a lot
easier to communicate
The Challenge
test results. It provides
RunKeeper is a leading health and fitness app actual insight.
for iOS and Android with more than 25 million
Localytics makes my
organically- acquired users. Looking to move
far beyond the running category, RunKeeper
life easier”
wanted to find ways to ignite enthusiasm and
- Jon Gillman,
bring awareness to using the app to track
Product Manager at RunKeeper
other activities like weight lifting, yoga
or biking.

The Solution

RunKeeper ran an A/B test of their start screen using Localytics to see if a design change
provoked users to log non-running activities. Specifically, they used funnel analysis tools to
split the reports with custom dimensions for the A/B test groups and analyze which
UI variation resulted in more non- running events logged.

16 5. A Real-Life A/B Test Example: RunKeeper

The Results

The winning version was 10 times more successful at driving users to log a non-running
event than the other version

Since the A/B test was exclusively done in their Android app, RunKeeper was able
to port the winning variation over to their iPhone app with extra confidence – saving
development time on iterations later on.

17 5. A Real-Life A/B Test Example: RunKeeper

6 WINNING IN THE MOBILE
MARKET
6 WINNING IN THE MOBILE MARKET

“The powerful
one-two-punch of analytics
and testing provides
undeniable results that are
hard for the higher-ups
to ignore.”

Investment in mobile marketing is expected to grow by 38%

over the next five years. With more brands crowding the app
space, there is far more competition for users’ attention.

In order to rise above the app noise, marketers need a strong

foundation of data and insights to formulate an app strategy.
Analytics give you the overall picture, but A/B testing takes
assumptions and turns them into actionable facts. The powerful
one-two-punch of analytics and testing provides undeniable
results that are hard for the higher-ups to ignore.

Whether you test in-app messages, push messages or app

content and features, experimenting with ongoing, simple A/B
tests on a variety of app elements can make the difference
between an app that drives user engagement, retention and
conversion and one that doesn’t live up to its promise.

19 6. Winning in the Mobile Market

 START YOUR FREE TRIAL

Trusted by top brands to provide actionable insight.

Start your free trial of Localytics today
Localytics is the leading marketing and analytics platform for mobile and web apps. Localytics works
with some of the world’s most well-known brands, like eBay, Salesforce, Microsoft and The New
York Times, helping them create great experiences for their customers and maximizing profitability,
engagement, lifetime value, and loyalty. Localytics provides analytics and marketing for
more than 1 billion devices across more than 20,000 apps.

For more information visit www.localytics.com or

contact [email protected] to request a demo.