THE ESSENTIAL

GUIDE TO MOBILE
APP TESTING
Tips, techniques and trends for launching
great mobile apps that delight your users
 01 INTRODUCTION

YOU DEVELOPED A MOBILE APP ... NOW WHAT?

B
y the time you finish reading this eBook, the mobile app landscape will
change. New OS versions will be released. A bunch of new devices will hit
the market. Cool new apps will go viral, gaining millions of users and setting
new standards in user expectations. And mobile app testing will become much
more complex and challenging.

Discouraged? Don’t be. The world of mobile is filled with excitement and
opportunity and anyone can have the next hit app! But to make a splash, your
app needs to stand up to use by thousands – if not millions – of users around
the world with different devices, operating systems, networks, connections and
more. To keep those users – your users – delighted, your app needs to be high
quality and thoroughly tested.

Despite the ever-changing mobile landscape, there are some constants in the
realm of mobile app quality that organizations of any size – from small startups
to major global enterprises – can study, understand and take advantage of.

In the following pages, we’ll discuss the mobile landscape, dispel some common
myths and outline proven strategies to help you deal with the exhaustive mobile
matrix. Ready to learn more? Let’s get started ...
 FUN FACT

93% OF U.S. SMARTPHONE USERS WILL

DOWNLOAD AT LEAST 1 APP THIS YEAR.

NEARLY 36% OF U.S SMARTPHONE USERS

WILL PURCHASE AN APP THIS YEAR.
(EMARKETER)
 03 MOBILE BOOM

THE MOBILE BOOM

By now, the mobile boom is obvious and it’s clear As mobile adoption increases, apps
that mobile apps are here to stay. Smartphones and will become even more critical to
tablets are rapidly becoming the primary method of the success of businesses. Companies
interaction for consumers and businesses worldwide. that recognize this trend and take
Smartphone penetration in the US reached 75% in 2015, actions to make the quality, security
with global penetration hitting 60%. But mobile goes and usability of their applications top
beyond smartphones and tablets. Apps are now being priority will find themselves with a
incorporated into cars, wearable technology and home tremendous advantage.
appliances (which are considerably less “mobile” but
still important in terms of applications.)
 04 MOBILE BOOM

THE EXPLOSION OF APPS CAN BE SEEN IN JUST ABOUT EVERY INDUSTRY, FROM CONSUMER USE TO B2B TOOLS:

• Retail: Location-aware mobile commerce, omnichannel • Gaming: From serious console games to casual mobile
offerings, in-store checkout and more games, gaming is a top app category that spans user
• Media: Magazines and newspapers are going digital and demographics
spreading to readers on every operating system • Business apps: CRM, ERP and HR systems are
• Travel: Mobile bookings, check-ins, maps and on-demand developing mobile access points
services • Productivity: Docs, spreadsheets and presentations are
• Education: Tablets and apps are increasingly being used increasingly moving to tablets
in classrooms and for on-site work training • Collaboration: Workers are ditching laptops and need
• Healthcare & fitness: From digital medical records and to access email, IM and other important features via
in-home care apps to fitness trackers and diet apps, the mobile
health, wellness and fitness world is going mobile • Smart TVs: Media, social, news, weather, sports and
• Finance: Apps do everything from on-the-go banking to gaming apps are in demand on smart TVs
real-time trading and portfolio analysis • Emerging tech: Smartwatches, in-car apps, lifestyle and
• Social: Social networks and geolocation meet-up apps fitness trackers, smart appliances, etc. need optimized
need to follow users wherever they go and well-testing apps
 FUN FACT

BY 2020 THERE WILL BE 6.1 BILLION

SMARTPHONES IN CIRCULATION,
OVERTAKING THE NUMBER OF ACTIVE FIXED
LINE SUBSCRIPTIONS WORLDWIDE.
(ERICSSON)
 06 MOBILE CHALLENGES

MOBILE CHALLENGES
The majority of users expect your app OTHER PROBLEM AREAS

to load within three seconds. If your app • OS & device: If you’re aiming for widespread adoption, then
you need to provide a seamless user experience regardless of
doesn’t launch fast enough, users will find
device or OS.
an alternative and many report that they
• Security & privacy: Companies that violate user privacy run
will never return. the risk of making headlines for all the wrong reasons. Users
expect their personal data to remain private and apps to be
Worse, a poorly performing app can have a 100% free of threats.
negative impact on a user’s overall opinion • Connection speed: Does your app run the same on 3G, LTE
and Wi-Fi connections? If the answer is no, expect upset users.
of your brand. A study by SOASTA found that
In some instances, a poor connections can be the difference
“almost nine out of ten Americans associate between a five-star and one-star review.
negative feelings with brands that have • Data usage: Users are wising up to the fact that some apps
poorly performing websites and mobile drain battery faster than others – and they’re abandoning
those offending apps in favor of more economical ones. When
apps (88%).”
developing an app, stay mindful of how it functions in all
situations.
Right or wrong, users have incredibly high
expectations when it comes to app quality.
 07 MOBILE CHALLENGES

EXTRA TIP
Want to know what your users are really thinking and saying
about your app? Don’t dig through app store ratings and
reviews by hand – Applause Analytics is a mobile app quality
tool that does it for you. Applause Analytisc crawls the ratings
and reviews in the Apple App Store and Google Play stores
and produces actionable data that will help you pinpoint WHY FAILING HURTS
trouble areas. Back in the day, one bad customer experience could
spread via word-of-mouth and reach a few prospective
LEARN MORE
customers. Thanks to social media, that same word-of-
mouth now travels at the speed of light and reaches
hundreds, thousands or even more users.

The other major difference is app store ratings. Imagine

a poor review stamped directly on your product – that’s
what happens with bad app store ratings.

“You can launch a beautifully designed native

application, but if it crashes, then it will receive a
poor rating and users will go elsewhere,” said Michael
Croghan, Mobile Solutions Architect at USA TODAY.
“Our goal is to launch nothing short of a 4.5 star app
every time – no exceptions. Anyone can read your app
store rating. There’s no way to hide poor quality in the
world of mobile.”
 08 FUNCTIONAL TESTING

COMMON FUNCTIONAL ISSUES

Like with all software, you need to make sure every facet MENU OPTIONS •• Menu options
of your app works as intended. Pay special attention to can often be difficult to access or
these key areas. decipher. Make sure menu items like
Help, About, etc. are easy to find,
SIGN-UP & LOGIN •• This may seem like a no-brainer,
navigate and select.
but if users cannot easily access your app, all your
efforts will be wasted. If your app requires a password ACTIONS •• Any problems related to
and user name, pay close attention to the fields and scrolling, selection, the back button,
make sure it’s easy for users to enter their information. etc. are bound to lead to trouble.
 09 FUNCTIONAL TESTING

“MUCH AS WE LOVE FREEDOM AND CHOICE, WE ALSO LOVE

THINGS THAT JUST WORK, RELIABLY AND SEAMLESSLY.”

CHRIS ANDERSON, WIRED

CONNECTION SPEED & CARRIER •• Testing on INTERRUPTIONS •• How does your app
only one carrier hasn’t been an option for a behave when the device battery is at full,
long while. Make sure your basic tests cover medium and low strength? Your app doesn’t
the most likely carriers, as performance can operate in a vacuum, so test how it works
vary greatly from one carrier to another. when interruptions like battery power, SMS,
MMS and video calls occur.
SCREEN SIZE •• Screen size discrepancies
should be a top consideration while testing ERROR MESSAGES •• Your error messages
your app. Does everything look good and should be clear, concise and actionable in the
work correctly on a range of screen sizes? eyes of the everyday user.
 10 FUNCTIONAL TESTING

EXTRA TIP
Committing to a single testing approach in today’s
multiple app ecosystems will result in missed bugs
and a poor quailty app. Good testing requires a
more comprehensive approach.

LEARN MORE
FUNCTIONAL TESTING
When it comes to app functionality, remind all
departments from design to development to QA
that making sure the product works is a shared
responsibility. Test early and test often – combining
automated and manual testing to cover your app
from every angle.

It is exceptionally important to test the functionality

of your app on a range of devices. If an app doesn’t
render correctly, it can mean navigation or other
actions are entirely impossible – making your app
essentially useless. And don’t forget regression
testing any time you change, remove or add a
feature. Use the app like you expect a new user to,
revealing potential problems.
 11 USABILITY TESTING

COMMON
USABILITY The goal of usability testing is to ensure a user can complete

ISSUES the tasks they are expected to complete. More importantly, they
must be able to do so easily and without becoming frustrated.
A good user experience can be thwarted by a number issues.

FUNCTIONALITY •• A mobile application must present the user

with the appropriate functionality. If the functionality of an
application is perceived as incomplete or inadequate, customers
may be lost.

LAYOUT & DESIGN •• Good layout and design allow a user to

easily complete tasks. Don’t offer so many features or content
that your app is difficult to navigate. Everything should be easy
to see and interact with on a small screen.
EXTRA TIP
To learn more about Mobile Usability, INTERACTION •• The flow of an app must be natural and
download our free whitepaper.
interactions should be easy. For example, if a button is placed
in the perceived ‘wrong’ location, users can become frustrated
LEARN MORE
and abandon the application. If links are too close together,
they’ll be hard to select.
 12 USABILITY TESTING

USABILITY TESTING
Not all users are created equal. If you want to get a true If you need help navigating the
sense of your mobile app’s usability, be sure to select usability testing landscape, consider
testers that closely mirror your target audience. working with a professional
usability expert. Experts can craft
Usability is highly subjective, so take care not to ask
surveys, analyze the raw data for
leading questions when crafting a usability survey and
meaningful takeaways and give
pay special attention when looking at the survey data.
you recommendations based on an
It’s imporant for you to seperate minor feedback from
understanding of user expectations
recurring issues that hinder the app’s usability for a
and current mobile standards.
large portion of users.
 13 LOAD TESTING

FUN FACT
Performance issues, crashes and
hangs account for roughly 15% to
30% of user complaints in the iOS
and Android marketplaces.

COMMON LOAD ISSUES

Load testing is another critical step in the successful launch
of an application. If the app’s performance is slow, users
will likely move to other apps and services.

OUTSIDE VARIABLES •• Be aware of mobile carriers and

data usage, which can affect the speed and performance of
the application.

MOBILE WEB •• Users expect mobile websites to respond

just as quickly as traditional sites. Be careful that your
mobile site isn’t so bulky that it’s slow to load and react.

ACCESS FROM ANYWHERE •• Mobile apps can be used

anywhere. Therefore they need to be tested with different
devices and carriers depending on the region to ensure a
consistent experience and load time.
 14 LOAD TESTING

LOAD TESTING
Before you begin load testing, decide what you want Though it may be tempting to rely
to gain from the effort. Do you want to find out how entirely on test scripts running
your app performs under steady traffic over time, if you synthetic loads, like in all cases, a
have a data leak, at what point your app breaks under machine alone isn’t good enough.
pressure, or when it slows down? There are several The most comprehensive approach to
types of load testing and each accomplishes a different load testing is to combine synthetic
goal. load with live testers and performance
engineers. This hybrid approach gives
you a human perspective on how your
app really looks and performs under
pressure and professional engineers
can make recommendations for
improvement.
 15 SECURITY TESTING

EXTRA TIP
To learn more about Mobile
Security Testing, download our
free whitepaper.

LEARN MORE

COMMON SECURITY ISSUES

The explosion of mobile presents an entirely ASK YOURSELF ABOUT:

new set of security challenges. While most • Confidentiality: Does your app keep private data private?
of the security tools and practices used for • Integrity: Can the data from your app be trusted and verified?
traditional web and desktop applications • Authentication: Does your app verify that users are who they
are equally applicable to mobile, there are say they are?

some unique concerns to keep in mind, • Authorization: Does your application properly limit user
privileges?
including lost or stolen devices, mobile
• Availability: Can an attacker take your app offline?
malware, data leaks, encryption and more.
• Non-Repudiation: Does your app keep records of events?
 16 SECURITY TESTING

SECURITY
TESTING With the rise of bring-your-own-device (BYOD) programs,
making sure your app isn’t the gateway for a hacker is crucial
for you, your reputation, your users and your company.

Security testing is a practiced skill, not necessarily one a general

QA team can adequately accomplish. Likewise, security testing
varies depending on what you’re testing. Are you testing a
mobile website, a web API, a backend server, all the above? If
no one on your QA team is willing to dig into security testing
on a deep level, find an expert who can fill the gap.

Another crucial aspect that falls under security testing is privacy

and transparency. Companies who access unnecessary device
features or use data in a way users aren’t aware of will suffer an
angry backlash when the truth comes out. When testing, make
sure your privacy policy and information about handling user
data is easy for your users to find and understand.
 “
SOFTWARE CAN BE CORRECT WITHOUT BEING
SECURE. INDEED, SOFTWARE CAN MEET EVERY
REQUIREMENT AND PERFORM EVERY SPECIFIED
ACTION FLAWLESSLY YET STILL BE EXPLOITED BY
A MALICIOUS USER. THIS IS BECAUSE SECURITY
BUGS ARE DIFFERENT FROM TRADITIONAL BUGS.
IN ORDER TO LOCATE SECURITY BUGS, TESTERS
HAVE TO THINK DIFFERENTLY TOO.
JAMES WHITTAKER
TESTING EXPERT
 18 LOCALIZATION TESTING

COMMON LOCALIZATION ISSUES

The worldwide proliferation of mobile apps Relying on translation software or outsourcing firms is
requires your products to be accessible not the same thing as proper localization. If your app isn’t
and contextual for users in a variety of being reviewed by native speakers or people who know the
markets and regions. But without proper local area, you could be missing major mistakes. Apps that
localization, companies can exclude, fail to fully account for different cultural understandings
confuse or offend entire subsets of users will not survive and thrive.
without even knowing it.
 19 LOCALIZATION TESTING

LOCALIZATION TESTING
LOCALIZATION TESTING SHOULD CONSIDER FACTORS SUCH AS:
• Content: Static and dynamic content like catalogs, search
results, metadata
• Dates: Is the date January 1 or 1 January?
• Characters: Différent länguages have ðifferent set∫ θf characters
• Postal codes: In some countries, postal codes contain letters
• Phone numbers: Different formats for different markets
• Direction: Some languages are written left to right, others are
right to left
• Currency conversion: Especially important for retailers
• Tax calculation: VAT, sales tax and others vary from country
to country
• Cultural context: Words and phrases may technically be
translated correctly, but they could have a different implied
meaning in another culture
 “
OUR GLOBAL CUSTOMERS HAVE DIFFERENT
DEMANDS. WE WANT PRODUCTS TO ‘FEEL LOCAL’
AND TO SUPPORT FEATURES THAT MAY BE
UNIQUE TO SPECIFIC MARKETS. AS WE SPECIALIZE
OUR PRODUCTS FOR CERTAIN MARKETS, IT
INTRODUCES MORE CHALLENGES FOR TESTING –
LIKE REQUIRING SPECIAL CULTURAL KNOWLEDGE.
PATRICK COPELAND
SR. ENGINEERING DIRECTOR, GOOGLE
 21 FRAGMENTATION

DEVICE
FRAGMENTATION
Perhaps the most difficult aspect
of the mobile testing matrix is
device fragmentation. Though
the iOS device matrix is growing
more than ever, fragmentation is
especially an issue for the Android
operating system.

Take, for example, the image to

the right. This is a data chart
by OpenSignal of over 24,000
separate Android device models
encountered in August 2015 (For
comparison, the same chart from
2012 found nearly 4,000 devices.)

If you’re concerned with quality

on cross-platform apps, you’ll
encounter a similarly complicated
matrix.
 22 FRAGMENTATION

OPERATING SYSTEM FRAGMENTATION

Although less daunting than the hardware matrix, ANDROID •• Even though Google is
the variety of mobile operating systems also poses a into version 6 of their OS, a large
challenge for dev and engineering teams whose goal is portion of users are still on 4.x and
to provide a consistent user experience across platforms. some even on 2.3.x.

Even within a single operating system, developers can iOS •• iOS users are traditionally
encounter fragmentation issues. Mobile users are not quick updaters, but developers should
always quick updaters – and some users who want the still support at least one (if not two)
latest OS version may not have access to it right away. back version of Apple’s mobile OS.
Developers need to ensure apps work on the newest OS
version and several past versions that are likely still in
use.
 23 FRAGMENTATION

IN-THE-WILD TESTING

In-the-wild testing is the practice of moving part of your QA out of the lab and into the
real world. Testing in real-world conditions provides a better perspective on how apps
will work in the hands of your real users. It also gives you access to a greater range of
devices, operating systems, versions and carriers. CLICK HERE TO LEARN MORE.

CARRIERS &
CONNECTIVITY
If you’ve ever experienced unexplained inconsistencies with the
performance of your mobile app, there’s a good chance that
carriers and connections are the culprit. Perhaps more than any
other criteria, issues associated with carriers are almost always
found outside of the lab environment.

Without moving a portion of your testing out of the lab and into
the wild, there’s no way to ensure quality across carriers and
locations. These situations are impossible to recreate accurately
in a lab.
 24 MOBILE APPS: NATIVE

NATIVE APPS
Native apps have a higher use rate and remain the only Since a unique native app needs
option if your app needs to access specific device APIs to be created for each operating
(such as the camera or address book). Native apps also system, your quality efforts also
offer a highly controllable custom experience because need to expand. Native apps should
they are developed specifically for a designated operating always be tested on a range of
system – so you can control exactly how the app looks devices and carriers associated with
on each device. The built-in exposure of being included that operating system, as well as on
in an app store is another draw for native apps. different versions of the OS.

Users of different operating systems

have different expectations, so listen
to what they’re saying after you
launch your app. With the ability of
users to rate and review your app in
the public app stores, there’s little
tolerance for poor quality.
 25 MOBILE APPS: WEB

THE RISE OF
MOBILE WEB Despite the rise and continued dominance of native apps,
mobile web is still extremely – and increasingly – important.
It’s also sometimes the better option.

Mobile web is attractive for several reasons. It is often cheaper

and easier than creating and maintaining native applications
because developers can create one source code that will function
across platforms.

By adopting responsive web design, teams can create one

cohesive experience that responds automatically to whichever
size device a user is on, from large screen desktops to the smallest
smartphones and every tablet in between. This is especially
important in today’s world of growing device fragmentation

EXTRA TIP and rising mobile web use. In fact, many of today’s mobile
Learn why responsive design is users feel that companies that don’t have sites that work well
important and how to implement
on mobile don’t care about quality.
a responsively designed site.

LEARN MORE
 26 TESTING COVERAGE

CONQUERING THE MOBILE MATRIX

Historically, when companies wanted to improve their The answer is simple – move a portion
app quality, they did so within the sterile environment of your testing out of the lab and into
of a lab – far removed from where their users live, the wild.
work and play. The evolution of mobile eliminates that
Professional in-the-wild testing gives
option as an effective solution on its own. If testing
companies access to experienced
for mobile apps is conducted exclusively in a central
testers around the world who will
location, teams can’t really be sure their apps will work
test your app in the same scenarios
in the hands of their actual users in the real world.
your users will actually experience it.
So what’s a tech leader to do? Hire testers in key Testers use real hardware, with real
geographic markets? Construct an elaborate QA lab? software, on imperfect connections.
Spend more on simulators and emulators? Declare their In short, they test under real-world
app quality “good enough?” conditions.
 27 TESTING COVERAGE

EMULATORS &
SIMULATORS
Mobile emulators and simulators are still an important testing
tool in that they enable developers to verify general functionality
and perform regular regression testing.

However, the very nature of emulators and simulators means

testing is occurring in an environment far removed from the
real world, with a series of actions performed by a mouse and
keyboard, not fingers on a touch screen.

The convenience of simulators and emulators can lull teams

into a false sense of security. But the advantages of such
EXTRA TIP
Some companies offer labs filled with tools are limited in scope, and should never be considered a
real devices for testing. These tests still substitute for real-world, on-device testing performed by live
rely on computer generated actions
testers. After all, emulators and simulators aren’t dealing with
and reports. Just because an app didn’t
crash doesn’t mean all the graphics rush hour in LA, testing in the fields of Iowa or holding a device
loaded correctly, there were no GUI in one hand and a hot cup of coffee in the other ... while on
bugs and links were easy to click with
a bus. Using emulators and simulators in tandem with in-the-
fingers. For that, you need real users on
a range of devices. wild testing will give you the best results.
 “
SO THERE I WAS, ON MY iPOD TOUCH, TRYING TO
GET TO A LIST OF USERS WHOSE NAME STARTED
WITH THE LETTER ‘I.’ IT WORKED GREAT ON A
SIMULATOR WITH A MOUSE, BUT WITH THE
ACTUAL iPOD, MY FINGER WAS TOO FAT TO CLICK
THE SINGLE LINE OF PIXELS.
MATT HEUSSER
QA CONSULTANT
 29 CONCLUSION

CONCLUSION
There was a time when the mobile matrix was
overwhelmingly complex and difficult to tackle. But
thanks to the evolution of in-the-wild testing, that’s
changed.

As the mobile market continues to grow, those brands

that emphasize quality and pay special attention to real-
world test coverage and feedback will enjoy increased
market share, profitability and – above all – user loyalty
and delight. Those who neglect quality will struggle to
keep up in a world filled with app options.

The future of mobile remains bright. Once reserved for

a tech-savvy niche, the use of mobile apps is now firmly
entrenched in the mainstream. There’s no turning back,
and that means mobile app quality must rapidly advance
to keep up with user demand and expectations.
 30 RESOURCES

ADDITIONAL RESOURCES

IN-THE-WILD TESTING •• For real-world success, you iPHONE & iPAD APP TESTING •• If you want your iPhone
need to test your app outside the lab and in the wild, or iPad app to thrive post-launch, you need to test it on
where your users live, work and play. the unique OS it was designed for. Learn the specifics of
iOS app testing and launch higher quality apps.
LEARN MORE
LEARN MORE

MOBILE APP QUALITY TOOLS •• Our app quality tools

help you stay more connected to your users and the
ANDROID APP TESTING •• The Android ecosystem is
one of the largest and most complex. With so many
health of your apps. From Mobile Beta Management to
different devices, manufacturers, carriers, locations
app store analytics and beyond, Applause has the tools
and versions, it’s hard to know where to begin testing.
you need to succeed in the high-stakes apps economy.
These eight tips will help.
LEARN MORE
LEARN MORE
 31 ABOUT APPLAUSE

ABOUT APPLAUSE

Applause is leading the app quality revolution by Applause app quality tools help companies stay connected
enabling companies to deliver digital experiences to their users and the health of their apps with Mobile
that win – from web to mobile to wearables and Beta Management and Applause Analytics.
beyond. By combining in-the-wild testing services,
The company is headquartered near Boston, with offices
software tools and analytics, Applause helps
in San Mateo, Seattle, Germany, Israel and Poland –
companies achieve the 360° app quality™ they need
with resellers serving dozens of international markets.
to thrive in the modern apps economy. Thousands
Since launching as uTest in 2008, Applause has raised
of companies – including Google, Fox, Amazon, Box,
more than $80 million in funding, generated triple-digit
Concur and Runkeeper – choose Applause to launch
revenue growth annually, made consecutive Inc. 500
apps that delight their users.
appearances and was named the 7th Most Promising
Applause automated and in-the-wild testing Company in America by Forbes in 2014.
services span the app lifecycle, including functional,
usability, localization, load and security.

100 Pennsylvania Ave.

Framingham, MA 01701
1.844.500.5556
www.applause.com