UTest Ebook Mobile Testing
UTest Ebook Mobile Testing
Table of Contents
Most Common Testing Challenges: What are the most frequent hang-ups
in terms of functionality, design, security and localization? Well share them
and give tips on how to avoid these pitfalls throughout the dev process.
Mobile Web vs. Native Apps: Theyre both mobile, but the testing
challenges of native apps are entirely different from those of mobile web.
Well explain which route is easier from a testing perspective.
Tips and Tricks: Get some helpful advice on obtaining quality crash
reports, using screenshot and video capture tools, as well as other tips and
tricks of mobile app testing.
Introduction
INTRODUCTION:
THE MOBILE APP BOOM
Whats at Stake?
By now, the mobile boom should be obvious to just about everyone (below are some hard numbers for those who need
a bit more convincing). What this means is that smartphones are rapidly becoming the primary method of interaction
for consumers and businesses worldwide. So whats at stake? Just the future of your business
377,900 iPhones are sold every day around the world
Key Questions
Before addressing the challenges of mobile app testing, it would be smart to first
understand the key questions. Here are a few questions youve likely asked
yourself and your company:
What are the testing challenges of native apps vs. the mobile web?
Are some types of testing, like functional and security, more critical than
other types of testing, like localization and usability?
1
0
COMMON ISSUES
IN MOBILE APP QUALITY
11
13
Context is critical for mobile applications. A thorough understanding of the users context and objectives is a must requirement.
Conduct ongoing user acceptance testing (UAT) throughout the design and development process, including testing on real devices.
Use simple navigation structures pointing to one specific task at a time.
Store data on the device selectively and archive data that is less frequently accessed on the server.
Effectively use multi-threading wherever possible to improve performance.
Use high contrast text color and select typefaces for maximum readability.
Provide clear status and feedback based on progress of task completion.
Manage content wisely. Wherever possible, crop large images and reduce the size of data files.
Use a consistent User Interface design that helps users maintain a familiarity with the application.
Get involved in real-time interaction via social media. This could include live Facebook or Twitter streams.
100
80
60
40
20
0
Test
Specification Design
Coding
Integration
Unit Test
Release Test
Post-Release
Functionality: A mobile application must present the user with the appropriate functionality. If the functionality of an application is
perceived as incomplete or inadequate, customers may be lost.
Layout & Design: Good layout and design allow a user to easily complete tasks. If a button is placed in the perceived wrong
place, users will get frustrated and might look for an alternative product.
Interaction: The flow of an application must be natural and allow the user to easily complete tasks. If a user believes they have
been sent to the wrong page and have to manually backtrack in order to find the page they were looking for, they can become
frustrated and abandon the application.
15
1
6
1
7
1
8
(Source: McAfee)
1
8
Content Static & dynamic content like catalogs, search results, metadata
Dates Is the date January 1 or 1 January?
Characters Diffrent lnguages have ifferent set f characters
Postal codes In some countries postal codes contain letters
Phone numbers Different formats for different markets
Direction Some languages are written left to right, others are right to left
Currency conversion Especially important for internet retailers
Tax calculation VAT, sales tax and others vary from country to country
1
9
20
Device F | R | A | G | M | E | N | T | A | T | I | O| N
Perhaps the most difficult aspect of the testing matrix is device fragmentation. This is especially true of the Android operating system, with its
seemingly countless permutations.
Take, for example, the image below. This is a data chart of nearly four thousand separate Android device models encountered by the dev team
at OpenSignalMaps over the course of a six month period. If youre concerned with quality on cross-platform apps, chances are your matrix will
look somewhat similar.
21
Operating System F | R | A | G | M | E | N | T | A | T | I | O| N
Although less daunting than the hardware matrix, the variety of mobile operating systems also poses a challenge for dev and engineering teams
whose goal is to provide a consistent user experience across platforms. The pie chart below illustrates the fragmentation that exists in just the
Android and iOS ecosystems. The chart outlines each operating systems required programming language.
Test abroad?
SIM cards?
Carriers?
Without moving a portion of your testing out of the lab and into the wild, theres
literally no possible way to ensure quality across carriers and locations. Heres a
good story from a European tech exec that pretty much sums it up:
We found, for instance, that one of our UK servers was not using the closest
node. It was sending information to Australia for no apparent reason. Without
testers physically present in those areas telling us how long it took to search,
stream and play music theres absolutely no way we would have known about
those issues.
24
Pros and Cons: Apps have the obvious downside of requiring more
development resources, especially to be truly optimized for each device. If a
company doesnt have enough resources to do this right, its better to have a
nice mobile site than a lame app. Jakob Nielsen
Mobile Web
25
Screen size
Custom themes
Interruptions
Many native apps have access to additional APIs. Those
connections need to be tested.
Mobile Web
No installation required
How does the site render in different browsers
iOS runs Safari
Android runs a stock browsers, but also supports
Chrome, Dolphin, Opera Mini and third party
browsers
Windows Phone runs IE
Blackberry runs a native browser, Opera Mini or Bolt
Requires an internet connection,
Connectivity varies by location
Load time is extremely important to mobile web users (source):
60% of users expect sites to load within three seconds
74% of users will only wait five seconds
When we asked them about future plans, the percentage of those offering a Web app or both remained the same, while many of those with nat
Lie Luo Global Intelligence Alliance
The Future:
Apps
Native apps still have a higher user rate and remain the only option if
your app needs to access specific APIs (such as the camera or
address book). Native apps also offer a highly controllable custom
experience because they are developed specifically for a designated
operating system so you can control exactly how it looks on each
device. Not to mention the built-in exposure of being included in an
app market.
For these reasons, native apps are not going away any time soon. But
mobile web does reduce app maintenance and allows for more
content. As a result, mobile web is emerging not necessarily as
competition to the native app, but as a potential ally. Companies are
now beginning to embrace either both types of apps or hybrid apps
which combine elements of the two into a single app.
ESPN ScoreCenter
A native app with updated scores within the app
Directs users to a mobile website for further articles
Lotte
100 pages written in HTML and used across platforms
A smaller number of custom developed native pages
Bank of America
A native app icon pushes users directly to the
banks mobile site
A Side-by-Side Comparison: Native Apps vs. Mobile Web vs. Hybrid Apps
Source: Worklight
A second downside of apps is that users have to install them. Our testing shows poor findability and usability in Apples Application Store,
and many users wont even bother downloading something at all for intermittent use. So ask yourself whether youre really offering something
within the hardcore mobile center of need: time-sensitive and/or location dependent, and whether your offer is truly compelling in this crowded
space. Most companies are never going to make it big in mobile. In some cases all they need is to make their main website somewhat
mobile- friendly. Many others should deliver a dedicated mobile site but not bother with apps.
- Jakob Nielsen
Mobile Web
2
9
In recent
50%77
of smartphone
The net result
[ofmonths,
tests roughly
loading
webpagesusers accessed native
apps
while
49%
of
used
the
mobile
web (comScore)
across 31 domains] is that using SPDY results
in a mean page load time improvement of 23% across these sites, compared to HTTP
Strides are being made to increase the speed of mobile web, which
would make it more competitive with native apps. [See sidebar]
Mobile web versus native app use fluctuates according to industry.
For example, in retail 51% of shoppers use mobile sites while only
28% use native apps. Remember, users have different expectations
for apps based on industry.
Some companies (like CNET and Kayak) are redesigning their
websites to mimic mobile sensibilities.
Mobile Web
3
0
ACHIEVING REAL-WORLD
TESTING COVERAGE
31
Manual vs.
Test Automation In-House vs.
Outsource
Partnering Testers
& Engineers
Agile Testing
Exploratory vs.
Test Cases
Managing
QA Teams
One solution: Hundreds of top companies from global enterprises to early startups have moved a portion of their mobile app testing outside
the lab and into the wild. This enables applications to be tested by professional testers, on real hardware, with real software, using imperfect
connectivity. In short, under real-world conditions. Thats not to suggest that other methods are insufficient. Lets take a closer look.
32
In-House:
Pros: In-house team makes communication easier;
professional testers who are tightly integrated into SDLC
Cons: Capacity constraints; testing may not match user
demographics; maintaining adequate collection of mobile
devices is prohibitively expensive
Outsourced Testing
Pros: Professional testers; access to range of devices
Cons: Hidden costs; not easily scalable; challenges in
communication and time zone; still lab-based testing
In-The-Wild Testing
Beta Testing
Pros: No direct costs; lives outside the test lab environment;
range of mobile devices, carriers, OS, locations
Cons: Users are not professional testers; provide subjective
feedback that is not diagnostic, which raises the cost for the
company; also puts unfinished product in front of customers
Crowdsourced Testing
Pros: Lives outside the test lab across real devices, carriers,
OS and locations; professional testers who mirror end users;
access to range of mobile devices & connections
Cons: There is a direct cost; requires careful partner vetting;
requires communication efforts between testers & developers
developing
a quality
application.
technology
enables
So there
I was, onand
mytesting
iPod Touch,
trying
to get Such
to a list
of users
whose name started with the letter I. It worked great on the sim
developers to verify certain functionality that is not specific to any device,
carrier or operating system.
Suddenly it hit me: This is different. Sure, all of the old GUI rules apply, but suddenly we have a new set of ways the application
Matt Heusser
They are also very useful in terms of usability, and especially design,
including data input, screen size, button use, etc. all from the convenience
of their own laptop. However, pointing and clicking with a mouse on a
desktop monitor isnt the same as using a finger on a small screen.
Remember, the very nature of emulators and simulators means testing of
applications is occurring in an environment far removed from the real world,
where actual users run and interact with those applications ON their
devices.
But recognize that this approach will never again be sufficient on its on.
For reasons already discussed, you'll need to move a portion of your app
testing efforts out of the lab and into the wild to mirror your user base:
Dont be fooled by the word wild when it comes to testing mobile. When you think of the term in-the-wild testing think of it as real-world vs.
lab conditions. This is not outsourcing or beta testing, and its definitely not suggesting you replace your existing QA or the solid
processes you have in place within your test lab. Rather, this is about complementing, scaling, and ensuring that a portion of your testing
mirrors your end users as closely as possible. The key is to align your in-the-lab testing with your in-the-wild testing. Matt Johnston, CMO,
uTest
5 THINGS THAT
YOU OUGHTA KNOW
36
Bite the hand that feeds you (a little). Dont be a jerk, but make sure you dont go passively along
with bad ideas. A weak willed tester is a bad thing. Make your concerns knownand know when
to fold. Patrick Copeland, Senior Engineering Director, Google
What doesnt work is assuming that because you are the test manager, or the tester, that you
know how your testing adds value to the company, that you know what the most important testing
is, and that you know the managers and executives are idiots because of what they are telling you
to test. Its not up to us to judge them for either being idiots or for making clumsy requests that
dont actually help them resolve their concern. It is up to us, however, to help them figure out what
they actually need, help them figure out how to meet that need, and then guide our testing to best
serve that need. Scott Barber, CTO, PerfTestPlus
The top reason we should care about diversity in our testing teams is because the demographic
of a computer user is more diverse than ever before. Lanette Creamer, Owner, Spark Quality
I suggest taking two weeks and actually measuring how the team is spending its time. Oh, not for reporting it is very important the team
stop the time-tracking after two weeks and not hand individual metrics into management for evaluation. Instead, we want to use the
numbers for improvement. For example, many of the people I talk to can spend 80% of their time in meetings, working on documentation,
working on compliance activities, doing email, and so on. That only leaves 20% of the time to test! Just pushing those numbers from
80/20 to 60/40 will double the amount of time the team spends actually doing testing. Matt Heusser, Writer and Consultant
Our job is vigilance. To lose vigilance is to abdicate our responsibility. Vigilance, in testing, means being a good skeptic. We must reject
certainty in any form. Were the Knights of May Be. To believe is to cease questioning; to fall asleep at our posts. James Bach, Author
and Consultant, Satisfice
37
Are developers and testers friends or foes? Im sure this varies from company to
company and team to team, but Id wager a sum of money that teams where the two are
friends dramatically outperform teams where theyre foes. Andrew Muns, President,
Software Testing & Performance
If test can work with development to improve the quality of the software prior to code
complete, that will improve the speed of the whole system. Realizing this, and having a
little bit of data to prove it, can help the entire system improve. Matt Heusser, and
Consultant
When people make a repeated error using my code, instead of asking why these people
are idiots, I learned to ask whats wrong with my software that causes the nice people to
look like idiots. Cem Kaner, Professor of Software Engineering, Florida Institute of
Technology
In mobile app markets today, users have an instant, strong, and public vote on the quality of an app through star ratings and comments.
Interestingly, they dont often complain about failed unit tests what matters is any impact on their experience. Be sure to focus on
integration, network fault tolerance, the device, other apps, and most importantly, user perception and expectations. Jason Arbon,
Engineering Director, uTest
"Engineering and quality best practices defined during the PC and Web eras are not well suited to the agile and mobile environments of
today. Only dinosaurs will blindly port the practice they are comfortable with to mobile it requires a re-imagining of engineering to be
competitive, efficient and relevant. Jason Arbon, Engineering Director, uTest
Find the right mixture of testers who are close enough to the product to become real experts with it
and specialist testers who are expert breakers. Call it whatever you like but product expertise and
testing expertise are the two ingredients that I find make for a good product. - James Whittaker,
Partner Development Manager, Microsoft
But even then, we dont assure quality; we question it on behalf of our clients. The people who are
building and managing the product assure quality. Michael Bolton, Principle, DevelopSense
If you change a core feature, you need to test everything again even if it was working and should
still be working because your other tests show it all fine. Michelle Sullivan, Founder, SORBS
Study your actual users: invite a handful of representative customers to your location and run them through simple usability studies of
your software. One day in the lab is worth a year in university lecture halls, in terms of actionable lessons learned. (And remember that
your usability lab can be a regular office or conference room as long as you shut the door.) Jakob Nielsen, Usability Expert and
Principle, Nielsen Norman Group
Expectations can be managed once you have established a credible track record of delivering product. Here data and process are
invaluable. If you can prove your points with data, then it makes it very hard to refute it, whereas hand waving just sets you up for
constant failure. From a process perspective, setting up a framework so that other teams know what is required to be completed to allow
the product to ship or what is the real definition of Feature Complete, for example, puts everyone in a better position. James Sivak,
Director of QA, Unidesk
I want software projects as a whole to run more smoothly and more predictably. I
really think thats what software testing is all about reducing the uncertainty of
software development and finding ways to muscle errors out of the process. A
process in which mistakes are harder than doing the right thing is the ultimate goal.
We cant eliminate them, but we can make doing the right thing to be the easiest
thing to do. James Whittaker, Partner Development Manager, Microsoft
I dont consider performance testing optional if youre making money online. Here is
how I would think about it: What is the cost of an outage to my company? Think
about it in terms of dollars lost as well as customers lost. Dan Bartow, VP of
Product Management, SOASTA
Teams need to be able to set aside time to look at everything from the proverbial
forty thousand foot view. Only then can they make true progress with goals that may
take a year to implement but that have huge ROI opportunities. James Sivak,
Director of QA, Unidesk
I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something
you MUST use, even if they dont work. A free tool may be freely abandoned. This gives you flexibility. James Bach, Author and
Consultant, Satisfice
To me, the biggest weakness [in the way companies test software] is not considering software testing anything but a (barely) necessary
evil. Testing is seen as something that could be done by a troop of monkeys, so serious testers are treated like third-class individuals. You
treat people as if they are stupid, then they will wind up acting stupid. Gerald Weinberg, Author and Consultant
4
0
41
Appendix
42
On automation: You probably guessed that automation had to play a role in this
solution. However automating scenarios on the phone or a tablet is complicated
when the core functionality of your application is to play back videos natively but you
are using an HTML5 interface which lives in the applications web view.
On device priority: To put it other way, when it comes to watching Netflix, any
device other than those ten devices can be classified with the high priority devices
based on their configuration. This in turn helps us to quickly identify the class of
problems associated with the given device.
On their testing team: We keep our team lean by focusing our full time employees
on building solutions that scale and automation is a key part of this effort. When we
do an international launch, we rely on crowd-sourcing test solutions like uTest to
quickly verify network and latency performance. This provides us real world
insurance that all of our backend systems are working as expected. These
approaches give our team time to watch their favorite movies to ensure that we have
the best mobile streaming video solution in the industry.
Even though the iOS operating system doesnt present as large of a matrix as Android
ESSENTIAL GUIDE TO
doesnt mean testing iOS apps is easy. iOS devices are supported by more than 100 carriers
iOS APP TESTING
worldwide, and the introduction of high res screens with the New iPad added yet another
Want
to
learn
the
ins
and
outs
of mobile app testing for the A
dimension to the matrix. Here are a few tools (most of which are already on the device) that
will make iOS testing on real devices easier and more effective:
uTests Essential Guide to iOS App Testing details the iOS testing matrix, UDID provisioning and important areas of focus for testing iOS apps (in
The console log is an iOS feature that includes information from every
application on the device. This log can help pinpoint if your app is being
adversely affected by other apps/software on the device. The console log
does not last very long so be sure to access it quickly following a crash,
otherwise the details about an issue may be lost. The console log can be
saved by connecting the device to a computer and accessing the Console
tab within device configuration utility.
The built-in screen shot command is a good tool to use to document bugs.
Holding the home and power button simultaneously will send a snapshot of the
devices screen to the iCloud (and subsequently to all your connected devices.
Windows8 apps can be tested in an iPad using Win8 Metro Testbed. This gave
Windows developers a jump start on the new market before Windows8 was released. It
also opens the market to developers without access to a Windows8 device. (Source)
Microsoft strictly adheres to their Metro style. Leading up to the release of Windows 8,
the company published Performance Tips for Metro Style XAML Apps. The paper
provides performance tips for metro style XAML apps. It provides guidelines for
developers to improve performance of their apps in the scenarios most important to
users. It assumes that you know basic XAML programming in C#, C++, or Visual Basic.
Download the paper at http://www.microsoft.com/en-us/download/details.aspx?
id=29886
Just for looks and beauty, I definitely favor the Windows 7 phone over Android.
Steve Wozniak
AND IN
CONCLUSION
46
Conclusion
Those companies who neglect testing in their mobile application
development lifecycle, do so at their own peril. There was a time when
the testing matrix was indeed too complex, too burdensome to be
completed using standard means. But thanks to the rapid evolution of
in-the-wild testing, all that has now changed.
Those companies that are leveraging in-the-wild testing - particularly
in the mobile apps and mobile web space - are gaining a competitive
advantage with each new release. As the mobile market continues to
grow sharply, those brands that make real-world testing coverage a
priority will enjoy ROI in terms of increased market share, profitability
and above all, user loyalty. Those who neglect testing will struggle to
keep up in a world filled with app stores, social media and increased
user expectations. Simple as that.
Either way, the future of mobile applications remains bright. Once
reserved for a tech-savvy niche, the use of mobile applications is now
firmly entrenched in the mainstream. Theres no turning back when it
comes to mobile apps, and that means the testing for mobile must
rapidly evolve to keep up.
4
7
About uTest
uTest provides real-world testing services for web, desktop and mobile
applications. By leveraging a community of 60,000+ professional testers
from 190 countries, uTest helps companies test their products under realworld conditions. Thousands of companies from startups to global
enterprises such as Google, Microsoft, HBO, Amazon and USA TODAY
turn to uTest to complement their in-the-lab testing, and to help them launch
better apps. uTests services span the entire software development lifecycle,
including functional, usability, security, localization and load testing.
The company is headquartered near Boston, with offices in Silicon Valley,
London and Israel. uTest has raised more than $37MM in funding and
consistently generates triple-digit annual revenue growth. The company won
the American Business Associations Most Innovative Company of 2011
award, and was named a Best Place to Work by the Boston Business
Journal two years in a row.
More info is available at www.utest.com or blog.utest.com. For more info on
mobile app testing, visit www.mobileapptesting.com
uTest, Inc.
153 Cordaville Road
Southborough, MA 01772
p: 1.800.445.3914
e: [email protected]
w: www.utest.com
About uTest
4
8