Scribd
Upload
162 views

Testing Your App: BCS Bristol Autumn School

This document discusses strategies for testing mobile apps. It begins by explaining why testing is important to identify bugs, ensure proper functionality, and build customer confidence. It then covers some of the unique challenges of mobile testing like device diversity. The document outlines elements of an effective test strategy including different levels of testing and considerations for the test environment. It also provides examples of tools that can be used for test automation, security testing, and accessing remote devices. Finally, it discusses approaches for testing different aspects of an app like the user interface, network performance, and security.

Uploaded by

Tanuja
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
162 views

Testing Your App: BCS Bristol Autumn School

This document discusses strategies for testing mobile apps. It begins by explaining why testing is important to identify bugs, ensure proper functionality, and build customer confidence. It then covers some of the unique challenges of mobile testing like device diversity. The document outlines elements of an effective test strategy including different levels of testing and considerations for the test environment. It also provides examples of tools that can be used for test automation, security testing, and accessing remote devices. Finally, it discusses approaches for testing different aspects of an app like the user interface, network performance, and security.

Uploaded by

Tanuja
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 33

BCS Bristol Autumn School

Testing your App

Jim Thomas
Director of Software Testing
TVS background

TVS Germany
TVS UK
(2011)
(2008)

TVS France
(2012)

T&VS India
(2011)

Delivering tailored solutions for software testing and


hardware verification
What we’ll cover

• Why test?

• Mobile testing challenges

• Developing a test strategy

• Testing different aspects of your App


Why test?

• Identifies bugs introduced in development


• Ensures there is no unwanted functionality
• Reduces development and schedule risk
• Reduces the cost of failure
• Confirms that the software works as desired
• Provides confidence in the software
• Measures quality, helping to decide when to
release software to the market
And if you don’t …

Fail to do a good job of testing and your


customers will end up doing it for you

Customers expect an App to work …..


and when it doesn’t there are other options just
a couple of clicks away
No hiding place …..

Testing is not just about finding bugs

it’s about gaining confidence in


the quality of your App
Mobile Testing Challenges
Device diversity and more

Web testing is a challenge

Mobile testing is even more of a challenge


And add to that …

Web App
servers servers
AUT

DB
servers
With more challenges

• Rapidly changing landscape – new devices


and OS versions
• Extreme time to market pressure for new apps
and versions
• Frequent updates to 3 rd party Apps
• Greater performance concerns – memory,
CPU, power
• Lack of mature testing tools
Developing a Test Strategy
We all know this ….
>150X
Relative Cost of Defect Fix

50X

20X
10X
1X 5X

Requirements Design Code Dev Test Accept Test Operation

Source: Boehm – Equity Keynote Address 2007


So shift ‘Testing’ left

High Level Low Level


Requirements Code Dev Test Accept Test
Design Design

Defects

High Level Low Level


Requirements Code Dev Test Accept Test
Design Design

Defect Detection Point


Elements of a test strategy

• Take account of development flow and release phases


• Levels of testing
– Unit
– Integration
– Application
– End-to-end system
• Test exit criteria
• Test environment
– Devices/emulators etc
• Test automation tools
• Resourcing
– In-house/Outsourced/Crowdsourced
Test environment – devices

• Time/cost constraints and device availability


limit devices to test on
• Master list of ‘must have’ and ‘good to have’
devices
• Group based on manufacturer/OS
version/screen size etc
• Identify candidate device in each group
• Test on emulators until App stable
• Full test on real candidate devices
• Sanity test on other devices
Test environment – remote devices

• Subscription based SaaS


• Remote access to hundreds of real devices
• Connected to real networks around the world
• Record & play/keyword based scripting
• Detailed reports

Perfecto mobile
DeviceAnywhere
Test environment - emulators

• Access to a large number of device


types/browsers
• Easily switch devices types
• Fully instrumented
• Lack quirks, faults
and characteristics of
real devices
Resourcing - crowdsourcing

• On-demand extension of testing resources


• For ‘in the wild’ testing
• Coverage across devices, networks, locations
and languages
• Can be cost effective and quick
• Feedback analysis required

uTest
Testing Different Aspects of Your App
Unit testing

• Should be first priority


• Separate UI and app code logic to make unit
tests more granular and faster
• Decouple unit tests from emulator to make
faster

Roboelectric, JUnit (Android)


OCUnit, GHUnit, OCMock (iOS)
Application testing

• Functional
– Does App perform intended functions?
• UI
– Does it meet platform requirements etc?
• Usability
– Is it easy to use/intuitive?
• Localisation
• Performance/load
• Security
– Authentication, authorization, privacy etc
Manual testing

• Traditional method of mobile testing


• Manual execution of test cases
• Visual verification of results
• Repetitive and time consuming
• Prone to human error
• Can be costly and slow when frequent
releases
• Still main option for UI, Usability,
Performance/load and Localisation testing
Automated testing

• Primarily used for functional testing


• Define tests once & execute multiple times
• Useful for regression and cross-platform tests
• Can reduce effort/time to market
• No generic automation framework available
for all mobile devices
• Some features can’t be simulated so will
require manual testing
Test automation tools

• Android • iOS

Robotium (free) MonkeyTalk (free)


MonkeyTalk (free) QTP (paid)
QTP (paid) Ranorex (paid)
Ranorex (paid) eggPlant (paid)
eggPlant (paid) iOS Instrumentation
DDMS (monitor CPU (monitor CPU and
and memory) memory)
Testing – the network challenge

• Multiple network technologies


• Protocol handling differences
• Mobile web proxies imposing restrictions
• Test over internet or LAN (network bypass) –
emulators will do this
• Also some emulators can internet access
mobile web proxy for testing
• Use real devices on location where possible
Security testing

• Threat Modelling
– identifying possible attacks and defining controls to
prevent them
– prevent/detect/mitigate
– defensive v adversarial

• OWASP Mobile App Threat Model (beta)


• Open Web Application Security Project
• Top Ten Mobile Risks
Threat modelling
Security testing tools

• Static analysis
– Clang static analyser (Objective C)
– FindBugs (Java)
– Veracode (all platforms)
• Dynamic testing
– OWASP Zed Attack Proxy
• Forensic analysis
– Android Debug Bridge
– iPad File Explorer
– SQLite Database Browser
And finally …
You are not alone

65%
Do not have the right tools to test
52%
Do not have the devices readily available

Do not have the right testing process/method


34%
29%
No mobile testing experts available
19%18%
Do not have in-house test environment

Not enough time to test

World Quality Report 2012-13


Capgemini, Sogeti, HP Software
Testing your App

o Mobile App testing is challenging


o Focus on building quality and security in
o Develop a clear test strategy
o Make use of test automation
o Manual testing and real devices are part of
every test strategy

Don’t leave testing to your customers


Thank you

Jim Thomas
[email protected]

TVS Test and Verification Solutions


SETsquared Business Acceleration Centre
Engine Shed
Station Approach
Temple Meads
Bristol
BS1 6QH

Checklists

• Android app testing check-list


• http://testeropine.blogspot.in/2012/10/testing-checklist-for-
android-app.html

• iOS app testing check-list


• http://testeropine.blogspot.in/2012/10/testing-checklist-for-ios-
app.html

• Scripts that automate many static security testing preparation


tasks for both iOS and Android
www.smartphonesdumbapps.com

You might also like