Mobile Application Testing

By Waqar Ahmad
Swaam Tech (Pvt) Ltd
Contact us: info@swaam.com for Ur App Testing

Note: The slide was presented in the Software Testing Course at NUCES-FAST Lhr

Classification of Mobile Applications
 Web Applications
 Could be Simple Markup (WML , HTML, XHTML,
XHTML MP) or AJAX Applications
 A Web Application may be Designed for
 A Single Type of Device or
 Multiple Different Devices with Customization for each
Device (or each Family of Devices)
 Multiple Different Devices—A Transcoder/Proxy Residing
Between Web Application Server and Device Transforms
Content based on Device User-Agent Capabilities

Classification of Mobile Applications
 On-Device Applications May be
 Developed for Better Portability
 Java ME
 FlashLite
 Native Platform Applications
 IPhone: Objective C
 Symbian: C/C++
 Windows Mobile: .Net CF
 Android : Java for Application and C/C++ for Low Level

Mobile App Testing—Challenges
 Device Fragmentation is a Big Challenge
 Devices Vary in Screen Size, Memory, Processing
Power, Hardware Features etc.
 Apple iPhone is Least Fragmented among All Mobile
Platforms
 Testing on All Target Handset/Devices
 Almost Impractical if Number of Target Handsets is Large
 Testing on All Target Operator Networks
 Almost Impractical if Number of Target Operators is Large
 Network Operator may Impose Certain Constraints

 Device Vendor (Not Necessarily Manufacturer) may
Customize a Device
 Changes may be Done in Both User Interface and
Functionality
 Many a Times Changes are Not Well-Documented
 Device is Not the Same as Released by Manufacturer
 Testing on All Versions of Operating System
 In Many Situations, it May Not be Possible to Upgrade the
Device to New OS Version
 Same Device Models may have Multiple OS Configurations
based on Target Market Requirements
 Localization
 Hundreds of Devices with Many Variants of Each Device

 Testing in All Web Browsers
 Browsers Significantly Vary Across Devices
 May Support HTML, xHTML, WML, AJAX
 Web Content Transcoders
 Web Content may be Transformed to Meet User-Agent
Constraints
 Transcoder may Not Respect User Experience Factors in
some Situations

 Inconsistencies in Specification Implementations
 Each Vendor is Responsible for Java ME Specifications
Implementations
 Implementations may Not be Consistent across Devices and
Vendors
 Performance Measurement
 Extremely Difficult to Benchmark Performance due to Highly
Fragmented Market
 Detection of Display Issues
 Human Involvement is Must
 Generally it is Not Possible to Purchase All Target Handsets
due to Financial Constraints

 Application Usability Testing
 End Users may Not even be Skilled at Using a Mobile
Device
 End User may Use Application in Multiple Contexts
 Driving in Rush Hour
 Walking in Crowded Market
 Multiple Input / Output Modalities
 Diverse User Interface Styles
 In Reality, A Testing Lab Cannot Simulate all Possible
End User Contexts

 Development and Deployment Platforms are Different
 Development Platforms
 Windows, Linux
 Deployment Platforms
 iPhone, Symbian, Android, Blackberry, Windows Mobile
 Emulators Do Not Capture All the Characteristics of a
Device
 Multiple Design Specifications may be Required for
Single Application due to Device Fragmentation—
Requires Test Cases Accordingly

Testing Techniques: General
Gather Useful Data
 Decide Target Handsets as Early as Possible in
Development Lifecycle (Better in Requirements Analysis
Phase)
 Document Incompatibilities in Devices e.g. Differences in
 Installation Processes
 Audio/Video Capabilities
 Memory, Processing Power, Screen Size etc.
 Document Device Software (Firmware) Limitations
 ‘Bugs in Firmware’ are Not Unusual
 Document Device Web Browser Constraints
 Supported Version of HTML
 AJAX Supported or Not

Gather Useful Data
 Better Acquire Meta Data of All Target Handsets from an
External Source
 Extensive Databases Containing Device Meta Data
 WURFL—An Open Source XML Database of Device Characteristics

 DeviceAtlas—A Proprietary Database of Device Characteristics

 Some Websites can Capture and Return HTTP Headers
 For instance, Access the Website using a Phone, Website will Return
Headers in HTTP Requests
 Useful when Data is Not Available in Databases and also to Verify
Already Available Data in Meta Data Source
 A Development House may also Maintain an Internal
Database Containing Data Not Available Otherwise

Classify Devices

 Classify Devices
 Category—1: All Devices that Support XHTML
 Category—2: All Devices that Support AJAX
 Category—3: All Devices that Support Java ME
 Pick Representative Device(s) from each Category for
Testing Purposes
 Prepare Test Cases based on Data Gathered about
Target Handsets

Perform Testing on Real Devices

 Start Testing on Real Devices as Early as Possible
 There is No Good Alternative to a Real Device for
Testing

Emulator is Useful but is Not a Real Device
 Testing on Emulators
 An Emulator is a Software Application that Imitates a Smartphlone
 A Device Emulator may be Used to Test Application on Desktop
 Some Emulators Emulate General Characteristics of a Platform (like
Java ME Emulator)
 Some Emulator Emulate Relatively Faithful Representation of Specific
Phones (BlackBerry Dev Kits)
 Emulator may or may Not Require Compilation for Target Platform
 Single Compiled Code for both Emulator and Target Devices
 Java ME
 Windows Mobile
 Different Compilers for Compilation for Symbian Emulator and Target ARM
Device
 Useful for Quick Testing in Development Environment
 Emulators are Not Real Devices
 Testing on Emulator does Not Guarantee Application Success on Real Device

Application UI Should Preferably be Consistent with
Device UI Style

 Test Application User Interface Consistency with
Device Interface Style
 Is the Application User Interface Consistent with User
Interface of Underlying Device Applications?
 Does the Application Offer Interaction Mechanism
Similar to Interaction Mechanism Generally
Implemented in Native Device Applications?
 Softkey Mapping
 Look&feel

Different Test Cases are Required to Cater Diverse
Input / Output Modalities
 Perform Testing for Various Input Modalities
 Touch Screen
 Testing with Single and Multiple Touch Input
 Testing with Virtual Keyboard
 Trackball/Track Wheel
 Numeric Keypad
 QWERTY Keypad
 4- or 5-Way Navigator
 Perform Testing for Various Output Modalities
 Different Screen Sizes

Test Applications in Real User-Contexts
 Test the Application in Relevant User Contexts
 User Context Involves User, Application and
Surroundings
 Design Various User Interaction Scenarios in Applicable
Contexts (user driving in rush hour, walking, in
classroom etc.)
 Test Applications in Real Contexts

Mobile Ecosystem Requirements must be Addressed
 Application are Generally Launched through Various
Application Stores (Mobile Ecosystem)
 Apple App Store, Nokia Ovi, Blackberry AppWorld, Android
Market, GetJar, Handango etc.
 Test Applications to Meet Requirements of various
Ecosystem Stakeholders
 Manufacturers/Vendors
 UI Guidelines from Apple for Iphone Applications
 Symbian Signed
 Operators
 Operators may also have some Requirements
 Third Parties
 Java Verified Process

Testing Techniques: Web Applications
Rules-Based Testing
 Rules-Based Testing with User-Agent Impersonation
 Create Testing Rules based on
 Device Capabilities
 Known Bugs in the Device
 Device Data in Data Sources (like WURFL, Internal Database)
 Application Requirements
 Test Cases are Written that Checks HTTP Response for Targeted
Content
 Impersonate a Device and Communicate with Web Server
 Change HTTP Headers like
 Accept: image/gif, image/jpeg
 User-Agent: ‘NokiaN95…….+Profile/MIDP-2.1 + …..’

 Check the Server Response Against Rules and Identify
Discrepancies, if any

Rules-Based Testing

 Rules-Based Tests
 Testing is Automated for Multiple Types of User-Agents
 Testing can be Done Using
 An HTTP Library for Sending Requests and Receiving
Response
 Web Browser (FireFox, Safari, Internet Explorer)
 Useful to View the Rendered Screen with some Limitations

 Rules Based Testing is Useful for Automated Testing of
 Web Applications that Deliver Content on Phone
 Regression Testing
 Web Content Accessibility

Transcoder Behavior must be Thoroughly Tested
 Transcoder Behavior
 Web Content may be Transformed by a Transcoder to
Meet User-Agent Constraints
 A Transcoder Resides Between User-Agent and Web
Server
 Cookies should Not be Shared
 Use Automated Tests for Communicating with Server in
Parallel
 User Experience Should Not be Affected
 Manual Testing—See and Record the Behavior

W3C mobileOK Standard

 W3C has Defined mobileOK Tests
 Tests for Checking Mobile Readiness (W3C Best
Practices) of a Website
 http://www.w3.org/TR/mobileOK-basic10-tests/

 W3C mobileOK Checker
 Performs mobileOK Tests on a Web Page to Determine
its level of Mobile-Friendliness
 http://validator.w3.org/mobile/

Testing Techniques: On-Device Applications
Unit Testing and On-Device Debugging

 Unit Testing Frameworks Exist for almost All
Platforms
 GUI Layer, Event Handling and System Testing is Not
Possible
 On-Device Debugging is Available on Most of the
Platforms

Custom Test Applications may Reveal Major Issues
 Custom Test Applications
 Can be Used to Test Critical Aspects
 Whether an Application has Access to Device File System
 Performance of Device Database System
 Write Applications to Diagnose Key Issues
e.g., Develop an Application that Helps in Testing
When
 Lot of Free Memory is Available on Device
 Very Limited Memory is Available on Device

Network Availability and Bandwidth
 Perform Application Testing in Different Network
Signal Strengths
 No Network
 Intermittent Connectivity
 Strong Network Signals Presence
 Application Testing May be Required in Various
Network Types
 2G, 3G

External Events may Affect Application Behavior

 Test Application against Events that may Occur during
Application Installation and Execution
 Phone Call Received
 SMS Received
 Battery Removed
 Device Switched Off
 Bluetooth Device Requested to Contact

Combine ‘Automated Input’ with ‘Manual Verification’
 Generate Automatic Events
 Key Press
 Pointer Press
 Touch
 Capture Screenshots during Event Generation Process
and Compare them

Track Behavior when App is being Used by Consumers

 Track Application Behavior during Various Lifecycle Stages
when in Production
 Installation
 Execution
 Removal
 Application may Send Important Data to Server that can be
Used to Identify and Fix some Bugs
 Java ME Includes some Parameters for this Purpose
 ‘GetJar ’ Offers Such a Service
 Application can Call Server and Report some Information
 http://www.getjar.com/

Testing Techniques: Commercial
Get Online Services for Devices Not Available In-house
 Testing on Real Devices that Developers Don’t Own
 A Service that Enables Application Testing on Real Devices
 Devices are Located on Remote Servers
 Devices are Connected to Various Operator Networks Worldwide
 Desktop Client Software/ Web Application Enables All Testing
Features Remotely
 Device Acquisition, Cleanup
 Application Installation, Execution, Removal
 Device Shutdown, Battery Removal and Installation

 Service Providers
 DeviceAnywhere (http://www.deviceanywhere.com/)
 Over 2000 Devices Connected to over 30 Worldwide Carrier
Networks across USA, Canada, UK, France, Germany, and Spain
 Perfecto Mobile (http://www.perfectomobile.com/)

Crowdsourcing is also an Option
 CrowdSourcing (Crowd + Outsourcing)
 Testing by Human Testers (Mobsters) Located Worldwide
 If Done Effectively, Real Usability Testing Results can also be
Obtained
 Testing in Real User Contexts is Possible
 How it Works?
 Developers and Testers are Registered with a Third Party Service
Provider
 Third Party Service Provider is Responsible for Coordination of All
Parties Involved in the Process (Application submission, monitoring
tools, results reporting etc.)
 Service Providers
 Mob4Hire (http://www.mob4hire.com/)
 GetJar (http://www.getjar.com/)
 uTest (http://www.utest.com/)

User Feedback is Important
 App Stores may Provide Useful Feedback from End
Users
 User Reported Bugs
 User Reviews on Usability
 User Ratings
 Data is Helpful for
 Quickly Fixing some Bugs
 Future Application Releases

Testing Tools: Web Applications
 HttpUnit
 Unit Testing for Web Applications
 HtmlUnit
 Unit Testing with JavaScript Support
 Urllib
 Provides APIs for Web Access
 Check a Web Application for Mobile Readiness
 mobiReady: http://ready.mobi/launch.jsp?locale=en_EN
 W3C mobileOk Checker:
http://validator.w3.org/mobile/

 User-Agent Capture Tools
 Captures Characteristics of User-Agent
 Content Type, HTML version Supported on Device etc.
 Pycopia: http://www.pycopia.net/webtools/headers
 FireFox and Others Browsers—Use User-Agent
Switcher Add-On. However, there are Some Issues
 Differences in how Desktop and Mobile Browsers Layout
Content may Render Testing Results Unreliable
 Performance Differences in Desktop and Mobile
Browsers
 Feature Differences in Desktop and Mobile Browsers

 Desktop Browser Automation Tools
 WebDriver (http://code.google.com/p/webdriver/)
 Java APIs that can be Integrated with Eclipse
 Selenium (http://seleniumhq.org/)
 Firefox add-on that records clicks, typing, and other actions to
make a test, which can be played back in the browser.
 Selenium RC can be Used in a Desktop Browser with HTTP
Header Emulation
 IPhoneDriver
 A WebDriver based Tool for Testing on UIWebView

Testing Tools: On-Device Applications
 iPhone
 Embedded Browser Component (UIWebView)
 iPhoneUnitTesting
 Memory Sweep (For memory snapshot of iPhone)
 Java ME
 J2MEUnit (For Unit Testing)
 JInjector (For System Testing)
 ASM (To Instrument the Code)

 Android
 Dalvik Debug Monitor Server (DDMS)
 Thread and Heap Information on Device
 Screen Capture on Device
 Incoming Call, SMS, Location Data Spoofing
 Monkey Tool (for Both Simulator and Device)
 Generate Random Streams of User Events Click and Touch

 Windows Mobile
 Hopper
 Simulates Input Stress by Rapidly Sending Keystrokes and
Screen Taps in a Random Fashion

 Symbian
 Bullseye
 Bullseye Coverage is a code coverage analyzer for C++ and C that tells how much
of the application source code was tested. Code coverage analysis is useful during
unit testing, integration testing, and final release.
 EUnitPro
 EUnit Pro enables creation and automation of unit and module tests for Symbian
C++ Software.
 Symbian AnalyseHeap Utility
 This tool analyses the contents of Symbian OS heaps to produce meaningful
representations of their structure and content.
 TRK (Target Resident Kernel)
 Enables On-Device Debugging
 Yucca Tools
 File browsing , Crash Monitor, Supported Fonts Preview, Memory Status etc.

Mobile Application Testing

Recommended

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Mobile Application Testing (20)

Recently uploaded (20)

Mobile Application Testing