Exercising and Scaling Up Mobile DevOps in the Enterprise

The Mobile DevOps Company
Ville-Veikko Helppi
Head of Demand Generation
ville-veikko.helppi@bitbar.com
Exercising and Scaling Up
Mobile DevOps in the Enterprise
WEBINAR

Mobile DevOps
•  The ‘Mobile DevOps’,
Process and Tools
•  Infrastructure & Architecture
•  Tips & Tricks for Scaling Up
•  Troubleshooting Device Lab
•  Demo
•  Summary – Q&A
Agenda
More information
about Mobile DevOps
can be found at
bitbar.com
© Bitbar Technologies 2017 – h7p://bitbar.com

•  On-Demand Mobile Device Infrastructure
•  Support Any Development & Testing Framework
•  Automate, Automate, Automate – To Save Time, Money and Hassle
•  Provide Unprecedented Scale and Throughput
•  Allow For Creative Use (Open API, Customizable Integrations)
•  Adopt Mobile Development Thought Leadership and Expertise
Support all development stages:
Prototyping Development Testing Production Monitoring
Successful Adoption of Mobile DevOps

•  Modern mobile app development is Agile, but
infrastructure does not support Agility well
•  DevOps practices are emerging in to the
intersection of Development, QA and Operations
•  DevOps practice emphasizes the collaboration
and communication of both software developers
and operations while automating the process of
software delivery and infrastructure changes
•  DevOps practice establishes a culture and
environment where building, testing, and
releasing software, can happen rapidly,
frequently, and more reliably.
Mobile DevOps and Agile
Development QA
Operations
Dev
Ops

SPEED WINS
It’s a simple fact. The faster you can deliver new applications, features and
upgrades to market, and collect feedback, the better your business
performs. For that you need higher development pace. And a lot of
automation.
THERE IS A DEMAND FOR MOBILE DEVOPS
DevOps practices and tools have revolutionized the pace and quality of
service delivery – but only for traditional server-based apps.
FRAGMENTATION WITH TOOLS
In mobile the delivery pipelines are more complex – in terms of tools and
used methods. Cohesive and well-integrated development tool flow
derives from Mobile DevOps practices.
MANUAL DOING IS NOT EFFECTIVE
Every stage requires a lot of manual work and silo’ed solutions from
various vendors and/or open source tools.
Why Mobile DevOps

Delivery Pipeline for Mobile Apps
Development
Source Code
Management
SCM
Build Test Deploy Production
Dev Tools:
•  IDEs
•  JIRA
•  Slack
•  etc.
SCM:
•  Git/GitHub/
GitLab
•  Bitbucket
•  MSFT
•  etc.
Continuous
Integration
and Build
Automated
Testing
Automation
for app store
deployment,
beta testers
& enterprise
app stores
Production
Monitoring
and feedback
to Dev

Delivery Pipeline with Mobile DevOps
Development
Source Code
Management
SCM
Build Test Deploy Production
Dev Tools:
•  IDEs
•  JIRA
•  Slack
•  etc.
SCM:
•  Git/GitHub/
GitLab
•  Bitbucket
•  MSFT
•  etc.
Continuous
Integration and
Build
Automated Testing Automation for app
store deployment,
beta testers
& enterprise app
stores
Production
Monitoring and
feedback to Dev
The Scope of Mobile DevOps

Mobile DevOps - User Experience
Development
/IDE SCM
Build
Automated
Testing
Deploy
/ Release
“Production”
Automated builds
for every code
change
Built binaries
automatically tested
Tested applications,
Screenshots for various devices
and languages, certificates,
metadata ready to push to App stores
Deploy
To Devices
Test apps in
real world
(Synthetic)
Apps Used by Users
•  Synthetic Monitoring
•  Real User Monitoring
Beta Testing
(Humans)

Exercising Mobile DevOps
Test
Deploy
Monitoring
SCM
Development
Build

Continuous Integration
CONTINUOUS
INTEGRATION

The ‘Shift-Left’ – What People Think It Is
Shi>-Le> TesAng
(happens earlier)
TradiAonal TesAng
(happens before release)

The ‘Shift-Horizontal’ Is The Way Forward

Hardware & Software for
Efficient and Scalable
Mobile DevOps

•  Parallelism, and Lots of Real Mobile Devices (Plug-and-Play)
•  Mobile Test Automation Frameworks and Appropriate Use of Those
•  Easy to Adopt, Deploy and Use Admin Configurability
•  Use of Virtual Machines & Containers (Cloud, On-Premise)
•  Open API and Ability to Integrate with Any Tools
•  Efficient Use of Resources
Support all development stages:
Prototyping Development Testing Production Monitoring
What Enables Scalability?

Three Deployment Options for
Test Automation with Devices
Public Cloud Private Cloud
On-Premise
Reserved &
User-Speciﬁc
Devices
Shared & Publicly
Available Devices
Hosted by Vendor
Hosted Locally

UNPRECEDENTED SCALE
By automating the creation, validation and deployment of the containers we
are able to deploy large number of identical device servers in matter of
minutes.
ALWAYS CLEAN EXECUTION ENVIRONMENT
New pristine container is started for each test session so that the environment
is always identical from one session to another.
FAST AND EASY TO UPDATE AND MAINTAIN
Any updates need to be made only to the scripts that create the container
images and the changes are orchestrated to all device servers
ALLOWS RUN TIME CUSTOMIZATION OF THE ENVIRONMENT
Because the tests are executed inside the container we can allow users to
customize the environment any way they like during the test session.
Why Containers
Host
Container 1
Container 2
Container 3
Container n
…

Because the tests are executed inside the container we can allow
users to customize the environment any way they like during the
test session.
This is done by using the run.sh script that is used to launch the
the container session.
Any shell command or tool that is installed on the container can be
called from run.sh script. For instance you can build anything with
Maven:
mvn clean test –DexecutionType=serverside
or install some library or tool with Brew
brew install python
Customizing The Environment
Host
Container 1
Container 2
Container 3
Container n
…

#!/bin/bash
startAppium(){
if [ "$(uname)" == "Darwin" ]; then
startAppiumOSX
elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
startAppiumLinux
else
echo "Unknown OS system, exiting..."
exit 1
fi
}
executeTests(){
echo "Extracting tests.zip..."
unzip tests.zip
if [ "$(uname)" == "Darwin" ]; then
echo "Running iOS Tests..."
mvn clean test -Dtest=IosAppiumExampleTest -DexecutionType=serverside
elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then
echo "Running Android Tests..."
mvn clean test -Dtest=AndroidAppiumExampleTest -DexecutionType=serverside
fi
echo "Finished Running Tests!"
cp target/surefire-reports/junitreports/TEST-*.xml TEST-all.xml
}
startAppium
executeTests
Example of run.sh Script
Host
Container 1
Container 2
Container 3
Container n
…

Example: iOS Virtual Machine Components
1.3.6, 1.4.3, 1.4.13,
1.4.16, 1.6.0, 1.6.3
+ Appium_Helper:
1.3.6, 1.4.3, 1.4.13
0.20.4

Test AutomaAon Frameworks
Dependencies and ‘Middleware’
iTunes
Programming Languages Core Tools

Example: iOS Virtual Machine Components
1.3.6, 1.4.3, 1.4.13,
1.4.16, 1.6.0, 1.6.3
+ Appium_Helper:
1.3.6, 1.4.3, 1.4.13

WGET
ideviceinstaller
Ios-webkit-debug-proxy
usbmuxd/iproxy
0.20.4
AddiAonal
frameworks
2.4.9
8.x / 7.x
Settings & Config to VM
•  Disable software updates,
Disable screensaver, Disable
sleeping, Disable Spotlight
mds agent, Remove dock
items, Disable hibernation,
Remove sleepimage
•  Remove swap files, Zero out
free space, Shrink disk
•  Set up /etc/hosts, Remove old
Keychains, Import keychain
•  Create ~/Library/MobileDevice
directory, Create ~/Library/
MobileDevice/Provisioning
Profiles directory
•  Import mobileprovision

Mobile Test Automation Frameworks
XCTEST/
XCUITEST
UIAUTOMATOR ESPRESSO APPIUM CALABASH
ANDROID No Yes Yes Yes Yes
IOS Yes No No Yes Yes
MOBILE WEB Yes Limited to x.y clicks No Yes No
LANGUAGE Obj-C/Swift Java Java Almost any Ruby
TOOL FOR TEST
CREATION
Xcode UIAutomator
viewer
Hierarchy
viewer
Appium.app CLI
(Human-Readable syntax)
COMMUNITY Apple Google Google Community Community

•  Easy to setup & easily extendable
•  Time-independent
•  Backward compatibility (supporting
various API levels)
•  A custom Instrumentation TestRunner
with special privileges
•  Works on any API levels
•  Thin layer on top of Android
Instrumentation Framework

•  Integral part of Xcode, works with both
real devices and simulator
•  XCTest allows developers to write tests
for components at any level
•  XCUITest provides rich UI testing
capabilities for apps
•  Objective-C / Swift
•  Nearly all frameworks (Appium, EarlGrey)
are working in conjuction with XCTest
•  Bitbar Testing products support XCTest +
XCUITest with all deployment options
XCTest /
XCUITest

•  Different Deployment Options
o  Hosted in various locations (data centers)
o  Co-location
o  On-premise
•  Infrastructure Support
o  Wireless carrier support
o  Rooted devices (optional)
o  Support for hardware add-ons (i.e. Bluetooth,
Wearables, etc)
•  Security (seriously!)
o  Isolated network
o  Dedicated hardware resources
o  VPN Support
o  OAuth/SSO
What Also Helps
Scaling Up?

Private
Cloud /
Enterprise
Support
•  Network Test Support
o  Network Performance Optimization
o  Charles Proxy
o  Fiddler
•  Performance
o  Built-in CPU / Memory tracking
•  Graphics Benchmarking Integration
o  FPS, GPU utilization, battery usage, etc
•  Value-Added Services
o  API Implementations
o  On-demand scripting
o  Video recording
o  Custom reporting / formatting

iOS Device Cluster Server
Mac OS 1.9 or later, i5 or greater CPU, 4GB RAM
or 16GM Minimum for Parallel Device Runs,
256GB Hard Disk Minimum
Android Device Cluster Server
Ubuntu Linux 12.04, i5 or Greater CPU, 4GB RAM
Minimum, 500MB Hard Disk, 10 Port USB 3.0 Hub
or 1USB 3.0 Port per Device, Motherboard with
BIOS that supports turning off USB Power Saving.
Up to 10 Android devices per Machine.
Web Front End Server
Ubuntu Server 12.04 LTS 64-Bit (Virtual or
Physical), 4-Core CPU Minimum, 16GB of RAM
Minimum, 500GB Hard Disk Space, 1GBit
Ethernet Connectivity
Database Server
Ubuntu Server 12.04 LTS 64-Bit (Virtual or
Physical), 4-Core CPU Minimum, 8GB of RAM
Minimum, 2TB Minimum of Hard Disk, Storage in
RAID Array, 1GBit Ethernet Connectivity
Other Equipment To Consider
Server Racks, USB Cables, PDUs, WiFi Routers,
UPS
Hardware Infrastructure
n

•  Devices
o  So many things can go wrong …
•  Infrastructural Hardware
o  Device Control Hardware
•  ADB and Instruments/Tools
•  Hard Disks/Storage
•  RAM
o  USB
•  Hubs
•  Cables
•  Network
o  WiFi
o  Cellular
•  Software
o  Jenkins/CI
o  SCM
o  Test Automation Frameworks
80%
The Most Typical Problems
10%
5%
5%

The most typical problems
•  Device doesn’t show up
o  CI/On-Premise Solution/etc
•  Device doesn’t react
•  Device doesn’t start tests/run
•  Device is out of battery/battery level low
•  Device shows notifications
Troubleshooting Mobile Devices
Daily Check-Ups for Mobile Devices
In order to maintain device farm, environment and all
aspects of SW & HW, the following daily checks are
recommended:
•  Are devices powered up?
•  Are all devices WiFi connected?
•  Do all devices have an active USB connection?
•  Are battery levels at least 50% (for every
device)?
•  Are there any pop-ups, system notifications, or
other system dialogs open on device screen?
•  Are screens unlocked?
We recommend checking every device 3 times/day.

Too Many Devices Connected /
Device Server
•  Too large device cluster
•  Maximum of ...
o  3 iOS devices per Mac Mini
o  10 Android devices per Linux server
•  Enough harddisk/storage for all servers
We recommend checking & cleaning
harddisks (with a shell script) every day.
Typical Mistakes (and How To Tackle)
Device Server Check-Ups
In addition to checking logs on server machines, it’s
highly recommended to check following things:
•  Diskspace on all machines (script)
•  Available RAM (script)
•  Do all devices have an active USB connection?
•  Available and consumption of CPU for peaks
•  Network conditions
We recommend automated scripts for checking
above-listed items every day.

Network Not Working Properly
•  Need to check if network is working
•  Network for ALL connected devices
•  Take WiFi down and up – try connection
with your browser
We recommend checking network connection
6-9 times every day.
Typical Mistakes (and How To Tackle)
Other Software Not Working Properly
Again, check all logs, that contain important
information about the system and health of the
environment:
•  Syslog (on all server machines)
•  Catalina out (on application server)
•  Jenkins logs (on device servers)
•  MySQL errors (and their logs)
We recommend checking these items every day.
In addition, it’s recommended to implement alarm
script to notify about any occurring problems.

PROBLEM: If none of the devices connected to one device server are running
tests (and you can’t initiate any manual session)
SOLUTION: Majority of issues seen with non-running tests are related to Jenkins
•  Check Jenkins log (https://hostname:8080/log/all)
•  Restart Jenkins – https://hostname:8080/safeRestart
•  Restart the device server
Troubleshooting - Mobile Devices
PROBLEM: Device is not running any tests.
SOLUTION: First, try if you can establish a manual test session with any of your
devices. If not, follow these steps:
•  Check that device is powered and visible on Jenkins
o  https://hostname:8080/android-devices or http://hostname:8080/ios-devices
•  Check that device is available in Bitbar On-Premise
o  https://cloudservername:9080/#service/admin/configuration

PROBLEM: Tests fail without running the test script successfully. Fail can happen
at init or any phase of the script execution.
SOLUTION: Find the exact error and follow these steps:
•  Retrying test run is a good idea to see if the error is random or reproducable
•  Check test run and device run views for logs and errors
•  Check console.log for the test run
•  Common failures: 1) app installation failed, 2) test zip structure is invalid, 3) Xcode didn’t start (iOS)
Troubleshooting - Mobile Devices
PROBLEM: None of devices are running any tests.
SOLUTION: Start with logs. In all cases, the logs contain information about errors,
wrong behavior and root cause of issues can be found in those. Start with these:
•  First check application server log (catalina.out)
•  Check available diskspace
•  Then restart Application server (sudo reboot now)

Troubleshooting – Mobile Devices

Test On Real Devices Automation -
Quickly and Efficiently
Use Existing Assets Now
and Future Tools
Platform Designed For
Agile Development
•  Test real world scenarios
on real, non-emulated
devices with 100%
certainty
•  See exactly what the user
experiences
•  Reduce time and
headache of device
acquisition and
management
The Key Things to Boost Mobile DevOps
•  Use any automation
framework or test
exercisers (e.g. Testdroid
App Crawler) to get instant
reports and results
•  Hit the ground running and
get actionable results on a
wide range of devices
•  Run more tests across
more devices in less time,
with fewer resources
•  Bring your own hardware,
tools and frameworks and
the Bitbar team will
integrate them into the
cloud
•  Bitbar solutions will not
lock you in to any tool or
automation framework
•  No vendor lock-in means
that you have a long term
and future proof platform
for all of your mobile
development and testing
needs
•  Bitbar integrates
seamlessly to your
development tools and CI
process
•  Instant results give
immediate feedback to the
development team
•  Faster iterations mean
more efficient
development, faster time
to market and more
satisfied users

Summary – Q&A
More information about mobile app testing,
mobile monitoring and mobile devops at
bitbar.com

Exercising and Scaling Up Mobile DevOps in the Enterprise

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Exercising and Scaling Up Mobile DevOps in the Enterprise (20)

More from Bitbar (8)

Recently uploaded (20)

Exercising and Scaling Up Mobile DevOps in the Enterprise