Testing Applications—For the Cloud and in the Cloud

8/19/2015
1
© 2013 IBM Corporation
Continuous delivery of software innovation on hybrid cloud
Testing Applications
For the Cloud and in the Cloud
@Al_Wagner
IBM Technical Evangelist
awagner@ca.ibm.com
1
Why do we care?
In the Cloud; For the Cloud

8/19/2015
2
2
Delivering
in today’s DevOps world
3
DevOps – the three principles
 Left to Right – Continuous Delivery
 Emphasize the performance of the entire system
 Right to Left – Continuous Feedback
 Create the right to left feedback loops
 Culture of Experimentation and Learning
 Continually experiment, take risks and learn from failure;
 Understand that repetition and practice is the pre-req to mastery.
Continuous Delivery
Whole team approach;
No blame culture;
Encourage experimentation & learning
Continuous Feedback

8/19/2015
3
4
The 1st step is solving the “legacy” challenges
Before DevOps, test
teams were fairly
successful at keeping
up with development
velocity
Test teams need to
change their behavior
in order to keep up
with increased
development velocity
Traditional test teams
can no longer keep up
with continuous delivery
5
Today’s reality for Testers
Testers want to spend LESS time
ibm.biz/TestingSurveyResults
In the Cloud

8/19/2015
4
6
Virtualizing today’s test labs
Modern test labs consist of…
Test Data
Management
Service
Virtualization
Deployment
Automation
Dynamic Infrastructure
In the Cloud
7
Service virtualization
making the unavailable available for testing
 Virtual components simulate the behavior
of a service or application during testing
– System dependencies
– Unavailable/inaccessible services
– Costly 3rd party access fees
 Virtual components run on commodity
hardware, private cloud, public cloud
 Each developer and tester can easily have
their own test environment
 Developers and testers can continue to use
current testing procedures and tools
 Streamline creation of production-like
environments and negative testing
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components
In the Cloud

8/19/2015
5
8
Continuous deployment
automating testing includes automating deployment
 Consistent deployments and rollbacks of applications
 Orchestration of changes across servers, tiers and components
 Configuration and security differences across environments
 Clear visibility: what is deployed where and who changed what
• Multi-Tier Application Models
• Scalable distributed automation
• Quality gates and approvals
Deployment Automation
In the Cloud
9
In the Cloud
“yes...BUT!”
I still need networking
and computing
resources to run the
deployed software!
Dynamic infrastructure

8/19/2015
6
10
Clouds are rolling in
Clouds are not a commodity.
Cloud computing doesn’t come out of the sky.
Clouds comes from physical hardware inside brick and
mortar facilities connected by hundreds of miles of
networking cable.
And no two clouds are built the same way.
11
Clouds deliver networking and computing power with speed!
privatepublic
On-Premises IT
hybrid

8/19/2015
7
12
Cloud services save you time!
Infrastructure (IaaS) Platform (PaaS) Software (SaaS)
more consumer control more vendor control
13
Automated test execution in the cloud
Many organizations are looking to the clouds to execute:
 Functional Tests
 Drive execution of functional tests in datacenters around the world without having
to deploy physical servers
 Reduce latency issues and long running tests
 Performance Tests
 Drive load around from datacenters around the world to measure global user
experience
 Scale user load on demand
In the Cloud
• Setup; tear down test execution
environments on demand at lower cost
• Distribute test execution around the
world with ease
Launch test execution
Log execution results

8/19/2015
8
14
Deployment automation and cloud accelerates testing
 Virtualizing Hybrid Test Labs - deploy applications in hybrid clouds
 Simultaneous execution of test stages – integration, functional,
performance, security testing
 Eliminates scheduling delays
 Eliminates hardware delays
 Setup; Tear Down on demand
In the Cloud
15
Deployment automation and clouds eliminate test lab bottlenecks
15
Rational Test
Virtualization
Server
- virtualized services
CICS
Tx
SAP External
Services
DB
Service
Virtualization
Deployment Automation
Env1
Env2
Customers
DB
App
Server
SAP
CICS
PaaS
(private cloud)
External
Services
SaaS
(public cloud)
Off premise
Off premise
DB
App
Server
App
Server
SAP
CICS
PaaS
3rd Party
Service
3rd Party
Service
Database
CICSSAP
Physical devices,
emulators, devices
clouds
Development Test Production
On premise
In the Cloud

8/19/2015
9
16
Deployment automation and cloud accelerates testing
 Virtualizing Test Workstations - deploy pre-configured virtual
desktops in the cloud
 Eliminate need to install, configure, license and maintain
workstations
 Standardize on a test automation platform
 Increases productivity; reduces down time
 Setup, Tear Down on demand
In the Cloud
17
Virtualizing testing workstations
Image
Repository
Artifact
Repository
Deployment
Automation
Private
Request Cloud
instance
Requests
Testing
Workstation
Returns
Workstation
I/P address
Dynamic Infrastructure
Request
workstation
image
Request specific
workstation configuration
Remote access to
virtual workstation
Operating system
& software
Configuration
files
Test
Assets
In the Cloud

8/19/2015
10
18
For the Cloud
What is quality software
in the cloud?
Clouds can be quickly stood up in Data Centers
anywhere in the world. This capability can expose risk to
the business if not considered in the test effort.
You need to make sure the software being released is:
• Resilient
• Reliable
• Secure
19
Testing for high availability
• Does failover routing work as expected?
• Does the system automatically scale?
• Are databases redundant and backed up?
• Are faults logged & people notified of outages?
Data Center
Pod
Cluster
Server/Data
For the Cloud

8/19/2015
11
20
Testing for resiliency
Quality is not only measured by how well code survives during
a long test run, but how quickly normal operations can be
returned after an error.
Testers need to assume remote services:
 will respond slowly,
 will respond with an error,
 or won’t respond at all
The most important production issue metric is
not “root cause” but “time to recovery”.
For the Cloud
21
Patterns for the cloud - Retry
For the Cloud

8/19/2015
12
22
Testing Retry pattern
Service A Service B500ms (timeout)
500ms (timeout)
RetryCount=2RetryCount=1
Server Log
Operation Exception – (date/time)
….
Server Log
….
For the Cloud
Timeout = 500ms
Max # of Retries = 3
extends Retry
23
extends Retry
Testing Retry pattern
Service A Service B
Server Log
….
500ms (timeout)
500ms (timeout)
200ms (passed)
• Use a virtual service to simulate
“Service B” responses along with
timeouts and retry attempts
• Confirm log entries are captured
RetryCount=0
For the Cloud
Timeout = 500ms
Max # of Retries = 3

8/19/2015
13
24
Patterns for the cloud – Circuit Breaker states
For the Cloud
25
Patterns for the cloud – Circuit Breaker
For the Cloud

8/19/2015
14
26
Testing Circuit Breaker pattern
Service A Service B
For the Cloud
State: closed
Circuit
Breaker
success
Parameters
invocationTimer = 0.1
resetTimer = 60_000
retryThreshold = 1
Testing the “closed” state:
• Straight forward and easy to test
• Validate “closed” circuit breaker state using real
service implementation or virtual service
requestIDState = closed
getState(requestID) returnState()
27
Service A Service B
For the Cloud
State: closed
Circuit
Breaker
failure
Parameters
resetTimer = 60_000
Service A Service B
State: half open
Circuit
Breaker
timeout
testCall()
success
success
updateState(halfopen)
updateState(closed)
resetTimer()
success
executeCall()
Testing for “half open” state:
• Hard to test negative in test labs and production
• Virtual services can help simulate the necessary
“negative” responses required

8/19/2015
15
28
For the Cloud
Service A Service B
State: open
Circuit
Breaker
fast fail
Parameters
resetTimer = 60_000
retryThreshold = 1
requestIDState = open
getState(requestID) returnState()
Testing the “open” state:
• Little harder if Service B is deployed
• Validate “open” circuit breaker state
• Setting requestIDState to “open”
• Routing request to stopped virtual service
29
Testing feature toggles in a continuous integration world
(aka feature flags; dark launches)
Release Toggles
 Hides incomplete features (dark launch)
 Short lived (removed when feature deployed)
 Pre-decided values across environments
Business Toggles
 Enables A/B testing
 Release to small control groups of people to get
feedback
 Enable and disable on the fly
For the Cloud

8/19/2015
16
30
Release toggles
For the Cloud
//pseudo-code
featureToBeReleased = true;
If “featureToBeReleased” {
do this():
} else {
do that();
}
 Hide features from being exposed until development and testing
is “Done”.
 Remove feature toggle from code base across all modified
components.
 Check to ensure “feature” is hidden or available as intended.
Be sure to toggle the UI layer!
31
Business toggles
For the Cloud
//pseudo-code
checkUser() {
boolean testgroup = checkDatabaseForUserGroup(userID);
if (testgroup) {
accessNewFeature();
} else {
useLegacyFunctionality();
}
}
 Enables organizations to “test” features before releasing to the
entire user community. (A/B Testing)
 Test for all user groups.
 Need to determine what combinations of features need to be
tested. It could be difficult and time consuming to test them all.
Be sure to check the UI presentation as well!

8/19/2015
17
32
Continuous feedback delivers insight on software quality
Build a system that allows you to make decisions
based on real data instead of just market direction.
 User behavior
 Customer sentiment analysis
 Performance metrics
For the Cloud
33
Testing for security
 Data Breaches
 Encrypt the data exchanged between services
 Data Loss
 Deploy redundant systems and test backups
 Insecure APIs
 Develop and Test 3rd party access using authorization services
 Denial of Service attacks
 Your organization could be billed for all the cloud services
 Malicious Insiders
 Protect your encryption keys
 Shared Technology
 You are not alone in “public” clouds
 to name a few…
For the Cloud
Don’t forget to scan your code for insecure coding practices.
Penetration test your applications.
If you are hacked; ALL your customers are impacted!

8/19/2015
18
34
Remember
It is important to test:
 what a product/service does
 how a product/service performs
 as well as how that product/service is managed
For the Cloud
35
ibm.com/devops

8/19/2015
19
36
ibm.com/devops

Testing Applications—For the Cloud and in the Cloud

Recommended

More Related Content

What's hot (18)

Viewers also liked (6)

Similar to Testing Applications—For the Cloud and in the Cloud (20)

More from TechWell (20)

Recently uploaded (20)

Testing Applications—For the Cloud and in the Cloud