The 11 Principles

of Sustainable
Continuous Testing
BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 As release cadences continue to accelerate, more organizations are looking to shift their testing
left and embrace continuous testing. By testing at each stage of the Software Development
Lifecycle (SDLC), DevOps teams can meet the need for speed without sacrificing quality.

Yet there are additional issues to consider aside from testing the application functionality:
• 56% of critical dependencies are unavailable.

• 50% of time is spent looking for test data.

• 64% of defects occur in the requirements phase.

To truly adopt continuous testing, teams must develop a continuous testing program that looks
at the whole picture and take these other considerations into account.

This eBook will delve into the 11 principles of sustainable continuous testing. These principles
will focus both on testing application functionality and application performance & security.

www.blazemeter.com | 2 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 Contents

Part 1: The Challenges of Continuous Testing.............. 4

The Testing Pyramid................................................................... 4
The Testing Prequel: The Requirements Phase............................... 5

Part 2 : 11 Principles of Continuous Testing.................. 5

Vizualized Environments ............................................................ 6
Test Data Management............................................................... 7
Test Automation........................................................................ 7
Pipeline Orchestration ............................................................... 8
API Testing................................................................................ 8
Performance & Load Testing ....................................................... 8
Security Testing ........................................................................ 9
Acceptance of Test-Driven and Behavior-Driven Developement....... 9
Automated Test Generation ........................................................ 9
Requirements Engineering......................................................... 9
Feedback Loops ...................................................................... 10

Part 3: Measure the Impact of Continuous Testing.......10

Part 4: Summary .................................................... 11

www.blazemeter.com | 3 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 Background: API and unit testing are at the base of the
test pyramid. These tests are smaller and

The Challenges to shorter, and typically created and run by the

developer. Teams need proper environments
Continuous Testing and fast feedback to find and fix problems
Continuous testing is the practice of testing within unit and API testing without interrupting
across every activity in the SDLC to uncover the workflow.
and fix unexpected behaviors. Yet continuous
testing can also happen before development Additionally, developers depend on the
even starts. availability of environments that are stable,
reliable, and always on. Therefore, dev
Let us take a closer look at two areas where teams require mock services and test data on
continuous testing makes a large impact: demand to seamlessly progress their software
throughout the testing pyramid and in the development and testing.
requirements phase of the lifecycle.
As the SDLC progresses, teams start building

THE TESTING PYRAMID more integration tests which consist of

functional and non-functional testing types.
One of the largest challenges to achieving
When the release gets closer to production,
continuous testing is creating the right testing
the developers and test engineers perform
strategy. To ensure quality at speed, testers need
end-to-end tests which include the highest
to cover scenarios across the entire test pyramid
scale testing types across most relevant
and in the right proportions.
platform combinations.

End-to-End
Individual Test Site

(10N of 1m Web Tests)

Dev & Testing Cost

Integration
(10N+1 of 1s Service Tests)
Black Box

White Box Unit

(10N+2 of 1ms of Code Tests)

Software Testing Pyramid

www.blazemeter.com | 4 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 Having the ability to drive the complete
11 Principles of
continuous testing plan from unit through
API, integration, and E2E, is imperative to
Continuous Testing
the success and velocity of the agile teams. To successfully accelerate and improve a code
release process through continuous testing,
THE TESTING PREQUEL: teams must adhere to certain principles to
THE REQUIREMENTS PHASE ensure that quality keeps pace. To embed
Yet it is not just the testing itself that poses quality in applications, teams need to add more
a challenge. Currently, 64% of defects modern testing practices in the form of small
originate in the requirements phase of the quality checks performed throughout the
SDLC. Ineffective communication at this stage application pipeline. This would enable teams
can introduce defects into the lifecycle before to test small sections of code constantly.
a single line of code is even written.
Achieving continuous testing does not just mean
focusing on test automation. Even after your API
Very often requirements are specified at a very
tests and GUI tests are automated and integrated
high level that is good enough for developers
into a continuous integration agent, those tests
and testers, but no one else. Without having
have dependencies that need consideration.
the domain subject matter expertise to fill in
Teams must satisfy the dependencies on test
the gaps in the requirements phase, team
data, interfaces, and environments before they
members are at risk of interpreting the same
can be executed. You must:
instruction differently.
• Remove environmental constraints.

On the flip side, requirements can be specified • Virtualize any interfaces you do not own
in an overly detailed fashion but take too long or control, or do not want to test.
to decipher and maintain over time. Not to • Utilize ephemeral testing environments.
mention developers and testers find it hard to
• Automate test data provisioning and
grasp so much information bundled together
management to feed your tests on-demand.
in a giant requirement.
• Automate your tests so they can run
against your virtualized interfaces using
The requirements definition process must
the appropriate test data.
improve to be unambiguous, complete,
and testable. • Have your pipeline orchestration engine
set up in such a way that there is no need for
manual intervention during the above steps.

• Focus on complementary disciplines.

www.blazemeter.com | 5 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 Let us explore these 11 principles of continuous So how do you coordinate testing through
testing in more depth. multiple environments when they are either
not always available or too expensive to test
VIRTUALIZED ENVIRONMENTS against? Virtualizing those environments allows
you to test your code without having to worry
Continuous testing means both testing more
about other systems and environments that you
frequently and hitting multiple environments
are not changing.
more frequently. If these environments are
not constantly available for testing, teams
Making those environments ephemeral,
will encounter bottlenecks.
available on-demand through service
virtualization, removes that constraint from
Some environments are accessible through
your development lifecycle. You control these
APIs, while teams can access others through
environments; you can spin them up as needed
various messaging interfaces. Those
and they will always be available.
environments could be built with modern
architectures, while others are monolithic
Unlike basic mocks, true virtualized services
legacy client/server or mainframe systems.
or “mock services” can provide sophisticated
testing scenarios by returning unexpected or
negative responses, which allow chaos testing
by emulating real-world conditions. With
virtualized services, your application is truly
go-live tested.

Testing using mock services Sample mock services testing report using BlazeMeter.

www.blazemeter.com | 6 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 TEST DATA MANAGEMENT TEST AUTOMATION
Finding the appropriate test data is an ongoing In a fully orchestrated application pipeline,
struggle for testers. If testing is the biggest today’s scripts can still fail due to factors not
bottleneck in the SDLC, then test data is arguably related to the application code. So, no one
the biggest bottleneck in testing! You must have will trust the results.
the right data and the appropriate diversity of
data for positive and negative testing scenarios. While it may seem obvious, it is important to
You won’t find all that diversity in production. note that you need reliable test automation to
And production data comes with inherent achieve continuous testing. Organizations that
compliance concerns. achieve continuous testing release on a daily or
bi-weekly basis and run tests that cover a variety
Increasingly, teams are turning to synthetic of platforms, test cases, and user conditions.
data generation to streamline their continuous
testing. Synthetic data enables teams to be With such robust test coverage, organizations
confident that no personally identifiable who automate their tests minimize defects
information is at risk in the data they are testing. while the delivery pipeline focuses more on
innovation. They spend no more than 20%
of their time focusing on any backlog.

Related Reading:
Perforce Test Coverage Guide >>

When organizations can automate 90%

or more of their tests, they can rest assured
that their tests are running reliably in the
background. This reassurance gives the
BlazeMeter Test Data Generation
space they need to focus their efforts elsewhere,
such as manual testing in the areas that need
Plus, gathering data from production, it or coming up with different ways to test new
and condition or provision it properly before application features.
sending it to your tests is a cumbersome process.
Only with synthetic data can you keep up with
the speed your application pipeline requires.

www.blazemeter.com | 7 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 PIPELINE ORCHESTRATION By testing at the unit and API levels as much as
possible, teams can find defects at earlier stages
Successful pipeline orchestration is critical
of development and reduce the cost of fixing
to achieving continuous testing. As part of
these defects in the future.
any DevOps adoption initiative, it is unrealistic
to expect to get to continuous testing without
the reliability and speed of a standardized
and automated pipeline.

A pipeline orchestrator is an automated

workflow tool that will run all your automated
tests and fully integrate with code deployment
activities as it moves through the pipeline.
Understanding your pipeline in more depth
enables your team to integrate continuous
testing attributes within each stage.
API Testing with BlazeMeter
Your pipeline orchestration must be integrated
with your automation suite, and your team
PERFORMANCE AND LOAD TESTING
must understand how it works, how to interpret
results, and how to make it scalable. Everyone Even when applications are functionally
involved in testing should also have full visibility running properly, you must radically change
into what is being run through the pipeline the way you think about performance testing
with full transparency. to truly test continuously.

As you shift testing to the left, you are testing

API TESTING
earlier in the lifecycle with less application and
API testing enables teams to properly align
infrastructure components. While these tests
their testing efforts with the testing pyramid.
are smaller by nature, they are also much greater
To achieve continuous testing, teams must
in volume.
strengthen their unit and API testing while
reducing their reliance on UI testing, especially
Make performance and load testing available
for business logic testing.
to everyone across all Agile teams. Developers
and testers alike must be able to create a

www.blazemeter.com | 8 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 performance test and run it by themselves, By creating tests to ensure that the various
without having to send requests to anyone else. acceptance criteria for each user story are
met, teams can build more focused tests
SECURITY TESTING within the sprints and developers can focus
more on meeting business objectives. Over
Security testing must play a central role when
time, teams will define much more detailed
strategizing a transition to continuous testing.
acceptance criteria, which will require more
While often treated as an end-of-the-line
comprehensive testing.
concern, security deserves to be placed in
the continuous testing stream at the very start.
AUTOMATED TEST GENERATION
Developers often unknowingly create The activity of manually designing and
security vulnerabilities. Continuously writing manual tests or automated test scripts
testing security is not just about policing is a major bottleneck.
those vulnerabilities; it is also about helping
developers better understand the security Teams should automatically generate the
of their code to prevent them from creation new acceptance-level tests for manual and
those vulnerabilities in the future. Security must automated testing in the beginning of the sprint.
become a partner, a trusted advisor, and expert, Then, when developers start checking in and
giving developers the training and tooling that merging the first working pieces of their code,
can be measured downstream. teams can run automated tests immediately.
This automation allows developers to get faster
Currently, security and testing are sitting on feedback on the quality of their code as they
the wrong side of the wall from the developers. check it into source control.
Security should not be seen as a conflicting
with quality. Rather, your security, testing, and REQUIREMENTS ENGINEERING
development teams should work in partnership
You must include all SDLC stakeholders in the
with each other.
continuous testing journey. That means finding
a better way to communicate and collaborate
ACCEPTANCE OF TEST-DRIVEN on the requirements. The earlier you include
DEVELOPMENT AND requirements engineering into your continuous
BEHAVIOR-DRIVEN DEVELOPMENT
testing initiative, the smoother the continuous
Test-driven development (TDD) and testing journey will be in the long run.
behavior-driven development are great
complements to continuous testing.

www.blazemeter.com | 9 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 FEEDBACK LOOPS the sentiment of end users and add them
to a continuous feedback loop back to
Feedback loops are critical to successful
developers and the business.
continuous testing. Dashboards in real time.
Must be automatic and entire team must have
You must also be able to track how many
access. Feedback loops across the entire
defects you have in pre-production and
SDLC, not just production, should be used as a
production, and pair this knowledge against
compass to help you navigate your continuous
the speed at which you can deploy. As people
testing transformation.
write increasingly better code, teams should
see the number of defects drop against a

How to Measure constant release cadence. Or better yet,

against an increasing release cadence.
the Impact of
Continuous Testing In assessing and measuring quality versus
quantity, factor in the following:
Every project requires metrics to determine
• Time to market.
progress and quality, and to identify business
• Value stream analysis.
value. This is no different for continuous testing.
• Timelines and costs of manual test

Some of the overarching benchmarks of execution.

determining progress and quality include • Regression testing.

customer adoption, engagement, and revenue.
Teams should seek to understand and quantify

Sample Performance Testing Report

www.blazemeter.com | 10 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)
 Start shifting left to developers by RELATED RESOURCES
democratizing testing tools and evolving testers
• Continuous Testing Meets Contract
into engineers. Every leader could and should
Testing—How to Shift Left in the
have their own view of what constitutes the
Cloud-Native Era
most valuable continuous testing metrics.
• BlazeMeter University:
Ideal metrics to track include:
Become A Continuous Testing Ninja
• Lead times.
• What Can Perfecto & BlazeMeter
• Releases.
Do For You?
• Production incidents.

• Business value.
ABOUT BLAZEMETER
A continuous testing plan must plan for future
The BlazeMeter Continuous Testing
aspects of development in an era of unabated
Platform is a complete solution for shift-left
change and speed. From the increased role
continuous testing. The platform includes
of AI to the Internet of Things, teams must use
UI functional testing, user experience
predictive analytics to ensure their applications
testing, API testing and monitoring,
are ready for the new technologies that are on
performance testing, test data, and
the horizon.
virtual services.

Summary All capabilities are deeply integrated in

an intuitive workflow designed for agile
Achieving continuous testing is a journey. teams and provide robust support for
But the principles of continuous testing popular open source tools. Delivered
discussed in this eBook will help you make in software as a service with support for
the journey a successful one. multiple clouds or private cloud, it is
simple to get started, intuitive to use, and
As teams become more comfortable with a powerful tool for delivering innovation
these continuous testing principles, your with quality and speed.
entire organization will begin to benefit from
them but in the short and long term. Visit www.blazemeter.com for a free
account, to request a demo, or learn more.

www.blazemeter.com | 11 BlazeMeter by Perforce © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0520KS22)