The Impact of

Continuous Testing
How Organizations Transform Their Testing
from Reactive to Innovative
Continuous testing is the preferred model for teams looking
What is Testing Maturity?
to create, test, and release high-quality applications today.
Testing maturity identifies where in the testing process steps and
If you are involved in the application development landscape,
methods can be improved upon. It is an indicator of how the testing
you know how important continuous testing is to ensure your
is designed, measured, managed, monitored, and how the results
applications not only meet expectations but exceed them.
are yielded.

Yet not all continuous testing strategies are created equal.

The benefit of using a testing maturity model is to continuously
Depending on where your team is in their level of continuous
refine and improve upon your team’s testing strategy to get the
testing implementation—and depending on where your
best application you possibly can. Without self-evaluation, you run
application is in the development lifecycle—your approach
the risk of your testing methods becoming stagnant and archaic,
to continuous testing may vary.
which will negatively impact the quality of your app.

This is where continuous testing maturity models come in.

Just as the integrity of your testing is important for the quality of
your app, so too is the structure of your testing maturity model
This guide will delve into continuous testing maturity models
important to the quality of your testing. Which brings us to setting
and their impact on different areas of business. It will also illustrate
up and establishing your model based on a few key pillars.
how adopting these continuous testing maturity models can help
take your organization’s strategy from reactive to
innovative, paving the way for more
successful applications that don’t just
meet expectations but exceed them.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

2 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
 PILLAR 3: INTEGRATED
The 5 Pillars of a Continuous
This pillar will see testing integrated into the software development
Testing Maturity Model lifecycle (SDLC). Tests become more organized,
Think of your testing maturity model as a two-phased approach — and objectives are determined based on priority. Testing is
the Capability phase and the Assessment phase. We will discuss the now considered an essential aspect of software development.
Assessment phase later, so for now let us focus and break down the This phase is also known as the Automated phase.
five key components also known as pillars of the Capability phase.
As the name suggests, the goals will vary depending on which level PILLAR 4: MANAGEMENT & MEASUREMENT
of capability your testing is at.
How do you know your tests are effective and doing what they are
supposed to? Implementing key measurements and quantifications
PILLAR 1: INITIAL of what a successful test looks like is the next step in your testing
This entry level pillar is to make sure the software runs successfully. maturity model. Reviews at each stage of the development process
This is the bare bones testing strategy, which lacks experienced are treated as tests. Regression testing becomes incorporated to
testers, testing tools, or necessary resources. There are no testing collect and record in a database for reference. Bugs are identified,
landmarks or processes established. Does the software work? Yes? logged, and given security levels based on severity. This phase is
Ship it — no QA checks are done prior. also known as the Orchestrated phase.

PILLAR 2: DEFINED PILLAR 5: OPTIMIZATION

The next step to take towards achieving testing maturity is to Now is the time to determine how well your testing process
establish debugging goals and policies. This means separating and is managed and defined. You should be tracking not only test
delineating testing and debugging to be conducted with intention. effectiveness but also how it impacts your budget. QA and
Testing should be done after coding to determine if the software defection prevention (as opposed to detection) are now used.
is meeting specifications at the project’s outset. This phase is also The team now has tools to support testing metrics, as well as test
known as the Basic phase. design and defect gathering. This phase is also known as the
Intelligent phase.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

3 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
Continuous Testing Maturity Models
Now that we have covered the 5 pillars of a continuous testing maturity model, let us take a look at some examples of maturity models in action.

FUNCTIONAL TESTING
Functional testing is the process of validating the functionality of a software application. Functional testing assesses whether the primary functions
of a software application are working and that they align with the overarching goals for the app. Functional tests result in ‘pass’ or ‘fail’ scenarios
because the features either work as designed or they do not.

Some examples of functional testing types include Unit Testing, Component Testing, Integration Testing, End-to-End Testing, Sanity Testing,
and Smoke Testing.

Functional Testing PHASE 1 PHASE 2 PHASE 3 PHASE 4 PHASE 5

Maturity Model Initial Reactive Proactive Optimizing Innovating

KPI: Platform Coverage 1 of each platform 15-29% 30-49% 50-79% 80-100%

KPI: Automation % ~10% ~33% ~66% ~75% 80%+

KPI: Automation Stability 0-29% 30-69% 70-95% 95%+ 99%+

Test new features only Add daily smoke Add scheduled sanity Add full regression
Quality Gates Add in-sprint testing
End of sprint only Add nightly regression Add smoke on every PR on every PR

Biannual / quarterly
Business Value Release every 2 months Monthly release Biweekly release Daily
release

Professional Services
Professional Services
SV and TD - required to
- Increase lab to meet Professional Services
Professional Services Professional Services shift left and perform
How Perforce Can Help desired feedback Introduce virtual lab to
Establish basic lab Increase lab size component level testing
window speed up feedback
advanced testing
Scriptless Mobile
introduced

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

4 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
The chart depicts the different phases (or pillars) of a functional You will see that in Phase 1 (or the Initial phase) of the Focus of
testing maturity model. Testing, load testing is done in an ad-hoc manner, with production
defects often getting reproduced. By Phase 5 (or the Innovating/
Take automation, for instance. As you can see, in Phase 1 (or the Optimization phase), load testing has become fully optimized to
Initial phase), only around 10 percent of the example is automated, the point where the team is pushing testing boundaries, performing
while in Phase 5 (or the Innovating/Optimization phase), at least chaos testing and UX testing to ensure the best
80 percent is automated. possible application quality.

In the Initial phase, teams are only testing new features and end When measuring the business value in this load testing maturity
of sprint only, while they can test a full regression on every pull model, in the Initial phase end users are already impacted and
request by Phase 5. the focus is merely on damage control. By the Optimization phase,
the ROI is achieving best-in-class applications.
LOAD TESTING
Load testing measures how well an application performs under
real-world conditions, such as peak traffic events like Black Friday
for retail apps. Load testing ensures that your application can
handle greater levels of usage and helps identify areas of potential
weakness in advance.

The chart on the next page depicts the different phases (or pillars)
of a load testing maturity model.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

5 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
 Load Testing PHASE 1 PHASE 2 PHASE 3 PHASE 4 PHASE 5
Maturity Model Initial Reactive Proactive Optimizing Innovating

Establish test suite

Test critical path before
Ad-hoc Performance and integrate to CI/CD Optimize test suite,
major events and Pushing the boundary
KPI: Focus of Testing Testing, Reproduce and isolation/ leveraging feedback
Performance Testing of testing
production defects component level testing from analytics
late in SDLC
as part of process

Leverage SLA
Defects identified Leverage SLA and
and benchmarks to Developers have
KPI: Performance Multiple sprints end of cycle, mostly benchmarks to identify
identify performance the ability to triage
Triage Response and hot fixes backlogged and work performance regression
regression within before PR
on prioritized items only during QA/UAT builds
development cycle

Add test on production

Add test on basic
like QA environment; Add component
QA environment;
Quality Gates Developer station only Add component level Add Production testing level testing as part
Establish early
test; Add Integrated into of PR process
performance baseline
CI/CD pipelines

ROI - Strategy is focused ROI - Focus on reducing ROI - Focus on

End users already ROI - Weaponizing
on risk mitigation, often responsiveness to increasing developer
impacted. ROI - Focus development as a
Business Value leading up to critical performance incidents productivity and driving
on damage control differentiator, achieving
events such as Black and increasing efficiency into the
best in class
Friday sales developer productivity process

Load Testing Load Testing

How Perforce Load Testing Load Testing Service Virtualization Service Virtualization Enterprise License
Can Help Professional Services Professional Services Test Data Test Data Agreement
Professional Services Professional Services

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

6 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
 The Impact of Continuous Testing Maturity Models
Now that we have delved into the details of functional and load testing maturity models, we will take
a closer look at the impact continuous testing maturity models can have on different areas of business, L4
starting with the business impact itself. INTELLIGENT
Optimization and
L3
BUSINESS IMPACT predictability
ORCHESTRATED
AND METRICS SUMMARY • Defect prevention

L2 Improved release agility • Test scoping and

• Test execution time
AUTOMATED maintenance effort
• Continuous testing a key • Test prioritization
L1 Improved dev/test agility;
part of DevOps pipeline
improved app quality • Dynamic test selection
BASIC • Defects due to
• Embrace shift left
Reduced cost; improved
L0 activities environment mismatch
code quality
INITIAL • Test automation level • Release quality: defect
• Requirements ambiguity
leakage to production
Unpredictable quality, • Well-defined automation
• Test design effort
high costs frameworks
• Test maintenance effort,
• Manual testing, minimal • Test execution time
change analysis effort
automation
• Test feedback response
• Test coverage
• High QA cost time to Dev
• Unit test coverage
• Depend on SME, • Defect leakage from
individuals to • Code quality
Integration
determine quality • Defect leakage from
• Test data provision time
• Non-scientific measures Dev to Test
• Test environment
of test and data coverage • Test data coverage
setup time
• Poor/unpredictable • Test environment
setup time
Capabilities

quality
• Start implementing shift
left activities

Maturity Level

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

7 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
BUSINESS IMPACT – INITIAL PHASE BUSINESS IMPACT – AUTOMATED PHASE
In terms of business impact, the Initial phase of a continuous testing In the Automated phase, there is an improvement in development
maturity model involves unpredictable quality and high costs. and test quality as well as an improvement in overall app quality.
Manual testing is the norm at this stage, with minimal automation. Shift left testing activities have been embraced, test automation
Rather than operating under the continuous testing model that has significantly increased, and test execution time has gone down.
gives every team member a role in ensuring quality throughout
the application development lifecycle, quality assurance is BUSINESS IMPACT – ORCHESTRATED PHASE
placed on individuals or specific SMEs—a much less efficient and
This phase is marked by an overall improvement in application
thorough approach that leads to poor and unpredictable quality.
release agility. Nearly everything is now automated, test execution
The measures of test and data coverage during the initial phase
time has significantly improved, and continuous testing has
are often un-scientific. These inefficient processes and lack of
become a key part of the DevOps pipeline. In addition, leaked
continuous testing lead to greater waste and higher costs.
defects are very rare during the Orchestrated phase.

BUSINESS IMPACT – BASIC PHASE BUSINESS IMPACT – INTELLIGENT PHASE

The Basic phase can be defined by reduced costs and improved
The final phase is marked by optimization and predictability.
code quality when compared to the Initial phase. It is also marked
Defects are prevented, there is a robust test scoping and
by an increase in test coverage, improved test environment setup
maintenance effort, and tests are properly prioritized. At this final
time, and an increased effort in test design and test maintenance
stage of the continuous testing maturity model, businesses can
efforts. Yet, there are often defects that still leak through to
trust in their process without worrying about quality assurance,
production (though they have decreased) as well as an ambiguity in
tests are fully automated, and resources are allocated properly
some testing requirements.
— ensuring the best quality applications and therefore the best
possible business outcomes.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

8 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
 PROCESS IMPACT AND METRICS SUMMARY L4
INTELLIGENT
Focus on predictive
L3 capabilities and optimization
ORCHESTRATED • Full testing/CD lifecycle
L2 Focus on CD integration data collection,
AUTOMATED • Greater level of test integration and x-lifecycle
automation ~60% data correlation in data
L1 Focus on CI integration
marts/warehouses
• Full automated
BASIC • Automated TDD
traceability -- • Use of advanced
• Automated test
L0 Focus on test optimization
automation from models –
Requirements models tied analytics, data correlation
• Requirements modeled
INITIAL GUI/APIs (F/P/S) ~40%
to releases, code, test and ML
in MBT tool data, virtual services and • Adaptive test (and
Mostly manual testing process • Automated testing tied
• Automated test design performance scripts automation) optimization
• Manual requirements to CI cycles
and unit tests • Full test process/ using ML on lifecycle data
definition and tracking
• Focus on In-sprint
• Some TDD/BDD lifecycle automation and • Self-healing tests assets
• Manual test design automated testing
• Some GUI/API test integration with (e.g. automation)
and execution
• Automated test data CD processes
automation (F/P/S) ~20% • Predictive testing
• Manual test data generation – sub-setting
• Scripted test data; • Focus on E2E automated capabilities, e.g. defect
management and masking
synthetic test data release testing prediction, adaptive test
• Manual builds, no CI • Data rich virtual services
• Code scans • Model-driven optimization
• Hand-coded • Modeled test env. test environment
• Algorithmic performance • Self-identifying A/B
virtual services definitions; automated management and
analysis (profiling) scenarios
• Manual test env provisioning and deployment of assets
• Recorded virtual services • DET, Test Data Science/
provisioning and deployments tied to CI
• Deployment and release Analytics skills
deployments • Scripted env provisioning • Automated release coordination across
and deployments promotion within complex release trains
• Manual release mgmt.
• Basic CI –
Capabilities

• Classic QA/manual release train • SDET/SRE skills

automated builds
tester skillset • QE/SDET skills
• Model-based tester (MBT)
and some QE skills

Maturity Level

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

9 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
PROCESS IMPACT – INITIAL PHASE PROCESS IMPACT – ORCHESTRATED PHASE
In the Initial phase of process maturity, the testing process is mostly During the Orchestrated phase of process maturity there is a focus
manual —from manual requirements and tracking to manual test on CD integration. There will be a greater level of test automation
design and execution and manual test data management. (around 60 percent)—including full automated traceability, full test
Virtual services are hand-coded and test environments are provided process/lifecycle automation and integration with CD processes.
and deployed manually. In addition, this phase includes manual SDET/SRE skills will be needed at this level.
release management utilizing a classic QA/manual tester skillset.

PROCESS IMPACT – INTELLIGENT PHASE

PROCESS IMPACT – BASIC PHASE
In the final phase of process maturity, there will be a focus on
In the next phase, there is a greater focus on test optimization.
predictive capabilities and optimization. Full testing CD lifecycle
Test design and unit tests are generally automated, there is around
data collection, use of advanced analytics, data correlation, and
20 percent API test automation, as well as some use of scripted and
ML, adaptive test optimization using ML, self-healing tests, and
synthetic test data. During this phase there will likely be a model-
predictive testing capabilities are the hallmarks of this final phase.
based tester (MBT) and some team members with some QE skills.
DET, test data science and analytics skills are incorporated into
this phase of the continuous testing maturity model.
PROCESS IMPACT – AUTOMATED PHASE
The Automated phase of process maturity is marked by a focus on
CI integration. At this stage, API test automation is around 40
percent, automation testing is tied to CI cycles, there is a focus
on in-sprint automated testing, and data rich virtual services are
utilized. There will be QE/SDET skills at this phase.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

10 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
 TECHNOLOGY IMPACT AND METRICS SUMMARY
L4
INTELLIGENT
Test platform integrated with
L3 analytics/ML platform
ORCHESTRATED • Data aggregation across

L2 TestpPlatform integrated lifecycle, use of intelligent

analytics
AUTOMATED with CD platform
• Tie with version control • Predictive analytics and
L1 Test platform integrated
solutions like Helix Core ML tools and frameworks
with CI platform
BASIC • A/B testing tools like • Machine learning for test
• TDD tools
Integrated test platform Optimizely optimization based on
L0 • Integration w/
• Unit testing and CI tools • X-release train lifecycle data
INITIAL APIM tools; API
• Synthetic test data artifact tracking • All test tools integrated
monitoring tools
Ad-hoc tools with
• Basic virtual services • All test tools integrated into analytics/ML
some integration • Sub-setting and masking
into CD platform platform with automated
• Requirements definition • API testing with of test data
centralized data collection
tools BlazeMeter • Data rich virtual services
• Spreadsheets/docs for • Perf testing and with dynamic data
test design, test data and automation tools handling
test env specification • Test environment • All test tools integrated
• Mocking/stubbing tools manifests defined in into CI platform
config tools tied to
• IDE, SCM tools and
deployment tools
build tools
• App release tracking
• Spreadsheet-based
release tracking
Capabilities

Maturity Level

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

11 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
TECHNOLOGY IMPACT – INITIAL PHASE TECHNOLOGY IMPACT – INTELLIGENT PHASE
When looking at continuous testing maturity model through a Finally, in the Intelligent phase of a technology maturity model,
technology-based lens, the Initial phase is often defined by a use the test platform is fully integrated with an analytics and ML
of ad-hoc tools with some (but minimal) integration. Spreadsheets platform. CDD/CDI tools are used for data aggregation across
are used for test design and release tracking, and mocking/ the lifecycle, predictive analytics and ML tools and frameworks
stubbing tools will be utilized. are in place, and machine learning is used for test optimization.
Automated centralized data location is also in place.
TECHNOLOGY IMPACT – BASIC PHASE
The Basic phase will include an integrated test platform, unit testing Bottom Line
tools, some test automation tools, basic virtual services, API testing,
Whether you are in the Initial phase of your maturity model or you
CI tools, and app release tracking.
have integrated many of the processes of the latter stages of the
maturity model, Perfecto is ahead of the curve when it comes to
TECHNOLOGY IMPACT – AUTOMATED PHASE helping teams fully adopt a continuous testing maturity model.
During the Automated phase, the test platform will be integrated
with the CI platform. TDD tools (such as Gherkin and Cucumber) From banking and finance, retail, healthcare, automative, and
and automation tools (such as Selenium) are used, as are data-rich telecom—to any other industry—Perfecto is your go-to partner
virtual services, API monitoring tools, as well as sub-setting and for continuous testing.
masking of test data.
Experience Perfecto in action today with our free trial or
TECHNOLOGY IMPACT – ORCHESTRATED PHASE by watching one of our product demos.

In the Orchestrated phase, the test platform is integrated with the

START TRIAL WATCH DEMO
CD platform. A/B testing tools are utilized, and artifact tracking
is in place.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

12 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)
ABOUT PERFECTO
Perfecto powers exceptional digital experiences by combining
the power of flexible test authoring, cross-platform execution, and
AI-driven intelligent testing into one secure, cloud-based web and
mobile testing platform.

Visit www.perfecto.io for a free trial, demo, or to learn more.

Perfecto by Perforce © Perforce Software, Inc. All trademarks and registered

13 | www.perfecto.io
trademarks are the property of their respective owners. (0720KS24)