AI and Software Quality:

Trends and Executive Insights

Why building trust in AI is essential to
ensuring end-to-end software quality
CONTENTS

Executive summary 3

Key findings 4

Everyone is integrating AI, and many are doing

so in untested waters 5

Is AI-augmented testing the solution to

emerging challenges? 7

Will testing still require humans in the future? 9

Investment in AI-augmented testing on the rise 11

How AI can be used to validate AI applications

for increased trust 12

Building trust in AI with Leapwork 13

2 AI and Software Quality: Trends and Executive Insights

Executive summary

In 2024, AI transitioned from an emerging technology Leapwork spoke to 401 senior and technical
to business-critical. Companies across industries professionals in the US and UK, half of which were
have rapidly integrated AI applications into their C-Suite executives, to understand two critical
operations to enhance efficiency, innovation, and components of the new AI era in the context of the
customer engagement. However, as AI becomes digital enterprise:
more deeply embedded in these processes, its
limitations and the challenges it introduces are 1 Testing of AI: How businesses will build trust
increasingly apparent. in the AI applications they are integrating

This report delves into these challenges, particularly

in the modern customer journey, where AI 2 Testing with AI: How businesses can improve
the efficiency and effectiveness of their
applications like chatbots are pervasive, and where
software testing
end-to-end quality is more important than ever.

While AI is widely adopted across sectors, This report provides decision-makers with a
concerns about its reliability, accuracy, and overall comprehensive overview of the current state of AI
effectiveness continue growing. These concerns and software quality, offering essential insights and
have been amplified by high-profile system failures solutions for businesses to adapt and consistently
over the past year, exposing critical vulnerabilities deliver exceptional user and customer experiences
in software systems. As a result, businesses are at scale, now that AI is a critical part of the equation.
increasingly re-evaluating how technologies are
tested, maintained, and trusted to deliver the
expected quality consistently.

At the same time, companies are increasingly using

AI-augmented testing tools to ensure applications
perform as expected across the customer journey.

This raises several questions: Can we fully trust AI

to address concerns about the very technology it
is meant to test? And how will AI-augmented tools
impact the role of humans in testing and shape the
Quality Assurance teams of the future?

3 AI and Software Quality: Trends and Executive Insights

Key findings
TESTING OF AI TESTING WITH AI

AI applications pose opportunities Businesses are turning to AI-

but also risks across the business augmented testing tools to solve
journey quality challenges

There is widespread adoption Only 16% believe their current

of AI, with 85% of organizations testing practices are efficient.
having integrated AI
applications into their tech
stacks in the past year.
Recognizing these inefficiencies,
However, 68% of those 79% of companies have already
have already encountered adopted AI-augmented testing
performance, accuracy, and tools, and 64% C-Suites trust
reliability issues. their results – technical teams
trust them even more (72%).

74% plan to further invest in

these tools within the next
There is a critical need for AI year.

testing
78% of companies see testing
AI as essential, which aligns
with the majority already The continued need for human
encountering significant issues.
oversight in AI-driven QA

However, 30% don’t believe their Despite advancements in AI-

current testing processes are up augmented testing tools, 68%
to the task of ensuring reliable of C-Suite executives believe
AI apps, indicating a substantial human validation will continue
gap in effective AI testing to be essential for ensuring
practices. quality across complex systems.

The adoption of AI-augmented

tools is not replacing human
roles but transforming them.
53% of C-suite executives
report an increase in new
positions requiring AI expertise,
emphasizing the evolving nature
of quality assurance in the AI era.

4 AI and Software Quality: Trends and Executive Insights

Everyone is integrating AI, and many are
doing so in untested waters

Amid a year of outages that have taken social Like any software, there is growing recognition that
media platforms offline, stopped fast food deliveries, AI applications need thorough testing to prevent
stranded passengers at airports, ceased bank business disruptions. An overwhelming 78% of
operations and postal deliveries, global conditions companies agree the AI apps need better testing;
are ripe for a potentially rocky AI revolution. in fact, 77% of C-Suite executives said testing is
critical to ensuring their performance, accuracy, and
With 85% of companies having integrated AI into their
reliability.
tech stack in the past year, there’s cause for concern
that the number of IT failures is on the rise. Already, “For all its advancements, AI has limitations, and I
68% of companies have faced performance and think people are coming around to that fact pretty
reliability issues with their AI applications. quickly,” says Robert Salesas, CTO at Leapwork. “The
rapid automation enabled by AI can dramatically
“There have been too many outages this year alone,
increase output, but without thorough testing, this
many of which affected millions of customers for
could also lead to more software vulnerabilities,
big brands. We’ve been given a wake-up call no one
especially in untested applications. It makes sense
can ignore,” says Christian Brink Frederiksen, CEO of
that C-Suite executives would be especially sensitive
Leapwork. “What makes digital infrastructure today
to this because of the implications for customer
so tricky to test is the copious amount of complex,
experience and negative publicity.
interconnected applications. A tiny error in one
application could have a monumental cascading
effect and shut down businesses.”

Have you adopted and integrated AI applications into your

technology stack in the last year?

No
15%
Yes Yes Yes
83% 88% 86%

Technology/ Finance Healthcare

Software and Banking
Count: 401

Yes Yes Yes

90% 92% 74%
Yes
85% Manufacturing Retail Government

5 AI and Software Quality: Trends and Executive Insights

What is the most common bug or issue you’ve encountered
with your AI applications?

Integration Security Incorrect System errors Performance

failures vulnerabilities AI-generated lags
responses

C-Suite 22% 22% 16% 11% 18%

Technical leads 19% 25% 18% 21% 12%

There’s an opportunity here for cross-industry Despite the urgent need for reliable AI, only 16%
collaboration to ensure more testing tools are up of companies believe their testing processes are
to scratch for the challenges of the modern world efficient.
where AI apps are more and more widespread.”
This reveals a troubling gap in quality assurance
For now, AI has limitations, and integration failure in that raises an important question: If today’s testing
particular was the most popular issue cited by 22% methods are falling short, how can the industry
of C-Suite executives. ensure that AI delivers on the promised benefits?

Integrating AI apps is a problem for companies

because of three main reasons: a resistance to
change within the company (20%), the difficulty
of managing the rapid pace of AI advancements
and updates (19%), and crucially, inconsistent
performance and reliability of AI applications (19%).

Unfortunately, sizable gaps remain for existing

testing resources and practices. For starters,
24% do not have a dedicated team or individual
responsible for testing AI apps, and 26% do not have
a commercial testing platform. Nearly a third (30%)
say outright that they do not believe their current
testing processes can ensure reliable AI apps.

6 AI and Software Quality: Trends and Executive Insights

Is AI-augmented testing the solution to
emerging challenges?

As companies grapple with the inherent bugs started adopting AI-augmented testing tools – an
and limitations of AI, a consideration emerges: overwhelming 79% now use them.
the potential of AI-augmented testing tools to
The fact that so many also trust the results of
effectively tackle the unique challenges posed by AI
these tools indicates an understanding amongst
applications.
early adopters of their possibilities and limitations.
While AI-augmented testing tools are gaining traction Especially as trust is even higher amongst technical
across industries, their true potential in enhancing AI leaders (72% vs. 64% for C-Suite).
reliability and performance is yet to be fully realized.
But a closer look at the survey results reveals gaps
Leapwork’s findings reveal that the trust is there -
between industries: there’s significantly more
68% of overall respondents say they trust the results
trust placed in AI-augmented testing within the
that AI-augmented tools provide - but isn’t there
technology industry (80%) than in retail (53%).
a paradox in using AI to address concerns about
the very technology it is meant to test? This is a “With retail, it’s easy to think about the mega
crucial aspect we’ll explore further in this report, as retailers and forget about the smaller boutique
leveraging these tools could be key to mitigating the vendors who have less familiarity with AI,” says
risks associated with AI integration. Salesas.

One thing is clear: To overcome the shortcomings of

existing testing processes, many organizations have

Do your testing processes currently incorporate AI-augmented

testing tools?

No
21%
Yes Yes Yes
81% 85% 76%

Technology/ Finance and Healthcare

Software Banking
Count: 401

Yes Yes Yes

87% 79% 74%
Yes
79% Manufacturing Retail Government

7 AI and Software Quality: Trends and Executive Insights

I trust results that AI-augmented testing tools provide

Strongly agree Agree Neither agree Disagree Strongly disagree

nor disagree

Technology/Software 35% 45% 19%

Finance and Banking 38% 28% 23% 7%

Healthcare 28% 43% 17% 10%

Manufacturing 29% 29% 26% 11% 5%

Retail 26% 27% 36% 7%

Government 23% 23% 46% 8%

0% 20% 40% 60% 80% 100%

“Not every retailer is tech-first like Amazon, and Still, as trust in these tools grows, it naturally raises a
there’s likely a cultural gap at play here: tech broader reflection on how adoption of AI-augmented
companies are at the forefront of AI development testing tools will affect the role of human testers in
and implementation, which means they get first the long term.
dibs on the talent who is more likely to have a
deeper understanding of the tools’ capabilities
and limitations. On a practical level, there are
also still many retail operations that rely on older
systems that don’t integrate seamlessly with AI-
augmentedtesting tools, and the stakes of failures
are high when any errors can directly impact
customer satisfaction and sales. Retail environments
themselves can be enormously diverse and complex
– and the customers even more diverse – which
might be giving professionals in the industry pause
about trusting AI’s ability to test all situations
accurately.”

8 AI and Software Quality: Trends and Executive Insights

Will testing still require humans in
the future?

Like every industry and trade impacted by AI, QA these tools can enhance their roles rather than
teams now face the question, ‘what will happen replace them. On the other hand, C-Suite executives
to humans?’ Leapwork’s findings suggest humans are looking at business operations more broadly,
are unlikely to disappear from the testing equation with optimism about how technology can improve
anytime soon. In fact, over two-thirds of C-Suite efficiency. While their perspectives may differ, both
executives (68%) believe that testing will need human groups agree that human input will remain a critical
validation for the foreseeable future, and almost part of the testing process.”
every single IT Director (92%) agrees.
“I believe that the synergy between AI and human
“There’s always going to be some variation in how expertise represents a transformative partnership in
technical teams and C-Suite executives perceive software testing. AI tools can significantly enhance
the need for human validation,” says Salesas. “For IT efficiency, allowing technical teams to focus on
teams, there’s a natural concern about job security innovation and ideation rather than the repetitive
as AI tools evolve, but the focus should be on how details of testing in an increasingly complex software

I believe that testing will continue to need human validation

for the foreseeable future

Strongly agree Agree Neither agree Disagree Strongly disagree

nor disagree

CIO 28% 36% 30% 5%

CTO 25% 48% 24%

Vice President of IT 29% 43% 27%

Director of IT 38% 54% 4%

IT Manager / Lead 28% 40% 18% 5% 9%

0% 20% 40% 60% 80% 100%

9 AI and Software Quality: Trends and Executive Insights

environment. However, no matter how advanced new roles specifically for people with AI skills, but
the tools become, the principle of requiring human almost as many (43%) have seen a reduction in roles
oversight and independent review will always be due to efficiency gains. C-Suite executives are much
essential to ensure accuracy and reliability.” more bullish, though, than technical leaders.

Interestingly, one sector is an outlier: manufacturing. Over half of C-Suite executives (53%) say the tools
While most respondents still believe in human have increased the number of new roles compared
validation, the number is a lot lower (56%) than in to just over a third (36%) of technical leaders.
other sectors like technology/software (79%), finance These perceptions also vary based on sector: 52%
and banking (83%), and healthcare (85%). of respondents in healthcare are seeing new jobs
created compared to only 36% in government.
“Manufacturing is centered today around
About 57% of respondents in manufacturing report a
maximizing automation – it’s all about
reduction in jobs.
standardization, repetitive processes, efficiency,
and a strong desire to keep costs low. This could
explain why respondents in the sector perceive less
need for human intervention. It speaks to a range of
different priorities, regulations, environments, and
operational characteristics of these sectors. Finance
and healthcare come with considerably strict and
unique compliance and safety requirements that
may be nudging those sectors towards a stronger
preference for human supervision.”

Whether or not AI-augmented testing will directly

impact headcounts is a question senior leaders
are still debating. Nearly half (45%) of overall
respondents say AI-augmented testing has created

How has the introduction of AI in testing changed your

team`s structure and roles?

Created new roles Reduced number Minor Significant

specifically for people of roles due to reorganisation reorganisation
with AI skills efficiency gains

C-Suite 53% 45% 32% 33%

Technical leads 36% 41% 38% 34%

10 AI and Software Quality: Trends and Executive Insights

Investment in AI-augmented testing on
the rise

No matter how AI impacts human testing roles, the may require further skills and financial investment to
trend is clear: AI-augmented testing tools continue fully take advantage of.”
to gain popularity. Most organizations (74%) foresee
increased investment in AI-augmented testing tools “I believe that a critical component to swaying
in their organization in the next year. The degree to executives who are unconvinced about the value of
which they agree varies from role to role. For example, AI-augmented testing will be to present tools that are
most CTOs (77%) expect increased investment into the intuitive to use, making them accessible not just for
tools, but far fewer CIOs (58%) do. technical teams but also for business users. When the
skills gap is so wide, you can’t afford for things to be
“Generally speaking, CTOs tend to focus much more difficult to adopt.”
on emerging technologies and how to apply them to
the business. That goes a long way to explain why they This leads us back to the pressing question: If easy-to-
might advocate more strongly for AI-augmented tools learn AI-augmented testing tools are the answer to
and AI in general. They’re thinking about the long-term buggy AI apps, is there a paradox in relying on the same
vision and staying competitive in that future.” Says technology that’s already causing issues to test itself?
Salesas. “CIOs, on the other hand, are more about To resolve this potential paradox lies in understanding
the day-to-day IT operations – particularly the short- that AI-augmented testing tools are not simply another
to-medium term. They manage a far larger remit of layer of AI—In the right context, they can be used to
operations and competing priorities. This might make address specific shortcomings of AI systems by providing
CIOs more reluctant to take any big leaps on tools that rigorous, unbiased assessments of their performance.

I foresee an increased investment in AI-augmented testing

tools in my organization over the next year

No
6%

Yes Yes Yes

Neutral 58% 77% 75%
20%
CIO CTO Vice President
of IT
Count: 400

Yes Yes
Yes 87% 78%
74%
Director of IT IT Manager /
Lead

11 AI and Software Quality: Trends and Executive Insights

How AI can be used to validate AI
applications for increased trust

This is where Leapwork’s approach comes into play. For instance, consider an AI chatbot designed
Leapwork’s approach to using AI to test AI is highly to answer user queries. If asked, “Do you offer
effective and designed to build trust - not just in your international shipping?”—a question that could be
AI applications, but across all the applications that phrased in various ways such as “Can you ship to
make up your business processes. other countries?” or “Do you deliver overseas?”—the
AI Validate block ensures that every response, which
Understanding how AI can might also vary linguistically, still accurately conveys
the correct answer: “Yes, we offer international
validate AI shipping to select countries.” By automating this
Leapwork’s AI capabilities are designed to assess validation process, Leapwork helps you avoid the
how well generative AI handles specific tasks and pitfalls of AI unreliability, providing a robust framework
responds to user-defined prompts. It does this by for ensuring that your AI applications deliver the right
comparing the AI-generated responses against results every time.
predefined, human-crafted expectations. This
ensures that the outputs of your AI applications are
consistent, accurate, and aligned with your intended
results.

This process helps to catch hallucinations, where AI

might generate incorrect or nonsensical responses.
By comparing these outputs against independent
data, Leapwork acts like a second pair of eyes,
objectively verifying the AI’s responses without the
biases the original AI might introduce.

12 AI and Software Quality: Trends and Executive Insights

Building trust in AI with Leapwork

Leapwork’s AI capabilities The Leapwork Test

AI Validate Automation Platform
AI Validate compares AI-generated responses with Leapwork’s approach to AI-augmented testing
expected outcomes to ensure consistency and is essential for ensuring the reliability of your AI
accuracy, catching errors like hallucinations and applications. But the capabilities of Leapwork go far
verifying that outputs align with intended results. beyond testing AI applications. Leapwork helps you
deliver better outcomes for your business by ensuring
AI Transform that your software meets the highest standards of
AI Transform reduces test creation effort and cost by quality and reliability. Leapwork’s end-to-end test
standardizing input data formats. This allows for quick automation platform ensures that every aspect of the
content translation, simplifying the testing process with customer journey is validated, by testing across all your
efficient text manipulation. business applications. This comprehensive approach
guarantees that your software, including newly
AI Extract integrated AI apps, deliver reliable and high-quality
AI Extract reduces the effort needed to test generative experiences consistently.
AI and unstructured text use cases by automating the
extraction and formatting of data from API responses,
improving data accuracy in systems such as CRM. Book a demo today and discover how
Leapwork can drive end-to-end quality
AI Generate software in your business.
AI Generate creates realistic and varied datasets.
This allows for comprehensive testing that accurately
reflects the environments your AI systems will operate in,
ensuring they perform reliably under diverse conditions.

Start D365 Retail Login Read Excel Verify D365 Match... AI Validate Pass

Source type Input Input

Data file Expected Value from input

Product Value from Expected

AdventureW...
Number Value from Value from input

Range
Price Value from
“A1:C8”, sheet:”Sheet1” Not Valid Fail

Product

Number

Price

13 AI and Software Quality: Trends and Executive Insights

AI and Software Quality:
Trends and Executive Insights

Report methodology
The research was conducted by Censuswide who
gathered responses from 401 respondents across US
and UK organizations. These included 201 C-Suite
executives (CTO/CIOs) and 200 technical leads
(including VP of IT, Director of IT, IT Manager, software
engineering leads, QA test manager/director).

The respondents were surveyed across sectors that

include technology/software, finance and banking,
healthcare, manufacturing, retail, and government.
The respondents were aged 18-55+. 63% of the
organizations were large (501-5000 employees),
whereas 37% were enterprise (5001+ employees).

14 AI and Software Quality: Trends and Executive Insights