AI-assisted Software Testing (3-hours tutorial)

© Vehid Geruslu (Vahid Garousi)
Vahid Garousi
AI-powered Software
Testing
(A hands-on tutorial – 3 hours)
1- Software Testing Strategist / Consultant
Testinium A.Ş., Türkiye
ProSys MMC, Azərbaycan
AzInTelecom, Azərbaycan
2- Professor of Software Engineering
Queen’s University Belfast, UK
vgeruslu
vehid.geruslu@gmail.com
www.vgarousi.com

The plan for: AI-powered Software Testing: Hands-on tutorial (3 hours)
● Introduction
○ About the speaker
○ History and evolution of software testing
○ Testing + AI: Two combinations:
■ Testing AI = Testing the AI

itself versus
■ AI Testing = Using AI to in software testing: to increase effectiveness, efficiency
■ Two separate issues and fields
● AI-powered Software Testing
○ What testing activities can AI help with, and how?
○ Insights and lessons learned from using AI in testing
○ Prompt engineering: The importance of the commands (‘prompts’) that we
give to the AI (LLM)
● Two categories of AI-powered testing tools
○ “General-purpose” AI testing tools: Like ChatGPT
○ “Special-purpose” AI testing tools: 80+ tools already in the market and new
tools coming to the market constantly …
● Conclusions:
○ Insights and implications: for software testing jobs, …
○ The future of this field…
Topics will be presented in a hands-on manner and with live
examples using ChatGPT
2
⏰ ~ 15 minutes
⏰ ~ 20 minutes
⏰ ~ 45 minutes
⏰ ~ 100 minutes
= 1.5 hours
⏰ Total = 180 minutes
Practical
exercises

About the speaker
Education:
PhD in Software Engineering, Carleton University, Ottawa, Canada, 2006
MSc in Software Engineering, University of Waterloo, Canada, 2003
BSc in Software Engineering, Sharif University of Technology, Tehran, 2000
1998-2013
2013- now
2019- now
2022- now
● I have been working in Software Engineering, Testing and QA
since 1998
○ (I wrote my first Test Automation code in 1998 using the IBM
Rational Functional Tester tool, to test an e-commerce web app)
● Origin: Azerbaijani Turkish
● Nationality: Canadian, British
● I have lived and worked in 4 different countries since 2000
● Work experience:
3
My area of

expertise (in short):
Improving software engineering processes and activities of software companies
...and increasing effectiveness / efficiency / quality

History and evolution of Software Testing, as a field
4
Many software companies around the world have started using artificial
intelligence (AI) in their testing and quality assurance activities...
Let's not be left behind!

Testing + AI: Two combinations…
5
System / component
/ model under test =
Aİ / ML models
System under test (SUT) =
Any software system
Aİ Testing:
Use of AI in software testing activities
Also called:
AI-powered testing
AI-powered testing
AI-augmented testing
AI-driven testing
AI-powered testing
…
Testing of Aİ
Testing of AI models
Testing of AI-enabled software
(software with one or more major AI component)
Tests (assesses accuracy /
precision of)
Tests
Getting
support
Aİ / ML models
/ LLMs
Purpose: To get support from AI in software testing
activities. Mainly: To increase effectiveness and efficiency
(and quality?) of testing
Purpose: To test the AI

and ML
models / software and verify that
they work correctly.
Or: AI-enabled
software

And an advanced emerging field 🚀: Testing AI using AI
6
Tests
(assesses accuracy and
precision of)
– and even improve !
An AI model / system Another AI model / system

Our plan  AI-powered Software Testing: Hands-on tutorial (3 hours)
● Introduction

itself versus
■ AI Testing = Getting AI to help with testing
○ Prompt engineering: The importance of the commands
(‘prompts’) that we give to the AI (LLM)
○ “Special-purpose” AI testing tools: 80+ tools already in the
market and new tools coming to the market constantly …
● Conclusions:
7
⏰ ~ 15 minutes
⏰ ~ 20 minutes
⏰ ~ 45 minutes
⏰ ~ 100 minutes
= 1.5 hours
Practical
exercises

What testing activities can AI assist with and how?
To analyze this, we need to refresh our knowledge of:
● SDLC = Software Development Life Cycle
● STLC = Software Testing Life Cycle
8

What testing activities can AI assist with and how?
● In almost all testing activities… 🤩
● (but the level of help from AI in different test activities can vary)
● (and the level of help could increase as we are advancing in time !)
9
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
³ Exercise´
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
System Under Test
(SUT)
Activity
Data/
Entity
Legend
Test-
Result
Reporting
Test management (planning, «
Test-Data and
Test-Environment
Management
Other test engineering activities (regression testing, test minimization, etc.)
M T
M T
M T
M T M T M T
M T M T
Manual AI tool
Tool (without AI)
Let’s add more details
to the STLC / process

AI-powered Software Test Management (Test Planning, etc.)
Important point: As software test engineers,
we should definitely review carefully what we
get from AI
10
Live demo +
practice
Creating Test Plans:
Prompt: My task is to test the following mobile banking app:
https://play.google.com/store/apps/details?id=com.pozitron.isc
ep
https://www.isbank.com.tr/iscep
For functional manual testing of this app, provide a draft test
plan document. (Türkçe cevap ver lütfen 🌎 )
Risk Analysis
Prompt: For the same mobile app, perform a detailed Risk
Analys, so that I can add to the test plan document

AI-powered Test Planning: Time for hands-on practice …
● It’s now your turn …
● Prompt: For functional manual testing of the following
mobile app: X, provide a draft test plan
One suggestion  But feel free to use any mobile app idea
● Lessons learned?
○ What are the advantages / benefits of using GenAI for Test
Planning?
○ What are the disadvantages / drawbacks of using GenAI for Test
Planning?
● Or (you can practice with these later):
○ Asking ask for draft test plans for Functional “automated” testing
○ Asking ask for draft test plans for Automated performance testing
○ Asking ask for draft test plans for Automated security testing
○ …
● Some other app ideas, as Systems Under Test (SUTs).
You can use GenAI to test them later 
11

AI-powered Software Test Management (Test Planning, etc.)
● In AI-powered Software Testing, we often need to have an iterative
interaction with the AI / LLM
● … to get the most help that we want, from the AI
12
Just like we ask our
colleagues (other software
engineers)
Yes, this is what I
wanted!
Is this what I wanted?
Can I get a better result?

AI-powered Software Test Management (Test planning, effort estimation, …)
Test effort estimation
● Prompt: For the same mobile app, for the Functional Manual Test Plan
above, perform test effort estimation
Importance of our prompts, to the GenAI:
● If we issue a general command, and we do not specify the type of effort
estimation approach, the AI

will choose any method
● But if we ask specific questions, we will get specific and often better / more
precise answers from the AI.
● Just like we ask our colleagues (other software engineers)
13
Live demo

AI-powered Software Testing: Let’s move to another testing activity
14
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
³ Exercise´
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
System Under Test
(SUT)
Activity
Data/
Entity
Legend
Test-
Result
Reporting
Test-Data and
Test-Environment
Management
M T
M T
M T
M T M T M T
M T M T
Manual AI tool
Tool (without AI)
What we will now learn
What we just learned

AI-powered Software Test-case Design
● What is test-case design (or just test design)?
● Various types (forms) of test cases?
○ tests in tabular format, automated test-code, etc.
● Example: We can issue the prompt to the GenAI
in different ways:
15
Test case design is a systematic /
structured approach to creating
test cases, outlining the test
inputs / steps / conditions, and
expected results to verify if a
software feature or functionality
is working as intended.
Which input fields?
We always need to give the
RIGHT direction to the AI tool
Conclusions:
• For many years to come, we still need to be in FULL CONTROL of how we
benefit from AI in our Software Engineering activities, and how it
generates artifacts for our tasks
• … until AI becomes capable enough to make near-zero mistakes!
• We don’t know when that will be: in 5, 10, 20 years?

AI-powered Software Test-case Design
● For test-case design, we also need to consider another important concept: coverage
● How many types of coverage do we have?
● In testing, there are two types of coverage: Test coverage, and code coverage
○ Prompt: I want you to do test-case design based on test (requirements) coverage. For this, I suggest you generate
requirements for the same above mobile app (İşCep) in form of user stories. Then use those requirements to
generate test cases that "cover" (test) all requirements. You can use tabular format for presentation, if that is
suitable. Respond in English
■ Option for later: Respond in Turkish.
○ Prompt: For the same İşCep Mobile Banking App, as the next activity, I want you to do test-case design based on
code coverage, this time.
■ For brevity, let’s only do this for the “Login” feature. To see code coverage in this example, generate back-
end code in Java for Login feature.
■ Then use generate test cases in tabular format to fully "cover" (test) the code. Note: I want test cases, and
not test code (yet)
■ When you design test cases in the process, provide details of which lines of code are covered by which test
cases. And explain how you are adding new test cases to cover the uncovered lines of code so far. Respond
in English
■ Option for later: Respond in Turkish. All code and variables / comments in it should also be in Turkish 16
Live
demo
Test coverage Code coverage
Percentage of requirements
tested during testing %
Percentage of code executed
during testing %

AI-powered Software Test-case Design (Exploratory Testing)
● But Exploratory testing ≠ Random (monkey) testing
● AI-powered Exploratory Testing: Do you think AI
could do / support Exploratory Testing?
○ AI cannot yet replace human test engineers in full execution
of Exploratory Testing, due to its current limitation of learning
“context”
○ But that could change in a few years 
○ For now, AI can support test engineers in Exploratory Testing.
Let’s ask the AI itself!
17
● Exploratory Testing: let’s recall what it is…
○ A common approach: using Test Charters 
How can you help in Exploratory testing in general?

More details on: AI assisting with or itself doing Exploratory Testing
18

AI-powered Software Testing: Auto generation of test-code from designed test-cases
● Prompt: For the same İşCep
Mobile Banking App:
○ I want you to “script” (write)
automated test code in Java
Selenium, using the test-cases that
you designed in the previous
interaction, for the “Login” feature
○ No need for POM.XML or other info.
Just give the Java Selenium code
○ (Code elements should all be in
Turkish)
● Insights / observations :
19
Test-case
Design
Test
Scripting
Test
Execution
Test Suites
(set of test
cases)
Scripted
test suites
³ E
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
Other test engineering activities (regression testing, tes
M T M T
M T M T
Manual AI tool
Tool (without AI)
Canlı
demo

● Prompt: Can you run each of the above manual or automated test
cases on the SUT?
● Prompt: For the “Login” feature of the same İşCep Mobile Banking
App:
1. First generate prototype back-end (production) code in Java
for the Login feature
2. Then do white-box test-case design to derive the list of test
cases (in tabular format), to have full “branch" coverage on
the code. Note: I want test cases, and not test code
3. Then, generate Selenium Java test code for those design
test-cases, clearly matching each test method to each
designed test cases
4. Run (in your simulation mode, obviously not on the actual
app) the Selenium Java test suite on the prototype back-end
code for the Login feature, and report the test output
Show details of each step as you proceed. Answer in all English
20
Test-case
Design
Test
Scripting
Test
Execution
Tes
Evalua
Test Suites
(set of test
cases)
Scripted
test suites
³ Exercise´
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
System
Test-D
Test-Env
Manag
Other test engineering activities (regression testing, test minimiza
M T M T M T
M T M T
Manual AI tool
Tool (without AI)
Canlı
demo
AI-powered Test Execution

AI-powered Test Execution
● As we saw, as “general-purpose” AI tool, GPT
does not help much in this test activity
● Prompt:
○ Show the names of a few AI-powered Test “Execution”
tools and their features
○ Note: I am not referring to AI tools which do test-code
maintenance (such as the self-healing feature), but
only those AI-powered tools that help in "execution" of
tests, running tests intelligently.
○ Explain each feature briefly 
● Note: Two categories of AI-powered testing
tools (we will explore this topic later in the tutorial)
○ “Special-purpose” AI testing tools: 80+ tools already in
the market and new tools coming to the market
constantly …
21

Prompt: For the same İşCep mobile banking app as the SUT:
○ If we want to test real SUT, the mobile app on an actual phone
○ What kind of other help can you provide for the “Test Evaluation” phase of the STLC? see attached
Prompt: For the same İşCep app as the SUT:
○ For the Test Evaluation phase, give one single concrete example of the help that you can provide
Let’s focus on Test flakiness…. A flaky test is a software test that yields both passing and failing
results despite zero changes to the code or test.
Prompt: Test flakiness is widely discussed in the testing industry. For the same İşCep app:
○ In the Test Evaluation phase, how would you help us identify and prevent test flakiness?
22
Canlı
demo
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
³ Exercise´
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
System Under Test
(SUT)
Activity
Data/
Entity
Legend
Test-
Result
Reporting
Test-Data and
Test-Environment
Management
M T
M T
M T
M T M T M T
M T M T
Manual AI tool
Tool (without AI)
AI-powered Test Evaluation

Prompt:
○ In the login page of a university online service…
○ I have received the following error message, which I believe is a bug
○ My question: Do you agree that it is a bug?
○ If you agree that it is a bug, generate a detailed bug report?
23
Canlı
demo
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
³ Exercise´
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
System Under Test
(SUT)
Activity
Data/
Entity
Legend
Test-
Result
Reporting
Test-Data and
Test-Environment
Management
M T
M T
M T
M T M T M T
M T M T
Manual AI tool
Tool (without AI)
AI-powered Reporting of Test Results and Defects

© Vehid Geruslu (Vahid Garousi) 24
Test-case
Design
Test
Scripting
Test
Execution
Test
Evaluation
Pass
Fail
Test Suites
(set of test
cases)
Test
Results
Bug
(Defect)
Reports
Scripted
test suites
³ Exercise´
(test)
Manual test
suites
Automated
test suites
(e.g., Junit)
Criteria-based
(Systematic)
Human
knowledge-based
(Exploratory)
System Under Test
(SUT)
Activity
Data/
Entity
Legend
Test-
Result
Reporting
Test-Data and
Test-Environment
Management
M T
M T
M T
M T M T M T
M T M T
Manual AI tool
Tool (without AI)
AI-powered Test-Data Management
● Test Data Management (TDM) is the process of …
○ planning, creating, storage, maintaining and deleting datasets used in software testing
● Prompt: For the same İşCep app as the
SUT:
○ Explain, using concrete examples, the
assistance that you can provide in test-data
management

AI-powered Test-Environment Management
● Test Environment Management (TEM) is a crucial process that ensures Test
Engineers have access to functional, stable, and usable environments for testing,
bug replication, and overall software quality improvement
25

AI-powered Test-Environment Management
● Prompt: For the same İşCep app as the SUT:
○ Explain, using concrete examples, the assistance that you can provide in Test-Environment
management
● Prompt: Define a comprehensive device matrix, for Test-Environment management
26

● Introduction

itself versus
○ Prompt engineering: The importance of the commands
(‘prompts’) that we give to the AI (LLM)
● Conclusions:
27
⏰ ~ 15 minutes
⏰ ~ 20 minutes
⏰ ~ 45 minutes
⏰ ~ 100 minutes
= 1.5 hours
Practical
exercises

Human-AI collaboration is a strategic
partnership between AI and human intelligence.
It combines the strengths of both to solve
problems, and improve outcomes.

● Introduction

itself versus
○ Prompt Engineering: The importance of the commands
(‘prompts’) that we give to the GenAI / LLMs
● Conclusions:
29
⏰ ~ 15 minutes
⏰ ~ 20 minutes
⏰ ~ 45 minutes
⏰ ~ 100 minutes
= 1.5 hours
Practical
exercises

The quality of the results we get from AI depends on the commands we give to AI
30

Prompt Engineering
● To get the quality response we want from AI, we must write effective
and high-quality prompts…
There are MANY resources on the internet:

Prompt Engineering – Job Positions !
32

● Introduction

itself versus
● Conclusions:
33
⏰ ~ 15 minutes
⏰ ~ 20 minutes
⏰ ~ 45 minutes
⏰ ~ 100 minutes
= 1.5 hours
Practical
exercises

Two categories of AI-powered testing tools
34
General-purpose Specific-purpose
(have been developed only for Software Testing)

Specific-purpose AI testing tools – An example
● Self-healing of UI tests: Has become a major trend in the industry
○ Self-healing UI tests use AI to automatically adapt to changes in the application's
UI, reducing manual test-code maintenance and ensuring tests remain valid even
with UI modifications.
● A few interesting videos:
○ youtube.com/watch?v=cWX2uvygLg4
○ youtube.com/watch?v=O8z_sBDFL5A
35

Specific-purpose AI testing tools – An example
● healenium.io
36
I am reviewing tools such as https://healenium.io/ . can
you give an estimate on extent of effort spent on
maintaining "locators" in Selenium tests
Old UI
Updated new UI

● Introduction

itself versus
● Conclusions:
37
⏰ ~ 15 minutes
⏰ ~ 20 minutes
⏰ ~ 45 minutes
⏰ ~ 100 minutes
= 1.5 hours
Practical
exercises

Insights and implications Human
Software Tester
AI Software Tester:
Will it replace us?
● Human versus AI?
Or:
● Human with (and) AI
(collaboration)?
We
are here
~2015 ~2040 ?
When?
(timeline)
38

Insights and implications
Human
Software Tester
AI
Software Tester
39

● Who learned what and where?
● Both can make mistakes 🐞 in all tasks, including in software testing activities
● What can each do faster and with better quality? (effectiveness and efficiency)
the World Wide Web contains over 100
billion indexed pages
100,000,000,000 +
40

● A typical Software Test Engineer asks herself:
○ “Should I do this testing activity on my own, or should I get help from AI?”
● What can we (test engineers) do faster and with better quality? (efficiency and productivity)
● What can AI do faster and with better quality?
● I have observed these in my own 2 years of AI-assisted Software Testing experience
In software testing, I want a table to support my own decision-making of when to use AI
and when not to.
Create a table showing: what we (test engineers) can do faster and with higher quality
(effectiveness and efficiency), and what AI can do faster and with higher quality
41

The future of the field: Levels of autonomy in software testing
with AI
● To what extent can AI help us? A little or a lot?
● First, let's look at the topic in another context: A well-known model in the
autonomous vehicle sector (levels of autonomy)...
● We can think of a similar model for software testing. Where do you think we
are?
42

The future of the field: Levels of autonomy in software testing with AI
● In your opinion, where are we in this model in the software
testing field?
● Let's ask GPT itself
43

The future of this field: How will AI change and impact the work of test engineers?
44

AI-powered Software Testing:
The Future is Here
(A hands-on tutorial – 3 hours)
* Also called: AI-driven (AI having a major / active role), AI-powered,
AI-enabled, AI-augmented, …
Vahid Garousi
(Vehid Geruslu)
1- Software Testing Strategist / Consultant
Testinium A.Ş., Türkiye
ProSys MMC, Azərbaycan
AzInTelecom, Azərbaycan
2- Professor of Software Engineering
Queen’s University Belfast, UK
vgeruslu
vehid.geruslu@gmail.com
www.vgarousi.com

AI-assisted Software Testing (3-hours tutorial)

Recommended

More Related Content

Similar to AI-assisted Software Testing (3-hours tutorial) (20)

More from Vəhid Gəruslu (20)

Recently uploaded (20)

AI-assisted Software Testing (3-hours tutorial)

Editor's Notes