Experience and best practices from automated software testing in the finance domain

1/37
Vahid Garousi Experience and best practices from automated testing in the finance domain
Experience and best practices from
automated software testing in the
finance domain
Vahid Garousi, PhD, PEng
Affiliations:
Associate Professor of Software Engineering (Senior Lecturer)
Queen’s University Belfast, Northern Ireland, UK
Director and Managing Consultant
Bahar Software Engineering Consulting Limited, UK
vahid@vgarousi.com
www.vgarousi.com
@vgarousi
vgarousi
Invited talk for the NatWest QCOE (NWG)
Testing Conference
Oct. 19, 2021
1:
2:

2/37
Outline
 My career background
 Past/ongoing software-testing projects …
 Experience and best practices from automated testing
in two finance projects (two of our clients/partners):
 Testing the web and mobile applications of BtcTurk A.Ş.:
Turkey's first and the world’s fourth Bitcoin and
cryptocurrency exchange platform. (Project running since
2020-)
 Testing the mobile applications of KuveytTürk Bank A.Ş.: a
large Turkish bank. (Project ran between 2018-2020)
 Note: The information, presented in this talk, have either
already been published as technical articles – and OK’ed by
the clients - thus are “unclassified” and there is permission
to share them
 Observations / retrospective
 … of best practices in automated testing in the finance
domain
 Q/A

3/37
My background (international experience)
Education:
PhD in Software Engineering, Carleton University, Ottawa, Canada, 2006
MSc in Computer Engineering, University of Waterloo, Canada, 2003
BSc in Software Engineering, Sharif University of Technology, Tehran, Iran, 2000
Work experience:
Associate Professor, Queen’s University Belfast, UK, 2019-
Managing Consultant, Bahar Software Engineering Consulting, since 2001-
Past:
Associate Professor, Wageningen University, Netherlands, 2017-2019
Associate Professor, Hacettepe University, Ankara, Turkey, 2015-2017
Associate Professor, University of Calgary, Canada, 2006-2014
Ethnic background: Turkish / Canadian
2013
2019
2015
2009

4/37
Consulting projects (a partial list)
 Improving capability and maturity of test automation projects
 Client: Testinium is a large test-services company in Turkey
 Project running since 2019-
 Test automation of web and mobile applications
 Client: BtcTurk A.Ş., Turkey's first and the world’s fourth Bitcoin and cryptocurrency exchange
platform
 Project running since 2020-
 Test automation of mobile applications
 Client: KuveytTürk Bank A.Ş., a large Turkish bank
 Duration: 2018-2020
 Improving test automation practices
 Client: HAVELSAN (Hava Elektronik Sanayi, Aviation Electronics Industries)
 Duration: 2015-2017
 Automated testing of law-management software suite
 Client: Innova IT Solutions, Ankara, Turkey
 Duration: 2016-2017
 Development and assessment of effective test automation infrastructure
 Client: MR Control Systems International, Calgary, Canada
 Duration: 2008-2011
 Decision-support for deciding “when to automate testing” and “what (test cases) to
automate”
 Client: Pason Systems Corporation, Calgary, Canada
 Duration: 2010-2012
 And many more…

5/37
Outline
 Experience and best practices from
automated testing in two finance projects
(two of our clients/partners):
 Testing the web and mobile applications of BtcTurk
A.Ş.: Turkey's first and the world’s fourth largest
Bitcoin and cryptocurrency exchange platform.
(Project running since 2020-)
 Testing the mobile applications of KuveytTürk Bank
A.Ş.: a large Turkish bank. (Project ran between 2018-
2020)
domain
 Q/A

6/37
Systems Under Test (SUT)
 Mobile app:
BtcTurk (BtcTurk.com) serves more
than 3.8 million registered users

7/37
Systems Under Test (SUT)
 Web apps:
 A simple version,
and a pro version

8/37
Setup of our Test Automation Team
Test Automation
Engineers
Test
Automation
Consultant
Client
Test Automation
Team-lead
Product owner,
and Software
Engineers
…
Systems Under
Test (SUT)
Test
Develop / update
Co-developing the test
automation strategy and
monitoring the entire test
process
PARTNERSHIP
SINCE 2019

9/37
Test automation strategy
GUI
API
Business Logic
Database
Calls
Calls
Returns data
Returns data
Class
(Unit)
Class
(Unit)
…
System Under Test (SUT)
and its architecture
GUI testing
API / service
testing
Unit tests
Developed by the
Software Engineers
at BtcTurk (not us)
Using the BDD
tool: Gauge
Test
Test
Test
Two approaches:
1- Model-based Testing
2-BDD (tool: Gauge)
Test automation
suites:
Some
example
test
artifacts
next…

10/37
Model-based Testing (MBT): Basics
 Let’s first see: What is
Model-based testing
(MBT)?
 Model-based testing is
the application of model-
based design for
systematic and
automated test-case
design and, optionally,
also for automated
execution of test cases.
 An example:
Valid login
• Nodes, or vertices: “States”
(pages) of the system to verify
• Edges or transitions: Events
such as valid login, click
Let’s hear from you: Have you ever used MBT in your projects? If yes, how have you
used it? Join at slido.com #777

11/37
Model-based Testing (MBT): Basics
 Developing the “behavior” of nodes/edges in Java using
the Selenium framework
public void e_valid_login() {
WebElement userNameElement =
methodsPage.findElement(By.id("username"));
userNameElement.clear();
userNameElement.sendKeys(email);
WebElement passwordElement =
methodsPage.findElement(By.id("password"));
passwordElement.clear();
passwordElement.sendKeys(password);
methodsPage.findElement(By.cssSelector(
"input[class$="login-page__submit-btn"][value="Sign In"]")).click();
}

12/37
Strategy for Model-based Testing (MBT)
 Our strategy: we designed two MBT test suites:
 An end-to-end test suite (like a smoke test): only the major paths
through the app
 A complete regression test suite: exercising all the pages and all
paths (five main screen, many tabs, and “many” possible paths)
 Details next…
 MBT test tool:
 An open-source popular tool, named GraphWalker
(graphwalker.github.io)
 It is in wide use, e.g., in companies such as Spotify, and testing the
information kiosks software of New York’s transportation authority
(bit.ly/MBTGuidingTestingDecisions)

13/37
MBT of BtcTurk mobile app
 End-to-end test suite (like a smoke test; mainly “Happy” paths):
Login
All currency
pairs
A pair’s
details page
My wallet
Buy
Sell
Buy Successful
Sell Successful
View latest
transactions
Buy/sell page
My wallet
My
transactions
Limit tab
Pair Command
Buy Command
Successful
Cancel
Command
My wallet
Sell Command
Successful
Start
Since team members
were all Turkish, Turkish
language is used for
naming ;) Also, UI has
two languages
English
translations

14/37
 The complete test suite: exercising all the
pages and paths):
 8 test models, 303 nodes, 567 edges (transitions)
Login LoginPro
Currency
pairs
My
wallet
My
Account
Deposit/
Transfer
(Funding)
Trade
Panel
Move
among
Tabs

15/37
 The complete test suite: how the 8 test models are connected
 One model for each app “page” (tab) 
Login, regular
version of app
Login Pro version
5
3
4
2

16/37
 The complete test suite: exercising all the pages and
paths): Full MBT run takes about 2 hours…
www.youtube.com/watch?v=dOgHQDnxnKw

17/37
MBT: What are its benefits?
 Let me ask you first…
 Tangible benefits:
 Covering more test paths (higher “path” coverage) with less effort 
Saving time and money
 Thus, leading to increased test effectiveness in detection of faults
 Intangible, but important, benefits:
 Ability to see the “big picture” of test-case design, and the models
being directly executable
 Making the work of test engineers more “interesting” / fun and more
organized (according to feedback from our team members)
 And much more…
One of our meetings in
Istanbul, 2019
(before COVID)
Join at slido.com #777

18/37
MBT: Challenges that we faced and how we addressed them
Challenges How we addressed them
Challenge 1- Choosing the “right” MBT tool
from amongst the very large pool of available
MBT tools
Tools Criteria
1-Matching test requirements- Essential 2a-Right level of model abstraction- Essential
TestModeller - Exhaustive activity diagram, resulting in repetition of nodes
(youtu.be/nctAQHsmjpI)
Failed the essential criterion 2
TestOptimal Supports web/mobile apps, but not specific for them Web page UI flow diagram
Tricentis Tosca Seems like a test-data management tool (youtu.be/f6aBpa95kLc). While the introduction on its
website mentions MBT, support for MBT is very limited. Not possible to design cycles and
complex flow/edge structures
-
SpecExplorer Support for MBT of web/mobile apps seems very limited. Most of focus is on API and unit
testing.
-
GraphWalker Specific for web/mobile apps Web page UI flow diagram
NModel - Test model is in a programmatic format, instead of visual
diagrams (doi.org/10.1007/978-3-642-05031-2_14)
TCases Support for MBT of web/mobile apps seems very limited. Most of focus is on test-case design
for input space exploration.
-
ModBat Focus is on API testing. No support for MBT of web/mobile apps. (fmv.jku.at/modbat)
-
MoMuT Focus is on embedded system testing. No support for MBT of web/mobile apps. (momut.org)
-
CrawlJax It produces as output a state-flow graph of the dynamic DOM states and the event-based
transitions between them. Focus is not on GUI testing of web apps
-
(1) the tool matching the test requirements, e.g., type of SUT (for us, this was web/mobile apps)
(2) tool being fit to the operating environment, e.g., “right level” of model abstraction, team’s expertise
(3) tool’s usability
(4) availability of support for the tool
(5) tool’s cost
We did a research on all knowledge and evidence shared
on the internet by experts (53 blogs and white papers),
about this topic. We synthesized the criteria for choosing
the right tool under into five categories:

19/37
MBT: Challenges that we faced and how we addressed them
Challenges How we addressed them
Challenge 2- We observed a general
shortage of knowledge and resources on
best practice and “design patterns” for
designing MBT models
Challenge 3-
By sharing the best practices, we have
identified a number of design patterns,
e.g., one model for each web/mobile app
page, clear separation of events (edges)
and assertions (nodes)
Four graph traversal algorithms (test-
path “generators”) are supported by
GraphWalker: Random, Weighted
random, Quick random, A* (A-star).
Which one shall we use?
We are investigating the different options
and assessing their strength, e.g.. Fault
detection effectives versus efficiency,
(time taken for test execution)

20/37
Let’s see another test approach in our BtcTurk project
GUI
API
Business Logic
Database
Calls
Calls
Returns data
Returns data
Class
(Unit)
Class
(Unit)
…
System Under Test (SUT)
and its architecture
GUI testing
API / service
testing
Unit tests
Developed by the
Software Engineers
at BtcTurk (not us)
Using the BDD
tool: Gauge
Test
Test
Test
Two approaches:
1- Model-based Testing
2-BDD (tool= Gauge)
Test automation
suites:
Reminder: Our test automation strategy:

21/37
BDD testing of BtcTurk apps, using Gauge tool
Develops
Gauge test-suite
Web
browser
Test
step
Test specification
file Test-step
implementation
Test
scenario
1...*
Test concept
1...*
1
1
In Java or
another
language System Under
Test (SUT)
Call
Call Return
Gauge
Call
Return
Test
outputs
Return
Anlayzes
Test enigneer
Call
A specification file:
(can have many test scenarios)
A scenario
A concept:
A step
Step implementations:
Examples from: docs.gauge.org/overview.html

22/37
BDD tools: Cucumber versus Gauge
 Behaviour-Driven Development (BDD) tools such as Cucumber are
powerful tools for automated testing
 But tools such as Cucumber have certain limitations:
 They often enforce strict syntax for test cases, like the “Given-When-
Then” format, which may not always be easy to write for all test cases.
 The more recent test-automation framework Gauge addresses that
limitation since it does not prescribe the BDD testing process with a
strict syntax.
 In Gauge, writing a test case is as easy as writing down the flow of test
cases in several itemized sentences in a natural language, like English.
 Thus, Gauge could be a better tool than Cucumber

23/37
 We started our test-suite development… But soon we faced a number
of challenges
 Challenge 1: See below… A tool is NOT enough. We need “systematic” test-case design
Terms are mostly in Turkish, but I am
sure you can see some issues …
Issues here:
 Duplications in test code
 Test logic: is, sometimes, hard to
understand (low understandability)
 Test code: Hard to read and hard to
maintain
  There should be a “better” way
of developing test code (scripts)…
Inside element “X”, write value of “Y”

24/37
 These challenges have also been observed by many other test engineers
 Challenge 1: Having a test tool is NOT enough. Do you agree?

25/37
 For this purpose, we devised a systematic test-case “design” approach, using test
models which we designed manually (essentially flow diagram of the web/mobile app)
 We derive four types of test cases from the model:
1. Transition from the current node (page) to its next neighbouring nodes (pages)
2. Node to itself (often, error checking, like: Invalid Login)
3. Input UI tests (in single node/page/unit level)
4. End-to-end testing: testing a path of nodes
https://www.slideshare.net/vahidgarousi/test-
automation-with-the-gauge-framework-experience-
and-best-practices-seetest-2020-conference
More details in…

26/37
 Example Gauge test suites:

27/37
 Another challenge:
 The test-script LOC of our Gauge test-suites soon expanded and reached more than 50K
test LOC
 So it was clear that, just like application (production) code-base, test code-base also
needs refactoring, maintenance and quality improvement (e.g., better test-code design /
architecture / modularity)
 Solution:
 We had developed an in-house tool for this need in our past projects, so we used it…
 Our GaugeDepend tool
 … is a dependency analysis
tool for visualizing and
exploring dependency
graphs for Gauge test suites.
 Such as visualization will
allow quality assessment of
test-automation scripts.
 Is open source:
github.com/vgarousi/
GaugeDepend
 Demo video:
youtube.com/watch?v=oehq
DgX8R8E
Test
specification
files:
Test
scenario: Test steps: Test concepts
(Gauge terminology):
Let’s zoom in…

28/37
Outline
A.Ş.: Turkey's first and the world’s fourth Bitcoin and
cryptocurrency exchange platform. (Project running
since 2020-)
2020)
domain
 Q/A

29/37
Automated testing of the mobile applications of KuveytTürk Bank
 App Under Test
(AUT):
 The app had 73
GUI screens
(pages)

30/37
 We used Appium to develop a large GUI test suite:
 In the first phase: 28 test classes in Java using Appium
 Testing 28 of the app’s total 73 GUI screens (pages) – the
most critical features (risk-based testing)
 In total, 16,351 test LOC (TLOC)
 In the 2nd phase: we tested all the GUI pages

31/37
 Some example test code (Java Appium framework):

32/37
Automated testing of KuveytTürk apps: Challenges observed and best
practices
 We observed a number of “classical” challenges,
e.g., the need to better design and develop the
test scripts
 Best practices that we applied:
 Use of test (design) patterns (xunitpatterns.com) such as: the
Four-Phase Test pattern (setup of test “fixture”, exercise / call
the SUT, result verification, and fixture teardown)

33/37
Automated testing of KuveytTürk apps: Challenges observed and best
practices
 Another best practice that we applied:
 Decoupling of test execution parameters from test scripts (an
example is below)

34/37
Testing the mobile applications of KuveytTürk Bank A.Ş.
 Our mobile device farm:
More details in:

35/37
Outline
since 2020-)
2020)
domain
 Q/A

36/37
Experience-based observations: Automated testing in finance
 We all know that quality of web and
mobile apps in finance are extremely
business-critical
 Automated testing of these apps are
crucial and critical to business success
 At the end of the day, we need a
combination of automated and manual
(exploratory) testing
 Each would find different types of
defects, and have pro’s and con’s
 With experience in these topics, we can
certainly plan and execute a successful
test approach
Azimo: Global
Money
Transfers
Australia and
New Zealand
Banking Group

37/37
Outline
since 2020-)
2020)
domain
 Q/A

38/37
BACK UP SLIDES

39/37
R&D Projects (in the UK, EU and Turkey)
 Two recent projects:
 XANDAR: X-by-Construction Design framework for Engineering
Autonomous & Distributed Real-time Embedded Software Systems
(2021-2024)
 Eight (8) partners from across Europe, including five industry and
three academic partners
 Budget: €5 million Euros
 I “lead” the Work Package (WP) on Verification and Validation (V&V)
 TESTOMAT (The Next Level of Test Automation)
 Project funded by the EU
 34 partners from across Europe
 Budget: €21 million Euros
 www.testomatproject.eu (2018-2020)
 And 12 other major projects…

40/37
BDD testing using Gauge tool: Challenges and solutions
 Opinions of other test engineers:
 “If you feel that Cucumber is a pain when used in large projects due to its
Given-When-Then syntax becoming messy in long run, Gauge can take out
that pain to some extent.
 Even here, the flexibility of free text can be a demerit as they don’t have a
defined format.
 But tests may end up looking like a messy text document unless imposed strict
restrictions”.
 https://medium.com/@divi6.1990/why-i-chose-gauge-over-cucumber-5b3de478d889
 “Gauge follows Markdown Syntax for scripting the tests in an unstructured
way - which means more liberty to make the test like a specific document and
also more risk of ending up with scripted tests in many different styles.
 Having some in-team standards will help to prevent any chaos.” [8]
 www.linkedin.com/pulse/things-consider-when-choosing-best-test-automation-framework-akbari

Experience and best practices from automated software testing in the finance domain

Recommended

More Related Content

What's hot (20)

Similar to Experience and best practices from automated software testing in the finance domain (20)

More from Vəhid Gəruslu (20)

Recently uploaded (20)

Experience and best practices from automated software testing in the finance domain

Editor's Notes