Using Big Data Analysis to Reduce Test Flakiness

®
WEBINAR: Using Big Data Analysis to
Reduce Test Flakiness
• January 20th, 2022 – Adam Sandman, Denis Markovtsev
We will be starting the webinar shortly, please stand by…
All phones will be automatically on mute until the Q&A.

®
2 | 4/6/2022 © Copyright 2006-2021 Inflectra Corporation
®
About Me
 Adam Sandman was a programmer
from the age of 10 and has been
working in the IT industry for the past
20 years.
 Currently Adam is a Director of
Technology at Inflectra Corporation,
where he is interested in technology,
business and innovation.
Twitter: @adammarksandman
linkedIn: https://www.linkedin.com/in/adamsandman/

®
®
About Me
 Denis Markovtsev works in the IT
industry for the past 18 years.
 Currently Denis is a Principal Software
Engineer at Inflectra Corporation. He is
one of creators of Rapise test
automation tool.
Twitter: @dmarkovtsev
LinkedIn: https://www.linkedin.com/in/denismarkovtsev/

®
Agenda
Why Is Test Automation So Hard?
Findings from the MOZ 500 Websites
So What Can We Do?
Some Tips and Tools
Questions?

®
It Began With a Question…

®
Why is Test Automation
So Difficult and Unreliable?

®
MOZ 500 Research Project
We Thought it Would be Good to See the Real World First!

®
®
Researching the Top 500 Websites
 Top 500 websites ranked
by their search visibility
 https://moz.com/top500

®
What Did We Find?

®
®
IDs Not Guaranteed to be Unique

®
®
Dynamic / Semi-Dynamic IDs

®
®
Wide Variety of Attributes in Use
Top Attributes:
1. class
2. href
3.style
4.target

®
®
Example: Dynamics Business Central
Top Attributes:
1. class
2. title
3.aria-label
4.controlname

®
®
Dynamic Classes & Closures

®
®
Hidden Frames & Duplicate Elements

®
What Can We Do?

®
®
Work with Developers (If Possible)
 Testers should ask developers to collaborate and listen to
what testers consider helpful changes for test automation
 Automation Tools should have analysis tools
 (Spy, query and analyze multiple DOM trees simultaneously)
 Ideally the developers and testers should be able to have a
common toolset that assists in joint working
 Note: This is not usually possible with packaged
applications such as Microsoft Dynamics, SalesForce, etc.

®
What If We Can’t?

®
Example #1: Dynamic ID
//button[@id='89EF7']/span
More examples:
<button type="button"
id="CustFreeInvoice_3_AccountingActionsTab_btn">
<div id="appGridQueryFilter_Component8738712>

®
®
Example #1: Some Alternatives
 Use attributes:
id, class, role
 Avoid auto generated ids and classes:
_lvv_3, _ariaId_24
 Avoid full xpath
/html/body/div[5]
 Index may be a good choice:
(//div[@role='option'])[1]
 Search for text
//span[text()='Log In']

®
®
OR… Multiple Locators
• Don’t just rely on IDs
• Pick the right approach for the element
• Sometimes text is best!

®
Example #2: Weak Attribute Usage
//form[@id='aspnetForm']/div[2]/div/div/nav/div[
2]/div/div/div[2]/div/div/div/div/div/div/div/di
v/div/div/div/div/button/span/span
Compare with
//button[@role='menuitem’]//span[@aria-
label='Customers']

®
Example #3: Hidden Layers

®
Example #3: Hidden Layers
Recorded:
//form[@id='aspnetForm']
Ways to fix:
//form[@controlname='Business Manager Role
Center’]
/div[contains(concat(' ', @class, ' '), ' spa-
view ') and position()=last()]/form

®
What Can Vendors Do?

®
Give Them Tools! To:
 Detect dynamic IDs
 Analyze identification power of attributes
 Analyze classes
 Identify Anchors
 Use Machine Learning (branded as AI but not really):
 Scan the application, compare to previous seen patterns
 Determine the right way to identify each element
 Trend is to try to make self-healing locators

®
Get Unique Sorted Attribute Values

®
Query and Compare Multiple Documents

®
Calculate Identification Power of Attributes

®
Define Anchors
//div[contains(concat(' ', @class, ' '), ' spa-
view ') and position()=last()]
Find nearest anchor to recorded element and include it in
XPath:
//div[contains(concat(' ', @class, ' '), ' spa-
view ') and position()=last()]
//button[@role='menuitem’]//span[@aria-
label='Customers']

®
Build Web Application Profile/Recorder
Configuration

®
End-Goal: Self Healing Tests

®
Our Experience
• Self-healing or manual
maintenance is Cure
• Recorder configuration is
Vaccine

®
Wrap Up & Final Thoughts
 Looking at the Moz 500 shows us the scale of the problem
 Current technology can help as long as we spend the time
to understand what is stable and what is transient in our
applications
 Future AI (really Machine Learning) can help – self healing
tests and intelligent object identification

®
Questions?

Using Big Data Analysis to Reduce Test Flakiness

Recommended

More Related Content

Similar to Using Big Data Analysis to Reduce Test Flakiness (20)

More from Inflectra (20)

Recently uploaded (20)

Using Big Data Analysis to Reduce Test Flakiness