SlideShare a Scribd company logo
Spot the difference Automating visual
regression testing
@11vlr
@11vlr
Spot the difference Automating visual
regression testing
@11vlr
• Previously a Software Engineer for
7 years (.Net C#)
• Currently a Senior Test Engineer
@Vizolution
• Code Club volunteer
• Organiser of SwanseaCon
• Co-organiser of DDD Wales
ABOUT ME
• Automated testing is more reliable, as
it is performed by tools and/or scripts.
• Automated testing is executed by
software tools, so it is significantly
faster than a manual approach.
WHY AUTOMATE END
TO END UI TESTS?
@11vlr
@11vlr
• Investment is required
• Automated testing is a practical option
when the test cases are run
repeatedly over a long time period.
• Automated testing entail
human observation and cannot
guarantee user-friendliness or positive
customer experience.
COMMON UI
AUTOMATION ISSUES
DOES NOT
IT CAN BE A SLIPPERY SLOPE
@11vlr
ICE-CREAM CONE ANTI-PATTERN
@11vlr
Unit
Service
UI
Manual &
Exploratory
Testing
£
£££
HOUR GLASS ANTI-PATTERN
@11vlr
£
£££
£
£££
UI
Unit
Manual &
Exploratory
Testing
@11vlr
WHEN YOU HAVE
UNIT BUT NO
INTEGRATION TESTS
@11vlr
TESTINGFRAGILE
TESTING PYRAMID
@11vlr
£
£££
UI
Service
Unit
Manual &
Exploratory
Testing
@11vlr
WHY NOT JUST STICK
WITH MANUAL
TESTING THEN?
• Manual testing allows for human
observation, which may be more
useful if the goal is user-friendliness
or improved customer experience.
UI
Service
Unit
Manual &
Exploratory
Testing
@11vlr
• Can be time intensive
• Is not accurate at all times due to
human error, hence it is less reliable.
• Is time-consuming, taking up human
resources.
• Is only practical when the test cases
are run once or twice, and frequent
repetition is not required.
MANUAL TESTING
WE DON’T HAVE TIME TO AUTOMATE
@11vlr
CATCH 22
@11vlr
We don’t automate
We don’t have time
BECAUSE BECAUSE
@11vlr
MANY OPTIONS
@11vlr
• Open source; anybody can see the
code and help
• Big community of support
• Can operate on almost any OS
• Multiple language support (Python,
Ruby, .Net, PHP, Java, JavaScript)
• Supports a range of browsers like
Opera, Chrome, IE (6,7,8,9,10,11),
Mozilla Firefox, Safari
SELENIUM
IT’S NOT A SILVER BULLET
@11vlr
@11vlr
• Speed to execute tests
• Render differently
BROWSER
COMPATIBILITY
ISSUES
EXECUTION SPEED
@11vlr
MAINTAINABILITY
@11vlr
@11vlr
HOW VALUABLE ARE THESE TESTS?
@11vlr
WHAT THE….






@11vlr
ALL IS NOT WHAT IT SEEMS






@11vlr
MAKING TESTS LESS
FLAKY
@11vlr
• Page object patterns
MAKING TESTS LESS
FLAKY
http://martinfowler.com/bliki/PageObject.html
@11vlr
• Qa-selectors
MAKING TESTS LESS
FLAKY
@11vlr
Qa- selectors
@11vlr
HOW GOOD ARE YOU AT SPOT THE
DIFFERENCE
@11vlr
HOW MANY DIFFERENCES CAN YOU FIND?
@11vlr
HOW MANY DIFFERENCES DID YOU FIND?
@11vlr
THERE WERE 8 DIFFERENCES
@11vlr
HOW DO YOU KNOW WHEN YOU’VE
FOUND THEM ALL?
@11vlr
CAN WE ADD MORE VALUE TO OUR
TESTS?
@11vlr
UI
Service
Unit
Manual &
Exploratory
Testing
WHAT IF WE COULD SIMPLY
@11vlr
equalsAssert
VISUAL TESTING
@11vlr
@11vlr
• Didn’t work straight out of the box
and/or contained lots of bloat
• Steep learning curve
• Small community so lack of support
• Needed to work with our current
automation framework
• Costs
ISSUES WITH OTHER
OFFERINGS
@11vlr
• Still in development
• Facing many challenges which are
because of the products we offer, the
ways in which we have developed
them and the way in which we test
them
ROLLING OUR OWN
@11vlr
• Browser rendering issues
RELIABILITY
@11vlr
EXECUTION SPEED
@11vlr
HUMAN VS MACHINE
@11vlr
HUMAN VS MACHINE
@11vlr
LETS ZOOM IN A LITTLE
@11vlr
LETS ZOOM IN A LITTLE BIT MORE…
@11vlr
GOT HIM!
@11vlr
HOW QUICK WAS THE MACHINE?
@11vlr
MAINTAINABILITY
Old code
@11vlr
MAINTAINABILITY
Old code
New code using some helpers
@11vlr
MAINTAINABILITY
New code using Specflow
New code using some helpers
@11vlr
EASILY IDENTIFY THE DIFFERENCES
@11vlr
ACCURACY
@11vlr
IMAGE QUALITY DIFFERENCE
@11vlr
CHALLENGES SO FAR
@11vlr
MAINTAINING BASE
IMAGES
Keep a base image for each driver
type
@11vlr
SCREENSHOTS
• Viewport (default - WebDriver
W3C standard)
• Full Screen (stitching)
• Capture images of just
elements
@11vlr
SCREEN RESOLUTIONS
@11vlr
SCREEN RESOLUTIONS
@11vlr
Responsive design
@11vlr
HARDWARE ISSUES
Setting a tolerance for the 256
different intensities
@11vlr
STORAGE
Database / Online storage – file
sizes (many images)
@11vlr
DYNAMIC CONTENT
@11vlr
DYNAMIC CONTENT
class=“attendeesLink"
@11vlr
MOVING ELEMENTS
• CSS animations
• Videos
@11vlr
VISUALLY CHECKING DOCUMENTS
@11vlr
PDF DYNAMIC
CONTENT
• A way to blanket content within
documents
UPCOMING CHALLENGES
@11vlr
@11vlr
MORE COMPARISON
OPTIONS
• Text only
• Layout only
ANOTHER TOOL TO ADD TO YOUR BELT
@11vlr
@11vlr
QUESTIONS?
http://bit.ly/VisualTesting
@11VLR
WWW.VIVRICHARDS.CO.UK
Ad

Recommended

Serverless testing-serverless-sydney-20181018
Serverless testing-serverless-sydney-20181018
Rowan Udell
 
Hi5 with HTML5
Hi5 with HTML5
Arif Nezami
 
Introduction to jOOQ
Introduction to jOOQ
Kostadin Golev
 
SharePoint Saturday San Antonio: Workflow 2013
SharePoint Saturday San Antonio: Workflow 2013
Sam Larko
 
Selenium Shots by Kyle J. Ginavan & Ryan Felton
Selenium Shots by Kyle J. Ginavan & Ryan Felton
Steven Chau
 
Common design principles and design patterns in automation testing
Common design principles and design patterns in automation testing
KMS Technology
 
DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
Andreas Grabner
 
Untangling - fall2017 - week 9
Untangling - fall2017 - week 9
Derek Jacoby
 
What TDD Can Teach Us About API Design
What TDD Can Teach Us About API Design
Jon Phenow
 
Defrag 2014 - Hybrid clouds with containers
Defrag 2014 - Hybrid clouds with containers
Avi Cavale
 
Better Page Object Handling with Loadable Component Pattern - SQA Days 20, Be...
Better Page Object Handling with Loadable Component Pattern - SQA Days 20, Be...
Sargis Sargsyan
 
Serverless Application Model - Executing Lambdas Locally
Serverless Application Model - Executing Lambdas Locally
Alex
 
Ruby Real life Perfomance
Ruby Real life Perfomance
viy123
 
Selenium
Selenium
Bryan Mikaelian
 
How Ninefold Tests for User Happiness
How Ninefold Tests for User Happiness
Ryan Clark
 
Migrate PHP E-Commerce Site to Go
Migrate PHP E-Commerce Site to Go
Weng Wei
 
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Oren Eini
 
Scaling small apps
Scaling small apps
Michael Brunton-Spall
 
Performance Testing w/ WebPage Test Private Instance (DrupalCamp Ohio)
Performance Testing w/ WebPage Test Private Instance (DrupalCamp Ohio)
Bill Condo
 
Gemboys
Gemboys
Filippo Dino
 
Surviving in a Microservices environment -abridged
Surviving in a Microservices environment -abridged
Steve Pember
 
Unit Testing Asp.net and Asp.net MVC
Unit Testing Asp.net and Asp.net MVC
Gil Zilberfeld
 
Synthetic web performance testing with Selenium
Synthetic web performance testing with Selenium
Andriy Samilyak
 
Web testing with Selenium
Web testing with Selenium
Sergio Sacristan
 
An introduction to Reactive applications, Reactive Streams, and options for t...
An introduction to Reactive applications, Reactive Streams, and options for t...
Steve Pember
 
Approaching APIs
Approaching APIs
Ross Singer
 
Apply dsl in agile testing
Apply dsl in agile testing
bqconf
 
Agile Testing
Agile Testing
Sargis Sargsyan
 
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Applitools
 
Testing ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NET
Ben Hall
 

More Related Content

What's hot (20)

What TDD Can Teach Us About API Design
What TDD Can Teach Us About API Design
Jon Phenow
 
Defrag 2014 - Hybrid clouds with containers
Defrag 2014 - Hybrid clouds with containers
Avi Cavale
 
Better Page Object Handling with Loadable Component Pattern - SQA Days 20, Be...
Better Page Object Handling with Loadable Component Pattern - SQA Days 20, Be...
Sargis Sargsyan
 
Serverless Application Model - Executing Lambdas Locally
Serverless Application Model - Executing Lambdas Locally
Alex
 
Ruby Real life Perfomance
Ruby Real life Perfomance
viy123
 
Selenium
Selenium
Bryan Mikaelian
 
How Ninefold Tests for User Happiness
How Ninefold Tests for User Happiness
Ryan Clark
 
Migrate PHP E-Commerce Site to Go
Migrate PHP E-Commerce Site to Go
Weng Wei
 
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Oren Eini
 
Scaling small apps
Scaling small apps
Michael Brunton-Spall
 
Performance Testing w/ WebPage Test Private Instance (DrupalCamp Ohio)
Performance Testing w/ WebPage Test Private Instance (DrupalCamp Ohio)
Bill Condo
 
Gemboys
Gemboys
Filippo Dino
 
Surviving in a Microservices environment -abridged
Surviving in a Microservices environment -abridged
Steve Pember
 
Unit Testing Asp.net and Asp.net MVC
Unit Testing Asp.net and Asp.net MVC
Gil Zilberfeld
 
Synthetic web performance testing with Selenium
Synthetic web performance testing with Selenium
Andriy Samilyak
 
Web testing with Selenium
Web testing with Selenium
Sergio Sacristan
 
An introduction to Reactive applications, Reactive Streams, and options for t...
An introduction to Reactive applications, Reactive Streams, and options for t...
Steve Pember
 
Approaching APIs
Approaching APIs
Ross Singer
 
Apply dsl in agile testing
Apply dsl in agile testing
bqconf
 
Agile Testing
Agile Testing
Sargis Sargsyan
 
What TDD Can Teach Us About API Design
What TDD Can Teach Us About API Design
Jon Phenow
 
Defrag 2014 - Hybrid clouds with containers
Defrag 2014 - Hybrid clouds with containers
Avi Cavale
 
Better Page Object Handling with Loadable Component Pattern - SQA Days 20, Be...
Better Page Object Handling with Loadable Component Pattern - SQA Days 20, Be...
Sargis Sargsyan
 
Serverless Application Model - Executing Lambdas Locally
Serverless Application Model - Executing Lambdas Locally
Alex
 
Ruby Real life Perfomance
Ruby Real life Perfomance
viy123
 
How Ninefold Tests for User Happiness
How Ninefold Tests for User Happiness
Ryan Clark
 
Migrate PHP E-Commerce Site to Go
Migrate PHP E-Commerce Site to Go
Weng Wei
 
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB
Oren Eini
 
Performance Testing w/ WebPage Test Private Instance (DrupalCamp Ohio)
Performance Testing w/ WebPage Test Private Instance (DrupalCamp Ohio)
Bill Condo
 
Surviving in a Microservices environment -abridged
Surviving in a Microservices environment -abridged
Steve Pember
 
Unit Testing Asp.net and Asp.net MVC
Unit Testing Asp.net and Asp.net MVC
Gil Zilberfeld
 
Synthetic web performance testing with Selenium
Synthetic web performance testing with Selenium
Andriy Samilyak
 
An introduction to Reactive applications, Reactive Streams, and options for t...
An introduction to Reactive applications, Reactive Streams, and options for t...
Steve Pember
 
Approaching APIs
Approaching APIs
Ross Singer
 
Apply dsl in agile testing
Apply dsl in agile testing
bqconf
 

Similar to Spot the difference - automating visual regression testing (20)

Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Applitools
 
Testing ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NET
Ben Hall
 
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
Applitools
 
The Holy Trinity of UI Testing by Diego Molina
The Holy Trinity of UI Testing by Diego Molina
Sauce Labs
 
Lean Quality & Engineering
Lean Quality & Engineering
Mek Srunyu Stittri
 
Getting Started with Visual Testing
Getting Started with Visual Testing
Applitools
 
SeConf2015: Advanced Automated Visual Testing With Selenium
SeConf2015: Advanced Automated Visual Testing With Selenium
adamcarmi
 
Getting started with Visual Testing using Applitools - @TPC, Feb2020
Getting started with Visual Testing using Applitools - @TPC, Feb2020
Anand Bagmar
 
Advanced Visual Test Automation with Selenium
Advanced Visual Test Automation with Selenium
adamcarmi
 
Designing Self-maintaining UI Tests for Web Applications
Designing Self-maintaining UI Tests for Web Applications
TechWell
 
Test Automation Techniques for Windows Applications
Test Automation Techniques for Windows Applications
Tabăra de Testare
 
Test Automation
Test Automation
nikos batsios
 
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Applitools
 
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
Applitools
 
Advanced Automated Visual Testing
Advanced Automated Visual Testing
adamcarmi
 
A Test Automation Platform Designed for the Future
A Test Automation Platform Designed for the Future
Applitools
 
Testing in agile
Testing in agile
Andrei Metelski
 
Saksham Sarode - Innovation Through Introspection - EuroSTAR 2012
Saksham Sarode - Innovation Through Introspection - EuroSTAR 2012
TEST Huddle
 
Discover the power of QA automation testing
Discover the power of QA automation testing
Softweb Solutions
 
Adam carmi
Adam carmi
CodeFest
 
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Modern Functional Test Automation Through Visual AI - webinar w/ Raja Rao
Applitools
 
Testing ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NET
Ben Hall
 
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
[webinar] Cutting-edge Functional UI Testing Techniques - w/ Adam Carmi
Applitools
 
The Holy Trinity of UI Testing by Diego Molina
The Holy Trinity of UI Testing by Diego Molina
Sauce Labs
 
Getting Started with Visual Testing
Getting Started with Visual Testing
Applitools
 
SeConf2015: Advanced Automated Visual Testing With Selenium
SeConf2015: Advanced Automated Visual Testing With Selenium
adamcarmi
 
Getting started with Visual Testing using Applitools - @TPC, Feb2020
Getting started with Visual Testing using Applitools - @TPC, Feb2020
Anand Bagmar
 
Advanced Visual Test Automation with Selenium
Advanced Visual Test Automation with Selenium
adamcarmi
 
Designing Self-maintaining UI Tests for Web Applications
Designing Self-maintaining UI Tests for Web Applications
TechWell
 
Test Automation Techniques for Windows Applications
Test Automation Techniques for Windows Applications
Tabăra de Testare
 
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Visual Testing: Quality and Speed -- by GIl Tayar, Applitools
Applitools
 
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
Triple Assurance: AI-Powered Test Automation in UI Design and Functionality
Applitools
 
Advanced Automated Visual Testing
Advanced Automated Visual Testing
adamcarmi
 
A Test Automation Platform Designed for the Future
A Test Automation Platform Designed for the Future
Applitools
 
Saksham Sarode - Innovation Through Introspection - EuroSTAR 2012
Saksham Sarode - Innovation Through Introspection - EuroSTAR 2012
TEST Huddle
 
Discover the power of QA automation testing
Discover the power of QA automation testing
Softweb Solutions
 
Adam carmi
Adam carmi
CodeFest
 
Ad

Recently uploaded (20)

Transmission Media. (Computer Networks)
Transmission Media. (Computer Networks)
S Pranav (Deepu)
 
UPDASP a project coordination unit ......
UPDASP a project coordination unit ......
withrj1
 
Making significant Software Architecture decisions
Making significant Software Architecture decisions
Bert Jan Schrijver
 
Advanced Token Development - Decentralized Innovation
Advanced Token Development - Decentralized Innovation
arohisinghas720
 
Introduction to Agile Frameworks for Product Managers.pdf
Introduction to Agile Frameworks for Product Managers.pdf
Ali Vahed
 
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
BradBedford3
 
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Philip Schwarz
 
wAIred_RabobankIgniteSession_12062025.pptx
wAIred_RabobankIgniteSession_12062025.pptx
SimonedeGijt
 
Insurance Underwriting Software Enhancing Accuracy and Efficiency
Insurance Underwriting Software Enhancing Accuracy and Efficiency
Insurance Tech Services
 
Microsoft Business-230T01A-ENU-PowerPoint_01.pptx
Microsoft Business-230T01A-ENU-PowerPoint_01.pptx
soulamaabdoulaye128
 
Code and No-Code Journeys: The Coverage Overlook
Code and No-Code Journeys: The Coverage Overlook
Applitools
 
Zoneranker’s Digital marketing solutions
Zoneranker’s Digital marketing solutions
reenashriee
 
HYBRIDIZATION OF ALKANES AND ALKENES ...
HYBRIDIZATION OF ALKANES AND ALKENES ...
karishmaduhijod1
 
A Guide to Telemedicine Software Development.pdf
A Guide to Telemedicine Software Development.pdf
Olivero Bozzelli
 
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
Hassan Abid
 
How to Choose the Right Web Development Agency.pdf
How to Choose the Right Web Development Agency.pdf
Creative Fosters
 
Open Source Software Development Methods
Open Source Software Development Methods
VICTOR MAESTRE RAMIREZ
 
Shell Skill Tree - LabEx Certification (LabEx)
Shell Skill Tree - LabEx Certification (LabEx)
VICTOR MAESTRE RAMIREZ
 
Complete WordPress Programming Guidance Book
Complete WordPress Programming Guidance Book
Shabista Imam
 
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
Natan Silnitsky
 
Transmission Media. (Computer Networks)
Transmission Media. (Computer Networks)
S Pranav (Deepu)
 
UPDASP a project coordination unit ......
UPDASP a project coordination unit ......
withrj1
 
Making significant Software Architecture decisions
Making significant Software Architecture decisions
Bert Jan Schrijver
 
Advanced Token Development - Decentralized Innovation
Advanced Token Development - Decentralized Innovation
arohisinghas720
 
Introduction to Agile Frameworks for Product Managers.pdf
Introduction to Agile Frameworks for Product Managers.pdf
Ali Vahed
 
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
BradBedford3
 
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Philip Schwarz
 
wAIred_RabobankIgniteSession_12062025.pptx
wAIred_RabobankIgniteSession_12062025.pptx
SimonedeGijt
 
Insurance Underwriting Software Enhancing Accuracy and Efficiency
Insurance Underwriting Software Enhancing Accuracy and Efficiency
Insurance Tech Services
 
Microsoft Business-230T01A-ENU-PowerPoint_01.pptx
Microsoft Business-230T01A-ENU-PowerPoint_01.pptx
soulamaabdoulaye128
 
Code and No-Code Journeys: The Coverage Overlook
Code and No-Code Journeys: The Coverage Overlook
Applitools
 
Zoneranker’s Digital marketing solutions
Zoneranker’s Digital marketing solutions
reenashriee
 
HYBRIDIZATION OF ALKANES AND ALKENES ...
HYBRIDIZATION OF ALKANES AND ALKENES ...
karishmaduhijod1
 
A Guide to Telemedicine Software Development.pdf
A Guide to Telemedicine Software Development.pdf
Olivero Bozzelli
 
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
Hassan Abid
 
How to Choose the Right Web Development Agency.pdf
How to Choose the Right Web Development Agency.pdf
Creative Fosters
 
Open Source Software Development Methods
Open Source Software Development Methods
VICTOR MAESTRE RAMIREZ
 
Shell Skill Tree - LabEx Certification (LabEx)
Shell Skill Tree - LabEx Certification (LabEx)
VICTOR MAESTRE RAMIREZ
 
Complete WordPress Programming Guidance Book
Complete WordPress Programming Guidance Book
Shabista Imam
 
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
Natan Silnitsky
 
Ad

Spot the difference - automating visual regression testing