SlideShare a Scribd company logo
3 Keys to Performance Testing at the Speed of Agile
Matching the imperative of high-performance
with the pace of modern software delivery
1. This session is being recorded
2. Discussion in chat panel
3. Q & A at the end of session
4. Follow-up email with links to presentation
Housekeeping
1. Agile Challenges and Anti-patterns
2. The Goal: Potentially Confidently Shippable Product
3. 3 Keys to Efficient Performance Testing in Sprints
4. Q&A Roundtable
What We’ll Cover
Today’s Presenter
Paul Bruce
Sr. Performance Engineer, Neotys USA
@paulsbruce @neotys
linkedin.com/in/paulsbruce
Quick Poll: When do you measure system performance?
[multiple choice]
● Just before MAJOR releases
● Just before MINOR releases
● A few times, throughout sprints
● As part of a Continuous Delivery pipeline
● No schedule, project-based, as-needed
System performance is imperative to all business models
14%
2%
6%
max
avg
min
Qualys - bit.ly/2D6OJWU
1 of every 3 organizations
take longer than 30 days
to patch/resolve security issue.
Slow Down
(Lose $$$)
Vulnerable
(Mitigate Risk)
?
Intel Newsroom - intel.ly/2B4DWL3 Databricks -bit.ly/2Dd51h4
Consistency: across all digital properties (web, mobile, API…)
Fast activity completion: checkout, subscribe, approve
Abandonment: page or mobile app uninstalls
Ubiquitous connectivity: WiFi, mobile, broadband…
What kind of performance do customers expect?
“Faster” Isn’t Good Enough Anymore
Value
Customer
Centricity
Business Technology
Accelerated Delivery
Small, frequent changes
Efficiency: skills, process, tools
“We didn’t bother with scalability because they wanted an MVP ASAP…”
“The stakeholders didn’t say they want to integrate this with [that other service]...”
“We’ll deliver documentation...if there’s time at the end of the sprint.”
“Security is for a hardening stage...which as usual, got eaten up by bug fixing…”
“We’ve been doing stand-ups for weeks, and no one mentioned performance criteria!”
“That isn’t in the definition of done, so the customer shouldn’t expect it in the product.”
“That’s someone else’s area of expertise...I’ll assign the task to them next.”
Agile “faster theater” anti-patterns
Finely-tuned batch sizes and automation solve many (but not all) problems.
● Resource intensive human-only tasks
○ Planning
○ Estimation
○ Prototyping
○ Code reviews
● Resource intensive automation tasks
○ Regression testing
○ Migrations / upgrades
○ Performance testing
● Trend-driven analysis
○ Build and release metrics
○ Production actuals (e.g. A/B testing)
○ Customer feedback / reviews / NPS
Real Work Is Lumpy, but Still Necessary
finalization
work
spin-up
finalization
work
spin-up
Small Batch SizeBig Batch Size
Same
transaction
overhead
“Right fit” is the biggest challenge.
1. What is ‘Progressive Testing’?
2. What needs to be performance tested (and when)?
3. The “right” performance test for the job
4. Example triggers and schedules
The Goal: Confidently Shippable Product
Progressive Testing Across the Delivery Pipeline
Feature X… API update Y… Patch N...For:
Time budget for testing:
seconds minutes hours
Days
Weeks
Each layer must be tested for performance:. API, database, web/mobile, networks, 3rd
parties, cloud infrastructure, failover/recovery behaviors.
What needs to be load tested?
Data Stores
Core microservices
3rd-party APIs
Load Balancers
Front-facing servers
Network / Edges
Short, fast performance
feedback loop
Longer burn / more complex
performance testing
API perf testing
DB latency testing
Windowed perf tests
Web & mobile testing
Spike/failover testing
Distributed geo load
.
The “right” performance test for the job
Real-world dependencies
Easy & fast
Takes time
The “right” performance test for the job
Development and Architecture
● Load (does it handle N users?)
○ Cloud migration baselines
○ Build-over-build performance trends
○ Meeting contractual SLA obligations
● Soak (are there leaks or degradations over time?)
○ Server RAM, CPU, disk
● Spike (how does it handle dramatic changes?)
○ Load balancing
○ Rate throttling
○ Garbage collection
Operations and Change Management
● Stress (where is the max?)
○ Capacity planning
○ EUX degradation and ajax timeout handling
● Configuration (is this tweak better or detrimental?)
○ Routing, SSL termination, database tuning
○ Logging, retention policies
● Network (what latencies are in the pipes?)
○ 3rd-party API SLAs
○ Service distribution (architectural latency)
○ Multi-cloud deployments
Example: Triggers and schedules for performance testing
Event Max CI duration DB API Web &
mobile
3rdParty Infra Network Regression
Code check-
in
10-15 min Sm Sm Sm
Merge 15-30 min Med Med Med
Overnight 1-4hrs Med/Lg Med/Lg Med/Lg Sm
Weekly Lg Lg Lg Sm Med Med Med
Pre- release Lg Lg Lg Lg Lg Lg Lg
Test Coverage (Sm / Med / Lg):
● Volume (VUs)
● Scope (# flows)
● Conditions (geo, wifi/LTE)Small
Medium
Large
“Right fit” is the biggest challenge.
1. Integrate performance into planning phase
2. Establish “right-sized” feedback loops
3. Reduce waste in scripting and analysis
3 Keys to Efficient Performance Testing
2017 study:
● Release every other day;
2x user stories w/
performance criteria
Key #1: Integrate performance into planning phase
bit.ly/performance-feature
How to “bake performance in” to user stories?
● “Back of card” / DoD - performance criteria
● “Think 4th dimensionally” → Time
● Concerns and guidelines, not testing performed
● Performance-specific:
○ Concurrency: “how many users at the same time?”
○ Conditions: “which devices from where?”
○ Capacity: “what resources will this take?”
Key #1: Integrate performance into planning phase
techbeacon.com/how-build-performance-your-user-stories
Todd DeCapua - @appperfeng
● Small load test on new features
● Critical-path performance regression
● Baselines and comparisons accessible to everyone
● Including metrics that tell the complete story:
○ The “efficacy” of the test (load health and bias)
○ The “impact” side of the story (server monitoring)
○ The “customer” side of the story (end-user experience)
Key #2: Establish Right-Sized Feedback Loops
Value vs. time cost
Risk Prevention
Early Detection
End-to-end visibility
Key #2: Establish Right-Sized Feedback Loops
Key #3: Reduce Waste in Scripting Time
● Record through a browser, low-code approaches
● Carry customizations during re-scripting / recording
● Build library of reusable test assets
● Reuse existing functional test assets
(“Self-healing” load scripts)
● Share test assets across teams and engineers
Easy & Intuitive
Efficient
Consistent
Realistic
Collaborative
Key #3: Reduce Waste in Scripting Time
1. Integrate performance into planning phase
2. Establish “right-sized” feedback loops
3. Reduce waste in scripting and analysis
3 Keys to Efficient Performance Testing
1. Agile Challenges and Anti-patterns
2. The Goal: Confidently Shippable Product
3. 3 Keys to Efficient Performance Testing in Sprints
What We’ve Covered
Q&A Roundtable
Paul Bruce
Sr. Performance Engineer, Neotys USA
@paulsbruce @neotys
linkedin.com/in/paulsbruce
Ad

Recommended

PHX - Session #2 Test Driven Development: Improving .NET Application Performa...
PHX - Session #2 Test Driven Development: Improving .NET Application Performa...
Steve Lange
 
Software Quality for Developers
Software Quality for Developers
Maira Bay de Souza
 
Scrum overview
Scrum overview
Robert Bastian
 
SbE - Requirements in an agile process
SbE - Requirements in an agile process
Chris Schotanus
 
Search microservice
Search microservice
Jean Carlo Machado
 
Continues Integration for Android
Continues Integration for Android
CODETE
 
Project Delivery
Project Delivery
CODETE
 
There and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metrics
Zbyszek Mockun
 
Integrated Dev And Qa Team With Scrum
Integrated Dev And Qa Team With Scrum
Ethan Huang
 
Test Driven Development
Test Driven Development
Hicham El Hammouchi
 
PHX Session #3 - "It Works on My Machine!" Closing the Loop Between Developme...
PHX Session #3 - "It Works on My Machine!" Closing the Loop Between Developme...
Steve Lange
 
Test Driven Development (TDD) & Continuous Integration (CI)
Test Driven Development (TDD) & Continuous Integration (CI)
Fatkul Amri
 
Shift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality Assurance
Zbyszek Mockun
 
Agile and test driven development
Agile and test driven development
Ahmed El-Deeb
 
Put "fast" back in "fast feedback"
Put "fast" back in "fast feedback"
Lars Thorup
 
Unit testing for project managers
Unit testing for project managers
Zend by Rogue Wave Software
 
Software management for tech startups
Software management for tech startups
Mahmoud Atef Abdelsamie
 
Test driven development
Test driven development
Sunil Prasad
 
Introduction agile scrum methodology
Introduction agile scrum methodology
Amit Verma
 
Agile vs. waterfall - The fundamentals differences
Agile vs. waterfall - The fundamentals differences
David Tzemach
 
Notes On Agile Development
Notes On Agile Development
Matt Griffin
 
Pair Programming: overview and concepts
Pair Programming: overview and concepts
Lior Kirshner-Shalom
 
The evolution of agile development process
The evolution of agile development process
David Tzemach
 
Introduction to TDD and BDD
Introduction to TDD and BDD
Luis García Castro
 
Overview of Site Reliability Engineering (SRE) & best practices
Overview of Site Reliability Engineering (SRE) & best practices
Ashutosh Agarwal
 
Agile methodology
Agile methodology
Dhruv Kumar
 
Agile Process Introduction
Agile Process Introduction
Nguyen Hai
 
What is this agile thing anyway
What is this agile thing anyway
Lisa Van Gelder
 
Comprehensive Performance Testing: From Early Dev to Live Production
Comprehensive Performance Testing: From Early Dev to Live Production
TechWell
 
Things to Be Considered Regarding Continuous Performance Testing.pdf
Things to Be Considered Regarding Continuous Performance Testing.pdf
RohitBhandari66
 

More Related Content

What's hot (20)

Integrated Dev And Qa Team With Scrum
Integrated Dev And Qa Team With Scrum
Ethan Huang
 
Test Driven Development
Test Driven Development
Hicham El Hammouchi
 
PHX Session #3 - "It Works on My Machine!" Closing the Loop Between Developme...
PHX Session #3 - "It Works on My Machine!" Closing the Loop Between Developme...
Steve Lange
 
Test Driven Development (TDD) & Continuous Integration (CI)
Test Driven Development (TDD) & Continuous Integration (CI)
Fatkul Amri
 
Shift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality Assurance
Zbyszek Mockun
 
Agile and test driven development
Agile and test driven development
Ahmed El-Deeb
 
Put "fast" back in "fast feedback"
Put "fast" back in "fast feedback"
Lars Thorup
 
Unit testing for project managers
Unit testing for project managers
Zend by Rogue Wave Software
 
Software management for tech startups
Software management for tech startups
Mahmoud Atef Abdelsamie
 
Test driven development
Test driven development
Sunil Prasad
 
Introduction agile scrum methodology
Introduction agile scrum methodology
Amit Verma
 
Agile vs. waterfall - The fundamentals differences
Agile vs. waterfall - The fundamentals differences
David Tzemach
 
Notes On Agile Development
Notes On Agile Development
Matt Griffin
 
Pair Programming: overview and concepts
Pair Programming: overview and concepts
Lior Kirshner-Shalom
 
The evolution of agile development process
The evolution of agile development process
David Tzemach
 
Introduction to TDD and BDD
Introduction to TDD and BDD
Luis García Castro
 
Overview of Site Reliability Engineering (SRE) & best practices
Overview of Site Reliability Engineering (SRE) & best practices
Ashutosh Agarwal
 
Agile methodology
Agile methodology
Dhruv Kumar
 
Agile Process Introduction
Agile Process Introduction
Nguyen Hai
 
What is this agile thing anyway
What is this agile thing anyway
Lisa Van Gelder
 
Integrated Dev And Qa Team With Scrum
Integrated Dev And Qa Team With Scrum
Ethan Huang
 
PHX Session #3 - "It Works on My Machine!" Closing the Loop Between Developme...
PHX Session #3 - "It Works on My Machine!" Closing the Loop Between Developme...
Steve Lange
 
Test Driven Development (TDD) & Continuous Integration (CI)
Test Driven Development (TDD) & Continuous Integration (CI)
Fatkul Amri
 
Shift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality Assurance
Zbyszek Mockun
 
Agile and test driven development
Agile and test driven development
Ahmed El-Deeb
 
Put "fast" back in "fast feedback"
Put "fast" back in "fast feedback"
Lars Thorup
 
Test driven development
Test driven development
Sunil Prasad
 
Introduction agile scrum methodology
Introduction agile scrum methodology
Amit Verma
 
Agile vs. waterfall - The fundamentals differences
Agile vs. waterfall - The fundamentals differences
David Tzemach
 
Notes On Agile Development
Notes On Agile Development
Matt Griffin
 
Pair Programming: overview and concepts
Pair Programming: overview and concepts
Lior Kirshner-Shalom
 
The evolution of agile development process
The evolution of agile development process
David Tzemach
 
Overview of Site Reliability Engineering (SRE) & best practices
Overview of Site Reliability Engineering (SRE) & best practices
Ashutosh Agarwal
 
Agile methodology
Agile methodology
Dhruv Kumar
 
Agile Process Introduction
Agile Process Introduction
Nguyen Hai
 
What is this agile thing anyway
What is this agile thing anyway
Lisa Van Gelder
 

Similar to 3 Keys to Performance Testing at the Speed of Agile (20)

Comprehensive Performance Testing: From Early Dev to Live Production
Comprehensive Performance Testing: From Early Dev to Live Production
TechWell
 
Things to Be Considered Regarding Continuous Performance Testing.pdf
Things to Be Considered Regarding Continuous Performance Testing.pdf
RohitBhandari66
 
Ensuring Effective Performance Testing in Web Applications.pdf
Ensuring Effective Performance Testing in Web Applications.pdf
kalichargn70th171
 
performance testing training in hyderabad
performance testing training in hyderabad
aparna3zen
 
performance testing training in hyderabad
performance testing training in hyderabad
madhupriya3zen
 
Performance Testing.3zen.pdf
Performance Testing.3zen.pdf
swathi3zen
 
Performance Testing Training in Hyderabad
Performance Testing Training in Hyderabad
rajasrichalamala3zen
 
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...
Codemotion
 
Heuristics of performance testing
Heuristics of performance testing
Pradeep Soundararajan
 
Continuous Performance Testing in DevOps - Lee Barnes
Continuous Performance Testing in DevOps - Lee Barnes
QA or the Highway
 
performance testing training in hyderabad
performance testing training in hyderabad
shaikmadarbi3zen
 
Incorporating Performance Testing in Agile Development Process
Incorporating Performance Testing in Agile Development Process
Michael Vax
 
Are You Ready For More Visitors Cognizant Gomez Jan20
Are You Ready For More Visitors Cognizant Gomez Jan20
Compuware APM
 
Magical Performance tuning with Gomez
Magical Performance tuning with Gomez
mcsaha
 
Neotys PAC 2018 - Ramya Ramalinga Moorthy
Neotys PAC 2018 - Ramya Ramalinga Moorthy
Neotys_Partner
 
Best Practices for Applications Performance Testing
Best Practices for Applications Performance Testing
Bhaskara Reddy Sannapureddy
 
[Webinar] Getting Started with Performance Testing
[Webinar] Getting Started with Performance Testing
Maharshi Shah
 
Performance Testing at Scale Techniques for High-Volume Services
Performance Testing at Scale Techniques for High-Volume Services
Knoldus Inc.
 
Performance testing
Performance testing
Chalana Kahandawala
 
Performance Engineering - how to start!
Performance Engineering - how to start!
Yoav Weiss
 
Comprehensive Performance Testing: From Early Dev to Live Production
Comprehensive Performance Testing: From Early Dev to Live Production
TechWell
 
Things to Be Considered Regarding Continuous Performance Testing.pdf
Things to Be Considered Regarding Continuous Performance Testing.pdf
RohitBhandari66
 
Ensuring Effective Performance Testing in Web Applications.pdf
Ensuring Effective Performance Testing in Web Applications.pdf
kalichargn70th171
 
performance testing training in hyderabad
performance testing training in hyderabad
aparna3zen
 
performance testing training in hyderabad
performance testing training in hyderabad
madhupriya3zen
 
Performance Testing.3zen.pdf
Performance Testing.3zen.pdf
swathi3zen
 
Performance Testing Training in Hyderabad
Performance Testing Training in Hyderabad
rajasrichalamala3zen
 
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...
Codemotion
 
Continuous Performance Testing in DevOps - Lee Barnes
Continuous Performance Testing in DevOps - Lee Barnes
QA or the Highway
 
performance testing training in hyderabad
performance testing training in hyderabad
shaikmadarbi3zen
 
Incorporating Performance Testing in Agile Development Process
Incorporating Performance Testing in Agile Development Process
Michael Vax
 
Are You Ready For More Visitors Cognizant Gomez Jan20
Are You Ready For More Visitors Cognizant Gomez Jan20
Compuware APM
 
Magical Performance tuning with Gomez
Magical Performance tuning with Gomez
mcsaha
 
Neotys PAC 2018 - Ramya Ramalinga Moorthy
Neotys PAC 2018 - Ramya Ramalinga Moorthy
Neotys_Partner
 
Best Practices for Applications Performance Testing
Best Practices for Applications Performance Testing
Bhaskara Reddy Sannapureddy
 
[Webinar] Getting Started with Performance Testing
[Webinar] Getting Started with Performance Testing
Maharshi Shah
 
Performance Testing at Scale Techniques for High-Volume Services
Performance Testing at Scale Techniques for High-Volume Services
Knoldus Inc.
 
Performance Engineering - how to start!
Performance Engineering - how to start!
Yoav Weiss
 
Ad

More from Neotys (20)

PAC 2020 Santorin - Giovanni Paolo Gibilisco
PAC 2020 Santorin - Giovanni Paolo Gibilisco
Neotys
 
PAC 2020 Santorin - Stijn Schepers
PAC 2020 Santorin - Stijn Schepers
Neotys
 
PAC 2020 Santorin - Edoardo Varani
PAC 2020 Santorin - Edoardo Varani
Neotys
 
PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner
Neotys
 
PAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis Chatzinasios
Neotys
 
PAC 2020 Santorin - Gopalkrishnan Yadav
PAC 2020 Santorin - Gopalkrishnan Yadav
Neotys
 
PAC 2020 Santorin - Hari Krishnan Ramachandran
PAC 2020 Santorin - Hari Krishnan Ramachandran
Neotys
 
PAC 2020 Santorin - Joerek Van Gaalen
PAC 2020 Santorin - Joerek Van Gaalen
Neotys
 
PAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur Jain
Neotys
 
PAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen Townshend
Neotys
 
PAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro Melendez
Neotys
 
PAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen Townshend
Neotys
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo
Neotys
 
PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez
Neotys
 
PAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark Tomlinson
Neotys
 
PAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli Aparna
Neotys
 
PAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan George
Neotys
 
PAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van Gaalen
Neotys
 
PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan
Neotys
 
PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux
Neotys
 
PAC 2020 Santorin - Giovanni Paolo Gibilisco
PAC 2020 Santorin - Giovanni Paolo Gibilisco
Neotys
 
PAC 2020 Santorin - Stijn Schepers
PAC 2020 Santorin - Stijn Schepers
Neotys
 
PAC 2020 Santorin - Edoardo Varani
PAC 2020 Santorin - Edoardo Varani
Neotys
 
PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner
Neotys
 
PAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis Chatzinasios
Neotys
 
PAC 2020 Santorin - Gopalkrishnan Yadav
PAC 2020 Santorin - Gopalkrishnan Yadav
Neotys
 
PAC 2020 Santorin - Hari Krishnan Ramachandran
PAC 2020 Santorin - Hari Krishnan Ramachandran
Neotys
 
PAC 2020 Santorin - Joerek Van Gaalen
PAC 2020 Santorin - Joerek Van Gaalen
Neotys
 
PAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur Jain
Neotys
 
PAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen Townshend
Neotys
 
PAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro Melendez
Neotys
 
PAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen Townshend
Neotys
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo
Neotys
 
PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez
Neotys
 
PAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark Tomlinson
Neotys
 
PAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli Aparna
Neotys
 
PAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan George
Neotys
 
PAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van Gaalen
Neotys
 
PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan
Neotys
 
PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux
Neotys
 
Ad

Recently uploaded (20)

Complete WordPress Programming Guidance Book
Complete WordPress Programming Guidance Book
Shabista Imam
 
Test Case Design Techniques – Practical Examples & Best Practices in Software...
Test Case Design Techniques – Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
 
Advance Doctor Appointment Booking App With Online Payment
Advance Doctor Appointment Booking App With Online Payment
AxisTechnolabs
 
arctitecture application system design os dsa
arctitecture application system design os dsa
za241967
 
Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factory
Maxim Salnikov
 
Simplify Insurance Regulations with Compliance Management Software
Simplify Insurance Regulations with Compliance Management Software
Insurance Tech Services
 
Introduction to Agile Frameworks for Product Managers.pdf
Introduction to Agile Frameworks for Product Managers.pdf
Ali Vahed
 
Humans vs AI Call Agents - Qcall.ai's Special Report
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
 
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
2nd Sight Lab
 
HYBRIDIZATION OF ALKANES AND ALKENES ...
HYBRIDIZATION OF ALKANES AND ALKENES ...
karishmaduhijod1
 
Automated Testing and Safety Analysis of Deep Neural Networks
Automated Testing and Safety Analysis of Deep Neural Networks
Lionel Briand
 
Y - Recursion The Hard Way GopherCon EU 2025
Y - Recursion The Hard Way GopherCon EU 2025
Eleanor McHugh
 
AI for PV: Development and Governance for a Regulated Industry
AI for PV: Development and Governance for a Regulated Industry
Biologit
 
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Philip Schwarz
 
Zoho Creator Solution for EI by Elsner Technologies.docx
Zoho Creator Solution for EI by Elsner Technologies.docx
Elsner Technologies Pvt. Ltd.
 
declaration of Variables and constants.pptx
declaration of Variables and constants.pptx
meemee7378
 
NEW-IDM Crack with Internet Download Manager 6.42 Build 27 VERSION
NEW-IDM Crack with Internet Download Manager 6.42 Build 27 VERSION
grete1122g
 
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
pcprocore
 
University Campus Navigation for All - Peak of Data & AI
University Campus Navigation for All - Peak of Data & AI
Safe Software
 
Why Edge Computing Matters in Mobile Application Tech.pdf
Why Edge Computing Matters in Mobile Application Tech.pdf
IMG Global Infotech
 
Complete WordPress Programming Guidance Book
Complete WordPress Programming Guidance Book
Shabista Imam
 
Test Case Design Techniques – Practical Examples & Best Practices in Software...
Test Case Design Techniques – Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
 
Advance Doctor Appointment Booking App With Online Payment
Advance Doctor Appointment Booking App With Online Payment
AxisTechnolabs
 
arctitecture application system design os dsa
arctitecture application system design os dsa
za241967
 
Azure AI Foundry: The AI app and agent factory
Azure AI Foundry: The AI app and agent factory
Maxim Salnikov
 
Simplify Insurance Regulations with Compliance Management Software
Simplify Insurance Regulations with Compliance Management Software
Insurance Tech Services
 
Introduction to Agile Frameworks for Product Managers.pdf
Introduction to Agile Frameworks for Product Managers.pdf
Ali Vahed
 
Humans vs AI Call Agents - Qcall.ai's Special Report
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
 
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
2nd Sight Lab
 
HYBRIDIZATION OF ALKANES AND ALKENES ...
HYBRIDIZATION OF ALKANES AND ALKENES ...
karishmaduhijod1
 
Automated Testing and Safety Analysis of Deep Neural Networks
Automated Testing and Safety Analysis of Deep Neural Networks
Lionel Briand
 
Y - Recursion The Hard Way GopherCon EU 2025
Y - Recursion The Hard Way GopherCon EU 2025
Eleanor McHugh
 
AI for PV: Development and Governance for a Regulated Industry
AI for PV: Development and Governance for a Regulated Industry
Biologit
 
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding 𝑢𝑛𝑓𝑜𝑙𝑑 as the Computational Dual of ...
Philip Schwarz
 
Zoho Creator Solution for EI by Elsner Technologies.docx
Zoho Creator Solution for EI by Elsner Technologies.docx
Elsner Technologies Pvt. Ltd.
 
declaration of Variables and constants.pptx
declaration of Variables and constants.pptx
meemee7378
 
NEW-IDM Crack with Internet Download Manager 6.42 Build 27 VERSION
NEW-IDM Crack with Internet Download Manager 6.42 Build 27 VERSION
grete1122g
 
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
pcprocore
 
University Campus Navigation for All - Peak of Data & AI
University Campus Navigation for All - Peak of Data & AI
Safe Software
 
Why Edge Computing Matters in Mobile Application Tech.pdf
Why Edge Computing Matters in Mobile Application Tech.pdf
IMG Global Infotech
 

3 Keys to Performance Testing at the Speed of Agile

  • 1. 3 Keys to Performance Testing at the Speed of Agile Matching the imperative of high-performance with the pace of modern software delivery
  • 2. 1. This session is being recorded 2. Discussion in chat panel 3. Q & A at the end of session 4. Follow-up email with links to presentation Housekeeping
  • 3. 1. Agile Challenges and Anti-patterns 2. The Goal: Potentially Confidently Shippable Product 3. 3 Keys to Efficient Performance Testing in Sprints 4. Q&A Roundtable What We’ll Cover
  • 4. Today’s Presenter Paul Bruce Sr. Performance Engineer, Neotys USA @paulsbruce @neotys linkedin.com/in/paulsbruce
  • 5. Quick Poll: When do you measure system performance? [multiple choice] ● Just before MAJOR releases ● Just before MINOR releases ● A few times, throughout sprints ● As part of a Continuous Delivery pipeline ● No schedule, project-based, as-needed
  • 6. System performance is imperative to all business models 14% 2% 6% max avg min Qualys - bit.ly/2D6OJWU 1 of every 3 organizations take longer than 30 days to patch/resolve security issue. Slow Down (Lose $$$) Vulnerable (Mitigate Risk) ? Intel Newsroom - intel.ly/2B4DWL3 Databricks -bit.ly/2Dd51h4
  • 7. Consistency: across all digital properties (web, mobile, API…) Fast activity completion: checkout, subscribe, approve Abandonment: page or mobile app uninstalls Ubiquitous connectivity: WiFi, mobile, broadband… What kind of performance do customers expect?
  • 8. “Faster” Isn’t Good Enough Anymore Value Customer Centricity Business Technology Accelerated Delivery Small, frequent changes Efficiency: skills, process, tools
  • 9. “We didn’t bother with scalability because they wanted an MVP ASAP…” “The stakeholders didn’t say they want to integrate this with [that other service]...” “We’ll deliver documentation...if there’s time at the end of the sprint.” “Security is for a hardening stage...which as usual, got eaten up by bug fixing…” “We’ve been doing stand-ups for weeks, and no one mentioned performance criteria!” “That isn’t in the definition of done, so the customer shouldn’t expect it in the product.” “That’s someone else’s area of expertise...I’ll assign the task to them next.” Agile “faster theater” anti-patterns
  • 10. Finely-tuned batch sizes and automation solve many (but not all) problems. ● Resource intensive human-only tasks ○ Planning ○ Estimation ○ Prototyping ○ Code reviews ● Resource intensive automation tasks ○ Regression testing ○ Migrations / upgrades ○ Performance testing ● Trend-driven analysis ○ Build and release metrics ○ Production actuals (e.g. A/B testing) ○ Customer feedback / reviews / NPS Real Work Is Lumpy, but Still Necessary finalization work spin-up finalization work spin-up Small Batch SizeBig Batch Size Same transaction overhead “Right fit” is the biggest challenge.
  • 11. 1. What is ‘Progressive Testing’? 2. What needs to be performance tested (and when)? 3. The “right” performance test for the job 4. Example triggers and schedules The Goal: Confidently Shippable Product
  • 12. Progressive Testing Across the Delivery Pipeline Feature X… API update Y… Patch N...For: Time budget for testing: seconds minutes hours Days Weeks
  • 13. Each layer must be tested for performance:. API, database, web/mobile, networks, 3rd parties, cloud infrastructure, failover/recovery behaviors. What needs to be load tested? Data Stores Core microservices 3rd-party APIs Load Balancers Front-facing servers Network / Edges Short, fast performance feedback loop Longer burn / more complex performance testing API perf testing DB latency testing Windowed perf tests Web & mobile testing Spike/failover testing Distributed geo load
  • 14. . The “right” performance test for the job Real-world dependencies Easy & fast Takes time
  • 15. The “right” performance test for the job Development and Architecture ● Load (does it handle N users?) ○ Cloud migration baselines ○ Build-over-build performance trends ○ Meeting contractual SLA obligations ● Soak (are there leaks or degradations over time?) ○ Server RAM, CPU, disk ● Spike (how does it handle dramatic changes?) ○ Load balancing ○ Rate throttling ○ Garbage collection Operations and Change Management ● Stress (where is the max?) ○ Capacity planning ○ EUX degradation and ajax timeout handling ● Configuration (is this tweak better or detrimental?) ○ Routing, SSL termination, database tuning ○ Logging, retention policies ● Network (what latencies are in the pipes?) ○ 3rd-party API SLAs ○ Service distribution (architectural latency) ○ Multi-cloud deployments
  • 16. Example: Triggers and schedules for performance testing Event Max CI duration DB API Web & mobile 3rdParty Infra Network Regression Code check- in 10-15 min Sm Sm Sm Merge 15-30 min Med Med Med Overnight 1-4hrs Med/Lg Med/Lg Med/Lg Sm Weekly Lg Lg Lg Sm Med Med Med Pre- release Lg Lg Lg Lg Lg Lg Lg Test Coverage (Sm / Med / Lg): ● Volume (VUs) ● Scope (# flows) ● Conditions (geo, wifi/LTE)Small Medium Large “Right fit” is the biggest challenge.
  • 17. 1. Integrate performance into planning phase 2. Establish “right-sized” feedback loops 3. Reduce waste in scripting and analysis 3 Keys to Efficient Performance Testing
  • 18. 2017 study: ● Release every other day; 2x user stories w/ performance criteria Key #1: Integrate performance into planning phase bit.ly/performance-feature How to “bake performance in” to user stories? ● “Back of card” / DoD - performance criteria ● “Think 4th dimensionally” → Time ● Concerns and guidelines, not testing performed ● Performance-specific: ○ Concurrency: “how many users at the same time?” ○ Conditions: “which devices from where?” ○ Capacity: “what resources will this take?”
  • 19. Key #1: Integrate performance into planning phase techbeacon.com/how-build-performance-your-user-stories Todd DeCapua - @appperfeng
  • 20. ● Small load test on new features ● Critical-path performance regression ● Baselines and comparisons accessible to everyone ● Including metrics that tell the complete story: ○ The “efficacy” of the test (load health and bias) ○ The “impact” side of the story (server monitoring) ○ The “customer” side of the story (end-user experience) Key #2: Establish Right-Sized Feedback Loops Value vs. time cost Risk Prevention Early Detection End-to-end visibility
  • 21. Key #2: Establish Right-Sized Feedback Loops
  • 22. Key #3: Reduce Waste in Scripting Time ● Record through a browser, low-code approaches ● Carry customizations during re-scripting / recording ● Build library of reusable test assets ● Reuse existing functional test assets (“Self-healing” load scripts) ● Share test assets across teams and engineers Easy & Intuitive Efficient Consistent Realistic Collaborative
  • 23. Key #3: Reduce Waste in Scripting Time
  • 24. 1. Integrate performance into planning phase 2. Establish “right-sized” feedback loops 3. Reduce waste in scripting and analysis 3 Keys to Efficient Performance Testing
  • 25. 1. Agile Challenges and Anti-patterns 2. The Goal: Confidently Shippable Product 3. 3 Keys to Efficient Performance Testing in Sprints What We’ve Covered
  • 26. Q&A Roundtable Paul Bruce Sr. Performance Engineer, Neotys USA @paulsbruce @neotys linkedin.com/in/paulsbruce