3 Keys to Performance Testing at the Speed of Agile

3 Keys to Performance Testing at the Speed of Agile
Matching the imperative of high-performance
with the pace of modern software delivery

1. This session is being recorded
2. Discussion in chat panel
3. Q & A at the end of session
4. Follow-up email with links to presentation
Housekeeping

1. Agile Challenges and Anti-patterns
2. The Goal: Potentially Confidently Shippable Product
3. 3 Keys to Efficient Performance Testing in Sprints
4. Q&A Roundtable
What We’ll Cover

Today’s Presenter
Paul Bruce
Sr. Performance Engineer, Neotys USA
@paulsbruce @neotys
linkedin.com/in/paulsbruce

Quick Poll: When do you measure system performance?
[multiple choice]
● Just before MAJOR releases
● Just before MINOR releases
● A few times, throughout sprints
● As part of a Continuous Delivery pipeline
● No schedule, project-based, as-needed

System performance is imperative to all business models
14%
2%
6%
max
avg
min
Qualys - bit.ly/2D6OJWU
1 of every 3 organizations
take longer than 30 days
to patch/resolve security issue.
Slow Down
(Lose $$$)
Vulnerable
(Mitigate Risk)
?
Intel Newsroom - intel.ly/2B4DWL3 Databricks -bit.ly/2Dd51h4

Consistency: across all digital properties (web, mobile, API…)
Fast activity completion: checkout, subscribe, approve
Abandonment: page or mobile app uninstalls
Ubiquitous connectivity: WiFi, mobile, broadband…
What kind of performance do customers expect?

“Faster” Isn’t Good Enough Anymore
Value
Customer
Centricity
Business Technology
Accelerated Delivery
Small, frequent changes
Efficiency: skills, process, tools

“We didn’t bother with scalability because they wanted an MVP ASAP…”
“The stakeholders didn’t say they want to integrate this with [that other service]...”
“We’ll deliver documentation...if there’s time at the end of the sprint.”
“Security is for a hardening stage...which as usual, got eaten up by bug fixing…”
“We’ve been doing stand-ups for weeks, and no one mentioned performance criteria!”
“That isn’t in the definition of done, so the customer shouldn’t expect it in the product.”
“That’s someone else’s area of expertise...I’ll assign the task to them next.”
Agile “faster theater” anti-patterns

Finely-tuned batch sizes and automation solve many (but not all) problems.
● Resource intensive human-only tasks
○ Planning
○ Estimation
○ Prototyping
○ Code reviews
● Resource intensive automation tasks
○ Regression testing
○ Migrations / upgrades
○ Performance testing
● Trend-driven analysis
○ Build and release metrics
○ Production actuals (e.g. A/B testing)
○ Customer feedback / reviews / NPS
Real Work Is Lumpy, but Still Necessary
finalization
work
spin-up
finalization
work
spin-up
Small Batch SizeBig Batch Size
Same
transaction
overhead
“Right fit” is the biggest challenge.

1. What is ‘Progressive Testing’?
2. What needs to be performance tested (and when)?
3. The “right” performance test for the job
4. Example triggers and schedules
The Goal: Confidently Shippable Product

Progressive Testing Across the Delivery Pipeline
Feature X… API update Y… Patch N...For:
Time budget for testing:
seconds minutes hours
Days
Weeks

Each layer must be tested for performance:. API, database, web/mobile, networks, 3rd
parties, cloud infrastructure, failover/recovery behaviors.
What needs to be load tested?
Data Stores
Core microservices
3rd-party APIs
Load Balancers
Front-facing servers
Network / Edges
Short, fast performance
feedback loop
Longer burn / more complex
performance testing
API perf testing
DB latency testing
Windowed perf tests
Web & mobile testing
Spike/failover testing
Distributed geo load

.
The “right” performance test for the job
Real-world dependencies
Easy & fast
Takes time

The “right” performance test for the job
Development and Architecture
● Load (does it handle N users?)
○ Cloud migration baselines
○ Build-over-build performance trends
○ Meeting contractual SLA obligations
● Soak (are there leaks or degradations over time?)
○ Server RAM, CPU, disk
● Spike (how does it handle dramatic changes?)
○ Load balancing
○ Rate throttling
○ Garbage collection
Operations and Change Management
● Stress (where is the max?)
○ Capacity planning
○ EUX degradation and ajax timeout handling
● Configuration (is this tweak better or detrimental?)
○ Routing, SSL termination, database tuning
○ Logging, retention policies
● Network (what latencies are in the pipes?)
○ 3rd-party API SLAs
○ Service distribution (architectural latency)
○ Multi-cloud deployments

Example: Triggers and schedules for performance testing
Event Max CI duration DB API Web &
mobile
3rdParty Infra Network Regression
Code check-
in
10-15 min Sm Sm Sm
Merge 15-30 min Med Med Med
Overnight 1-4hrs Med/Lg Med/Lg Med/Lg Sm
Weekly Lg Lg Lg Sm Med Med Med
Pre- release Lg Lg Lg Lg Lg Lg Lg
Test Coverage (Sm / Med / Lg):
● Volume (VUs)
● Scope (# flows)
● Conditions (geo, wifi/LTE)Small
Medium
Large
“Right fit” is the biggest challenge.

1. Integrate performance into planning phase
2. Establish “right-sized” feedback loops
3. Reduce waste in scripting and analysis
3 Keys to Efficient Performance Testing

2017 study:
● Release every other day;
2x user stories w/
performance criteria
Key #1: Integrate performance into planning phase
bit.ly/performance-feature
How to “bake performance in” to user stories?
● “Back of card” / DoD - performance criteria
● “Think 4th dimensionally” → Time
● Concerns and guidelines, not testing performed
● Performance-specific:
○ Concurrency: “how many users at the same time?”
○ Conditions: “which devices from where?”
○ Capacity: “what resources will this take?”

Key #1: Integrate performance into planning phase
techbeacon.com/how-build-performance-your-user-stories
Todd DeCapua - @appperfeng

● Small load test on new features
● Critical-path performance regression
● Baselines and comparisons accessible to everyone
● Including metrics that tell the complete story:
○ The “efficacy” of the test (load health and bias)
○ The “impact” side of the story (server monitoring)
○ The “customer” side of the story (end-user experience)
Key #2: Establish Right-Sized Feedback Loops
Value vs. time cost
Risk Prevention
Early Detection
End-to-end visibility

Key #2: Establish Right-Sized Feedback Loops

Key #3: Reduce Waste in Scripting Time
● Record through a browser, low-code approaches
● Carry customizations during re-scripting / recording
● Build library of reusable test assets
● Reuse existing functional test assets
(“Self-healing” load scripts)
● Share test assets across teams and engineers
Easy & Intuitive
Efficient
Consistent
Realistic
Collaborative

Key #3: Reduce Waste in Scripting Time

1. Agile Challenges and Anti-patterns
2. The Goal: Confidently Shippable Product
3. 3 Keys to Efficient Performance Testing in Sprints
What We’ve Covered

Q&A Roundtable
Paul Bruce
Sr. Performance Engineer, Neotys USA
@paulsbruce @neotys
linkedin.com/in/paulsbruce

3 Keys to Performance Testing at the Speed of Agile

Recommended

More Related Content

What's hot (20)

Similar to 3 Keys to Performance Testing at the Speed of Agile (20)

More from Neotys (20)

Recently uploaded (20)

3 Keys to Performance Testing at the Speed of Agile