The DevOps Dance - Shift Left, Shift Right - Get It Right

®
The DevOps Dance:
Shift Left, Shift Right
Get It Right
Presented by
Adam Sandman

®
2 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
®
About Me
 Adam Sandman was a programmer
from the age of 10 and has been
working in the IT industry for the past
20 years.
 Currently Adam is a Director of
Technology at Inflectra Corporation,
where he is interested in technology,
business and innovation.
 Adam lives in Washington, DC, USA

®
Agenda
The DevOps Dance
 What is Continuous Testing?
Shifting “Left”
Shifting “Right”
Stuck in the Middle with You
 Assessing risk and the decision to release
Questions?

®
®
Takeaways
1. How you can shift left by adding more testing to your
DevOps Pipeline
2. Ways to shift right by adding monitoring and tracing for
applications in production
3. Putting it all together into an integrated end to end test
management process
4. Some tools both open source and commercial that can
help along the way

®
The DevOps Dance
What is Continuous Testing?

®®
What is DevOps Anyway?
(By Devops.png: Rajiv.Pant derivative work: Wylve - derived from Devops.png: , Link, originally by Gary Stevens)

®®
The DevOps “Lifecycle”
(By Devops.png: Rajiv.Pant derivative work: Wylve - derived from Devops.png: , Link, originally by Gary Stevens)

®®
What is Continuous Testing?
 Continuous testing is the process of executing tests as part
of the software delivery pipeline to obtain immediate
feedback on the business risks associated with a software
release candidate.
 Originally limited to automated tests in the CI portion
 Should include all kinds of testing, throughout CI/CD pipeline
 For Continuous testing, the scope of testing extends from
validating bottom-up requirements or user stories to
assessing the system requirements associated with
overarching business goals.
1. https://en.wikipedia.org/wiki/Continuous_testing

®®
Adding - Continuous Testing

Adding - Continuous Testing
TESTING
TESTING
TESTING
TESTING

®®
Key Facts About Continuous Testing
 Without Continuous Testing, you won’t realize significant
value from DevOps.
 Pushing low quality code into production faster = bad
 Lack of visibility and feedback to assess risk = bad
 Chaos engineering without testing = simply chaos!
 Continuous Testing Myths
 Continuous Testing ≠ 100% test automation
 All testing should be done earlier in the lifecycle
 No testing is needed, just have production monitoring

®®
Some Best Practices
 Continuous means testing is happening before, during, and
after each software change is made.
 Collaborate on requirements
 Validate changes constantly (e.g.: regression testing)
 Dev/Test Pairing (ex: exploratory testing, reviewing test
cases)
 Automated testing in CI
 Continuous improvement of test approach
 Review of customer feedback and business metrics

®®
Introducing The DevOps Dance
TESTING
TESTING
TESTING
TESTING

®®
Shift-Left
TESTING
TESTING
TESTING
TESTING
SHIFT LEFT

®®
Shift-Right
TESTING
TESTING
TESTING
TESTING
SHIFT RIGHT

®
Shifting Left
Testing Early, in the CI Pipeline

®®
What Does “Shift Left” Mean?
 Shifting Left is about removing downstream blockers and
finding and fixing defects closer to where they are
introduced.
 Rapid, automated testing to prevent known issues reappearing
 Identifying areas of risk earlier to know where to focus later:
 Performance
 Security
 Integration
 Infrastructure
 Usability
 Early exploration of features to understand business issues/risks

®®
Continuous Integration (CI)
 Ensure that you have a robust set of “smoke”
tests that can run after every CI build
 Unit, API and Automated UI tests

Unit Testing
Plugins Available for:

®®
User Interface Testing
• Automated Tests • Manual Tests
You need both to really test and verify!

®®
Non-Functional Testing
• Don’t leave these to the end of the process, before deployment
• Do just enough of each type of testing early in the pipeline to
determine if further testing will be needed later.

®®
Exploratory Testing: Where & Why
Good testers find issues that no one knows about
Exploratory testing finds those edge cases
It’s the unknown ‘unknowns’ that trip you up
It’s particularly useful when the functionality is still
being formed in early sprints

®®
Some Tools
 Dependency checkers – DependencyCheck
 Static Code Analysis – Parasoft, SonarQube
 Unit Tests – xUnit (jUnit, Nunit, PyTest, etc.)
 Performance Tests – JMeter, Gatling, NeoLoad
 API Testing – Postman, Rapise, SoapUI
 Security Testing – OWASP ZAP
 UI Testing – Rapise, Selenium
 Exploratory Testing – SpiraCapture

®
Shifting Right
Monitoring in Production, after CD

®®
Shifting Right
 Shifting Right is about leveraging customers and access
to production data to test effectively and support
feedback loops
 Enabling testing activities
 Customer feedback on features (ex: A/B and Canary Testing)
 Monitoring and healing production issues
 Usability and experience (ex: usability testing, UI/UX)
 Customer satisfaction (ex: NPS ratings, engagement, adoption)
 Monitoring business metrics to “test” value of changes

®®
Continuous Deployment (CD)
Automating the process from release ready, tests passed, to
production, post-release checks.

Business Monitoring
Business
metrics can
be a useful
indicator of
a problem
that system
monitoring
may have
missed!

®®
Some Tools
 System Monitoring – Dynatrace, Solarwinds, Cloudwatch
 Customer Helpdesk – ZenDesk, KronoDesk, Remedy, etc.
 Customer Behavior – Google Analytics, Splunk, Matomo
 Business Monitoring
 Traditional BI Platforms – PowerBI, Splunk, Cognos, Domo, etc.
 AI-Driven Data Analytics – Qlik, Tellius, etc.
 Data Visualzation – Tableau, etc
 Customer Satisfaction – Delighted, Promoter, AskNicely, …

®
Stuck in the Middle with You
Assessing Risk and Deciding Whether to Go Live

®®
There Is Never Enough Time for Testing
• How Do We Know if We Can Release into Production?
• How Do We Know We Have Tested the Right Areas?
• We Don’t Have Enough Time to Test Everything!

®
Testing = Manage Technical Risk

Focus on Areas Identified Earlier in Pipeline
Unit Tests
API Tests
Security Tests
Performance Tests
Exploratory Sessions
Shift Left Tests Deeper Testing
Deeper Functional
Testing
Deeper Performance
Testing
Focused Exploratory
Sessions

®®
Example: Performance Testing
Early Load Testing
• Development hardware
• No isolation
• 10 VUs, 10,000 requests
• Analyze trends
• Is this CI build faster or
slower than before?
• Should we spend more time
testing it later?
Full-Blown Load Testing
• Replica of production
hardware
• Dedicated environment
• Isolated networks
• 10,000 VUs for 4 hours
• What is the maximum load,
throughput, bottlenecks
• Do we meet our SLAs?

Hierarchy of Testing Risk
Happy Path, Safe Environment
Realistic Paths, Safe Environment
Edge Cases, Safe Environment
Edge Cases, All Environments
If it breaks here, then
no point trying further
levels in the hierarchy

®
All The Automated Tests
Pass
Should We Ship?

®®
User Stories
Requirements
Automated Testing
Exploratory Testing: Navigating Edges

®®
The Actual System

®®
The Actual System
Edge Cases

®
Evaluating Risk & Deciding
The decision to release is a
business not technical decision

®
Wrap Up & Final Thoughts
1. Integrate Shift-Left techniques into your CI Pipeline
 Address risk areas identified early to focus deeper testing
2. Add Shift-Right techniques into your post-deployment
infrastructure, so that testing doesn’t end after live
3. Have a risk-based approach to deciding when to release
functionality into production

®
Questions?

®
Thank you for attending this
workshop/session.
Please fill out an evaluation form.

The DevOps Dance - Shift Left, Shift Right - Get It Right

Recommended

More Related Content

What's hot (20)

Similar to The DevOps Dance - Shift Left, Shift Right - Get It Right (20)

More from Inflectra (20)

Recently uploaded (20)

The DevOps Dance - Shift Left, Shift Right - Get It Right