No Devops Without Continuous Testing
1 like885 views
The presentation by Arthur Hicken emphasizes the importance of continuous testing in the DevOps framework, highlighting the need for real-time collaboration between development and operations teams throughout the software lifecycle. It identifies challenges such as rapid deployment speeds and the complexity of testing in evolving environments, advocating for the use of automated tools, service virtualization, and effective test management to improve quality and predictability. Ultimately, the document calls for a shift in testing paradigms to accommodate the fast-paced nature of modern software development.
No Devops Without Continuous Testing
- 1. No DevOps Without Continuous Testing Arthur Hicken - Parasoft 1
- 2. Your Presenter Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame and SQLi Hall-of-Shame http://bit.ly/sqlishame Follow him @codecurmudgeon Blog: http://codecurmudgeon.com Web: http://parasoft.com
- 3. Agenda • What the shift from automated to continuous means • How disruption requires changes to how we test software • Addressing gaps between Dev and Ops • Technologies that enable Continuous www.iist.org 3 www.psqtconference.com
- 4. Continuous 4 www.psqtconference.com • Build, Integration, Testing, Release, Delivery, Deployment • Builds on DevOps movement • Enables DevOps reality • High dependency on automation: – processes – assessment – decisions
- 5. DevOps “DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.” – The Agile Admin • No more throwing code over the wall • Design for deployment 5 www.psqtconference.com
- 7. Automated vs Continuous • Just being automated isn’t continuous • Scheduling / frequency • Intermittent manual dependencies • Decisions points – Is it done? – Is it good? – Is it safe? – Can I push it out the door? 7 www.psqtconference.com
- 8. Complexity a Barrier to Testing 3rd Party System Evolving Component Mainframe Scheduled Access
- 9. API Economy Pieces/Parts come from everywhere Definitions changing (upgrades, new versions) Not always available Can’t test with real data without causing problems 9 www.psqtconference.com
- 10. We must Re-invent Test • The penalty for faulty software is increasing – Brand erosion – All industries at risk for total transformation • Technology is faster and more distributed – Containers – Microservices
- 11. Fixed Fixed Impact of Test in the Evolving SDLC Plan Develop Build Test Release Test as a time-boxed event is the prevailing method “Are we done testing?”
- 12. Impact of Test in Evolving SDLC Plan Develop Build Test Release Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test Build Test “Does the release candidate have an acceptable level of risk?”
- 13. Gaps between Dev and Ops What is “done”? Fully testing with external dependencies Full environments and all their variants 13 www.psqtconference.com
- 14. Speed of Deployments Rapid deployments Deliver features and bug fixes Deployments outpace ability to test Too many changes Since we didn’t test enough, Devs are mired in bug fixes Glitches and outages can happen 14 www.psqtconference.com
- 15. QA and Testing • Cannot keep up with the rate of change • Quality lags behind • Do we even know what to test? • Complex technologies including message protocols and services 15 www.psqtconference.com
- 16. Enabling Technologies • Stubs • Service Virtualization • API testing • Test data management • Environment management • Self-service test environments 16 www.psqtconference.com
- 17. How to do it better ? • Give absolute priority to automated tests • Invest time into designing interfaces (API) • Use API (service) testing tools to cover interfaces • Measure the quality of the test
- 18. How to do it better ? • Use stubs and mocks to improve: – Facilitate test automation – Increase code penetration
- 19. How to do it better ? • Use service virtualization to improve testing automation – Isolate at the message layer – Simulate functional scenarios and performance conditions
- 20. How to do it better? Test management •Environments •Data •Dependencies 20 www.psqtconference.com
- 21. • The Challenge – Performance issues found late in the SDLC resulting in large rework and cost – Middleware component able to handle large load but available downstream dependent systems were unable to provide throughput needed Reduced testing time by 27-days for a major commercial bank Capacity Constrained staged environments Database Service Mainframe Application 1000 tps
- 22. Database Servic Mainfram e Applicatio n Reduced testing time by 27-days for a major commercial bank 1000 tps Virtual Asset Virtual Asset Virtual Asset Virtual Asset • The Solution – Identified key use-cases and emulate the behavior of dependent systems. Asset performance setup to match production response values – API and Load testing tools used to validate system under test. Virtual assets changed to model different scenarios • The Business Benefit – Able to predict performances issues and fix before full Perf testing. – Able to run different scenarios not easily achieved with production environment (excessive response times). – Setup for ‘performance test’ reduced to near zero
- 23. – The Challenge • Multiple teams using the same test database • Teams not respecting data integrity & others test data records • Regression tests consistently failing. Takes >1 hour to determine that it was due to “data changes”. • “Real problems” were getting lost in the noise Shared Databas e Eliminated 83% of the configuration time for a major telecom company Test data management for complex transactions ✔ ✖ ✖ Shared Database
- 24. – The Solution • Setup Virtual Assets to model the SQL queries and use API testing tool to manage automated nightly regressions against both virtual assets and live systems – The Business Benefit • Test teams able to focus on ‘real regressions’ and separate out data integrity issues from functional test failure Eliminated 83% of the configuration time for a major telecom company Virtual Asset Virtual Asset Virtual Asset ✔ ✔ ✔
- 25. Reduced wait time for test team by 60% for a major media conglomerate • The Challenge – Large agile development effort to adopt Service Oriented Architecture (SOA) – High risk project but the Test team “stuck waiting for the first build” – Development of functionality was not easy to coordinate as different teams had different schedules; not all finished at the same time Agile/Parallel development limited by system dependencies Iteration Iteration Iteration Team A Team B Team C Iteration Iteration Iteration Iteration Iteration Iteration Iteration Current Development/Testing dependencies
- 26. Reduced wait time for test team by 60% for a major media conglomerate • The Solution – Use descriptions of the new services (WSDL, XSD, example JSON payloads) to build Virtual assets prototyping the new functionality. – Test team builds tests with against the prototypes with API testing tool and the independent development tests use the prototypes to perform early stage Integration Testing • The Business Benefits – Met business goals and timelines, were able to test functionality “as soon as” it was available. Practiced TDD against prototypes to get a head-start on ‘full system testing’ Iteration Iteration Iteration Team A Team B Team C Iteration Iteration Iteration Iteration Iteration Iteration Iteration ✔✖✔✖✔✖
- 27. Extended testing hours for a stock trading broker • The Challenge – Leverages 3rd party (out sourced) system for trade exchanges – Many teams need access, so time is scheduled into 4 hour time windows, and could only test during market hours Capacity Constrained staged environments Database Servic e Mainfram e Applicatio n
- 28. Extended testing hours for a stock trading broker • The Solution – Use Test environment manager tool to setup ‘Test Silos’ where each team can do 24x7 testing against virtual assets Database Mainfram e Applicatio nEnvironment Manager
- 29. Conclusions Emulates dependencies for the Test Environment • Reduces complexity for early stage testing • Increases predictability On-Demand “Test Anytime, Anywhere” • Automated Provisioning for different use-cases • Automated Test Data Management/Simulation Reduce time to configure for test Realistic reusable assets Reduce hardware costs and system access fees
- 30. Q&A • Web – http://www.parasoft.com • Blog – http://alm.parasoft.com – http://codecurmudgeon.com •Social –Facebook: https://www.facebook.com/parasoftcorporation –Twitter: @Parasoft @CodeCurmudgeon –LinkedIn: http://www.linkedin.com/company/parasoft