Continuous Performance Testing
Download as PPTX, PDF1 like612 views
This document discusses continuous performance testing (CPT) and Grid Dynamics' CPT solution called Jagger. It begins with an introduction to CPT and the benefits of including performance tests in the continuous integration process. It then describes the key parts of a CPT solution including the testing platform, test data management, and environment management. The document focuses on Jagger, providing an overview and specifications of Grid Dynamics' CPT platform. It concludes with a customer success story demonstrating how Jagger helped a large e-commerce site address performance issues and pass sign-off tests.
Continuous Performance Testing
- 1. 1 Private and Confidential1 Private and Confidential Continuous Performance Testing Mikhail Klokov, QA Technical Director 08/10/2016
- 2. 2 Private and Confidential • Section 1: Introduction to CPT • Section 2: CPT solution • Section 2: Jagger CPT Solution • Section 4: Customer Success Story 2 Agenda
- 3. 3 Private and Confidential Why performance testing? 2012: Research showed that Amazon would lose $1.6 billion in sales every year if its site took one more second to load. 2013: 39% of e-retailers claimed they lost money last year due to performance or stability problems. 2014: The web performance monitoring company Catchpoint Systems looked at aggregate performance on Black Friday and compared it to the same timeframe in 2013.The results are notable: desktop web pages were 19.85 percent slower, while mobile web pages were a whopping 57.21 percent slower. 2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday traffic during 2015’s Cyber Monday peak traffic times.
- 4. 4 Private and Confidential Section 1: Introduction to CPT
- 5. 5 Private and Confidential Performance only as sign off Commit Build Unit Smoke Regression Integration Release deployment Performance tests and sign off Security sign off UAT sign off Functional E2E Release Candidate
- 6. 6 Private and Confidential Performance impact of each change Add new feature Functionality check Performance impact
- 7. 7 Private and Confidential Implementation issues Cost Lack of skills Availability of adequate performance testing environments Immature continuous integration and continuous delivery practices No focus on performance tests, because other tests are “more important
- 8. 8 Private and Confidential Commit Build Unit Smoke Perf Smoke Regression Perf Regression Integration Perf E2E Include performance tests to CI loop Release deployment Performance sign off Security sign off UAT sign off Functional E2E Release Candidate
- 9. 9 Private and Confidential Principles #1: Divide and conquer: stage your test execution #2: One throat to choke: target APIs for all testing goals #3: Keep your eyes on the prize: get your risks and KPIs right #4: Automate everything: one-click automation must be end-to-end #5: Analyze this: discover, visualize and deliver performance insights #6: Retain all results: value of data grows over time
- 10. 10 Private and Confidential CPT goals Certify that the application meets performance targets. Plan capacity and manage growth. Track useful performance metrics. Identify load-related weaknesses and track SLA break points.
- 11. 11 Private and Confidential Section 2: CPT Solution
- 12. 12 Private and Confidential Solution parts CPT PLATFORM - test framework, results storage, metrics collection, and visualization TEST DATA MANAGEMENT - data manipulation to establish the CPT process ENVIRONMENT MANAGEMENT - controlling the performance test environment life cycle
- 13. 13 Private and Confidential CPT platform Create test loads for our SUT (System Under Test) Collect metrics and store the results Provide an analytics and visualization engine Integrate with the CI/CD pipeline
- 14. 14 Private and Confidential Test data management Generate data based on existing subsets Always use fresh test data Compare test results on test data and production Mask any sensitive test data
- 15. 15 Private and Confidential Environment management Cloud Provisioning on demand Automated configuration Automated deployment
- 16. 16 Private and Confidential Section 3: Jagger, Grid Dynamics’ CPT platform
- 17. 17 Private and Confidential Bird’s Eye View on Jagger
- 18. 18 Private and Confidential Summary view
- 19. 19 Private and Confidential Trends between different test sessions
- 20. 20 Private and Confidential Detailed report: Parameters during test execution
- 21. 21 Private and Confidential Test environment configuration
- 22. 22 Private and Confidential Jagger specifications • General • Performance test in Jagger – Java project • To develop test, some basic knowledge in programming is required • Simple tests can be implemented just by XML editing • How to start • Download and start test project example: https://github.com/griddynamics/jagger/wiki/Getting-Started • Download test project with customization examples: https://github.com/griddynamics/jagger/wiki/Samples
- 23. 23 Private and Confidential Section 4: Customer Success Story
- 24. 24 Private and Confidential Preconditions Big eCommerce site – Top 3 on US Retailer Load during Holiday season – 5.4k TPS for cluster Only sign-off performance testing process Regular issues on release stage which cause release postponed and expensive fixes New project for critical service – new search engine and catalog
- 25. 25 Private and Confidential What was done Evaluating specific SLAs for search service Creating test data based on user request from production Implementing tests and configure automation execution on build server Introduction Performance testing to CI loop
- 26. 26 Private and Confidential Results Catch performance issues of search engine on stage of development Successfully pass performance sign-off as part of full site No failures during high season
- 27. 27 Private and Confidential Thank you! Mikhail Klokov: [email protected] Grid Dynamics blog: blog.griddynamics.com Follow up on twitter: @griddynamics Jagger: jagger.griddynamics.net We are hiring! griddynamics.com/careers