Tips and tricks for performance testing CI. Лайфхак для безперервного процесу тестування навантаження, Микита Кричко
Download as PPTX, PDF1 like94 views
This document provides guidance on continuous performance testing. It discusses establishing a performance testing pipeline that runs tests automatically as part of a CI/CD process. Key aspects covered include storing results in a database, generating reports to identify trends and compare builds, and using layer-specific and scenario-based reports to find bottlenecks. The document also addresses challenges like dealing with outlier results and detecting anomalies between test runs.
Tips and tricks for performance testing CI. Лайфхак для безперервного процесу тестування навантаження, Микита Кричко
- 2. WHAT IS YOUR ROLES IN PROJECT • DEVELOPER • QA • PM • Other ? WHAT IS YOUR EXPIRIENCE IN PERFORMANCE TESTING? WHAT DO YOU WANT FROM PERFORMANCE TESTING? AGENDA • Timeframe? • What to skip? • What to focus on? WHO ARE YOU PEOPLE?
- 3. 24.05.2018 / 4 TIP and TRICKS in CONTINIOUS PERFORMANCE TESTING
- 4. PERFORMNCE TESTING AGENDA • Timeframe? • What to skip? • What to focus on? GOALS BASE-LINE Understand current status of application performance RESEARCH Find problems and bottlenecks CONTROL Execute after each iteration of development, to control that performance does not decrease
- 5. CONTINIOUS PERFORMANCE TESTING AGENDA • Timeframe? • What to skip? • What to focus on? GOALS FOR CONTINIOUS PERFORMANCE TESTING DEFINE • Issue without human interaction • Trend: is it better or worse • Place of problem • Responsible person DECREASE TIME • Find problems and bottlenecks • Recreate problems • Diagnostic problem
- 6. REQUIREMENTS TO SYSTEM AGENDA • Timeframe? • What to skip? • What to focus on? WHEN IMPLEMENT CI is already implemented Dedicated environment for PT Easy deployment process • Application, DB, system, parts of the system • Test data PT scenarios are ready Data storage for result Access to all monitoring's and logs
- 7. HOW TESTS APPEARS Deployment Health check Functional smoke test Performance testing pipeline Save results to DB Save logs to DB Automatic analysis Generate report /result/ dashboards Separate scenario Etalon test Test on target load Increase load Stress test Combined scenario (flow) Etalon test Test on target load Increase load Stress test
- 8. • Response time • Network delay • Network traffic • Disk I/O • CPU usage • per server • per process • Memory usage • per server • per process • OtherAGENDA • Timeframe? • What to skip? • What to focus on? MEASUREMENTS METRICS x server x process
- 9. • Any DB • Table for RAW data from performance tool • Table for resource-log data • Table for aggregated statistic • Add column – TPS & RPS • Save RAW result for 30 days • Primary key and foreign key should be timestamp (round to 1 sec) AGENDA • Timeframe? • What to skip? • What to focus on? RESULT STORAGE
- 10. AGENDA • Timeframe? • What to skip? • What to focus on? PRIMARY KEY Time stamp milisecond Request name CPU (server 1) CPU (server 2) 1526288760 001 R_1 1526288760 054 R_2 1526288760 112 56 1526288760 176 R_3 1526288760 245 54 1526288760 540 R_4 1526288760 664 57 1526288760 782 56 1526288760 891 R_5 1526288760 953 R_6 During one second
- 12. AGENDA • Timeframe? • What to skip? • What to focus on? PERFORMANCE TESING PIPELINE Functional test (pre-commit, integration, functional, GUI tests) Deployment to performance environment Smoke test (integration, functional tests) PT: Etalon test PT: separate test on target load PT: step separate test to stress load PT: step combined test to stress load Save results and generate report
- 13. Fast identify layer of problem PT: smoke test – check successful deployment – functionality should work PT: Etalon test – fast check on minimal load (1 TPS) AGENDA • Timeframe? • What to skip? • What to focus on? WHY DO ALL THAT TEST?
- 14. PT: Separate test on target load (simple scheme) Response time = REST + SOAP + SQL and REST RT = SOAP + SQL + delay SOAP RT = SQL + delay TESTS: 1. SQL 2. SOAP 3. REST AGENDA • Timeframe? • What to skip? • What to focus on? WHY DO ALL THAT TEST? x number of scenarios
- 15. Fast identify layer of problem PT: step separate test to stress load PT: step combined test to stress load AGENDA • Timeframe? • What to skip? • What to focus on? WHY DO ALL THAT TEST?
- 16. AGENDA • Timeframe? • What to skip? • What to focus on? HOW FIND BOTTLE NECK IN SHORTEST TIME REST SOAP SQL
- 17. STEP BY STEP AGENDA • Timeframe? • What to skip? • What to focus on? HOW BUILD PROCESS OF CI IN PT Manual performance testing CI of Performance testing Automated execute Save results (log) history Automated analysis Automated reports Prepare Execute Analysis Report Upgrade
- 18. AGENDA • Timeframe? • What to skip? • What to focus on? AUTOMATED EXECUTE Jmeter generator
- 19. AGENDA • Timeframe? • What to skip? • What to focus on? AUTOMATED REPORTS Jmeter generator GENERATE OWN REPORT (Overview) SAVE REPORT
- 20. AGENDA • Timeframe? • What to skip? • What to focus on? COMPARE REPORTS USER CI TOOL PT TOOL Application + Infrastructure Report tool DB Monitoring tool START TEST START TEST GENERATE LOAD GENERATE DATA SAVE DATA CALCULATE DATA GENERATE REPORTRETURN DATA RETURN REPORT
- 21. 0 5 10 15 20 25 21-11-17 22-11-17 23-11-17 24-11-17 25-11-17 26-11-17 27-11-17 28-11-17 Positive trends Positive Negative WHY DO ALL THAT TEST?DIFFERENT TRENDS OF RESULTS 0 5 10 15 20 25 21-11-17 22-11-17 23-11-17 24-11-17 25-11-17 26-11-17 27-11-17 28-11-17 Negative trends Positive Negative
- 22. TRUE or FALSE – only for Boolean Boolean is only for defined result for one measure Its possible in case of SLA But PT has much more measurements: • Response time • Network delay • Network traffic • Disk I/O • CPU usage • per server • per process • Memory usage • per server • per process • Other AGENDA • Timeframe? • What to skip? • What to focus on? PROBLEM WITH RESULTS
- 23. WHAT VALUE WE SHOULD USE FOR MEASURE: • Minimum • Average • Mode • Median • Quantile • Maximum AGENDA • Timeframe? • What to skip? • What to focus on? PROBLEM WITH RESULTS
- 24. AGENDA • Timeframe? • What to skip? • What to focus on? PROBLEM WITH RESULTS https://mathwithbaddrawings.com/2016/07/13/why-not-to-trust-statistics/
- 25. WHO KNOWS what is BOXPLOT? AGENDA • Timeframe? • What to skip? • What to focus on? QUASTION TO AUDITORY
- 26. PROBLEM WITH RESULTS 10 ms 20 ms 20 ms 20 ms 30 ms 30 ms 30 ms 40 ms 40 ms 40 ms 60 ms 200 ms All results: 10 - 200 Results Average: 45 Median: 30 IQR (most of 50 %): 20-40 IQR±2 σ (97%): 10-60 0 10 30 50 60 100 150 200
- 27. AGENDA • Timeframe? • What to skip? • What to focus on? BOXPLOT
- 28. AGENDA • Timeframe? • What to skip? • What to focus on? TRENDS REPORTS 5 10 20 30 50 80 90 100 5000 2500 0
- 29. AGENDA • Timeframe? • What to skip? • What to focus on? TRENDS REPORTS date
- 30. AGENDA • Timeframe? • What to skip? • What to focus on? COMPARE REPORTS 10 20 30 40 50 60 70 80 90 100 500 2500 0
- 31. LAYER REPORT
- 32. AGENDA • Timeframe? • What to skip? • What to focus on? LAYERS REPORT
- 33. AGENDA • Timeframe? • What to skip? • What to focus on? LAYERS REPORT
- 34. WHY DO ALL THAT TEST? 0 1 0 1 0 0 1 0 0 1 1 0 1 0 1 0 1 1 0 1 1 0 1 1 0 Test Scenario_1 Test Scenario_2 Test Scenario_3 Test Scenario_4 Test Scenario_5 Operation_1 Operation_2 Operation_3 Operation_4 Operation_5 HEATMAP MATRIX
- 35. WHY DO ALL THAT TEST?COMPARING WITH TARGET MEASUREMENTS
- 36. WHY DO ALL THAT TEST? 0.1 0.99 0.2 0.99 0.3 0.15 0.9 0.25 0.1 1 0.95 0.3 0.95 0.3 0.86 0.2 1 0.75 0.4 0.7 0.7 0.1 0.99 0.77 0.25 Test Scenario_1 Test Scenario_2 Test Scenario_3 Test Scenario_4 Test Scenario_5 Operation_1 Operation_2 Operation_3 Operation_4 Operation_5 HEATMAP MATRIX
- 37. WHY DO ALL THAT TEST? 0.75 0.6 0.7 0.3 0.75 0.8 0.78 0.2 0.6 0.8 0.4 0.2 0.7 0.78 0.4 0.32 0.3 0.2 0.2 0.32 REST Response time SOAP Response time SQL Response time CPU usage RAM usage REST Response time SOAP Response time SQL Response time CPU usage RAM usage COMPARE REPORTS CORRELATION MATRIX
- 38. WHY DO ALL THAT TEST? COMPARE REPORTS DIFFERENCE BETWEEN BUILDS 0.75 0.6 0.7 0.3 0.8 0.78 0.2 0.4 0.2 0.32 REST Response time SOAP Response time SQL Response time CPU usage RAM usage REST Response time SOAP Response time SQL Response time CPU usage RAM usage 0.73 0.6 0.7 0.3 0.9 0.58 0.2 0.4 0.5 0.32 REST Response time SOAP Response time SQL Response time CPU usage RAM usage REST Response time SOAP Response time SQL Response time CPU usage RAM usage 0.02 0 0 0 -0.1 0.2 0 0 -0.3 0 REST Response time SOAP Response time SQL Response time CPU usage RAM usage REST Response time SOAP Response time SQL Response time CPU usage RAM usage Old build New build Difference
- 40. ANOMALIES Anomalies: • Software • Hardware (VM or resources) • Time • Test data
- 41. OTHER RECOMENDATIONS Don’t believe yourself and results Re-run test : • This day, but change time • Next day same time • Different test data • Wait , maybe it will be fix by itself Bug report: • Change step-to-reproduce into conditions • Prepare magic button for developer • as usual: add more log and more profiling information as you can
- 44. AGENDA • Timeframe? • What to skip? • What to focus on? QUESTIONS