Ad
More Related Content
What's hot (20)
Similar to Building functional Quality Gates with ReportPortal (20)
Ad
More from Dmitriy Gumeniuk (11)
Ad
Recently uploaded (20)
Building functional Quality Gates with ReportPortal
- 1. Building Quality Gates and automated decisions in CI/CD pipelines. What we learned and built on the top of a regression of 200,000 test cases Dmitriy Gumeniuk EPAM Systems Head of Testing Products CEO ReportPortal.io
- 2. Dmitriy Gumeniuk Head of Testing Products @ EPAM Systems Based in New York Metro Area Leads development of a portfolio of accelerators focused on the usage of Machine Learning and Neural Networks in test automation. While having 16 years of experience in software development, has been contributing to DevTestOps communities by actively speaking at events and organizing the DelEx Conference, aimed to inspire DevTestOps practices. CEO ReportPortal.io /ghdmitry /ghdmitry Dm itr iy Gu me niu k
- 3. ReportPortal.io DelEx Conference Dmitriy Gumeniuk Drill4J.github.io Healenium.io TDSpora.ai QASpace JIRA plugin ReportPortal.io – AI-powered Test Automation Dashboard Drill4J – Test Gap and Impact analysis, regression minimization tool Healenium – self-healing capabilities for Selenium- based test cases TDSpora.ai – ML-based synthetic data generation based on production data QASpace – JIRA plugin for neat Test Case Management
- 5. CODE BUILD TEST REALESE DEPLOY Quality Gates Quality gates are checkpoints at which the project's quality is assessed. They involve a set of predefined criteria that must be met before a software product can progress through the development lifecycle. These gates are used to ensure that the project is meeting the established quality standards and to identify and address any issues that may arise during the development process.
- 6. REALESE DEPLOY Quality Gates Benefits of establishing 1. Improved quality: Quаlity Gates maintain high standards, resulting in reliable and stable products with better user satisfaction. 2. Early identification of issues: Detect potential problems early in the development process, reducing the cost and effort involved in fixing defects. 3. Increased efficiency: Streamline the development process by enforcing clear criteria of acceptance, leading to more efficient use of resources. 4. Better communication: Promote a shared understanding of quality expectations among team members at all stages, fostering improved teamwork. 5. Enhanced risk management: Mitigate risks by ensuring software meets predefined quality criteria before release, reducing the likelihood of issues impacting users. 6. Faster time-to-market: Identify and address issues early, shortening the time it takes to bring a high-quality product to market. 7. Informed decision-making: Quality Gates provide data-driven insights on software quality, enabling stakeholders to make informed decisions regarding product releases.
- 7. CODE BUILD TEST REALESE DEPLOY Quality Gates Quality gates Non-functional: - Execution time - Test Flakiness - Code quality (static code analysis) - Code Reviews - Environment stability - Performance testing - Security testing - Accessibility testing Functional: - Pass/fail ratio - Feature/component coverage - User flows health - Defect detection rate
- 8. What’s in reality? Test Report – build 34.5.2 Testing_Team_Lead QA Manager Test Report – build 34.5.2 Dear Manager, The most popular Quality Gates form:
- 9. What’s in reality? “Oh Jeez…. I supposed to get it on Wednesday…” MANAGER AT THIS TIME:
- 10. What’s in reality? TEAM LEAD AT THIS TIME:
- 11. What’s in reality? “Okay Guys, let’s run our Quality Gates check-list…”
- 12. What’s in reality? The most common Quality Gates form: - Emailable reports - Check-list calls PROS: - They do exist CONS: - People factor: - People are lazy - People make mistakes - Time factor - No automated decision RELEASE REJECT
- 13. CODE BUILD TEST REALESE DEPLOY
- 14. TEST 200 000test cases 63 days of execution time in a single thread 21 VMs x 6 threads x 18 hours 550 CI/CD jobs to run UNRELIABLE RESULTS AUTOMATION MAINTENANCE EFFORT AUTOMATION STATUS TRACKING ANALYSIS AND MANAGING EFFORTS Triaging failed test reports with a goal to identify product, automation or system issues Aggregation of all automated testing results and visibility to the actual status Ability to track the effort to stabilize and maintain the automated tests High risk of sequence failure and processing time makes TA results unreliable source of data
- 15. Regression
- 19. This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X
- 24. #1 Consolidation of results
- 25. age of failed #1 Consolidation of results
- 26. #2 Triage of failed results
- 27. #2 Triage of failed results Product Bugs Automation issues Environment issues
- 28. #2 Triage of failed results Product Bugs – actual value of test automation in finding product bugs Automation issues – technical dept of QA automation team (outdated tests, broken locator, incorrect test data, waiters, etc.) Environment issues – infrastructure limitations and inconsistency (configuration issues, limited budget, unstable connection, version mismatches, etc.)
- 29. #2 Triage of failed results Goal: - categorize each and every failed test case Expectations Reality
- 30. #2 Triage of failed results Improved visibility returns as investments into the QA infrastructure Goal: - categorize each and every failed test case Why? - Known issues - Multiply failures because of the one specific issue - Just boring Solution - ML algorithms to triage known issues Result of ML categorization: - Known issues categorized automatically - Unknown issues are grouped for bulk investigation - Team focuses on new failures - Released team effort is focused on stabilization and improvements.
- 31. This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X OR the percent of defect type for ALL test > 3% Total AB OR the percent of defect type for ALL test > 0% Critical PB
- 32. #3 Test Case attributes enrichment @Test @TestCaseId("TNG.1.0.1") @Attributes(attributes = { @Attribute(key = ”Priority", value = ”critical") , @Attribute(key = ”Role", value = ”Admin") , @Attribute(key = ”Flow", value = ”configuration") , @Attribute(key = ”Component", value = ”DCHP"), @Attribute(key = ”IP-Type", value = ”IPv6") }) public void testDHCPconfigurationForIPv6() { ... Assert.assertEquals(1, 1); } Goal: - Establish traceability between tests and requirements. How: - Adopt `Documentation as a Code` principals – tag test cases with attributes - Establish the best practice of test case implementation - DOD: test attributes in Acceptance Criteria
- 33. #3 Test Case attributes enrichment
- 35. #3 Test Case attributes enrichment
- 36. #3 Test Case attributes enrichment
- 37. #3 Test Case attributes enrichment This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X OR the percent of defect type for test with > 1% Total AB component:payment OR the percent of defect type for test with > 2% Product Bugs severity:trivial OR the percent of defect type for ALL test > 3% Total AB OR the percent of defect type for ALL test > 0% Critical PB OR new failures for exist layer:API
- 38. #3 Test Case attributes enrichment This Gate fails: IF Percent of failure for ALL test > 0% OR the amount of all test cases for ALL tests < X OR the percent of defect type for test with > 1% Total AB component:payment OR the percent of defect type for test with > 2% Product Bugs severity:trivial OR the percent of defect type for ALL test > 3% Total AB OR the percent of defect type for ALL test > 0% Critical PB OR new failures for exist layer:API Automation Product Environment
- 40. #1 Consolidation of results
- 42. webhook