Keeping Your Continuous Test Automation Continuously Valuable
Download as PPTX, PDF0 likes68 views
When executing test automation at scale and continuously the value tends to decline over time, the team should follow recommended practices to keep their tests with high value. Getting the value from your test automation is fundamental for fast feedback, risk reduction and return on investment from your testing activities. Getting the value from your test automation is fundamental for fast feedback, risk reduction and return on investment from your testing activities. Once developing the test scenarios, teams cannot stop monitoring and ensuring that their tests continuously bring value, are not flaky, and can support the latest functionalities in your web and mobile apps. Teams often “forget” about their tests once they have been developed and integrated into the CI pipeline regardless of the value they bring. Key Takeaways: -How to make smart decisions regarding which test scenarios to automate? -What are the criteria for a test to get included in the CI and continuous testing pipeline? -How to continuously maintain the tests and optimize your suite so it continues to bring value? -See a live demo of smart reporting and analytics that can serve as a monitoring and test maintenance tool.
Ad
More Related Content
What's hot (20)
Similar to Keeping Your Continuous Test Automation Continuously Valuable (20)
Ad
More from Perfecto by Perforce (17)
Ad
Recently uploaded (20)
Keeping Your Continuous Test Automation Continuously Valuable
- 1. Perfecto by Perforce © 2020 Perforce Software, Inc. Keeping Your Continuous Test Automation Continuously Valuable
- 2. 2 | Advanced Codeless Testing for Web Apps perfecto.io Today’s Speaker: Eran Kinsbruner • Chief Evangelist and author at Perfecto • Blogger and speaker • 19+ years in development & testing • Author of “The Digital Quality Handbook” and “Continuous Testing for DevOps Professionals” • Twitter: @ek121268 • Blog: continuoustesting.blog • Website: perfecto.io
- 3. perfecto.io3 | Perfecto by Perforce © 2020 Perforce Software, Inc. Digital Innovation Highlights
- 5. perfecto.io World Quality Report 2019-2020 Maturing DevOps and Agile actually requires more testing!
- 6. perfectomobile.com What Is Continuous Testing? Continuous testing is the process of executing automated high value tests as a part of the software delivery pipeline in order to obtain feedback on business risks associated with a software release upon every code change.
- 7. perfecto.io7 | Perfecto by Perforce © 2020 Perforce Software, Inc. Optimize Based on Data Recurring Patterns for Inefficiencies Licenses In-use Disconnected Orchestration issues Network Availability Data Backend issues Objects Coding Time Other Scripts & FW issues • 10% of devices causing 80% of lab issues Networking Stability Config Lock Lab issues Backend 30% Lab 15%Orchestration 15% Scripts & FW 40% What’s wrong with my scripts? What’s wrong with my executions? What’s wrong with my lab? What’s wrong with my backend?
- 8. perfecto.io8 | Perfecto by Perforce © 2020 Perforce Software, Inc. Noise Reduction – Continuous Testing Enabler “About 10 – 20 defects per 1000 lines of code during in- house testing, and 0.5 defect per KLOC in production” (Sogeti Labs) ~1 real defect per developer per day Regression suite of 10,000 tests == ~20% failures Out of 2000 failures, only 20 are actual bugs. 1980 are false negatives! Reporting noise is a risky & bad practice. •Reporting crashes, errors in functionality, performance issues – Dev Friendly •Lab issue, object not found, popups, alerts, orchestration, etc. should never be taken to Dev
- 9. perfecto.io9 | Perfecto by Perforce © 2020 Perforce Software, Inc. • Think continuous value. • Think maintenance and complexity to create, execute, monitor, and maintain (ROI). • Test reliability over time. Test Automation Certification If { (Test Scenario) == Candidate to add Value add { and x– It detect defects} { and – It passes across multiple platforms with consistent result} { and – Its execution time is < 5 minutes } Then { Include in Test Suite & & CI } Else { } Run such certification once in every 2-3 software iterations.
- 10. perfecto.io10 | Perfecto by Perforce © 2020 Perforce Software, Inc. Measuring Test Automation ROI Value Value of Test Automation •Cycle testing time per iteration •Test automation types (functional, non-functional, unit) •Error detection efficiencies •Product coverage (key business scenarios) Cross-Team Collaboration •Three personas sharing load (resource, smart utilization of platforms) •Manual testing % •Backlog of test automation between iterations (in-sprint automation) •Test debugging and analysis time per cycle Test Execution •Automation within CI success rate •Execution across platforms success rate •Test redundancy/duplication/waste •Coverage across environments (test, production)
- 11. perfecto.io High-Level Comparison Between Tools and Practices Test Framework Supported Dev Languages Supported Platforms Supported Test Frameworks Setup and Execution Integrations Breadth of Testing Options Maturity, Documentation, Support Required Skillset Cloud and Execution at Scale Selenium/Appium WebDriver Java, C#, Java Script, Python, Ruby, Objective- C Chrome, Safari, Firefox, Edge, IE/iOS/Andr oid Mocha JS, Jest, other super set on top of Selenium (Protractor, WebDriverIO, etc.) Download relevant driver, set up a grid, network and location impacts execution speed Plenty of integrations (CI, CD, reporting, visual testing, cloud vendors) End-to- end, security, unit, Robust community, multiple bindings, best practices Coding skills required (SDET Oriented) Perfecto fully supports Selenium and its WebDriver configurations. Local execution requires setting up a Selenium grid XCUITest/Espresso/ Headless/Cypress Objective C/Java/Jav a Script Chrome, Electron NA Embedded into IDEs, headless bundles a browser in the FW CI/CD UI/Unit Good documentation and code samples Dev Oriented Built-in Chrome/Firefox browsers in headless, Perfecto Cloud supports scaling Espresso/XCUITest Codeless Irrelevant, based on record and playback All Proprietary UI with underlying Selenium WebDriver APIs Mostly SaaS/browser plugin installation Limited Functional /UI Growing, limited No coding skills required (Business Tester Oriented) Perfecto supports codeless in the cloud BDD Java, Ruby, JS, Kotlin All Junit, Selenium, Appium Open source, Maven/Gradle /TestNG Plenty + APIs (e.g. Rest Assured) Functional Robust community, docs, adoption Step-definition development in code is required/scenarios are no-code (Mix of Business Tester and SDETs) Perfecto Quantum is a web/mobile BDD framework
- 12. perfecto.io12 | Perfecto by Perforce © 2020 Perforce Software, Inc. • Fast feedback • > Coverage of platforms • < Execution of time constraints The Need for Scale
- 13. perfecto.io13 | Perfecto by Perforce © 2020 Perforce Software, Inc. Scaling Best Practices What to Test (Scenarios + Platforms)? Why These (Value)? When in the Cycle, How Often (CI/Outside)?
- 14. perfecto.io14 | Perfecto by Perforce © 2020 Perforce Software, Inc. • Stick to time and coverage objectives • Burst to lower overall execution time • Include more platform permutations in the cycle • Continuously monitor your tests • Improve test suites based on history • Retire obsolete test scenarios • Data-driven decision making — release/tests/platforms • Scale wisely inside and outside your CI • In CI: • Subset of functional and non-functional (security, performance, accessibility) • Build acceptance, unit, API • Outside CI: • Niche tests | Flaky/inconsistent test scenarios | Unchanged areas based testing Concepts for Scaling Test Automation
- 15. perfecto.io15 | Perfecto by Perforce © 2020 Perforce Software, Inc. • History of failures • Test vs. device issue • Specific areas failures • Script development practices • Popup handling • Element not found • Are the tests brining any value? • Platform health? Lab? Test Data Analysis – Act Upon Early Warnings
- 16. perfecto.io16 | Perfecto by Perforce © 2020 Perforce Software, Inc. • Testing with Selenium (Java) the Boston Globe RWD site at scale • Smart analytics with CI dashboard and cross-platform comparison • Technologies used: • IntelliJ Idea • Selenium • TestNG/Maven • Perfecto Live Demo
- 17. perfecto.io17 | Perfecto by Perforce © 2020 Perforce Software, Inc. Create Your Own RCA?
- 18. perfecto.io18 | Perfecto by Perforce © 2020 Perforce Software, Inc. The 4 Key Pillars of Continuous Testing in DevOps This is where money is spent (Time, Tools, Resources) This is where value is being realized, quality is improved Unified solution serves DevOps with greater effectiveness.
- 19. Thank You! Visit perfecto.io or follow us @perfectomobile
Editor's Notes
- #2: How to Scale Mobile and Web Test Automation in DevOps
- #7: focuses on business risk and providing insight on whether the software can be released. Beyond test automation, Continuous Testing also involves practices such as aligning testing with your business risk, applying service virtualization and stateful test data management to stabilize testing for Continuous Integration, and performing exploratory testing to expose “big block” issues early in each iteration
- #11: Value of test automation Cycle testing time per iteration Test automation types (Functional, Non-Functional, Unit) Error detection efficiencies Product coverage (Key business Scenario’s) Collaboration 3 personas sharing load (resource, platforms smart utilization) Manual testing % Backlog of test automation between iterations (In-sprint automation) Test debugging and Analysis time per cycle Execution Automation within CI success rate Execution across platforms success rate Test redundancy/Duplication/Waste Coverage across environments (Test, Production)