Comparing Agile QA Approaches to End-to-End Testing
- 1. COMPARING AGILE QA APPROACHES TO END-TO-ENDTESTING Daria Mehra, Director of QE at Quid Talk at Lean StartupWeek, 11/03/16
- 2. Why test at all? ■ Can’t we write bug-free software? – Industry Average: "about 15 - 50 errors per 1000 lines of delivered code.” (Steve McConnell, Code Complete) ■ Can’t we fix the bugs as the users report them? – Are you Facebook? ■ Cost of fixing the bug goes up with time from inception – Prevent it at no cost*, fix it pre-release for $, fix it post-release for $$$$. ■ “Hope is not a strategy.” (Google SRE)
- 3. Buggy math ■ Mercedes recently ran an ad saying that its new car has 100 million lines of code.That’s impressive—until you realize that on average, any piece of software has a few bugs per thousand lines of code, so the car probably has thousands of vulnerabilities. – Chris Dixon of Andreessen Horowitz, replying to Obama’s 6 tech challenges ■ 100 mln loc * 15 bugs/kloc => 1.5 mln bugs! ■ Ford F-150 pickup, 150 mln loc => 2.25 mln bugs ■ Assume 0.001% probability of bug in a given month ■ Get 1 bug per working day…
- 4. How to move fast ■ Remember “Move fast and break things”? ■ Slogan v2: “Move fast and build things”! ■ Because you don’t move faster with a broken leg. ■ Quality vsVelocity is a false dichotomy ■ Quality improves as waste is removed, leading to velocity – Kaizen: “change for the better” – What Agile was called in 1986
- 5. Bimodal IT: where do you land? http://techbeacon.com/gartners-bimodal-it-mistake-devops-can-deliver-velocity-quality Y u so slow? Don’t do that “Fast” won’t last Awesome. How?
- 6. Path to quality Velocity: High. Except for when it breaks. Quality: ????? Velocity: High. Quality: High. https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/
- 7. Who needs end-to-end tests? ■ http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/ ■ https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html ■ End-to-end testing considered harmful. If that’s all you do. (Steve Smith) ■ Unit tests are great! At Quid: 5000+ unit tests, 108 e2e user-workflow tests.
- 8. 2 unit tests, 0 integration tests If this video won’t play in the slide deck, watch it here: http://thisotplife.tumblr.com/post/137415092034/2-unit-tests-0-integration-tests
- 9. needs end-to-end tests! ■ What’s Quid? – Quid is a platform that searches, analyzes and visualizes the world’s collective intelligence to help answer strategic questions ■ Huge data set + upload your own – Data-dependent execution ■ Search -> NLP -> knowledge network -> viz ■ Rich UI with many workflows: – Lots of ways to slice and dice the data ■ Import and export and browsers, oh my! ■ Must test important user workflows daily in CI
- 10. Two roads diverged ■ Manual-only testing is not a feasible path: slow, expensive, unreliable. ■ Two ways I’ve done it: full automation with Selenium, and RainforestQA.
- 11. Selenium approach ■ Automated end-to-end testcases in Python / Ruby / etc, executed by driving browser actions via Selenium framework ■ Often with a layer of structured natural language on top (Gherkin or Cucumber) ■ Pros: – Fast execution -> quick feedback loop – Runs in CI on developer branches and master -> always releasable master ■ Cons: – Tied to the front-end implementation, your CSS is now your API – Finicky, tends to suffer from timing issues with page state – Hard to debug failures, is this a product regression or a test issue? “Element is not clickable at point (x, y). Other element would receive the click: …” – Misses regressions that human would notice – Takes engineering skill to add and maintain testcases
- 12. Release process w/Selenium UnitTests + FunctionalTests + End-to-End SeleniumTests
- 13. RainforestQA approach ■ Automated end-to-end testcases in unstructured natural language (English), executed by crowdsourced human testers via RainforestQA service ■ Pros: – Easy to add and maintain testcases, non-engineers can do it – Easy to debug test failures (screenshot + human comment) – No dependence on front-end implementation, only user-visible workflow – Human eye – Parallel scalable execution ■ Cons: – Slower test execution at human speed – Annual contract expense – Runs on staging and/or production, not on dev branches
- 14. Release process w/RainforestQA UnitTests + FunctionalTests End-to-End RainforestQATests
- 15. Why we picked RainforestQA ■ Not either/or: – Quid uses Selenium in production monitoring checks ■ Scaling out Rainforest testcases was SO MUCH EASIER ■ Solved regression testing for a complex product ■ Nightly testing is close enough to CI/CD for us – Next up: early access testing with feature flags ■ Allows QE team to stay small ■ Promotes collaboration on quality between PM, CS, QE and Dev ■ Frees up Quality Engineer time for “artisanal” exploratory testing ■ … And that has made all the difference.
- 16. Ask me anything ■ Daria Mehra, Director of QE at Quid ■ Look us up: www.quid.com ■ Email me: [email protected] ■ Read my post on quality process: http://tinyurl.com/dmehra-ikea ■ Read about how I spend my days: http://tinyurl.com/dmehra-day ■ And we ARE hiring!