Advancing Testing Using Axioms
Download as PPTX, PDF1 like634 views
- The speaker proposes 16 "test axioms" that are intended to provide a framework for testing approaches and represent principles that are context-insensitive and self-evidently true. - The axioms are grouped into three categories: stakeholders, design, and delivery. The speaker argues the axioms can help testers think critically about testing and identify flaws in arguments. - It is argued that process improvement models are not effective for improving testing because there is no consensus on best practices and processes must be tailored to context. True improvement requires understanding why current approaches are used given the context.
![[HCMC STC Jan 2015] Workshop Of Context-Driven Testing In Agile](https://cdn.slidesharecdn.com/ss_thumbnails/nhatdo-vuduongworkshopcontext-driventestinginagile-150208225650-conversion-gate02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Advancing Testing Using Axioms (20)
Ad
More from SQALab (20)
Recently uploaded (20)
Advancing Testing Using Axioms
- 1. AdvancingTesting Using Axioms @paul_gerrard Paul Gerrard [email protected] gerrardconsulting.com
- 2. Helping clients transform their testing through INNOVATION, COACHING and LEADERSHIP Our CLIENTS – Want to be agile rather than follow Agile dogma – Have a pragmatic approach and are focused on delivery – Want a solution that fits, not a badly fitting suit.
- 3. Agenda • It all started with a thought experiment • There are no agreed principles in testing • What does an Axiom look like? • The sixteenTest Axioms • Test process improvement is a waste of time • ImprovingTesting • Conducting aTest Assessment (if I have time) • Close
- 4. It all started with a ‘thought experiment’
- 5. Are there a set of axioms that provide a framework for all test approaches?
- 6. What is an Axiom?
- 8. 2 + 2 = ??? Something so self-evidently true it requires no proof
- 9. Example Axiom: “The shortest distance between two points is a straight line”
- 10. We believe Axioms are INFALLIBLE (but we are prepared to be proven wrong) We KNOW that heuristics are FALLIBLE (cf the Context-Driven School)
- 11. “Testing as the conscience of the project” Testers tend to take principled stances
- 12. We all have our favourite ‘testing sayings’ but… There is no agreed set of laws so our profession is based on weak foundations.
- 13. We need to start with a definition of ‘test’ (American Heritage Dictionary) Test: (noun) • A procedure for critical evaluation; • A means of determining the presence, quality, or truth of something; • A trial.
- 14. OK – that’s the intro… Let’s play: “Hunt the Axiom”
- 15. Diverse principles promoted by diverse set of ‘experts’
- 16. Amazon offers 3557 choices
- 17. Differing backgrounds and experience of ‘experts’
- 19. When some of them were written
- 21. Early focus on BUGS
- 22. Testing is an intellectually difficult activity
- 25. Testing builds confidence …and so on…
- 26. Test thinking has evolved from simple beginnings
- 27. Testing as a lifecycle activity Model-based testing Test-obsessed Testing as exploration Digital Assurance… Agile testing Test-first, test-driven Risk-based testing Testing as learning
- 28. What could the Test Axioms look like?
- 29. From any perspective, they are self-evidently true
- 30. They apply to any test approach
- 32. They are not practices Practices are always context-sensitive
- 33. A non-Axiomatic testing approach is ‘unsound'
- 36. Different approaches reflect a different emphasis on the same axioms
- 37. Good test approaches align with the Axioms or… The Axioms areWRONG
- 38. Test Axioms • Formulated as a context-neutral set of rules for testing systems • They also represent some critical thinking processes required to test any system • There are clear opportunities to advance the practice of testing using them • Testers Pocketbook: testers-pocketbook.com • Test AxiomsWebsite testaxioms.com
- 40. Axiom Name Each Axiom has a name and a one-line summary that states the principle.
- 42. Test Basis Axiom Testers need sources of knowledge to select things to test
- 43. Test Oracle Axiom Testers need sources of knowledge to evaluate actual outcomes or behaviours
- 44. Fallibility Axiom Our sources of knowledge are fallible and incomplete
- 45. Scope Management Axiom If you don’t manage scope, you may never meet stakeholder expectations
- 46. Design Axiom Test design is based on models
- 47. Coverage Axiom Testing requires a coverage model or models
- 48. Delivery Axiom The usefulness of the intelligence produced by test determines the value of testing
- 49. Environment Axiom Test execution requires a known, controlled environment
- 50. Event Axiom Testing never goes as planned; evidence arrives in discrete quanta
- 51. Prioritisation Axiom Testing needs a mechanism for ordering tests by value
- 52. Execution Sequencing Axiom Run our most valuable tests first – we may not have time to run them later
- 53. Repeat-Test Axiom Some repeated tests are inevitable
- 54. Good-Enough Axiom The scope of testing and acceptance are always compromises
- 55. Never-Finished Axiom Testing never finishes; it stops
- 56. Value Axiom The value of testing is measured by the confidence of stakeholder decision making
- 58. How to group the Axioms? • Would it be helpful to group the Axioms in some way? – To give them a structure – To relate them to the ‘real world’ • How do the Axioms relate to the test process?
- 60. The three axiom groups Stakeholder Value Scope Fallibility Good-Enough Delivery Repeat-Test Sequence Environment Event Never-Finished Design Basis Coverage Prioritisation Oracle
- 61. The three groups - summarised • Stakeholder – These are our customers – what do they want from testing? – What intelligence?When? How? How often? • Design – What do we plan to deliver? – How do we plan to deliver it? • Delivery – How do we manage the process of delivery? – How do we communicate intelligence?
- 62. How can we use Test Axioms? • Test Axioms are not beginners guides • They can help you to think critically about testing • They expose flaws in other people’s thinking and their arguments about testing • They generate some useful by-products • They help you to separate context from values • Interesting research areas! • First Equation of Testing,Testing Uncertainty Principle, QuantumTheory, Relativity, Exclusion Principle... • You can tell I like physics.
- 63. Test Process Improvement is a Waste of Time!
- 64. How to improve… • I want to improve my (insert any activity here) • _______ people improvement • _______ organisation improvement • _______ process improvement Changing people (like me) and organisation (like my company) is so hard – let’s not even think about it.
- 65. • There are no “practice” Olympics to determine the best • There is no consensus about which practices are best, unless consensus means “people I respect also say they like it” • There are practices that are more likely to be considered good and useful than others, within a certain community and assuming a certain context • Good practice is not a matter of popularity. It’s a matter of skill and context. The delusion of ‘best practice’ Derived from “No Best Practices”, James Bach, www.satisfice.com
- 66. The delusion of process models (e.g. Capability Maturity Model) • Google search – “CMM” – 22,900,000 – “CMM Training” – 15,200 – “CMM improves quality” – 66 • So – lots of consultants making money out of training (and assessment)? • Does CMM improve quality?
- 67. Coordinate measuring machine This talk Another of my talks Another of my talks Reducing defects by 50% improves quality 2x???
- 68. The delusion of process models (e.g. CMM) • One of my clients … – CMM level 3 and proud of it (chaotic, hero culture) – Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping) – 40+ recommendations, only 7 adopted – they couldn’t change – How on earth did they get through the CMM 3 audit?
- 69. “I believe that a scientist looking at nonscientific problems is just as dumb as the next guy” “It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong” Richard P. Feynman Physics quotes…
- 70. “I believe that a process consultant looking at non- process problems is just as dumb as the next guy” “It doesn't matter how beautiful your process model is, it doesn't matter how smart you are. If it doesn't agree with reality, it's wrong” Me! Process quotes Using process change to fix cultural or organisational problems is not going to work Improving test in isolation is not going to work either
- 72. First Equation of Testing Axioms + Context + Values + Thinking =Approach
- 73. Why you are where you are Context + Values + Thinking =Approach <- your values <- your context <- your thinking <- your approach
- 74. Making change happen <- recognise <- hard to change <- could change? <- just do some <- your approach Axioms + Context + Values + Thinking =Approach
- 75. Using the axioms and questions • Axioms represent the critical things to think about • Associated questions act as checklists to: – Assess your current approach – Identify gaps, inconsistencies in current approach – QA your new approach in the future • Axioms represent the WHAT • Your approach specifies HOW • Use the same questions for STRATEGY
- 76. Why is the equation useful? • Separation of Axioms, context, values and thinking • Tools, methodologies, certification, maturity models promote approaches without reference to your context or values • No thinking is required! • Without a unifying test theory you have no objective way of making lasting change.
- 77. One context, multiple approaches • Given context, practitioners can promote different approaches based on their values • Values are preferences or beliefs – Pre-planned v exploratory – Predefined v custom process – Requirements-driven v goal-based – Standard documentation v face-to-face comms. • Some contexts preclude certain practices • “No best practices”
- 78. Axioms allow (ensure) different approaches and expose positions • Separating axioms, context and values clarifies positions, for example: – ‘Structured’ test advocates have little (useful) to say about Agile contexts – Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance • The disputes between these positions is more about values than practices in context.
- 80. Conducting a Test Assessment • We might research test documentation and records • But most information will be gathered through interviews or meetings of key staff • Here are some guidelines • Each Axiom has some suggestions for good questions.
- 81. Planning information gathering • Need to engage people at all levels of the organisation • Start with the most senior and work downwards • Senior managers: more focused – 30-45m • Practitioners: allow 45-60 minutes • Schedule information gathering to complete in less than two weeks.
- 82. Planning interviews • Three interviews per day are comfortable – More than three per day can make it hard to keep up with note-taking and assimilation. • If helpful, write a brief note that introduces you, your objective and agenda • Send to all prospective interviewees so they can prepare • Prepare a list of the questions you would like answers for – use as a checklist.
- 83. Conducting sessions • What problems (relevant to the scope of the assessment) do people have? • What might be potential solutions to the problem? • Do they have any questions to ask us? • Advise them of next steps and how you intend to feed back our assessment, and timetable.
- 84. What questions should we ask? • Axiom questions make good interview questions – Most are open questions that require more than a yes/no – Some questions are more appropriate for managers than practitioners – and vice versa • Interviewees may be nervous – They may feel that they are under investigation and their ability is in question – Assure them that there is no hidden agenda • Always treat information gathered in interviews with sensitivity and respect the privacy of the individuals concerned.
- 85. Testing needs stakeholders (p64) Summary: Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide Consequence if ignored or violated: There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience. Questions: – Who are they? – Whose interests do they represent? – What evidence do they want? – What do they need it for? – When do they want it? – In what format? – How often?
- 86. Test design is based on models (p68) Summary: Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make Consequence if ignored or violated: Tests design will be meaningless and not credible to stakeholders. Questions – Are design models available to use as test models? Are they mandatory? – What test models could be used to derive tests from theTest Basis? – Which test models will be used? – Are test models to be documented or are they purely mental models? – What are the benefits of using these models? – What simplifying assumptions do these models make? – How will these models contribute to the delivery of evidence useful to the acceptance decision makers? – How will these models combine to provide sufficient evidence without excessive duplication? – How will the number of tests derived from models be bounded?
- 87. Close • The Axioms exist to help you understand what’s going on • Process improvement models are not going to help you improve • If you want to improve: – UnderstandWHY you behave the way you do – Given a current culture, the environment, the context is what drives most behaviour – It’s easier to change the context than culture – Use your own thinking, not someone else’s.
- 88. AdvancingTesting Using Axioms @paul_gerrard Paul Gerrard [email protected] gerrardconsulting.com