James Christie CAST 2014 Standards – promoting quality or restricting competition?
0 likes•3,069 views
The document discusses the complexities and challenges associated with testing standards, particularly ISO/IEC 29119, arguing that while standards can provide a common understanding, they may also impose restrictions and limit competition. It critiques the inflexible nature of these standards and illustrates the potential pitfalls of regulatory capture and economic rent in the testing industry. Additionally, it highlights the necessity for genuine industry standards that serve both practitioners and the market effectively.
James Christie CAST 2014 Standards – promoting quality or restricting competition?
- 1. Standards – promoting quality or restricting competition? James Christie 1a
- 2. James Christie jack of all trades, master of some Test Manager & Consultant Developer & Business Analyst Information Security Manager Project Manager Computer Auditor investment accountant trainee chartered accountant 1b
- 3. Image courtesy Stuart Miles/FreeDigitalPhotos.net Standards are a good thing Shared understanding/definitions Connecting the world Repository of collective experience Protect suppliers from unscrupulous competition Protect customers from unscrupulous suppliers 2a Can be attached to or referenced by contracts Increase professional discipline
- 4. Image courtesy Keerati/FreeDigitalPhotos.net But testing standards ? “ISO/IEC29119 relates to bullshit” James Bach Search for Michael Bolton & testing standards. “If you need ISO29119, you probably couldn't make good use of it. If you could make good use of it, you probably don't need it.” Iain McCowatt My objection here is to their status as standards, rather than their content. 2b
- 5. What I learned from economics 1. There’s far too much unjustified certainty around from people who should know better. 2. Everything I learned about economics is either rubbish, or simplistic (but interesting). Probably. I'm not certain (see #1). 3. The assumptions you start with dictate the conclusions you arrive at. 4. In trying to understand what's happening start by following the money (cui bono). 5. Dig below the surface. Always. 3a
- 6. Images courtesy Stuart Miles/FreeDigitalPhotos.net What is economic rent? capital land labour 3b
- 7. Image courtesy Stuart Miles/FreeDigitalPhotos.net What is economic rent? “There are two ways to become wealthy; to create wealth or to take wealth from others.” Joseph Stiglitz, Columbia University, Nobel prize winner in Economics. “The payment to a factor of production in excess of what is required to keep that factor in its present use.” David Riccardo Unearned income? Or income that’s not consistent with free market returns? 3c
- 8. Images courtesy coldesign, Stuart Miles/FreeDigitalPhotos.net Theft Rare talent Monopoly profits Insider dealing Asymmetric knowledge Inappropriate regulation Patent ambush Collusion over standards 3d What is economic rent? Free money
- 9. Images courtesy Stuart Miles & Chris Sharp /FreeDigitalPhotos.net Rent Seeking (it’s the name of the game) 3e Influencing public policies for personal or factional gain A negative sum game
- 10. Images courtesy Stuart Miles & digitalart /FreeDigitalPhotos.net Rent Seeking (it’s the name of the game) The prisoner’s dilemma 3f Influencing public policies for personal or factional gain
- 11. Rent Seeking (it’s the name of the game) Many small losers A few big winners 3g Images courtesy Stuart Miles, cooldesign & Vlado /FreeDigitalPhotos.net Influencing public policies for personal or factional gain
- 12. Rent Seeking (it’s the name of the game) Influencing public policies for personal or factional gain Testers & customers Many big losers A few big winners Large testing consultancies? 3h Images courtesy Stuart Miles, cooldesign & Vlado /FreeDigitalPhotos.net
- 13. Image from Tom Toles / Washington Post Regulatory Capture Don’t break the law. Make the law. “As a rule, regulation is acquired by the industry and is designed and operated primarily for its benefits.” George Stigler, University of Chicago, Nobel prize winner in Economics “Regulation was often sought by industries for their own protection, rather than being imposed in some ‘public interest’.” Barry M Mitnick, University of Pittsburgh 4a
- 14. Favourable legislation Cognitive capture Revolving doors Licensing 4b Regulatory Capture Don’t break the law. Make the law.
- 15. Images courtesy Sura Nualpradid /FreeDigitalPhotos.net Regulatory Capture - Licensing 4c 1970s,10% of US workforce in licensed occupations 2008, 30% of US workforce in licensed occupations (Morris Kleiner)
- 16. “Imagine an industry where qualifications are based on accepted standards, required services are specified in contracts that reference these same standards, and best industry practices are based on the foundation of an agreed body of knowledge – this could easily be the testing industry of the near future.” Stuart Reid, convenor of ISO 29119 Working Group Implications – the vision of standards 5a
- 17. “Imagine an industry where qualifications are based on accepted standards, required services are specified in contracts that reference these same standards, and best industry practices are based on the foundation of an agreed body of knowledge – this could easily be the testing industry of the near future.” Stuart Reid, convenor of ISO 29119 Working Group Implications – the vision of standards 5a
- 18. Implications – the vision of standards Have you ever felt insignificant? 5b
- 19. Implications – the vision of standards 5c
- 20. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d1
- 21. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d2
- 22. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d3
- 23. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d4
- 24. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d5
- 25. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d6
- 26. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d7
- 27. Implications – the vision of standards Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5d8
- 28. Implications – the vision of standards 5e How government & big corporations might see “craftsmen” testers? Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers?
- 29. Implications – the vision of standards How the standards lobby wants to see “serious” testers? Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5f Image courtesy Ambro /FreeDigitalPhotos.net
- 30. Implications – the vision of standards What we might get; drudgery, but we’re proud to say we do ISO standard drudgery. Irrelevant “craftsmen” as opposed to…? Serious corporate testers? Responsible professionals? Certified testers? 5g
- 31. Implications – the Healthcare.gov fallout 6a “Test Maturity and the Obamacare Website Contractors to Government should have a high CMMI and TMMI Level proving that they not only have processes of high maturity but continuously operate in a mature environment. The CMMI and TMMI are required by many Department Of Defense and U.S. Government contracts, since it will give the Government the peace at heart that they are dealing with a mature Company operating with best practices.” Images courtesy Stuart Miles/FreeDigitalPhotos.net
- 32. Implications – the Healthcare.gov fallout 6b “Apply standards judicially … and selectively apply frameworks such as CMMi and ITIL to embrace their value-adding processes and functions. Service providers themselves should define expectations regarding the application and adherence to international standards like ISO 29119 for software testing … to avoid a situation where a solution or service has never been tested and operational readiness is questionable.” Submission to federal government by professional body? Sales blurb? or “Lessons Learned From Healthcare.gov's Troubled Launch”
- 33. Implications – the Healthcare.gov fallout 6b “Apply standards judicially … and selectively apply frameworks such as CMMi and ITIL to embrace their value-adding processes and functions. Service providers themselves should define expectations regarding the application and adherence to international standards like ISO 29119 for software testing … to avoid a situation where a solution or service has never been tested and operational readiness is questionable.” Sales blurb? or both Oct 15 2013 Dec 12 2013 Submission to US Environmental Protection Agency “Lessons Learned From Healthcare.gov's Troubled Launch”
- 34. Implications – the Healthcare.gov fallout 6c “What the Outsourcing Industry Can Learn from the HealthCare.gov Fiasco. Set and adhere to standards. Never put out a product to the market without adhering to international standards. There is ISO 29119 for software testing and ISO 20000 for service management.” “We base our approach to test strategy and planning on ISO 29119 … to which we are a contributor… (So) you are assured your test strategy will reflect the complexity of any given development project.”
- 35. Implications – the Healthcare.gov fallout 6d “Infuse Test Factory is delivered on pre- built centre of excellence model. The test factory is an industrialized test delivery service based on best of breed process, people, and technology. The service can be applied to functional testing and non-functional testing. Built on a proven methodology and a certified TMMi Level 3 testing service; the Test Factory is built to ISO 29119 & ISO25010 standards. The Infuse Test Factory provides an efficient and effective service with guaranteed outcome.” (for 2000 automated tests)
- 36. Implications – the Healthcare.gov fallout 6d “Infuse Test Factory is delivered on pre- built centre of excellence model. The test factory is an industrialized test delivery service based on best of breed process, people, and technology. The service can be applied to functional testing and non-functional testing. Built on a proven methodology and a certified TMMi Level 3 testing service; the Test Factory is built to ISO 29119 & ISO25010 standards. The Infuse Test Factory provides an efficient and effective service with guaranteed outcome.” (for 2000 automated tests)
- 37. 6e Marketing ISO 29119 – the treasure of the standards
- 38. 6e Marketing ISO 29119 – the treasure of the standards Mit der richtigen Schatzkarte in der Tasche ist der Weg oft ganz einfach – das gilt auch fürs Softwaretesten. Im Spätsommer 2013 sind die ersten drei Teile der neuen Norm ISO/IEC/IEEE 29119 erschienen. With the right treasure map in your pocket it’s easy to find your way – that also goes for testing. The first three parts of the new standard ISO 29119 are coming in late summer 2013.
- 39. 6e Marketing ISO 29119 – the treasure of the standards Mit der richtigen Schatzkarte in der Tasche ist der Weg oft ganz einfach – das gilt auch fürs Softwaretesten. Im Spätsommer 2013 sind die ersten drei Teile der neuen Norm ISO/IEC/IEEE 29119 erschienen. With the right treasure map in your pocket it’s easy to find your way – that also goes for testing. The first three parts of the new standard ISO 29119 are coming in late summer 2013.
- 40. Marketing ISO 29119 – an appeal to fear 6f “Imagine something goes noticeably wrong. How easy will you find it to explain that your testing doesn’t comply with international testing standards? So, can you afford not to use them?” Stuart Reid
- 41. Marketing ISO 29119 – asymmetric information 6g Can you tell a lemon from a Lamborghini? James Christie “Buyers are unclear on what is 'good test practice' “ Stuart Reid Lamborghini image courtesy Teerapun /FreeDigitalPhotos.net
- 42. COBIT Framework has no insistence on “best practice” Countless references to ISO standards for; - Risk management - Security - Release management - Configuration management - Service level management - Incident management - Problem management - Business continuity - etc No mention of testing standards No insistence on detailed scripts or test cases None at all! 7a What do the auditors say?
- 43. “Every IT environment is unique and represents a unique set of risks. The differences make it increasingly difficult to take a generic or checklist approach to auditing.” 7b What do the auditors say? The Snowflake Theory of IT Audit Institute of Internal Auditors Global Technology Audit Guide, Management of IT Audit, 1st edition, 2006 ISO standards are not mentioned except in an appendix “… for consideration”.
- 44. “Internal auditors should not expect organizations to fully implement PMBOK, PRINCE2, COBIT, or any other large set of best practices. Rather, they should expect to see that these practices have been customized and integrated into the organization’s project management methodology.” Institute of Internal Auditors Global Technology Audit Guide, Auditing IT Projects, 2009 (current) 7c What do the auditors say?
- 45. Wrapping up – we have to speak out 8a “Buyers are unclear on what is 'good test practice' “ The debate must not be framed as a false choice OR
- 46. Wrapping up 8b American Society of Mechanical Engineers v. Hydrolevel Corporation Response of standards bodies? Consensus is “general agreement, characterized by the absence of sustained opposition… by any important part of the concerned interests”. Need for consensus regarding both standards and their interpretation.
- 47. Wrapping up 8c Email: [email protected] Twitter: @james_christie www.clarotesting.wordpress.com www.clarotesting.com Image courtesy Stuart Miles/FreeDigitalPhotos.net