Continuous Testing - The New Normal
- 1. T1 Session 10/27/2016 10:15:00 AM Continuous Delivery with Cloud-Based Testing Methodologies Presented by: Tom Wissink Intervise, Inc. Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 - [email protected] - http://www.starcanada.techwell.com/
- 2. Tom Wissink Intervise, Inc. Tom Wissink worked for Lockheed Martin (LM) managing, developing and testing software intensive systems for 35 years. He was a Lockheed Martin Senior Fellow when he left Lockheed Martin to became a consultant for Intervise, Inc. in February 2016. From January 2010 until becoming a consultant he was the LM Corporate Director of Integration, Test and Evaluation. Throughout his Lockheed Martin career he worked on programs like the Space Shuttle, Hubble Telescope, GPS and several Satellite Command and Control centers. Tom is a member of the National Defense Industrial Association (NDIA) and was a past chairperson for the Industrial Committee on Test & Evaluation (ICOTE) as well as a past co-chair of the Development Test & Evaluation (DT&E) committee. He has presented at the Aerospace Testing Seminar as well as at the STAREAST and STARWEST conferences.
- 3. 9/22/2016 1 SESSION ABSTRACT Some notions of continuous testing (CT) have been applied in software development methodologies for a while but it was never called by that term. Another term sometimes used for CT is parallely p testing. While some have mastered CT, most of us struggle with how to transform our current testing approaches to CT approaches and align them with evolving development methodologies. Join Tom Wissink as he discusses current examples of CT implementations across different software development methodologies (agile, waterfall, incremental) and describes where parallel or CT type testing yields the best benefits. Arguably the most challenging methodology that demands CT testing is DevOps. DevOps requires all phases of testing to be done quickly and in parallel with the development process and some contend that testing continues into actual operations. Leave this i ith b tt d t di f CT d h thi h b b t l d i 2 session with a better understanding of CT, and how this approach can be best leveraged in your development environment.
- 4. 9/22/2016 2 AGENDA • INTRODUCTION • SURVEY OF ROLES AND METHODS• SURVEY OF ROLES AND METHODS • TEST TYPES DEFINED • DEVELOPMENT METHODOLOGIES • CONTINUOUS TESTING (CT) APPROACHES • “THE TOOLCHAIN” • SUCCESSES WITH CT • WRAP-UP 3 SURVEY •WHAT IS YOUR ROLE? • TEST/QA ENGINEER •WHAT DEVELOPMENT METHOD(S) ARE YOU USING? •WHAT TYPE OF SOFTWARE? • WEB APPS• TEST/QA ENGINEER • SW DEVELOPER • PROJECT LEADER • BUSINESS ANALYST/USER • MANAGER • OTHER METHOD(S) ARE YOU USING? • WATERFALL • AGILE • DEVOPS • HYBRID • OTHER • WEB APPS • CLIENT/SERVER APPS • MOBILE APPS • LARGE SOFTWARE SYSTEMS • OTHER? OTHER OTHER 4
- 5. 9/22/2016 3 TEST TYPES DEFINED Goal: High test coverage for each test type or identify risk for reduced coverage Unit Testing (White-Box) Statement, Condition, Decision & Path Metric is Logic Coverage – tools available Integration Testing (Interface Testing: this is generally very weak) Identify threads through system Threads part of incremental/build plansThreads part of incremental/build plans Early look at Technical Performance Measures – TPM’s (stability, performance, RMA, etc) Metric is Interface coverage – external & internal interface 5 TEST TYPES DEFINED (CONT’D) Functional Testing (Black-Box) Boundary Value Analysis, Output Forcing, Equivalence Class Partitioning, Cause & Effect Graphing, Combinatorial, etc. S SMetric is Function coverage – SRS requirements, etc System Testing (Official Sell-Off) Types - Scenario-Based, Risk-Based, Exploratory, Model-Based, etc Includes several categories of requirements – HMI, Non-Functional (Performance, RMA, Configuration, Installation), etc. Metric is Requirements coverage – System Spec, Ops Concept/Scenarios, etc A t T ti (C t T ti ) 6 Acceptance Testing (Customer Testing) Operational Test & Evaluation (OT&E), Beta, etc Metric typically owned by Customer but good to know
- 6. 9/22/2016 4 DEVELOPMENT METHODOLOGIES 7 Effective and Efficient Test Lifecycle in any Software Development Methodology (WF, Incremental, etc.) 8
- 7. 9/22/2016 5 Effective and Efficient Test Lifecycle in any Software Development Methodology (Agile, DevOps, etc.) 9 Effective and Efficient Test Lifecycle in any Software Development Methodology (Agile, DevOps, etc.) See SAFe at www.scaledagileframework.com ART – Agile Release Trains ATDD – Acceptance Test Driven Development WSJF – Weighted Shortest Job First 10
- 8. 9/22/2016 6 CONTINUOUS TESTING APPROACHES 11 Defects – Are teams spending too much time logging, triaging or analyzing defects? What about time spent on defects that aren’t “real” “Continuous Testing: Shift Left and Find the Right Balance” spent on defects that aren t real defects—where there is a misunderstanding between the test and the code? Or what if they could prevent entire schools of defects from ever being created in the first place? 12
- 9. 9/22/2016 7 Test Management – Are teams spending time manually crafting status reports and rolling up test execution results? Or do they have a “Continuous Testing: Shift Left and Find the Right Balance” execution results? Or do they have a tool that provides that information in real time and allows stakeholders to drill down as needed? How do teams know if they are on schedule with their test effort, behind schedule or even ahead of schedule? 13 Test Automation – How efficient are teams at re-executing existing tests? Do they run most or even all of them “Continuous Testing: Shift Left and Find the Right Balance” manually? If they’ve automated tests, are they focused only on functional tests at the user interface layer, or are they running functional API-layer tests, performance tests and even security tests? Do they have a robust and maintainable test automation framework? 14
- 10. 9/22/2016 8 Analytics – How do teams know which tests they should run, when and even why they are running those tests at those times? How good is their test “Continuous Testing: Shift Left and Find the Right Balance” times? How good is their test effectiveness–meaning, are they running the fewest number of tests that find the largest number of problems? Impact analysis is critical in selecting the right sets of tests to execute whenever they get a new build. 15 Test Environments – Are teams constantly waiting on test environments to be provisioned and configured properly? Do they run tests and discover ft th f t th t th t t i t “Continuous Testing: Shift Left and Find the Right Balance” after the fact that the test environment wasn’t “right,” so they have to fix the environment and then re-run all the tests again? Do they hear from developers, “It works on my machine!” but it doesn’t work in the test environment? 16
- 11. 9/22/2016 9 Service Virtualization – Are teams waiting for dependent systems to become available before they can “really” test? Are they using a “big bang” approach to conduct end to end “Continuous Testing: Shift Left and Find the Right Balance” a big bang approach to conduct end-to-end system testing, where they throw all the components together and hope they work and interact properly? Can teams test exception and error scenarios before going to production? Are they testing the easiest parts first just because they are available, and delaying the high-risk areas for the end of the testing effort?g 17 “Continuous Testing: Shift Left and Find the Right Balance” Test Data – Do teams have the needed sets of production-like test data to ensure they are covering the right test scenarios? Are there exception and error scenarios that we can’t execute because they don’t have the right sets of test data? 18
- 12. 9/22/2016 10 “Continuous Testing: Shift Left and Find the Rightg Balance” by Marianne Hollier on DEVOPS.com 19 “Understanding DevOps – Part 4: Continuous Testing and Continuous Monitoring” by Sanjeev Sharma on sdarchitect.wordpress.com 20
- 13. 9/22/2016 11 A VENDORS VIEW OF CD WITH CT Continuous delivery (CD): a SW engineering approach in which teams producein which teams produce software in short cycles… oContinuous Delivery is not Continuous Deployment oRelies on 3 foundations: • Configuration management, • Continuous integration, • Continuous Testing 21 A VENDORS VIEW OF A TOOLCHAIN 22
- 14. 9/22/2016 12 SOME OPEN-SOURCE CATEGORIES/TOOLS IN A TOOLCHAIN 23 MY SET OF TOOL CATEGORIES IN A TOOLCHAIN Project M Requirements Gathering & Versioning Configuration MManagement Gathering & Management Versioning Management SW Development & Testing* Tools Continuous IntegrationMonitoring 24 Testing Tools * Test Automation tools work and provide value
- 15. 9/22/2016 13 SUCCESS WITH CT IN WATERFALL AND AGILE 25 10 COMPANIES KILLING IT AT DEVOPS FROM TECHBEACON.COM •1. AMAZON •2 NETFLIX •6. FACEBOOK •7 ETSY•2. NETFLIX •3. TARGET •4. WALMART •5. NORDSTROM •7. ETSY •8. ADOBE •9. SONY PICTURES ENTERTAINMENT •10. FIDELITY WORLDWIDE INVESTMENT 26
- 16. 9/22/2016 14 WRAP-UP • CT can be used in any development methodology and provide value A b t t l h i i i l t f l CT• A robust toolchain is crucial to successful CT and CD • Good testing practices are still important and maybe more so in CT as in any other development methodology 27 • DevOps demands CT to achieve desired outcomes of faster delivery with high quality