Improving developer tester collaboration with microsoft visual studio 2010

Editor's Notes

• #2: Walk-in slide. Advance to the next slide to begin the presentation.
• #3: HIDDEN SLIDE FOR SPEAKER PREP ONLYAudience Definition:Software Developers, Development LeadsWhat are they like? Software developers are problem solvers, tinkerers, artists and engineers all rolled up in one. They are continuously building something for the first time. No matter what they are building there is some element of it—or all of it—that is new, that they have never done before. Developers are analytical and believe, to a certain extent, that there is an art form to software development.Why are they here?Developers attend this session because they want to learn how to make their work life better. They have to work with testers every day, and anything that improves how they receive a bug, and how they can fix it and never see it again is an improvement.What keeps them up at night? Bugs. Software developers are a proud people. They don’t like to find out that they introduced a bug into the code. The goal is bug-free code. Obviously this is impossible, because some bugs are the result of bad requirements or changes in scope. If a bug is found it can be massively disruptive to what they believe is a more creative process—writing new code. No software developer wants to spend time fixing bugs; time spent on bugs is like paying taxes.How can you solve their problem?Since the majority of time spent fixing bugs is in (a) reproducing them and (b) finding the root cause, and not in actually coding the fix, you need to give them tools that enable them to do these two things—reproduce the bug and find the root cause faster. What do you want them to do? Install Visual Studio 2010 Ultimate and start using IntelliTrace in their local debugging sessions. Work with their QA or test team to get them to begin using Visual Studio Test Professional 2010 so that bugs that are filed are more actionable.How can you best reach them? Demonstration. Show them what IntelliTrace does and how it can help them. Show them the entire workflow—from discovering a defect and filing a bug, to quickly locating the cause, to automating the functional test to identify if the bug ever returns.How might they resist?If the new tools and capabilities appear to have a lot of setup overhead they may quickly reject it as something they don’t have time for. Additionally, the need to get another team to use a new tool—the QA team—may appear to be a non-starter.
• #4: HIDDEN SLIDE FOR SPEAKER PREP ONLYSituationDevelopers and testers frequently work in separate but connected teams. Each role is managed separately, perhaps only connecting at the Director of Development or Vice President of IT level. The development team is responsible for implementing a set of requirements, while the QA team is responsible for validating the requirements were met; however there is often little direct interaction between the developers and the testers and often a skill and pay gap that divides the two roles. In 90% of IT organizations it’s obvious the developers are more highly valued and usually more highly skilled.ProblemSince the developers and testers are managed separately, they are frequently given different direction. There is nothing forcing the two roles to interact, and likely the division of reporting structure creates a greater boundary between the two roles. The level of collaboration is very low, and too often the level of blame is too high.ImplicationThere is erratic code quality, dependent on the individual developer with a high escaped bug rate, frequent regressions and performance problems. As a result there is a high-level of mistrust between testers & developers. When bugs are found, they are often difficult to reproduce due to poor bug documentation. In some environments, coding mistakes are caught by the developers before handing off to testing, but consistency still depends on individual discipline. Since developers don’t understand the testing role there’s often a lack of trust. In addition, the high complexity of the typical enterprise environment means developers assume the testers incorrectly configured the software or made some other mistake that explains away the bug.SolutionVisual Studio 2010 introduces a new set of tools and capabilities that enable testers to easily create rich, actionable bugs when a defect is found. The level of data collected and attached to the bug enables developers to quickly take action and locate the root cause of the bug. IntelliTrace™ enables the developer to step through the code execution from the test environment as if it was currently running—similar to a digital video recorder (DVR). The data collectors can also capture environment and configuration settings, and developers can see the actual execution of the program using Lab Management 2010. Once the developer resolves the bug they are able to create a Coded UI Test which can be added to an automated regression test suite, ensuring that if the bug is ever reintroduced, it is quickly discovered.
• #5: What is the most common bug resolution used right now by your development team? Is it ‘Cannot Reproduce,’ or ‘Not a Bug,’ or ‘Won’t Fix?’ If you stack ranked all of the bug resolution by how commonly they were used, where would ‘Fixed’ appear? Imagine if you could move ‘Fixed’ to the top of the list? How would your software development process change if resolutions like “Not a Bug” and “Won’t Fix” became rarities? What if “Cannot Reproduce” became obsolete?Welcome. My name is [SPEAKERNAME] and I am a [SPEAKER TITLE].
• #6: The first step in resolving a bug is reproducing it reliably. Once the bug is reproduced, a developer can use tools like the Visual Studio debugger to step through the code execution, watching variable states and execution flow to identify the cause of the bug. Unfortunately, all too often bugs are difficult to reproduce. If you cannot reproduce a bug, how can you be expected to fix the root cause? The reasons for not being able to reproduce the bug could be varied. Some bugs are triggered by inputs that are difficult to recreate, or poorly documented in the bug report; often a tester will perform actions or steps that weren’t explicit in the test case. Without knowing what the tester did it becomes very difficult to reproduce the bug. In some cases, bugs are the result of the speed at which the tester performed the actions—a reproduction that is often very hard to identify until you see the tester in action. In some cases the bug is the result of environmental differences between the development environment and the test environment—which could include different operating systems, service packs, different versions of required runtimes, and even the version of the software being tested.
• #7: All of these things lead to a lot of wasted effort on your team, and create a divide—a chasm—between the developers and the testers. The reality is that you share a common goal. Both developers and testers want to ship high-quality software that meets the expectations of the customers and stakeholders. How can you reduce the waste of bug Ping-Pong, and increase the level of collaboration between developers and testers. Sure you can engage in some crazy team building outing where you have the developers have to fall backwards and the testers have to catch them…but is that really going to help?
• #8: Visual Studio 2010 is going to help. Today you are going to see three (3) ways that developers and testers can use the tools and capabilities of Visual Studio 2010 to:Create rich, actionable bugs that enable any developer to learn enough about the bug to take immediate action.Replay the code execution from a test run to reproduce a bug and quickly find the root cause, even if the development environment is different than the test environment.Create an automated functional test that will quickly discover the bug if it reemerges.
• #9: In order to move ‘Fixed’ to the top of the resolution list you have to start by looking at how software is tested and how defects are reported. In most environments today the test or QA team uses a set of tools that enable them to do their job, but which don’t integrate into the tools used by the development team. This is the first fracture in the system and a contributor to costly bug ping-pong. In an ideal world the development team and the test team would each use tools that integrated with one another and enables easily passing information back and forth.
• #10: Microsoft Test Manager 2010 is for testers what Visual Studio is for developers. That is to say, where Visual Studio is an IDE – an integrated development environment, Test Manager is an ITE – an integrated test environment. This is the interface that a tester will use to create test cases, organize test plans, track test results, and file bugs when defects are found. Test Manager is integrated with Team Foundation Server, and is designed to improve the productivity of testers. While I am not going to do a deep-dive of all that Test Manager can do, it is important to understand how it integrates with the Visual Studio Agents to make the most of test case execution, and ensure that when a tester files a bug, it is actionable with very little work on the tester’s part.
• #11: Visual Studio Agents 2010 are set of technology—included with Visual Studio 2010—that enable tasks to be performed on behalf of one or more users. In the context of developer-tester collaboration, the most important Agent is the Test Agent, which includes a set of diagnostic data adapters, or DDA’s. There are several DDA’s which each server a different purpose. During test execution DDA’s can be used to:Collect a video capture of the tester’s machine.Collect the steps performed during the test execution.Collect system information from any or all of the machines involved in the test execution.Collect an action recording—a recording of the keyboard strokes and mouse clicks performed by the tester.Collect an IntelliTrace file which can be later used to replay the events that occurred during the test run.DDA’s can be used to impact the system under test (e.g. network emulation), and are extensible as well; so if there’s a piece of information which you find would be helpful for your developers to have from test runs, you can easily write your own DDA which collects that data for inclusion with the actionable bug.While this is not an exhaustive list of the DDA’s and their capabilities, these are the ones that will provide the most value when filing a bug; these are the DDA’s that will automate the collection of information that will turn a normal, somewhat helpful bug into a rich, actionable bug.
• #12: To define an actionable bug I’d like to examine a typical bug. When a tester files a bug they record information they think is important or will help whomever the bug is assigned to understand it better. The problem starts right here. You might say that Developers are from Mars, testers are from Venus. That is to say, the roles of these individuals, and what is important to them and the language they speak is so different that they may as well be from different planets. How can anyone expect a tester to accurately provide the information that a developer needs to do their job? As a result a typical bug fails to contain the necessary detail for a developer to understand the bug, find the root cause and fix it.
• #13: An actionable bug, on the other hand, is a bug that is full of data that will aid the developer in understanding the bug, and enable them to take immediate action toward fixing the bug. An Actionable bug contains enough information that whomever it is assigned to is not blocked from taking action to fix the bug.Let me demonstrate how to use Microsoft Test Manager 2010 to execute a test, find a defect and file an actionable bug.
• #14: See DemoMate file: CreateActionableBug.demo 10-mins
• #15: We have to understand the types of testers and who we are targeting with these tools.We’ve had specialist testing tools in the past and have improved on them.However the big leap with Test Elements is the support for the generalist tester.
• #16: +In General, Manual Testing is supported by the Microsoft Test and Lab Manager that is included in Visual Studio Test Elements 2010 and Specialist testing is supported by Visual Studio.
• #23: As a developer, the most important thing you need to see in a bug is information, and the more the better. An actionable bug is tremendously helpful. No longer do you have to hope that the tester properly documented the steps to reproduce the bug, or the system information—those things are collected automatically and added to the bug. No longer do you have to walk down the hall and ask the tester to run the test case again so that you can watch and see what happened with your own eyes—the screen capture and video capture do that for you. But that still may not be enough to do the most important thing—find the root cause of the bug quickly.
• #24: To quote - Maurice Wilkes, the builder and designer of the EDSAC, the first computer with an internally stored program,“…the realization came over me with full force that a good part of the remainder of my life was going to be spent in finding errors in my own programs.”Even with all the information provided by an actionable bug, unearthing that nugget of information—the root cause of a defect—may be the most difficult thing to do. Imagine if you could go back in time and watch the code execution happen as the bug was discovered. Think of how much time you would save if you could rewind and fast forward through the test execution.
• #25: IntelliTrace is a new capability in Visual Studio 2010 Ultimate that enables developers and testers to collaborate by sharing a historical perspective. When a test case is executed using Test Manager 2010, you have the option of collecting an IntelliTrace file using one of the DDAs. The IntelliTrace DDA collects information about the events that executed during the test run. The level of event data collection is adjustable via DDA configuration, and may be fairly coarse grain collection, or much more fine grain collection. When a tester files a new bug, the IntelliTrace file is automatically attached and available to the developer when they open the bug in Visual Studio. This enables the developer to see the exact code execution that occurred during the test run.
• #26: You could think of this as being like a DVR—a digital video recorder—for developers. When you attach the Visual Studio debugger to an IntelliTrace file you are able to move through the code execution similarly to a DVR—move forward, backward, pause, stop, rewind, etc. Similar to a DVR, you can’t change the code execution that occurred—after all, it already happened, and you can’t change the past—but you are able to examine the code execution more easily. If you miss something, simply “rewind” back to the event to see what happened. This enables you to more easily and more quickly identify the issues that caused the bug.
• #27: Demo: IntelliTrace.demo 10-mins
• #28: Functional testing refers to the testing of a specific action of function of the code. A function test is typically designed to identify if a particular requirements has been fulfilled, if a feature works, or if a user story can be completed without error.
• #29: Currently about 70% of functional testing is done manually—that is, a software tester follows a script to execute a series of steps to verify the outcome of a test. Microsoft Test Manager 2010 is an example of a tool that someone doing manual testing would use to be more productive. Imagine if you could still get the benefits of functional testing without the overhead and resource cost of doing in manually.Consider a regression test—a test designed to identify if a bug fix begins to fail. With a regression test (and even other functional tests) you are repeatedly testing something that was known to work at one point. These too are typically done manually. This is a massive resource hit. Software testers spend countless hours testing functionality that works, solely to ensure it still works. Imagine how much more productive they could be if they could focus their efforts on creating and running new test cases that covered parts of the system not currently tested instead of spending their time testing the same thing, over and over again.
• #30: Comp Photo: http://www.istockphoto.com/stock-photo-1629680-the-cyber-robot-hand.phpA new capability in Visual Studio 2010 Premium is the ability to create a Coded UI Test—a test that automates the testing of the user interface of an application. Coded UI Tests work similarly to unit tests, in that they are written in a programming language—such as Visual Basic or C#—perform some action or series of actions, validate one or more assertions, and are easily repeatable with predictable results. The difference from a unit test is that, while a unit test is intended to test an individual unit of code, such as a single method call, the Coded UI Test is intended for automating a functional test, including the validation of the outcomes within the test. Visual Studio makes it easy for you to create Coded UI Tests be enabling you to use an action recorder to capture your actions—keyboard strokes and mouse clicks—or to simply import an existing action recording created with Test Manager 2010 as part of a test case execution. In either case, the actions recorded are converted into code—Visual Basic or C#. This creates a great collaboration point for developers and testers. Consider the following workflow:A developer writes code to fulfill a requirement.A tester uses Test Manager 2010 to test the implementation and discovers a defect.The tester files an actionable bug.The developer fixes the bug.The tester verifies the fix by successfully completing the test case.The developer (or a test automation engineer) imports the Action Recording from the last test run and creates a Coded UI Test.Every time a new build is deployed to the test environment, all of the Coded UI Tests are executed to ensure there are no regressions.By automating functional tests, including regression tests, you are able to free up testing resources to focus on creating and executing new tests to ensure complete test coverage of your software. This is like adding an automated tester to you team. One that is never late for work, never complains, works at incredible speed and who will take on additional testing work as fast as you can create the Coded UI Tests.
• #31: Demo CodedUITest.demo 10-mins
• #32: Visual Studio Lab Management 2010 takes developer and tester collaboration to the next level. It offers an out-of-the-box solution fully integrated with Visual Studio. It leverages virtualization for application lifecycle management and enables you to Create even richer bugs. With a single click, you can take an environment snapshot and attach a link to it in the bug. Developers can quickly debug issues on real testing environment by connecting to environment snapshot. In case the QA team is still holding on to the test environment, network isolation allows you to create multiple copies of environments without having to reconfigure machines or your application.Once you have created automated test cases, use lab management to automate the end-to-end workflow that builds your app, provisions an environment for application deployment in minutes instead of hours or days, deploys your application through a distributed workflow based on Team Foundation Build and runs automated test suites. The test results are stored in TFS for further analysis and reporting.Furthermore, teams spend a lot of time on tasks that don’t contribute to the over all quality of the application or project success. Areas where teams maybe wasting precious cycles is around setting up and tearing down of environments. Our customers tell us that they spend up to 40% of the test cycle time setting up machines. Lab Management empowers the team members to quickly create complex multi-machine environments. Since Lab Management leverages virtualization, you get all the benefits of virtualization, such as, hardware consolidation which lowers your total cost of ownership.
• #34: At the beginning of this session I asked what your most common defect resolution is currently. More importantly I asked you to consider what the impact to your business and your software projects would be if you could move “Fixed” to the top of the list, and resolutions like ”Cannot Reproduce” to the bottom of the list.
• #35: Today you saw three (3) ways that developers and testers can collaborate to deliver high quality software more easily, and ensure “Fixed” is far and away the most common defect resolution by using the capabilities of Visual Studio 2010.Visual Studio 2010 enables you to:Create rich, actionable bugs that enable any developer to learn enough about the bug to take immediate action.Replay the code execution from a test run to reproduce a bug and quickly find the root cause, even if the development environment is different than the test environment.Create an automated functional test that will quickly discover the bug if it reemerges.