Automating The Process For Building Reliable Software

Editor's Notes

• #2: Introduce yourselfHi my name is Simon Watson, and I am a field applications engineer for Vector Software in London. My role is to support our customers in Europe in the use of VectorCAST, our software test automation tool. I also work closely with our partners and distributors helping them to promote VectorCAST, our test automation tool solution.Before joining Vector Software, I was working as a developer and system integrator writing embedded software to provide secure communications products for financial services companies.The presentation for today will discuss how we can go about automating, and hence reducing the cost of, developing reliable software. Most of the cost and effort involved in developing reliable software goes into the software process around each line of code. By process we mean traceability back to design and requirements and also verification that we have completely tested the system to an appropriate level for the required reliability. Looking at these elements of the process the verification aspect is the most expensive. Hopefully, there will be some pointers in this presentation that you can take into your current projects. If you have any questions at any point, please free to stop me and ask them. Opening QuestionsBefore we go on, I have some questions, just to give me an idea of the kind of audience we have in the room.How many projects are represented here today and are any of those safety or mission critical?How many of you are software testers?How many of you are software developers ? Out of curiosity of those of you that develop software, do you also unit test your own software?How do you unit test your software? Are you using a tool, or it automatedWhich languages are you using on these projects, C, C++, Ada?What internal or external standards or processes are used in developing software?How does your company know that your product is ready to ship to the customer?
• #4: Vector Software has the opportunity to work with a huge variety of customers and understand how their testing process works. One of the key metrics that stands out to us, is how much it costs to build reliable software. Typically, we see customers saying that using a manual process, the cost of testing their software represents about 50-60% of their budget. In fact, we have this quote from Lockheed Martin in the US, a gentleman by the name of Mark Hall. Industry wide, 50% of a software budget is used for FAA, level A, software structural testing.” This is the reason our company was formed 20 years ago, to try and find ways to reduce the significant cost of validating and testing high reliability applications.
• #7: This slide shows us the manual unit and integration testing process.We choose a file that contains a function we want to test.Since we can only execute a complete application, to satisfy the linker we have to build stubs to replace all the called/dependent functions that are not in our source file, we have to supply all the global objects and a driver to call the function under test.Then we have to define input data that we are going to provide as parameters to the function under test, initialisation for global objects and input data returned by stubs that our function callsThen we have to define at least one expected value that is some kind of output or result from the function under test. That might be a C return value, a value passed by reference to our called function, a global that is modified or a value passed into a called function that has been stubbed. The expected value is a test that determines whether the function behaved correctly.We have to add lines of instrumentation to record coverage outputWe compile and link and run the test harness program, debug it as necessary and then record the results.We do that multiple times for a given function under test to obtain 100% coverage with correct and malicious input values chosen both to prove correct operation and to try and break the function.We repeat that process for other source files.We repeat that process for groups of source files to test a complete class or project.
• #8: In the previous slide we talked about writing driver code to call a function under test and stubs to replaced called functions. How much code do we typically need ?Looking at Microsoft Windows NT4 as a sample project, this was supposed to be Microsoft’s secure server platform; that application had 6 million lines of code and Microsoft wrote 12 million lines of test code to verify it. So that is twice as many lines of test code as application code. Even so, we all know how many defects NT4 shipped with. Looking at safety critical applications, typically we need not 2 but 5 lines of test code per line of application code to achieve statement and branch coverage, rising to 10 lines of test code to achieve 100% MCDC coverage.As you can see there is a significant amount of test code to manage and maintain, in addition to the application code.Furthermore that test code is more likely to contain errors than the application code because the application code was written according to a process but there is no defined process for writing the test code.Our experience working with customers is that, for manual testing, 75% of the testing effort is spent building test harnesses and just 25% of the effort goes into running the test cases.Since 50% of the overall budget is spent on testing, that means more than 30% of the total software budget is spent on preparing to test.You can see that any improvement we can make in the process of building test harnesses will bring significant cost savings to the project.
• #9: Lets look at some of the problems that come from the manual test harness process.Since every developer writes their own test framework, 2 developers will write 2 different test harnesses for the same test. If they peer-review or share their work, they will waste time understanding or disagreeing with each other’s testing work. When a developer leaves the team, part of the test process knowledge will leave with them.When test harnesses are built manually, developers will typically write some kind of in-house scripts to automate the execution, the collecting of results and the flagging up of test cases that failed to obtain the expected results. What you will find is that those scripts end up strongly dependent on the project or the execution platform in some way. The more sophisticated those in-house scripts are, the more probably they wont be portable, for example from execution under Linux to Windows, or from project to project.Test harnesses will be locked to the syntax of the functions that are being tested and of the stubs of the dependent functions. When the parameter list changes as the application is developed, the test harness will need to be edited to match the changes. When we move from the unit test phase to the integration test phase, the original test harness will need to be updated because the list of stubs will change and we risk losing the original unit test framework. If system testing encounters a bug that is fixed in the source code, the unit test probably will not be updated because the system tester didn’t write the unit test framework and doesn’t know how to make the change. All these factors lead to manual testing becoming unmaintainable in practice.
• #11: In this slide we are looking at the test harness construction diagramatically :Your project is the set of source files to the left had side, categorised in those 3 colours, red, blue and purple.The red files are the files containing functions we want to test. When we talk about a unit we mean a C or C++ source file or an Ada package. The white rectangles are those functions and the test harness will make those functions visible for test cases. The blue files are source code we have decided not to include in our test. That might be because they have not been written yet or because excluding them, and replacing them by stubs, will allow better control over the testing process. The tool will need to write stubs for all the functions in the blue files that are needed to satisfy the linker.The purple files are code that we want to call when we run the test cases. These usually contain libraries or operating system functionality.The green section is the test driver that will stimulate the function under test. Again this will be written by the automation tool.
• #18: Lets take a look at a concept called basis path analysis. How many people have heard of the cyclometric code complexity measurement by McCabe?How many people are familiar with basis path analysis – which is the application of this measurement to testing?The cyclocmetric code complexity of a piece of code is equal to the number of decision points plus 1.As an example, a straight line function has a complexity of 1. Every deviation from the straight line for an if statement or while loop or for loop adds 1. Each case statement beyond the default inside a switch statement adds 1.The theory is that complexity is directly correlated with errors. Many organisations place a limit on the complexity of their software when developing the code but never use the metric any further in the software process.The key concept behind the basis path analysis is that decision outcomes should be tested independently. So the number of basis path tests will be equal to the number of basis paths which will be equal to the cyclometric code complexity.Basis path will also tell us the minimum number of test cases that are required to verify every independent path in a code segment.Finally, all of the paths in a basis path set should be linearly independent. This means that no path in the basis path set should be definable using other basis paths in the set.
• #19: If we return to our example we can now see that the code complexity for this function will be 3. We get this from the fact that there are 2 ‘if’ conditionals + 1. We then also get our 3 linearly independent basis paths, (False, True), (False, False) and (True, True). These paths translate to the values for X and Y, (0,1), (0,0) and (1,1). As can be seen here, the first case will execute the scenario resulting in the NULL pointer dereference and so correctly identify the coding error.A useful measure when validating an existing test set, is to use Basis Path coverage in conjunction with statement and MC/DC coverage. This can quickly generate missing test cases to ensure the code has been covered completely.
• #26: Again a further screen shot, showing some graphing of the results. This is to give you an idea of how powerful visualising the data can be in reviewing the health of a system. We can see straight away the test environments and hence source code that needs attention.
• #27: There are many approaches to developing systems. IF we recall the graph we review earlier, the later we identify a defect in our system, the costlier it is to fix this defect. A common approach to reduce bug detection cost, is to introduce code inspections, or code static analysis. However there are approaches to software development that can produce even faster bug detection. The techniques are knows as Extreme Programming or Agile techniques. In particular, Test Driven Development.