2012_en_using_vectorcast_to_satisfy_software_verification_and_validation_for_iso_26262_wp
2012_en_using_vectorcast_to_satisfy_software_verification_and_validation_for_iso_26262_wp
Introduction
Competition in the automotive industry is intense. Successful companies must constantly
innovate by introducing new features, many of which contain significant amounts of software.
The automobile has been transformed from primarily a mechanical device, into an integrated
machine with embedded software in all major systems including: engine control, power train,
suspension, braking, and entertainment.
Controlling the costs of automotive embedded systems is extremely important for automotive
industry suppliers since there is a much higher volume of software than other safety-critical
industries like avionics and railway.
Software testing has traditionally been very expensive, but the cost of finding software bugs
now versus the direct costs and damaged product branding associated with recalls makes
thorough testing a necessity in the automotive industry.
VectorCAST.com
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
MISRA
Static analysis has been a big part of the automotive application development process since
the advent of the Motor Industry Software Reliability Association (MISRA) C standard.
“Guidelines for the Use of the C Language in Vehicle Based Software” is a document which
was first published in 1998 to promote safe use of the C language in the automotive industry.
It contains rules defining a subset of the C language that is now widely accepted as a model for
good programming practice. “MISRA C++: 2008 Guidelines for the use of the C++ Language in
Critical Systems” was published in 2008 to define similar rules for the C++ language.
ISO 26262
ISO 26262 is a Functional Safety standard currently under development, titled "Road vehicles --
Functional safety”. The standard is an adaptation of the Functional Safety standard IEC 61508
for Automotive Electric/Electronic Systems. Part 6 of the ISO 26262 standard addresses the
recommendations for dynamic software testing and verification as part of the standard for
software development.
Recommended activities include both unit level and system level testing, such as functional
tests (requirement-based tests and partition tests) and structural coverage tests.
VectorCAST is also used for a variety of robustness testing activities such as range and out-of-
bounds testing.
Additionally, VectorCAST tools support capture and reporting of structural code coverage for
all Automotive Safety Integrity Levels (ASIL) required by ISO 26262.
ASIL is the automotive specific risk-based approach for determining product risk classes. Risk classes are defined
as Level A though D, with ASIL D representing the highest risk.
VectorCAST.com Page 2
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
The VectorCAST/C++ test tools for C and C++ unit and integration testing, combined with
VectorCAST/Cover for system-level test verification, provide a complete dynamic test suite for
host, simulator, and target level testing.
Please note that ISO 26262 is presently in the last stages of acceptance and specific recommendations may
change before it is homologated.
R Recommended activity
HR Highly recommended activity
ISO 26262 proposes a waterfall approach to testing, with a clear demarcation between unit
testing activities and system testing activities. The document thus addresses these two levels
of testing separately.
To achieve this goal, the standard recommends the following unit testing methods to be
implemented (see ISO 26262 Table 12 – Methods for Software Unit Testing).
ASIL Supported by
Methods
A B C D VectorCAST
1
If using VectorCAST/C++ under the control of the debugger or in conjunction with other tools.
2
VectorCAST can use the data from models to generate test cases.
VectorCAST.com Page 3
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
Using VectorCAST/RGW (a module of VectorCAST/C++) each test case can also be linked to a
specific software requirement downloaded from a requirements management tool such as
IBM® Rational® DOORS®. Once the link is established, that information can also be uploaded to
the requirements database, along with the test status information, making it even easier to
monitor which requirements were met – and which were not.
All of these activities can easily be performed with VectorCAST/C++. Extreme values can be
specified according to the functional range or type, and illegal values can be easily specified.
The exact range of each data type is duly tested by VectorCAST/C++ in the target environment,
so the boundary values are known with precision and can be used to generate boundary test
cases automatically. These can also be done in a combinatorial mode, which multiplies the
number of effective test cases that are derived from a single set of data – entirely
automatically.
VectorCAST.com Page 4
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
ASIL Supported by
Methods
A B C D VectorCAST
The standard also makes provisions for testing based on the structure of the code itself
(generation and analysis of equivalence classes and analysis of boundary values) activities
highly recommended whenever testing a system to levels ASIL B, ASIL C, and ASIL D.
VectorCAST.com Page 5
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
These types of activities are also highly automated in VectorCAST/C++, which enables users to
quickly build test cases based on ranges and lists of values. These inputs can be executed in a
non-combination, linear mode, or the tool can use all of the input combinations possible to run
tests. Execution of these complex test cases is done automatically whether they are executing
in a host, simulator, or target environment.
VectorCAST also features a partition test case generator to automatically create additional test
cases on a provided domain.
This can be done easily within VectorCAST/C++, on both the range of the variable type and the
functional range. It can be further automated by automatically generating MIN-MID-MAX test
cases, which set all values to their minimum, median and maximum values, respectively. The
minimum and maximum values are determined by testing the range of every type present in
the program on the target board or simulator. Thus, using the tool on either the board or on a
simulator will guarantee that the range of boundary values tested through automatically
generated MIN-MID-MAX tests is valid in the system, whether it is 8, 16 or 32-bit.
These two tools can also test approaching values, illegal values, and even special values such
as Not-A-Number (NaN), positive and negative infinity on floating-point variables.
VectorCAST.com Page 6
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
With VectorCAST, creating a test case is a straightforward matter, with no need for scripting.
Signals can be raised for exceptions thrown and pointers can be voluntarily left un-initialized to
see if the code will be protected against this type of occurrence. Performing “what-if”
scenarios are easily generated, while manual test or script-based tools would require
significant amounts of test code to be developed.
All test cases are kept outside the test harness until needed, which means that test cases can
be created and deleted without the need for recompiling the code, which maximizes
productivity when compared to other “similar” tools.
The goal is clearly intended to be 100% coverage based on the coverage criteria selected. As
stipulated by ISO 26262 Table 14, Note 4, a rationale is to be given for the level of coverage
achieved (e.g.; for accepted dead code or code segments depending on different software
configurations), or else code not covered can be verified using complementary methods (e.g.;
inspections).
At the unit level, three different criteria can be chosen for code coverage as indicated below
(see ISO 26262 Table 14 - Structural Coverage Metrics at the Software Unit Testing)
ASIL Supported by
Methods
A B C D VectorCAST
VectorCAST.com Page 7
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
It should be noted that these levels of coverage are progressively more difficult to achieve.
When using a tool like VectorCAST (which enables users to test MC/DC, Branch and Statement
in isolation), the following combinations of criteria should be used in order to achieve a
structural coverage level compatible with the letter – and spirit - of ISO 26262.
VectorCAST/C++ (as well as VectorCAST/Cover) has a simple to use code coverage viewer. The
coverage viewer indicates through symbols and color codes whether the code (a) is entirely
covered (in green), (b) is partially covered (in orange), or (c) is not covered (in red). Leaving the
cursor in place on any on the lines covered enables users to see which test cases cover specific
lines.
VectorCAST.com Page 8
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
In the case of targets with sufficient memory and liberal timing constraints, code coverage can
even be animated – literally “replaying” how the code coverage was achieved during
execution. For more restricted environments, code coverage can also be run in modes that
save on memory space and/or have less of an impact on timing issues. The VectorCAST code
coverage instrumentation and data collection has several options that allow the user to
customize to maximize resource efficiency for special applications.
It should be noted that although a single test case evaluating to false at this line will not cover
additional lines that may be contained within that ‘IF’ statement. Likewise, if this ‘IF’ statement
has an ‘ELSE statement attached to it, a test case evaluating to true will not cover the contents
of that ‘ELSE’ statement. However, either the true or false test case will cover the ‘IF’
statement per se.
VectorCAST/C++ (as well as VectorCAST/Cover) fully supports Branch coverage, either stand
alone, or in combination with other criteria of code coverage. To comply with ISO 26262,
VectorCAST can produce both Statement and Branch levels of coverage during a single test
execution.
VectorCAST.com Page 9
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
One should demonstrate that by changing the value of ‘i’ while keeping the value of other sub-
conditions stable, the end value will change.
This task can be very arduous even for the most experienced engineer. However, VectorCAST
provides a very efficient way of doing this type of testing through its VectorCAST/MCDC
module. A truth table is automatically generated which indicates clearly which test case pairs
are required to achieve MC/DC coverage, and then flags which test cases and test case pairs
have been provided.
VectorCAST.com Page 10
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
Although ISO 26262 does not make the distinction, it is advantageous to consider the testing
of modules (units that are combined to represent a functional process, but not the whole
software) as module testing or integration testing (which is the term used on
www.vectorcast.com). A test performed on the entire applications is often referred to as a
system test.
In ISO 26262, the process should be to progressively integrate different units together based
on the particular hierarchy of the software until the embedded software is fully integrated at
system level (section 10.4.1). It thus suggests that both module testing and system testing shall
be undertaken.
VectorCAST.com Page 11
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
Methods for Software Integration and Testing and for Deriving Test Cases
Just like during the unit testing phase, a number of methods are recommended for software
integration testing. These are listed below (see ISO 26262 Table 15 – Methods for Software
Integration Testing).
ASIL Supported by
Methods
A B C D VectorCAST
Likewise, the methods for deriving test cases for software integration testing are listed in
below (see ISO 26262 Table 16 – Methods for Deriving Test Cases for Software Integration
Testing).
ASIL Supported by
Methods
A B C D VectorCAST
5
1a. Analysis of Requirements HR HR HR HR
5
1b. Generation and Analysis of Equivalence Classes R HR HR HR
5
1c. Analysis of Boundary Values R HR HR HR
5
1d. Error Guessing R R R R
ISO 26262 Table 16 – Methods for Deriving Test Cases for Software Integration Testing
These methods are exactly the same as those defined in the unit testing section. In this
section, we will focus on highlighting the useful features of VectorCAST that specifically
address integration testing.
3
If executing test cases from the control of the debugger.
4
VectorCAST can use the data from models to generate test cases.
5
Applicable to VectorCAST/C++ only.
VectorCAST.com Page 12
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
VectorCAST/C++ offers real integration testing – the files that are put together become, in
essence, a larger, integrated module. Thus, if tests are executed on one function in a first unit,
and if this unit itself calls another function in the second unit, both functions will be tested and
both will generate code coverage. As in the case of unit testing, there is no script to write in
order to generate an integration test environment – everything is automatically generated.
System testing
Once the integration reaches system level, the usual build system usually will take over, and
test case input will be done through other means, such as signal generation, pushing buttons
at a console, or maybe with a simulator. During system testing, VectorCAST/Cover is used to
capture the code coverage (see two sections below).
VectorCAST.com Page 13
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
ASIL Supported by
Methods
A B C D VectorCAST
6
To be supported by VectorCAST in 2012.
One frequent question asked about VectorCAST/Cover is how the tool exports the code
coverage data from different targets. In fact, the tool can export the coverage data through
three broad methods: (a) save to a file (particularly useful in case a file system is present or a
simulated file I/O can be established), (b) send through a port (such as a serial port), and (c)
store the data in a memory buffer that is then saved with the help of the debugger or another
means.
Reporting
ISO 26262 specifies creation of a number of documents, such as a Software verification
specification and a Software verification report.
The VectorCAST products produce a variety of test artifacts based on unit test, integration test,
or system test. These reports can be generated in either Text format or HTML. They can be
saved outside VectorCAST, and have been used to comply with a variety of standards such as:
IEC 61508, CENELEC, DO-178B, etc.
VectorCAST.com Page 14
Using VectorCAST to Satisfy Software Verification and Validation for ISO 26262
Certification
Both VectorCAST/C++ and VectorCAST/Cover can be certified for compliance activities with
ISO_26262. Please contact Vector Software for more information.
Conclusion
The VectorCAST embedded software testing platform provides tools for automating testing
and code coverage activities in a way that makes complying with MISRA and ISO_26262
requirements much more efficient. All of these tools can export their individual reports in
HTML or Text, which have been used successfully in the past to comply with a number of
demanding industrial standards.
VectorCAST/Lint is used to perform static source code analysis on C/C++ codebases and can
automatically identify problems at their source prior to compiling. VectorCAST/Lint is
configured for checking the MISRA C, MISRA C 2004, and MISRA C++ 2008 standards.
VectorCAST/RGW permits the flow of data between a requirements management tool such as
IBM® Rational® DOORS® and the VectorCAST testing tool. Through a simple and intuitive
interface, developers can quickly link requirements to VectorCAST test cases.
VectorCAST/RSP enables the execution of a test harness on simulator or a target board. The
process is entirely automated, so test cases can be executed individually or as a group by a
simple click of a mouse or the command line. The execution itself requires no user input.
VectorCAST.com Page 15