Test Management: Part II

Software Testing: INF3121 / INF4121

Summary: Week 7
Test organisation
Independence | Tasks of the test leader and testers

Test planning and estimation

Activities | Entry and exit criteria | Estimation | Strategy and approach

Test progress monitoring and control

Configuration and management
Risk and testing
 project title

Part I: Close-ended questions

Question 1
Which of the following metrics would be most useful to
monitor during test execution? project title

a. Percentage of test cases written

b. Number of test environments remaining to be configured
c. Number of defects found and fixed
d. Percentage of requirements for which a test has been
written
Question 1: Answer
Which of the following metrics would be most useful to
monitor during test execution? project title
Test progress monitoring
Give feedback and visibility of the test effort / activities
Monitoring information can be collected manually / automatically
Metrics can be used to measure exit criteria

Common test metrics

Question 1: Answer
Which of the following metrics would be most useful to
monitor during test execution? project title

a. Percentage of test cases written

b. Number of test environments remaining to be configured
c. Number of defects found and fixed
d. Percentage of requirements for which a test has been
written
Question 2
In a test summary report, the project’s test leader makes
the following statement: project title
“The payment processing subsystem fails to accept payments from
American Express cardholders, which is considered a must-work
feature for this release.”
This statement is likely to be found in which section?
a. Evaluation
b. Summary of activities
c. Variances
d. Incident description
Question 2: Answer
This statement is likely to be found in which section?
Test reporting project title
Summarising information about the test effort
What happened during a test period?
E.g. dates when exit criteria were met

Analysed metrics to support decisions about future actions

Adequacy of the test objectives for that test level
Adequacy of the test approaches taken
Effectiveness of the testing with respect to its test objectives
Question 2: Answer
This statement is likely to be found in which section?
project title

“The payment processing

subsystem fails to accept payments
from American Express
cardholders, which is considered a
must-work feature for this release.”
Question 2: Answer
In a test summary report, the project’s test leader makes
the following statement: project title
“The payment processing subsystem fails to accept payments from
American Express cardholders, which is considered a must-work
feature for this release.”
This statement is likely to be found in which section?
a. Evaluation
b. Summary of activities
c. Variances
d. Incident description
Question 3
During an early period of test execution, a defect is located,
resolved and confirmed as resolved by re-testing, but is seen
project title
again later during subsequent test execution.
Which of the following is a testing-related aspect of
configuration management that is most likely to have broken
down?
a. Traceability
b. Confirmation testing
c. Configuration control
d. Test documentation management
Question 3: Answer
Which is a testing-related aspect of configuration
management most likely to have broken down? project title
Configuration Management
Establish and maintain the integrity of the products of the software

Integrity  Assurance of the accuracy and consistency of data through the life cycle

Ensure that all items of testware are:

Identified
Version controlled
Maintain traceability
throughout the test process
Tracked
Question 3: Answer
During an early period of test execution, a defect is located,
resolved and confirmed as resolved by re-testing, but is seen
project title
again later during subsequent test execution.
Which of the following is a testing-related aspect of
configuration management that is most likely to have broken
down?
a. Traceability
b. Confirmation testing
c. Configuration control
d. Test documentation management
Question 4
You are working as a tester on a project to develop a
point-of-sales system for grocery stores. Which of the
project title
following is a product risk for such a project?

a. Arrival of a more reliable competing product on the market

b. Delivery of an incomplete test release to the first cycle of
system test
c. An excessively high number of defect fixes fail during re-
testing
d. Failure to accept allowed credit cards
Question 4: Answer
Which of the following is a product risk for a point-of-
sales system? project title
Risks
Possibility of a negative / undesirable outcome
Possible problems that may endanger the objectives of the project stakeholders
Risk analysis and management aid in managing possible negative outcomes

Two main categories of risks

Product risks
Project risks
Question 4: Answer
Which of the following is a product risk for a point-of-
sales system? project title
Product Risks
Potential failure areas in the software

Risks specifically related to the quality of the product

Failure-prone software
Software / Hardware that could case harm to the user
Poor software characteristics
Software that does not perform its intended functions
Question 4
You are working as a tester on a project to develop a
point-of-sales system for grocery stores. Which of the
project title
following is a product risk for such a project?

a. Arrival of a more reliable competing product on the market

b. Delivery of an incomplete test release to the first cycle of
system test
c. An excessively high number of defect fixes fail during re-
testing
d. Failure to accept allowed credit cards
Question 5
A product risk analysis meeting is held during the
planning period. Which of the following determines the
project title
level of risk?

a. Difficulty of fixing related problems in code

b. The harm that might result to the user
c. The price for which the software is sold
d. The technical staff in the meeting
Question 5: Answer
What determines the level of risk during a product risk
analysis meeting in the planning period? project title
Determining the level of risk
Likelihood  What are the chances?
Impact  What are the consequences?
Question 5
A product risk analysis meeting is held during the
planning period. Which of the following determines the
project title
level of risk?

a. Difficulty of fixing related problems in code

b. The harm that might result to the user
c. The price for which the software is sold
d. The technical staff in the meeting
Question 6
You are writing a test plan using the IEEE 829 template
and currently completing the Risks and Contingencies
project title
section.
Which of the following is most likely to be listed as a
project risk?

a. Unexpected illness of a key team member

b. Excessively slow transaction-processing time
c. Data corruption under network congestion
d. Failure to handle a key use case
Question 6: Answer
Which of the following is most likely to be listed as a
project risk? project title
Project risks
Risks that surround the project's capability to deliver its objectives

Risks specifically related to aspects of the project processes

Organisational factors
Shortage of people, skill, training | Communication and attitude problems
Technical issues  Problems defining the right requirements
Supplier issues  Third-party failures
Question 6: Answer
You are writing a test plan using the IEEE 829 template
and currently completing the Risks and Contingencies
project title
section.
Which of the following is most likely to be listed as a
project risk?

a. Unexpected illness of a key team member

b. Excessively slow transaction-processing time
c. Data corruption under network congestion
d. Failure to handle a key use case
Question 7
You and the project stakeholders develop a list of product risks and
project risks during the planning stage of a project. What else should
you do with those lists of risks during test planning? project title

a. Determine the extent of testing required for the product risks and the
mitigation and contingency actions required for the project risks
b. Obtain the resources needed to completely cover each product risk with
tests and transfer responsibility for the project risks to the project
manager
c. Execute sufficient tests for the product risks, based on the likelihood and
impact of each product risk and execute mitigation actions for all project
risks
d. No further risk management action is required at the test planning stage
Question 7: Answer
What else should you do with those lists of product and
project risks during test planning? project title
Risk-based approach during the planning stage
Product risks
Determine the extent of testing to be carried out
Prioritise testing in an attempt to reveal critical defects as early as possible
Project risks
Identify risk items
Determine likelihood and impact for each item
Define mitigation and contingency actions for the identified risks
Question 7: Answer
You and the project stakeholders develop a list of product risks and
project risks during the planning stage of a project. What else should
you do with those lists of risks during test planning? project title

a. Determine the extent of testing required for the product risks and
the mitigation and contingency actions required for the project risks
b. Obtain the resources needed to completely cover each product risk with
tests and transfer responsibility for the project risks to the project
manager
c. Execute sufficient tests for the product risks, based on the likelihood and
impact of each product risk and execute mitigation actions for all project
risks
d. No further risk management action is required at the test planning stage
Question 8
According to the ISTQB Glossary, a product risk is related
to which of the following? project title

a. Control of the test project

b. The test object
c. A single test item
d. A potential negative outcome
Question 8: Answer
According to the ISTQB Glossary, a product risk is related
to which of the following? project title
Product risk: ISTQB definition

In contrast to a project risk

"A risk related to management and control of the project "
E.g. lack of staffing, strict deadlines, changing requirements, etc.
Question 8: Answer
According to the ISTQB Glossary, a product risk is related
to which of the following? project title

a. Control of the test project

b. The test object
c. A single test item
d. A potential negative outcome
Question 9
Typical test strategies include:
project title
1. Risk-based testing
Testing directed to areas of greatest risk
2. Failure-based testing
Test more in failure-prone areas
3. Management-directed testing
Test first the areas indicated by the org. management

Which are true and which are false?

Question 9: Answer
Risk-based testing
Idea: Can organise test effort to reduce the residual levelproject
of risktitle
Use of risk analysis to prioritise and emphasise appropriate tests
Guide planning, specification, preparation, and execution
Analysis and risk identification starts early in the project
Involves both preventive (mitigation) and damage control (contingency) measures
Mitigation  Testing to provide opportunities to reduce likelihood of defects

Contingency Testing to identify work-arounds for the defects that aren't discovered
Question 9: Answer
Failure-based testing
Idea: Can organise test effort based on failure-prone areas
project title
Identifies areas most likely to hold defects, based on complexity
Some functionality is harder to implement than other
These areas are more vulnerable to defects

Starts early in the project and conducted throughout the life cycle
New problems / challenges will always emerge

Management-directed testing
This is not a defined test strategy in the ISTQB syllabus
Question 9: Answer
Typical test strategies include:
project title
1. Risk-based testing TRUE
Testing directed to areas of greatest risk
2. Failure-based testing TRUE
Test more in failure-prone areas
3. Management-directed testing FALSE
Test first the areas indicated by the org. management

Which are true and which are false?

Question 10
In an incident report, the tester makes the following statement:
"At this point, I expect to receive an error message explaining the rejection of this invalid input
project title
and asking me to enter a valid input. Instead, the system accepts the input, displays an
hourglass for between one and five seconds, and finally terminates abnormally, giving the
message [Unexpected data type: 15. Click to continue]"

This statement is likely to be found in which of the following

sections of an IEEE 829 Incident Report?
a. Summary
b. Impact
c. Item pass / fail criteria
d. Incident description
Question 10: Answer
This statement is likely to be found in which of the
following sections of an IEEE 829 Incident Report?
project title

“At this point, I expect to receive an

error message explaining the rejection
of this invalid input and asking me to
enter a valid input. Instead, the system
accepts the input, displays an hourglass
for between one and five seconds, and
finally terminates abnormally, giving the
message [Unexpected data type: 15.
Click to continue]"
Question 10: Answer
In an incident report, the tester makes the following statement:
"At this point, I expect to receive an error message explaining the rejection of this invalid input
project title
and asking me to enter a valid input. Instead, the system accepts the input, displays an
hourglass for between one and five seconds, and finally terminates abnormally, giving the
message [Unexpected data type: 15. Click to continue]"

This statement is likely to be found in which of the following

sections of an IEEE 829 Incident Report?
a. Summary
b. Impact
c. Item pass / fail criteria
d. Incident description
Question 11
According to the ISTQB Glossary, what do we call a
document that describes any event that occurred during
project title
testing which requires further investigation?

a. A bug report
b. A defect report
c. An incident report
d. A test summary report
Question 11: Answer
According to the ISTQB Glossary, what do we call a
document that describes any event that occurred during
project title
testing which requires further investigation?
Incident
Discrepancies between actual and expected test outcomes

Incident management
The process of recognising, investigating, taking actions, and disposing of incidents
Incident report
A document reporting on any event that occurred during testing
Requires further investigation
Question 11: Answer
According to the ISTQB Glossary, what do we call a
document that describes any event that occurred during
project title
testing which requires further investigation?

a. A bug report
b. A defect report
c. An incident report
d. A test summary report
Question 12
A product risks analysis is performed during the planning stage of
the test process. During the execution stage, the test manager directs
projectrisk
testers to classify each defect report by the known product title it
relates to (or by 'other').
Once a week, the test manager runs a report that shows the
percentage of defects related to each known product risks, and to
unknown risks.
What is one possible use of such a report?
a. To identify new risks to system quality
b. To locate defect clusters in product subsystems
c. To check risk coverage by tests
d. To measure exploratory testing
Question 12: Answer
What is one possible use for a report showing defects
related to known product risks and unknown risks?
project title
Objectives of an incident report
Provide developers with feedback about incidents to:
Enable identification, isolation, and corrective measures
Provide test leaders with feedback about incidents to:
Track system quality
Measure test progress
Provide ideas for test process improvement
Identify potential new risks to system quality
Question 12: Answer
A product risks analysis is performed during the planning stage of
the test process. During the execution stage, the test manager directs
projectrisk
testers to classify each defect report by the known product title it
relates to (or by 'other').
Once a week, the test manager runs a report that shows the
percentage of defects related to each known product risks, and to
unknown risks.
What is one possible use of such a report?
a. To identify new risks to system quality
b. To locate defect clusters in product subsystems
c. To check risk coverage by tests
d. To measure exploratory testing
 project title
Part II: Exercises and
Open-ended questions
Exercise 1
Go to the following website and find an incident:
project title
www.adam.goucher.ca/parkcalc/

Fill out the attached incident report

Exercise 1: Incident Report
Date:______________
Project:________________
Tester:___________________________
project title

Program / Module:______________________ Build/Revision/Release:__________________

Software Environment:_____________________ Hardware Environment:________________
Status of the incident:_________________________________________________________

Number of Occurrences:________ Severity:_______ Impact:______ Priority:_____________

Detailed Description:__________________________________________________________
__________________________________________________________________________
Expected result / Actual result: __________________________________________________

Assigned To:__________________________________________
Incident Resolution::____________________________________
Exercise 1: Incident Report

project title
The seminar slides are made by

Yulai Fjeld ydfjeld @ uio.no

Master student
Department of Informatics
University of Oslo

Previously taught courses

Systemutvikling (INF1050), Universitet i Oslo
Software Testing (INF3121/4121), Universitetet i Oslo
Systemutvikling (ADSE2200), Høgskolen i Oslo og Akershus