Software Testing

Practical Skills for Testing Practitioners © 2009

Review

Yue Zhao
Consulting Scientist
Institute for Software Research
Carnegie Mellon University
Objectives
ƒ Explore the knowledge of review, help people understand:
• The definition and category of review
• The category of peer review
• The workflow of inspection
• Testing the specification
• Examining the code

© 2009, CMU-ISR 2
Why Review
ƒ The most important reason should be that reviewing can
save money.
• We can test software as early as possible even before software
can be run. That means reading the specifications, the designs
and the code to find defects.
ƒ The other reasons include:
• Sharing knowledge
• Educating project members
• Providing information to make decisions
• Gathering information of process improvement

© 2009, CMU-ISR 3
What Is Review
ƒ Definition
• An evaluation of software element or project status to ascertain
discrepancies from planned results to recommended
improvements.

ƒ The category of reviews

• Education Review
• Prepare Review
• Peer Review
• State Review

ƒ The one we often use in the software development process

is the peer review to find defects in software products.

© 2009, CMU-ISR 4
The Categories of Peer Reviews
ƒ The categories of peer reviews:
• Inspection
• Team Review
• Walkthrough
• Peer Desk Check
• Ad Hoc Review

ƒ The difference among these types of peer reviews lies

in the formality level.
• Inspection is the most formal one, then team review,
walkthrough, peer desk check and Ad hoc review being the
least formal approach.
• The more formal the method is, the more defects we will find.
• But more formal reviews also cost more.

© 2009, CMU-ISR 5
Inspection
ƒ Inspection
• Also called formal inspection, is a structural process.
• Uses rigorous rules that must be applied.
• The process of inspection is made up of seven stages:
planning, overview meeting, preparation, meeting, rework,
follow-up and analysis.
• Each stage has its own inputs, outputs and participants.

© 2009, CMU-ISR 6
Inspection Roles
ƒ Moderator
• The one who keeps discussions on track during the meeting
stage and ensures follow-up occurs.
ƒ Reader
• The one who describes the deliverables during the meeting
stage.
ƒ Recorder
• The one who writes down problems during the meeting stage.
ƒ Inspector
• Those who examine the deliverables to find defects.
ƒ Author
• Those who provide deliverables to be reviewed.

© 2009, CMU-ISR 7
Inspection Rules
ƒ Some important rules that must be applied:
1. Author shouldn’t act as moderator, reader or recorder.
2. Moderator shouldn’t act as recorder.
3. Keep the inspection team small:3-7 participants will be
better.
4. Inspectors should be sensitive to colleagues.
5. The goal of meeting is to find problems not to solve problems.
6. It should take no more than 2 hours to have the meeting.
7. At the meeting, the inspection team should keep a suitable
speed : 150-200 SLOC/hour or 3-4 pages/hour.

© 2009, CMU-ISR 8
Workflow of Inspection(1)

Planning

Output
Inspection package, Input
schedule meeting(s)
Overview meeting

Checklists, Input
initial version of deliverables, Preparation
supporting docs

Output
Input
Problem list Meeting

Initial version of Output

deliverables
Lessons learned
Inspection report
problem log

© 2009, CMU-ISR 9
 Workflow of Inspection(2)

Problem log Input

initial version of Rework
deliverables

Output
Input
Corrected deliverables
Problem logs
Follow-up

Output

Baselined deliverables

Input
Analysis

Output

Information of process
improvement

© 2009, CMU-ISR 10
Workflow of Inspection(3)
ƒ Planning
• Participants: Author and Moderator
• Objectives: determine objectives, select moderator, identify
inspector, schedule meeting, prepare and distribute inspection
package.
ƒ Overview meeting
• Participants: all the members that will take part in the
inspection.
• Objectives: get participants familiar with the deliverables to be
reviewed, including its features, context, background and so
on.
ƒ Preparation
• Participants: Inspectors
• Participants examine the deliverables line by line to find defects
and write them down in the problems list.

© 2009, CMU-ISR 11
Workflow of Inspection(3)
ƒ Meeting
• Participants: Author, Moderator, Inspectors, Reader, Recorder
• At the meeting, the Reader describes segments of the product
one by one. Participants listen to the reader and see if there
are new problems aside from those in the problem list. The
Author confirms whether the problems are defects. The
Recorder writes down the answer to the problem listed or new
defects found at the meeting. At the end of the meeting, all
vote to draw a conclusion of the appraisal of the product.
ƒ Rework
• Participants: Author
• After the meeting, author fixes the confirmed defects.
ƒ Follow-up
• Participants: Author, Moderator
• Moderator examines corrected deliverable and author checks it.

© 2009, CMU-ISR 12
Workflow of Inspection(3)
ƒ Analysis
• In this stage, the following tasks will be done:
1. Analyzing the cause of the defects.
2. Measuring effectiveness and efficiency.
3. Making improvements of the QA process.

© 2009, CMU-ISR 13
Team Review
ƒ Team Review
• Also called technical review, is a light-weight inspection.
• Has rigorous rules that must be applied like that of inspection.
• The process of the team review is like that of inspection, but no
analysis stage is included.

© 2009, CMU-ISR 14
Walkthrough & Peer Desk Check
ƒ Walkthrough
• The inspection rules are not applied so rigorously. The author
can lead the meeting.
• Process is like inspection, but with no preparation and no
formal follow-up stage.

ƒ Peer desk check

• A nonstructural approach of review.
• Examination by a single reviewer.

© 2009, CMU-ISR 15
Ad Hoc Review
ƒ Ad hoc review
• Get informal feedback from any team member.
ƒ An Example
Here is a scenario: Susan, who is a programmer, is examining her
code and John is another programmer. The story follows:
• Susan: ”Hi, John, Could you do me a favor?”
• John: ”Of course, I’m now free”
• Susan:” There is something wrong with my program, but I’m
not sure what it is”
• John: ”Okay! Let me have a look!”…A few minutes later, ”Oh,
perhaps here it is, you have the wrong variant, you should use
i instead of j”
• Susan: ”Oh, yeah! It is! Thank you very much”
• That is what we call ad hoc review. As you can see, it may
happen anytime, anywhere, among any people.

© 2009, CMU-ISR 16
Review Type To Used
ƒ Suggestion
• The more important or risky the deliverables to be reviewed
are, the more formal of a review should be chosen. For
example:
1. Considering the importance of SRS (Software Requirement
Specification), most project teams choose inspection or at least
team review as the approach, as more information will be
discussed in “Testing the specification.”
2. Most project teams choose inspection or team review to examine
high-level design, low-level design and critical code.
3. Most project teams choose walkthrough or peer desk review as
the approach of examining generic code.

© 2009, CMU-ISR 17
High-Level Review of The
Specification(1)
ƒ Objectives
• Find specific defects such as large fundamental problems,
oversights, omissions, etc.
ƒ Can use the following techniques:
• Put yourself in the user’s shoes: meeting the user needs is
what quality means.
• Research existing standards and guidelines.
• Review and test similar software.

© 2009, CMU-ISR 18
High-Level Review of The
Specification(2)
ƒ If you are a user
• Examining the specification from the customer perspective,
you can ask yourself the following questions…. if I was the
customer:
1. What functions do I need?
2. Are all the functions I need included in the specification?
3. Are there functions conflicting with the former ones?
4. Are the functions easy to use?
5. How about the performance?
6. Are the functions secure?
etc
• Gaining some familiarity with the field in which the software
applies, would be a great help.

© 2009, CMU-ISR 19
High-Level Review of The
Specification(3)
ƒ Existing research standards and guidelines
• Testers may refer to the existing standards and guidelines
when performing high-level reviews of specifications.
Following are some of them:
1. Organization standards, terminology and conventions: software
should adopt the common terms and conventions used by the end
users.
2. Industry standards: In some industries there are strict standards
that software must follow.
3. Government standards.
4. Security standards.
etc
• Testers should consider the relative standards as part of the
specifications.
• When reviewing the specification, testers should test whether
the correct standards are being used and none is overlooked.

© 2009, CMU-ISR 20
High-Level Review of The
Specification(4)
ƒ Review and test similar software
• Nothing is better than hands-on experience. You can get useful
information from similar software. This could be:
1. Former versions of your software
2. Similar software of your organization
3. Competitors’ products
• When reviewing similar software, you may pay attention to the
following questions and think about whether this will impact your
testing.
1. Will there be more or less features?
2. How about the scale of code?
3. Will your software be more or less complex?
4. How about testability?
5. How about the performance, security and other non-functional
attributes?

© 2009, CMU-ISR 21
Low-Level Review of The
Specification(1)
ƒ A Good specification has the following attributes:
• Complete: Is anything missing or forgotten? Is it thorough? Does it
include everything necessary to make it standalone?
• Accurate: Is the proposed solution correct? Does it properly define
the goal? Are there any errors?
• Precise, Unambiguous and Clear: Is the description exact and not
vague? Is there a single interpretation? Is it easy to read and
understand?
• Consistent: Is the description of the feature written so that it doesn’t
conflict with itself or other items in the specification?
• Relevant: Is the statement necessary to specify the feature? Is there
extra information that should be left out? Is the feature traceable to
an original customer need?
• Feasible: Can the feature be implemented with the available
personnel, tools and resources within the specified budget and
schedule?
• To be continued….

© 2009, CMU-ISR 22
Low-Level Review of The
Specification(2)
ƒ A Good specification has the following attributes:
• Code-free: Does the specification stick with defining the
product and not the underlying software design, architecture
and code.
• Testable: Can the feature be tested? Is enough information
provided that a tester could create tests to verify its
operation?
ƒ When examining the specification, you can ask yourself if
the words and pictures you are reviewing have these
attributes.

© 2009, CMU-ISR 23
Low-Level Review of The
Specification(3)
ƒ List of problem words
• When examining the specification, pay more attention to the
following words and the context that are ambiguous or
unclear. There are chances that they may lead to defects.
1. Always, Every, All, None, Never: these words denote
something as certain or absolute. Make sure that it is,
otherwise think of cases which violate them.
2. Certainly, Therefore, Clearly, Obviously, Evidently: these
words tend to persuade you into accepting something.
Please think about it carefully.
3. Some, Sometimes, Often, Usually, Ordinarily, Customarily,
Most, Mostly, Etc. And so forth, And so on, Such As, Good,
Fast, Cheap, Efficient, Small, Stable: these word aren’t
testable. They must be further defined to explain exactly
what they mean.

© 2009, CMU-ISR 24
Low-Level Review of The
Specification(4)
ƒ List of problem words
1. Handled, Processed, Rejected, Skipped, Eliminated: these
words may hide functionality which should be specified.
2. If…then: the statements miss else clause.

© 2009, CMU-ISR 25
Examining the code
ƒ Examining the code means reading the code line by line to
find defects as soon as the program or part of it has been
finished.
ƒ Reasons
• Find defects as early as possible.
• Make the program more testable.
• Software tester can get familiar with the application.
ƒ Approach
• Static white-box testing: many software project teams choose
inspection as the approach of examining core code and walkthrough
or peer desk review of generic code.
• It would be much better if your team adopts the following rules when
examining the code.
1. The reviewer should have knowledge of the used programming
language.
2. One must refer to programming guidelines and standards.

© 2009, CMU-ISR 26
Programming Standards and
Guidelines
ƒ Programming standards and guidelines are collections of
best experience of programming.
ƒ Repeating those experiences will be a great help when
coding and examining the code.

© 2009, CMU-ISR 27
Checklist
ƒ According to programming standards and guidelines, there is
always a checklist for reviewers to refer to when examining the
code.
ƒ When examining the code, a reviewer should ask himself if the
code satisfies the requests listed in the checklist. If not, it is
likely that there are defects.
ƒ A segment of a Java Checklist Example (Adopted from Java
Conventions:
http://java.sun.com/docs/codeconv/CodeConventions.pdf)

© 2009, CMU-ISR 28
Where Can Standards Be Obtained
ƒ Available sources
• National and international standards
1. American National Standards Institute(ANSI): www.ansi.org
2. International Engineering Consortium(IEC): www.iec.org
3. International Organization for Standardization(ISO): www.iso.org
4. National Committee for Information Technology Standards(NCITS):
www.nctis.org
• Professional organizations
1. Association for Computing Machinery(ACM): www.acm.org
2. Institute of Electrical and Electronics Engineering(IEEE): www.ieee.org
• Software Vendors

ƒ Things you should keep in mind

• Different programming languages and different project teams may adopt
different standards and guidelines. The one which is suitable for your
project is the best one.
• Standards and guidelines should be examined and renewed after each
project has been finished.

© 2009, CMU-ISR 29
An Example
ƒ Here is a piece of code of the initial version of the NextDate
Function given in lecture 1. When I finished coding, I asked one of
my colleagues to review the code and he read it line by line and
found a defect. Where is it?

© 2009, CMU-ISR 30
Summary
ƒ We can use reviews to examine software before it is finished.
ƒ Reviews can save money.
ƒ In software development, we often use peer review.
ƒ Inspection is the most formal review method and is usually
applied when reviewing important deliverables.

© 2009, CMU-ISR 31