3

September, 2008

The Magazine for Professional Testers

350 Pantone 362�

80% 60% 40% 20% 100% 80% 60% 40% 20%

cmyk

c:63 c:47 c:32 c:16 c:76 c:61 c:46 c:30 c:16

m:0 m:0 m:0 m:0 m:0 m:0 m:0 m:0 m:0
y:70 y:52 y:35 y:17 y:100 y:80 y:60 y:40 y:20
b:61 b:46 b:31 b:15 b:11 b:9 b:7 b:5 b:3
printed in Germany

Test Techniques in practice -

free exemplar

Do they help?
Why do we often test without them?
www.testingexperience.com
ISSN 1866-5705

© iStockphoto
 Editorial
I hope you have enjoyed the summer time and spent some days at the beach or the
mountains. I couldn’t. I really would love to spend some days at the beaches of Gran
Canaria, just laying down and having a nice cold beer and fish or a paella at the “Las
Canteras”-beach promenade for dinner … But that’s life! I had to work.

In this issue we focus on test techniques. Test techniques are the key to success. It
doesn’t matter if you use them systematically or just based on your experience perform-
ing exploratory testing. We are lost without these techniques and experiences. Some
of us use regularly equivalence partitioning, boundary check, error guessing, decision
tables and paiwise for our test and perform them systematically. Some of us don’t.
If you have a look on the market there are a lot of books and seminars which explain
these techniques. My impression based on my old days as tester and actually as man-
ager of a “testing company” is that most people involved in this field don’t use to apply
them for the daily work. Why?

My personal opinion is that the market is leaded by tools companies that don’t use them
directly. If you have a look they support test management, test processes, test automa-
tion but not really test techniques. Test techniques as the key for the success may have
been forgotten for a long time.

In this new issue there are some articles that give you tips for the leisure time, for test
case design, test management and share experiences or new ideas. Please enjoy them.
We have started a serial of interviews to presenting interesting people worldwide and
their work, thoughts and strategies that influence our field.
With the second issue we have achieved over 80,000 readers! I want to thank all of you
for the support and for spreading the magazine in your community.
We are working hard on a new website with community character that should appear
in a few weeks. We will start with a beta version and hope that all of you find a lot of
bugs!
I wish you a pleasant time with testing experience.

Yours sincerely

José Manuel Díaz Delgado

www.testingexperience.com The Magazine for Professional Testers 3

 Contents
Editorial............................................................................................................................... 3
News................................................................................................................................... 7
The benefits of modeling in a large-scale test integration project: A case study.......... 8
by Graham Bath
SW quality improvements by using quality KPIs............................................................ 12
by Dr. Hartwig Schwier
Are test design techniques useful or not?...................................................................... 15
by Hans Schaefer
Test Process Improvement using TMMi.......................................................................... 21
by Erik van Veenendaal, Rob Hendriks, Jurian van de Laar and Bart Bouwers
Interview Norbert Bochynek............................................................................................ 26
Software Testing – the future?........................................................................................ 29
by Geoff Thompson
What Testing Cannot Do.................................................................................................. 34
by Gerald M. Weinberg
Two Weeks to Better Testing this Summer..................................................................... 36
by Rex Black
The Boundary Value Fallacy............................................................................................ 38
by René Tuinhout
Testing Techniques and the Theory of Constraints....................................................... 44
by Ladislau Szilagyi
Index Of Advertisers......................................................................................................... 47
Improving the testing process using the TMMi.............................................................. 48
by Erik van Veenendaal
10 tips for successful testing!......................................................................................... 52
by Dominic Maes and Steven Mertens
Explaining Security Testing as Real Trousers................................................................. 54
by Bogdan Bereza-Jarociński
Method for Reducing a Generic Software Routine into a Mathematical Function...... 58
by Danilo Berta
Database Testing - Testing The Backend........................................................................ 65
by Gulshan Bhatia
Strategies for Testing Telecommunication Services...................................................... 70
by Bárbara Martínez , Jesús del Peso and Lourdes Calvo

SW quality improvements
What Testing by using quality KPIs
by Gerald M. Weinberg
Cannot Do

19
36
© iStockphoto

© iStockphoto
by Dr. Hartwig Schwier

4 The Magazine for Professional Testers www.testingexperience.com

 Control integration and test activities in your project.................................................... 74
by Erwin Engelsma Pantone 350 Pantone 362�

Diffusion Methodology for Regression Test Analysis..................................................... 77

by Oron Dilmoni 100% 80% 60% 40% 20% 100% 80% 60% 40%

cmyk cmyk
Test design techniques were not invented to bully testers!.......................................... 82
by Leo van der Aalst
c:79 c:63 c:47 c:32 c:16 c:76 c:61 c:46 c:30
Masthead..........................................................................................................................
m:0
y:87
m:0
y:70
m:0
y:52
86
m:0
y:35
m:0
y:17
m:0
y:100
m:0
y:80
m:0
y:60
m:0
y:40
b:76 b:61 b:46 b:31 b:15 b:11 b:9 b:7 b:5
The Test Maturity Model Integrated (TMMi ®)................................................................ 87
by Brian Wells
Size Does Matter in Process Improvement.................................................................... 91
by Tanja E.J. Vos, Jorge Sánchez Sánchez and Maximiliano Mannise
Personnel evaluation based on tests............................................................................. 96
by Andrey Konushin
Practicing Test Techniques - A Discussion...................................................................... 98
by Yaron Tsubery

Method for Reducing a Generic Software Routine

into a Mathematical Function
... and Possible Applications for Black-Box (Functional) Software Testing

42
© iStockphoto

© iStockphoto
by Danilo Berta

The benefits of modeling in a large-scale test integration project:

A case study

25
by Graham Bath

www.testingexperience.com The Magazine for Professional Testers 5

© iStockphoto

European Tour
Application Security Testing
2 days testing course by Manu Cohen-Yashar

3rd - 4th November Berlin

e s 6th - 7th November Amsterdam
ac
pl 1st - 2nd December London
e d 4 - 5th December Stockholm
th

mit
Li
We are living in a world of data and communication, in which the most valu-
able asset is information. There is no doubt that today’s applications must be
secure.

Security Standards are created to insure that products will implement security
measures to protect their data.

Security is an “all-inclusive” term, which means it must be implemented “ev-

erywhere”, in all levels:

ff Users: Train your users and build awareness to help them to reduce the
risk of performing irresponsible actions which will be used by the attacker.
Make sure your UI helps your user to take the correct decisions.

ff Infrastructure: Firewalls, Network Admin, Host & Server Hardening, Net-

work traffic encryption etc.

ff Application: Authentication, Authorization, Input validation, Encryption,

Configuration management, Parameters manipulation, Auditing, Error
Handling etc.

The application must be designed and implemented while taking security is-
sues into consideration. We have to remember that the attacker needs to find
just one security breach while we have to protect everywhere.

Leaving one of the above levels unhandled will result in a completely

unsecured product.

Application security is not just another feature. You can not just turn it on.

Application security demands a lot of thinking. Threat modeling and a lot of

design work must be done. Many concrete actions must follow in every phase
of the development cycle.

Security Testing:
Testing is a crucial part of Security Development Lifecycle.
The tester must understand methodology of secure development. He has to
build a security test plan using the threat modeling documentation.

The tester has to understand the Hacking mechanics. He has to get out of the
box and think like a hacker.
The tester has to know the security testing methodology. The hacker must be
diligent and work systematically to find security breaches.

All this and more will be taught in the course.

1250,- EUR
(plus VAT)

Please register by
email [email protected]
Pantone 295
Knowledge Transfer Pantone 279
or fax +49 30 74 76 28 99
100% 80% 60% 40% 20% 100% 80% 60% 40% 20%

cmyk cmyk

c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14
News
Improve Quality Services and iSQI launch IREB Requirements Certification in The
Netherlands and Belgium
Requirements engineering is a key discipline in requirements engineering. The members of requirements engineering with regard to deter-
for ensuring project success. The function of the Board are independent, internationally mining, analysing, specifying, documenting,
requirements engineering during the devel- recognized experts in economics, consulting, testing, and managing requirements.
opment process is to determine the system research and science and include Chriss Rupp
requirements, to document them adequately, (chair) and Suzanne Robertson. IREB is a reg- International Software Quality Institute
to test them and to manage them throughout istered non-profit association. (iSQI)
their entire life cycle. The requirements engi-
neering discipline has recently been receiving Improve Quality Services The certificate is awarded to anyone who pass-
much attention. Not at least of course as a re- es the exam which is prepared by the IREB.
sult of all the outsourcing that is taken place. Improve Quality Services is the leading course The certificate is an impartial confirmation of
provider in the areas of test- the level of knowledge the holder has attained
The International Require- ing and quality management during his training in requirements engineer-
ments Engineering Board in The Netherlands and Bel- ing. The international exam institute iSQI is
gium. For a number of year responsible the examination (www.isqi.org).
A number of years ago the Improve Quality Services The iSQI certifies IT personnel all over the
International Requirements has already been running Re- world. It also amongst others runs examina-
Engineering Board has been quirement Engineering and tions for ISTQB and QAMP.
founded (www.certi- Management courses.
fied-re.de) within the They have recently More information
Requirements Engi- extended their services
neering discipline. The to also provide the The IREB Requirements Certification Founda-
stated aim of the Inter- “Certified Profession- tion courses in The Netherlands and Belgium
national Requirements al for Requirements will start this autumn. More information can
Engineering Board Engineering Founda- obtained via the web site of Improve Quality
(IREB) eV is to help tion level” course, Services: www.improveqs.nl, or by sending an
standardise basic and compliant to the IREB email to [email protected].
continuing training in requirements engineer- requirements. The course is intended for any-
ing by providing syllabuses and examinations. one who works with requirements in their pro-
Ultimately, it seeks to improve actual practices fessional lives. It covers the fundamentals of

Stephan Goericke (iSQI) and Erik van Veenendaal (Improve Quality Services) signing the agreement.

www.testingexperience.com The Magazine for Professional Testers 7

 The benefits of modeling in a large-scale test integration project:
A case study

by Graham Bath
© iStockphoto

Abstract makers is generally more complex. properly. But we’ve been around long enough
This case study describes the benefits obtained It’s against this background of “systems of to know that these new developments follow
and the lessons learned from introducing a systems” that model-based approaches was a cycle which starts with euphoria, passes
model-based approach into one of europe’s evaluated and introduced. This article de- through the “valley of dissolusionment” and
largest system integration testing projects, scribes a number of the benefits which were finally reaches the a level of practical usage
which includes off-shore components and a achieved and the strategy which was followed. (which is generally somewhere between those
staff of more than 400. Since the paper is based on industrial experi- two extremes).
An industrial-scale research project with ence, it would come as no suprise to learn that At T-Systems, Test Factory we were acutely
three distinct phases was established to man- there were problems along the way, so I’ll be aware when we set up our study that using a
age the test process modifications resulting sharing some “lessons learned” as well. model based approach for achieving our test-
from adopting a model-based approach and ing objectives could well end up like chasing
to ensure that practical and business factors 2 Deciding on an approach: the next pot of gold. So we need to be care-
remained in focus throughout. In this paper ful and not be too euphoric at the start and we
Somewhere over the rainbow…
the three phases are described in terms of the adopted an iterative approach, where learn-ing
There’s an old saying that if you can find the
concept used for achieving defined objectives, from experience was a central element.
point where a rainbow ends you’ll find a pot
the lessons learned, and the actual benefits ob- Oh, and we needed a realistic objective: “To
of gold. The problem is, if you set off towards
tained. achive synergies and efficiencies by introduc-
your rainbow’s end you’re chasing a moving
The results of the case study show that the ing a model-based approach and integrating
target, which may ultimately disappear before
model-based approach is an effective instru- this into our established standard testing pro-
your very eyes. We never actually reach that
ment for enabling a wide range of improve- cess; from planning through to execution.”
mystical point where the pot of gold is buried.
ments to be achieved. The customer require- In IT we’ve all been there before haven’t we?
ments process has been integrated into the 2.1 The starting point
We’ve chased pots of gold called “tools give
testing process and a seemless, tool-supported The existing test process is based on the de-
full test automation at the press of a button”,
process from requirements through to test scription provided by the Interna-tional Soft-
“mainframe applications will be entirely re-
specification has been achieved. ware Testing Qualifications Board (ISTQB). It
placed by e-commerce” and, more recently,
was not an objective to replace this test pro-
“just send everything offshore; it’s sure to be
1 Introduction: cess, but we certainly wanted to make it op-
more efficient”. We probably all know by now
erationally more efficient. The diagram below
The trend to highly integrated systems that there’s more to it than that. Yes, tools can
illustrates the aspects of the test process which
Monolithic systems are a dying breed. The help automate certain types of testing efficient-
were identified as candidates for efficiency
trend these days is towards system architec- ly. Yes, e-commerce is a major step forward
improvements (note that this diagram does not
tures which are sometimes best described as for user-centric applications and, yes, off-
show all of the steps in the test process).
“systems of systems”. The reasons for this shore can yield substantial benefits if managed
could fill a whole article, but certainly the
benefits of software re-use, the availability
of standard software packages, the flexibility
and scalability offered by heterogenous archi-
tectures and the “mix and match” possibilities
offered by web-based software services (SOA)
are among the principal driving factors. But
there’s a price to be paid for developing archi-
tectures like this; the task of testing and pro-
viding quality-related information to decision

8 The Magazine for Professional Testers www.testingexperience.com

The primary areas for potential improvement were identified as: 3 Phase 1: “Let’s get modeling”
• the customer requirements interface In this first phase we looked carefully at the issue of modeling in gen-
• the analysis and specification of test cases eral and established an overall concept for introducing model-based ap-
• the use of off-shore organizational components proaches to our test process.

2.2 The concept 3.1 The overall concept

To investigate the benefits of using models in testing a research project Central elements of the concept were:
was set up which works closely with both operational projects and the • Use of UML 2.0 to model the test basis. In this step (which we
local university, (in this case the Technical University of Munich). The called “design for test” ) the business requirements provided in
clear intention was to obtain buy-in from the projects right from the the customer’s solution documents were captured as UML2.0 Se-
start, avoid the problems of developing new concepts in “ivory towers” quence Diagrams and Class Diagrams.
and to make use of existing academic expertise in the field of model- • Use of the UML-2 Testing Profile (U2TP) to take over the UML
ing. diagrams and extend them for test purposes. The result is a Test
The project plan we developed had three phases, each with approxi- Model (equivalent to a test specification) which contains informa-
mately six month’s duration. The objectives for each phase were only tion about the System under Test (SUT) and each test case.
set after learning from the previous phase. • Use the Test Model as a basis for test execution. This could be
The three phases we identified are shown in the table below: performed either manually or (potentially) used by tools for the
generation of automatically executable tests.
Project phase Name of phase • For each of these steps the roles and tasks for on-shore and off-
shore staff are clearly identified.
1 “Let’s get modeling“
• Identification of business processes for implemention in UML.
2 “Integrate, integrate“ These needed to be stable, well understood processes and the re-
3 “Automate, where it makes sense“ sponsible test designers needed to be willing to take on the piloting
task in a “live” project.
Details of the project phases will be discussed in later chapters. Before • Training in UML 2.0 provided by the Technical University of Mu-
we started with phase 1 though, it was important to choose a suitable nich. A total of 12 experienced staff were selected to perform the
pilot project. phase 1 modeling
• Selection of the Enterprise Achitect (AE) tool for the UML 2.0
2.3 Choosing a pilot project modeling. The selection decision was based primarily on ease of
The objectives we set ourselves could only be achieved if a representa- use and low license costs.
tive project was used as a pilot. Of course, it’s possible to reduced risks • Modeling of 30 selected business processes as UML 2.0 activity
by using a relatively simple project for piloting, but what value are the diagrams.
results afterwards? Will they scale to the types of large, complex proj- • Modeling of the messaging and communications between indi-
ect we have to test? vidual systems using UML 2.0 sequence diagrams.
What we needed from our pilot project were a number of attributes • Getting feedback from industry: The concept was presented at the
so we could be sure the approach was going to provide benefits when International TTCN-3 conference in Berlin in 2006 and at the Sys-
rolled out. We selected a major testing project as our pilot because it tem Engineering Conference “SE08” in Munich in 2008.
gave us good coverage of these attributes. • The basic concept for phase 1 is shown in the diagram below.

3.2 Strategies for off-shore compared

Required attributes for pilot Characteristics of the chosen
Before we could really settle on the concept outlined above we first
project
needed to consider the issue of off-shoring, which is an essential ele-
Technical complexity: Large Over 70 inteconnected applica- ment of our testing organization. What would be the best way to capture
numbers of applications and in- tions the customer’s requirements as a test specification so that our off-shore
terfaces. teams could use it efficiently and effectively? We looked at three dif-
Business processes complexity: Mostly complex business pro- ferent options:
Several applications communi- cesses running over multiple ap- 1. Text-based: This is the existing baseline. Textual test cases were
cate with each other to imple- plications. Many “end-to-end” generated onsite, translated along with other specification docu-
ment business processes tests are performed. ments and then sent to offshore for execution.
2. Model-based on-shore: Test design and specification both con-
Organizational complexity: Geo- The project is performed at sev-
ducted on-site and then executed off-shore.
graphically distributed organi- eral locations in Germany and in
3. Model-based partial off-shore: Test design conducted on-site.
zation, including off-shore ele- India.
The design is then refined and the test specification created off-
ments
shore. After a review with the on-shore team the test are then ex-
Real business case: There has to With over 400 testing profession- ecuted off-shore.
be a reason for doing this. We’re als involved in the project, any The text-based approach (option 1 above) in which test cases are trans-
not interested conducting a pure- improvements to efficiency will lated from German into English was known to be inefficient (that’s one
ly academic exercise. lead to substantial savings. of the primary reasons for looking at modeling). Here are a few reasons
why:
2.4 The chosen pilot project • Lower proportion of work carried out off-shore.
The fundamental task of the pilot project is to provide functional and • High levels of communication due to queries regarding the test
operational acceptance testing of the en-
tire system stack prior to entering produc-
tion. Regular releases of software are tested
and quality-related information provided on
which production decisions are taken. For
each release the testing organization receives
a number of text-based “solution documents”
from the business owners which are used as
the test basis.

www.testingexperience.com The Magazine for Professional Testers 9

 specifications and incorrectly raised de- lowing problems arose and a number a lessons customer and the testing organization, it would
fect reports. were learned: also help to strengthen our modeling process
• Lower than expected levels of productiv- • The modelling task was actually more and make it more precise compared to our
ity. time-intensive than predicted, despite the earlier approach of using Enterprise Architect
• Insufficient knowledge transfer. training courses and the availability of with modeling guidelines.
• The translations were often a major expert support. We realized quite quickly After discussions with our customer the po-
source of error and off-shore teams often that modelling guidelines are absolutely tential benefits of project phase 2 were clear;
found themselves (incorrectly) interpret- essential, especially regarding the levels if we could integrate the MID-Innovator mod-
ing their meaning. of abstraction required. elling tool into our testing process we would
• Insufficient knowledge transfer. • The use of Enterprise Architect (EA) as have a seemless, model-based interface with
• The off-shore staff were not involved in a modelling tool proved to be a double- our customer. The test organization would
the testing process early on, which low- edged sword. EA is easy to learn and use no longer need to model the customer’s text-
ered motivation. and allows the modeller complete free- based requirements and significant gains in
dom to capture requirements as models. efficiency could be achieved. In addition, the
3.3 Benefits of the model-based ap- This freedom, however, can cause prob- use of this particular tool held the prospect of
proaches lems, even when modelling guidelines are being able to automatically generate test cases
A structured questionnaire was used to support available. First of all, the modeller has to from the models.
the results obtained from the first phase. Even keep referring to those guideleines and,
though this evaluation was primarily subjec- secondly, control mechanisms have to 4.1 Concept for Phase 2
tive in nature, it enabled all those participating be in place to ensure that they are being Introducing a new tool like MID-Innovator
in the innovation project to express their views correctly applied. One particular problem into the testing process triggered a number of
on the benefits obtained. Did they “feel” that area was fixing the level of abstraction re- integration tasks (hence the name given to this
there had been improvements made? Could quired in the model. This started to erode phase!):
they perform their work better than before? some of the efficiency gains. • Integrate test-specific aspects into the re-
Was the effectivenss of the testing now bet- • The levels of defects detected using quirements specification process. To do
ter or worse than before? Questions like these model based test specifications with off- this we would implement a Domain Spe-
were important for giving us the confidence shore testing teams remained relatively cific Language (DSL) so that both cus-
for taking further steps. constant. tomer and test organization could agree
Here are the primary benefits which emerged on a common domain-specific language
from the survey: 3.5 Just a minute: why is the test or- for modelling. The test organization
• Increased efficiency. In some areas, ef- ganization doing this? would assist in creating this DSL so that
ficiencies of over 50% were achieved Phase 1 of the project was initiated in order our standard test process could be taken
compared to a testing approach without to improve the efficiency of our testing orga- into account.
modeling. These efficiencies arose from a nization’s process. One of the principal tasks • Integrate the MID-Innovator tool into our
number of different sources (see below). performed to achieve this was arguably, how- own tool set. This would mean creating a
• Better communications. Perhaps one of ever, not a testing task at all. The modeling of new XML-based interface between MID-
the most significant benefits was in bet- requirements in UML was a necessary step Innovator and our standard test manage-
ter communications within distributed to enable other benefits to be gained for the ment tool.
testing teams. Testing staff in Germany testing organization. In principal it’s a desir- • Integrate the modelling tool into our ex-
and India could communicate via UML able skill for a testing organization to be able isting testing process. This would mean
diagrams instead of pure text. There to model requirements, but we were still left altering a number of steps in the process,
were fewer misunderstandings; fewer at the end of phase 1 with this basic testing including test planning and test specifica-
telephone conferences and a general im- overhead. tion.
provement in the process of off-shore • Integrate the modified test process into
governance. 4 Phase 2: “Integrate, integrate” off-shore components of our organiza-
• Higher motivation. Levels of motivation During phase 1 of the project we became aware tion
for off-shore staff increased because they of similar kinds of project being conducted by • Establish a knowledge management
became far more involved in the test pro- our customer. Their idea was to model their concept which would ensure that best
cess as a whole. requirements so that development organiza- practices and lessons learned from the in-
• Better knowledge capture. The capture tions could use them. For us this was welcome novation project could be shared across
of requirements documents as models re- news; it would be a relatively small conceptual the entire testing organization. To achieve
lieved the problem of “tacit” knowledge step to also include the testing organization in this a Wiki would be developed using the
where particular individuals are in posses- their plans. Confluence product.
sion of key business process know-how, The contours of the next phase in the project • Model the same test scenarios used
which is often not fully documented. began to take shape when we learned that a in phase 1 to allow comparisons to be
• Stimulus for offshore. In general a modelling tool had already been selected by made.
model-based approach proved to be a the customer. The tool, MID-Innovator, would
significant “enabler” for the off-shore not only provided the “bridge” between our The basic concept and objectives for phase 2
testing process. Whilst many papers are summarized in the diagram below:
and presentations have highlighted the
automation-based benefits of a model-
based approach to testing, the benefits we
obtained pointed to significant improve-
ments in other aspects of the software de-
velopment process such as in knowledge
management, team motivation and off-
shore governance.

3.4 Lessons Learned from Phase 1

Not everything worked out absolutely as ex-
pected in this first phase. In particular, the fol-

10 The Magazine for Professional Testers www.testingexperience.com

Note that the option to generate test cases automatically from the MID-Innovator-
based models does not mean that the option to define test cases manually, using
UML (as in phase 1) or indeed with any other valuable approach is eliminated.

4.2 Benefits and Objectives of Phase 2

At the time of writing this paper the second phase of the innovation project was
under way. The benefits we expect from this phase include:
• Efficiency gains resulting from the use of a common DSL
• Efficiency gains resulting from the use of a common modelling tool
• Fewer defects found as a result of requirements problems
• More efficient generation and administration of test cases
• Easier monitoring and verification of achieved test coverage levels
• Better sharing of knowledge across the testing organization

In addition to the benefits noted above, we want to provide answers to a number of

questions:
• How easy is it to create a common DSL and what are the primary areas to pay
particular attention to.
• What proportion of the lessons learned are applicable specifically to the pilot
project itself and what can be defined generically so that other projects can
benefit?
• Will the modelling tool be able to generate a high-value set of test cases based

Biography
on risk, or will expected efficiencies be eroded by an “explosion” of automati-
cally generated test cases.
• Does our modified testing process handle requirements changes efficiently?
• How does this approach compare to the results of phase 1, where models were Graham’s experience in testing spans
created in UML using Enterprise Architect. over 25 years and has covered a wide
• Is our customer satisfied with the seamless model-based approach? range of domains and technologies. As a
The results of phase 2 will be documented as a business case which will enable test manager, he has been responsible
management to decide on a general roll-out and the objectives of phase 3. for the testing of mission-critical systems
in spaceflight, telecommunications and
5 Phase 3: “Automate where it makes sense” police incident-control. Graham has de-
Assuming that we are satisfied with the results from phase 2, the next phase planned signed tests to the highest level of rigor
will look at automation. As the name for this phase suggests, we will be performing within real-time aeropsace systems such
evaluations of automation concepts which integrate to our testing process. as the Tornado and Eurofighter military
Modular test automation concepts will be evaluated. These may be based on key- aircraft. As a principal consultant for the
words defined in the DSL “meta-model”, or may make use of existing concepts. T-Systems Test Factory he has mastered
Where a clear business case can be made for automation this will be implemented, the Quality Improvement Programs
potentially using our off-shore organization. of several major German companies,
If we have been able to a well-defined DSL in phase 2, the opportunity may also primarily in the financial and government
exist for using tool-support to generate automatically executable test cases. There sectors. In his current position, Graham
are already some products available to do this, but for the moment we’d like to let is responsible for the training pro-
those products mature a little. gramme and for introducing innovative
One of the principal objectives to be answered in this phase will be to define model- testing solutions to his company’s large
ing frameworks at a high enough level of abstraction to make the benefits generally staff of testing professionals.
available in other projects and for other customers.
Graham is a member of the authoring
team for the new ISTQB Advanced Level
6 Summary and conclusion
Certified Tester and is a long-standing
We’re experienced enough to know that “magic solutions” rarely become reality
member the German Testing Board,
and that the day is unlikely to come where we can represent requirements in model
where he chairs the Advanced Level and
form, press button “B” and then watch all the tests run automatically. Recalling
Expert Level Working Parties.
the rainbow anecdote at the start of this paper, we’re a long way off from this par-
ticular “pot of gold”. However, we have shown in this industrial scale pilot project
Together with Judy McKay, Graham has
that some major benefits can be obtained from modeling in a complex large-scale
co-authored the recently published book
integration project.
“The Software Test Engineer’s Hand-
Here are the main benefits in summary.
book”, which is a study guide for the
ISTQB Test Analyst and Technical Test
• Modeling enables the benefits of off-shoring to be realized.
Analyst Advanced Level certificates.
• Models can build bridges between customer and test organization, especially
if Domain Specific Languages and common modeling tools can be utilized.
Major efficiency gains can be achieved as a result.
• Modeling is an effective instrument in achieving effective knowledge manage-
ment.
• Modeling can help introduce rigour into testing, especially if tools can be
properly integrated into the test process.

I look forward to informing you of progress. Who knows, we may find the occa-
sional “pot of gold” along the way!

www.testingexperience.com The Magazine for Professional Testers 11

 SW quality improvements by using quality KPIs
by Dr. Hartwig Schwier
© iStockphoto

1. Introduction In addition, all quality product indicators con- across business segments was derived from
Océ supplies digital printing systems, software tribute to a global Q-indicator, which repre- the number of installations (software licenses
and services for the production, reproduction, sents the overall product quality of R&D Ap- reports) and the impact on the overall business.
distribution and management of documents in plication SW. The product weights were set as 1, 3, 5 and 9.
small and wide format for professional users A weight of “one” is meant as “nice to have
in offices, educational institutions, industry, The defined quality indicators reflect the actual / limited importance” and a weight of “nine”
advertising and the graphics arts market. “quality in use” of the software products from stands for “business crucial”.
a customer point of view. Besides the number
To make the print workflow faster and more of defects and change requests, trends, priori- 2.1.2. Standard SW quality metrics
efficient R&D Application Software within ties or backlogs of the open field reports are Despite the fact that the usage of metrics and
Océ develops and maintains modular docu- also considered. KPIs is highly recommended, real practical
ment workflow software. These products are examples for software KPIs and their applica-
commercial-off-the-shelf (COTS) software 2. Definitions tion can seldom be found in literature. There-
products and licensed to customers. The base fore, we analyzed state-of-the art metrics defi-
products can be configured by options and ad- 2.1.1.1. Problem reports nitions (see references listed in the appendix),
ditional components can be added to support As the quality indicators should reflect the ac- selected the most promising for our purposes
integration with system and work processes tual “quality in use” of the software products and tailored them for the Océ R&D Applica-
already in place across the full range of print- from a customer perspective, the evaluation of tion Software processes to cover all relevant
ing environments. the metrics is based on a corporate-wide defect quality aspects, e.g. number of problems,
database, which contains all customer and ser- trends, severity of problems and time-to-solve
In 6 R&D sites worldwide approx. 220 engi- vice records worldwide of released products. (backlog). Product-specific parameters allow
neers develop and test about 10 product lines Each database record includes priorities, time making the definitions applicable to the vari-
with independent life cycles. stamps, descriptions, responsibilities and the ety of all 10 products groups.
actual status for the problem solving process.
At the beginning of 2007 software quality key The relative weight for each metric was de-
performance indicators (KPI) on product level 2.1.1.2. Product and product groups rived systematically from experience and ob-
were introduced As problems are reported in relation to prod- servations, e.g. starting from initial statements
ucts, options, components or configurations, it like “too many open problems” or “response
• to report the quality of the software prod- is necessary to define exactly what is under- time to solve is too long”.
ucts stood by product or product group.
• to initiate software product quality im- In addition, to define a global quality indicator, In table 1 the defined metrics, the description
provement activities the relative importance for each product line and the relative weight are summarized.

Metric Weight Description

TCP 9 Trend of Customer Problems measures the deviation from reference level. Defined as unilateral capability index
and displayed as pseudo control chart.
PCI 9 Problems per Customer Installation
ASCP 3 Average Severity of Customer Problems
Backlog 5 Backlog of open problems (response-time-to-solve)
CRI 1 Change Requests per Customer Installations
NCR 0 Number of Change Requests (normalized)
CRISIS 0 Number of Crises (absolute measure)
Table 1: Defined metrics, their description and the their relative weight

12 The Magazine for Professional Testers www.testingexperience.com

2.1.3. Definition of Quality indicator
The q-indicator for a product as well as the
global quality indicator are defined either as
weighted sum of the metrics or as weighted
sum of the product q-indicators.

For the product q-indicator the definition is as

follows

Figure 2: Graphical presentation of the Global Q-Indicator (27.48, green) of December 2007 as a pie-chart.
and the global Q-indicator as
The Q-indicator is reported monthly
as red (poor), yellow (acceptable) and Q-Indicator Dec-06 to Dec-07
green (good), but also in more detail in
a percent range from 0 (very good) to 120
100 (very poor). In Figure 2 the global
Q-KPI (27.48, green) of December 2007 100
is shown as a pie-chart. The color of each
pie displays the status of a product group 80
and the size of each pie represents the

Q-KPI %
The interpretation of the quality indicators is impact on the global Q-indicator respec- 60
identical and straight forward (see table 2 for tively.
details) 40
3. Findings and Improvement Re-
Value ranges Interpretation and sults 20
color assignment
0
0 ≤ m, q, Q ≤ 30 Quality level is good 3.1. Findings Dec-06

Jan-07

Feb-07

Mar-07

Apr-07

May-07

Jun-07

Jul-07

Aug-07

Sep-07
Oct-07

Nov-07

Dec-07
30 < m, q, Q ≤ 60 Quality level is ac- After the introduction at the beginning of
ceptable, but to be 2007, a detailed analysis showed that the
Months
improved number of problems was too high, that
there was a strong deviation from the Q-Indicator 30% Level 60% Level 100% Level
60 < m, q, Q ≤ 100 Quality level poor (to Figure 3: Achieved improvements of the overall Q-Indicator in
reference values, that the average prob- the year 2007.
be corrected) lem priority was too high and the prob-
lem fixing time too long. The following 3.3. Product quality indicator
Table 2: Interpretation of the metrics and quality Figures 4, 5 and 6 show the significant im-
root causes were identified:
indicators
provements over the period of about 12 months
• Inconsistencies in the problem database for one product.
2.1.4. Reporting and in the problem handling process, e.g. The green polygon in Figure 4 describing the
The indicators are reported on a monthly ba- no systematic record handling or limited quality in 2007 is almost completely included
sis. Figure 1 shows the typical cover page of attention on record maintenance. in the 30% level (yellow line). In addition, the
such a report, which summarizes all important • In the planning of product releases, the area defined by the green polygon is only a
findings per product in a radar diagram. The attention was more directed on new fraction of the area defined by the red poly-
smaller the size of the area defined by the blue functionality than on product quality im- gon (December 2007) indicating the degree of
product line, the higher the quality of the prod- provements. quality improvement.
uct.
The status of a product group is displayed also The agreed actions were straight forward: The reduction of number of problem records
in the pie-chart; the size of the pie represents • Purge the database with the support of of about 66% could be achieved as a combined
the impact on the overall Q-Indicator. service and development. action of four (!) bug-fix releases and addition-
• Initiate and define product qual- al maintenance activities of purging the prob-
ity improvements explicitly either as lem database (see figure 5)
part of the regular functional enhance-
ments or as separate releases. The columns in Figure 6 represent the number
of open problems per priority. Besides a signif-
3.2. Improvement results icant reduction in open problems (as discussed
before) the distribution of the data of Decem-
3.2.1. Global Q-indicator ber 2007 in green can obviously be fitted as a
By December 2007, the overall Q- Gaussian distribution with a mean value close
indicator had improved by about to priority 3. From a customer point of view,
40% from 45 (yellow) to 27 (close this means to our experience a high product
to green). Figure 1 shows the result quality in functionality, performance and us-
of December 2007 as a pie-chart and ability.
Figure 3 displays the improvements
achieved in 12 months.
Figure 1: Product quality KPI displayed as radar diagram

www.testingexperience.com The Magazine for Professional Testers 13

 Product 2: Radar chart (improvements)

TCP
100

80

60

40
CRI PCI
December-06
20
December-07
0 30% Level
60% level
100% level

ASCP Backlog

Figure 4: Radar chart indicating the improvements

Product 2: Pseudo control chart for trend analysis

120

100
Biography
Open Problems (normalized)

80 Product 2
Reference
Hartwig Schwier is head of corporate
60 30% Line software test and validation at Océ. He
60% Line has been working in software engineer-
40 Trend line ing for more than 20 years covering
software development, architecture and
20
quality assurance. In his recent position
0
he is responsible for software testing at
12/31/06 04/30/06 08/31/07 12/31/07 six R&D sites worldwide and for the time
Months being he is also general manager of Océ
Figure 5: Decrease in the number of problems for one major product. Software Romania in Timisoara.

Product 2: Open Problems December 2007 versus December 2006

100
Number of problems

80

60 December-07
40 December-06

20

0
1 2 3 4

Priority

Figure 6: Open Problems and average problem severity

4. Summary
During 2007 software quality indicators were introduced in a systematic way. The
defined KPIs describe several aspects of product quality, e.g. the number of prob-
lems, the severity of problems and the problem handling. It was proven that qual-
ity indicators can be used for reporting as well as for initiating and controlling
improvement activities.
To be successful a systematic approach, regular reporting and the support from all
departments involved is needed.

5. Appendix: Literature
The introduced metrics are based on state-of-the art definitions and adapted to the
available data and defined processes at R&D Application Software.

The following sources were used:

• Kan, H. - Metrics and models in software quality engineering

2nd edition, Addison Wesley, 2003
• Galin, Daniel - Software Quality Assurance
Addison Wesley, 2004
• Craig, Rick D; Jaskiel, Stefan P. - Systematic Software Testing
Artech House, 2002

14 The Magazine for Professional Testers www.testingexperience.com

 Are test design techniques by Hans Schaefer

useful or not?

© iStockphoto

Many people test, but few people use the well- some input is ”special” or wrong, it fails. For ues into sets that you think are equivalent. The
known black-box and white-box test design consumer software, when confronted with fail- idea of equivalence means that you think the
techniques. The technique most used, how- ures, the user will often ask, ”How could any- program will handle equivalent values in, prin-
ever, seems to be testing randomly chosen one have forgotten to test this obvious thing?” cipally, the same way. YOU think! This means
valid values, followed by error guessing, ex- someone else may find a different partition. It
ploratory testing and the like. Could it be that Examples: depends a bit on viewpoint and experience.
the more systematic test design techniques are -- The input should be numeric: Test one Then you select one value per equivalence
not worth using? typical correct value. For the incorrect class for testing. IN PRINCIPLE, handling the
one, choose a value which is too high input in the same way means that you can as-
I do not think so. When safety-critical soft- (extremely high for example) or a nega- sume the program is executing the same lines
ware is produced, it is tested very systemati- tive number. Zero is a typical destructive of code. For example, for a bank transfer, it
cally using the good well-known techniques. value, often difficult to handle for a pro- should not matter if the amount is 100, 140,
Standards recommend or require doing so. gram. Trying a non-numeric input is also 145, 150, or something like that. Or for check-
Therefore there must be some value. What a valuable choice. ing validity of a date input, at least every day
kind of value? -- The input is text: Try a usual text with between 2 and 27 should work the same way.
usual length as correct. Try special char-
The most important one is representativeness: acters, a missing text (no input) or a text A different use of this technique is backwards:
When using a systematic technique, we test which is too long as wrong possibilities. When I review test plans, I look at the values
representative and special values. We test for -- A group of allowed values: Try the most planned to be used in the test. Then I try to
two reasons: To find as many problems as pos- common one as correct, and something figure out what equivalence classes might be
sible, and to assess the quality of the product, else that is wrong. For the wrong one you behind them, which other equivalence classes
i.e. we test with destructive values and with could choose a value that is ”near correct” there may be, and then I search for the other
representative values. or that was correct in an earlier version or test values covering these other equivalence
is correct in other software. classes (and often find ”holes in the test”).
The other much-needed quality of testing is -- In any case: Try to leave out inputs, and
accountability: We need to be able to tell our try repetitions. Here are the main rules:
clients what we have tested, what not and why. -- If an input is numeric, choose a value that
Or what we tested more or less. The value of this technique: You will at least is too small (wrong); one that is too high
get some impression of how error handling (wrong) and one that is correct (right).
When the software works or fails for one works with your software. Error handling is -- If an input is logical, then test both true
value, we assume, very often rightly so, that often badly designed and implemented, and and false.
it will do the same for other values. The ba- also badly tested by programmers. Thus the -- If an input is a time value, do as with nu-
sis for this is called equivalence partitioning. wrong values have good value when it comes meric values, but include a fourth class:
The benefit: We do not unnecessarily duplicate to discovering trouble. impossible values (like 35th of Decem-
testing effort. ber).
As you see, this technique leaves many choic- -- For discrete inputs, test every allowed
But how can we do it? es, e.g. which of the discrete allowed values value plus one wrong.
you should choose. Or if you should choose -- Test every input in two ways: given and
Let me introduce a few methods, to make it a numerical value that is too high or too low. not given.
easier. Thus we have another method: -- Try correct and wrong data type for in-
puts. (Example: texts in a number field,
First method: Test one right and one Second method: Equivalence partition- Chinese characters for an ASCII text field
etc.)
wrong value ing
-- If an input can be repeated, try zero, one,
In many cases software works relatively well Somehow, this is ”the mother of all testing”.
more than one repetitions.
if your inputs are correct or ”mainstream”. If The idea is to partition any set of possible val-
-- The pessimist rule: If you think some-

www.testingexperience.com The Magazine for Professional Testers 15

 thing might not be equivalent, then par- Every new variable will duplicate this table! 1. The test reaches every (reachable) state
tition any such equivalence class into 2. The test passes through every (reachable)
subclasses. (This creates a more thorough One way to cut down on this is testing only transition
test, but also more testing work). Real lower-level combinations: Observation has 3. The test should cover every possible
pessimists would test every value, but shown that pairwise combinations detect most event in every state
then they would never finish testing. errors. Less errors by combinations of three 4. The test should cover combinations of
values, and so on. There are many tools sup- transitions after each other.
If you look at other systematic testing tech- porting pairwise combinations, where all pairs
niques, many of them are based on equiva- of values from different parameters are gener- All these test design methods are easy to ap-
lence classes. ated automatically. The technique is popular, ply if the state machine is defined in a formal,
This technique is easy to learn and use, and it as it saves a tremendous amount of work. machine-readable way, and if a test design tool
saves a lot of duplicated test effort. You test is applied. Actually, most of the model-based
one value per equivalence class only. Combi- As an example, here is the table of all pairwise testing is about testing state machines.
nations of equivalence classes may be tested as combinations of three logical inputs:
a more advanced technique. For most inputs, State machine testing tests only the state ma-
if the data type and the conditions are given, Test case Input 1 Input 2 Input 3 chine logic in itself. If data can vary, they must
equivalence partitioning is easy: It is the same number be tested using other techniques, like equiva-
over and over again. In principle, just use the lence partitioning (boundary value analysis
checklist above! 1 Y Y Y (see later) or logical combination testing).
There are tools, which support the technique. 2 Y N N
CASEMAKER is one of them. 3 N N Y The first method is not very valuable. The
main value is in finding states that are not
The trouble with this technique: 4 N Y N reachable, even if they should be, or require-
-- Logical combinations are not very well ment problems (misunderstandings, unclear
covered. However, pairwise combination testing may requirements etc.). From the start state or from
-- The technique is blind for trouble when be dangerous, as it may overlook errors. This initialization, the state machine is fed with in-
the implementation of the program is is especially true if there are implicit depen- puts in order to get it to assume every state. If
done in a completely different way than dencies. An example of such dependencies some state seems unreachable, the problem is
you think it may be. is the validity checking of a date input: The researched.
-- The technique may be too complicated number of days in a month is dependent on the
when experienced people use the pessi- month and even the year. However, if there are The second technique has a value: It checks if
mist rule a lot. no known dependencies, then pairwise testing the state machine “works”, i.e. if every defined
-- Some values may find more problems or is a good first choice. It may even be applied if transition gives a reasonable result. An exam-
may be more representative than other some values are not logical, but discrete. ple is the answer to the question what should
values. happen if someone presses this or that button.
Example for general combination testing with The test design starts again from the start state,
Therefore we may need more. This is de- discrete values: and now in every state, every defined transi-
scribed in the remainder of this paper. There One input is the month, another one is a color. tion is tested. Typically, these are the “valid”
are several techniques. The month falls into three categories: 31 days, transitions. An example would be to switch on
30 days, and February. your mobile phone, and then test everything
The colors may be red, blue, black, and white. there is in the user manual that has to do with
Third method: Logical combination test- All combinations of these two inputs are 12. state (e.g. the menu selection).
ing
Logical combinations are dangerous. In many A more difficult technique is cause-effect The third technique will find that not every
cases there is unclear thinking: Some com- graphing. This technique is only useful if sup- possible event has a defined reaction for every
binations are forgotten. Distributed logic, ported by a tool. Otherwise it is too difficult to state. The technique described above, testing
implemented in different places, may have apply. The analysis may introduce more prob- one right and one wrong, is an example of this:
holes or side effects. The conclusion is: Logi- lems than are found. The “wrong” events are often forgotten. How-
cal combinations must be tested. The problem:
ever, if there is a “water-tight” specification,
Combinatorial explosion. The number of tests There are more techniques available for logi- testing in every state has probably been done
would grow by the power of two per new com- cal combination testing, but these would be already. To continue the mobile phone exam-
bination. This may be OK for small problems, too complicated for a short article. ple, this technique would require trying every
but if the number of parameters exceeds about
button in every state of the phone (considering
five, this tends to generate too much work.
Fourth method: State-transition testing only buttons as inputs here).
Many programs are defined as state-transition
As an example, here is the table of all pairwise
diagrams, state charts, or state machines. The The last technique is difficult and possibly
combinations of three logical inputs:
technique is popular and supported by UML. A leads to indefinite amounts of work: One may
corresponding testing technique exists. It was test two, three, n events after each other, start-
Test case Input 1 Input 2 Input 3 defined back in the 1960s and 1970s. ing from each and every state. This kind of
number testing finds “long-term corruption”, i.e. the
1 N N N Examples for state machines are: state machine somehow depends on several
-- A mobile phone. It may be on, off, may events after each other, and on the order of
2 N N Y
be a phone, a camera, a database, and these events. Even hidden extra states imple-
3 N Y N SMS machine etc. mented may be found. But the amount of work
4 N Y Y -- A traffic light: The state is often visible as for this is probably too much. This technique,
the displayed light. if applied at all, requires tools.
5 Y N N
-- Most home appliances. Buttons introduce
6 Y N Y state transitions. However, a “trick” may be applied: When test-
7 Y Y N -- The status accounting in a defect report- ing state machines, one test should be chained
ing and tracking system. after the other, with no restart in between. This
8 Y Y Y
These are the different coverage criteria: will at least cover some of the combinations

16 The Magazine for Professional Testers www.testingexperience.com

 h e d ate!
Save t

TESTING DAYS GERMANY 2008

Compuware and Diaz & Hilterscheid invite you to attend a free event series for
testing professionals in autumn 2008:
•26th September · Duesseldorf • 29th September · Frankfurt
•30th September · Munich • 1st October · Hamburg
Presentations include a mini tutorial on test design by „Mr. Test“ Hans Schäfer,
case studies and much more. Events will be followed by a get together
for testing professionals to exchange experiences.

More details on www.compuware.de/testingdays

and thus be more destructive. Here is a list of (implicit) boundaries to check Error guessing or fault attack
up: This method concentrates on typical problems
In the mobile phone example, we may try to Buffer sizes with the program. There is an assumption that
press buttons one after each other, in all pos- Table sizes the tester has some idea about what typical
sible different orders. (People who tried this First and>> last elements in buffers, ta- problems are. Examples include:
sometimes had to take back their phones for bles, files etc. • Boundary values (covered by boundary
service). With these mobile phone examples, Memory boundaries value analysis)
I ignore that the phone has some more states, Allowed number of repetitions • National and special characters for text
depending on incoming calls etc. Thus, in re- Message sizes values
ality, the test is even more complicated than Lengths of output fields • Missing and duplicated input
outlined here. File lengths • Too much input
List lengths • Low memory
Fifth method: Boundary value and do- Transition in time (over the hour, day, • Unavailable resources
year) • And more.
main analysis
Timeouts
Some values are better than others in finding
A tester should always keep a log about typical
trouble. Boundary values have this property.
This kind of testing requires programming ex- problems. Checking the existing defect log is
This is because people have problems express-
perience. You need to know what is going on a good idea.
ing boundaries correctly, and because pro-
behind the scene.
grammers tend to use the wrong comparison
Error guessing is the oldest test method. Over
operator (less or equal instead of less, for ex-
An example boundary problem in PowerPoint time, much of it has been systematized and has
ample). Another problem is counting: Bound-
for Mac, WORD ART: flown into the other methods described here.
ary value analysis is good against off-by-one
But a tester should always put in “the little ex-
errors.
-- Fill in characters until the maximum (just tra” for concentrating on typical problems.
hold down some key!).
Boundary value analysis can very easily be
-- When the maximum (about 128) has been White-box testing
combined with equivalence partitioning.
reached, no more characters can be input, This method is typically used for unit or com-
When choosing values in an equivalence class,
but no error message is displayed. Key- ponent testing where there is access to the pro-
the boundaries are prioritized.
board input is just ignored. gram code. The tester tries to cover all code.
There are many different coverage criteria, but
The technique asks us to test two or three val-
What is missing? an absolute minimum should be to cover most
ues near the boundary: The boundary itself and
There are many more techniques. Some of the of the branches in the code. This will assure
one or two values as near as possible either
most useful or often discussed ones are these: that most of the statements or lines of code are
side of the boundary. I used to test all three
- Random and statistical testing also executed. The question is: How much do
of them: Below and above the boundary as
- Exhaustive testing you know about a line of code that is never
well as the boundary value itself. This requires
- Error guessing executed?
less thinking and is safer. I call this ”the Fri-
day afternoon method”. If you have less time - White-box testing
- The test oracle The test oracle
and experienced people, you may just test two
Testing requires you to interpret the system’s
values: The boundary and another one nearby
Random and statistical testing reaction to the inputs. You compare the output
which shall be in the other equivalence class.
Software is digital. It may fail for any value. with the expected output. But who tells you
This requires more thinking. I call this the
Equivalence class partitioning is a black- box the expected output?
”Monday through Thursday method”.
technique. It tends to overlook things imple-
mented in the program unknown from the The test oracle!
When it comes to boundaries in several di-
mensions, like the border of a window, you outside. For example “Easter eggs”, special
conditions triggering very special behavior. It is defined as a mechanism that decides if
test each linear boundary with three values:
This means, in principle, that any value could a result is right or wrong. In principle, you
Two (on n, for n-dimensional problems) on the
be tested and has some chance of discovering look into the specification. But often, results
boundary, one inside or outside, in the other
trouble. On the other hand, a randomly se- are complicated and interpretation is difficult.
equivalence class. Domain testing is interest-
lected test normally has a low defect-finding It would be nice to have an automatic oracle.
ing when boundaries are not geometrically
ability. It only works if there are very many This would be another program, but it could
vertical or horizontal. Then you save a lot of
tests. Statistical testing tries to improve ran- be a spreadsheet implementation for the math-
combinatorial testing. However, for three or
dom testing by concentrating on values, which ematics, or an older version of the program. If
four dimensions, the technique soon becomes
will occur more often in practice. However, in this is not available, you may test with “easy
difficult.
order to apply it, the tester must design a usage values”, where manual calculation is easy. Or
profile for the application under test. This may you might deviate from full checking and just
Boundary value analysis should not only be
be a huge endeavor or impossible. do a plausibility check. In the worst case, the
applied to user inputs. There are many other
Anyway: If anything can easily be tested by only thing you do is check that the program
(more implicit) inputs, and they should be test-
generating random inputs, and checking the does not crash.
ed. James Bach, a testing guru, defines a few
more rules. Boundary value analysis should output can also be automated, then this method
also be used to is promising. Summary
Test design techniques are valuable. You have
-- question the validity and interpretation of Exhaustive testing to design your test cases anyway, so why not
requirements, This is the extreme case: Every possible com- do it systematically. The benefit is: If some-
-- discover (implicit) boundaries the pro- bination of input values is tested. In principle, body asks how you did it, you are able to de-
grammers overlooked, this should find all problems. In practice apl- scribe it, plus your reasoning behind it. Your
-- learn about boundaries that emerge from lying this method is not possible. There are test will be accountable. And you may be able
the interactions among sub-systems, plainly too many possibilities. to improve over time.
-- discover the absence of boundaries where
such absence creates the opportunity for To randomly fire some shots at the program is
performance and reliability problems. not testing!

18 The Magazine for Professional Testers www.testingexperience.com

 Biography
54 years old. Specialist in software testing
1979 M. Eng.from Technical University of Braunschweig, Germany. Computer science,
railway signalling
1979 - 1981: Development of real time process control software at the Fraunhofer-
Institute in Karlsruhe, Germany.
1981 - 1987: Work with Center for Industrial Research in Oslo, Norway. Development
of parts of an IDE-tool (Systemator). Later developing test tools and leader of quality
improvement program. Consulting and lecturing about software quality.
1987 - 2007: Independent consultant in software testing and testing improvement
matters. Guest lectures at several universities in Norway about Quality Assurance
and Software Testing. Public and in-house seminars in Software Review and Testing
in Scandinavian and European countries. Regularly speaking at conferences. Sev-
eral best paper awards. (Best presentation award at CONQUEST 2003, best paper in
2004).
1998 - 1999 Test coordinator in Norwegian Telecom’s Y2K project.
1999 ISEB Certified Tester (Foundation Level)
2004 ISTQB Certified Tester (Foundation and Advanced Levels)
2003 - 2007 Chairman of Norwegian Testing Board
1992 - 2007 Active participant in running museum trains in Norway, certified for safety
critical services on steam locomotives.

ASTQB
Advantages no other
software tester certification can match
WHY SHOULD YOU AND YOUR COMPANY CHOOSE
ISTQB Certification in the U.S.A.

ISTQB SOFTWARE TESTER CERTIFICATION?

IT’S PRACTICAL, SO IT COMPLEMENTS THE WAY YOU WORK. Theory is
important, but ISTQB certification is designed for your everyday
software testing challenges.

IT’S GLOBAL, SO IT CAN GROW WITH YOU. Developed by more than 100
global software testing experts, and offered in more than 30 countries,
ISTQB certification has more than 80,000 certified testers worldwide.

IT’S CREDIBLE, SO OTHERS WILL RESPECT YOUR CHOICE. ISTQB offers its
certification syllabi and testing glossary at no cost, and lets you choose
your own education.

IT’S TRUSTED, SO YOU’LL FEEL GOOD ABOUT YOUR CHOICE BOTH NOW AND
IN THE FUTURE. ISTQB is the world’s only not-for-profit organization
dedicated solely to providing practical, globally-accepted software tester
certification in more than 30 countries.

LEARN MORE ABOUT ISTQB’S ADVANTAGES FOR YOU

AND YOUR COMPANY AT WWW.ASTQB.ORG, THE U.S.
BOARD FOR ISTQB.

www.testingexperience.com The Magazine for Professional Testers 19

We Change. Your Chance.

Noch Fragen?
Sie erreichen
uns unter:
+49 (0) 711/

Egal, wo Sie herkommen:

972- 80000

Wichtig ist, wo Sie hinwollen.

Die Deutsche Telekom ist in Bewegung: Gestalten Sie den Wandel der vernetzten Welt mit – in
einem der führenden IT- und Telekommunikations-Unternehmen. Bundesweit suchen wir Senior
Testmanager/Projektleiter und Senior Consultants, die unsere anspruchsvollen Themen und
Projekte mitgestalten. Unser Wandel ist Ihre Chance!

Arbeiten bei der Deutschen Telekom Senior Consultant Test Factory (m/w)
‚ T-Systems ist die Geschäftskundensparte der Deutschen Telekom
ID 2008-6750
und bietet hochwertige Dienstleistungen und integrierte Lösungen
für Informations- und Kommunikationstechnik (ICT) aus einer Hand.
Das erwartet Sie bei uns
‚ Testconsulting für internationale Projekte
Senior Testmanager/Projektleiter- ‚ Erarbeitung geeigneter Strategien und Konzepte zur Optimierung
des bestehenden Testprozesses und Beratung zur Aufbau- und
Test Factory (m/w) ID 2008-3615 Ablauforganisation von Testprojekten
‚ Unterstützung unseres Fachvertriebes bei Presales-Aktivitäten
Das erwartet Sie bei uns
‚ Verantwortung für internationale Testprojekte Das erwarten wir von Ihnen
‚ Leitung unserer Testteams (onsite/offshore) sowie Steuerung ‚ Branchen-Know-how aus einem unserer Zielmärkte (Public,
der Testplanung, -koordination und -durchführung Telekommunikation, Automotive)
‚ Fachliche Personalführung der Projektmitarbeiter ‚ Erfahrung im Testconsulting oder Testmanagement
‚ Durchführung des Projektcontrollings ‚ Kenntnis und praktische Anwendung von Standards und Methoden
‚ Unterstützung unseres Fachvertriebes bei Presales-Aktivitäten, wie z.B. TPI®, CMMI, ISTQB, ISO 9126
wie z.B. fachliche und inhaltliche Angebotserstellung ‚ Kommunikationsfähigkeit und Kundenorientierung
‚ Verhandlungssichere Englischkenntnisse und eine hohe
Das erwarten wir von Ihnen Reisebereitschaft
‚ Mehrjährige Erfahrung im Management von Testprojekten
‚ Kenntnis der Testmethodik und Testtools sowie QS-Verfahren Wir freuen uns auf Ihre Online-Bewerbung mit Angabe der Stellen-ID
‚ Kommunikationsfähigkeit, hohe Kundenorientierung unter www.telekom.com/your-chance
‚ Gutes Verhandlungsgeschick
‚ Verhandlungssichere Englischkenntnisse und eine hohe Nähere Informationen zur
Reisebereitschaft T-Systems Enterprise Services GmbH unter www.t-systems.de
 Test Process Improvement using TMMi
by Erik van Veenendaal, Rob Hendriks, Jurian van de Laar and Bart Bouwers
© iStockphoto

Introduction Overview of the TMMi maturity levels Each maturity level contains a comprehensible
More and more organisations make efforts to Just like the CMMI staged representation, the set of process areas. The process areas for ma-
improve their software development process- TMMi has a staged architecture for process turity level 2 are shown in figure 1. Experience
es. The reference model that is most often used improvement. It contains levels that an organi- has shown that organisations are most suc-
is the Capability Maturity Model Integration sation passes through as its testing process cessful when they focus their test process im-
(CMMI). In this model practices that are relat- evolves from one with an ad-hoc and unman- provement efforts on a manageable number of
ed to verification and validation activities are aged nature to a mature and controlled process process areas at a time. Because each maturity
described, but the level of detail is too limited with defect prevention as its main objective. level forms a necessary foundation for the next
from the viewpoint of the test professional. To Achieving each level ensures that adequate level, the decision to skip a maturity level is
fill this gap, the Test Maturity Model Integra- improvements have been made as a foundation usually counterproductive. On the other hand,
tion (TMMi) has been developed by the TMMi for the next stage. The internal structure of the test process improvement efforts should focus
Foundation [www.tmmifoundation.org], using TMMi contains testing practices that can be on the needs of the organisation in the context
the TMM framework as developed by the Illi- learned and applied systematically to support of its business environment.
nois Institute of Technology as one of its major quality improvement in incremental steps.
sources. The TMMi provides a structured ap- There are five levels in the TMMi that define a (5) Optimisation
proach for test process improvement. Testing maturity hierarchy and an evolutionary path to (process optimisation,
as defined in the TMMi is applied in its broad- test process improvement. defect prevention and
control)
est sense to encompass all software quality-re-
lated activities. Within Europe the number of
companies using the TMMi is increasing.
Practical experiences are positive and show
(4) Management and
that the TMMi and its predecessor TMM sup- Measurement
port the process of establishing a more effec- (measuring, evaluation
tive and efficient test process. Testing becomes and advanced reviews)
a profession and a fully integrated part of the
software development life cycle. Applying the
TMMi maturity criteria will improve the test-
ing process and has a positive impact on prod- (3) Integration
uct quality, test engineering productivity, and (organisation, training,
test execution lead-time. life cycle, non-functional
In this paper we give our recommendations testing, peer reviews and
life cycle)
on how to organize and execute a successful
test improvement project, using the TMMi as
a reference. We’ve gathered the practical do’s,
(2) Managed
don’ts and examples that we consider to be the
Test Policy and Strategy
most important success factors, based on our Test Planning
own experience. For a complete description of Test Monitoring and
the TMMi and its guidelines we refer to the Control
web pages of the TMMi Foundation. Test Design and Execution
Test Evironment
(1) Initial
Figure 1: TMMi maturity levels and process areas level 2

www.testingexperience.com The Magazine for Professional Testers 21

It is possible that a process area at a higher ma- achieving a higher maturity in testing. In the ity for an improvement project with the role
turity level may address the current needs of end, a mature testing process should contrib- of test manager, especially when operational
an organisation or project. ute to the business goals of the organisation. issues constantly demand attention. We also
Predictability, higher productivity, efficiency experienced the benefits of working in pairs:
The TMMi improvement project and effectiveness are examples of goals that the external consultant teaming with the in-
The TMMi is not just a theoretical model but an organisation may want to achieve with test ternal test manager. We discovered that with a
a well-defined set of process areas and goals improvement. The organisation itself will have good balance between process and project, the
based on practical case studies. Although the to determine exactly which goals to strive for. improvements were immediately embedded in
model is quite easy to understand, implement- For one of our customers we have developed the organisation. Inevitably the external con-
ing it in an organisation and thereby improving a ‘vision document’ in which we described sultant leaves, and then the organisation must
the test process is not always a straightforward the envisioned future as an end goal of the be able to continue the improvements on its
task. On average it takes approximately two improvement initiative. Creating such a docu- own. Therefore we recommend that the own-
years to achieve TMM(i) level 2. To support ment is important to reach a common under- ership of the improvement project and the test
organisations doing test process improvement, standing about the ‘why’ of the improvement process is given to the internal staff, not to the
a number of recommendations are provided, project, to get commitment and to guide the consultant. A common pitfall of consultants
based on our practical experiences. They have improvement project along its way. ‘driving’ the improvement project is that they
been major contributors towards success in a fail to adapt to the maturity level of their cus-
test process improvement project. A typical PLANNING PHASE tomer. Some organisations want to use their
TMMi improvement project has the following own strengths to achieve improvements: to
phases: Organising the project team what extent do they accept the involvement of
-- Initiation (determine goal and scope) A common problem is that improvement ini- an external consultant? It is difficult to achieve
-- Planning (define the project in terms of tiatives get insufficient attention and resources real commitment when the changes are (per-
structure, resources, ownership) to really make a difference. We have seen very ceived to be) ‘forced’ from the outside. Make
-- Implementation (developing the deliver- positive results when the project is organized clear agreements on responsibilities, also for
ables, like procedures and templates) around the quick scan’s recommendations. a consultant it is important to know what he
-- Deployment (publication and roll-out of Each recommendation is assigned to a Recom- is accountable for and what the scope of his
deliverables, institutionalisation) mendation Workgroup. The workgroup imple- involvement is.
ments the improvements and deploys them.
INITIATION PHASE Since the changes are to be deployed in test Create stakeholders involvement
projects, each workgroup consists of repre- Not all stakeholders may be obvious at first
Determine the current status sentatives from all test projects. Each work- sight. Therefore we recommend a stakeholder
“If you don’t know where you are, a map won’t group reports to the test improvement project analysis at the beginning of the improvement
help.” (Watts Humphrey). Before starting an manager, who in turn reports to the steering project. The people whose participation or
improvement project it is necessary to know group. It is crucial that the steering group has contributions we need are stakeholders, but
which maturity level the existing test process- members that will support the improvements also the people who should have a certain
es have. Therefore the current situation has and benefit from them. If no sufficient capacity interest in our results. Management, or more
to be assessed. A formal assessment may not is allocated to staff the improvement project specifically the sponsor or champion who
be required to determine the current status; a team, and the team members are not enabled supports the improvement project, evidently
quick-scan is usually sufficient. A small group to spend time on improvement activities, then needs to be informed and involved. Also think
of people in the organisation is interviewed all resources will soon be occupied with their about quality assurance, the owner of the qual-
and the current set of documentation is col- other daily operations instead of improving the ity system and those responsible for the test
lected and checked. The result is a report con- process. infrastructure. Other disciplines may be in-
taining a current maturity profile (see figure 2 volved or affected as well, such as a tailored
for an example) and recommendations for im- Communication test training for developers.
provement. The report is presented to manage- One of the most important parts of the im-
ment and this is usually the basis for a TMMi provement project is communication. Improv- The maturity of the development processes
improvement project. ing means introducing change. People will It is important to not only look at test process-
only change if they really understand why es but also at the processes which testing is
change is necessary, so inform them, continu- highly dependent on. Without mature support-
Process Area Score Conclusion
ously. People see their colleagues participating ing processes it will be very difficult or even
Test policy & 3.9 Not satisfied in the improvement project and want to know impossible to improve the test process. Make
strategy what is going on. They, as well as manage- sure that project management, configuration
Test planning 4.6 Not satisfied ment, must be informed on a regular basis. management, requirements management and
Publishing periodic newsletters showing the risk management get enough attention in the
Test monitoring 4.2 Not satisfied
progress and having regular department meet- organisation that you are improving. Lack of
Test design & 5.4 Partly satisfied ings are ways to do this. Another best practice project management will cause test manage-
execution is to involve these persons in reviews of the ment to fail. If the organisation is not used to
Test environment 6.6 Partly satisfied project’s deliverables. In other words: share working on projects, it will be difficult to em-
what you are doing to create a buy-in. You can bed the testing activities in a structured way.
Figure 2: TMMi Maturity Profile document your approach in a communication When applying test design techniques it is
plan, e.g. with a mind map. recommended to have a set of requirements.
Determine the goal Also changes in the requirements should be
It is desirable to clearly define the goals of the Process ownership well managed and the test team should be
improvement project. These goals should be With their knowledge and experience, external notified. Finally, not having a configuration
directly related to the organisational business consultants can make a valuable contribution management system often results in defects
goals to enable management commitment. If to an improvement project. Especially if they that cannot be reproduced and defects reoccur-
you want to get people involved and commit- are allocated to the improvement project on a ring that were supposed to be solved. These
ted, you will have to explain to them where fulltime basis, they can really be the stimula- types of problems can’t be solved within the
the improvement efforts should lead. Adher- tors of process improvement in terms of effort test process itself. Consider starting a sepa-
ence to the TMMi is not a goal in itself. The and progress. In one project we experienced rate improvement project for these supporting
model can provide guidance and practices in the difficulties of combining the responsibil- processes, or just don’t start a test process im-

22 The Magazine for Professional Testers www.testingexperience.com

 Potsdam Chamber of Commerce and
Industry, Germany
©CrazyCat/photocase.com
m

September 24 – 26, 2008

Register Now!
www.conquest-conference.org

25 % discount for testing experience readers

The conference program is included in this issue of testing experience.

• Service Oriented Architecture

• Business Process Engineering
• Secure and Safe Software-Based Systems
• Secure Software Development
• Model-Driven Engineering
• Requirements Engineering
• Verification and Validation
• Testing
• Metrics and Measurements of System Quality
and of Development Processes, Analytical
Models of Software Engineering
• Project Management
• Configuration Management

Keynotes
Partner Country
• Libby Affen, Matrix Global Israel (Talpiot), Israel
• Bjoern Brauel, Software AG, Germany
• Andreas Kindt, Deutsche Telekom AG, Germany
• Ingolf Krüger, University of California, San Diego, USA
• Andreas Spillner, Hochschule Bremen, Germany
& Karin Vosseberg, pdv.com Beratungs-GmbH, Germany ISRAEL

Organizer Your Contact

Tanja Brütting
[email protected] [email protected]
International Software Quality Institute Tel +49 9131 91910-16 www.isqi.org

www.testingexperience.com The Magazine for Professional Testers 23

provement project at all if the organisation is to be addressed, this paper will only focus on be clear and concise. The business language
not ready for it. the process areas that are essential for making that is typically used in mission statements
a good start. At maturity level 2 the test strat- and business strategies is often too abstract
Alignment with other improvement initia- egy and policy are the first and most important to clearly explain what testing is about. It is
tives subjects that give direction to the entire im- better to use testing terminology, e.g. by sum-
Within organisations other improvement ini- provement project. This is one of the strengths marizing the main themes that have focus in
tiatives can be on-going, e.g. software process of the TMMi: its top-down approach that starts the organisation, like ‘risk based testing’, ‘test
improvement based on the CMMI. Beware of with the organisation’s view on testing, and us- training’ or specific (non-)functional quality
the ‘yet another initiative’ syndrome! A de- ing this as the basis for further improvements. characteristics, like reliability and usability.
partment can only handle a certain amount of Metrics are important to measure the effect of
change at a time. This highly determines the the changes and drive further improvements. Test strategy
change velocity. To prevent that duplicate ac- A test policy is implemented in the organisa-
tivities are executed it is recommended to map Test policy tion by a test strategy. The TMMi requires
how the test improvement activities contribute Within the TMMi the process area ‘Test policy the definition of a test strategy for each pro-
to other improvement activities and vice ver- and strategy’ requires that an organisation de- gramme in the organisation. This strategy pro-
sa. This mapping should aid project managers fines a test policy that states the organisation’s vides clear insight into how the products are
in working with improved processes in their view on testing. It answers questions like tested throughout their life cycle. It has to fit
projects. ‘Why do we test?’; ‘How important is qual- into the organisation’s development process
ity?’; ‘Are we time-driven or quality-driven?’. (e.g. RUP, RAD, V-model, DSDM). The strat-
IMPLEMENTATION PHASE Typically the answers to these questions are egy contains a definition of test levels and the
related to the business goals or strategy of the organisational groups that execute them, the
During this phase all planned changes, typical- organisation. The test policy is an important approach to regression testing, entry and exit
ly based on the recommendations from a quick basis for (management) commitment to all the criteria, and more. A sample strategy is shown
scan, are implemented. When possible, use activities in the organisation related to quality in figure 3. Make sure that you create a strategy
existing processes and best practices as much and testing. In one of our improvement proj- that is easy to explain to stakeholders, mainly
as possible. Sometimes a thorough process is ects, for example, we derived the test policy your testers. The strategy must be aligned with
already common practice in the organisation; directly from the so-called “One Page Devel- the organisation’s test policy.
it just needs to be documented. opment Strategy” that was already available
Although in practice probably all process ar- and deployed at a higher organisational level.
eas of the applicable maturity level will have However, keep in mind that a test policy should

Unit testing Integration testing System testing User acceptance testing

Objective Unit meets its design. System is ready for system System meets specification System meets specification
Ensure low level quality. test: quick functional check. (FD). (FD).
Unit tests run automated for System does not show regres- System meets business re-
each build (regression test) Integrated units meet inter- sion. quirements and expectations.
face specifications/global System is “fit for use” in its
design/architecture. business context.
System does not show regres-
sion.
Responsible Development Development System test team Acceptance testers in busi-
ness units
Entry criteria Developers are challenged to All unit tests pass. FDs approved. Requirements system tested.
write unit tests by TLs and All unit tests pass. No open defects with severity
Seniors, following a unit test No open defects with severity major or higher.
strategy. major or higher.
Exit criteria All unit tests pass (build does All unit tests pass. 100% Test cases executed. System meets acceptance
not fail). Planned code reviews ex- No open defects with severity criteria.
ecuted and rework done. major or higher. No regression.
No open defects with severity Business approval for pro-
major or higher. duction deployment.
Techniques Ad hoc testing. Ad hoc testing / error guess- Black box techniques. Error guessing.
ing Error guessing. Some black box techniques.
Tools JUnit Jira HP Quality Center (Test Jira
Director + QTP)
Jira
Figure 3: Sample test strategy for a V-model process.

Metrics or analysed. Goal-Question-Metric (GQM) is document overview should be established. In

Measuring the progress and quality of testing a good method to define a useful set of mea- one TMMi project the documents were initial-
as well as establishing performance indicators surements. With this method every metric is ly delivered separately by different teams for
for the test process are important to monitor the derived from a specific goal and implements a their particular improvement action. When all
status and to show the benefits of the improve- specific question from a stakeholder related to individual documents came together it was ob-
ment process. Think about metrics in the area that goal. Thus a metric is only defined if there vious that an overall document, tying all parts
of efficiency, effectiveness and deployment. A is a clearly related need for it. Keep metrics as together into one total procedure from the start
common pitfall is that a (too) large number of simple as possible! to the end of the test process, was missing.
metrics is defined. As a result the collection of Once this overall procedure had been devel-
measurement data requires a lot of effort and Documentation structure oped, it became the starting point of a struc-
we often see that the collected data is not used In an early phase of the improvement project a ture or hierarchy for all the other documents in

24 The Magazine for Professional Testers www.testingexperience.com

the quality system. By thinking about a solid consuming part of the improvement project. be directly embraced. When applying changes,
structure for your documents at the beginning You can publish a procedure or template, but be prepared for resistance. Don’t change more
of your project, you can save a lot of time on that doesn’t mean that it will be immediately than necessary, take small steps and communi-
reworking and restructuring later. used by everyone. Often the problem in im- cate the changes as often as needed and in any
provement projects is not the availability of way you can think of. An improvement project
DEPLOYMENT PHASE documentation but the lack of adherence to the is not successful until it has been deployed by
described way of working. To achieve a ma- people motivated to improve.
Accessibility of deliverables turity level, at least 80% of the people in the
Make sure that all documents, both procedures organisation should work according the docu- We thank Richard Grooff for his contributions
and templates, are easily accessible for all tes- mented process. If existing procedures are not to the initial versions of this paper.
ters and other stakeholders. An intranet page is followed, try to solve the root cause first. Only
a very effective way to provide a clear, prefer- when it’s clear why current processes are not
ably graphical structure, allowing users to eas- being followed is it worthwhile to change and
ily access and download the documents they enforce them.
need. In one project we depicted the documen- The throughput time of the change process
tation structure as a graphical presentation of does not only depend on the amount of chang-
a ‘pyramid’, with the overall process at the es, but also on the organisation, its culture and
top, and all related and supporting documents, the running development project(s). It may be
such as work instructions and templates in the wise to look for a suitable pilot project to see
basis. This test framework became a familiar if the changes have the intended effects and
artefact in the quality system and soon every- if the procedures are really feasible in day-to-
body recognized and used ‘The Pyramid’ on day practice. But such a pilot candidate must
the intranet as the starting point in their search be available – preferably in one of its early
for a document or template. stages – and willing to participate. Also in an
ad-hoc culture, where local heroes have a large
The change process influence, changes leading to a more struc-
Deployment is the most difficult and time tured and disciplined way of working may not

Biography
Erik van Veenendaal is the founder and director of Improve Quality Services BV. He is an internationally recognized testing expert with over
20 years of practical experience. He is the author of numerous papers and a number of books, including the best-sellers ‘The Testing Prac-
titioner’, ‘ISTQB Foundation’ and ‘Testing according to TMap’. Currently Erik is the vice-president of the ISTQB and the vice-chair for the
TMMi Foundation. For his contribution to the testing profession he received the “European Testing Excellence Award” in December 2007.

Rob Hendriks has more than 12 years of experience in software quality, with great emphasis on software testing. The last few years he
has been active as a senior consultant in projects for consumer electronics, professional technical systems and administrative systems.
He regularly runs courses in the area of inspections and testing (ISEB Foundation and ISEB Practitioner accredited), and is specialised in
the field of test design techniques. Currently he fulfils the role of operational manager within Improve Quality Services Ltd.

Jurian van de Laar has more than 13 years of experience in software engineering. As a senior consultant at Improve Quality Services, he is
involved in several TMM/TMMI and CMMI improvement programs. He had a leading role in achieving TMM Level 2 at Philips Healthcare in
2007. Jurian is an accredited teacher of the ISTQB Foundation and ISEB Practitioner training. He is member of the Syllabus working party
of the Belgium and Netherlands Testing Qualifications board.

Bart Bouwers is a Senior Test Consultant at Improve Quality Services. He has over 13 years of experience in software development and
software testing, in several positions, in both technical automation and information systems domains. Bart is a certified ISEB Practitioner.
Currently he is running a TMM Level 2 improvement project at a Dutch bank.

www.testingexperience.com The Magazine for Professional Testers 25

Interview Norbert Bochynek
Director IT Strategy

Not every reader of this article is familiar with DSGV. Can you Which challenges is your department facing at the moment?
please give us a brief overview of the tasks and goals of DSGV?
The fusion of FinanzIT with the Sparkassen Informatik processing to
The Deutscher Sparkassen- und Giroverband (German Savings Bank form FinanzInformatik has been completed. The consolidation process
Association, DSGV) is the umbrella organization of the Sparkassen- of ten data centers in 1998 to one in 2008 has therefore been achieved
Finanzgruppe. This includes 446 savings banks, 7 Landesbanken, 10 successfully. This fusion makes it possible for us to achieve sustainable
Landesbausparkassen, 12 public insurance companies and many more synergies and these now have to be increased.
financial service providers.
DSGV represents the interests of the Sparkasse financial group, or- The development in the IT of the regional banks will be changed sig-
ganizes the formation of intention within the group and determines nificantly by the current influences. Until there is clarity here about the
the strategic direction. For this purpose it partly operates in an affili- business models and fusions, not much will change in the IT collabora-
ation with the regional associations and additional institutions of the tion between the regional banks.
Sparkasse financial group.
DGSV is the carrier of the central educational facilities of the Sparkasse Which challenges do you see particularly for the IT in this crisis
financial group: the German Sparkasse Academy and the college of the among the (affected) banks?
Sparkasse financial group, the University of Applied Sciences. Other
community institutions are the Association for the Economic Support The cost and margin pressure as well as new legal requirements by the
by the Sparkasse financial group, the Eberle-Butschkau Foundation and legislator will have the effect that banks will have to pay closer atten-
the Sparkasse Foundation for International Cooperation. tion to the costs, optimize their IT and handle their project challenges
It also administers the institute-securing programs according to the much more effectively. The optimization will not only lead to the imple-
Deposit Protection and Investor Compensation Law and the liability mentation of new and improved systems but also to the slimming down
association formed for this purpose, along with the insurance funds of of the processes and a corresponding training of the employees. One
the bank transfer centers and the insurance funds of the Landesbaus- bank or another will surely look for near-shoring and off-shoring pos-
parkasse banks. sibilities and accordingly keep the costs within certain limits.

As the director, you are responsible for IT is decisive for the success, regardless of
the IT strategy of DSGV? In what way The development in the IT of the regional whether a bank is more or less affected by
are you supporting your member insti- the subprime crisis. We can see that some
tutions? banks will be changed significantly by banks have a clear economic advantage
the current influences. compared to others due to their superior-
My department provides support during ity in IT. A good example in Europe is the
the rendering of nationwide IT concepts. Spanish Santander Group.
These serve as recommendations for the
member institutions but are not legally binding. Among other things, You have addressed a few subjects that I would like to examine
my division implements projects to develop control systems for IT, for individually.
the invoicing service of Sparkasse and for the trade with the used li- You’re speaking about the use of new, improved systems. We’re ob-
censes. serving that some of the new and prevailing systems and applica-
We also coordinate the implementation of regulatory requirements (e.g. tions in the banking world still don’t have the necessary maturity,
the compensation tax) as well as the business strategies (financial con- which has the result that the old systems have to continue to be
cepts) that are developed at DSGV to IT. Here we are working espe- operated simultaneously at least for a while. Doesn’t this result in
cially closely with FinanzInformatik. higher costs after all and make IT inefficient?
Employees of the institutions, FinanzInformatik, and association part-
ners are collaborating on all of our projects and look for the best solu- Well, there will always be problems during the employment of new
tions and concepts. technologies and systems. There is no IT project that is not faced with
these challenges. With some projects the problems can be solved in the
prescribed or planned time period, but not with most of them. The plan-
ning of IT projects has to be improved. Everyone in IT has to face this

26 The Magazine for Professional Testers www.testingexperience.com

problem – it’s not a specific problem for a particular group. processes. In addition, an effective IT process management is indis-
pensable that effectively controls the procedures with the aid of suitable
About the other point: it is known that there are providers who are not measured quantities – so-called key performance indicators (KPIs) -
offering “the heart of the matter” and that their systems don’t perform in and therefore continuously improves them. This is an incredibly impor-
the way they are supposed to in the end. But I think that the market will tant aspect in the quality management of banks.
be regulated on its own even in this respect. No IT management board
will want to carry additional Quality management and quality assurance play a
costs in the long-term. The central role especially in large migration projects.
effectiveness and efficiency To what degree can you prescribe to the member
of the IT added value chain The effectiveness and efficiency of the IT institutes how the quality management has to be
is a central goal. One might added value chain is a central goal. arranged or that they have to use established meth-
believe that some providers ods to determine test cases?
and IT executives think that
the delivery date is more We can’t prescribe anything and that is intentional.
important than the quality. But in actuality the total costs of systems But as an association, we can define certain approaches that concern
that were not tested sufficiently are much higher than those of systems the entire association. If you find a majority here and they are finalized,
that were subjected to a professional and sustainable quality assurance. then they’re usually implemented too.
These additional costs then strain the budgets in the IT business and of The persons in charge of IT at the Sparkasse financial group are aware
course the yields of a bank. of how much significance the quality of systems and applications has.

There are certainly also deadlines that definitely have to be met. But We recommend to our members that they support their employees dur-
here the cost/use ratio should be seen in relation to the risk that is being ing their daily work, e.g. through targeted continuing education. Two
created. In the past we have seen that decisions were made based on a perspectives are very important here: For one, the employee expands

missing risk analysis that afterwards had very costly effects. But I think his personal profile and is developing it in a future-proof manner. For
that the IT managers and stakeholders are aware of this situation. another, the loyalty to the company as well as its character are support-
ed. The lack of qualified IT personnel will increase over the next years.
You spoke about the processes. The effectiveness of IT should be If one’s own employees are not offered the opportunities for continuing
supported by new, adjusted processes, among other things. Do you education or if they are not trained appropriately for their tasks, there
rely on international standards or models for this purpose in your is the danger that they will leave the company to develop their future
association? elsewhere.

We support our members with the studies and results from pilot projects
and benchmarking analyses of other banks. We don’t close ourselves off You mentioned the training and education of employees to guar-
towards trends or new discoveries and developments. antee a sustainability of the changes in IT, to have the employees
CMMI, SPICE and especially ITIL are in a boom, sometimes with more properly complete their tasks and to form a good connection to the
or less intensity. I think that we are well-advised to provide our mem- company. There are movements on the subject of certification on a
bers with the necessary support in worldwide level; the offers are very diverse. What is your opinion
IT process management. In this area about this?
we work together with very experi-
The lack of qualified IT enced partners so that our member As I said, it’s important to me that the employees are qualified for their
personnel will increase institutes remain competitive. tasks. This is the prerequisite to counter certain risks in the life of the
But the processes are just one side project.
over the next years.
of the coin. The employees and All regular testers and test managers should at least receive a basic train-
their know-how in filling these pro- ing – and by this I don’t just mean the IT staff but also the specialized
cesses with life are the other, almost departments. The ISTQB certified tester training with almost 100,000
more important aspect. They have to be trained, supported and coached certified people worldwide forms a foundation for this. At the same time
professionally so that these changes have a sustainable effect on the everyone is also aware that a certificate alone isn’t everything. The em-

www.testingexperience.com The Magazine for Professional Testers 27

 ployees have to practice and collect their experiences and here profes- Yes, definitely. The attention of IT was always directed towards attacks
sional coaching is very helpful. What is important in this respect is that from the outside – knowing quite well that the enemy can also be on
the colleagues have initially been told what they can do and should do the inside! Especially the security of data at companies and the protec-
and how it is done – professionally. The continued development of each tion of applications will continuously gain in significance and there are
individual – like other continuing education – is planned by the respec- enough examples that show that this is necessary.
tive supervisors. Several steps are provided in the ISTQB certification This perspective of security had not been widely distributed so far,
series that round off the performance of a professional tester and test where one had to defend against attacks not only on the outside but the
manager. inside as well. Yes, this is a topic.

Other relevant certifications For years there has been an increased amount of off-shoring and
are offered as well, such as it’s important to me that the near-shoring in the private bank economy. Do you also see this as a
e.g. according to IREB for employees are qualified for possible model for the public banks?
professional requirements their tasks
engineering or the certifi- Of course. Steps in this direction have already been taken by one or
cate developed by ISSECO another of the public banks. This is another development that one can’t
on the topic of application close off to.
security. The certification of ISSECO is something that IT has needed Nonetheless this also has to be quite well considered. Depending on
for a long time already. Aside from that ISSECO was founded by Ger- which country and cultural circle they enter, they will be confronted
man companies, among others, at the top of which is SAP AG. The with different problems. These can also be mastered, that’s certain.
transfer of these special skills has to be regulated and it especially has Some off-shoring or near-shoring projects fail for the same reasons as
to be kept at a high standard. one’s own projects locally. Those don’t always function either, unfor-
tunately.
I would like to emphasize again that the certification alone isn’t ev-
erything. A certain practical experience is required. With QAMP the The characteristic risks that have their origin in the distance, communi-
certifications are supplemented and rounded off by this important but cation and culture have to be minimized. This specific risk management
previously missing perspective. Because QAMP certifies that the per- can only be realized with a partner who is working with the local shor-
son has not only attended a variety of classes and passed tests but also ing partner. These partners have to be selected very carefully, but in this
has at least two years of practical experience in this environment. respect these projects require a very specific planning.

What has been done in this area in the last few years was and continues It also has to be made clear, however, that not all subjects are suitable
to be necessary. for an off- or near-shoring project – pertinent experiences have already
been made in this area.
I would like to bring up application security again. Here most peo-
ple think of firewalls, penetration, etc., or hardware security. Few Thank you very much for your time and the interview.
see the application itself as a weak point. Do you think that this
topic will be booming soon? You are welcome.

Development &
Testing Outsourcing
in Latin America

www.bicorp.biz
© iStockphoto

28 The Magazine for Professional Testers www.testingexperience.com

 Software Testing – the future?
by Geoff Thompson
© iStockphoto

When I was set the challenge of writing an ar- Initially I thought here we go again, testing “I wouldn’t go live yet”– yes I heard that said
ticle on the future of software testing, having put the bugs in etc. but then I understood. I by one Test Manager one day – and this was
got over the initial creative panic, I was won- wonder if it was more that there was a lack all he said
dering how to start. Then I found the following of the right information from the test organisa-
headlines in two well known online computer tion that would have enabled the right decision None of these provide even the slightest hint
journals:- to be made by the BA and BAA management. of what the impact of going live is! How criti-
If the test organisation had a grip on the real cal are the 150 tests left outstanding? What’s
‘BA: software failures key in T5 fiasco’ issues that mattered to the management team the error finding profile? Why is testing still
Computing 8 May 2008 and were able to communicate the real risks finding errors now? These are just some of the
‘Update: lack of software testing to blame for maybe a different decision would have been questions that these statements generate but
Terminal 5 fiasco, BA executive tells MPs’ taken, or at least the decision that was taken can’t answer.
Computer Weekly 9 May 2008 would have been made based upon the right
information, with complete awareness of the So, if we get the information wrong, wrong de-
In summary both articles made the same point. impact. cisions will be made. There was however one
During questioning by MP’s Willie Walsh (BA very good example of a situation where the
Chief Executive) reported that construction To me, testing’s role is the provision of data right information was provided in the wrong
work had not completed in time squeezing the to enable go live decisions to be made (I have way, and therefore ignored. The 1986 Chal-
window for any real testing. The management heard this termed as Trusted Advisor and Proj- lenger disaster was predicted – if you knew
met, identified and agreed the risks and issues ect Intelligence). Yes we might be involved that the Space Shuttle would definitely ex-
this creates and decided that the risk was ac- in the decision but ultimately the decision plode killing all of the astronauts, would you
ceptable to enable a go live as planned. belongs to the Sponsor and his management have given the go ahead to launch. NASA did.
team, but the data used in the decision will pre- Several engineers had voiced concerns about
In hindsight Walsh said “If I were to pick one dominately be derived from test activity and the effect of the temperature on the resilience
issue I would have done differently, it is that, must be sufficient to enable the right decisions of the rubber O-rings that sealed the joints of
having recognised the importance of testing to be made. the Solid Rocket Boosters (the tubes attached
and having designed six months of testing, we to the side of the Shuttle – that are full of fuel
subsequently compromised on that.” Given this is a major element of our role; I sug- at take off). They argued that if the O-rings
Computer Weekly 9 May 2008 gest that we need to get this right or the future were colder than 53 °F (12 °C), there was no
of software testing will be very dim. We are in guarantee they would seal properly. This was
Why, did it interest me? Quite simply the man- the best position, or at least we should be, in an important consideration, since the O-rings
agement of the project did exactly the right a project to gather real data, and to establish had been designated as a “Criticality 1” com-
thing; project management and risk manage- what the right data may be, to enable the accu- ponent—meaning that there was no backup for
ment were used to assess whether slippage in rate reporting of project progress information. them and their failure would destroy Challeng-
earlier parts of the lifecycle meant that reduc- Far too often today a test team arrives and sits er and its crew. They also argued that the low
ing test activity and timescales would have an in its cocoon only appearing at the end of the overnight temperatures would almost certainly
impact on the quality of the finished article. test cycle to announce they haven’t finished result in Solid Rocket Boosters temperatures
Obviously the risk was perceived to be low but can’t give anything helpful to say other below their redline of 40 °F (4 °C). However,
as a decision was taken to reduce testing and than perhaps statements like: they were overruled, the launch happened, the
go live on time (a decision now regretted by O-rings failed leaking rocket fuel, and the rest
Walsh). “You can’t go live because...” is history.

What I also found interesting was that one of “We have 150 tests still to run” Interestingly the right information was iden-
the papers saw software testing as the issue. “We are still finding errors” tified but if I am very critical, using words

www.testingexperience.com The Magazine for Professional Testers 29

like ‘no guarantee’ instead of ‘an absolute The data requirement is never static, so we also the reality is that the exam candidates leave
guarantee’ when the management board were need to be conscious of the changing needs for with an understanding of the basic vocabulary
under significant pressure from the US Presi- information as a project progresses. and approach to testing, but in most cases no
dency to launch, didn’t really emphasize the real practical capability, and certainly no un-
facts strongly enough. In the same situation, To enable the future of software testing we derstanding of how to communicate outside of
with the same information, I might well have have to influence people’s approaches to Test the test team.
also decided, like NASA did, to launch. Now Management and the provision of informa-
I know it’s a word thing, but in reality if we tion? Is it a qualification thing, or maybe stan- Standards
can’t express ourselves and ensure that our dards, there are test models out there and what There are a few out there, for example BS7925
message is understood all the great work we role do tools play in this arena? The following – 1&2 (the British standard for Component
have done to improve the processes we use sections look at each of these individually and Testing and its associated glossary), IEEE829
and align vocabulary will be wasted and we their relevancy to the issue. (the Test Documentation standard) etc. etc.
will never grow as an industry and will always
be seen as the also ran profession! It’s all about Software Testing Qualifications In their favour Standards are developed by
communication. You may or may not know that I have been in- a very wide working group (these can be as
volved in Software Testing Qualifications for a many as 1000 contributors) who build and re-
Sometimes, alright I concede, most of the time few years now! Firstly working with ISEB (In- view the content. In most cases involvement
the information we need to provide is not what formation Systems Examinations Board) and comes from right around the globe. However,
the Programme wants to hear, but it’s our job lately internationally with the ISTQB (Interna- by necessity they are generic and some con-
to ensure the right information gets through to tional Software Testing Examinations Board). cessions have to be made to gain global agree-
the right person and that they are able to un- So I guess I am a little bias, but I believe there ment (in fact I understand that the latest version
derstand and make informed decisions. We is value in them. However, there does seem of IEEE829 couldn’t be released as planned as
need to be prepared to be the bearers of good to be some polarisation in the industry as to the detail was not agreed) and provide one so-
and bad news. the value of these types of qualification, some lution for everyone, so they are useful but they
calling for them to be banned, others suggest- can’t provide the whole answer.
If testing can’t provide the right information ing that having got them you are able to sell
it actually adds little value, and that I suggest yourself as a competent tester. Standards don’t, in any example I have seen,
maybe the situation we find ourselves in a lot indicate how to provide and what information
of organisations today, we all work very hard The reality is that a good tester or test manager is needed to communicate effectively with the
at our profession but no one outside of it seems needs many components of skills and knowl- programme, although they can help. For ex-
to understand what we do and why. edge combined to make them successful. In ample in IEEE829, when it discusses entry
my view the qualifications account for about and exit criteria and closure reports looking at
Whilst working on one client site I remember 10% to 15% of the knowledge required for the planned activity against actual activity and les-
interviewing a Test Manager who was most practical application in the workplace. sons learned, although it mentions these key
upset that the Programme team had asked him elements of a test project it doesn’t provide
to stop issuing his daily report, for what he At ISTQB Foundation level there is a short any help in pinpointing the right detail.
described as stupid reasons. When I met and training course (3 days) followed by an exam.
interviewed members of his programme team However the quick learners amongst us could They are however excellent process models,
they said that the report was stopped because very quickly understand what is required and that provide guidance on how to establish your
from day one the Test Manager had had the pass these exams, maybe even without ever test project.
project flagged as red, what was requested was testing anything, there is no minimum experi-
that the report included relevant data (such as ence requirement to take the Foundation. Test Models
where the issues lay, what element of the soft-
There are many models available for testers to
ware was not working etc.) or was stopped. Intriguingly ISEB have recently revised their review against. Most consultancies will have a
The Test Manager saw no reason to include single Practitioner Exam, into three, the Inter- Test Maturity Model of their own, most sadly
the extra data so stopped issuing it. Even to- mediate (a mandatory common module which leading to creating a sales opportunity, and not
day the Test Manager cannot understand what contains their view of the generic detail that actually there to help the organisation that has
he did wrong and blames the programme team is required for the advanced level),Test Man- been reviewed against the model to improve
for the delays in delivery because they did not agement and Test Analyst, but only prescribe what they do. The nearest commercial model
listen to him! a 3 day training course followed by an exam. that really does set out to help companies is
There is a minimum entry requirement, but the TPI® model provided by Sogeti. However
If Software Testing is ever to be seen as a pro- they do not include practical test experience, this is now a little out of date; it is based upon
fession and be respected in the future then we more a passing of time between taking the their in-house test method T-Map which has
should focus on the information we need to Foundation exam. been updated and those updates have not been
enable the Programme to make the right deci-
reflected within TPI®.
sions. ISTQB, on the other hand, have just launched
their Advanced level role-based qualifications, Probably the best model available today to
Now I am not suggesting that I know the an- and like ISEB there are three of them, Test help ensure good processes in use across test
swer. Each person will have differing require- Manager, Test Analysts and Technical Test projects is the new TMMi Model. This model
ments, and most importantly their require- Analysts. However unlike ISEB there is no has been developed by the TMMi Foundation
ments may be very different to what they common module, as these are role based(for (www.tmmifoundation.org), an international
request individually when asked as part of a example the role of a Test Manager in a re- organisation whose aim is to establish a non
team (I call this the pack syndrome – when a view is very different to that of a test analyst) commercial test process assessment model
group influence individual perspectives and with a minimum training period of 5 days each for the industry. The TMMi model takes its
results. If you get separate agreement to an ac- followed by an exam, with basically the same input from many places including the origi-
tion from each individual in a group of people, entry criteria as ISEB. nal Bernstein TMM, and others including the
when you put them together the actions may
Gelperin and Hetzel’s Evolution of Testing
and often do change), but it is our job to iden-
Model [Gelperin and Hetzel], which describes
tify what they are and focus our approach to So I believe qualifications have a very impor- the evolution of the testing process over a 40-
testing in order to deliver these. We need to be tant part to play in the education and verifica- year period, Beizer’s testing model, which de-
conscious of the pack syndrome. tion of software testers and test managers. But scribes the evolution of the individual tester’s

30 The Magazine for Professional Testers www.testingexperience.com

 International Software Quality Institute

ARE YOU ALREADY CERTIFIED?

The International Software Quality Institute (iSQI GmbH), headquartered in Erlangen and Potsdam (Germany), develops internationally
accepted certification standards for advanced vocational training in the area of software quality. In order to optimize and safeguard the
skills and abilities of software professionals, iSQI was founded in 2004 as a subsidiary of the German non-profit organisation ASQF e.V.
and works together with international organizations, e.g. ISO (International Organization for Standardization) or EQN (European Quality
Network).

iSQI certifies IT personnel in more than 35 countries. With more than 3000 exams per year in Germany alone, iSQI is one of the most
important personnel certifier in the area of software quality. The advantages of personnel certification are primarily the secure, compa-
rable qualification of professionals across national boundaries and language barriers.

The iSQI certification program includes standards such as

ISTQB® Certified Tester, Foundation Level

ISTQB® Certified Tester, Advanced Level (Test Manager, Functional Tester, Technical Tester)
iSQI® Certified Professional for Project Management
iSAQB® Certified Professional for Software Architecture
IREB® Certified Professional for Requirements Engineering
iNTACS™ certified ISO/IEC 15504 Provisional Assessor
iNTACS™ certified ISO/IEC 15504 Competent Assessor
TTCN-3® Certificate
Certified Innovation Manager
iNTCCM® Certified Professional for Configuration Management
QAMP® Quality Assurance Management Professional
iNTCSM® – International Certified Professional for IT-Security Management
ISSECO® – International Certified Professional for Secure Software Engineering
T.I.S.P.© – TeleTrusT Information Security Professional
V-Modell® XT

In addition, iSQI hosts seminars and organizes different international software

conferences, for example the „Conference on Quality Engineering in Software
Technology“ (CONQUEST). It provides a platform for experts and practitioners from
industry and science to exchange experiences and ideas and to discuss the newest
advancements in their special field.

The CONQUEST conference program is included in this issue of testing experience.

www.isqi.org
SETTING QUALiTY STANDARDS
ALL OVER THE WORLD

www.testingexperience.com The Magazine for Professional Testers 31

thinking [Beizer], research on the TMM car- the following:- So it’s not something a Test Manager can do
ried out in the EU-funded MB-TMM project, alone, we have to work together to ensure that
and international testing standards, e.g. IEEE A. Accurate information provision, in the in the increasingly complex world of software
829 Standard for Software Test Documenta- language of the recipient development we understand what it is we all
tion [IEEE 829]. The testing terminology used B. Building quality in, rather than testing it need, and ensure we deliver it.
in the TMMi is derived from the ISTQB Stan- out, prevention rather than detection
dard Glossary of terms used in Software Test- C. Ensuring that quality is considered and If the test manager for Terminal 5 had provid-
ing [ISTQB]. not seen as a burden, alongside time and ed the following information to Willie Walsh
cost, which seem to have become the new at the point that the decision to reduce testing
The model is structured much like CMMi with buzzwords occurred I wonder if the result may have been
5 levels of maturity. At level 2 of the model different.
projects begin to be assessed on their capabil- As I have said already the great thing about
ity to measure progress through testing. How- Qualifications is that they have established Test Manager –“Mr Walsh we have reviewed
ever like the CMMi model it is not prescriptive some common terminology amongst testers, progress and remaining activity and can report
as to what is measured just that measurement but that terminology is still not translatable that should you reduce testing in the way sug-
occurs, although help is provided regarding across the lifecycle. So we have four languages gested the following will happen:
the minimum is expected. as a minimum being spoken in projects today,
we have the Business (client), the designers, In the first 5 days of operation:
Tools the developers and the testers, and in an ITIL
Now to some these are the panacea, I prefer (IT Infrastructure Library) organisation is pos- 1. Over 23,000 bags will get lost, leaving
to look at tools as the slaves of the process. sibly a fifth. you with a bill well in excess of £300,000
When it comes to information provision there to reconnect them with passengers
does seem to have been thought put in most We have projects that sit together but don’t 2. 500 flights will definitely be cancelled
examples I have seen, to the data that the tool talk to each other. We also have so called Ag- 3. Staff will not be able to park due to issues
provides. The Dashboard seems to be the latest ile projects starting to show how greater col- with the security system, and so will not
fashion accessory for any company wishing to laboration brings big benefits, is this the way arrive on site on time
set its stall in the test management arena. HP forward? 4. 28 lifts will not be operating
has one in Changepoint, Compuware in ODM 5. You will definitely be pulled up in front
etc, etc. In principle these provide data derived I think the way forward for information pro- of the UK Parliament to explain yourself
from actions within the system. vision is actually to learn something from the
Agile world and start to talk to each other, I recommend we delay launch for a minimum
Tools can be excellent providers of data, but and not to work in silos. As I have already of a week so we can resolve these issues.”
that data is only as good as the way it is col- explained, everyone’s requirement for infor-
lected. A common problem, for example, mation is different and so the closer projects Would he have agreed to the same launch
with Test Management tools is that the tool is work together and people mix the more we can date? That I cannot say, but at least if he had,
implemented straight out of the box, with no understand what our requirements are. he would have understood the issues he was to
configuration; testers then load test cases into face with the press and public.
the tool with no views on how progress will In a world of increased outsourcing or off-
be tracked, so they load the minimum data to shoring this becomes increasingly harder to But you may say hindsight is a great thing. If
allow them to do their job. Then they try to achieve. This is especially relevant when the we could all see into the future we could per-
track progress, but realise that the data needed test resources, or development resources, work haps all determine these types of issues easily,
to do this hasn’t been loaded, so they can track across the other side of the world in a differ- but how do we do that prior to delivery?
certain elements, so unfortunately in my expe- ent time zone. It is as important, if not more
rience the reporting in these situations is very important, that in these situations proper and Project delivery is not difficult, it is simply the
poor. regular communications are established. It’s bringing together of professionals in their field
not easy but if success is expected then effort to deliver to a single goal. If that team works
So with some real thought up front as to what needs to be put in to ensure that each party is together then, for example, how much can a
information/data is required the tools can be aware of their goals and provides the right in- developer learn from a business analyst if they
configured to ensure that data is a by product formation to enable measurement of progress talk about how the system should actually
of the process (e.g. does not become time con- to be tracked and understood. work, or where it always doesn’t!
suming to collect and report).
Lots of organisations seem to think that by A project manager who just cares about meet-
Each approach listed above contributes in a putting people in buildings in different coun- ing delivery dates regardless of setting, and
small but very useful way in helping the Test tries and arming them just with a process they meeting success criteria is a fool in my book,
Manager recognise the data that is required, will get great results. This ignores the need to and one that shouldn’t be working in our in-
and to some extent the way that data should be work as a team and communicate, and so in dustry. Success criteria are the goals that are
communicated, however this doesn’t extend most instances is a strategy doomed to failure. the starting point for information; and may be
across the different roles that exist within a different for each different level in the project,
project. It doesn’t prescribe what it is a devel- I think it worthy of a stop here to review an- for example:
opment manager needs to know or the Busi- other element we need to consider. We (and
ness Sponsor. in this instance I am not just referring to test 1. Success for the Sponsor may be a system
resources, I refer to anyone involved in a soft- that saves or makes them £3bn a year. In
So having identified the data, the key is how ware development project) need to accept that that instance the information provided
it is presented back as information in the right Software lies at the heart of almost every busi- needs to include data to reflect achieve-
format at the right time. ness process. Everything from how an organi- ment when it goes live
sation communicates internally and externally,
This particular issue does seem to reside in the to the way it delivers products and services, 2. Success for a Test Manager may be that
Test Managers camp, so I would like to focus to how it manages administrative processes. there is an agreed risk based approach
there for now. Software can liberate or debilitate an organisa- to testing established at the outset of the
tion’s efficiency, effectiveness, risk exposure, project, and all high priority test cases are
I believe that the future of testing depends on risk exposure, and ultimately, its profitability. agreed up front and are completed during

32 The Magazine for Professional Testers www.testingexperience.com

 test execution, but let’s think about that. So to be successful a Test Manager
needs to have a prioritised list of test cases, which they ensure are completed
during test execution.

This is an altogether too familiar success criterion today but maybe one that
doesn’t help point B above (building quality in, not testing it out), maybe a
better success criteria could be:

a. Test activity defined that reducing the amount of test execution to one
week (a tall order but if the team works together and identifies up front
where things may go wrong and stop them occurring. Using an analogy,
if you knew early enough that your wall was about to fall down, would
you write a test and then go away until it fell? Then check it had, raise a
bug report and pay a lot of money to have it repaired? Or would you do
something as soon as possible to stop it falling and keep the cost, time
and quality at the right levels? Translating that into a project, if we know
the objectives of a project manager include time, cost and quality we need
to find ways of preventing rather than detecting issues as this is the most
economical approach to delivery, and to meet all three objectives).
Biography
Test managers and project managers must also reflect progress towards achieving
these criteria. But we shouldn’t forget that throughout the lifecycle of a project Geoff is Consultancy Director of Experi-
these will change as each new issue and risk is resolved, and so by necessity, must mentus (www.experimentus.com). He
the information provision. has been actively involved in testing for
over 20 years covering Test Analysis, Test
Teamwork is absolutely key to achieving project delivery in the future, and ensur- Management, Test Programme Manage-
ing software testing continues to deliver value. Sadly this is something that gets ment and Test Strategy Development.
very little daylight in IT. We ensure all of our people are technically competent
but we don’t really consider that they need to be taught how to manage people and Geoff is a trained Test Maturity Model
how to build working teams. Good developers are promoted and asked to manage (TMM) assessor and can perform Test
people, and it’s the same in testing. How many testers ever get sent on a project Maturity assessments. Obtaining the
management course to learn the basic project management processes? Very few, in accolade as Test Manager of the Year
fact in an organisation I worked in it was considered a course that was too advanced 2004, Geoff was also one of only three
for test resources to attend! However developers and designers aspiring to become testers in Europe nominated for The
Project Managers were allowed to attend. European Testing Excellence Award in
2003.
I think a software delivery project is much like an orchestra, if each part works
together in harmony it can be beautiful, but if only one element works on their own Geoff is a recognised thought leader in
the result will be disastrous. We need to work together so that the software delivery his field. He is a founder member of the
orchestra works, and delivers beautiful results. ISEB Software Testing Board, Interna-
tional Software Testing Qualifications
So now let’s go back to where I started. Good information provision comes from Board (ISTQB) and the TMMi Foundation,
being part of a team who communicate regularly and fully understand the progress of which he is currently Treasurer. Geoff
through the different lifecycle stages, all the way through to the live environment. was directly involved in the creation of
the original ISEB Foundation syllabus
At this point I can’t help but remember the quote – ‘“You cannot manage what you in Software Testing in 1999, managed
cannot (or do not) measure”. (anon). So we must know: the delivery of the ISEB Practitioner
syllabus in 2002 and is currently the UK
• What do we need to measure and does the information and importance of the representative to the ISTQB - formulat-
information change during the lifecycle of the project ing and managing the new Foundation,
• We must still capture the number of test cases, tests run, tests passed/failed, but Advanced and Expert level syllabi.
need to identify how to translate this information for the different audiences
we have Geoff is also Chairman of The UK Testing
• We need to capture ‘quality requirements’ and therefore identify how to mea- Board and Vice Chairman of the BCS
sure meeting them. They must be clearly and concisely defined, easily under- SIGiST (Specialist Group in Software
stood, not ambiguous and easily demonstrated Testing).
• Testers must encourage the breakdown of the silos. Why not work within the
development team and help them test? Geoff’s expertise is widely documented
through articles and publications. He co-
No one can define today what the future of testing is but by learning to commu- authored what is currently classified as
nicate more effectively and by becoming an integrated element of the project, I ‘the best selling testing book on Amazon.
believe the future of testing is assured. co.uk’, ISEB – A Software Testing Foun-
dation. Geoff is also a frequent speaker
at major industry events nationally and
internationally.

www.testingexperience.com The Magazine for Professional Testers 33

 What Testing Cannot Do by Gerald M. Weinberg
© iStockphoto

An excerpt from Chapter 2 of Gerald M. Wein- If you want reliable information, a and b to the risk. If Benito delays the product while
berg’s July 2008 release, Perfect Software— probably won’t do the job, but c at least has a waiting for more tests on the logistics system,
And Other Illusions About Testing, ISBN:978- chance. Gathering information on what a pro- he might enter the market too late. Or, he might
0-932633-69-9. Reprinted by permission of gram actually does when used is one form of go broke spending money on testing. So, we
Dorset House Publishing. To save 20% off the what some people call “testing.” always have to consider money and time in the
$23.95 list price before September 1, 2008, decision about whether to test: Testing cannot
visit www.dorsethouse.com/offers/te.html, Information doesn’t necessarily help reduce be done in zero time and cannot be done for
e-mail [email protected], or call (212) risk. free.
620-4053. Managers in the software business frequently Sometimes, the information produced also
have to make risky decisions, which can be adds to the risk. If the developers have in-
“Have no fear of perfection—you’ll never made less dangerous if based on answers to formation that something isn’t working well,
reach it.” questions such as the following: they may want to spend time fixing it. From
—Salvador Dali, Spanish Surrealistic Painter the point of view of Benito, this could increase
(1904-1989) • Do we ship now? Later? Ever? the risk of spoiling something that was work-
• Do we cancel the project? ing well enough to use.
Imagine you are an executive in charge of a • Do we continue the project and add more You might think the real risk is in having a
multimillion-dollar logistics system. The com- resources? development organization you can’t control,
pany president, Benito, is a dynamic, forceful • Do we reduce the scope of the project? but here’s another danger. If people sue you
character known for firing people in staff meet- • Do we attempt to bring the product to a because of an error in software you devel-
ings when they don’t give him the answers he wider market? oped and sold, they can subpoena the records
wants. He is eager to have the system rolled • Do we sell this version of the product at of your development process. If the records
out. He’s putting pressure on you. You go to a lower price? indicate that your testers found this bug and
the project manager, who reports to you, and you didn’t fix it, you’re in more trouble than if
ask, “How’s progress on the system?” The Of course, such decisions could be made in they’d never found it in the first place. Some-
project manager replies, “I don’t know.” the complete absence of information—and times, ignorance really is (legal) bliss.
What would you do now? Quite likely, you’d they frequently are. More often, they’re made The same blissful-ignorance principle trans-
want more information than “I don’t know.” after the decision-maker has obtained some lates down to the individual manager, even
There are various ways to acquire it: bit of information, while resolutely ignor- if no lawsuit is involved. If there’s a problem
ing other information that might be obtained. with the product in the field and the manager
a. Maybe torture will work. Since testing is an information-gathering pro- can honestly say, “I didn’t know,” the reper-
Pull out the project manager’s fingernails one cess, there’s always a question of whether it’s cussions usually are less severe than if that
at a time until she squeals, “Oh, yes, I do know. worthwhile to pay for more testing. And that is manager knew but didn’t do anything about
It’s great!” If necessary, proceed to toenails. another decision. it. The moral: Think carefully about what may
Someone in the organization has the authority qualify as too much information. But think
b. Maybe promoting a toady will work. to make decisions, with or without informa- even more carefully about which of your goals
Fire the project manager and promote some- tion. If that person will make the same deci- are foremost. Are you trying to produce a suc-
one who says, “I do know.” Repeat this step sion no matter what additional information cessful product, or avoid lawsuits, or simply
as needed. arises, there is little point in testing to gather advance your personal career?
more information. If Benito has decided to
c. Maybe gathering information will work. go forward with the new logistics system as
Look for some information on what the pro- is, why bother getting him more information?
gram actually does when someone tries to use Why bother testing?
it in a realistic way. In fact, sometimes doing more testing adds

34 The Magazine for Professional Testers www.testingexperience.com

 Application Security
www.diazhilterscheid.com
Photo by Steven M. Smith, www.stevenMsmith.com

How high do you value your and your customers’ data? Do your
applications reflect this value accordingly? Accidental or deliberate
manipulation of Data is something you can be protected against.

Talk to us about securing your Systems. We will assist you to incor-

porate security issues in your IT development, starting with your
system goals, the processes in your firm or professional training
for your staff.
[email protected]

Biography
Gerald Weinberg is the author of more
than forty books spanning all phases of
the software development life cycle. For
nearly fifty years—through his writing,
speaking, teaching, and consulting—
Weinberg has helped generations of
software professionals to improve their
productivity. Based in New Mexico, he
is a principal of Weinberg and Weinberg,
consulting to consultants, start-ups,
government agencies, and Fortune 500
firms. Visit www.geraldmweinberg.com
to learn more.
© iStockphoto

Co-Founder of ISSECO (International Secure Software Engineering Council)

www.testingexperience.com The Magazine for Professional Testers 35

 Two Weeks to Better Testing this Summer
by Rex Black
© Katrin Schülke

Many of us will spend this summer relaxing, critical—quite the contrary—and it doesn’t Read a Book on Testing
which is always good. However, why not in- mean we know how to write them well. Most Most practicing testers have never read a book
vest a little of your vacation time in improv- test groups have opportunities to improve their on testing. This is regrettable. We have a lot
ing your testing operation. After all, if you’re bug reporting process. we can learn from each other in this field, but
like most testers, you are time constrained When RBCS does test assessments for clients, we have to reach out to gain that knowledge.
and need to make improvements quickly that we always look at the quality of the bug re- (Lest you consider this suggestion self-serv-
show fast results. So here are three practical ports. We focus on three questions: ing, let me point out that writing technical
ideas which you can put into action in just 1. What is the percentage of rejected bug books yields meager book royalties. In fact,
two weeks, which will make a noticeable dif- reports? on an hourly basis it’s more lucrative to work
ference when you tackle that next big, post- 2. What is the percentage of duplicate bug bagging groceries at a supermarket. Other
summer project. reports? benefits, including the opportunity to improve
3. Do all project stakeholder groups feel our field, are what motivate most of us.)
Get Hip to Risk-Based Testing they are getting the information they need There are many good books on testing out
I have a simple rule of thumb for test execu- from the bug reports? there now. Here’s a small selection, any one
tion: Find the scary stuff first. How do we do If the answer to questions one or two is, “More of which you could work your way through
this? Make smart guesses about where high- than 5%,” we do further analysis as to why. during a beach vacation.
impact bugs are likely. How do we do that? (Hint: This isn’t always a matter of tester com-
Risk-based testing. petence, so don’t assume it is.) If the answer I have read each of these books (some of
In a nutshell, risk-based testing consists of the to question three is, “No,” then we spend time which I also wrote or co-wrote). I can promise
following: figuring out which project stakeholders are be- you that, if you need to learn about the topic
1. Identify specific risks to system quality. ing overlooked or underserved. Recommen- in the left column of the table, reading one of
2. Assess and assign the level of risk for dations in our assessment reports will include the books in the right column will repay you in
each risk, based on likelihood (techni- ways to gets these measures where they ought hours and hours saved over the years, as well
cal considerations) and impact (business to be. Asking the stakeholders what they need as teaching you at least one or two good ideas
considerations). from the bug reports is a great way to start— you can put in place immediately.
3. Allocate test effort and prioritize (se- and to improve your relationships with your
quence) test execution based on risk. coworkers, too.
4. Revise the risk analysis at regular inter-
vals in the project, including after testing
What You Want Books to Read
the first build.
You can make this process as formal or as in- General tips and techniques Pragmatic Software Testing, Rex Black
formal as necessary. We have helped clients for test engineers A Practitioner’s Guide to Software Test Design,
get started doing risk-based testing in as little Lee Copeland
as one day, though one week is more typical. Object-oriented testing Testing Object-Oriented Systems, Robert Binder
For more ideas on how, see my article, “Qual-
Web testing The Web Testing Handbook, Steve Splaine
ity Risk Analysis,” in the Library at www.
rbcs-us.com, or my books Managing the Test- Security testing Testing Web Security, Steve Splaine
ing Process (for the test management perspec- How to Break Software Security, James Whittaker
tive) or Pragmatic Software Testing (for the Dynamic test strategies and T-Map Next, Tim Koomen et al
test analyst perspective). techniques How to Break Software, James Whittaker
Test management Managing the Testing Process, Rex Black
Whip Those Bug Reports into Shape Systematic Software Testing, Rick Craig
One of the major deliverables for us as testers
Test process assessment and Critical Testing Processes, Rex Black
is the bug report. But, like Rodney Danger-
improvement Test Process Improvement, Martin Pol et al
field, the bug report gets “no respect” in too
many organizations. Just because we write ISTQB tester certification Foundations of Software Testing, Rex Black et al
them all the time doesn’t mean they aren’t The Testing Practitioner, ed. Erik van Veenendaal

36 The Magazine for Professional Testers www.testingexperience.com

 Testing is a skill.
While this may come as a surprise
to some people, it is a simple fact.

Fewster, Graham: "Software TestAutomation

Software Test Automation"

Biography TOBIUS
With a quarter-century of software and
systems engineering experience, Rex
Black is President of RBCS (www.rbcs-us.
Premium provider of
com), a leader in software, hardware, ICT Services & Solutions
and systems testing. For over a dozen
years, RBCS has delivered services in - Fast & Flexible Staffing Services
consulting, outsourcing and training for - Professional Testing Solutions
software and hardware testing. RBCS - Project Portfolio Management
clients save time and money through im-
proved product development, decreased
tech support calls, improved corporate
reputation and more. As the leader of
RBCS, Rex is the most prolific author
practicing in the field of software testing Put us to the test at www.tobius.be
today, having written four popular books,
with two new books (and a new edition
of Managing the Testing Process) on the
way. Rex is the President of the Interna-
tional Software Testing Qualifications
Board and a Director of the American
Software Testing Qualifications Board.

Subscribe at

Pantone 350 www.testingexperience.com Pantone 362�

100% 80% 60% 40% 20% 100% 80% 60% 40% 20%

cmyk cmyk

c:79 c:63 c:47 c:32 c:16 c:76 c:61 c:46 c:30 c:16
m:0 m:0 m:0 m:0 m:0 m:0 m:0 m:0 m:0 m:0
y:87 y:70 y:52 y:35 y:17 y:100 y:80 y:60 y:40 y:20
b:76 b:61 b:46 b:31 b:15 b:11 b:9 b:7 b:5 b:3

www.testingexperience.com The Magazine for Professional Testers 37

 The Boundary Value Fallacy

by René Tuinhout

© Mirko Raatz - Fotolia.com

Introduction been implemented correctly. Furthermore, the 1. Every system input (or system output)
Boundary Value Analysis is a popular test tech- article introduces a new technique, B3VA, to can be divided into several classes;
nique to use: It is easy to comprehend and easy solve this issue in Boundary Value Analysis. 2. The system processes every possible in-
to explain to testers and non-testers alike. put value in a class in exactly the same
Part of the ease in comprehension is based Equivalence Partitioning way as it processes every other input
on the elegant and simple examples that can The Boundary Value Analysis (BVA) test value in that class.
be used to explain the technique: A boundary technique is based on the Equivalence Parti- (or: the system has produced every pos-
value is introduced, and it’s explained that tioning (EP) test technique. The basic idea of sible output value in a class in exactly the
three values are needed to test the boundaries EP is quite simple:”If you expect the same re- same way as every other possible output
thoroughly. sult from two tests, it’s sufficient to run one value in that class has been produced.)
This article suggests that three values are, con- of them”1. This idea is based on two assump- Based on these two assumptions it follows that
trary to popular believe, not enough to perform tions: it’s sufficient to test one value per class2:
a black box test to prove that a boundary has

An example:
An insurance system accepts clients aged 99 years old or younger. All other potential clients are not accepted.
Ages are entered into the system under test in whole years3.

The above functional description leads to two classes:

1. People aged 99 and younger will be accepted (this is called a valid class);
2. People aged 100 and older will not be accepted (this is called an invalid class);

This can be represented in a figure (+ means: valid, - means: invalid):

+
- -

99

When using EP, the system in the example can be tested by using (for example) the values:
15 (expected result: client accepted) and 120 (expected result: client not accepted).

(These values are arbitrarily chosen, just as long as for each class a value is tested. Therefore, also the values
89 and 160, or for that matter 20 and 200 could have been chosen.)
Both classes (the valid and the invalid class) have now been tested.

1
Essential Software Test Design, Torbjörn Ryber 2007.
2
In the following examples and elaborations EP based on input values will be used. However, the examples and elaborations can be extrapolated for EP based on output
values.
3
To keep the explanation as simple as possible, other obvious boundaries (no ages below zero, no alphanumeric input, no non-integer input, etc.) have been omitted in the
example.

38 The Magazine for Professional Testers www.testingexperience.com

Boundary Value Analysis is where a programming error could most eas-
While executing tests it was discovered some ily occur. Therefore this is also the place to test IF age <=99
the most thoroughly. In the Boundary Value THEN accept client
defects seemed to cluster around the values
ELSE refuse client
outlining the equivalence classes. It’s not hard Analysis test technique this means to test with
END IF
to see how this came about: The values outlin- test values on, just below, and just above the
ing the equivalence classes (the boundaries) boundary to detect any possible programming
are being incorporated into the programming fault made in the relational operator (<, <=, =, the defects that could exist regarding the re-
code during programming. In the example >, >=, <>). The use of three values is based on lational operator are programming faults in
above, focusing on the “younger than or equal the following: the <= sign. This operator could wrongly be
to 99 years old”-boundary, somewhere in the programmed as: >, >=, <, <> or = instead of
code it could say:”IF age <= 99”. In our example, that could be written down in <=4,5.
Since this boundary is actually in the code, this pseudo code as:

Fault detected Fault detected

Possible fault Test value Actual Result Expected Result per value using three values
98 Refuse Accept Yes
> 99 99 Refuse Accept Yes Yes
100 Accept Refuse Yes
98 Refuse Accept Yes
>= 99 99 Accept Accept No Yes
100 Accept Refuse Yes
98 Accept Accept No
< 99 99 Refuse Accept Yes Yes
100 Refuse Refuse No
98 Refuse Accept Yes
= 99 99 Accept Accept No Yes
100 Refuse Refuse No
98 Accept Accept No
<> 99 99 Refuse Accept Yes Yes
100 Accept Refuse Yes
The table above shows that a programming fault in the relational operator will always be detected when using three test values (on, just below and just
above the boundary). It also shows some programming faults will not be detected when testing using only two values6.

BVA summary operator? The evidence presented in the table Software Component Testing”, and defines it-
In summary, the use of BVA comes down to: seems to support it. Also, quite a lot of litera- self to “cover only the lowest level of indepen-
1. Identify equivalence classes7 ture shows that testing using BVA should be dently testable software”. It identifies black
2. Determine Boundary Values done using three values8. box techniques, but it doesn’t identify BVA
3. Specify test cases using three test values as being such a technique (nor does it identify
for each boundary. So, is testing with three values really enough? BVA as being a white box technique).
To answer this question, we should find out In daily practice, BVA is used for white box as
The Boundary Value Fallacy if BVA is a white box or a black box testing well as black box testing.
However, is the above correct? Are three val- technique. ISTQB defines BVA as:”A black
ues really enough to test if a failure occurs as a box test design technique in which test cases And that’s where a possible issue occurs:
result of an incorrectly programmed relational are designed based on boundary values”9. Let’s have another look at the example pre-
BS7925-210 describes itself as a “Standard for sented earlier:

4
The incorrect introduction of a “NOT” is not included in the possible defects, since this is covered in the defects mentioned (e.g. “NOT <= 99” equals “>99”).
5
Of course, other defects could be introduced, e.g. typing “999” instead of “99”, typing “99z” instead of “99” etc. These defects will, however, not be discussed since
boundary value analysis focuses on the possible defects mentioned in the text. (To test for the other defects presented in this footnote, other test techniques could be used.)
6
E.g. If “= 99” was programmed instead of “<= 99”, this fault would not have been detected testing only with the values 99 and 100 (since the actual outcome (produced by
the code with the fault in it) would be the same as the expected outcome). The same goes for the incorrect programming of “< 99” instead of “<= 99”. This programming error
would not be detected if the test was executed using only the values 98 and 100.
7
The way in which the identified equivalence classes should be represented (in a picture, a table or even using another technique) is not relevant for this article.
8
Please refer to the Literature reference.
9
Standard Glossary of terms used in Software Testing, Version 2.0 (dd. December 2nd 2007), Produced by the ‘Glossary Working Party’ International Software Testing Qualifi-
cations Board.
10
Standard for Software Component Testing, Working Draft 3.4, Date: 27 April 2001 produced by the British Computer Society Special Interest Group in Software Testing
(BCS SIGIST).

www.testingexperience.com The Magazine for Professional Testers 39

 An insurance system accepts clients aged 99 years old or younger. All other potential clients are not accepted.
Ages are entered into the system under test in whole years11.

The above functional description leads to two classes:

1. People aged 99 and younger will be accepted (this is called a valid class);
2. People aged 100 and older will not be accepted (this is called an invalid class);

This can be represented in a figure (+ means: valid, - means: invalid):

Equivalence Classes:
+
+ -

99
BVA test values:
+
+ -
98 100
99

When contemplating this example, it occurs detect a fault if >, >=, <=, = or <> 100 is pro-
that 99 being the boundary value is based on IF age < 100 grammed instead of < 100?
an assumption. The assumption being that the THEN accept Let’s have a look:
given functional description (“people aged 99 client
ELSE refuse
and younger will be accepted, people older Based on the functional design (being the test
client
than 99 will not be accepted”) is implemented END IF12
basis for a black box test) which states “An
as: insurance system accepts clients aged 99 years
old or younger and refuses to accept clients
IF age <=99 In black box testing, for which BVA is often older than 99 years old”, 99 will be identified
THEN accept used, the former implementation is as good as as the boundary in a black box test13. The three
client the latter: Since it is black box testing, the test
ELSE refuse values to test using the BVA technique will
professional has no insight into the implemen- therefore be 98, 99 and 100:
client
END IF
tation of the functional design.
This leads to the following table when using
However, this need not be the case. The func- However, does BVA still find possible faults in BVA with three values:
tionality could as well be implemented as: the relational operators if the latter implemen-
ta-tion is chosen by the programmer? Does it

Fault detected Fault detected

Possible fault Test value Actual Result Expected Result per value using three values
98 Refuse Accept Yes
> 100 99 Refuse Accept Yes Yes
100 Refuse Refuse No
98 Refuse Accept Yes
>= 100 99 Refuse Accept Yes Yes
100 Accept Refuse Yes
98 Accept Accept No
<= 100 99 Accept Accept No Yes
100 Accept Refuse Yes
98 Refuse Accept Yes
= 100 99 Refuse Accept Yes Yes
100 Accept Refuse Yes
98 Accept Accept No
<> 100 99 Accept Accept No No
100 Refuse Refuse No

11
To keep the example as simple as possible, other obvious boundaries (no ages below zero, no alphanumeric inputs, no non-integer inputs, etc.) have for clarity, again, been
omitted in the example.
12
Again, assuming the input being integer has been thoroughly tested and implemented elsewhere in the code.
13
Even if “If age < 100” is programmed. In black box testing, the tester is unaware of the implementation; hence the functional design is accepted as leading. The functional
design mentions 99, therefore 99 will be identified as the boundary.

40 The Magazine for Professional Testers www.testingexperience.com

As shown in the table above, if “<> 100” was possible fault in the relational operators used. • When the black box test basis states “In-
(wrongly) programmed instead of the intended As shown, the flaw could lead to a wrong test put greater than or equal to X”
“< 100”, the actual results and the expected re- result if “<” was intended to be programmed, • When the black box test basis states “In-
sults are identical for all three BVA test cases. and “<>” was actually programmed. However, put less than X”.
Therefore, the fault will not be detected! there are more situations in which the flaw
occurs14,15: A remedy
Where does this flaw in BVA occur? • When the black box test basis states “In- In the example, the non-detection of the fault
Based on the above it can be concluded that put less than or equal to X” can be easily remedied by adding an extra test
using three test values in BVA will, for black • When the black box test basis states “In- value (101), as shown in this table:
box testing, not always be enough to find a put greater than X”

Fault detected Fault detected

Possible fault Test value Actual Result Expected Result per value using three values
98 Refuse Accept Yes
99 Refuse Accept Yes
> 100 Yes
100 Refuse Refuse No
101 Accept Refuse Yes
98 Refuse Accept Yes
99 Refuse Accept Yes
>= 100 Yes
100 Accept Refuse Yes
101 Accept Refuse Yes
98 Accept Accept No
99 Accept Accept No
<= 100 Yes
100 Accept Refuse Yes
101 Refuse Refuse No
98 Refuse Accept Yes
99 Refuse Accept Yes
= 100 Yes
100 Accept Refuse Yes
101 Refuse Refuse No
98 Accept Accept No
99 Accept Accept No
<> 100 Yes
100 Refuse Refuse No
101 Accept Refuse Yes
By adding the extra test value (101) the fault (“<>” being programmed instead of “<”) will, even when using Boundary Value analysis as a black
box technique, be found (in the table above this is shown for the 99-boundary value used in the example presented earlier).

To extrapolate this example to an overall solution, for every situation in which the fault occurs the tables below suggest a solution.

When FD states: This could be im- But also as: Fault will not be de- Instead of: Fault will be de-
plemented as: tected using BVA if tected when testing
... is programmed with extra value:
“If input less than or equal to X“ IF input <= X IF input < (X+1) <> (X+1) < (X+1) (X+2)
“If input greater than X“ IF input > X IF input >= (X+1) = (X+1) >= (X+1) (X+2)
“If input greater than or equal to X“ IF input >= X IF input > (X-1) <> (X-1) > (X-1) (X-2)
“IF input less than X“ IF input <= X IF input <= (X-1) = (X-1) <= (X-1) (X-2)

In an example:

When FD states: This could be im- But also as: Fault will not be de- Instead of: Fault will be de-
plemented as: tected using BVA if tected when testing
... is programmed with extra value:
“If input less than or equal to 99“ IF input <= 99 IF input < 100 <> 100 < 100 101
“If input greater than 99“ IF input > 99 IF input >= 100 = 100 >= 100 101
“If input greater than or equal to 99“ IF input >= 99 IF input > 98 <> 98 > 98 97
“IF input less than 99“ IF input <= 99 IF input <= 98 = 98 <= 98 97

Please be aware that the (X+1), (X+2), (X-1) and (X-2) in the table below should be interpreted as:
X+1: The closest value on the upper side of X with regard to accuracy and type of the boundary value X.
X+2: The closest value on the upper side of (X+1) with regard to the accuracy and type of the boundary value X.
X-1: The closest value on the lower side of X with regard to the accuracy and type of the boundary value X.
X-2: The closest value on the lower side of (X-1) with regard to the accuracy and type of the boundary value X.
FD: Functional design. “Fault will not be detected” means “Fault will not be detected if using BVA (3 values)”

www.testingexperience.com The Magazine for Professional Testers 41

A solution: B3VA
Now it’s not only possible to conclude there’s a flaw in BVA when BVA is used in black box testing, but based on the tables above a solution can
be presented:
The BVA-technique can, for black box testing, be extended to Black Box BVA (B3VA):
1. Identify equivalence classes16.
2. Determine test values using the Boundary Value technique.
3. Determine the possible operators used:

When the black box design In formula Then add an extra value In formula All test values overview
states17 next to the value on the
“Less than or equal to X” <= X upper side of the boundary X+2 X-1,X,X+1,X+2
”Greater than X” >X upper side of the boundary X+2 X-1,X,X+1,X+2
”Greater than or equal to X” >= X lower side of the boundary X-2 X-2,X-1,X,X+1
”Less than X” <X lower side of the boundary X-2 X-2,X-1,X,X+1
If none of the phrases represented in this the accuracy and type of the boundary value. • 99 is still the boundary value (X); how-
table is used in the design, no extra value E.g.: In the example used before the value 101 ever
needs to be added (all test values then be- is added as an extra upper value because: The closest value to the upper side of 99 would
ing X-1, X, X+1). • 99 is the boundary value (X); then have been 99.1 (X+1).
4. Specify test cases, using three or four test • Integers (non-decimal numerics) are The closest value to the upper side of 99.1
values per boundary, based on the table used. would have been 99.2 (X+2).
in 3. The closest integer to the upper side of 99
therefore is 100 (X+1). The closest integer to Rule of thumb
Please be sure to add possible extra values in the upper side of 100 then is 101 (X+2). When considering the B3VA technique as pre-
the same way boundary values are found, and sented above, an elegant rule of thumb springs
to interpret the X, X+1, X+2, X-1 and X-2 Should one decimal have been accepted in our to mind. For this rule of thumb, please be
–values in the table correctly: With regard to example, then: aware that a boundary value always separates
two equivalence classes18, 19, (the boundary
If this rule of thumb is applied to the example used earlier, it shows the following. value itself being in one of the two classes).
To test the 99-boundary introduced in the example: Keeping this in mind and considering this ar-
+ ticle, it follows that for each boundary value:
1. EP: + -
15 120 1. When testing using Equivalence Parti-
99 tioning (EP):
One (random) value for each of the two
Two test values, 15 (to test the valid class) and 120 (to test the invalid class), are cho- equivalence classes separated by a bound-
sen as test values. The boundary value (99) is not chosen as a test value22. ary value must be chosen as test value;
2. When testing using Boundary Value
2. BVA: + Analysis (BVA):
+ -
• One test value must be chosen (near-
98 100 est to the boundary) in one of the two
99
equivalence classes20.
Three test values are chosen: • Two test values must be chosen (one
• Two values in the valid class (98, 99) are chosen as test values on and one nearest to the boundary)
• One value in the invalid class (100) is chosen as test value23. in the other of the two equivalence
+ classes20
3. B3VA: + - 3. When testing using Black Box Boundary
98 100 101 Value Analysis (B3VA):
99 • Two test values must be chosen (both
nearest to but not on the boundary) in
Four test values are chosen: one of the two equivalence classes20.
• Two values in the valid class (98, 99) are chosen as test values, as in BVA • Two test values must be chosen (one
• One value in the invalid class (100) is chosen as test value, as in BVA on and one nearest to the boundary)
• One extra value in the invalid class (101) is chosen as test value24. in the other of the two equivalence
classes20.
16
The way in which the identified equivalence classes should be represented (in a picture, a table or even using another technique) is not relevant for this article.
17
In any form
18
All examples so far have shown a boundary value separating a valid and an invalid class. However, it can be shown the described also holds when the boundary value sepa-
rates two valid (or two invalid) classes.
19
Please be aware even a boundary value that is a class in itself (e.g. “customers are only accepted if they are 99 years old) can be seen as separating two classes (but
twice: The two classes “99” (+) and “< 99” (-) are separated by the boundary value. Also the two classes “99” (+) and “> 99” (-) are separated by the boundary value).
20
Referring to the two equivalence classes being separated by the boundary value. A boundary value that is a class in itself does not occur for B3VA.
21
If the boundary value is an equivalence class in itself, then values must of course be chosen as follows: 1 on the boundary, 1 in the class on the left of and 1 in the class on
the right of the boundary, but the latter two nearest to the boundary.
22
It could have been chosen though. In that case the valid class would have been tested using the 99-value, dismissing the need to use 15 as a test value (because the valid
class in that case is already tested using the 99-value).
23
The spread of values over the classes is due to the example. In another example, the spread might have been diffe¬rent. (E.g. If “If < 99” was to be (white box) tested using
BVA, 99 would have been in the invalid class. Still, 98, 99 and 100 would have been chosen as test values, thus testing one valid class value and two invalid class values.)
24
In every (!) example using B3VA, two values for each class would have been chosen. E.g. in the example presented in footnote 26: if “Accept if age less than 99 years old”
was to be (black box) tested using B3VA, the extra test value to choose using B3VA would have been 97. The boundary test value 99 would have been in the invalid class. Thus,
97 and 98 would have been tested, being two values in the valid class, and 99 and 100 would have been tested, being two values in the invalid class.

42 The Magazine for Professional Testers www.testingexperience.com

Summary and conclusion boundary is suggested. For strict bound-
In this article the Equivalence Partitioning For example: In the example used in this aries, in an example three test values
technique (EP) and the Boundary Value Anal- article (“a person less than or equal to per boundary are presented (just below
ysis technique were briefly described, and a 99 years old is accepted for an insur- a boundary, on a boundary, just above a
BVA shortcoming was presented: The pre- ance, persons older than 99 years are boundary). However, it is suggested later
sented flaw in BVA can lead to the possible not accepted”), the tester/test manager on that test values in the same class as
non-detection of a fault, and thus to extra risk, could choose to make the: the boundary itself could be omitted “if
if the (black box) test basis contains functional • 98 and 100 test values: the tester does not believe they are nec-
descriptions in the form: to be “Must test” test values (thus essary“. The book stresses the need for
• “… should be less than (or equal to) …” covering EP); equivalence partitioning and boundary
or • 99 test value: value analysis to “be complimented by
• “should be greater than (or equal to) …”. to be a “Should test” test value use of white box and, in many cases, oth-
(thus covering BVA); er black box test design approaches.” The
Subsequently, a solution for this shortcom- • 101 test value: risk involved in BVA black box testing,
ing, the Black Box Boundary Value Analysis to be a “Could test” test value (thus or the need for four test values in some
(B3VA), was introduced and rules for using covering B3VA). (black box) circumstances is not identi-
this technique were presented. It was shown based on the risks involved25, thus fied.
that, by introducing an extra test value adher- deciding to execute a certain test value • TMap Next, Tim Koomen et. al., Tutein
ing to the B3VA rules, the BVA-flaw is solved based on the risk accepted by manage- Nolthenius 2006.
by using B3VA. ment, the time available, test progress This book distinguishes between “nor-
etc. mal BVA” (testing using three values)
B3VA and Risk-based Testing: and “BVA light” (testing using only two
Furthermore, one could argue that knowledge values). It does indicate the risk involved
boundary. All examples given are white
about EP, BVA and B3VA can be used in test in testing with only two values.
box, exactly within the scope described
projects to make a well-founded choice re- However, it does not identify the risk
in the standard:”…so the standard is de-
garding test coverge for black box testing: involved in BVA black box testing, nor
liberately limited in scope to cover only
• Choose EP if the only issue that needs to does it identify the need for four test val-
the lowest level of independently testable
be covered is the correct functioning of ues in some (black box) circumstances.
software. Because the interpretation of
the system under test with respect to its and name for the lowest level is impre-
classes, regardless of the boundaries; cise, the term ‘component’ has been cho-
• Choose BVA if not only the correct func- sen rather than other common synonyms
tioning of classes has to be proven, but such as ‘unit’, ‘module’ or ‘program’ to
also a certain amount of certainty (not avoid confusion with these more com-
100%!) needs to be established that the mon terms and remain compatible with
system under test functions correctly them.”
around and on its boundary values; The standard (therefore) does not elabo-
• Choose B3VA if not only the correct func- rate on BVA black box testing.
tioning of classes has to be proven, but • Essential Software Test Design, Torbjörn
also certainty needs to be established that Ryber, Mediaprint Uddevalla AB, Sverige
the system under test functions correctly 2007
around and on its boundary values. This book deals with “design techniques
… most important for functional tests”
Testing a system under test using BVA implic- and explains “…this section builds on
itly tests this system for EP. the British Standard 7925:2 for compo-
In the same way, testing a system under test nent testing”. The book explains BVA in
using B3VA implicitly tests this system for a short way and suggests in its text and
BVA. tables that two test values per boundary
In a picture: are enough. However, in a graphical de-
scription of BVA for one of the two bor-
ders in the figure, three test values are
This enables suggested.
3
B VA the tester/test However, the risk involved in BVA black
manager to not box testing, or the need for four test val-
BVA only choose ues in some (black box) circumstances is
EP
an appropriate
technique for •
not identified.
Practical Software Testing: a process ori-
Biography
a (black box) ented approach, I. Burnstein, Springer-
René Tuinhout, Logica, is a senior test
test, but also Verlag New York.
advisor and test coach with over 11
to allocate test In this book BVA is introduced as an extra
years of experience in software test-
priorities. to EP, it states “if a tester is viewing the
ing. René advises organizations on test
software under test as a black box with
management, test change management,
well defined inputs and outputs, a
structured testing, and he manages test
Literature good approach to selecting test inputs is
(change) projects.
• British Standard 7925-2, Standard for to use a method called equivalence parti-
René is an ISTQB Practitioner and tu-
Software Component Testing, Working tioning”, thus defining EP (and with that
tors courses in Test Techniques, ISTQB
Draft 3.4, 27 April 2001 also BVA) as a black box technique. For
Foundation Level and several other test-
This Standard uses three test values per testing ranges using only two values per
related subjects.

25
Must, Should and Could are chosen arbitrarily in this example, but should in real life be based on a
product risk assessment, of course.

www.testingexperience.com The Magazine for Professional Testers 43

 Testing Techniques and the Theory of Constraints

by Ladislau Szilagyi
© iStockphoto

Introduction
In the context of newly emerging testing In his bestseller “What is this
schools and methodologies (see Derk-Jan de thing called THEORY OF
Grood’s Test Goal - Result driven testing4), the CONSTRAINTS and how
topics of Test strategy, tactics, testing mission should it be implemented?”1,
are hot and much debated in the worldwide Eliyahu Goldratt, the author
testing community. of TOC, points out the es-
sence of his theory:
Also, in a real software project’s day-to-day
context, there are always constraints affecting 1. Identify the Constraints.
the testing goals’ optimal fulfillment. Remember that to identify
How do we deal with these constraints? Here, the constraints also means
we must use our testing techniques practical to prioritize them according
skills! Appropriate testing techniques must be Fig 1. Constraint example to their impact on the goal.
used as mandatory key elements in this pro- 2. Decide How to Exploit the Constraints.
cess of dealing with testing constraints, and How can we deal with constraints? Here, we Now that we decided, how we are go-
Eliyahu Goldratt’s way of thinking in Theory must use our testing techniques practical ing to manage the constraints, and how
of Constraints may provide us with a proven skills! It is obvious that the mastery of vari- should we manage the vast majority of
methodology to be applied. ous testing techniques can help us find a way the system’s resources which are not con-
out of such situations. New testing techniques straints? We should manage them so that
must be used as mandatory key elements in everything that the constraints are going
What is a testing process constraint? this process of dealing with testing process
According to Webster’s dictionary, a constraint to consume will be supplied by the non-
constraints. However, without an appropri- constraints.
is “the state of being checked, restricted, or
ate methodology, we may randomly jump to 3. Subordinate Everything Else to the
compelled to avoid or perform some action”.
using another (possibly unsuitable) testing Above Decision. It’s obvious we still
I prefer to describe a constraint as “the conflict
technique, and precious time will be wasted. have room for much more improvement.
(between two actions), affecting the optimal
I suggest that the thinking mechanism behind Whatever the constraints are, there must
fulfillment of an intended goal”.
Theory of Constraints may guide us as a prov- be a way to reduce their limiting impact.
en methodology. 4. Elevate the Constraints. But, can we
An example will help! Consider the following
common situation: We are asked to perform stop here? There will be another con-
efficient and effective testing. Apparently, The Theory of Constraints straint, but if we continually elevate a
that means we should obtain high test cover- The Theory of Constraints (TOC) is an overall constraint there must come a time when
age in order to find as many bugs as we can management philosophy. It is based on the ap- we break it. Whatever we have elevated
(effective) and we should minimize test time plication of scientific principles and logic rea- will no longer be limiting the system.
(efficient). In order to obtain high test cover- soning to guide human-based organizations. Will the system’s performance now go to
age, we should execute as many test cases as TOC is one of the favorite buzz-words of the infinity? Certainly not. Another constraint
we can, whilst at the same time, to minimize current business management strategy school. will limit its performance and thus…
test time, we should execute only some of the 5. Go Back to Step1.
test cases (fig.1). That’s a testing process con- Obviously, it is out of the scope of this article
straint; We can’t have both! to offer a detailed description and explana- TOC Thinking Processes
tion of TOC. Instead, we will try to cover only The Thinking Processes emerged as TOC
some of the main ideas behind this theory. practitioners needed to identify the core con-

44 The Magazine for Professional Testers www.testingexperience.com

straints and how to manage or elevate them. ficient and effective
They needed the answers to three simple ques- testing”)
tions: 5. Identify and validate
• What to change? the Assumptions ( in
• What to change to? order to satisfy the
• How to cause the change? Need we must obtain
our Want because of
The thinking process steps of TOC uses some our Assumptions )
exotic terms: 6. Propose Solutions
1. Define the system – build the Intermedi-
ate Objectives Map The assumptions are basi-
2. Analyze the mismatches - build the Cur- cally “why” we must obtain
rent Reality Tree our Want, and finding er-
3. Create a transformation – applying the roneous assumptions is the
“Evaporating Clouds” method key to breaking the conflict.
4. Design the future – build the Future Real- Assumptions “hide” un-
ity Tree derneath the Want-to-Need
5. Plan the execution – build Prerequisite edges, and the Needs-to-
Trees Common Objective edges.
There are also assumptions Fig 3. Propose Solution
The decisive step to obtain the conflict reso- that underlie the Conflict entity itself— why
lution is the so-called “Evaporating Clouds” we believe we can’t have both Wants simul-
method (in fact, a method to “evaporate” con- taneously.
flict). Let’s try to detail the underlying think- Let’s target the “execute only the most impor-
ing process,by taking a look under the hood of tant test cases” Want. We assumed that in order
the TOC Thinking Processes! to “minimize testing time” we must “execute
only the most important test cases”. Well, must
Evaporating Clouds or may execute? What if we imagine that per-
A “Cloud” is based on Necessary Condition haps there is another way to minimize testing
Thinking: we must “execute as many test cases time? Here, our testing skills shall guide us to
as we can” (wanted) as a necessary condition search for alternatives…and one alternative is
to obtain “high test coverage” (needed). We the risk-based testing approach! Yes, we could
have already seen that when two “wants” ap- make a quality risk analysis (see5), prioritize
pear to be mutually exclusive, there is a con- the product features and come up with testing
flict. To solve this conflict, the way forward is tactics involving various levels of testing (ex-
to recognize that our “wants” are motivated by tensive, balanced, according to opportunity)
underlying “needs”. We “want” to execute as and implying different time slices assigned
many test cases as we can because we “need” to the test cases, depending on the risk of the
feature being verified (“time-boxing the test Fig 4. “Evaporated” conflict
to obtain high test coverage.
cases” fig.3). tree of inter-dependent testing goals (fig.5).
Let’s build a graphic representation of our Oops, we just got our Solution, it satisfies our
Conflict (fig.2). need to minimize the testing time (fig.4)! Tactics, on the other hand, are supposed to
tell us “How we are supposed to reach the ob-
Strategy, tactics and the practical jectives.” In other words, tactics answers the
use of TOC to remove testing pro- “How?”. For any meaningful testing action
cess constraints we should be able to ask: “Why are we do-
When we discuss the testing process, it’s ing this? What is its purpose?”. The answer to
best to consider the whole picture. In the these questions is the strategic goal. Likewise,
test process, the following levels define for any meaningful testing strategic goal we
and apply test activities: should be able to answer: “How do we ob-
tain this? What actions are needed in order to
1. Test Plan (scope, approach, resourc- achieve it?”. The answer to these questions is
es and schedule of intended test the tactic action. That means that there must be
activities)
2. Test Strategy (high-level de-
scription of the test levels to be
performed and the testing within
those levels)
3. Test Tactics (practical ac-
Fig 2. Needs and wants tions performed by the tester
when testing)
The steps to follow in order to apply the
“Evaporating Clouds” method are:
Strategy and tactics are different en-
1. Identify the Wants (“execute as many test
tities and are planned and performed
cases as you can”, “execute only the most
in distinct time intervals. Strategy is
important test cases”)
setting the goal. In other words, the
2. Identify the Conflict (can’t have both!)
strategy sets the “What for?”. There is
3. Identify the underlying Needs ( “obtain
no such thing as a ‘unique test goal’,
high test coverage”, “minimize testing
but obviously there is a primary goal
time”)
and there are many secondary testing
4. Identify the common Objective (“ef- Fig 5. Testing goals tree
goals, more exactly represented as a

www.testingexperience.com The Magazine for Professional Testers 45

a corresponding tactic action behind any stra- constraints connected to the test activities, technique in the process of ‘evaporating the
tegic goal, (fig.6). where often the constraint’s root cause is the clouds” of the constraint’s conflict. In any such
wrong option for an inappropriate situation, try to draw a quick “needs-wants”
testing technique. Here we are for- graph and then start to identify the wrong as-
tunately on solid ground, it depends sumptions, leading to the “evaporating cloud”
only on us, the testers, to remove solution candidates, selected from the testing
these constraints. techniques you know. Always try to select a
“need” that can be satisfied using some other
The ISTQB Foundation & Ad- testing technique than the one currently used.
vanced syllabi3 describe a complex
taxonomy of testing techniques. For instance, we may be confronted with the
These are the possible candidates task of testing a dialog which is used to con-
to be used when searching for alter- figure a system, and containing combinations
natives in the “evaporating clouds” of groups of radio-button choices. Here, the
method. In our previous example constraint may emerge as the conflict between
on how to remove a constraint, we too many combinations to be tested and the
saw a ‘high-level’ conflict situation limited testing time available. As possible “so-
(“execute as many test cases as you lutions” we may use some testing technique
Fig 6. Strategic testing goals and tactic actions tree can” versus “execute only some of candidates:
the test cases”). • combinatorial testing techniques (or-
Usually we cannot modify the test strategy thogonal arrays, all-pairs testing, t-way
during the testing session. The right time to testing6, including the use of tools like
update it will be after the testing session, when Functional Testing techniques: AllPairs, PICT, FireEye)
we learn the lessons (success or failure) from 1. Specification based (Black box) • cause-effect graphing and decision ta-
the testing mission. However, if we are do- • Equivalence partitioning bles, to identify particular rules that could
ing Exploratory testing, we may modify our • Boundary values analysis be hidden behind some particular groups
strategy during the testing session according • State transition testing of combinations
to what we have learnt up to now. In either of • Cause-effect graphing • classification tree method, to quickly vi-
these two cases, at the stage of performing the • Decision tables testing sualize test ideas and find relevant test
practical test tactic actions, we will be surely • Syntax testing values
confronted with constraints affecting the test- • Classification tree method
ing goal’s optimal completion. • Orthogonal arrays, All pairs Keep in mind that the testing context (avail-
• Use Case testing able time, test environment, tester experience,
Let’s look more closely at the various catego- 2. Structure based (White box) client expectations and feature criticality) will
ries of testing process constraints. Among the • Statement testing have a serious influence in our choice.
types of constraints impacting the testing mis- • Decision testing
sion’s optimal completion, most of the types • Branch testing Well, to be honest, our quest does not stop
are related to: • Condition testing here. There is usually more than one constraint
1. Time (too short) • Multiple condition testing affecting the completion of the testing mission.
2. Cost (too high) • Condition determination Our success will be critically influenced by our
3. Quality (expectations too high – zero de- testing ability to pick the most important constraint!
fects!) • LCSAJ Here, too, TOC has developed a methodology
4. Resources (lack of, insufficient, not • Path testing to help us identify the critical constraint (but,
trained) 3. Defect based that’s another story to be told…please consult
5. Test environment (incomplete, not set • Taxonomy the referenced books on TOC1,2). And, finally,
up) 4. Experience based remember that the testing constraints removal
6. Test tools (lack of, not suited, difficult to • Error guessing is a continuous process, like the Deming’s
maintain) • Checklist based PDCA cycle. Once we removed a constraint,
7. Test activities (none or wrong testing • Exploratory we will surely find another constraint that af-
technique applied) • Attacks fects our testing mission’s optimal comple-
5. Static analysis tion.
The first six categories of constraints must be 6. Dynamic analysis
handled by test managers in their continuous Keep TOC’s thinking processes as a valuable
risk mitigation activities throughout the test- Non-functional Testing types: asset in your first-aid testing kit!
ing process life cycle. This is not always an 1. Accuracy testing
easy task; you must admit that it’s hard for a 2. Suitability testing
3. Interoperability testing
References
test manager to: 1. Eliyahu Goldratt - Theory of Constraints,
• request some more testing time (testing 4. Functional security testing
North River Press
cost is implied too) 5. Usability testing
2. William Dettmer - Strategic Navigation:
• criticize a manager for a wrongly imposed 6. Accessibility testing
A Systems Approach to Business Strat-
‘good enough’ level of product quality 7. Technical security testing
egy, ASQ Press
• ask for an extra tester to help the current 8. Reliability testing
3. ISTQB - Foundation & Advanced level
testing mission 9. Efficiency testing
syllabus
• ask for some more hardware to set-up the 10. Maintainability testing
4. Derk-Jan de Grood - TestGoal, Result
test environment 11. Portability testing
driven testing, Springer
• convince the company’s top management 5. Rex Black – Critical Test Processes, Ad-
to buy a new testing tool Obviously, at the tactical ‘low-level’ in our dison-Wesley
• instantly become an expert in using a new testing session, we will encounter different 6. Yu Lei & others - IPOG: A General Strat-
testing tool kinds of conflict, much more dependent on egy for T-Way Software Testing, Engi-
the current testing context. Here, we must rely neering of Computer-Based Systems,
I will focus here only on the last category, the on our ability to choose the suitable testing 2007

46 The Magazine for Professional Testers www.testingexperience.com

 Index Of Advertisers

ASTQB 19
Business Innovations 28, 61
Chouchair 64
Compuware 17
Díaz & Hilterscheid 35, 79, 90, 100
dpunkt.verlag 49
Gran Canaria 99
iSQI 23, 31, 69, 75
ISTQB 50-51
Kanzlei Hilterscheid 86
PureTesting 93
RBCS 68
Biography Sela Group 2
Ladislau Szilagyi holds a BS in Computer SQS 66
Science & Mathematics, 1978, Bucha- TE Knowledge Transfer 6
rest University, Romania and has more
TOBIUS 37
than 30 years of working experience in
IT. He worked until 1991 at the Research T-Systems 20
Institute for Computer Science, Bucha-
rest, authoring multitasking real-time
kernels and process control systems. He
has been involved in software testing
since 1995 and now works as Test Man-
ager at Totalsoft, Romania.

His main specialized areas of consulting

and training are Quality Management,
Testing and Software Process Improve-
ment. He contributed several articles to
the Carnegie Mellon’s Software Engi-
neering Institute Repository (https://seir.
sei.cmu.edu/seir/), covering topics such
as CMMI, Testing, Metrics and Theory of
Constraints. He is ISTQB CTFL, member
of ISTQB South East European Testing
Board and an active ISTQB trainer.
© iStockphoto

Subscribe at

Pantone 350 www.testingexperience.com Pantone 362�

100% 80% 60% 40% 20% 100% 80% 60% 40% 20%

cmyk cmyk

www.testingexperience.com The Magazine for Professional Testers 47

 Column

Improving the testing process using

the TMMi
by Erik van Veenendaal

The holiday period is always a good time to rethink issues. My column was this time
written on the beautiful island of Bonaire in the Dutch Caribean. Facing the ocean
Erik van Veenendaal is a leading and drinking a local beer, I’m trying to focus my mind on test process improvement
international consultant and trainer, ………
and recognized expert is the area of
software testing and quality manage- Different angles
ment. He is the director of Improve Since the latests developments around the TMMi are a hot topic in the testing com-
Quality Services BV. At EuroStar 1999, munity, this is time to dedicate my column around “improving the testing process”.
2002 and 2005, he was awarded the Interesting to note that the first ISTQB expert level to be released (scheduled for
best tutorial presentation. In 2007 he early 2009) is called “Improving the testing process”. Many organizations, whether
received the European Testing Excel- using traditional development models or applying a type of agile development, are
lence Award for his contribution to the trying to improve their testing. To me there are various angles to work from. The
testing profession over the years. He one that seems to get most attention is process orientation. Before exploring this
has been working as a test manager more in depth I would like to point out that other approaches such people-oriented
and consultant in software quality for (“just make sure your testers are top of the class and they will do a better job”) or
almost 20 years. automation-oriented (“using a unit-test framework and/or automating the regres-
He has written numerous papers and sion tests are proven ways to improve the effectiveness and efficiency of testing”).
a number of books, including “The Don’t just focus on processes, balance your improvement efforts!
Testing Practitioner”, “ISTQB Founda-
tions of Software Testing” and “Test- The need for improving the testing process
ing according to TMap”. Erik is also a Many organization use a reference model for test process improvement such as TPI
former part-time senior lecturer at the (related to the Dutch testing standard TMap) or TMMi (related to the well-known
Eindhoven University of Technology, the process improvement model CMMI). Reasons for test process improvement are by
vice-president of the International Soft- now clear to most professional testers:
ware Testing Qualifications Board and • Larger and more complex systems are being developed
the vice chair of the TMMi Foundation. • IT-systems that play a critical, and sometimes even safety critical, role in the
society
• Despite numerous attempts system and software development have not been
able to produce low defect density systems
• The amount of time spent on testing in large projects is at least 30%, but often
it sums up to 40% or even 50% of the total project effort

In addition we are facing challenges due to short time-to-market requirements,

strong competition, new technologies and of course everything around outsourc-
ing. In short improving the testing process is not an option; it is mandatory for those
who want to survive and be successful in business. I do not care so much whether
one uses TPI or TMMi, both have their strong and weaker areas. More important is
the right approach consisting of amongst others a clear mission or policy (“what is
it that we are trying to achieve?”), a thorough improvement plan, a practical angle,
management commitment and adequate resources.

New developments around the TMMi

Traditionally TPI has been strong in various industries, such a finance. Recently
TMMi is gaining “market share” very rapidly. One of the main reasons being its
strong relationship in every way to the CMMI. Achieving TMM level 2 almost au-
tomatically ensures compliance with the CMMI level 3 process areas Verification
en Validation.

48 The Magazine for Professional Testers www.testingexperience.com

 The TMMi Foundation has recently been established to co-
ordinate, speed up and professionalize the developments
around the TMMi. Most important areas of attention are:
1. Full development and maintenance of the TMMi model.
(Currently the TMMi framework and level 2 definition are
freely available at www.TMMiFoundation.org.)
2. The development of a formal certification scheme. Like
with the CMMI it shall be possible to formally rate accord-
ing to a predefined procedure the TMMi maturity level of an
organization. Of course the requirements for such a scheme
are highly related to the experiences with the CMMI.
3. The establishment of requirements for TMMi (lead)
assessors; those that formally want to perform TMMi as-
sessments. One needs to have enough knowledge and skills
regarding testing, auditing, process improvement and of
course TMMi.
4. The TMMi Foundation will in addition be a forum
where persons and organizations can exchange experiences
and run discussions regarding the TMMi and its usage in
the real-life.

Results and uptake

All together a development that will enhance the testing pro-
fession. A test department can now achieve a formal, but most of all meaningful, quality mark. I’m aware of many companies that are involved in
testoutsourcing that are studying how to become a formal TMMi accredited organization.

I already know organizations that during their road to higher testing maturity have been capable to decrease the test execution lead time and at
the same increase their test effectiveness. They also support this claim by having concrete metrics!. Is this utopia? I don’t think so with the right
approach and attitude it can be done!!

Back to my beer and a refreshing dive in the ocean….

For queries or comments regarding ISTQB expert level, TMMi or the TMMi Foundation you can contact Erik van Veenendaal (eve@improveqs.
nl)

'RAHAM"ATH*UDY-C+AY !NDREAS3PILLNER4ILO,INZ

4HE3OFTWARE4EST%NGINEER³S(ANDBOOK "ASISWISSEN3OFTWARETEST
!3TUDY'UIDEFORTHE)341"4EST!NALYST !US UND7EITERBILDUNGZUM#ERTI½ED4ESTER
AND4ECHNICAL4EST!NALYST &OUNDATION,EVELNACH)341" 3TANDARD
!DVANCED,EVEL#ERTI½CATES ~BERARBUNDAKT!U¾AGE
PAGES3OFT#OVER 3EITEN&ESTEINBAND
53 E$
)3".     )3".    

.%5
+LAUS(yRMANN-ARKUS-~LLER 4IM+OOMEN,EOVANDER!ALST
,ARS$ITTMANN*yRG:IMMER "ART"ROEKMAN-ICHIEL6ROON
!UTOMOTIVE30)#%IN0RACTICE 4-AP‡.EXT
3URVIVING)MPLEMENTATIONAND!SSESSMENT 0RAKTISCHER,EITFADENF~R
PAGES3OFT#OVER ERGEBNISORIENTIERTES3OFTWARETESTEN
53 3EITEN&ESTEINBAND
)3".     E$
)3".    
2OCKY.OOKDPUNKTDEVELOPMENT

*UDY-C+AY *OHANNES,INK
-ANAGINGTHE4EST0EOPLE 3OFTWARETESTSMIT*5NIT
!'UIDETO0RACTICAL4ECHNICAL-ANAGEMENT 4ECHNIKENDERTESTGETRIEBENEN%NTWICKLUNG
PAGES3OFT#OVER ~BERARBUNDERW!U¾AGE
53 3EITEN"ROSCHUR
)3".     E$
)3".    

WWWROCKYNOOKCOM

2OCKY.OOKISREPRESENTEDBY/2EILLY-EDIA)NCANDBYDPUNKTVERLAG4OORDERPLEASECONTACT

INTHE53AND#ANADAORDER OREILLYCOM 2INGSTRA†E"À$ (EIDELBERGÀFON
INTHE5+%UROPE-IDDLE%ASTAND!FRICAINFORMATION OREILLYCOUK FAXÀE MAILBESTELLUNG DPUNKTDE
WWWDPUNKTDE
www.testingexperience.com IN'ERMANY!USTRIA3WITZERLANDBESTELLUNG DPUNKTDE
The Magazine for Professional Testers 49
© iStockphoto

A Community

90,000 Cert
Are you on the

www.ist
*July 2008
50 The Magazine for Professional Testers www.testingexperience.com
y Worldwide!

tifications*
e right track?

tqb.org
www.testingexperience.com The Magazine for Professional Testers 51
 10 tips for successful testing!

by Dominic Maes and Steven Mertens

© iStockphoto

Software development is often a balanced 2. Select a testing method which fits zation is large enough (i.e. multiple IT projects
exercise in time and quality management. your needs per year) you can even consider setting up a
For most companies it is a constant struggle Iterative development, e.g. V-Model, Agile, test factory, a separate department conducting
to complete tasks on time and within budget, Extreme Development… Each Software de- independent and uniform testing of all your
usually resulting in quality being neglected. velopment process needs its own adapted test existing applications and new projects.
When applications are finally released into process; select the testing method correct for
production, users are often confronted with your application. You and your management 4. Review, review, review, review …
only basic functionalities or even the complete team will benefit from a process that not only Testing should start early in the development
omission of functionalities due to the lack of reveals the business/project risks but provides process, you cannot afford to wait for code to
development time. the comfort of knowing that the testing team start testing your application. Make sure you
How do you set up or improve your software has mitigated these risks. Pay attention to your start reviewing your documents right from the
testing process to reduce the business risk of project specific characteristics, there is no such outset; as well reviewed documents tend to re-
poor application implementation? thing as a ‘universal test approach’. Processes sult in 65% less defects being carried forward
Here are a few simple tips to get you ahead in that work for one company or project may be into later phases of the development cycle.
quality, saving you time and money: totally inappropriate for another. This can amount to huge saving of time and
• Do you need to react quickly and make money. Don’t ‘just send’ your documents ask-
1. Start early and be prepared! lots of changes? ing colleagues to review them without expla-
The key to a successful software develop- • Is the goal of the project to add function- nation or preparation, instead give individuals
ment project is preparation. Projects often ality to an existing application? specific assignment on the documents that you
start as simple ideas with enthusiastic devel- Many different aspects of your development want them to review. Have someone check the
opers committing hours of effort prior to in- process can influence the correct choice of the technical aspects, another, the documents on a
depth investigation, this results in unforeseen testing approach. No matter which direction higher level and yet another against the layout
problems causing delays and project overrun. you choose, ensure that your test process is standards used within the company. This way
Testing should be introduced from the outset, structured and started at an early stage. you can avoid holding review meetings where
this way you have an objective look at the only grammar mistakes have been highlighted.
project feasibility and its risks prior to com- 3. Be objective Reviewing is an exhausting task, so only re-
mitting valuable effort. In addition the test or The test team should have an autonomous role views for short periods, one hour maximum,
quality department should be involved at an within the organization as it is often difficult at a review rate of 10 pages for the period,
early stage to review requirements and design to work within the project itself. Some project make sure you are not disturbed. Avoid doing
documentation. Reviewing helps to highlight managers tend to neglect the test teams advise reviews in a crowded or noisy office environ-
defects and if conducted early in the process when their project deadline is at stake and ment, close your e-mail…. Make sure you be-
will save money by reducing the cost of ex- quality is no longer important. When there is a come an expert in your reviewing methods. It
pensive repairs or corrections. Defects found delay in a development, many project leaders takes time to learn ‘how’ to review and code in
in the requirements phase are on average 100 also have a tendency to use the time allocated a professional and organized way.
times cheaper to correct than the same defects for test execution to perform further develop-
found in production. Preparing your test ex- ment activities, hence testing is usually skipped 5. Prioritize
ecution is paramount to getting your project and not just postponed. This can have serious It is generally accepted that testing everything
completed in the shortest possible timescale. consequences for the overall quality of the is impossible. In a normal application or sys-
Valuable time can be saved during the test ex- project which often ends-up in rework when tem, there are far to many functionalities and
ecution phase by specifying your test scripts the application is placed in production. The to many possibilities to test every single one
and creating a test scenario before the applica- additional work results in delays and further of them. An application can take several man-
tion under test is delivered. cost to the overall project. When your organi- years to test so it is important to test those

52 The Magazine for Professional Testers www.testingexperience.com

aspects of your application where the risk of In our experience, NOW is a good time to es- phase. You should always have your ‘old’ test
damage or frequency of use is expected to be tablish metrics, the sooner you start collecting cases ready for immediate re-use, this way you
the highest. A thorough risk assessment should data the faster you can start improving. can save 30% to 50% in overall test effort.
be performed before setting up the test strate-
gy. Select test techniques according to the risks 7. Know when to stop testing. 9. Communicate
you want to reduce. Build a strategy based on When can you suspend test execution? In many companies there is an ‘It is not MY
the risks that are involved in the project. As a rule, you should stop ’Bug Hunting’ responsibility’ attitude. Communication is
Remember: No Risk, No Test. when the cost of finding an additional defect poor between the end-users, architects, devel-
is no longer in balance with the risk you’re opers, testers and management. Too often this
6. Measure! exposed to. Under normal circumstances, you kind of behavior leads to misinterpretation in
Two important questions which should be need to set exit criteria to decide in advance requirements, unwanted functionalities and
posed. when testing should stop, for example, define unsatisfied end-users. Communication is key
• What do you know about your testing a criteria that will stop testing when 90% of to ‘quality’.
process? the available test scripts has been executed Listen to the end-users and get involvement of
• What do you want to know about your with a maximum of 5 open defects. Make sure the complete team to obtain the best results.
testing process? your project criteria are clearly defined. Dis- Be diplomatic and have constructive discus-
Most of the time, it’s easy to establish the cost cuss your exit criteria at the start of the test sions.
to solve a defect in your application (at differ- project and incorporate them within your test Software development is constantly balancing
ent stages), but what is the cost and how effi- plan, this will avoid discussions at a later date. budget, quality and scope.
cient is your team in finding these defects? When no exit criteria are defined, testers tend
The key to answering these questions is: - start to continue testing until all defects have been 10. Last but not least: keep it simple!
measuring. found or solved. Often companies become lost in administra-
Gather data to monitor your test process, ef- tive procedures when considering testing. An
ficiency and effectiveness of your test depart- 8. Preserve. efficient test process should excel in its sim-
ment. The strength of software testing partially lies plicity, no mountains of red tape, nor tedious
Utilize metrics which provide the information in the structured preservation of testware. ways of giving your quality team and develop-
you need to know about your process, i.e. test On completion of testing, take some time to ers a hard time. Choose an approach that fits
scenario density, defect density, number of de- evaluate and assess your deliverables and pre- your project or company and does the job: as-
fects found in the first month of production, serve them for future use. Test cases should be suring quality for your applications.
etc… stored in such a way that they can be reused in
The question is often asked ‘When is the best later projects- or even during the maintenance Good luck!
time to start measuring?’

Biography

Dominic is a senior test consultant at TOBIUS NV. Steven is Manager of the Competence Center Testing at TOBIUS
Dominic has a master degree in commercial and financial sci- NV. Steven has a bachelor in management and insurances. Since
ences. Since 1997 Dominic is working as a professional test con- 1998 Steven gained experience in various industries such as
sultant. He started from junior tester over test coordinator and test the pharmaceutical industry, gas & electricity, social security,
manager to an expert level, specialized in both test processes and insurance, banking, retail, telecom, industry and government in
tools. Dominic gained experience various industries and sectors, Belgium, the Netherlands and Luxemburg. He has an excellent
such as retail, telecom, manufacturing, finance and government knowledge of testing methodologies such as TMap®, STBox™ and
in Belgium and the Netherlands. Dominic has excellent knowledge Test Process Improvement frameworks. Steven is one of the three
of TMap® and TMap Next®, and is both ISEB Practitioner and creators of STBox™. He has experience in coaching people, giving
TMap® Certified. He is a requested speaker at colleges, universi- training, test management and team leadership, determining test
ties and seminars. strategies, writing test plans and practical implementation of test
methodologies. Steven is ISEB Practitioner Certified since 2005.
He is a frequent speaker at colleges, universities and seminars.

www.testingexperience.com The Magazine for Professional Testers 53

 Explaining Security Testing as Real Trousers

by Bogdan Bereza-Jarociński
© iStockphoto

Motto: with the wholesaler who placed this enquiry, Security Definitions
“I reckon our bard wasn’t expecting flames it was clear that no description of security re- Here are some notes that were found by the
to shoot out of the floor unexpectedly,” said quirements existed - it was up to the producer cleaners on the boardroom floor in the morn-
Cohen. to convince the wholesaler that the product ing of the next day – they were the result of a
Truckle shrugged theatrically. was both “safe” and “secure” - and to create prolonged brainstorming activity.
“Well, if you’re not going to expect unexpected the definition of security on the fly. “Colour matters, but not for security - or
flames, what’s the point of going anywhere?” Some chaotic discussion at the clothes maker’s should it be well visible, like signal red for
(Terry Pratchett “The Last Hero”) boardroom followed. example?”
- Secure trousers must not endanger moun- “Safety = trousers do not malfunction in a way
Abstract taineers’ safety. This means perhaps that they which puts their user in danger. If they for ex-
“Can you perform security testing for us? “, have to have two legs, not just one leg, and be ample fall apart while stored in a storeroom
asked a prospective customer. waterproof enough to protect their users from which is too damp, this is a problem, but not a
“Sure, we are test specialists. Give us your se- rain and snow - said the Chairman. safety problem”.
curity requirements, and we will test them! “, - Sorry, but I think two trouser legs is more Some caps from beer bottles were found in
was the answer. like trousers’ functionality, not their “secu- the dustbin. They were the result of a discus-
In reality, the situation is not that simple. In rity”, whatever it means. Security would be sion on the difference between mountaineers’
this respect, security is like usability: you can rather that these legs are long enough to go all “safety” and “security”, which continued until
seldom expect customers to know all about it the way down to the boots, and wide enough the wee hours and was terminated only when
and get a free ride on their requirements. You to have room for thick fleece trousers under the amount of consumed beer seriously ham-
may have to provide your customer with infor- them - said the Vice-chairman. pered the participants’ ability for deep philo-
mation about what “security” actually is and It must have been a democratic company if sophical reasoning.
how it should be tested. anyone ventured to challenge the Chairman’s “Our problem is, we have to prove that trousers
Besides, whereas security requirements may opinion, which was good - otherwise this story are secure”. The word “prove” was crossed out
be simple (“no unauthorised access shall be could not have been created at all. and “test” written above it; someone pencilled
possible”), testing for compliance with them - “I guess you have a point here”, the Chair- “verify” beside the whole sentence.
may not be. Security testing is very much man agreed. “We have to know what secure means, then we
looking for unexpected side-effects were no- - However, what you tell is functionality, too. have to prove that it is”.
one (except hackers or crackers) expects them, “Sure enough, trousers for mountaineers must “You cannot prove it unless you actually use
which requires minute technical knowledge cover them well enough to give protection them”- note was still glued to another note,
and plenty of “error-guessing”. For some rea- against the elements, and have room enough stating that “too expensive to prove first when
son, all this is called “penetration testing” in a to be used over all other clothing, but this does all is ready for mass production, we must prove
security context. not mean they are necessarily secure. Secure it from the very beginning, otherwise wrong
trousers must be reliable: they must not break fabric and thread may be ordered!”.
and fall apart, for example, or dissolve in cold An officially looking document lay on the
What Is “Security”?
water, or have parts you can stumble on etc.”, table beside the Chairman’s leather armchair.
There was once a small company making
continued the Chairman. It was mostly empty, but contained one phrase
clothes. One day they got an interesting pro-
- “OK, folks, we cannot solve this now. Let’s printed in fat style: “What is meant by secu-
posal: to produce jackets and trousers for
have a meeting in the afternoon - is 3 o’clock rity is defined by the way we have verified it,
mountaineers. One clause in this proposal re-
OK for everybody? - and let’s have some tech- which is described in Appendix 17”.
quired that “you must prove that your products
nical people with us”, concluded the Chair-
fulfil our safety and security requirements”.
man.
However, no such requirements were explic-
itly stated. After some telephone conversations

54 The Magazine for Professional Testers www.testingexperience.com

Where to Look for Security (and Safety) reality happen in the mountains, and my tech- - “We do have a lower proposal”,, continued
Bugs nical person would need to guess what can go the technical manager, “complete security test-
Rumours about the trouser issue started to wrong from the technical perspective. When ing for only 700 thousand, but I really believe
spread across all levels of the company. Em- something breaks, it will be necessary to deter- the risk would be too high. And a really ex-
ployees began to discuss it. An overheard con- mine why - again, technical experience would pensive one, for three million, but that would
versation was recorded: be necessary for that”, thought the production mean risking that the trousers are too expen-
- “You know, Frank, this trouser security manager. sive or unable to be sold, or our profit margins
stuff reminds me of when I once tried to in- - “And the difficult part”, continued the thought dangerously low.”
stall some extra electrical appliance in my car process in the production manager’s harassed Really impressive, the way the technical man-
and thought I could save some money by do- brain, “is that testing in the mountains requires ager convinced the Managing Director. He
ing it myself. I had to connect some electrical whole trousers as well as real or at least re- was well prepared for this meeting and pro-
wires and it worked perfectly! However, you alistic imitations of mountain environment. duced convincing, supporting numbers with
got an electric shock in your fingers when you My technical person will have to follow my his reasoning. Just like we are used to in SW
touched anything in the car afterwards. I kind mountaineer everywhere and keep an eye on testing in general, aren’t we?
of did too much: the stuff worked all right, but what happens with the trousers. Difficult, that
it did something more, too. And what it did one! “ Dragon’s Voonerables
was definitely not safe for you!” “What went wrong before”, said some pessi-
- “So you mean, for these trousers it could be How Much to Test Security? mistic employee at our clothing manufacturer,
something like we make them good, but some - “Never!” shouted the Managing Director and “may go wrong in future, too. “
idiot adds a build-in parachute which opens rammed his fist into the shiny desktop. Some impressionable manager who happened
when wind is strong enough. And then some- - “Never, never, never!” he repeated and shook to be within earshot thought it was a great idea
one gets injured because this parachute opens the offensive document violently. and suggested creating a list of all remembered
suddenly in strong wind during climbing. You It was the trouser development project’s bud- past security-related failures, then making
do too much, and it may result in things becom- get proposal. a checklist out of it. Here are some excerpts
ing less secure than when you do too little.“ - “Two bloody million just to be able to tell the from this checklist.
- “Absolutely. This security stuff will be a bit -ing wholesaler that our trousers are secure? “
like finding out where you can expect unex- came the next broadside. Implementation Faults
pected things to happen. And, let me confess, I - “Let him put his ass into those trousers him- A consignment of party dresses was once sewn
even checked the difference in the dictionary, self and let him go to the mountains, he would with seams so flimsy that some dresses simply
and it looks to me now like some things may be able to see for himself if they are good fell to pieces when partying was wild enough.
be unsafe when insecure, and some may be in- enough or not!” In some cases users considered this almost a
secure when unsafe!” - “The problem is”, retorted the technical required functionality, but in some cases a se-
The Company’s network administrator who manager, “that the wholesaler wants to be curity (and safety) hazard.
had worked for an Internet company before it convinced that our trousers are secure without Metal zips were once used in winter jackets.
went bust two years ago. having to make a mountain trip himself. Be- There were complaints that they sometimes
- “Security without network connection is sides, he says that for his customers, his own froze and could not be opened. Just a minor
easy”, he said. mountain-walking habits may be not enough. flaw for ordinary clothes, perhaps a security
- “You cannot easily compare trousers to com- Many customers are half his age and three (and safety) issue for mountaineering clothes.
puter systems, because they are not connected times his physical temperament, and they have It was decided that only plastic zips will be
to anything or to one another. If they were, to be convinced that our trousers will serve used, and that they will have to be tested for
anyone could at any time try to make your them even in extreme conditions. “ durability in low temperatures.
climber’s trousers disappear, or put patches on There was silence, an ill-omened silence, and
them where there should be seams, or simply finally Managing Director asked very, very Insecure Design
get into the same pair of trousers while some- quietly: A similar failure list was created for design-
one uses them already. This is about what we - “OK then, if it is so important, why spend related failures. Generally, it was agreed that
call “viruses” and “hacking” in our computer just two million? Why not ten million? If we the shape of trousers must give maximum mo-
world. With trousers, it will be easy: no con- did that, we would be still surer that we really, bility and must not create any points of high
nectivity, no software configurations… really really know how secure our trousers are?” tension and risk of fabric splitting. Besides,
easy!” But sarcasm did not bite. The answer was all details such as zips, buttons and clips must
prompt and very much to the point. be designed so that they can be operated even
Security Testing - “Well, this is exactly how we reasoned while with thick gloves. Ahem... was it functionality,
Because security seemed to be very important making this budget. We got three mountaineers perhaps?
to the wholesaler, the manager responsible for here on a short-contract basis and they made
production and quality assurance decided to for us what we call a “usage profile”: how we Unexpected input, shared code libraries, In-
assign a separate team to this job. “The first guess these trousers will actually be used. Then ternet applications
thing”, he thought, “is that you have to ensure we made an estimation of how much security Putting a hot gasoline stove instead of your leg
that the trousers do not create any dangers is needed. You can make perfectly secure into a trouser leg could definitely be classified
specific for mountaineering. That means you trousers, but they would cost a huge amount as an “unexpected input” and as quite a major
need someone who knows about mountaineer- of money to produce. So our estimation was hazard, but it was decided as unavoidable.
ing, otherwise it would all be based on blind based on how much extra we guessed custom- On the other hand, other “unexpected inputs”,
guesswork. On the other hand, it is not enough ers would be willing to pay for trousers with such as different kinds of fabric used for un-
to ask a mountaineer, who knows much about our “security certificate” logotype. And how dergarments coming into contact with trou-
what he or she does in the mountains, but per- much these trousers would be allowed to fail sers’ fabric was considered a legitimate safety
haps little on how trousers are made. A moun- under extremely extreme conditions without issue.
taineer can make the trousers break - good if it compromising the value of the certificate. Fi- Decision was unanimously taken to have a
happens before mass production has started! - nally, we tried to figure out how and how much closer look at other forms of unusual usage
but he cannot tell us why it has happened: Was to test in order to achieve that level of quality with possible safety and security consequenc-
the fabric too weak, or the thread, or the seams with sufficient confidence. And two million is es, such as what happens when the trousers are
too thin, or does perhaps the sewing machine’s what we got. “ left for several days in a frozen tent.
needle weaken the fabric?” More silence, but not so much ill-omened as
- “A mountaineer needs to guess what can in before.

www.testingexperience.com The Magazine for Professional Testers 55

Usability forget to test the resistance of the whole trou- compliance with known requirements. Many
Usability, defined as “ease of use” and “suit- sers, and not just their components. Direction requirements were not security-related (price,
ability for intended usage”, is indeed impor- of typical pressure, points where extra pressure availability, pattern, suitability for produc-
tant for mountaineers’ clothes! But who knows is applied (like knees) must be identified: in tion process etc.). Some were security-related,
what matters for usability? Definitely not the other words, you need an “operational profile” such as tear resistance, resistance to wind and
clothing manufacturer, and not the wholesaler. for trouser users, just like for Internet users! water, characteristics in prolonged damp con-
It is the end-user’s opinion that matters. ditions and in temperatures below zero.
Here, the mountaineers who belonged to the Organisational aspects
security-testing team produced a long list of From the beginning, it was thought that the Integration Testing
what was important. This list was immediately way the trousers should be used need not be After parts of garment were sewn together,
given to those responsible for the trouser de- explained to anybody, or at least be left to they were put to tests checking their resis-
sign so that - hopefully - the product would the users themselves. However, finally it was tance, shape etc. Some faults were discovered
fulfil all usability requirements from the be- agreed that even the manufacturer had some and could be removed at this stage. The type
ginning, thus eliminating the need for ex- responsibility for telling customers how to of thread used for main the seams happened
pensive re-design and re-configuration of all use trousers securely: by providing appropri- to be too thick and impaired the fabric’s tear
equipment on the trouser production line. Here ate cleaning and storage instructions, for ex- resistance. One type of needle used in sewing
are some excerpts from this list. ample, and a sufficiently detailed description machines made holes, which were difficult to
of how much load and performance to expect keep waterproof afterwards.
• You must be able to put on and take off from them.
your trousers with thick gloves on your System Testing
hands, in complete darkness, in con- A few first pairs of trousers left the production
Security Testing Process
fined space (tent or bivouac), or in strong line! They were minutely inspected for manu-
Armed with all definitions, checklists, tools for
wind. facturing faults, extensively tested for all kinds
exercising fabric’s resistance and other weap-
• Trousers must be equipped with a loop of performance, checked for accordance with
onry from the quality assurance arsenal, the
for securing, so that they do not fall down initial design. Acceptance testing was started
manufacturer’s personnel stood braced for the
or get blown away. “Is that usability or in parallel: the next visit to Ben Nevis region
challenge of entering the realm of production
function, by the way? “, thought one took place.
of trousers for mountaineers. And they suc-
mountaineer while preparing this list. ceeded - no doubt (?) due to their meticulous
• Otherwise, there should be no loose ele- High-level Integration Testing
preparations and thought processes described
ments outside which could catch in other F… (they really meant “heck”)! Suspenders’
above. There are quite a few manufacturers
equipment or rocks. clips tended to fasten in jacket’s inner pocket
of good, robust, secure trousers for mountain-
• A flap enabling you to easily answer a zip. The distance between jacket’s lower hem
eers, so you can easily visit the nearest shop
“call of nature” must be provided. It must and trousers’ upper hem was too little, snow
with outdoor equipment and study the result.
never open accidentally, and must never could get in. The trouser legs were too narrow
Below follows the description of how they did
freeze, either! By the way, function or for those huge plastic boots used for crampons.
it. Nobody would disclose the well-guarded
characteristics - again? The side-pocket zip was sharp and could make
industrial secret whether this process descrip-
• All buttons, zips and clips must be made holes in down sleeping bags, should someone
tion preceded or followed its actual first-time
of plastic so that they are not cold and choose to sleep with trousers on.
execution, but here it is, and the manufacturer
never freeze. This plastic must not be- If anyone suspects that some of these faults
is now well-prepared for more challenges of
come brittle even after long exposure to were actually discovered during acceptance
the same kind.
very low temperatures. test hikes at Ben Nevis, I must confess this.
• Pockets (a long treatise on pockets fol- They were still, however, typical high-level
Requirements Engineering
lowed, omitted here). integration faults!
The production manager went hiking to the
mountains. A famous mountaineer was invited
Load and performance Acceptance Testing
to the factory and participated in the process,
Mountaineers clothes must be robust and While the contract mountaineer was busy
bribed by free trousers for his next expedition
withstand quite brutal usage. Tear and wear acceptance-testing-mountaineering, some of
to the Himalayas (money for that was taken
must be gradual, so that weaknesses are easily his colleagues got free trousers for their winter
from the company’s marketing budget). Trou-
visible and security degradation predictable. climbs in the Alps (beta-testing). Young per-
sers already available on this market were
Never must mountaineering equipment break sonnel from production and marketing depart-
bought and analysed in detail.
suddenly. ments got some free trousers, too, for use while
A list of requirements was produced. This list
From the customer’s (and even seller’s) point walking, hiking or jogging (alpha-testing).
was not perfect, but could be completed later
of view, it is of course important that garments Oops! A female climber discovered some prob-
on, as forgotten requirements were discovered,
are long-lived, withstand storage conditions lems with the location of the “call-of-nature”-
some of them as late as during monitoring of
well etc, but these aspects of their performance flap. Instead of complete re-design, a separate
customers’ complaints.
are not important for security. model for women was introduced.
Mountaineers use their clothes in extreme Design
conditions, which must be taken into account Monitoring in Operation
As described in section [Insecure Design], the
when measuring their performance. Damp Buyers were given a “registration card” to fill
design of trousers was carefully verified. Be-
conditions, wind, water and temperatures well in (voluntarily). If they answered some ques-
sides, all rules used in the design process were
below zero are common. tions, their “registration cards” participated in
re-used later on as test cases to check whether
On the other hand, perfectly water-proof and a lottery, where mountain tents were the main
the final product fulfilled them.
wind-proof trousers may be too heavy and prizes.
One pair of trousers was quickly made and
stiff, and their “breathing” ability may become These registration cards, as well as of com-
worn - by the mountaineer, followed by the
poorer, too. A proper balance must be defined plaints from users later on, were analysed -
panting production manager - during a one-day
and achieved in this respect. among other aspects - for security-related is-
hike in the Ben Nevis region. Some obvious
Good news is, the fabric’s, thread’s, button’s sues.
design flaws could be identified and removed
and clip’s strength, resistance to friction, water immediately afterwards.
etc. are readily measurable and some data on
what is appropriate for mountaineering clothes Component Testing
is available. The important thing was not to All components were carefully checked for

56 The Magazine for Professional Testers www.testingexperience.com

Books
Besnard, J., F. Developing Software for Safety Critical Systems
Cole, E. Hackers Beware: Defending Your Network From The Wiley Hacker
Friedman, M., A., Voas, J., M. Software Assessment: Reliability, Safety, Testabil-
ity
Gardiner, S. (ed) Testing Safety-Related Software: A Practical Handbook
Ghosh, A., K. E-Commerce Security: Weak Links, Best Defenses
Hatton, L. Safer C: Developing Software for High-Integrity and Safety-Critical
Systems
Herrmann, D., S. Software Safety and Reliability: Techniques, Approaches, and
Standards of Key Industrial Sectors
Jamsa, K. Hacker Proof: The Ultimate Guide to Network Security
Leveson, N. Safeware: System Safety and Computers
McClure, S., Scambray, J., Kurtz, G. Hacking Exposed
McGraw, G., Felten, E. Securing Java: Getting Down to Business with Mobile
Code
McGraw, G., Viega, J. Building Secure Software
Pipkin D., L. Halting the Hacker: A Practical Guide to Computer Security
Redmill, F. System Safety: Hazop and Software Hazop
Schiffman, M. Hacker’s Challenge: Test Your Incident Response Skills Using 20 Biography
Scenarios
Sindell, K. Safety Net: Protecting Your Business on the Internet Bogdan has experience in teaching
Smith D., J. Achieving Quality Software - Including its Application to Safety-re- testing, requirements and other SW
lated Systems engineering subjects since 1994 in Swe-
Van Der Meulen, M. Definitions for Hardware and Software Safety Engineers den, Poland, Denmark, Finland, Spain,
Villemeur, A. Reliability, Availability, Maintainability and Safety Assessment Luxembourg and the Czech Republic. He
Whittacker, J. How to break software pioneered the ISEB Foundation qualifica-
Whittacker, J., Thompson, H. How to break SW Security tion in Sweden (1999), in Poland (2004),
Wiegers K., E. Software Requirements then ISTQB® Foundation in Poland (in
both English and Polish) in 2006-2007,
and now in Russia.
Organisations, companies, services and standards
Board of Certified Safety Professionals, www.bcsp.org
Besides that, Bogdan has been involved
BS 7799, www.bsi.org.uk
in work with standards and syllabi.
Bugtraq, mailing list at www.securityfocus.com
Cigital, www.cigital.com (books, training and white papers on security)
He holds both the ISEB Practitioner and
COAST Software Vulnerabilities Testing Group, www.cerias.purdue.edu/coast/
full ISTQB Advanced Certificates.
projects/vuln_test.html
Today, he cooperates closely with the
Common Criteria, http://csrc.nist.gov/cc (“Common Criteria for IT Security
WCSQ 2008 organization.
Evaluation, Common Language to Express Common Needs “)
IEEE Standard for Software Safety Plans IEEE Std 1228-1994 (www.ieee.org)
Internet Security Systems, www.iss.net
LogiGear, www.logigear.com (Internet security testing, training)
NASA-STD-8719.13A (SW Safety Standard), www.hq.nasa.gov
Packetstorm Security, http://packetstorm.linuxsecurity.com
“Penetration Testing”: www.google.com gives 128,000 results
Safeware Engineering Corporation, www.safeware-eng.com (“applied research and
development for real-time, safety-critical systems”)
Security Focus, www.securityfocus.com
Security Innovation, www.sisecure.com (security: tools, training, testing)
www.hackers.com
www.hackersoftwarehacker.net
www.insecure.org

Dedicated Tools
GFiLANguard, www.all-internet-security.com/free_protocol_analyzer.html
Holodeck, www.sisecure.com/products.html
Internet Security Systems, www.iss.net
Nessus, www.nessus.org (free security scanner)
SAINT, www.saintcorporation.com (vulnerability scanner)
SATAN (Security Analysis Tool for Analysing Networks), www.fish.com/satan
www.insecure.org/tools.html

www.testingexperience.com The Magazine for Professional Testers 57

 Method for Reducing a Generic Software Routine
into a Mathematical Function
... and Possible Applications for Black-Box (Functional) Software Testing

© iStockphoto
by Danilo Berta

1. Abstracts methods to test the same function, sampling

some of the input values of the function (in (2.1) f (x1 x 2 ...x n ) = x n x n −1 ...x1 where x1 x2 ...xn
This article describes a method that can be used
some statistical way) in order to verify whether are generic alphanumeric characters.
to reduce a general software routine, which ac-
cepts input of alphanumeric values and returns the outputs of the same function for these input
values are as expected. This method, which is If we wanted to draw a Cartesian graph of this
alphanumeric output values, into a mathemati-
strictly speaking a black-box method, can also function, the first problem we would encounter
cal function. A generic software routine takes
be applied to any routine with any number of is how to allocate an order to the input and out-
and produces alphanumeric strings as inputs/
inputs and outputs; it’s clear that in these cases put string, to be able to define the metrics (the
outputs, whereby there is no way to define an
we have to deal with multi-variable functions. intervals) to use for the X and Y-axes. Intui-
order in these strings (e.g. it is not possible to
In this article, we will not deal with statisti- tively, we can imagine that this function would
determine between two strings which of them
cal sampling methods, but will only describe be the diagonal that splits the Ist quadrant in 2
comes before or after the other).
a method for associating a generic string with parts of 45°, as shown in the picture below:
In order to explain the matter in more detail,
a number in an unambiguous way, and will The intervals are labeled as s1, s2,...,s10,
we will consider a simple software routine
then discuss how to apply this method for the meaning that each point represents an input
which produces as output the reverse string
test and the regression test of generic software string and s(n)< s(n+1). However, we don’t
that it has taken as input. This routine can be
routines. know, for example, whether the input string
defined as a mathematical function in this way
f (x1 x 2 ...x n ) = x n x n −1 ...x1 , where xn xn −1...x1 are “ASER” is less than (or greater than) the string
“ATER”.
generic alphanumeric characters. If we want 2. Introduction with a simple example We need to define a method to associate a
to draw a Cartesian graph of the function, Let’s start with a very simple example to so we
unique number to a string and a method to de-
we need to know if, for example, the string can follow the course of the subsequent dis-
duce the string that corresponds to that num-
xn xn −1...x1 comes before or after the other string cussion with a more technical and mathemati-
x n x1 ...x n −5 , or any other string which we can
ber.
cal approach.
use as input of the function.
If we were able to reduce a software routine The first thing that comes to mind is to number
To keep things simple, we assume that we
into a well-defined function, we would also the single string elements progressively. If, for
have a really straightforward simple software
be able to use the well-known mathematical example, we suppose that the single elements
routine that produces
that compose our input/output strings shall be
as output exactly
the first three letters of the alphabet, ‘A’, ‘B’
the string it takes as
and ‘C’, we can compile the following table:
input, whatever the
string may be. This
function will be:

Fig 1: Identity Function Graph Example

58 The Magazine for Professional Testers www.testingexperience.com

 Number String Number String Number String Number String So, starting from the last remainder (step #4)
and continuing with the other remainders , we
1 A 13 AAA 25 BBA 37 CCA have found the numeric string “1123”, and us-
2 B 14 AAB 26 BBB 38 CCB ing the table in Fig. 2.3 we can now replace the
3 C 15 AAC 27 BBC 39 CCC numbers with letters as follows:

4 AA 16 ABA 28 BCA 40 AAAA 1→A

5 AB 17 ABB 29 BCB 41 AAAB 2→B
6 AC 18 ABC 30 BCC 42 AAAC 3→C
7 BA 19 ACA 31 CAA 43 AABA In this way, the numeric string “1123” becomes
8 BB 20 ACB 32 CAB 44 AABB “AABC”, which is the string corresponding to
9 BC 21 ACC 33 CAC 45 AABC the number 45 in the Table 1.
10 CA 22 BAA 34 CBA 46 AACA In the following paragraph, mathematical
11 CB 23 BAB 35 CBB 47 AACB proof of these rules is presented.
12 CC 24 BAC 36 CBC 48 ...
3. Mathematical proof of the rules
Table 1: Association of string with numbers ies with any number of elements. The higher First of all, we will start with some useful defi-
the number of elements in the dictionary, the nitions, as used before:
The table should be read in this way: number greater will be the number we deduce for
1 is linked to string “A”, number 2 is linked “short strings” (whereby “short strings” means 1. A dictionary is a collection of elements
to string “B”,..., number 46 is linked to string strings with a low number of base elements). a1,a2,...,an composing the string. It’s
“AACA” and so on. written as follows: D{a1,a2,...,an}.
Clearly, it’s possible to go on indefinitely to It’s a little trickier to find the “opposite rule”, 2. A base of the dictionary D is a number
compile the table above. We need to define i.e. the rule to transform a generic number into equal to the number of elements of the
a rule to calculate the corresponding number a string composed only of elements of some dictionary.
from the string. well defined dictionary. For now, we introduce 3. A string is valid with regard to diction-
the rule and apply it with an example: ary D if it’s composed exclusively by ele-
The rule is as follows: ments of D. It’s not valid if there is at
1. Divide the number for the base of the dic- least one element that does not belong to
Start with the base association of string with tionary, considering both the result of the dictionary D.
number: division and the remainder of the divi- 4. The length of a string is a number equal
sion. For example, for dictionary {A,B,C} to the number of elements the string is
Number String the base is 3, as previously stated. composed of.
2. If the remainder is zero, subtract 1 from 5. The numbers equivalent to the diction-
1 A the result and replace the remainder num- ary’s elements (in the following NEDE)
2 B ber with the base. are defined by a matrix as follows:
3 C 3. If the result is greater than or equal to the
base, continue with the division, applying
NEDE Element
Table 2: Association of Base String with numbers rules (1) and (2) until the result is zero.
In our example, this will happen just af- a1 = 1 a1
As a hypothesis, we will suppose that all ter we have found a result of the division a2 = 2 a2
strings considered must come from any com- that is less than 3 and after exactly 4 it- a3 = 3 a3
bination of the base letters {A,B,C} which in erations.
4. Write down the string composed by the ... ...
the following we will call the dictionary. So,
string “ACABCCAACC” is valid, while string remainder of the division, starting from ap = p ap
“ASDFGHAAABBCRR” is not valid, because the last, and written sequentially.
it contains the elements S,D,F,G,R that are not 5. Replace the number in the string, which Table 3: Association of Base String with numbers
in the defined dictionary. was derived as described in point (4), with
the letter in the base association string to It is presumed that all a where j=(1,2,...,L) are
The rule is expressed in the following formula, number of the dictionary (see Table 2). not zero. This is a fundamental hypothesis for
which for now we will describe by example what follows.
(and we will explain later in detail). To explain this in more detail, we will use the Please, note that all aj for all j=(1,2,...,L) are
number 45 as an example to verify whether the strictly less than base B of the dictionary.
(2.2)AABC = 1*33 + 1*32 + 2*31 + 3*30 = rule is correct. We will expect to find string The first role does not have to be proven, but
1*27 + 1*9 + 2*3 +3*1 = 45 “AABC”, using the dictionary {A,B,C}. must be taken as a definition of the algorithm
used to associate a number to a string. So, we
This is exactly the number that we find for Here is an example of the algorithm’s applica- can state the following:
string “AABC” in Table 1 above. If we exam- tion: Definition #1: Given a dictionary D, it’s pos-
ine the method of obtaining a value for string sible to associate an unique number N0 to any
“AABC” it’s evident that it’s very similar to 1. 45/3 = 15 remainder 0. Subtract 1 from valid string a1a2a3,…aL for this dictionary by
the way that numbers are usually transformed 15: 15-1 = 14 that is greater than 3, so we applying the following rule:
into base 2, 8 or 16 or any other base into deci- have to continue. Replace the remainder
mal. So, there’s nothing new or magic about it. 0 with 3. (3.1) N0 =
The number “3” we have used in our formula 2. 14/3 = 4 remainder 2. No need to sub-
tract. 4 is greater than 3, continue. a 1 B L −1 + a 2 B L − 2 + ... + a L B 0 = a 1 B L −1 + a 2 B L − 2 + ...
is simply the base of the dictionary, or, in other
words, the number of base elements in the dic- 3. 4/3 = 1 remainder 1. The result is less
tionary {A,B,C} is exactly equal to three. than 3. Oneadivision
1B
L −1
+ amore
2B
and
L−2
+ ...then
+ a we 0
L B = a 1B
L −1
+ a 2 B L − 2 + ... + a L because B 0 = 1
will stop.
This method can be extended for dictionar- 4. 1/3 = 0 remainder 1. Stop here.

www.testingexperience.com The Magazine for Professional Testers 59

Where B is the dictionary’s base and aj is the This results in the following formula:
numerical equivalent to the generic elements we have: N 2 = N 3 B + a 2
aj of the dictionary, where j=(1,2,...,L) and L is (3.7)
the length of the string. Put: N 4 = 0
Starting from this definition, it’s possible to N L − r = N L − r B + a r with a r = 0 means:
deduce the rule for transforming a generic we have: N 3 = N 4 B + a 1 ≡ a 1
number into a string composed only by ele- N L−r = N L−r B
ments of the dictionary. In the example L = 4, (L-1) = 3. The next N4
value is exactly equal to 0 (zero), because we In other words:
If we start dividing the number N0 by base B, divide N3 (that is less than B) by B, obtaining
we will obtain: 0 with remainder N3 = a1. We have to stop the
process at this point. (3.8) N L − r = N L − r B = ( N L − r − 1) B + B
(3.2) Remember that we have supposed that all aj
where j=(1,2,...,L) are not zero.
This formula demonstrates the tip explained
N0 a above.
= a 1 B L − 2 + a 2 B L −3 + ... + a L −1 + L In formula (3.5), the aj where j=(1,2,...,L) are
B B the remainder of the subsequent division of
number N0 for the base of the dictionary, while 4. Summary of the rules
With the position: In this paragraph, we will summarize the re-
the Nj where j=(1,2,...,L) are the results of the
division. sults obtained above and define the rules for
(3.3) transforming a string into a number and, vice
N 1 = a 1 B L − 2 + a 2 B L −3 + ... + a L −1 From the above discussion we can deduce the versa, a number into a string, with reference to
following rule to extract from a number N0 a a given dictionary.
string of length L composed only of elements
We have: Rule #1: Transform a string into a number.
from a given dictionary D{a1,a2,...,an}.

1. Define the dictionary and the number- 1. Define the dictionary and the elements of
(3.4) N0 a equivalent elements. the dictionary. The elements in the string
= N1 + L must only be composed of any combina-
B B 2. Divide number N0 for the base B of the
dictionary, considering both the result of tion of elements of the selected diction-
the division and the remainder of the di- ary.
that is equivalent to N 0 = N 1 B + a L vision, and continue in this way until the 2. Define the NEDE (numerical equivalents
result of the division is zero. This hap- to dictionary elements) matrix
pens after L steps. 3. Apply the following formula:
Continuing in this way, we obtain:
3. Starting from the last remainder create a
string composed by the remainders of the
N 0 = N1 B + a L N 0 = a 1 B L −1 + a 2 B L − 2 + ... + a L
divisions.
4. Consider the remainders of the divisions
N 1 = N 2 B + a L −1
as the numbers equivalent to the diction- Where B is the dictionary’s base and aj is the
N 2 = N 3 B + a L−2 ary’s elements, and replace these num- numerical equivalent for the generic elements
bers with the equivalent elements of the of the dictionary, where j=(1,2,...,L) and L is
dictionary, using the “NEDE” table. the length of the string.
(3.5) .........
N0 is the number corresponding to string
Now, all this works fine as long as we find only “a1a2...aL”, that was searched.
N L − 2 = N L −1 B + a 2 remainders of the divisions that are non zero.
However, what happens if the remainders of Rule #2: Transform a number into a string.
N L −1 = a 1 the division are zero?
NL = 0 1. Define the dictionary and the numerical
We will use the following simple tip. If the re- equivalents.
sult of the division of X through Y is exactly 2. Divide number N0 for the base B of the
From the last equation in (3.5), i.e. NL-1 = a1 it N, with remainder 0 (zero), it can also be writ-
should be evident that if we start from equation dictionary, considering both the result
ten as follows: of the division and the remainder of the
(3.1), we divide N0 by B(L-2). For a better un-
derstanding, an example is given below where division; in each subsequent division re-
(3.6) place the dividend with the result of the
we apply (3.5) for the case where L = 4.
previous division, and continue in this
N X way until the result of the division is zero.
= N remainder 0 → = ( N − 1)
Start with: N 0 = a 1 B 3 + a 2 B 2 + a 3 B + a 4 Y Y This happens after L steps, where L is the
remainder Y length of the string.
3. If during execution of the steps described
Put: N 1 = a 1 B + a 2 B + a 3
2
in point 2, you find a remainder equal to
we have: N 0 = N 1 B + a 4 zero, replace the result with (result -1)
In other words, the result of X should be and the remainder 0 with the base B of
Y the dictionary.
(3.5 ex)
also be considered as equal to N-1 with re- 4. Starting from the last remainder, create a
mainder Y. For example: 15/5 = 3 remainder 0, string composed by the remainders of the
Put: N 2 = a 1 B + a 2 but it’s also 15/5 = 2 with remainder 5. divisions.
we have: N 1 = N 2 B + a 3 5. Consider the remainders of the divisions
So, if we found some aj equal to zero in the as the numerical equivalents to the dic-
division process in (3.5), we now apply the tip tionary’s elements and replace these
Put: N 3 = a 1 and replace the result with (result -1) and the numbers with the equivalent elements of
remainder with the base B of the dictionary. the dictionary, using the “NEDE” table.

60 The Magazine for Professional Testers www.testingexperience.com

 We train in Latin America -
ISTQB Certified Tester Training
[email protected]
© iStockphoto

www.testingexperience.com The Magazine for Professional Testers 61

 The resulting string is the one that was following command: 6. The plot of the “string reverse” func-
searched and corresponds to number N0. tion
perl NumberToString.pl <input
The first column is the X input numerical
number> <dictionary>
5. Software implementation of the equivalent of the input strings in the third col-
rules For example:
umn, while the second column is the Y output
I have prepared two Perl scripts that imple- numerical equivalent of the output strings in
ment both processes described. Rule #2 de- perl NumberToString.pl 637549 “A the fourth column (that is the real output of the
scribed above is suitable to be implemented B C D E” reverse string function, or whatever function
in a recurring way using Perl (or your favorite you decide to call).
programming language), but I prefer to strictly which returns the string: ABECDEADD. The
follow the process described to be sure that it same considerations as for script #1 are valid These scripts were tested on Perl version
will work correctly. for this one. A DOS script called runNumber- v5.10.0 built for MSWin32-x86-multi-thread
All the scripts can be downloaded from link: ToString.bat could also be used to run the for the Windows Vista operating system, and
http://www.bertadanilo.netsons.org/, or direct- script (avoiding having to bore yourself with Perl version v5.8.8 built for i486-linux-thread-
ly from: http://www.bertadanilo.netsons.org/ writing the dictionary, which is usually fixed), multi on debian version (Kurumin). These are
Upload/perl_TestingExperience.zip. passing only the string input parameters, as not really intended to control the formal valid-
follows: ity of the inputs used (wrong numerical format,
Another Perl script was built with the purpose wrong dictionary format, and so on); therefore
of generating the mathematical-equivalent runNumberToString.bat 637549 please take care to use the right format, other-
function of some software routine. I took as wise the result cannot be guaranteed.
an example the (very simple) routine that pro- Script #3. This script, called EvaluateFunc-
duces as output the reverse of the string it has tion.pl implements both of the rules. It takes Now, at last, we will attempt to plot a Carte-
taken as input. as input a dictionary in the form of an array of sian diagram of a software function that works
chars separated by spaces, and returns the ta- with a list of symbols (strings). To keep the
All these scripts were prepared with the sole ble of the results of the function that should be task simple, we have decided to consider the
purpose of providing a “proof of concept”. In used to plot the Cartesian diagram of that same reverse string function. Using the command
the following, there is a brief explanation of function. The function is written in the code.
how they work. All the scripts are released as In detail, to use the script you will need to: perl EvaluateFunction.pl “A B C
is, under GPL License. D E” 100 > out.txt
• Personalize the function to generate the
Script #1. This script, called StringToNum- data in order to plot in the main routine we have in file out.txt a table of 100 points
ber.pl implements rule #1. It takes as input of the program. As a default, the function that can be used to plot the “reverse string”
a string and a dictionary in the form of an ar- is the “string reverse” that produces as function. For this purpose, you may use your
ray of chars separated by spaces, and returns output the reversed input string. For ex- favorite spread sheet or any other software you
the numerical equivalent to the input string, as ample: ABC becomes CBA. usually use to draw graphs.
per rule #1 explained above. To run the script, • Call the program as follows:
from DOS or *IX system, you need to run the The plot looks like Graph 1.
following command: perl EvaluateFunction.pl <dic-
tionary> <number of point> The Y axis shows the numbers and not the Y-
perl StringToNumber.pl <input string value, to make the graph easier to man-
string> <dictionary> The output will be a table of data (separated by age. Considering that the function is simply a
tabs) with <number of point> points with the x “reverse string” it is easy to imagine the cor-
For example: input string and y output string and equivalent responding string value of the Y interval num-
numerical value of the software function. ber.
perl StringToNumber.pl ABECDEADD A graph with more points (in this case 500
“A B C D E” For example, the output of command points), but with fewer details, is shown be-
low.
which returns the number 637549 perl EvaluateFunction.pl “A B C
D E” 10 The graph with 10000 points looks like Graph
You can avoid having to explicitly call the 3.
“perl” interpreter before the script name if will be a table of 10 values as follows:
you give the execution permission (chmod +x 7. Conclusions: What does all this have
StringToNumber.pl) on *IX system to the X Y X Y
script. In Windows, it depends on how your
to do with software testing?
num num num num So, we have probably found a method to “re-
system is configured. I will not bother you any
1 1 A A duce” a software routine into a mathematical
further with these details. A DOS script called
2 2 B B function. Clearly, what we have described can
runStringToNumber.bat could also be
“easily” be further generalized.
used to run the script (avoiding having to bore 3 3 C C What does all this have to do with testing? At
yourself with writing the dictionary, which is
4 4 D D the current moment absolutely nothing; the
usually fixed), passing only the string input pa-
5 5 E E main idea, as you can guess, is the following:
rameters, as follows:
if we know what the mathematical equivalent
6 6 AA AA function behind the software routine under
runStringToNumber.bat ABECDEADD
7 11 AB BA test is, we can easily “test it” under a (small or
Script #2. This script, called NumberTo- 8 16 AC CA large) sample of input testing points (the “X”
String.pl implements rule #2. It takes as in- axis).
9 21 AD DA Now, the problem is that nobody (also, and
put a number and a dictionary in the form of an
array of chars separated by spaces, and returns 10 26 AE EA in particular, no one who wrote the software)
the numerical equivalent to the input string, as knows of the function. I think that nobody
Table 4: Example knows the profile of the “reverse string” func-
per rule #2 explained above. To run the script,
from DOS or *IX system, you need to run the tion graphics presented above, or that the “re-
verse string function” is a well-known and

62 The Magazine for Professional Testers www.testingexperience.com

 widely used feature in a lot of software pro-
grams.
So, one might ask what this method could be
useful for? It could be useful in regression test-
ing, in particular for the testing of single soft-
ware components and for the test of (probably
huge) amounts of database data, especially for
systems that cannot be easily accessed, for ex-
ample, systems who are located in different,
non-integrated environments and for which
access to the data for the software under test
can only be produced only by extraction (e.g.
in CSV format). In regression testing you can
use the information collected during the “first
test run” to clearly define the expected behav-
ior of the function under test. You can there-
fore transform it into a numeric function (one
or more functions) and keep it as reference for
the other entire regression test or retest. The
Graph 1: Shape advantages are:

a. You collect input and expected output

data in numerical format. Knowing the
numerical input and output data, you can
at any time recalculate the inputs and out-
puts needed for the software routine (see
NumberToString.pl proof of the concept
script). Clearly, you must also save the
dictionary used in the transformation.

b. After you have run the software on the

input data, you can transform the real
output data into a table of numbers and
draw the “real function”. This function
must then be compared with the “ex-
pected function” (i.e. performing a point-
by-point comparison between the two
functions). If any differences between
the two are found, these can be analyzed.
The operation can be easily done using
any mathematical software that can draw
graphics.

For example, to keep things simple, if you

Graph 2: Shape enter users onto a system and would like to
verify that all the users’ fields are correctly
loaded onto the database, you can proceed as
follows:

1. Prepare the file with the data to be load-

ed.
2. Transform all the columns for which you
want to verify correct entry into the ex-
pected functions.
3. Load the data using the software under
test
4. Extract the loaded data from the data-
base and transform it into a function (i.e.
a real function, which expresses the real
software behavior); then compare the two
functions.

This has been done for the first time. For the
following test, you can automatize the process.
Starting from the input and output numbers
(which you saved in the first run), you proceed
as follows:

a. Extract the input strings from the input

numeric data and from the dictionary.
Graph 3: Shape b. Run the software on the inputs and get

www.testingexperience.com The Magazine for Professional Testers 63

 the outputs; transform the outputs into “comparator software” to perform a particular that there aren’t any. The subject occurred to
numbers and draw the function(s). comparison, because all you need is software me one day when I was reading a (very bor-
c. Compare the expected output (numbers) that can draw graphics starting from a table of ing) functional analysis for some software. I
with the real output (numbers) and find (x;y) values (or also for more variables). thought along the lines: “Wouldn’t it be nice
the differences (if any). if we could reduce a big analysis into some
Acknowledgements mathematical functions! How we can do this?
Clearly, there will be no major differences if I’m grateful to my friend and colleague Gian- “...
we compare this method with a more tradi- maria Gai for the help he has given me when If any reader knows about some other articles
tional “string comparator”. With this method, I wrote this article: he also suggested that and/or books on similar methods, please let me
you will probably not have to develop new I include some references. The problem is know.

Biography
Berta Danilo graduated in Physics at Turin University (Italy) in 1993 and first started to work
as a Laboratory Technician, but soon he switched to the software field as an employee of a
consultancy company working for a large number of customers. Throughout his career, he
worked in banks as Cobol developer and analyst, for the Italian treasury, the Italian railways
(structural and functional software test), as well as for an Italian automotive company, the
European Space Agency (creation and test of scientific data files for the SMART mission),
telecommunication companies, and the national Italian television company. This involved
work on different kinds of automation test projects, software analysis and development
projects.
With his passion for software development and analysis – which is not just limited to testing
– he has written some articles and a software course for Italian specialist magazines.
His work gives him the possibility to deal mainly with software testing; he holds both the
ISEB/ISTQB Foundation Certificate in Software Testing and the ISEB Practitioner Certificate;
he is a regular member of the British Computer Society.
Currently, he works as Test Manager for an Italian Editorial Company.

64 The Magazine for Professional Testers www.testingexperience.com

 Database Testing - Testing The Backend
by Gulshan Bhatia
© iStockphoto

Database systems play an important role in The database design process is decided by determine if the logical data model is complete
nearly every modern organization. The ulti- finding the requirements and needs of the end and correct. Confirm the design with the busi-
mate objective of database analysis, design, user. Uncertainty about understanding the re- ness community in a design review process.
and implementation is to establish an electron- quirements can be reduced only after signifi- Create a logical Entity Relationship Diagram
ic data store corresponding to a user’s concep- cant analysis and discussions with users. Once (ERD) to graphically representthe data store.
tual world. the user requirements are clear, the process of Determine if the data model is fully docu-
Functionality of the database is a very criti- behavior implementation consists of the design mented (entities, attributes, relationships) At-
cal aspect of application’s quality; problems and construction of a solution domain follow- tributes have correct datatype, length, NULL
with the database could lead to data loss or ing the problem domain. Because of the dif- status, default values. General discussion of
security violation, and may put a company at ficulties associated with the changing require- business rules to be enforced by database level
legal risk depending on the type of data being ments, the database developer must attempt constraints should be carried out e.g.
stored. Applications consist of a database and to develop a database model which closely
improving quality of data in an organization is matches the perception of the user, and deliver • Not null constraints
often a daunting task. a design that can be implemented, maintained • Check constraints
and modified in a cost-effective way. • Unique constraints
A database should be evaluated throughout Diagrammatic representation using entity-re- • Primary key constraints
the database development lifecycle to prepare lationship diagrams, object models, data flow • Foreign key constraints
a quality database application. Data in a da- diagrams, allows the information described in
tabase may be input from and displayed on a a visual format in a meaningful way. Business rules to be enforced by triggers and
number of different types of Systems. Each of procedures should be discussed along with the
these types of systems has unique system limi- Database testing is one of the most challeng- Business rules to be enforced by application
tations, which may dictate how data should be ing tasks to be done by software testing team. code. After this the normal forms should be
formatted in your database. A database should A Database Tester, by understanding the ref- tested with the help of test data. Testing physi-
be evaluated based on the factors such as data erential integrity and database security, and cal database design includes testing table defi-
integrity, consistency, normalization, perfor- by having a good grasp on the various tech- nitions, constraints, triggers and procedures.
mance, security and very important - the ex- nologies and data formats used to transfer and Black Box testing techniques like Boundary
pectations of its end users. retrieving the data from the database, can test value analysis can be used. We can test the ta-
We can attempt to maintain the quality stan- database to avoid problems. Testing should be ble definition by testing the column definitions
dards in a proactive way and it involves sur- included in various phases of database devel- and constraints that have been imposed:
veying the customer to determine their level of opment lifecycle. The cycle typically consists Database constraints can be checked as fol-
satisfaction with the product so that potential of several stages from planning to designing, lows:
problems can be detected early in the develop- testing and deployment.
ment process. In the first phase of database development pro- 1. Primary Key- Write Test Case to insert
cess, requirements are gathered; checklists can duplicate value in Primary Key column.
Once the development is over we can ensure be used as part of the evaluation process for 2. Insert record to violate Referential Integ-
that the database conforms to predefined stan- the database specification. rity Constraint.
dards and guidelines. It compares the required After gathering requirement and understand- 3. Delete record to violate Referential Integ-
features or attributes (problem domain) with ing the need for the database, a preliminary rity Constraint.
the results of development (solution domain). list of the fields of data to be included in the 4. Insert NULL value in NOT NULL col-
When deviations from the problem domain database should be prepared. We should have umn.
are found, they are resolved and the process is complete information about what information 5. Insert values to violate check constraints
modified as needed. is required by the client and what type of fields by inserting values outside input do-
are required to produce that information. Next mains.

www.testingexperience.com The Magazine for Professional Testers 65

 Klkfi`Xcj
Y\i#
N \ [ e \j[ Xp(jkFZkf
F e f]
`e^Xj\i`\j
n\Xi\_fc[ fdgc\d\ek
_`Z_Z
klkfi`Xcjn k_\
\ g i\j\ e kXk`fej]ifd ^
k_ Y\`e
%K_\j\Xi\
Zfe]\i\eZ\ ejk`klk\f]
@
_\c[XkK_\ ^`e\\ij%
e `Z X c<e
D\Z_X

JH:Zfe]\i\eZ\)''/ s Dfe[Xp)0k_J\gk\dY\iXe[Kl\j[Xp*'k_J\gk\dY\i sH<@@:fe]\i\eZ\:\eki\Cfe[fe

E\m\ikff Yljp6
K_\ifc\f]k\jk`e^`e`dgifm`e^gif[lZk`m`kp%
JH::fe]\i\eZ\)''/
N\Ëi\cffb`e^kfZi\Xk\Xi\XcYlqq`e)''/n`k_X]i\j_#e\nm\el\Xkk_\H<@@:fe]\i\eZ\:\eki\%8e[JH:LBn`ccY\X_`m\f]XZk`m`kpk_`j
p\Xin`k_jfd\]XjZ`eXk`e^jg\Xb\ij]fin_Xkgifd`j\jkfY\XZ_Xcc\e^`e^#Xe[gfk\ek`XccpZfekifm\ij`Xc#Zfe]\i\eZ\k_\d\1e\m\ikff
YljpÆk_\ifc\f]k\jk`e^`e`dgifm`e^gif[lZk`m`kp%
<m\ipfe\e\\[jkfglcck_\`in\`^_k`eXefi^Xe`jXk`fe%@]Xnfib\iY\\[f\jeËkZfeki`Ylk\kfX_`m\#k_\\ek`i\Zfcfep`jXki`jbXe[ef$fe\
^\kjXep_fe\p %K_\jXd\ZXeY\jX`[f]fi^Xe`jXk`fejn_fj\[\gXikd\ekjXi\eËkZfeki`Ylk`e^\]]\Zk`m\cpkfk_\Ylj`e\jjXck_fl^__fe\p
c\m\cjXi\eËkX]]\Zk\[ %
N`k_`ej`^_k]ifddXep`e[ljki`\j#`eZcl[`e^ÔeXeZ`Xcj\im`Z\jXe[\ek\igi`j\dXeX^\d\ek#JH:LBn`cckXZbc\k_`jjk`ZbpjlYa\Zk_\X[fe
Xe[[`jZljj\oXZkcpn_Xkk_\k\jk`e^Zfddle`kpZXe[fkfYffjkgif[lZk`m`kp`ek_\Ylj`e\jjjgXZ\%
9\k_\Y\\Ëjbe\\j`epfliZfdgXep¿

Fi^Xe`j\[Yp
M\el\ ;\c\^Xk\j
DflekYXkk\eJl`k\Xe[ GiXZk`k`fe\ijXe[Ylj`e\jj[\c\^Xk\j1
<c`qXY\k_N`e[jfiIffd# ([Xp1™+''"M8Kg\ig\ijfe
,k_Õffi#H<@@:fe]\i\eZ\:\eki\#Cfe[fe )[Xpj1™.,'"M8Kg\ig\ijfe
=fidfi\[\c\^Xk\fi\o_`Y`kfi`e]fidXk`fe#
Klkfi`Xc;Xp$™*,'"M8Kg\ig\ijfe gc\Xj\ZfekXZkk_\Zfe]\i\eZ\k\Xd%
;Xk\j <o_`Y`kfij G_fe\1"++' )'.++/+-+.
Dfe[Xp)0k_J\gk\dY\iXe[ K_\i\Xi\XeldY\if]\o_`Y`kfigXZbX^\j <dX`c1lb7jhj$Zfe]\i\eZ\j%Zfd
Kl\j[Xp*'k_J\gk\dY\i XmX`cXYc\#`eZcl[`e^jgfejfij_`g[\Xcj% N\Yj`k\1nnn%jhj$Zfe]\i\eZ\j%Zfd&lb

66 The Magazine for Professional Testers www.testingexperience.com

For relational databases queries are written using SQL. We can test da- Description : Insert a record with I555 as Item code.
tabase queries by identifying different situations and data values. SQL Expected Result : record should be saved in database.
conditions can be tested using Decision Condition Coverage:
Test Case ID: TCcheck _itemcode3
a. Select statements use conditions in Where Clause and Having Objective: To evaluate check constraint on Item code in item table.
Clause for Group By columns. Description : Insert a record with invalid Item code as I000.
Expected Result : error message should be displayed.
i. Conditions written using AND logical operator requires (T,T),
(T,F), (F,T) outcomes for two operands to be tested. Test Case ID: TCcheck _itemcode4
ii. Conditions written using OR logical operator requires (F,F), Objective: To evaluate check constraint on Item code in item table.
(T,F), (F,T) outcomes to be tested. Description : Insert a record with I556 as Item code.
Expected Result : error message should be displayed.
Testing requires that every condition affecting the result takes all pos-
sibleoutcomes at least once. Test Case ID: TCcheck _Description1
Objective: To evaluate NOT NULL constraint on description column
b. Testing SQL statements involving NULL values. in item table.
Description : Insert a record with no value for description column.
i. Requires Testing conditions with each operand in the condi- Expected Result : error message should be displayed.
tion taking NULLvalue.
ii. For a Group By Clause, NULL values have to be considered. Test Case ID: TCcheck _price
Objective: To evaluate check constraint on price column in item table.
c. For subqueries, include test cases to return zero and more rows. Description : Insert a record with price=-10.
d. SQL statements like Update, Insert, Delete also need to be tested Expected Result : error message should be displayed.
for conditions.
Test Case ID: TCcheck _delete
Apart from testing the table definitions and SQL statements a database Objective: To evaluate referential integrity constraint on item code col-
tester should test the triggers, procedures and functions. These objects umn in item table.
can be unit tested by finding various paths of execution in the code and Description : insert a record in sales table with item code I001. Delete
functionality can be tested by executing the code - providing required record from item table where item code=I001.
inputs and checking the output generated. Expected Result : error message should be displayed.
Lets see how can we design test cases to test table definition i.e. column
definitions and constraints. Refer to ‘Item’ table storing details of items SALES Table
in stock. Details of sales orders placed for various items are stored in Test Case ID: TCcheck _itemcode
another table ‘Sales’. The table definitions are as follows: Objective: To evaluate references constraint on itemcode column in
sales table.
ITEM Table Description : Insert a record with item code not existing in item table.
Col Name Datatype Size Constraints Expected Result : error message should be displayed.

Itemcode Char 4 1001 to 1555 Primary Key Test Case ID: TCcheck _orderID
Description Varchar 20 NOT NULL Objective: To evaluate check constraint on Order_ID column in sales
Price int >0 table.
Description : Insert a record with Order_ID as 1001.
QOH int Expected Result : error message should be displayed.
ROL int QOH>ROL>0
Now consider a requirement for displaying details of all the items for
SALES Table which qty ordered is>10. We can find the required details by writing
Col Name Datatype Size Constraints SQL query:
Order_ID Char 4 Should begin with O Select item_code
Item_code char 4 References Items Table from sales
QTY int >0 where qty>10
Order_date datetime To test this query we can prepare a test sales table having records with
ROL int QOH>ROL>0 qty>10. When executed, this query should return all the matching re-
cords. Similarly we can check the query by preparing a table not having
To test the constraints, we can design test cases as follows: any matching records.

ITEM Table We can unit test the triggers, procedures and functions by finding vari-
Test Case ID: TCcheck _itemcodePK ous paths of execution in the code. Consider the following PL/SQL
Objective: To evaluate Primary key constraint on Item code in item block:
table.
Description : Insert two records with I001 as Item code.
Expected Result : second record should not be saved in database.

Test Case ID: TCcheck _itemcode1

Objective: To evaluate check constraint on Item code in item table.
Description : Insert a record with I001 as Item code.
Expected Result : record should be saved in database.

Test Case ID: TCcheck _itemcode2

Objective: To evaluate check constraint on Item code in item table.

www.testingexperience.com The Magazine for Professional Testers 67

1: Declare 1
Bonus number:=0; --required values assumed
Sales number:= 6000;
Target number:= 5000; 2

2: Begin
differ:=sales-target; 3
3: if differ>0 then
4: if differ>100 then
R1 4
5: bonus:=1000; R3
6: else
5 R2 6
bonus:=500;
7: end if;
7
8: end if;
9: update salesman
set incentive=bonus 8
where empcode=1;
10: End; 9

No. of Edges: 11
No. of Nodes: 10 10

No. of Regions: 3
No. of Predicate Nodes: 2 Control Flow graph
Cyclomatic Complexity : 3

There are 3 independent paths. We can design the test cases to test each path using statement coverage and decision-condition coverage.

Apart from testing constraints, triggers and procedures; database security also needs to be checked. We need to see how database can be made
secure. Test data plays very important role in database testing process. Database developers and administrators spend a lot of time in creating test
data sets to examine database performance. Data generation tool can be used to avoid this and automatically create test data. Tool can also be used
to stress test database server by creating a continuous set of requests to the server.
Thus all the factors must be considered to improve quality of database and in turn improving the entire business process.

9Get your product to the market

9Avoid a recall
9Improve your reputation
9Streamline your process
9Build a better testing organization

01. Consulting
x Planning, Testing and Quality Process Consulting
x Based on Critical Testing Processes
Biography x Plans, Advice and Assistance in Improvement
Miss Gulshan Bhatia currently works
at WebTek Labs Pvt. Ltd. - a leading 02. Outsourcing
Software Testing company in India. She x On-site Staff Augmentation
has a broad background in the field of
Information Technology and Software
x Complete Test Team Sourcing
Testing. Gulshan is involved in con- x Remote Project Execution
ducting Software Testing Trainings at
corporate and public level; designing
and updating the courseware; providing
03. Training
consultancy for Software Testing projects x ISTQB and Other Test Training
using Manual Testing Techniques and x Exclusive ISTQB Test Training Guides
Automation Tools. Her interests include
Database Testing, Web Application Test-
x License Popular Training Materials
ing and Telecom Testing.

68 The Magazine for Professional Testers www.testingexperience.com

 International Software Quality Institute

Quality Assurance
Management Professional

Best prepared.
Tailor-made Modular System
Three certificates:
1) Requirements Engineering (IREB® CPRE, FL)

2) Software Test (ISTQB® CT, FL)

3) Work environment specific module of choice:

- Test Management (ISTQB® CT AL-TM)

- Configuration Management (iNTCCM®)
- Project Management (iSQI® CPPM)
- Software Architecture (iSAQB® CPSA)
- Secure Software Engineering (ISSECO®)
(Other certificates need to be recognized by iSQI.)

Experience evidence:
At least two years of practical experience in
software quality assurance management required.
Annual re-certification.

www.qamp.org
www.testingexperience.com The Magazine for Professional Testers 69
 Strategies for Testing
Telecommunication
Services

© Hans-Peter Reichartz /www.pixelio.de by Bárbara Martínez , Jesús del Peso and Lourdes Calvo

For a telecommunications operator, the final • Hardware elements which include: user addition, the single elements usually have be-
outcome of its activity is a series of services equipment and terminals (such as mobile haviours complex enough individually.
to be offered to the public. The quality of these phones, fixed phones, personal comput- However, this complexity is not only due to
services and their perception by end users is ers, PDAs or TV set-top boxes) and the these internal elements. It is also due to the
essential for the operator’s business model. hardware elements that compose the core fact that they are public services, and therefore
This article covers the issue of the design and network, (such as routers, switches, gate- they must be able to interoperate with other
execution of large test campaigns for telecom- ways, radio stations, etc).. similar services of different operators in order
munication services, and does so from a prag- • Software elements which include: the to provide a whole service.
matic point of view. In addition, some basic programs and software systems that con-
design principles for the design and construc- trol hardware elements, which can be of Testing Telecommunication Services
tion of general testing tools for telecommuni- very different types (such as clients, serv- Due to the high complexity of telecommunica-
cation services are discussed. ers, databases, directories, codecs, etc).. tion services, the phase of certification and test-
ing becomes crucial. However,
Introduction testing of telecommunication
First of all, it is necessary to services is also a very complex
define precisely what is meant task for several reasons:
by telecommunication service. • It is necessary to model
One possible definition of this the behaviour of some of the
term is found in the Telecom- elements of the service and
munications Act of 1996 [1]: frequently one or more final
“The offering of telecommu- user also have to be simulated
nications for a fee directly to in order to be able to perform
the public, or to such classes complete tests. In other words,
of users as to be effectively it is necessary to build behav-
available directly to the pub- ioural models for the whole
lic, regardless of the facilities service and for some of its
used.”; where “Telecommuni- components.
cations”, in turn, is defined as • It is also necessary to veri-
“the transmission, between or fy multiple conditions on some
among points specified by the of the elements and equipment
user, of information of the us- Figure 1: Example of architecture of a telecommunication service (IMS). composing the service.
er’s choosing, without change • The number of different
in the form or content of the situations and scenarios that
information as sent and received.” • Several sets of logical elements, like the have to be tested is also very large.
communication protocols through which • It is necessary to test the new releases
Telecommunication services are very com- these elements communicate, or the inter- of the service efficiently. Therefore it is
plex types of service. Their complexity is due faces and reference points through which very important to be able to ensure repro-
to both structural and functional reasons. For this communication takes place. ducibility of the testing scenarios and to
example, from the structural point of view, guarantee fast execution and analysis of
telecommunication services are composed of a Therefore, the complexity of telecommunica- the test campaigns.
high number of elements. These elements can tion services comes mainly from the multiple
be of very different types: interactions established between the different
elements that compose the whole system. In

70 The Magazine for Professional Testers www.testingexperience.com

Therefore, test campaigns are usually very Using Testing Standards: TTCN-3 costs of doing this are unaffordable in terms of
large, and are composed of a large number TTCN-3 (Testing and Test Control Notation time and effort in our testing scenarios.
of individual tests. Test parameterisation is version 3) [2] is a standard test definition lan- To achieve our goal, a more friendly and scal-
usually very complex and great flexibility is guage, created by ETSI (European Telecom- able formalism is needed to model telecom-
required for test definition, configuration and munication Standards Institute) [3]. TTCN munication services behaviours.
execution. was initially designed for the specification
and execution of telecommunications proto- In addition, TTCN-3 test campaigns involve
One important problem to be solved is con- col tests. However, since version 3 it supports the execution of several test cases. These test
nected with the reutilisation of code: since a virtually any kind of tests, including software cases may be parameterised to allow the ex-
telecommunication service is composed of testing. Therefore, TTCN-3 is beginning to be ecution of different types of tests to simulate
many different elements, it is necessary to popular not only in telecommunication testing a set of testing scenarios of interest. As men-
combine various independent test libraries environments, but also in other sectors such tioned in the previous sections, one of the
easily to build specific test tools for new ser- as consumer electronics, automotive and soft- main characteristics of testing telecommuni-
vices. ware industries. cation services consists of the large-size test
One of the main reasons for adopting the campaigns: they usually have several hundred,
Another problem that must be tackled is the TTCN-3 standard is the reutilisation of test even thousands, of different tests which have
provision of tests: the execution of telecom- code and libraries. As telecommunication to be implemented and executed.
munication service tests requires provisioning
complex configuration information to achieve
a successful service simulation. Before ex-
ecuting any test, it is necessary to provision
it, which means to configure each one of the
elements that compose the telecommunication
service. This implies, for example, the modifi-
cation of databases or directories and even the
manual configuration of some equipment. Af-
ter execution of the tests it is also necessary to
undo the previous provision in order to leave
Test
Campaign
Initial
Figure 3: Phases of the definition and execution of complex telecommunication services tests.
State
Figure 1: Phases of the definition and execution of complex telecommunication services tests.

Global services are composed of multiple types of To deal with the execution of large test cam-
simple
Provisioning elements and equipment, it is fundamental to paigns and with the high complexity of service
test Pre- be able to combine several test components behaviours to be tested, we have proposed to
ready to quickly build new testing systems for new separate the execution of tests from their defi-
telecommunication services. nition, as shown in Figure 3.
Specific
Provisioning
TTCN-3 defines a test language which can We propose to make the test cases as general
Ready for be compiled for multiple software and hard- as possible. Test cases must also be highly pa-
Test Execu-
ware platforms; it has been defined with code rameterised. Our main goal is to control test
tion
reutilisation as one of its main design goals. behaviours through parameterisation as much
TTCN-3 allows this reutilisation through a set as possible. Therefore, we distinguish two
of standard interfaces ─ which make it possi- types of parameters:
Test ble to build test adaptation libraries and reuse • Module level parameters, which are com-
Specific
Execution
Global them in different systems─, and the concept of mon to the whole test campaign and con-
Unprovisioning Unprovisioning ATS (Abstract Test Suite), which contains the tain general configuration information
logic defining the tests, expressed in a system- – these parameters are defined as modu-
Figure 2: Execution of tests including automated independent way, and is composed of several lepars in TTCN-3 terminology–.
service provision phases. reusable modules. • Test case level parameters, which are
In addition, some standardisation organisa- specific to each of the individual tests of
the system ready for the next test execution, as tions, such as ETSI or 3GPP (3rd Generation the campaign – defined as the formal pa-
described in Figure 2. Partnership Project) [4] are adopting TTCN-3 rameters of test cases –.
as a way for defining conformance test suites Therefore, test campaigns are defined by a set
for their specifications. This is the case, for of general configuration parameters and one
To be able to efficiently carry out tests for tele- example, for the conformance ATSs of SIP or more databases, which contain the specific
communication services, we have proposed (Session Initiation Protocol) [5], IPv6 [6] or parameters that define each of its individual
─and are currently using at Telefónica I+D─ a WiMAX [7] by ETSI. tests.
solution based on two fundamental principles:
• Using standard languages and tools for Modelling Complex Behaviours for Test- We propose to extract the main part of the
the definition and implementation of test- complexity of behaviours from TTCN-3 code
ing systems.
ing to external tools, which help us to specify tests
Despite the advantages of using TTCN-3 as
• Modelling complex behaviours of ser- and perform complex simulations.
testing language, it is not enough to define and
vice through high level models and for- Therefore, we have decided to make test cases
execute complete telecommunication services
malisms, which allows us also to highly as simple and general as possible and to pa-
tests. The TTCN-3 language ─as many other
automate both the definition and the ex- rameterise them completely. This means that
testing and programming languages─ is a rea-
ecution of tests. TTCN-3 test cases contain the code which is
sonably low-level language. This means that
• Using data-driven models of testing to necessary for the execution of quite generic
it allows to define tests to a very high level of
increase code reusability and to simplify tests, but the main part of the behaviour of the
detail, but it is difficult to define complex be-
test development and management. test is driven by test parameters.
haviours: using TTCN-3 every single step of
This implies that the number of parameters
the execution of tests must be coded, but the

www.testingexperience.com The Magazine for Professional Testers 71

needed for each test case is now very high duce the number of database fields; normally TTCN-3 development is needed.
─ there could even be hundreds of them ─. this message is a complex PIXIT (Protocol
However, as the number of parameters of the Implementation eXtra Information for Test- In a test case developed with the data-driven
test cases increases, it becomes more difficult ing), which means a PIXIT defined by a struc- approach, the messages sent and received,
to assign them their corresponding values. the number of
Sometimes these parameters are even too iterations and
Test Design
complex to be specified manually. the timers be-
tween iterations
For these reasons we have decided to use a set Test implementation are read from
of external tools that allow us to: a database, as
• Use a high level model for the specifi- TTCN3 TTCN3 described in
ATS ETS
cation of tests, which makes it possible Specification Compilation
Figure 6. Mes-
to easily define complex behaviours and sages are usu-
DB
service simulations. Specification
DB ally represented
• Automate the codification of parameters. by complex
PIXITs, in or-
These external tools separate the specification der to minimise
Test
of tests from their execution and contain the Execution
the number of
main part of the logic that models the behav- PIXITs handled.
iour of services under test, using, for example, With this meth-
Figure 4 : Data driven approach and TTCN-3.
rule-based systems – in which test behaviours odology all protocol messages are defined in
can be specified declaratively as sets of rules the ATS, and subsequently it is possible to
–. The external tools also offer a simplified tured type. configure any send/reception sequence defined
model and specific user interfaces for the Applying this philosophy, a testing design by the test design using a database. Therefore,
specification of large test campaigns, integrat- modification has a much lower impact in test- only one test case is required, which consists
ing the use of databases of test cases. They are ing implementation. It is enough to modify of a loop with n-iterations ‘send/reception’,
also responsible for the generation of the pro- a database, which could be an Excel file, for configurable through a database.
vision information and parameters and for the example.
automation of provision execution. The data-driven concept applied to TTCN-3 It is possible that there are test scenarios where
provides the following benefits: data are unknown at specification time, and
In Figure 3 we can see how the specification therefore they must be obtained during execu-
and execution of tests is separated: there ex- • Reduction of development costs: it is not tion time. For these scenarios, an import/ex-
ists a first phase in which the specification of necessary to undertake a new TTCN-3 port mechanism can be implemented.
tests takes place using a set of external tools. development each time the test design is
This phase produces as its result one or more modified.
databases which contain the parameters that • Version control and defect
begin
define each one of the individual tests of the probability minimisation:
test campaigns. The second phase consists of once the initially devel-
the execution of the previously simplified test oped ATS is debugged, it is
send
cases using the parameters contained in test not necessary to make any message
databases, which usually involve several pro- further ATS versions with
vision steps. each test design modifica-
tion and its debugging. Recieve
message
TTCN-3 data-driven testing • A person without TTCN-3
TTCN-3 development is mainly related to the skills can specify the tests.
testing implementation process. • Testing specification can
As a result of the TTCN-3 specification activ- be automated, for example send
message
condition
send
message
ity, an ATS (Abstract Test Suite) is obtained. by visual basic macros, set
This ATS is transformed into an ETS (Execut- of rules, etc. This makes
able Test Suite) after compiling, and is used in the specification task easi-
the testing execution process. er, and reduces the amount Recieve Recieve
message message
of repetitive work and the
If the traditional TTCN-3 development meth- probability of error.
odology is used, a testing design modification • Lower impact of SUT
implies repeating the TTCN-3 specification (System Under Test) up-
end
and compilation activities. In this situation, the dates in test implementa-
following disadvantages are found: tion: when there is a design
• Development costs: every time a de- modification in the SUT, it Figure 5 : Traditional TTCN-3 test cases.
sign modification is done, a person with is enough to modify the testing database Previously received data is exported to a data
TTCN-3 skills is needed to update testing rather than adapting the ATS. buffer, and is imported to the data buffer to be
implementation. • Reusability of test cases: it is possible to sent. This can be configured in the database by
• ATS version control: the number of ATS validate different services with the same means of import/export flags.
versions increases significantly; debug- test case by applying small configuration If any data calculation is needed during execu-
ging tasks become more complex, and changes. tion, then the appropriate keyword is defined
the probability of error increases. to be used in the scenario’s definition in the
Using the traditional methodology, as shown database.
With a data-driven approach, the ATS is de- in Figure 5, a test case is defined as a pre-de-
signed to read all values used during the ex- termined sequence of sent/received messages, Conclusion
ecution from a database. with some condition statements. This kind of The complexity and large dimension inherent-
As shown in Figure 4, the ATS fetches a com- test case implies that each time the sent/re- ly associated to telecommunication services
plete message from each field in order to re- ceived messages sequence is changed, a new testing imply special requirements to the test-

72 The Magazine for Professional Testers www.testingexperience.com

 of minimising resources and costs, compared
to a traditional solution.
Telefónica I+D is currently using this method-
ology to certify telecommunication services,
and it has obtained positive results by improv-
ing considerably the efficiency and quality of
the testing process.
SEND RECEIVE
MESSAGE MESSAGE
References
N
DB [1] “Telecommunications Act of 1996,” Pub.
ITERATIONS
(complex PIXITs) LA. No. 104-104, 110 Stat. 56, 1996.

[2] “TTCN-3 Home page”; http://www.ttcn3.

org/.

[3] “ETSI”; http://etsi.org.

TIMER
[4] “3GPP”; http://www.3gpp.org/.
Figure 6: Data driven TTCN-3 test cases. [5] “Conformance Test Specification for
SIP (IETF RFC 3261)”; http://webapp.
ing process of a telecommunication operator. etsi.org/WorkProgram/Report_WorkItem.
Telefónica I+D has chosen TTCN-3 to imple- asp?WKI_ID=19801&curItemNr=8&to
Within this context, the main prerequisites in ment the testing systems because it has been talNrItems=12&optDisplay=12&titleTy
the testing process are: necessity of simulating specifically designed for testing and certifica- pe=all&qSORT=HIGHVERSION&qET
the service performance, flexibility and repeat- tion, and it allows the reutilisation of test code SI_ALL=&SearchPage=TRUE&qETSI_
ability of the scenarios’ configuration, and ef- and libraries. In order to make the testing pro- STANDARD_TYPE=’TS’&qINCLUDE_
ficiency and reproducibility of test campaigns cess based on TTCN-3 more flexible, external SUB_TB=True&qRAPTR_
for new releases of the service. tools (rule-based tools) have been introduced NAME=Fischer&qRAPTR_ORGANISATI.
to separate the specification of tests from the
To deal with these requirements, Telefónica execution. These tools reproduce the service [6] “TC MTS-IPT: IPv6 Testing an eEurope
I+D has developed a testing methodology by using behavioural models. Project”; http://www.ipt.etsi.org/deliverable.
based on two basic principles: use of standard The combination of TTCN-3 testing environ- htm.
languages and tools for the definition and im- ment with the service’s behaviour modelling
plementation of testing systems, and replicat- has been successfully achieved by introducing [7] “Open IP Testing Library”; http://www.
ing complex service behaviours through high- a new approach: TTCN-3 data driven testing. ipt.etsi.org/STF295-ph1/home.asp.
level models and formalisms. This approach has several advantages in terms

Biography
Jesús del Peso graduated in Telecom- charge of managing
munication Engineering at the Technical a team working on
University of Madrid where he is currently the development
PhD candidate in Computational Mathemat- of testing tools for
ics and Artificial Intelligence. He is involved protocols of mobile
in researching in the field of multiagent communications
systems, semantic technologies and the based on TTCN-3
Semantic Web. (testing environ-
He has applied several IA techniques in ment defined by
several projects during his career as consul- ETSI). She has been
tant, and for the last four years he has been also involved in the
responsible for the design and development design, develop-
of several simulators and test specification ment and applica-
and execution test systems at Telefónica tion of other testing systems in the area of
Bárbara Martínez graduated in Techni-
I+D for environments ranging from SS7 (Sig- mobile communication, specially related to
cal Telecommunication Engineering at
nalling System number 7) to IMS (Internet Intelligent Network (IN).
Polytechnic University of Madrid. She has
Multimedia Subsystem). She graduated in Telecommunication Engi-
worked as a tester since 1999.
He has used TTCN-3 as the basis of his neering at Superior Polytechnic Institute of
As an ISTQB Foundation Level Certified
work in testing for the last three years, Zaragoza, (Spain) and made a M. Sc. thesis
Tester, she has developed different testing
participating in the last two TTCN-3 User on optical communication in Denmark Tech-
tools using TTCN2 and TTCN3 specification
Conferences. nical University. In 2000 she joined “Testing
languages.
Environment for Mobile Networks” Division
She has also taken part in the TTCN3 Users
in Telefónica I+D performing tasks related
Conference 2008, in Madrid.
Lourdes Calvo is a project manager at to specification and development of testing
Telefónica I+D in the “Service’s Monitor- to Intelligent Network services.
ing Platform” Division. She is responsible Currently, Lourdes is also involved in Eu-
for projects on testing of mobile services ropean projects related to testing such as
(Telefónica MoviStar). She is currently in Panlab and PII.

www.testingexperience.com The Magazine for Professional Testers 73

© iStockphoto

Control integration and test activities in your project

by Erwin Engelsma

Introduction upon mishaps or unexpected events I will show below (see Fig 1) what these ele-
I suppose we all know the problems that proj- • High level of cognitive dissonance need- ments and the MID look like. I would like to
ects have when they start to integrate products ed before project management sees prob- stress that the MID differs from a Gantt chart
from multiple development teams. Partly fin- lems in that the MID provides necessary technical
ished products that seemed to work correctly (or other) relationships, and does NOT plan
in isolation do not work well when integrated. So-called ‘Agile’ or ‘Evolutionary’ methods activities.
Testers find problems more rapidly than devel- help to reduce this problem. Using Evolution- In order to make the MID easily creatable (no
opers can solve them (or even assign them to ary methods you may find faults quicker (you special tools needed), we use only a few sym-
subprojects), and all of a sudden a project that detect them sooner after the development team bols that you can find in Word, and connectors,
seemed to run on time gets delayed. Depend- introduced them). so we can easily shift the elements.
ing on the experience of the project team, tes- Philips Healthcare uses evo-
ters may get blamed, or may get asked to test lutionary methods to develop
less, in order to meet the project deadline. This products and within this con-
usually results in bad product quality going text my group applies a method
to the customer, which will generate a busi- that:
ness headache in the future. The problem gets
more difficult to handle if the project devel- • Controls integration steps
ops software from several software disciplines based on pressing techni-
and hardware (electronics) or mechanics, or cal relationships
employs several teams. Many different causes • Makes visible (graphical-
may underlie this phenomenon. To mention a ly) how the project team
few: will integrate partly com-
Fig 1 Example of MID
pleted products
• Measurement of project progress in terms • Plans development of test means, meth-
of ‘phases’ completed, like: ‘Requirement ods and environment Discussion of Deliverables
document completed’, ‘Design document • Monitors project progress in terms of Fig 1 gives a simple example of an MID.
what the teams measurably achieved On the left it depicts Deliverables. Each De-
ready’
liverable has its own identification, from which
• Overly optimistic view of the design
complexity The graphic overview that displays the needed you can derive:
• Designs not clear enough, ambiguous, or information we call the ‘Master Integration • What is the content (Functionality, Inter-
Diagram’. faces)
not communicated well enough
• Which development group is responsible
• Misunderstandings between people from
for creating it
various disciplines Master Integration Diagram description • What level of quality does it have on de-
• Overly optimistic planning The Master Integration Diagram (MID from
livery
• Insufficient or skipped (time pressure!) now on), makes use of the following ele-
The Deliverable describes a product in a cer-
test ‘moments’ ments:
tain, not necessarily completed, state. Howev-
• Insufficient synchronization between de- • A Deliverable (D) from a development
er, the stakeholders know what state it has.
liverables from various teams team
In Fig 1 each ‘document’ symbol with a ‘D’
• Insufficient control over interfaces (lack- • An Integration Moment (IM) between
indicates the Deliverable. In reality you may
ing formal verification) deliverables
add some references, e.g. to an Excel sheet
• Insufficient design modeling • A Test Moment (TM) for integrated de-
giving details.
• Insufficient attention paid to test method- liverables
Each Deliverable may have its own ‘sub’
ology and environments • An Increment Delivery Moment (IDM)
MID, which charts how the team develops
• Essential parts of the test environment not for completed functionality that the proj-
this Deliverable. In this way the development
developed, or have insufficient quality ect will deliver to a customer
teams can use the MID in a scalable way, and
• Insufficient information present to act

74 The Magazine for Professional Testers www.testingexperience.com

 ISSECO
SECURE SOFTWARE ENGINEERING
®

ISSECO® – INTERNATIONAL CERTIFIED PROFESSIONAL FOR SECURE SOFTWARE

ENGINEERING

In times of daily internet attacks, customers have grown accustomed to relying heavily on their software security.
Every security breach can lead to a damaged reputation which in turn can result in customer loss. The development
of secure software requires special knowledge of the entire software development process. Together with some of
the best names in the field, iSQI has developed the Certified Professional for Secure Software Engineering.

CONTENT:
The Certified Professional for Secure Software Engineering guarantees knowledge about aspects of security in all
phases of software development (software lifecycle):

· Requirements Engineering
· Design & Specification
· Programming & Testing
· Evaluation, Distribution & Maintenance

TARGET GROUP:
· Software Developers & Development Managers, Requirement Engineers, Quality Managers, Security Managers

CERTIFICATION:
· Certificate “Certified Professional for Secure Software Engineering”

FOR MORE INFORMATION VISIT: WWW.ISSECO.ORG

MEET US AT CONQUEST 2008 / www.conquest-conference.org

The conference program is included in this issue of testing experience. International Software Quality Institute

www.testingexperience.com The Magazine for Professional Testers 75

show only the details that interest the rest of may also use this symbol to indicate a delivery ing colors or other indicators, in order to indi-
the team. to an MID at a ‘higher level’. In that case, other cate information relevant to your organization.
‘light-weight’ acceptance criteria may apply. A I have not discussed the relationship between
Discussion of Integration Moments product may have several of these moments, MID and Configuration Management.
Fig 1 depicts three Deliverables that have enabling evolutionary development.
some technically or logically necessary reason References
for integrating them together in the Integra- Applying the method Dieter Arnouts: Test management in different
tion Moment. This means that if one of the Unless the relevant stakeholders understand Software development life cycle models, Test-
teams does not deliver any of the Deliverables, the MID and wholeheartedly support it, it ing Experience, June, 2008
the Integration Moment cannot take place. will not do much good and just gather dust in
The combination simply will not work. The some drawer. We therefore introduce the role Timo Käkölä; Juan Carlos Dueñas Lopez,
Integration Manager (new project role, see: of Integration Manager. A designer or senior Research Issues in Engineering and Manage-
Applying the method) only groups elements tester may fulfill this role. As a first step he ment: E.S. Engelsma: Chapter 12: Incremental
together that have this relationship. makes an overview of the stakeholders for Systems Integration within Multi-disciplinary
Each Integration Moment has its own identifi- this project. He may then create an overview Product Line Engineering using Configuration
cation which describes in detail what activities of what the integration activity will look like. Item Evolution Diagrams
are needed to integrate the Deliverables. As an Then he will organize ‘beamer sessions’ with
Example: The three Deliverables may stand the stakeholders (or a subgroup, depending on
for a motor movement control involving a: the exact content of the discussion) to fill in
• Motor the details of the diagram. In a beamer session
• Electronic motor controller/driver a secretary uses the PC to display the draw-
• A software layer that generates move- ing, and to quickly make changes as proposed
ment profiles and agreed by the team. You may use a white-
Obviously, I could sum up a very large number board for this. The Integration Manager makes
of examples. certain that all persons present understand the
technical issues, and the added value of testing
Integrating means: at a certain moment. He also makes sure that
• Generating the software designers can design and develop (or specify)
• Doing build tests on the software the relevant test equipment.
• Connecting the electronics
• Showing ‘basic functionality’ Critical success factors
• And so on, depending on your organiza- The method will only work if the project ad-
tion heres to a number of critical success factors.
These are:
Discussion of Test Moment • Stakeholders understand the way of
In order to test, the tester needs access to the working with the MID and support it
delivered product and he must also have the • Involvement of all stakeholders
test environment up and running. Most impor- • Minimum level of abstraction at which
tantly, he must know the purpose of this test. testers can partake in the discussion
In our example that would mean: • The motivation to see development of
• Test documentation test tooling as real product development
• Supplying a (laboratory) power supply • Cooperation across disciplines
• Organizing a local PC connected to the • Good insight into what you will develop
electronics
• Designed test scripts
• Creating and downloading test scripts
Practical results
Within Philips Healthcare we have used the
Biography
• Having a means of verifying that the method several times.
motor makes the expected movements, The second Dutchman to pass the ISEB
Without exception, we found this leads to
has vibration within the accepted range Practitioner level with ‘distinction’, he
much better insight for stakeholders to under-
and any other measurement equipment now works as Test Architect for Philips
stand what they contribute to the integration
needed to test any parameter the project Healthcare. He has worked with ITEA
process. The project manager had better con-
should control at this time projects doing research into product
trol, and projects could be completed faster.
• And some other standard tools like re- family development related test issues,
Also, due to faults that the testers found at an
porting and so on and has given several presentations at
early stage, developers did considerably less
The Deliverable leading straight into the Test SQS congresses and other organizations.
rework than in earlier comparable projects,
Moment symbolizes all equipment and other In the past he was also employed as
which is what led to faster project comple-
test environment needs that a team must spe- system designer for MRI equipment, and
tion. Furthermore the MID helped manage
cifically develop or buy. Naturally, the Test was head of a testing department for 8
unexpected behaviors in the outside world like
Moment also has a document it refers to which years during which he introduced TMM
a change in requirements or unexpected tech-
describes the test purpose, the types of test and started a group building Automated
nical problems. In the latter case, the project
done, acceptance criteria and so on. Test Equipment. He spends most of his
team could easily identify which technical ar-
spare time studying and playing the
eas the technical problem affected.
clarinet, and performing with his Klezmer
Discussion of Increment Delivery Mo- band at weddings and festivals, or he
ment Conclusions studies something else. Those ‘in the
This symbol indicates when the teams have Good use of the MID contributes to better proj- know’ about NLP-related aspects of
developed sufficient functionality to deliver ect control and faster project throughput, with documenting will notice NLP’s influence
this to a customer. This may include a formal inherently better quality of the final product. on his article.
validation, and possibly acceptance tests by the Most people do not find it difficult to learn,
customer, and handing over of user documen- and do not need additional tooling. In practice
tation as defined by your organization. Teams you can extend the ideas shown above by us-

76 The Magazine for Professional Testers www.testingexperience.com

 Diffusion Methodology for Regression Test Analysis

by Oron Dilmoni
© iStockphoto

Abstract The methodology is meant for organizations doesn’t mean we should test the whole system.
Regression tests were always a very hard which have a lot of regression test cycles, The term ‘regression test’ refers to the overall
task. especially for global companies dealing with process as it will be detailed in this article and
The difficulties stem from the fact that you rollouts. the methodology which offers a mechanism to
touch the code and you should select the rel- analyze the change and decide about the next
evant test cases for retesting. Introduction steps).
But, what are those test cases? How can you Regression tests are a very important part on
decide which test case to retest and which one our lives as SW testers. Note: It should be noted that the proposed
to skip? How can you be sure that you ana- We come across them almost everywhere and methodology is a very useful tool. The way
lyze the system correctly and select all the test all the time: after bug fixing, after change re- to use this tool (including the scales, formu-
cases that were affected by the changes? And quests have been implemented, after version las, risk analysis, etc. proposed here) might be
what happens if there were several changes? upgrades, etc. changed in order to suit the specified organiza-
IEEE defines regression testing as “…the way The first question we should ask is: “When” tion.
to catch a large class of these bugs quickly should we perform regression tests?
and efficiently. Regression testing focuses on As a matter of fact, surveys show that only Regression Test Flow & Stages
ensuring that (theoretically) everything that about 30% of software development relates In principle, as mentioned in the abstract,
used to work still works.” to new developments, and the rest are mainte- when inside the project, we must make deci-
But how can you do it quickly and efficient- nance, support, bug fixing, version upgrades, sions regarding the regression tests quickly.
ly? etc. Therefore, our main work as SW testers Therefore, we do not want them to be taken
is to test changes in existing systems, which randomly, by the wrong persons, on the basis
The answer, surely, must be based on “Risk means regression tests. of wrong circumstances or missing informa-
Analysis”, but we must remember that the rea- tion. So the idea is to create a consistent analy-
sons for retesting are the changes in the code. There are a lot of definitions for the term itself, sis method for all code changes.
So the most risky issue should be the changes, but there is no doubt that it basically deals with
and accordingly you should manage all your changes in the code. The flow is divided to 4 stages:
risk analysis based on this. IEEE defines the term as follows: “Many
Regression
products go through several releases, and can Methodology
If you are a global company dealing with roll- be ported to many platforms (hardware and/
outs, then your problem is much more com- or operating system). Every line of code writ-
plicated .The SW is developed for one coun- ten or modified offers opportunity for bugs to
try or branch and then it is being customized Regression
creep in. Regression testing is the way to catch Regression
Evaluation
and propagated to another country or branch. a large class of these bugs quickly and effi- Regression
Evaluation
However, by customizing the code for the ciently. Regression testing focuses on ensur- Evaluation
second country or branch (the first rollout), ing that (theoretically) everything that used to
what happened to the first country? What tests work still works.” Regression
should I do in order to ensure that it will not The FDA GPOSV (January 11,2002) definition Tests
Regression
Regression
be impacted? And what will happen in the sec- reads: “Regression testing is the rerunning of Tests
Tests
ond, third (and so on) rollouts? test cases that a program has previously ex-
ecuted correctly and comparing the current
This article describes a new methodology result to the previous result in order to detect Regression
Regression
named “The Diffusion Methodology for Re- unintended effects of a software change.“ Report Regression
Report
gression Test Analysis”, which enables you to Therefore, we can absolutely conclude that Report
manage a quick and efficient regression test regression tests should be done after every
mechanism. change in the code. (As we will see later, this Figure 1 – Regression Tests Basic Flow & Stages

www.testingexperience.com The Magazine for Professional Testers 77

Stage 1: Regression Methodology – Each but just like when we define regression tests, it each change in the project and decide about the
organization shall once create its own meth- should be focused on the processes potentially required regressions efficiency and quickly. At
odology regarding the regression tests, based affected by the change. the regression evaluation stage, the unaffected
on regulations and the company policy. The The Risk Analysis should first find the affected processes are basically sifted out, and the other
methodology shall define the whole process, processes and focus on these. The other types affected processes which are likely to be risky
responsibilities, and milestones, and it shall of risk are not relevant if the process was not are grouped.
describe how we are going to handle each type affected (they might be relevant if the process
of “When”? is affected). The other ‘high-risk types’ might This is done in 3 stages:
be integrated in the methodology and re-tested 1. Building the Regression Influence Map
Stage 2: Regression Evaluation - Analyzing the as a company decision but not enforced. 2. Recursive Diffusion (3 cycles)
impact of each change (when it occurs) based 3. Filtration (after deciding about the re-
on the review of the relevant documentation So, how we will find the affected processes? gression threshold)
and in accordance with the specific strategy
‘event’. Building Project Influence Map Building Regression Influence Map
As the regression decision should be require- At this stage, we have changes that were done
Stage 3: Regression Tests - Regression tests ment-based and not testing-based, the idea is to the system, and we have the influence map
should be performed and should be focused on to already build an influence map in the speci- which was designed in the specification stage.
several aspects: new processes, changed pro- fication phase. While defining all the business These should now be combined into one ta-
cesses, deleted processes, affected processes processes of the system and all the require- ble.
and bug fixing. ments, the system analysts should build a very
simple map which will point out the reciproca- This could be done using the following simple
Stage 4: Regression Report – Summarizing tion between each 2 processes. This pre-anal- table:
the results and deciding about more cycles as ysis is done by the professional analysis team
deemed necessary. at the right stage and, apart from saving time Process RCat Affected RFactor
later on, it helps the project team to working (Req.) processes
Stage 1 - Regression Methodology consistently. This is because the whole team
will work with the map and decisions will not 1. Order to 3 2 60
Each organization or sometimes each project,
be taken randomly by the first person to get it. Cash
shall define how it is going to deal with the
Regression tests issue. The organization shall 2. Inventory 4 --- 80
adopt a set of rules which will later help to This might be done by using the following Management
take the decisions quickly. This article might, simple table: 3. Turning on 5 --- 100
for example, be adopted as a methodology, but the system
even if this is the case, the organization shall Process (Req.) Affected
still consider specific issues which are only processes Figure 2 – Project’s Processes influence Map
relevant to it.
1. Order to Cash 2
When building the Regression methodology, The Process column and the Affected Pro-
you should consider at least four important is- 2. Inventory Management --- cesses column already exist. We only have to
sues: 3. Turning on the system --- evaluate the changes and fill in the RCat &
1. Risk Analysis methodology for Regres- RFactor columns.
sion tests. Figure 2 – Project’s Processes influence Map The RCat is the (Regression Category), which
2. Project Influence Map. is represented by one of 3 options (3 - Process-
3. Cause & Results plan. In the Process column we should map all the es that were not changed since the last tests; 4
4. Responsibilities. processes from the process tree. In the “Af- - Processes that were configured and changed
fected processes” column, the team should by system configuration mechanisms; 5 - Pro-
Risk Analysis for Regression analyze only the processes which will be af- cesses which were changed and involved de-
While thinking on the first issue, we can deter- fected directly by any change in the process. velopment/changes in the code.):
mine that Risk Analysis will definitely help the (As we are working with a diffusion method, RCat = 3 or 4 or 5
organization to identify the processes which it is very important to map only the direct in-
were not changed at all and those which were fluence). And the RFactor:
not even impacted. RFactor = 20 * RCat
Let us agree that: Cause & Result plan
Risk = Severity * Probability As part of the Regression Methodology, the At that point, we can already easily find the
project management should think about as processes which were directly affected by
Then under the assumption that the tests will many situations of unexpected changes as pos- sorting the influence table.
cover all the system and the system worked sible, and define what the required actions for In fact, all processes marked with RFactor =
fine before the changes, the probability for each situation are. 100 are risky/ affected and must be retested.
bugs in processes which were not changed is: All processes marked with RFactor = 60 are
Probability -> 0 Responsibilities not directly affected.
As part of the Regression Methodology, the
So we can conclude that: project management should identify the exact Recursive Diffusion
Risk = Severity * 0 persons who will have the right to perform or In the Diffusion stage, the objective is to find
=> Risk = 0 to approve each operation. For example, who the processes that might be affected transitive-
will build the influence map, who will ap- ly (not directly). For example, if process A af-
prove changes in that map, who is going to fects directly process B and process B affects
And therefore, we can define in our Regres- approve bug fixing, who is going to analyze directly process C, then process C might be af-
sion methodology that the severity should be the required regression tests for that change, fected by any changes in process A transitive-
considered only if the process is affected. and so on. ly. The depth of the transitive relation might be
controlled by the organization according to the
The Regression Strategy is based on Risk Stage 2 - Regression Evaluation number of recursive evaluations performed.
Analysis. The risk analysis might consider any Once we have the Regression methodology, The recursive evaluation is done only to pro-
risk types (finance, business, regulations, etc.) the infrastructure is ready and we can handle cesses classified as Rcat 4 or 5, because pro-

78 The Magazine for Professional Testers www.testingexperience.com

© iStockphoto

Wachsen Sie mit uns!

Die Díaz & Hilterscheid Unternehmensberatung GmbH (www.diazhilterscheid.de) unterstützt seit 1998 individuell
in den Beratungsfeldern Financial Services, IT Management & Quality Services, Training Services und IT Law.

Die Mehrzahl unserer Kunden sind Kreditinstitute, Versicherungen und IT-Dienstleister für Finanzinstitute. Im Zuge
unseres weiteren Wachstums suchen wir zur Verstärkung unseres Bereichs IT Management & Quality Services zum
nächstmöglichen Termin mehrere:

Senior Consultants IT Management & Quality Services (m/w)

für unsere Kunden aus der Finanzwirtschaft
Deutschland
Europa

Ihr Profil:
• Fundierte Ausbildung oder Studium (Informatik, BWL, Mathematik, etc.) sowie mehrjährige Berufserfahrung als
IT-Consultant in einem Fachbereich bzw. in der Organisation eines Finanzunternehmens, eines finanzwirtschaftli-
chen IT-Dienstleisters oder in einem Beratungsunternehmen in entsprechenden Projekten
• Erfahrung in der Führung von großen Teams (Projektleiter, Teilprojektleiter, Testmanager)
• Kenntnis und praktische Anwendung von Methoden und Standards, wie CMMI®, SPICE, ITIL®, TPI®, IEEE, ISO
9126
• Eigeninitiative, Verhandlungs- und Gestaltungsgeschick, repräsentatives Auftreten
• Hohe Reisebereitschaft
• Gute Englischkenntnisse in Wort und Schrift

Wir bieten:
• Interessante und anspruchsvolle IT-Projekte in den Bereichen IT-Prozesse, MA-Qualifikation und Coaching, Soft-
waretest sowie Toolauswahl und -einführung
• Weiterentwicklung in einem flexiblen Unternehmen mit eigenen Gestaltungsspielräumen
• Attraktives, erfolgsbezogenes Gehalt

Ihr Kontakt:
Bitte senden Sie Ihre aussagekräftige Online-Bewerbung an unseren Bereich Personal
([email protected]). Ihre Fragen im Vorfeld beantworten wir gerne (+49 (0)30 74 76 28 0).

www.testingexperience.com The Magazine for Professional Testers 79

cesses classified as 3 are not affecting anything can contribute “0.8” or “1.0” to the RFactor Note: Even though the process looks com-
(3 = processes that were not changed). It is value of the affected process. plicated, the effort might be assimilated very
recommended that the phase is performed in This means that if RFactor is higher than 80, easily and its contribution will have an impact
another 2 recursive cycles according to the fol- the process might be changed directly by cus- very soon.
lowing formulas: tomization, or it is at least 2-3 times directly
• For cycle II: affected, or it has more than 20 2nd level af- Stage 3 - Regression Tests
RFactor’ = RFactor’ + RFactor/10 fections. So, it will be good to decide that: When working with a strong methodology, the
Where RFactor’ is the RFactor of any process Regression Threshold < 80 Regression Tests themselves become the very
in the ‘Affected Processes’ column of the orig- easy part. At this stage, you should only ex-
inal process. However, if we analyze the RFactor further, ecute the selected test cases according to the
we will find that even if the RFactor = 68, it tables from previous stages.
• For cycle III: means that the process is directly affected, or it Of course, it is necessary to track the execu-
RFactor’’ = RFactor’’ + RFactor’/100 has more than 10 2nd level affections. tion and to identify whether there are any ar-
Where RFactor’’ is the RFactor of any process So, it will be good to decide that: eas with a lot of problems. If we identify, for
in the ‘Affected Processes’ column of the 2nd Regression Threshold < 68 example, a process which has a lot of failures,
level of affected processes. Therefore, it is recommended to define the Re- it is recommended to go back to the influence
gression Threshold as: table and to re- evaluate the affected processes
Note: In very solid organization, it is possible Regression Threshold = 68 of this process.
to perform more and more recursions. The for- (But in any case it will be: 60 < Regres- In some cases, the above methodology might
mula in each cycle shall be changed only by sion Threshold < 68) prove to be a much stronger tool, since you can
another division by 10, because the level of perform the diffusion only on the identified
influence becomes more and more indirect and Let’s take the following example: failure process and also identify precisely the
therefore the risk reduces accordingly. affected processes at all levels. In this case, it
Process RCat Affected RFactor would be recommended to reduce the Regres-
Filtration (Req.) processes sion Threshold for that evaluation and execute
At this stage, we already have an updated in- more tests.
fluence map, where the RFactor has actually A 60
become the decisive criterion. The higher the B 60.8
Stage 4 - Regression Report
RFactor, the higher the risk that the process is C 61 Like every other test, the regression cycle
affected by the changes.
D 61.6 should be summarized and reported (by pro-
We might find a lot of RFactor values for the
ducing a test report). The more important role
different processes in the table. All we have E 61.8
at this stage of dealing with regression is to
to do is to define the RFactor threshold and to F 62 decide whether more rigorous operations are
filter the table accordingly – only processes
G 62.4 required.
whose RFactor is higher than the threshold
We should remember that sometimes pro-
should be retested. H 62.8
cesses are being added, and they should also
I 64 be added to the influence map with all their
The role of the filtration process is to filter all
J 68 relations. Likewise, sometimes processes are
the unaffected processes.
K 70 deleted, and they should also be removed from
As stated in the introduction, unaffected pro-
the other process relations. In addition, as pro-
cesses should not be retested (no matter what L 76 cesses are updated, they might affect more
the severity is):
M 80 processes or stopping affecting others.
If RFactor = 60 => No regression on
N 88 So, after finalizing the cycle, it is necessary to
this process
go back to the influence map and check if it
O 90 needs to be updated.
And since all processes which might be direct-
P 96
ly affected must be retested again,
we have to identify the highly risky processes Q 98 Advantages
which might be directly affected or which have R 100 The advantage of this methodology is first and
a lot of 2nd level impact: foremost to have a methodology for Regres-
S 106 sion Tests, when usually organizations do not
If RFactor > 100 => Regression must
be executed function in this field with an approved meth-
Figure 4 – Example: Regression influence map odology, but simply let projects flow.
All processes which are between 60 and 100 Apart from this, there are more specific advan-
• Process A – was not affected at all by any tages:
will be filtered by the ‘Regression Threshold’. change – will not be retested.
This threshold might be decided based on the • All the changes are handled by the same
• Process B – was affected only once in mechanism, no matter if it is a major
existing RA, considering also the severity of 2nd level from one customization – will
the process (requirement). version upgrade, minor version, change
not be retested. request, bug fixing, or extensions and so
If RFactor > ‘Regression Threshold’ • Process H – was affected only in 2nd
=> Regression must be executed on.
level (2 times by changes in the code and • The thinking part is done earlier (during
If RFactor < ‘Regression Threshold’ one time from customization) – will not
=> No regression on this process the specification), which will save time
be retested. during later stages, which are much more
• Process I – was affected only in 2nd critical for the project.
The Regression Threshold should be decided level (4 times by changes in the code, or
by the organization and might be changed ac- • This thinking is done consistently by the
5 times by customizations) – will not be correct persons and not randomly by the
cording to different needs. In principle, all the retested.
processes which are not affected are allocated wrong persons.
• Process J – was affected directly by cus- • The methodology enforces good practic-
an RFactor of 60. Each “affection” of the first tomization or even if not directly, then it
recursive cycle can contribute “8” or “10” to es and correct procedures, so the projects
was affected at least 8 times by 2nd level become well documented and structured.
the RFactor value of the affected process, and – will be retested.
each affection of the second recursive cycle • The risks are reduced through mapping
the affected processes exactly, so there

80 The Magazine for Professional Testers www.testingexperience.com

 are much fewer cases of failure after regressions.
• The scope of the regression tests might be calculated much earlier (while plan-
ning the changes), and it is possible to create very accurate plans.
• The methodology helps to get the correct decisions for unexpected occurrenc-
es during the project life cycle (as the map tool might be used for reactions to
failures detected and for re-planning), so the organization becomes much more
flexible for changes.
• The methodology enables a unique language and unique procedures within
the organization, since all projects and all team members work with the same
methods and with the same actions.

Summary
see Fig. 5

References
[1] – FDA General Principles for Software Validation (January 11,2002).
[2] - GAMP Good Practice guide: Testing of GxP systems (2005)
[3] - GAMP 4 “Guide for Validation of Automated Systems” –issued in December
2001
[4] - IEEE STD 829-1998, “Software Test Documentation” – updated in 1998.
Biography
Oron Dilmoni has been working in the
software testing area for 10 years. Oron
started is career as a SW testing engi-
neer and during the years he became a
Testing Project Manager, Senior Testing
Project Manager and Practice Manager.
During his career, Oron managed dozens
of SW testing projects in various fields,
sizes and types. In his latest role, Oron
was a branch manager at Tescom Israel.
Oron established a nearshoring branch
supplying SW testing services for the
Israeli market.

Regression Flow & Stages

Done only once per

Methodology

organization or per
Regression

Risk Analysis for Create Influence Cause & Results Define Respon- Decide about
Regression Map Plan sibilities Regression project. (Should be
Threshold maintained during the
project)
Change
Occur
Regression

Take influence Map identify the changed Perform 3 cycles of Filter the processes
Evaluation

and Classify all pro- processes and mark Diffusion table to RFactor > Re-
cesses their Rcat = 4 or 5 gression Threshold

Done consistently
Regression

Test only filtered Track results and for every change

processes perform re-evaluation occured during
as needed the product life
Tests
Regression

Create STR Decide of more rig- Update the Influ- Wait for the next
orous operations ence Map change
Report

are needed

Change
Occur
Figure 5 – Summary chart for the entire Diffusion Methodology

www.testingexperience.com The Magazine for Professional Testers 81

 Test design techniques were not invented to
bully testers!

by Leo van der Aalst

© iStockphoto

Frequently, clients and even testers complain Having determined the characteristic to be and time-consuming business? It would seem
that using test design techniques is a difficult tested and the test intensity, one or more suit- this is a matter of ‘unknown, unloved’!
and time-consuming business. If they can get able test design techniques can be selected to
away with it, they would prefer not using any create the test cases (See Figure 1: From test The quantity and abstract nature of techniques
techniques at all! That’s a pity, because these goals to test cases, below). is an important cause of their being ‘unloved’.
techniques represent the only way to realise
the agreed test strategy in a demonstrable way.
This article provides you with the tools to se-
lect one or more suitable techniques.

Substantiate the test strategy with test

design techniques
After the test goals and product risks have
been established via a product risk analysis,
the resulting test strategy should be substanti-
ated, according to the intensity of testing for
a specific combination of characteristics and
object parts.

IN MORE DETAIL Figure 1: From test goals to test cases

To avoid overloading the tester such that he
• A test goal is a success criterion for TMap Next [Koomen et al, 2006] describes a ‘can’t see the wood for the trees’, it is better
the test assignment specified in the large number of techniques and a great many not to teach or explain every single test design
customer’s language. other techniques can be found in books and on technique. Practice has shown that if the tech-
• A product risk is the chance that the Internet – including some created by tes- niques are explained well, with examples from
the product fails in relation to the ters themselves. Of course, variations of any a tester’s own immediate work environment,
expected damage if it does so. technique can be also generated. the tester ‘suddenly’ does not feel they are so
• The test strategy is the distribution difficult and sees the benefits of using them.
of the test effort and test intensity If an unsuitable technique is selected or no The reason for this is that using examples
over the combinations of charac- techniques are used, all of the previous steps from the tester’s work environment eliminates
teristics and object parts aimed at will have been in vain, and it becomes very the abstraction of a technique, and allows the
finding the most important defects difficult to make a judgement as to whether the tester to see its practical application and moti-
as early as possible and at the low- test goals have or have not been realised – with vates him to put into practice at his workplace
est costs. the attendant risks. For instance, the chance of what he has learned.
• The intensity of testing is light, production disruptions grows - unfortunately
average or thorough; it is part of still a common situation. And in terms of Also some organisations want more ‘certainty’
the test strategy. test governance, test goals cannot be traced and assurance that their testers have an ad-
• Characteristics include amongst through to test cases. equate knowledge of test design techniques.
others functionality, user-friendli- Solutions include encouraging or even man-
ness and security. dating their in-house testers to acquire formal
Are test design techniques difficult and
• Object parts are usually the certification or asking for external certified
sub-systems of the application time-consuming?
testers when testing.
software. Why is it that clients and testers often consider
that using test design techniques is a difficult

82 The Magazine for Professional Testers www.testingexperience.com

Often, when the use of test design techniques In Figure 2, the activity ‘Take measures’ may
is described as time-consuming, people forget include: IN MORE DETAIL
that the techniques may be used ‘incorrectly’. • Required test basis not available
The aim is not for the tester to design every -- ask designers to adapt the test basis Technique is imposed
possible test case, but rather that he selects a so that the technique can be used Not every company, industry or specific
specific technique in relation to the selected -- ask testers to adapt the test basis so application allows a tester to ‘freely’
test strategy - aiming to achieve the highest that the technique can be used select a technique; it may be prescribed.
possible ‘defect-finding chance’ with the least -- organise information sessions to An example from the aviation industry il-
possible number of test cases. In practice, we achieve a usable test basis. lustrates this situation. Aircraft can only
find that testers frequently make the wrong use software that aviation authorities
choice. As a result, an excessive number of • Inadequate knowledge and skills on the have found to be “safe” for aviation pur-
test cases are designed. This is an important tester’s part poses. To this end, the American Radio
cause for considering the use of techniques to -- train the testers in the proposed tech- Technical Commission for Aeronautics
be time-consuming. nique (RTCA) and the European Organization
-- select another technique because it for Civil Aviation Equipment (Eurocae)
is a better match with the tester’s have developed a standard: DO-178B
IN MORE DETAIL knowledge and skills. for America and ED-12B for Europe.
These standards classify software
Defect-finding chance • Labour-intensiveness disproportionate to systems according to the consequences
Let’s say that you have a ‘travel reserva- the time available for the aircraft and its passengers if a
tion system’ with the following param- -- make the test less intensive system should fail. The consequences
eters and equivalence classes: -- make more time available. range from ‘no negative impact’ (level
Number of days : 8; 15; >15 E) to ‘catastrophic’ (level A). DO-178B
Amount (euros) : <500; 500-1000; Clearly, the proposed measures must be agreed and ED-12B require the coverage type
>1000 with the client, since they may have an impact “decision points modified condition/
Membership card: none; silver; gold; on the agreed result, the risks to be covered, decision coverage (MCDC)” for level A
platinum the estimated costs, and/or the planned time. testing. The American and European
Departure date : workday; weekend; aviation authorities, the Federal Aviation
bank holiday The technique that is the result of using this Administration (FAA) and the European
You need 3x3x4x3=108 test cases method of selection is a suggestion only; there Aviation Safety Agency (EASA), accept
to test all possible combinations (the may, of course, be reasons for selecting an- this standard for certifying aviation
complete decision table). If you use other technique. The selection diagram is just software systems.
the technique ‘pairwise testing’, you a tool. Finally, sometimes, a certain technique
only need 13 test cases (if using the is imposed on a tester, for reasons of industry
‘Allpairs’ tool1). regulation or standardisation.

Research conducted by the National

Institute of Standards and Technology
[Kuhn, 2000] shows that 98% of all
defects are found when pairwise testing
is used. This is because just 2% of all
defects are caused by a problem in the
combination of three parameters or
more! In other words, 12% of all pos-
sible test cases will be enough to detect
98% of all defects in the above example.
Start

Table 1 Table 1
Choosing the best possible technique Select a combination of characteristic Remember the technique
After the test strategy is determined, suitable and test intensity as proposed on the intersection

techniques must be chosen. This is not always

easy - after all, we must take a large number of Table 2 Table 2
variables into account: Remember the required test basis
Select the row with the proposed technique
• characteristic as mentioned in the row

• test intensity
• test basis
• knowledge and skills of the testers Does the required Y Do testers Y Is the technique’s Y
have the knowledge labour-intensiveness
• labour-intensiveness of the technique. test basis coincide with
the available one?
and skills to implement proportionate to the
the technique? available time?

The flow chart illustrated in Figure 2: Tech-

nique selection diagram is a valuable means N N N

of making the technique selection, in conjunc- Select the

Take measures
tion with the information in Table 1: Proposed technique

technique for a specific combination of char-

acteristic and selected test intensity, and Table End
2: The required test base for a proposed tech-
nique. Figure 2: Technique selection diagram

1
The ‘Allpairs’ tool was created by James Bach and can be downloaded from http://www.satisfice.com. Another tool is ‘Pict33’ by Microsoft®, which can be downloaded
from http://www.pairwise.org. The DaimlerChrysler tool, ‘Classification tree editor’, can also be used; this can be downloaded from http://www.systematic-testing.com.

www.testingexperience.com The Magazine for Professional Testers 83

 Test intensity
Characteristic
Light coverage Average coverage Thorough coverage

Checklist
DCoT-equivalence classes
PCT-test depth level 1 DCoT-pairwise testing
Manageability PCT-test depth level 2
UCT-checklist PCT-test depth level 3
ET
EG

DCoT-equivalence classes
Checklist DCoT-pairwise testing
Security SEM-modified condition/decision coverage
EG Penetration test
ET

UCT-checklist PCT-test depth level 2 RLT-operational/load profiles

Usability
EG UCT-paths UCT-decision points

RLT-operational/load profiles
Continuity EG RLT-operational/load profiles
ET

DTT- condition/decision coverage DTT- multiple condition coverage

DCoT-pairwise testing
Functionality DCoT-equivalence classes (+ boundary values)
ECT-modified condition/decision coverage
- detail ECT- condition/decision coverage DCoT-N-wise testing
ET
EG ECT-multiple condition coverage

DCoT pairwise testing

DCoT-N-wise testing
DCyT (life cycle of the data) CRUD
DCyT (life cycle of the data) CRUD (extra Rs)
DCoT-equivalence classes DCyT (integrity rules) decision coverage
DCyT (integrity rules) modified condition/decision
Functionality SYN-checklist (limited) PCT-test depth level 2
coverage
- overall UCT-checklist SYN (prioritised list)
RLT-operational/load profiles
EG SEM-condition/decision coverage
SEM-modified condition/decision coverage
UCT-paths
UCT-decision points
ET

Functionality SYN-checklist (limited) SEM-condition/decision coverage

SEM-modified condition/decision coverage
- validations EG SYN (prioritised list)

PCT-test depth level 2

SYN-checklist (limited) Usability test
User-friendliness SYN (prioritised list)
EG (possibly in lab)
UCT-checklist

Infrastructure RLT-operational/load profiles

EG RLT-operational/load profiles
(suitability for) ET
RLT-operational/load profiles
UCT-paths
UCT-checklist UCT-decision points
PCT-test depth level 2
DCoT-equivalence classes DCoT-N-wise testing
DCoT-pairwise testing
Suitability PCT-test depth level 1 DCyT (life cycle of the data) CRUD (extra Rs)
DCyT (life cycle of the data) CRUD
UCT-checklist DCyT (integrity rules) modified condition/decision
DCyT (integrity rules) decision coverage
EG coverage
ET
PCT test depth level 3
RLT-operational/load profiles
Performance EG RLT-operational/load profiles
ET
Checklist
Random sample functional tests Functional regression test
All functional tests
Portability Random sample environment Important environment combinations
All environment combinations
combinations ET
EG
RLT-operational/load profiles
Efficiency EG RLT-operational/load profiles
ET

Table 1: Proposed technique for a specific combination of characteristics and selected test intensity

Please refer to Table 2 for the meaning of the abbreviations. See TMap Next [Koomen et al, 2006] for comments on the test design techniques.

Comments on the terms used in Table 1:

Portability - functional tests
When testing portability, a random sample of functional tests, the regression tests or all test cases can be executed in a specific environment with
increasing test intensity.
Environment combinations
Testing portability determines whether the system runs in various environments. Environments may consist of different parts, such as hardware
platform, database system, network, browser and operating system. If the system needs to be able to run on 3 (versions of) operating systems under
4 browsers (or browser versions), you already have 3 x 4 = 12 environment combinations to test.
Penetration test
The penetration test aims to find gaps in the system’s security. It is usually executed by a so-called ‘ethical hacker’.
Usability test
A test in which the users simulate business processes and test the system. Statements about the test object’s user-friendliness are made by observing
the users during the test. A specifically configured and controlled environment, which includes e.g. video cameras and a room with mirrored glazing
for observers, is also called a usability lab.

84 The Magazine for Professional Testers www.testingexperience.com

 Test basis

All types Individual Structured CRUD ma- Structured Operation- Input and Input and Use cases
of test conditions functional trix, data descrip- al profiles, output output
Technique
basis or decision specifi- integrity tion of load specifica- specifica-
tables, cation rules business or profiles tions, busi- tions,
without (pseudo operating ness rules attribute
structure code) processes descrip-
tions
Checklist x x x x
decision table test
x x x
(DTT)
data combination test
x x x x
(DCoT)
error guessing (EG) x x x x x x x x x
exploratory testing (ET) x x x x
elementary comparison
x x
test (ECT)
functional tests x x
Data cycle test (DCyT) x x
Environment combina-
x x
tions
Penetration test x
process cycle test (PCT) x x x x x x
real life test (RLT) x x x
semantic test (SEM) x x x
syntactic test (SYN) x x x
usability test x x
use case test (UCT) x x x x

Table 2: The required test base for a proposed technique

References
• [Koomen et al, 2006]
Koomen, T., Aalst van der, L., Broekman, B., Vroon, M. (2006), TMap® Next for result-driven testing, Tutein Nolthenius, ‘s-Hertogenbosch,
Netherlands, ISBN 90-72194-80-2, www.utn.nl
• [Kuhn, 2000]
Kuhn, D.R., Wallace, D.R., (2000), Failure modes in medical device software: an analysis of 15 years of recall data, National Institute of
Standards and Technology, Gaithersburg, MD 20899 USA, http://csrc.nist.gov/staff/Kuhn/final-rqse.pdf
• TMap® is a registered trademark of Sogeti Nederland B.V.

Biography
Leo van der Aalst has more than 20 years of testing experience and developed
amongst others services for the implementation of test organisations and for test
outsourcing.

He is co-author of TMap Next, designed the EXIN tasks for TMap Next certification and
holds a readership position for quality&testing at the ‘Fontys Hogescholen’.

Besides all this, Leo is a much sought-after teacher of test training, a regular speaker
at national and international conferences, and he is the author of several articles.

www.testingexperience.com The Magazine for Professional Testers 85

YK:
Masthead
11, 0, 18 EDITOR
Díaz & Hilterscheid
Unternehmensberatung GmbH
Kurfürstendamm 179
10707 Berlin, Germany

Phone: +49 (0)30 74 76 28-0

Fax: +49 (0)30 74 76 28-99
E-Mail: [email protected]

Díaz & Hilterscheid is a member of “Verband der Zeitschriftenverleger Berlin-Brandenburg e.V.”

EDITORIAL
José Díaz ISSN 1866-5705
B:
LAYOUT & DESIGN In all publications Díaz & Hilterscheid Unternehmensberatung GmbH makes every ef-
Katrin Schülke fort to respect the copyright of graphics and texts used, to make use of its own graph-
ics and texts and to utilise public domain graphics and texts.

5, 137, 175 WEBSITE

www.testingexperience.com All brands and trademarks mentioned, where applicable, registered by third-parties
are subject without restriction to the provisions of ruling labelling legislation and the
ARTICLES & AUTHORS rights of ownership of the registered owners. The mere mention of a trademark in no
[email protected] way allows the conclusion to be drawn that it is not protected by the rights of third
parties.
PRINT RUN
12.000 printed issues The copyright for published material created by Díaz & Hilterscheid Unternehmensber-
approx. 80.000 downloads atung GmbH remains the author’s property. The duplication or use of such graphics
Kanzlei Hilterscheid or texts in other electronic or printed media is not permitted without the express con-
ADVERTISEMENTS sent of Díaz & Hilterscheid Unternehmensberatung GmbH.
[email protected]
The opinions expressed within the articles and contents herein do not necessarily ex-
SUBSCRIBE press those of the publisher. Only the authors are responsible for the content of their
www.testingexperience.com/subscribe.php articles.

ntone: PRICE
free of charge
No material in this publication may be reproduced in any form without permission.
Reprints of individual articles available.

5u

Berlin, Germany

IT Law
Contract Law

German
English
French
Spanish
© Katrin Schülke

www.kanzlei-hilterscheid.de

86 The Magazine for Professional Testers www.testingexperience.com

 The Test Maturity Model Integrated (TMMi ®)
Measuring our Capability to Deliver!

by Brian Wells
© iStockphoto

In a recent exercise I assessed the test process- has matured greatly over the last 2 decades of approaching elements. These are still evolv-
es of four CMMi Level 5-accredited organiza- and organizations are very actively looking to ing.
tions using one of the then available test ma- develop/implement/improve the capability of
turity models. It may not be a surprise that all their test processes across the test and project Testing as a separate, professional activity
four organizations failed to achieve the same life cycles and across the organization. All this only really emerged in the 1990’s but is devel-
levels of capability for their test processes! against the back drop of the changing nature of oping the “hunger” for standards to evaluate
However, while effective, the cosnsitency and testing and of Software Engineering overall. their process much more quickly.
effectiveness was open to question as, unlike
Software Engineering generally, test does not It is also valid that Test activities (including all In this time, we have seen many developments
have a uniformly accepted standard to measure “forms” of testing; Reviews, Static, Dynamic in testing and software delivery; “fashion”
itself against (and against others!). Test activities) are the prime mechanism to times as well as very practices etc. We have
demonstrate levels of required quality (how- also seen many changes in approaches; often
I asked myself the question “If test disci- ever defined) – you could argue that Test is a driven by wider pressures such as economic
plines have improved so much in recent de- Quality Assurance function, but that is another globalization or customer expectations!.
cades, why do we not have the capability to discussion! Throughout all this, the widely held view is
accurately assess our capabilities (internally that TESTING is now a major, publicly ac-
and against industry norms), realize our Since the 1950’s, Software Engineering has knowledged professional activity (and profes-
weaknesses and use this information to im- been a developing entity. In the beginning sion?) within Software Engineering.
prove?”. there was (arguably) chaos which led to the de-
velopment of standard approaches for specific Given the proportion of IT spending on test ac-
It is because there is not an accepted Standard elements of the life cycle (often driven by indi- tivities, are these Software Engineering evalu-
Reference Model within the public domain viduals/organizations). Throughout the 1970’s ation models (i.e. CMMi) sufficient to assess
that is complete, fit for purpose, adaptable, and 1980’s these started to be published and test activities in adequate depth? How mature
useable, cost effective and able to satisfy the accepted (i.e. the V Model). This standardiza- and how professionally capable are the testing
requirements of the industry for today and to- tion proceeded at pace and the industry identi- processes overall? How can this be demon-
morrow. fied a need to develop an overarching ability strated? Should we be concerned if we cannot
to evaluate the capability (or maturity) of the accurately evaluate in a standard, transparently
In this article I will outline: overall Software Engineering process; not just comparable manner?
the individual elements.
• The Battleground – Looking at the back- Until recently, there was little for test to refer-
drop to the emerging awareness that we In the 1990’s this culminated in the emergence ence to provide robust standards and yardsticks
need to have the ability to assess the ca- of standard reference models to measure ca- to accurately measure our process capability
pabilities of test processes pabilities of overall process maturity (CMM, by. Taking CMMi as an example, there are
• The origins of our species – Looking at CMMi, ITIL, ISO 9000, etc). These have some elements that will evaluate testing, veri-
the origins of Test Reference Models developed into internationally accepted stan- fication and validation, but this is only in the
• The emerging TMMi model – Looking at dards and public accreditations for organiza- context of the Software Engineering process
the TMMi® Test Reference Model – de- tions globally (although the original intention overall. Little attention is paid to evaluating
veloped by test practitioners for all test may have differed i.e. CMM was developed in depth the capability of test and test-related
practitioners! to measure the US Government contractors quality processes and abilities.
but was subsequently used to measure the
The Battleground! process). These defined a framework of pro- The IT testing industry is developing stan-
It is true to say that Testing as a discipline gression linked to capabilities and maturity, dards for the process elements (i.e. ISTQB,
encompassing the developing “standard” ways BS 7925-2 etc). However, there is a real need

www.testingexperience.com The Magazine for Professional Testers 87

to supplement the existing Software Engineer- supplement one of the most common Software We thought not! It is not enough, however,
ing capability models with one specifically Engineering Models used globally. just to publish a model. Too ensure the model
targeted at Test providing us with the ability to is correctly used to assess organizations, the
evaluate the capabilities of a not inconsequen- The TMMi® provides a full framework to be Foundation has also published the TMMi®
tial chunk of Software Engineering to a greater used as a reference model during test process Assessment Method Accreditation Require-
degree than hitherto we have done. improvement. It does not provide an approach ments (TAMAR).
for test process improvement such as the IDE-
As a member of the Testing profession for over AL (Initiating, Diagnosing, Establishing, Act-
2 decades I personally believe the profession ing, Learning) model (Software Engineering
needs a standard, internationally recognized Institute/Carnegie Mellon). (5) Optimisation
Defect Prevention
reference model. It is only in this way that we, Test Process
as a profession, can measure our capabilities. The process areas for each maturity level of
To do this, I also believe that this should be the TMMi® are shown below.
under the umbrella of an independent body
providing an unbiased, independent role to the Note that the TMMi® model does not have
Test profession. Others think the same way! a specific process area dedicated to test tools (4) Management and
and/or test automation. Measurement
The origin of the (TMMi®) Species! Test Measurement
There are a number of existing test “models” Software Quality Evaluation
Advanced Peer Reviews
that have emerged in the past. These include,
amongst the others:

• Boris Beizer’s Test Model (3) Defined

• Gelperin and Hetzel’s Evolution of Test- Test Organisation
ing Model Test Training Program
Test Life Cycle and Integration
• (elements of) Capability Maturity Model
Non-Functional Testing
(CMM) & Capability |Maturity Model – Peer Review
integrated (CMMi)
• Test Process Improvement (TPI) model –
Tim Koomen/Martin Pol This defines the requirements for an assessment
(2) Managed method, using the TMMi® model, to comply
• Test Maturity Model (TMM) – Illinois Test Policy and Strategy
Institute of Technology with an industry standard: ISO 15504. If de-
Test Planning
Test Monitoring and Control
sired, organizations can submit their Assess-
While all are developing the ability to assess Test Design and Execution ment Method and supporting documentation
test process capability and indicating that the Test Evironment for the Foundation to check compliance with
Test profession acknowledges the need they (1) TAMAR. If successful, a formal independent
have limitations of one form or another. This recognition (accreditation) will be issued.
has prevented “market penetration” and uptake
as a developing de facto global standard. Within TMMi® test tools are treated as a sup- Following on from this thought process, or-
porting resource (practices) and are therefore ganizations that have accredited assessment
Test practitioners, have used these models to part of the process area where they provide methods can also submit details of Assessors
try and evaluate weaknesses in test processes support, e.g. applying a test design tool is a and Lead Assessors to be similarly indepen-
based on industry “knowledge” and extensive supporting test practice within the Test Design dently verified by the Foundation as skilled
expertise and experience. However, while any and Execution at TMMi® level 2 process area, and experienced enough to undertake assess-
offering had good elements, none fully satisfied and applying a performance testing tool is a ments (and be publicly accredited).
the need for a comprehensive, understandable supporting test practice within the Non-Func-
and flexible generic model that can adapt. tional Testing at TMMi® level 3 process area. All this provides an independent framework
to the industry at large with the means to be
In 2004, a group of Test practitioners from While it is structured along the lines of CMMi, publicly recognized as using a robust, repeat-
across the world got together to discuss our it draws heavily from many other sources, in- able assessment method to evaluate test pro-
experiences and the needs of the Test profes- corporates proven good principles, best prac- cess capability against a standard reference
sion going forward. This group recognized the tices and extensive experience from industry model (thus allowing easier comparison etc.)
need for a standard test (process) capability practitioners and responds to industry needs. using publicly accredited resources – a useable
reference model and, more importantly, an in- global standard!
dependent body to develop a standard offer- The preamble and a full definition of TMMi®
ing. They agreed this should take the best of Level 2 (Managed) are already published. It is The TMMi Foundation
what is available and be enhanced based on the anticipated that the full definition for TMMi® Set up as a “Not for Profit” company regis-
collective experience of the group and others. Level 3 (Defined) will be available in Septem- tered in Ireland, The TMMi ® Foundation is
ber 2008 with TMMi® Levels 4 & 5 following dedicated to improving test processes and
Out of these discussions, they decided to cre- by Q1-Q2 2009. While this may not be soon practice. The focus of the foundation’s ac-
ate the TMMi Foundation as a non-profit mak- enough for many, the definitions take much tivities is the development of a robust model
ing organization run by test professionals for effort and are reviewed extensively and rigor- for test process as¬sessment/improvement in
the testing profession and beyond. The prime ously to ensure they are complete and fit for IT organizations. We are looking to de¬velop
purpose was to create a standard Test Refer- purpose! international support and sponsorship from in-
ence Model; TMMi®! dustry leaders.
Is it enough to publish the Standard TMMi®
Reference Model without offering to provide Its raison d’être is:
The Emerging TMMi® Model
an independent scheme to ensure that orga-
It was decided to structure the TMMi® model
nizations have the capability to undertake 1. To develop an International TMMi®
in line with CMMi (staged as the initial deliv-
assessments against the TMMi® model to a Standard for Test Process Maturity As-
erable). This was because it would aid under-
robust standard? sessments Model, including the develop-
standing and it would seamlessly support and

88 The Magazine for Professional Testers www.testingexperience.com

 ment of an assessor’s scheme and any qualification requirements (including
provision of accredited training, examinations and management of accredita-
tion).
2. To facilitate international acceptance of the TMMi® Standard via recognized
international bodies and place the standard in the public domain.
3. To promote and support a single, publicly available, international Test Process
Maturity Assessment scheme for the benefit of IT testing professionals and
Senior Management.

The TMMi Foundation has the following specific objectives:

• Identifying and securing ownership of the TMMi® standard and the ongoing
Intellectual Property rights
• A standard TMMi® Model that can be used in isolation or in support of other
process improvement models such as the CMMI.
• Assessment Method Accreditation for TMMi® in accordance with ISO15504
and the process to certify commercial assessment methods against the TMMi®
Assessment Method Application Requirements (see website).
• An independently managed data repository to support TMMi® accreditation
for assessment methods and (lead) assessors and also submitted assessment
results.
Biography
Brian is a senior IT consultant specializ-
I would like to emphasize that the work of the Foundation is intended to be avail-
ing in Test & Validation and is a recog-
able in the public domain, available to all and to provide independent, unbiased
nized thought leader within the industry.
services to all.
One of his initiatives was to recognize
The current board of directors of the TMMi Foundation come from the UK, Ireland,
that the test industry needs an industry-
Holland, Denmark and the USA/India. In addition, there is a rapidly growing list of
wide, fit for purpose, standard Reference
interested parties across the globe; both individuals and organizations.
Model for Test process capability. He was
instrumental in formulating the TMMi
How does the Foundation work? There is a Board of unpaid Directors that man-
Foundation and, as Chair, provides the
age specific areas of our work (technical development, finance, marketing etc.).
impetus to its members in their activi-
Funding is through donations and fees for services offered (method and assessor
ties as well as strategic direction and
accreditation).
management.

Is it worth it? He is fully experienced in the definition

It is very well theorizing and postulating to the industry at large that the TMMi® and implementation of strategic test-
Model is the best thing since sliced bread! ing solutions for clients. This includes
defining, communicating and implement-
However, the need for a standard Reference Model is being increasingly demon- ing corporate testing policy & strategy,
strated by increasing evidence that there is a fast growing need by the industry at processes & procedures, standards,
large to be able to evaluate, to a common reference model, test and test-related test (design) techniques, metrics and
quality for a variety of reasons including: monitoring programs, measurement of
Return on Investment (ROI), implement-
• We need to know our weaknesses – improve efficiency and effectiveness of the ing test tools, training, resources and
test function (with ensuing financial and other savings!) organization and much more. His opera-
• We need to know where we are (often compared to industry sector benchmark- tional experience covers industry sectors
ing) including Banking, Finance, Insurance,
• What do we need to do to gain a certain level of capability/maturity? Retail and Communications. He also ac-
• Evaluate capabilities of 3rd party suppliers/managed services (often compara- tively understands emerging trends and
tively as part of tendering process) requirements in the marketplace.

This can be greatly enhanced by adopting a standard reference model to measure Within his current role at Experimentus
test processes against which it provides confidence in the assessment ratings (irre- Ltd, Brian provides comprehensive,
spective of assessment providers assuming the method/assessment team is accred- flexible solutions for all aspects of Test
ited) and allows easy comparison at all levels. These observations are supported by & Validation to all levels of a corporation
the experiences of assessment practitioners. which is both targeted based on priority/
risk and achievable while remaining flex-
It only confirms us in our view that the TMMi® model has the potential to provide ible in its approach. This is based on a
the standard by which Test will be judged by - we think it is worth it! clear ability to understand the business
requirements/drivers, risks and priori-
Interested? ties.
The Foundation needs the help of the international community, and we are keen to
develop a dialog at all levels. Brian has successfully completed the
ISEB Foundation & Practitioners Certifi-
If you are interested in the products the Foundation is developing, providing con- cates in Software Testing and is a certi-
structive feedback and/or in supporting the work of the Foundation, please contact fied trainer of the Foundation course.
us via our website at www.tmmifoundation.org.

www.testingexperience.com The Magazine for Professional Testers 89

 Training with a View
Kurfürstendamm, Berlin © Katrin Schülke

08.09.08-11.09.08 Certified Tester Advanced Level - TESTMANAGER German Berlin

08.09.08-10.09.08 Certified Tester Foundation Level German Düsseldorf
15.09.08-17.09.08 Certified Tester Foundation Level German Berlin
22.09.08-24.09.08 Certified Professional for Requirements Engineering - German Berlin
Foundation Level
29.09.08-02.10.08 Certified Tester Advanced Level - TESTMANAGER German Berlin
06.10.08-08.10.08 Certified Tester Foundation Level German Berlin
20.10.08-23.10.08 Certified Tester Advanced Level - TESTMANAGER German Berlin
20.10.08-22.10.08 Certified Tester Foundation Level German Frankfurt
27.10.08-29.10.08 Certified Tester Foundation Level German Berlin
03.11.08-06.11.08 Certified Tester Advanced Level - TESTMANAGER German Ratzeburg
03.11.08-05.11.08 Certified Professional for Requirements Engineering - German Berlin
Foundation Level
10.11.08-12.11.08 Certified Tester Foundation Level German Hannover
10.11.08-12.11.08 Certified Tester Foundation Level English Berlin
24.11.08-26.11.08 Certified Tester Foundation Level German Berlin
01.12.08-04.12.08 Certified Tester Advanced Level - TESTMANAGER German Berlin
08.12.08-10.12.08 Certified Tester Foundation Level German München
08.12.08-10.12.08 Certified Professional for Requirements Engineering - German Berlin
Foundation Level
15.12.08-17.12.08 Certified Tester Foundation Level German Berlin

also onsite training worldwide in German, English, Spanish at

http://training.diazhilterscheid.com/ [email protected]

90 The Magazine for Professional Testers www.testingexperience.com

© iStockphoto

Size Does Matter in Process Improvement

by Tanja E.J. Vos, Jorge Sánchez Sánchez and Maximiliano Mannise

Abstract ITI’s mission includes as a main goal to build 1.3. Their projects (and their problems)
Based on experiences in implementing testing bridges between the knowledge and technol- Many times enterprises come to us because
process improvements in different SMEs, we ogy acquired through research and the need of they have problems they think they can solve
found that the most common methodologies simple, practical and robust solutions for the through testing. Problems are mostly the same
are aimed at much larger organizations and IT-related industry. throughout companies:
are therefore difficult to implement in a com- In order to reach this goal, ITI supports en- -- Customers complain because they find
pany profile that has limited resources and low terprises with tailored formation, technology many bugs or because the functionality
maturity in the testing processes. We therefore transfer, standard solutions and consulting provided does not correspond to their
propose a series of simple and concrete actions projects. needs.
that SMEs can do to spend little resources and -- Time delays and budgets discussed with
get fast results while getting prepared for fur- 1.2. The enterprises (SME or microSME) the customer are not reached. Testing is
ther, more formal improvement processes. Many of the enterprises we support are small often sacrificed because of delays in other
Keywords: software, testing, SME, process, software companies from Valencia, Spain. parts of the project.
improvement Those companies generally share the follow- -- Bugs, regardless if they are found through
ing characteristics: testing or reported by customers, are not
1. Introduction -- They sell a personalized software product managed properly, .
This article describes our experiences we for specific end users -- Too many hours are spent on correcting
gained during the technology transfer and con- -- The software has grown from initially defects and/or misunderstandings in re-
sulting services projects run with the aim to just a small, transparent and easy-to- quirements.
help enterprises solve a “testing issue” detect- maintain piece of code programmed by -- The multiple versions of a product are
ed by themselves, or just to help them improve up to two persons to a huge opaque mon- difficult to manage and/or test.
their practices in software testing. ster impossible to maintain by a group of -- There is little or no information about the
In the next few paragraphs we will describe developers. quality of the products developed.
the background and characteristics of those -- Their customers are used to specifying
projects. vague requirements for new develop- 1.4. Software testing and its levels
ments and having their demands imple- For any company (also for SMEs) engaged in
1.1. ITI (Instituto Tecnológico de Informáti- mented. This usually means having mul- software development, whatever the method-
ca) tiple versions of the same software. ology used, testing should be a fundamental
ITI is a non-profit research institute located -- There are no written requirement docu- part both to ensure that products are developed
inside the Universidad Politécnica de Valencia ments, and if there are any, they have properly (verification) as that products meet
(UPV) campus. Its main goal is to apply the little detail, structure or information. the user’s requirements (validation).
knowledge obtained from scientific investiga- -- There are no structured testing processes In any case, proper testing during the develop-
tion and innovation in state-of-the-art technol- nor people specifically dedicated to test- ment cycle satisfies ([HMS+07]):
ogy in small and medium enterprises (SME) ing. In fact, many times there are no spe- -- Testing starts in the early stages of the
related to information technology and commu- cific activities related to testing. software life cycle
nication (IT). -- There is little information and manage- -- You should not only test the final code
The institute was constituted by seven research ment with regard to software bugs. If but also requirements, manuals, docu-
groups, mostly lead by university professors. there is, it usually focuses on bugs de- ments, etc..
The experiences described in this article come tected by customers. -- Testers are involved in requirements re-
from the SQUaC group, which focuses on the trieval.
quality, usability and software certification do-
main.

www.testingexperience.com The Magazine for Professional Testers 91

The typical levels of testing during an applica- 2.2. Process improvement in SMEs, in prac- agencies or large companies; in these cases
tion’s life cycle are: tice the project is simply regarded one more cost
Unit Testing (or components testing). The The main problem with earlier models is that without real improvement expectations. On
source software is often divided into units they were originally designed for larger orga- the other hand, when responsible people of the
more or less isolated, also called programs, nizations and are not easily adaptable to struc- company think the improvement project is a
modules, components or classes. Unit testing tures much smaller than our SMEs. The first real opportunity to optimize their internal pro-
aims at ensuring that each of these units meets reason is that the cost and duration of an evalu- cesses and systems in practice, the chances of
its specification separately. This test is usually ation process are not compatible with available obtaining positive results increase.
done by the same developer who writes the resources in an SME. In addition, their level In the case of Spain studies as [INTECO08]
code. of maturity, especially with regard to soft- highlight the particular importance of SMEs
Integration Testing. Once the units have been ware testing is usually very low; for example, (representing more than 99% of companies in
written and tested the next step is to link them in a quick initial assessment most SMEs we the software industry) and show that method-
to create a complete system: this is called in- worked with only reached the minimum TPI ologies such as SPICE and CMMI are quite
tegration. The purpose of integration testing level of maturity (level A) in the area of office well-known, mainly because of grants from
is to discover defects in the interfaces and environment. government agencies for its adoption. How-
interactions between different components or Several studies (such as [BJ94]) show that a ever, it seems that the companies are more in-
systems. large number of process improvement projects terested in certification for being renamed in
System Testing. Once the components are in- in SMEs based in models such as CMMI found front of third parties than in the improvement
tegrated and functioning together properly the serious problems, and that a significant per- of internal processes: there is knowledge about
next step is to consider the full functionality of centage of these problems (53%) is related to the existence of standards and their use outside
the system as a whole, which is called system the size of the enterprise; Success of a project Spain, but they do not know the details of their
testing. This type of testing is necessary be- to improve processes increases with the num- implementation in real projects and anything
cause the lower levels of testing (unit and inte- ber of persons working on software processes related to its adoption by an enterprise. As a
gration) used a partial view of the system and inside the company. As indicated in [LS99] in result, the level of adoption is very low. The
are not representative of all the actual working this type of projects and size of companies, it barriers that SMEs wield are the high costs of
conditions. System testing, usually conducted is impossible to collect enough data to make a adopting the model, the changes in working
by a team not invlolved in developing the sys- solid statistical analysis, being in such cases methods they involve, the need for changes in
tem, can refer to both functional requirements much more important the human resources, the organization structure, or the difficulty of
(checking that the system performs the desired involving key people (those who possess the evaluating benefits of implementing them.
functions) and non-functional requirements, knowledge) from the beginning. In short, the experiences of SMEs with process
such as performance, load, reliability, usabil- On the other hand ([Mc00]), the stiffness and improvement methodologies are often prob-
ity, etc. formality of these methodologies involve for lematic due to excessive investment in time,
Acceptance Testing. The purpose of accep- SMEs an increase in bureaucracy that be- tools, etc. carried by applying models that are
tance testing is to give confidence to the end comes a liability to the flexibility and innova- designed for larger organizations with more
user that the system will operate as expected. tion that these companies need in a dynamic resources and greater maturity in these proce-
The basis for acceptance testing is the docu- environment on which they virtually have no dures. These difficulties become an abyss that
ment specification requirements, checking control. To decrease the strictness of the meth- separates the SME from process improvement
in practice that the system meets all of them odologies, it is not enough to reduce these dis- (Figure 1).
properly. This testing is often totally inde- advantages.
pendent of other levels of testing and of the A key factor in the success or failure of such 3. Preparing the improvement models
implementation of the system and can be car- projects lies in the motivation that drives the
for SMEs
ried out by the end users of the system (with SMEs to undertake this process of improve-
Given all those problems which arise when
the possible participation of members of the ment ([BWL98]). For some of them it is
applying the improvement models to SMEs,
development team). “something they must do” faced by pressure
there have been various attempts to adapt or
from major customers such as government
create new models specifically oriented to the
2. Process improvement and SMEs

2.1. Process improvement methodologies

There are plenty of methodologies and tech-
niques to improve processes in general and
testing processes in particular (some of them
mentioned in [Swi00] and [Ku06]).
Methodologies such as SPICE [EDM97], de-
veloped by ISO / IEC, are intended to be a
standard for evaluating software processes in
general. Others such as CMMI [CKS03] are
even more comprehensive, as they cover all
development and maintenance of the product
and integrate different areas of knowledge
that had previously been treated separately as
software engineering, systems engineering or
supply.
Other authors have realized that insufficient
attention was paid, from these global models
to software testing, and developed method-
ologies for improvement that are focused on
that aspect, as Test Improvement Model (TIM)
[ESU97] Test Organization Maturity [TOM],
Testing Maturity Model (TMM ) [BSC96], or
Test Process Improvement (TPI) [KP99]. Fig. 1. In the abyss between the SMEs and process improvement methodologies many resources are lost: money
spent, tools left over, etc.

92 The Magazine for Professional Testers www.testingexperience.com

 Tea testers look at
aspects of dried & brewed
tea leaves and the liquor

Likewise, PureTesting goes into various

aspects & nuances of so�ware testing

We build innovati ve, end-to-end solutions,

and manage critical testing processes,
and reduce total cost of producing quality so�ware

• Banking & Finacial Services • Datacom & Telecom

• Pharmaceuticals • Embedded Systems
• eLearning • EAI & ISVs

Test Consulting • Testing Services • Testing Products

India • USA • UK • NZ
www.puretesting.com
+91 (120) 4621010; [email protected]

Global Software Test Consulting The&Magazine

www.testingexperience.com Services company
for Professional Testers 93
characteristics of small businesses. proach: testing on the toilet [TotT]. 4.3. Other aspects
In general, they consist of simplifications and • Give them freedom to carry out testing In addition to these two points, some basic
/ or adjustments to more complex models as they feel convenient. In fact, program- courses on testing can be useful to increase
like CMMI (which is used as a reference in mers are continuously confronted with awareness of the need of it in their business,
[ITMark], especially implanted in Spain) or new technologies and techniques that but too much could be counter-productive if
SPICE / ISO 15504 (which is taken as a basis they have to use to solve many problems; the SME is overwhelmed by the amount of
for Rapid Assessment for Process Improvement they constantly seek, learn and apply things that they are NOT doing.
for Software Development, rap [RTCH00]), or these new technologies. So why wouldn’t We must be prepared because the SMEs are
a combination of them (as does [OWPL]). they do the same with unit testing frame- sometimes reluctant to accept that freedom
Other models, such as Minimal Test Practice works and other tools available? (“let them do what they think they should”)
Framework (MTPF) [KRN05] are specifically • Provide them with necessary resourc- at first. However, it is easy to explain that if
focused on the activities of software testing es. First of all they need time allocated we start from a situation in which virtually
and try to adapt to the specific nature of SMEs: specifically to testing, both to learn and to nothing is being done, every step is already a
scarcity of resources, resistance to internal investigate the best methods and to really big step on the path to process improvement.
change, flexibility, and so on. do unit testing. Usually they will already It is important that SMEs understand that at
In Spain [INTECO08], [ITMark] seems to have the environment (hardware, facili- this early stage, the most important factor is
have special significance although there are ties, etc.) required for testing because of motivation.
also initiatives such as the SoftAragón pro- their normal work. But if the extra effort So, after some time (we propose around 6
gram, which has developed a reduced CMMI to realize unit testing is not planned, you months), the company will have acquired
model to be applied to Aragonese SMEs, or will most likely not apply the necessary some knowledge and experience in testing,
MESOPYME [CCS+02], specially oriented time. and we can start with more ambitious plans: to
towards the implementation phase of the im- implement a more formal methodology as TPI,
provements. To complement their knowledge, you may more specific courses depending on the needs
All these adaptations make improvement need some kind of training designed to explain of SMEs, and so on.
models (and, in particular, the testing improve- in general terms what unit testing is as well as
ment) more easily applicable to SMEs, but an overview of existing techniques and tools, 5. Conclusions
these specific methodologies may not be suf- without going too much into detail, and while Following both advices we present we be-
ficient, especially because of the lack of matu- leaving the programmers the choice on how to lieve, based on our experience, that an SME
rity and awareness that we found in practice in implement this knowledge in their work envi- can shortly be in a much better situation to
enterprises. Is there nothing we can do inside ronment. successfully undertake a more ambitious test-
the SMEs to ease the implementation of a test- ing process improving, such as implementing
ing processes improvement methodology? We 4.2. Staff exclusively dedicated to testing a more formal improvement methodology, in-
think there is something we can do. This would involve hiring at least 1 person that crease testing resources and so on.
is exclusively devoted to high-level testing. In any case, they suppose a relatively small in-
4. Preparing the SMEs for testing pro- His/Her only goal should be finding errors. vestment and any progress in the maturity of
On top of finding these erros he or she he will the company on testing-related aspects that is
cess improvement
also uncover needs in order to carry out his/ achieved by implementing these practices rep-
Before you start implementing any of the im-
her work properly (such as better definition of resents a major breakthrough compared to our
provement models seen earlier (indistinctly
requirements, management defects, risk anal- usual starting position.
the general models or those tailored to small
ysis, planning, etc.) and to help improve the Some of the SMEs we worked with are begin-
organizations) in an SME with the character-
company’s development processes. ning to implement (partially) these simple tips,
istics that we saw at the beginning (no formal
Ideally, the person responsible for that task and the first feedbacks we get are giving quite
testing processes, no dedicated staff, etc.), we
would be someone with knowledge and previ- positive results. After enough time has passed
propose preparing the company with two very
ous experience in software testing. However, we will study and present the experiences and
practical and simple actions:
taking into account the current job market and results achieved in practice.
1. - Unit testing done by programmers
the difficulty in finding and recruiting profes-
2. - Staff exclusively dedicated to testing
sionals with those requirements (especially
References
SMEs) we can accept a person with no previ-
4.1. Unit testing done by programmers [HMS+07] HAMBLING, B.; MORGAN,
ous experience in testing. In general, it would
The sole responsibility of developers in terms P.; SAMAROO, A.; THOMPSON, G.; WIL-
be someone with a technical profile (e.g. a
of testing would be doing some unit testing on LIAMS, P. Software Testing. An ISEB Foun-
computer engineer) and certain skills in other
the code they develop, leaving to their choice dation. The British Computer Society, 2007.
areas: communication, planning tasks, etc. In
what to test and how to do it. Let us trust the [KP99] KOOMEN, T.; POL, M. Test Process
this situation being a self-learner will be very
programmers’ knowledge: they know what Improvement: A practical step-by-step guide
important. In any case, remember that these
they want to program, so let them check if they to structured testing, Addison-Wesley, 1999.
suggestions seek to lead not to a final testing
are doing so properly. [Swi00] SWINKELS, R. A Comparison of
structure within the organization but rather to
The following points are needed in order to TMM and Other Test Process Improvement
a basic preparation prior to undertake a more
succeed with this approach: Models. Frits Philips Institute. Technical Re-
formal improvement process.
• Force programmers to do unit testing. port 12-4-1-FP, 2000.
The person assigned to that task could be
After all, the programmers’ job is not [Ku06] KULKARNI, S. Test Process Maturi-
someone who already belongs to the company,
simply to produce programs but rather to ty Models – Yesterday, Today and Tomorrow.
although in practice it is often very difficult for
produce error-free quality code. And one Proceedings of the 6th Annual International
a person who already has prior responsibilities
way to achieve this is through testing. Software Testing Conference, Delhi, India,
assigned to suddenly stop all his/her previous
• Motivate programmers to perform march 2006.
tasks and focus solely on testing. Hiring a new
testing. Different techniques can be used. [ESU97] ERICSON, T.; SUBOTIC, A.; UR-
person avoids this problem but also offsets
For example, hold regular meetings spe- SING, S. TIM-a test improvement model.
other problems such as increased costs or lack
cifically dedicated to testing (“every Fri- Software Testing, Verification and Reliability
of knowledge that someone from outside the
day morning”); perform informal semi- 7(4): 229-246, 1997
organization may have.
nars to explain the progress made (“look [TOM] Gerrald Consulting web site. TOM™
what we got/found this week”), or use - Testing Organisation Maturity Model
other means to share that information, http://www.gerrardconsulting.com/default.
as Intranets [LS99] or even Google’s ap- asp?page=/tomoverview.html

94 The Magazine for Professional Testers www.testingexperience.com

[BSC96] BURNSTEIN, I; SUWANASSART, T.; CARLSON C.R. The Development of a Testing Maturity Model. Proceedings of the Ninth In-
ternational Quality Week Conference, San Francisco, 21-24 may, 1996.
[CKS03] CHRISSIS, M. B.; KONRAD, M., SHRUM, S. CMMI: Guidelines for Process Integration and Product Improvement. Addison-Wesley,
2003.
[EDM97] EL EMAM, K., DROUIN, J.N.; MELO, W. SPICE: The Theory and Practice of Software Process Improvement and Capability Deter-
mination. Wiley-IEEE Computer Society Press, 1997.
[INTECO08] Estudio sobre la certificación de la calidad como medio para impulsar la industria de desarrollo del software en España. Instituto
Nacional de Tecnologías de la Comunicación, april 2008.
[BJ94] BRODMAN, J. G.; JOHNSON, D. L.., What Small Businesses and Small Organisations say about CMM? Proceedings of the 16th Inter-
national Conference on Software Engineering, Sorrento, Italia, may 1994.
[LS99] LIED, H.J.; STLHANE, T. Experience from process improvement in a SME, EuroSPI’99, 1999.
[Mc00] McADAM, R. Quality models in an SME context: A critical perspective using a grounded approach. International Journal of Quality &
Reliability Management, Vol.17, Issue 3, pp. 305-323, 2000.
[BWL98] BROWN, A.; WIELE, T.V.D.; LOUGHTON, K. Smaller enterprises’ experiences with ISO 9000. International Journal of
Quality & Reliability Management, Vol. 15, Num. 3, pp. 273-285(13), 1998.
[OWPL] OWPL Software Process Improvement for VSE, SME and low maturity enterprises; version 1.2.2. FUNDP-CETIC – Software Quality
Lab – 2000-2006. http://www.cetic.be/article393.html
[ITMark] ESI web site. IT Mark. http://www.esi.es/index.php?op=15.1.2
[KRN05] KARLSTRÖM, D.; RUNESON, P.; NORDÉN, S. A Minimal Test Practice Framework for Emerging Software Organisations. Software
Testing, Verification and Reliability, VL 15, NO 3, PG 145-166, John Wiley & Sons, Ltd, 2005.
[RTCH00] ROUT, T.; TUFFLEY, A; CAHILL, B; HODGEN, B. The Rapid Assessment of Software Process Capability. Proceedings of SPICE
2000, 2000.
[CCS+02] CALVO-MANZANO, J. A.; CUEVAS, G.; SAN FELIU, T.; DE AMESCUA, A.; GARCÍA, L.; PÉREZ, M. Experiences in the Applica-
tion of Software Process Improvement in SMES. Software Quality Control 10, 3 (Nov. 2002), 261-273.
[TotT] Google Testing Blog. Introducing “Testing on the Toilet”. http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html

Biography
Tanja Vos is director of the SQUaC (Software Quality, Usability and Certification) research group of the ITI (Instituto Tecnológico de In-
formática). She has a master in Computer Science and a PhD on verification of distributed systems, both from the University in Utrecht.
Tanja has worked as a researcher and professor at various universities like the University of Utrecht (The Netherlands), Universidad Mayor
de San Simón (Bolivia), University of Cambridge (UK) and the Mediterranean University of Science and Technology in Spain. Dr. Vos has
participated in research projects funded by the European Commission, the Spanish government or by industry. She has more than 10
years of experience in the area of software quality and testing.

Jordi Sánchez is an engineer in Computers Science by the UIB (University of Balearic Islands); currently he works at the SQUaC (Software
Quality, Usability and Certification - http://squac.iti.upv.es) group of the ITI (Instituto Tecnológico de Informática), in Valencia, on projects
related to software quality: testing, usability, accessibility, etc. Previously he has worked for more than 10 years in the financial sector,
studying and applying new technologies to collaborative environments in Intranet and Internet, always from the point of view of user expe-
rience. He also writes on his personal web page, http://jordisan.net

Maximiliano Mannise is member of the SQUaC (Software Quality, Usability and Certification) research group of the ITI (Instituto Tecnológico
de Informática). He had a degree of Computer Engineering from the Universidad Católica del Uruguay. He worked nine years at IBM
Uruguay, over two years as a Test Project Manager. Since 2007, he is with ITI and works as a consultant in process and software testing
improvement and as a researcher in the area of software testing. Maximiliano is interested in practical test process improvement applied
to software organization.

www.testingexperience.com The Magazine for Professional Testers 95

 Personnel evaluation based on tests
by Andrey Konushin
© iStockphoto

It can be asserted that software testing as an information about a candidate. Evaluation of Time spent on expert evaluation equals the
independent technological discipline appeared this data is still an expert task with low levels time spent on evaluation by exam.
in the natural evolution of program engineer- of formalization. Results depend on the ex- The formulas are based on the assumption that
ing knowledge. Russia, with a noticeable lag, perience and professional instincts of the test employees are rated on an hourly basis. The
repeats processes which are taking place in manager. time spent by an examiner on performing di-
West. America, for example, has already real- Suppose that a recently hired employee has no rect tasks and the time spent on references cost
ized the need for “independent testing”. Over knowledge in testing or the knowledge is not the same.
time, western specialists have begun to treat systematic and based purely on previous expe-
quality problems with much more responsibil- rience. This can result in problems in teaching Expert evaluation cost:
ity. After years of evolution, testing had been this employee.
transformed into a rather structured techno- The International Software Testing Qualifica-
logical process that requires not only human tions Board (ISTQB) is now just starting to
resources, but also special software and hard- pick up speed in Russia and is set to become
ware which cannot be assumed as available to one the main trends in the training and certifi-
every company. Apart from that, testing is a cation of software testers. Training programs
specific science that requires specific qualifi- are independent from vendors. More than where - cost of expert hour of personnel evalu-
cations. The experience is that in Russia today 120 professionals in software testing from all ation work, - cost of examiner hour of work,
such qualifications are rarely taken. Even in over the world took part in the development - number of testers being evaluated.
the West there are not many educational in- of syllabi, which gives them a high level of
stitutions that teach testing. In Russia they are reliability. Apart from that, the syllabi are in- Test evaluation cost:
absent. And not every programmer is able to ternational and generally accepted: they exist
become a good tester [1]. in more than 20 countries and several tens of
In the Russian provinces the situation with thousands of specialists are certified [2]. The
personnel selection is much more difficult than certification scheme provides qualification
in the larger cities. This is principally the result based on exams.
of low standards, the evolution of the testing There is a good reason to evaluate personnel
branch and a lack of teaching organizations. periodically. Moreover, evaluations need to
In spite of this, there are periodical seminars be processed for both the new and existing where - cost of exam development by expert,
and trainings for software testers, all of which testers. World trends show that evaluation is - cost of examiner hour of work, - number of
are in part non-standard because the trainers assumed to be in the form of exams [3]. A testers being evaluated.
lead the seminars using their own subjective properly constructed exam permits an objec- Suppose that a testing department consists of
program. Thus when selecting the personnel in tive evaluation of qualification levels shown 10 testers. Evaluation is held twice a year. The
these provincial areas department leaders are as a quantitative ratio. It is a laborious task to cost of examiner work is 14 €. The cost of an
faced with the question of how to evaluate an conduct such exams; maybe even more labori- expert is 22 € per hour. Evaluation of one em-
employee when he/she has no experience or ous than an expert assessment of qualification ployee takes 1 hour in both variants consid-
knowledge in testing. Not only is the current levels. But exams have an advantage: they can ered. The expert is able to develop an exam
qualification level of the candidate important be applied more than once and to multiple em- in 5 hours.
here, but also his/her professional status. Hav- ployees concurrently. This can save precious
ing experience means you can teach employ- manager’s time, as shown by the following
ees based on your own strengths. A personal calculations.
evaluation which can be assessed in all testing Assume that expert evaluation and exam eval-
areas can help here. However, such evalua- uation give the same result – objective knowl-
tions at this stage only provide a way to obtain edge of the employee’s qualification level.

96 The Magazine for Professional Testers www.testingexperience.com

Of course, results processing should be automated as much as possible. The time
spent on automation is not taken into account in the formulas shown above. It is
possible to assume that automation consists of the elementary report on the number
of correct and incorrect answers. If we assume that one exam can be used twice
(e.g. twice a year or two exams alternating over two years) and evaluation is needed
twice a year, then we have the following results per year.

Biography
25 years old. Specialist in software
Thus, with some assumptions, calculations show that the most effective method of testing. ISTQB Certified Tester, Advanced
evaluation is exams, other conditions being equal. In this article a software testing Level.
department is considered. However, it is easy to see that these calculations are ap-
plicable to different key industries that face the problem of evaluation of person- Master of techniques and technologies
nel. in informatics and computer engineering,
Vladimir State University, Russia.
1. “CIO” magazine №10, 19.12.2002, http://www.cio-world.ru/
offline/2002/9/22721/ Since 2004: Test Manager of Inreco
2. International Software Quality Specialists Certification Scheme, A. Konushin, LAN, a software development com-
V. Vershinin, Contemporary problems of economics and new research tech- pany (http://inrecolan.com). His main
nologies: interuniversity collection of proceedings. Part 1 / VZFI Vladimir research interests include information
Branch. – Vladimir, 2006. ISBN 5-93350-126-3 management and system analysis both
3. Personnel evaluation methods, http://www.management.com.ua/hrm/hrm029. in software testing and related fields.
html
Since 2006: Lecturer of Information Sys-
tems & Information Management chair
of Vladimir State University. In 2008
Andrey developed and implemented the
“Software testing” course for the Univer-
sity based on ISTQB Syllabus.

Andrey is one of the founders of the

Russian Software Testing Qualifications
Board. Since 2006 he is the President of
RSTQB.

www.testingexperience.com The Magazine for Professional Testers 97

Yaron’s Forecourt
Practicing Test Techniques -
A Discussion by Yaron Tsubery

Dear colleagues and readers,

Welcome back to our joint forecourt, I hope that you’re already equipped with a
glass of chilled wine or a pint of cold beer… or what ever you choose in order to
relax, willing to read, and to share your thoughts on today’s subject…
Test Techniques. Does just hearing the expression make you feel uncomfortable?
Or are you still relaxed? Do you feel secure discussing Test Techniques? Or do you
dismiss it as “not for us”? When I find myself speaking about Test Techniques with
my employees (especially newbies – no offence…) or even sometimes with my
managers (who are R&D managers or Business managers), they look at me as if I
just fell from the sky… It’s clear which camp they are in – they are uncomfortable
talking about them.
Why do you think this is so? What is it about Test Techniques that makes it so “ter-
rifying”? Well, I haven’t done a thorough research on this subject from the psycho-
logical angle, but let us look at some practical aspects related to Test Techniques
that I’ve been engaged with. Biography
Yes, you, who don’t feel comfortable with Test Techniques or are “terrified” by Yaron Tsubery has been a Director of
them, you should be. It is not so easy to understand them, it is not easy to under-
QA & Testing Manager at Comverse for
stand how to implement them in a real project and it is not easy to estimate how
much time it will take. It takes time for practitioners to get use to the fact that for seven years. Yaron is in charge of a large
each project they must spend time, effort, and stimulate discussions on Test Tech- group that includes Testing Team Lead-
niques, while you – the “terrified” people – probably don’t even consider them in ers and Test Engineers. The group deals
your plans! with Functional, Non-functional, Load &
Performance tests, some of which are
Well, if you just think that it’s enough to go to training, learn the theory from the done off-shore. The group has much
books, and get to know some practical aspects from your professional and skilled experience in managing and controlling
tutors who will give you only a clue about it, you’re partly right. They will give you
acceptance tests. Yaron is also known as
the theory, but you’ll not be in the real world.
The real world is where you have to make this effort part of your plan, and then a manager who has built testing groups
justify this plan to a bunch of unconvinced peers and managers. The first time you in his past and also acted as a consul-
do that, you have no proof that this effort will buy you anything. You may need to tant to testing managers while they were
get used to see lots of gloomy faces of your peers and manager/s who are convinced building their testing groups.
you are just spending their time and money. Yaron has been working in software
Yes, the time will come and you’ll be able to prove the message of the cost ef- since 1990, and has more then 15 years
fectiveness of investing time on Test Techniques, you’ll be able to show the gaps in testing field as a Test Engineer, Testing
between your test coverage before and after, which tests caught the most important TL, and Testing Manager. His original
bugs in the shortest time. Yes, it will be hard in the beginning, there will be doubts profession was Systems Analyst.
all around – even you will have some (I had once, in a very stressed project) – don’t Yaron worked in Product Management,
give up, keep on with your mission! Project Management and Development
So, what does it take to use test techniques? Firstly, you have to have the knowledge for at least 3 years before becoming a
and skills to implement these techniques. This is no small feat. It’s not enough to Director of QA & Testing.
learn the techniques; you have to have implemented them and understood their ben- Yaron is a member of ISTQB (Inter-
efits. Secondly, you need the understanding that we must invest time in the analysis national Testing Qualifications Board
phase, to come up with the best test coverage for the project. Thirdly, you need to – www.istqb.org) and is the President
be wise in how to communicate it with your peers! and founder of the ITCB (Israeli Testing
Well, at the end of the day, dealing with Test Techniques is also a matter of maturity Certification Board – www.itcb.org.il). He
– in order to reach the high quality that we require from the products our company is a member of IQAMF (Israeli QA Manag-
produces, we need to have a mature testing organization too! Speaking the same ers Forum) and in SIGiST Israel.
language – “engineering” – with our peers, and showing test coverage by using the Yaron has been invited as a speaker to
best test techniques for each area/domain, will enhance the maturity of the product, some important international confer-
and achieve respect for the test organization, and recognition of its maturity. ences to lecture on subjects related to
Summarizing this short column, I would like to open it for discussion by asking testing.
few questions which came up while writing this paper and before…: Do we use test Yaron has lots of experience in banking
techniques in our organizations? Do we have the knowledge? Can we estimate and business processes. For the last 7 years
later measure the profit of using test techniques? What is the impact on our daily job he has implemented best practices in a
and on the test organization maturity? Can you share and advise how to leverage the field where he is in charge of producing
use of test techniques? Are there experts who can support using test techniques, or complex systems for telecommunication
even oppose (with proven reasons)? companies such as Vodafone, T-Mobile,
Well, that’s it for today. I’ve had my couple glasses of red dry wine; I hope that you NTT DoCoMo, Verizon Wireless etc’;
enjoyed the time as well. I’m sure now that even if you’re relaxed, you still have Main systems in Mobile technology:
enough strength to share your knowledge and experience; please don’t think twice, i-mode, Mobile GW and Billing Proxy, in
your inputs are important for others in the forecourt, and I’ll be more thankful if SI projects: Content Data Management
you’ll send them to the following account, prepared especially for our discussions: solutions, Video Solutions, IPTV and IMS.
[email protected]

Yours truly,
Yaron

98 The Magazine for Professional Testers www.testingexperience.com

www.testingexperience.com The Magazine for Professional Testers 99
 © Pitopia / Klaus-Peter Adler, 2007
What about
your intellect?
CaseMaker – the tool for test case design and test data generation
If you are ISTQB Certified Tester, you can get a free version of the new client software for 12 months.
Please register at www.casemaker.eu/istqbct

ISTQB
Certified Tester?
Get a FREE
Version

www.casemaker.eu
100 The Magazine for Professional Testers www.testingexperience.com