Software Testing Basics

Unit 1

Software Testing

Software Testing is a method to check whether the actual software product matches expected
requirements and to ensure that software product is Defect free. It involves execution of
software/system components using manual or automated tools to evaluate one or more properties
of interest. The purpose of software testing is to identify errors, gaps or missing requirements in
contrast to actual requirements.

Some prefer saying Software testing definition as a White Box and Black Box Testing. In simple
terms, Software Testing means the Verification of Application Under Test (AUT). This Software
Testing course introduces testing software to the audience and justifies the importance of
software testing.

Why Software Testing is Important?

Software Testing is Important because if there are any bugs or errors in the software, it can be
identified early and can be solved before delivery of the software product. Properly tested
software product ensures reliability, security and high performance which further results in time
saving, cost effectiveness and customer satisfaction.

What is the need of Testing?

Testing is important because software bugs could be expensive or even dangerous. Software
bugs can potentially cause monetary and human loss, and history is full of such examples.

 In April 2015, Bloomberg terminal in London crashed due to software glitch affected
more than 300,000 traders on financial markets. It forced the government to postpone a
3bn pound debt sale.
 Nissan cars recalled over 1 million cars from the market due to software failure in the
airbag sensory detectors. There has been reported two accident due to this software
failure.
  Starbucks was forced to close about 60 percent of stores in the U.S and Canada due to
software failure in its POS system. At one point, the store served coffee for free as they
were unable to process the transaction.
 Some of Amazon’s third-party retailers saw their product price is reduced to 1p due to a
software glitch. They were left with heavy losses.
 Vulnerability in Windows 10. This bug enables users to escape from security sandboxes
through a flaw in the win32k system.
 In 2015 fighter plane F-35 fell victim to a software bug, making it unable to detect targets
correctly.
 China Airlines Airbus A300 crashed due to a software bug on April 26, 1994, killing 264
innocents live
 In 1985, Canada’s Therac-25 radiation therapy machine malfunctioned due to software
bug and delivered lethal radiation doses to patients, leaving 3 people dead and critically
injuring 3 others.
 In April of 1999, a software bug caused the failure of a $1.2 billion military satellite
launch, the costliest accident in history
 In May of 1996, a software bug caused the bank accounts of 823 customers of a major
U.S. bank to be credited with 920 million US dollars.

What are the benefits of Software Testing?

Here are the benefits of using software testing:

 Cost-Effective: It is one of the important advantages of software testing. Testing any IT

project on time helps you to save your money for the long term. In case if the bugs caught
in the earlier stage of software testing, it costs less to fix.
 Security: It is the most vulnerable and sensitive benefit of software testing. People are
looking for trusted pr
 .ducts. It helps in removing risks and problems earlier.-
 Product quality: It is an essential requirement of any software product. Testing ensures a
quality product is delivered to customers.
 Customer Satisfaction: The main aim of any product is to give satisfaction to their
customers. UI/UX Testing ensures the best user experience.
Testing in Software Engineering

As per ANSI/IEEE 1059, Testing in Software Engineering is a process of evaluating a software

product to find whether the current software product meets the required conditions or not. The
testing process involves evaluating the features of the software product for requirements in terms
of any missing requirements, bugs or errors, security, reliability and performance.

Types of Software Testing

Here are the software testing types:

Typically Testing is classified into three categories.

 Functional Testing
 Non-Functional Testing or Performance Testing
 Maintenance (Regression and Maintenance)

Types of Software Testing in Software Engineering

Testing Category Types of Testing

 Unit Testing
 Integration Testing
Functional Testing
 Smoke
 UAT ( User Acceptance Testing)
  So on

 Performance
 Endurance
 Load
 Volume
Non-Functional Testing
 Scalability
 Usability
 So on

 Regression
Maintenance  Maintenance

This is not the complete list as there are more than 150 types of testing types and still adding.
Also, note that not all testing types are applicable to all projects but depend on the nature &
scope of the project. To explore a variety of testing tools and find the ones that suit your project
requirements.

Testing Strategies in Software Engineering

Here are important strategies in software engineering:

Unit Testing: This software testing basic approach is followed by the programmer to test the
unit of the program. It helps developers to know whether the individual unit of the code is
working properly or not.

Integration testing: It focuses on the construction and design of the software. You need to see
that the integrated units are working without errors or not.

System testing: In this method, your software is compiled as a whole and then tested as a whole.
This testing strategy checks the functionality, security, portability, amongst others.

Program Testing
Program Testing in software testing is a method of executing an actual software program with
the aim of testing program behavior and finding errors. The software program is executed with
test case data to analyse the program behavior or response to the test data. A good program
testing is one which has high chances of finding bugs.

7 Principles of Software Testing

1. Testing shows presence of defects

2. Exhaustive testing is not possible
3. Early testing
4. Defect clustering
5. Pesticide paradox
6. Testing is context dependent
7. Absence of errors fallacy

Background

It is important that you achieve optimum test results while conducting software testing without
deviating from the goal. But how you determine that you are following the right strategy for
testing? For that, you need to stick to some basic testing principles. Here are the common seven
testing principles that are widely practiced in the software industry.

To understand this, consider a scenario where you are moving a file from folder A to Folder B.

Think of all the possible ways you can test this.

Apart from the usual scenarios, you can also test the following conditions

 Trying to move the file when it is Open

 You do not have the security rights to paste the file in Folder B
 Folder B is on a shared drive and storage capacity is full.
 Folder B already has a file with the same name, in fact, the list is endless
  Or suppose you have 15 input fields to test, each having 5 possible values, the number of
combinations to be tested would be 5^15

If you were to test the entire possible combinations project EXECUTION TIME & COSTS

Here are the 7 Principles:

1) Exhaustive testing is not possible

Yes! Exhaustive testing is not possible. Instead, we need the optimal amount of testing based on
the risk assessment of the application.

And the million dollar question is, how do you determine this risk?

To answer this let’s do an exercise

In your opinion, Which operation is most likely to cause your Operating system to fail?

I am sure most of you would have guessed, Opening 10 different application all at the same time.

So if you were testing this Operating system, you would realize that defects are likely to be
found in multi-tasking activity and need to be tested thoroughly which brings us to our next
principle Defect Clustering

2) Defect Clustering

Defect Clustering which states that a small number of modules contain most of the defects
detected. This is the application of the Pareto Principle to software testing: approximately 80%
of the problems are found in 20% of the modules.

By experience, you can identify such risky modules. But this approach has its own problems

If the same tests are repeated over and over again, eventually the same test cases will no longer
find new bugs.

3) Pesticide Paradox
Repetitive use of the same pesticide mix to eradicate insects during farming will over time lead
to the insects developing resistance to the pesticide Thereby ineffective of pesticides on insects.
The same applies to software testing. If the same set of repetitive tests are conducted, the method
will be useless for discovering new defects.

To overcome this, the test cases need to be regularly reviewed & revised, adding new & different
test cases to help find more defects.

Testers cannot simply depend on existing test techniques. He must look out continually to
improve the existing methods to make testing more effective. But even after all this sweat & hard
work in testing, you can never claim your product is bug-free. To drive home this point, let’s see
this video of the public launch of Windows 98

You think a company like MICROSOFT would not have tested their OS thoroughly & would
risk their reputation just to see their OS crashing during its public launch!

4) Testing shows a presence of defects

Hence, testing principle states that – Testing talks about the presence of defects and don’t talk
about the absence of defects. i.e. Software Testing reduces the probability of undiscovered
defects remaining in the software but even if no defects are found, it is not a proof of correctness.

But what if, you work extra hard, taking all precautions & make your software product 99% bug-
free. And the software does not meet the needs & requirements of the clients.

This leads us to our next principle, which states that- Absence of Error

5) Absence of Error – fallacy

It is possible that software which is 99% bug-free is still unusable. This can be the case if the
system is tested thoroughly for the wrong requirement. Software testing is not mere finding
defects, but also to check that software addresses the business needs. The absence of Error is a
Fallacy i.e. Finding and fixing defects does not help if the system build is unusable and does not
fulfill the user’s needs & requirements.

To solve this problem, the next principle of testing states that Early Testing
6) Early Testing

Early Testing – Testing should start as early as possible in the Software Development Life Cycle.
So that any defects in the requirements or design phase are captured in early stages. It is much
cheaper to fix a Defect in the early stages of testing. But how early one should start testing? It is
recommended that you start finding the bug the moment the requirements are defined. More on
this principle in a later training tutorial.

7) Testing is context dependent

Testing is context dependent which basically means that the way you test an e-commerce site
will be different from the way you test a commercial off the shelf application. All the developed
software’s are not identical. You might use a different approach, methodologies, techniques, and
types of testing depending upon the application type. For instance testing, any POS system at a
retail store will be different than testing an ATM machine.

What is Software Testing Life Cycle (STLC)?

Software Testing Life Cycle (STLC) is a sequence of specific activities conducted during the
testing process to ensure software quality goals are met. STLC involves both verification and
validation activities. Contrary to popular belief, Software Testing is not just a single/isolate
activity, i.e. testing. It consists of a series of activities carried out methodologically to help
certify your software product. STLC stands for Software Testing Life Cycle.

STLC Phases

There are following six major phases in every Software Testing Life Cycle Model (STLC
Model):
STLC Model Phases

1. Requirement Analysis
2. Test Planning
3. Test case development
4. Test Environment setup
5. Test Execution
6. Test Cycle closure

Each of these stages has a definite Entry and Exit criteria, Activities & Deliverables associated
with it.

What is Entry and Exit Criteria in STLC?

 Entry Criteria: Entry Criteria gives the prerequisite items that must be completed before
testing can begin.
 Exit Criteria: Exit Criteria defines the items that must be completed before testing can
be concluded

You have Entry and Exit Criteria for all levels in the Software Testing Life Cycle (STLC)

In an Ideal world, you will not enter the next stage until the exit criteria for the previous stage is
met. But practically this is not always possible. So for this tutorial, we will focus on activities
and deliverables for the different stages in STLC life cycle. Let’s look into them in detail.

Requirement Phase Testing

Requirement Phase Testing also known as Requirement Analysis in which test team studies the
requirements from a testing point of view to identify testable requirements and the QA team may
interact with various stakeholders to understand requirements in detail. Requirements could be
either functional or non-functional. Automation feasibility for the testing project is also done in
this stage.
Activities in Requirement Phase Testing
  Identify types of tests to be performed.
 Gather details about testing priorities and focus.
 Identify test environment details where testing is supposed to be carried out.
 Automation feasibility analysis (if required).

Test Planning in STLC

Test Planning in STLC is a phase in which a Senior QA manager determines the test plan
strategy along with efforts and cost estimates for the project. Moreover, the resources, test
environment, test limitations and the testing schedule are also determined. The Test Plan gets
prepared and finalized in the same phase.
Test Planning Activities

 Preparation of test plan/strategy document for various types of testing

 Test tool selection
 Test effort estimation
 Resource planning and determining roles and responsibilities.
 Training requirement

Deliverables of Test Planning

 Test plan/strategy document.

 Effort estimation document.

Test Case Development Phase

The Test Case Development Phase involves the creation, verification and rework of test cases
& test scripts after the test plan is ready. Initially, the Test data is identified then created and
reviewed and then reworked based on the preconditions. Then the QA team starts the
development process of test cases for individual units.
Test Case Development Activities

 Create test cases, automation scripts (if applicable)

 Review and baseline test cases and scripts
 Create test data (If Test Environment is available)
Deliverables of Test Case Development

 Test cases/scripts
 Test data

Test Environment Setup

Test Environment Setup decides the software and hardware conditions under which a work
product is tested. It is one of the critical aspects of the testing process and can be done in parallel
with the Test Case Development Phase. Test team may not be involved in this activity if the
development team provides the test environment. The test team is required to do a readiness
check (smoke testing) of the given environment.

Test Environment Setup Activities

 Understand the required architecture, environment set-up and prepare hardware and
software requirement list for the Test Environment.
 Setup test Environment and test data
 Perform smoke test on the build

Deliverables of Test Environment Setup

 Environment ready with test data set up

 Smoke Test Results.

Test Execution Phase

Test Execution Phase is carried out by the testers in which testing of the software build is done
based on test plans and test cases prepared. The process consists of test script execution, test
script maintenance and bug reporting. If bugs are reported then it is reverted back to
development team for correction and retesting will be performed.
Test Execution Activities

 Execute tests as per plan

 Document test results, and log defects for failed cases
  Map defects to test cases in RTM
 Retest the Defect fixes
 Track the defects to closure

Test Cycle Closure

Test Cycle Closure phase is completion of test execution which involves several activities like
test completion reporting, collection of test completion matrices and test results. Testing team
members meet, discuss and analyze testing artifacts to identify strategies that have to be
implemented in future, taking lessons from current test cycle. The idea is to remove process
bottlenecks for future test cycles.
Test Cycle Closure Activities

 Evaluate cycle completion criteria based on Time, Test coverage, Cost,Software, Critical
Business Objectives, Quality
 Prepare test metrics based on the above parameters.
 Document the learning out of the project
 Prepare Test closure report
 Qualitative and quantitative reporting of quality of the work product to the customer.
 Test result analysis to find out the defect distribution by type and severity.

V Model

V Model is a highly disciplined SDLC model which has a testing phase parallel to each
development phase. The V model is an extension of the waterfall model wherein software
development and testing is executed in a sequential way. It is known as the Validation or
Verification Model.
Key Software Engineering Terms:
SDLC: SDLC is Software Development Life Cycle. It is the sequence of activities carried out by
Developers to design and develop high-quality software.

STLC: STLC is Software Testing Life Cycle. It consists of a series of activities carried out by
Testers methodologically to test your software product.

Waterfall Model: Waterfall model is a sequential model divided into different phases of
software development activity. Each stage is designed for performing the specific activity.
Testing phase in waterfall model starts only after implementation of the system is done.

Example To Understand the V Model

Suppose, you are assigned a task, to develop a custom software for a client. Now, irrespective of
your technical background, try and make an educated guess about the sequence of steps you will
follow, to

Phases of Software
Activities performed in each stage
Development
 Gather as much information as possible about the details &
Requirement specifications of the desired software from the client. This is
Gathering stage nothing but the Requirements gathering stage.
  Plan the programming language like Java, PHP, .net; database like
Oracle, MySQL, etc. Which would be suited for the project, also
Design Stage
some high-level functions & architecture.

 After the design stage, it is build stage, that is nothing but actually
Build Stage code the software

 Next, you test the software to verify that it is built as per the
Test Stage specifications are given by the client.

 Deploy the application in the respective environment

Deployment stage

 Once your system is ready to use, you may require to change the
Maintenance stage code later on as per customer request

All these levels constitute the waterfall method of the software development lifecycle.

Problem with the Waterfall Model

As you may observe, that testing in the model starts only after implementation is done.

But if you are working in the large project, where the systems are complex, it’s easy to miss out
the key details in the requirements phase itself. In such cases, an entirely wrong product will be
delivered to the client and you might have to start afresh with the project OR if you manage to
note the requirements correctly but make serious mistakes in design and architecture of your
software you will have to redesign the entire software to correct the error.

Assessments of thousands of projects have shown that defects introduced during requirements
& design make up close to half of the total number of defects.
Also, the costs of fixing a defect increase across the development lifecycle. The earlier in life
cycle a defect is detected, the cheaper it is to fix it. As they say, ―A stitch in time saves nine.‖

Solution: The V Model

To address this concern, the V model of testing was developed where for every phase, in the
Development life cycle there is a corresponding Testing phase

 The left side of the model is Software Development Life Cycle – SDLC
 The right side of the model is Software Test Life Cycle – STLC
 The entire figure looks like a V, hence the name V – model

Apart from the V model, there are iterative development models, where development is carried
in phases, with each phase adding a functionality to the software. Each phase comprises its
independent set of development and testing activities.

Good examples of Development lifecycles following iterative method are Rapid Application
Development, Agile Development.
Code Review vs. Code Walkthrough vs. Code Inspection

Walkthrough : It is an informal meeting for evaluation or informational purposes. Little or no

preparation is usually required.

Inspection: An inspection is more formalized than a 'walkthrough', typically with 3-8 people
including a moderator, reader and a recorder to take notes. The subject of the inspection is
typically such as a requirement spec or a test plan, and the purpose is to find problems and see
what's missing, not to fix anything. The result of the inspection meeting should be a written
report.

Peer reviews: peer reviews are composed of walkthroughs and software inspections and integral
to software product engineering activities. Peer reviews include structural review process,
standard of excellence product checklist, defined role of participants and the forms and report.

Based on the extent of the formality review are classified in to 3 types.

1. Informal review (Peer Review) - One on one with peers, no agenda or

preparation time Informal reviews do not have the rigor of a formal review. The
reviews may not be planned, or metrics may not be captured to measure review
effectiveness. They are typically used to confirm understanding, test ideas,
brainstorm, etc.

-Team includes: 1 programmer + reviewer (1-2 programmers OR tester)

-Objective: is to review the code and look for problems
-Process/Phases: make sure that 4 elements (looking for problems, following of
rules, preparation for review and report) are in place

2. Semiformal review (Walk Through) - Author walks the participants through

work product, defects are pointed out by participants, solutions may be
discussed for defects found. This could be a one person review also. Semiformal
/ Informal reviews shall be considered for agile based projects.
-Team includes-2-7 members, here Author=Presenter=programmer, reviewers (must
include atleast 1 programmer as reviewer)

3. Formal review (Inspection) - Meeting is planned in advance, facilitated by

a moderator, follows a proper process, participants come prepared for the
meeting. Reviewers are selected based on skill and knowledge.

-Team includes: moderator, author, presenter=reader, inspectors=reviewers,

recorder

Why are code reviews done?

Code review is especially beneficial for identifying security flaws. Specialist application
packages can aid this approach. Automated code review simplifies the process of evaluating
source code for vulnerabilities such as buffer overflows, race conditions, memory leaks, size
violations, and duplicate statements. Another common application of code review is testing.

Code reviews should be integrated into an organization’s existing process.

For Example — A code review should be started if a team uses task branching processes after all
the code has been produced and all the automated tests have been run and passed before the code
is merged upstream.

How does Code Inspection work?

Participants in inspection activities follow a predetermined process and have clear duties to play.

An inspection team comprises three to eight people who serve as moderators, authors, recorders,
readers, and inspectors. A designer, for example, can serve as an inspector during code
inspections, whereas a quality assurance representative can serve as a standard enforcer.

 Planning — The moderator plans the inspection. The inspection team is given relevant
materials, and after that, the team schedules the inspection meeting and works together.
 Overview meeting — The author gives the inspection team members a brief summary of
the project and its code if they are unfamiliar with it.
 Preparation — After that, each inspection team conducts a code inspection using a few
inspection checklists.
 Inspection meeting — During this meeting, the reader goes through the work output,
section by section, and inspectors point out any flaws.
 Rework — The author modifies the work output following the inspection meeting’s
action plans.
 Follow-up — Hold a meeting with the team after the code inspection to discuss the
reviewed code.

How does Code Walkthrough work?

 Code walkthroughs are a type of peer review. The code’s author often leads a meeting
attended by other team members.
 A few team members usually are provided a copy of the code a few days before the
meeting. The author offers the document under review while the attendees manually run
test cases against the code. Members of the team may also ask the author any pertinent
questions.
 All attendees share and discuss their individual results with the author.