Methods of Software Testing

There are two basic methods of performing software testing:

1. Manual testing
2. Automated testing
Manual Software Testing
As the name would imply, manual software testing is the process of an individual or individuals
manually testing software. This can take the form of navigating user interfaces, submitting
information, or even trying to hack the software or underlying database. As one might presume,
manual software testing is labor-intensive and slow. There are some things for which manual
software testing is appropriate, including:
 User interface or usability testing
 Exploratory/ad hoc testing (where testers do not follow a 'script', but rather testers 'explore'
the application and use their instincts to find bugs)
 Testing areas of the application which experience a lot of change.
 User acceptance testing (often, this can also be automated)
The time commitment involved with manual software testing is one of its most significant
drawbacks. The time needed to fully test the system will typically range from weeks to months.
Variability of results depending on who is performing the tests can also be a problem. For these
reasons, many companies look to automation as a means of accelerating the software testing
process while minimizing the variability of results.
Automated Software Testing
Automated software testing is the process of creating test scripts that can then be run
automatically, repetitively, and through many iterations. Done properly, automated software
testing can help to minimize the variability of results, speed up the testing process, increase test
coverage (the number of different things tested), and ultimately provide greater confidence in the
quality of the software being tested.
There are, however, some things for which automated software testing is not appropriate. These
include:
 End user usability testing is not typically a good candidate for automated testing.
 Tests which will not be run more than a couple of times are typically not a good candidate for
automated tasting, since the payoff of in test automation comes after many test executions.
 Tests for areas of the application which experience a lot of change are also not a good
candidate for automation since this can lead to substantial maintenance of test automation
scripts. Such areas of the application may be more effectively tested manually.
It is important to note that test automation is software, and just like the software you are building
for internal or external customers, it must be well-architected. A good test automation
architecture, such as a keyword-driven testing framework, will reduce the overall cost of
ownership of your test automation by minimizing maintenance expense and increasing the
number of automated tests, allowing your organization to run more tests (and achieve higher
quality) for the same investment of time and money.

[1]
Testing is a process used to help identify the correctness, completeness and quality of developed
computer software. With that in mind, testing can never completely establish the correctness of
computer software.
There are many approaches to software testing, but effective testing of complex products is
essentially a process of investigation, not merely a matter of creating and following rote
procedure. One definition of testing is "the process of questioning a product in order to evaluate
it", where the "questions" are things the tester tries to do with the product, and the product
answers with its behavior in reaction to the probing of the tester. Although most of the
intellectual processes of testing are nearly identical to that of review or inspection, the word
testing is connoted to mean the dynamic analysis of the product—putting the product through its
paces.

The quality of the application can and normally does vary widely from system to system but
some of the common quality attributes include reliability, stability, portability, maintainability
and usability. Refer to the ISO standard ISO 9126 for a more complete list of attributes and
criteria.

Testing helps is Verifying and Validating if the Software is working as it is intended to be

working. Thins involves using Static and Dynamic methodologies to Test the application.

Because of the fallibility of its human designers and its own abstract, complex nature, software
development must be accompanied by quality assurance activities. It is not unusual for
developers to spend 40% of the total project time on testing. For life-critical software (e.g. flight
control, reactor monitoring), testing can cost 3 to 5 times as much as all other activities
combined. The destructive nature of testing requires that the developer discard preconceived
notions of the correctness of his/her developed software.

Software Testing Fundamentals

Testing objectives include

1. Testing is a process of executing a program with the intent of finding an error.

2. A good test case is one that has a high probability of finding an as yet undiscovered error.
3. A successful test is one that uncovers an as yet undiscovered error.

Testing should systematically uncover different classes of errors in a minimum amount of time
and with a minimum amount of effort. A secondary benefit of testing is that it demonstrates that
the software appears to be working as stated in the specifications. The data collected through
testing can also provide an indication of the software's reliability and quality. But, testing cannot
show the absence of defect -- it can only show that software defects are present

[2]
Testing Start Process

Testing is sometimes incorrectly thought as an after-the-fact activity; performed after

programming is done for a product. Instead, testing should be performed at every development
stage of the product. Test data sets must be derived and their correctness and consistency should
be monitored throughout the development process.

If we divide the lifecycle of software development into “Requirements Analysis”, “Design”,

“Programming/Construction” and “Operation and Maintenance”, then testing should accompany
each of the above phases. If testing is isolated as a single phase late in the cycle, errors in the
problem statement or design may incur exorbitant costs. Not only must the original error be
corrected, but the entire structure built upon it must also be changed. Therefore, testing should not
be isolated as an inspection activity. Rather testing should be involved throughout the SDLC in
order to bring out a quality product.

Testing Activities in Each Phase

The following testing activities should be performed during the phases:

1. Requirements Analysis

- Determine correctness
- Generate functional test data.

2. Design

- Determine correctness and consistency

- Generate structural and functional test data.

3. Programming/Construction

- Determine correctness and consistency

- Generate structural and functional test data
- Apply test data
- Refine test data

4. Operation and Maintenance

- Retest

Requirements Analysis

The following test activities should be performed during this stage:

1.1 Invest in analysis at the beginning of the project - Having a clear, concise and formal
statement of the requirements facilitates programming, communication, error analysis and test
data generation.

[3]
The requirements statement should record the following information and decisions:

a. Program function - What the program must do?

b. The form, format, data types and units for input.

c. The form, format, data types and units for output.

d. How exceptions, errors and deviations are to be handled.

e. For scientific computations, the numerical method or at least the required accuracy of the
solution.

f. The hardware/software environment required or assumed (e.g. the machine,

theoperating system, and the implementation language).

Deciding the above issues is one of the activities related to testing that should be performed
during this stage.

1.2 Start developing the test set at the requirements analysis phase - Data should be generated
that can be used to determine whether the requirements have been met. To do this, the
input domain should be partitioned into classes of values that the program will treat in a similar
manner and for each class a representative element should be included in the test data.

In addition, following should also be included in the data set:

(1) boundary values

(2) any non-extreme input values that would require special handling.

The output domain should be treated similarly.

Invalid input requires the same analysis as valid input.

1.3 The correctness, consistency and completeness of the requirements should also be analyzed -
Consider whether the correct problem is being solved, check for conflicts and inconsistencies
among the requirements and consider the possibility of missing cases.

[4]
Design

The design document aids in programming, communication, and error analysis and test data
generation. The requirements statement and the design document should together give the
problem and the organization of the solution i.e. what the program will do and how it will be
done.

The design document should contain:

 Principal data structures.

 Functions, algorithms, heuristics or special techniques used for processing.

 The program organization, how it will be modularized and categorized into external and
internal interfaces.

 Any additional information.

Here the testing activities should consist of:

- Analysis of design to check its completeness and consistency - the total process should be
analyzed to determine that no steps or special cases have been overlooked. Internal interfaces,
I/O handling and data structures should specially be checked for inconsistencies.

- Analysis of design to check whether it satisfies the requirements - check whether both
requirements and design document contain the same form, format, units used for input and
output and also that all functions listed in the requirement document have been included in the
design document. Selected test data which is generated during the requirements analysis phase
should be manually simulated to determine whether the design will yield the expected values.

- Generation of test data based on the design - The tests generated should cover the structure as
well as the internal functions of the design like the data structures, algorithm, functions,
heuristics and general program structure etc. Standard extreme and special values should be
included and expected output should be recorded in the test data.

- Re-examination and refinement of the test data set generated at the requirements analysis
phase.

The first two steps should also be performed by some colleague and not only the
designer/developer.

[5]
Programming/Construction

Here the main testing points are:

- Check the code for consistency with design - the areas to check include modular structure,
module interfaces, data structures, functions, algorithms and I/O handling.

- Perform the Testing process in an organized and systematic manner with test runs dated,
annotated and saved. A plan or schedule can be used as a checklist to help
theprogrammer organize testing efforts. If errors are found and changes made to the program, all
tests involving the erroneous segment (including those which resulted in success previously)
must be rerun and recorded.

- Asks some colleague for assistance - Some independent party, other than the programmer of the
specific part of the code, should analyze the development product at each phase. The
programmer should explain the product to the party who will then question the logic and search
for errors with a checklist to guide the search. This is needed to locate errors the programmer has
overlooked.

- Use available tools - the programmer should be familiar with various compilers and interpreters
available on the system for the implementation language being used because they differ in their
error analysis and code generation capabilities.

- Apply Stress to the Program - Testing should exercise and stress the program structure, the data
structures, the internal functions and the externally visible functions or functionality. Both valid
and invalid data should be included in the test set.
- Test one at a time - Pieces of code, individual modules and small collections of modules should
be exercised separately before they are integrated into the total program, one by one. Errors are
easier to isolate when the no. of potential interactions should be kept small. Instrumentation-
insertion of some code into the program solely to measure various program characteristics – can
be useful here. A tester should perform array bound checks, check loop control variables,
determine whether key data values are within permissible ranges, trace program execution, and
count the no. of times a group of statements is executed.

- Measure testing coverage/When should testing stop? - If errors are still found every time the
program is executed, testing should continue. Because errors tend to cluster, modules appearing
particularly error-prone require special scrutiny.

The metrics used to measure testing thoroughness include statement testing (whether each
statement in the program has been executed at least once), branch testing (whether each exit from
each branch has been executed at least once) and path testing (whether all logical paths, which
may involve repeated execution of various segments, have been executed at least once).
[6]
Statement testing is the coverage metric most frequently used as it is relatively simple to
implement.

The amount of testing depends on the cost of an error. Critical programs or functions require
more thorough testing than the less significant functions.

Operations and Maintenance

Corrections, modifications and extensions are bound to occur even for small programs and testing
is required every time there is a change. Testing during maintenance is termed regression testing.
The test set, the test plan, and the test results for the original program should exist.

Modifications must be made to accommodate the program changes, and then all portions of the
program affected by the modifications must be re-tested. After regression testing is complete, the
program and test documentation must be updated to reflect the changes.

SDLC models

Introduction

There are various software development approaches defined and designed which are
used/employed during development process of software, these approaches are also referred as
"Software Development Process Models".

Each process model follows a particular life cycle in order to ensure success in process of
software development.

Waterfall Model

Waterfall approach was first Process Model to be introduced and followed widely in Software
Engineering to ensure success of the project. In "The Waterfall" approach, the whole process of
software development is divided into separate process phases.

The phases in Waterfall model are: Requirement Specifications phase, Software Design,
Implementation and Testing & Maintenance. All these phases are cascaded to each other so that
second phase is started as and when defined set of goals are achieved for first phase and it is
signed off, so the name "Waterfall Model". All the methods and processes undertaken in
Waterfall Model are more visible.

[7]
The stages of "The Waterfall Model" are:

Requirement Analysis & Definition: All possible requirements of the system to be developed
are captured in this phase. Requirements are set of functionalities and constraints that the end-
user (who will be using the system) expects from the system. The requirements are gathered
from the end-user by consultation, these requirements are analyzed for their validity and the
possibility of incorporating the requirements in the system to be development is also studied.
Finally, a Requirement Specification document is created which serves the purpose of guideline
for the next phase of the model.

System & Software Design: Before a starting for actual coding, it is highly important to
understand what we are going to create and what it should look like? The requirement
specifications from first phase are studied in this phase and system design is prepared. System
Design helps in specifying hardware and system requirements and also helps in defining overall
system architecture. The system design specifications serve as input for the next phase of the
model.

Implementation & Unit Testing: On receiving system design documents, the work is divided in
modules/units and actual coding is started. The system is first developed in small programs
called units, which are integrated in the next phase. Each unit is developed and tested for its
functionality; this is referred to as Unit Testing. Unit testing mainly verifies if the modules/units
meet their specifications.

Integration & System Testing: As specified above, the system is first divided in units which are
developed and tested for their functionalities. These units are integrated into a complete system

[8]
during Integration phase and tested to check if all modules/units coordinate between each other
and the system as a whole behaves as per the specifications. After successfully testing
the software, it is delivered to the customer.

Operations & Maintenance: This phase of "The Waterfall Model" is virtually never ending
phase (Very long). Generally, problems with the system developed (which are not found during
the development life cycle) come up after its practical use starts, so the issues related to the
system are solved after deployment of the system. Not all the problems come in picture directly
but they arise time to time and needs to be solved; hence this process is referred as Maintenance.

Advantages

The advantage of waterfall development is that it allows for departmentalization and managerial
control. A schedule can be set with deadlines for each stage of development and a product can
proceed through the development process like a car in a carwash, and theoretically, be delivered
on time. Development moves from concept, through design, implementation, testing, installation,
troubleshooting, and ends up at operation and maintenance. Each phase of development proceeds
in strict order, without any overlapping or iterative steps.

Disadvantages

The disadvantage of waterfall development is that it does not allow for much reflection or
revision. Once an application is in the testing stage, it is very difficult to go back and change
something that was not well-thought out in the concept stage. Alternatives to the waterfall model
include joint application development (JAD), rapid applicationdevelopment (RAD), synch and
stabilize, build and fix, and the spiral model.

Iterative Model

An iterative lifecycle model does not attempt to start with a full specification of requirements.
Instead, development begins by specifying and implementing just part of the software, which can
then be reviewed in order to identify further requirements. This process is then repeated,
producing a new version of the software for each cycle of the model. Consider an iterative
lifecycle model which consists of repeating the following four phases in sequence:

[9]
A Requirements phase, in which the requirements for the software are gathered and analyzed.
Iteration should eventually result in a requirements phase that produces a complete and final
specification of requirements. - A Design

phase, in which a software solution to meet the requirements is designed. This may be a new
design, or an extension of an earlier design.

- An Implementation and Test phase, when the software is coded, integrated and tested.

- A Review phase, in which the software is evaluated, the current requirements are reviewed, and
changes and additions to requirements proposed.

For each cycle of the model, a decision has to be made as to whether the software produced by
the cycle will be discarded, or kept as a starting point for the next cycle (sometimes referred to as
incremental prototyping). Eventually a point will be reached where the requirements are
complete and the software can be delivered, or it becomes impossible to enhance the software as
required, and a fresh start has to be made.

The iterative lifecycle model can be likened to producing software by successive approximation.
Drawing an analogy with mathematical methods that use successive approximation to arrive at a
final solution, the benefit of such methods depends on how rapidly they converge on a solution.

The key to successful use of an iterative software development lifecycle is rigorous validation of
requirements, and verification (including testing) of each version of the software against those
requirements within each cycle of the model. The first three phases of the example iterative
model is in fact an abbreviated form of a sequential V or waterfall lifecycle model. Each cycle of
the model produces software that requires testing at the unit level, for software integration,
for system integration and for acceptance. As the software evolves through successive cycles,
tests have to be repeated and extended to verify each version of the software.

[10]
 V-Model

The V-model is a software development model which can be presumed to be the extension of the
waterfall model. Instead of moving down in a linear way, the process steps are bent upwards
after the coding phase, to form the typical V shape. The V-Model demonstrates the relationships
between each phase of the development life cycle and its associated phase of testing.

Verification Phases

1. Requirements analysis:

In this phase, the requirements of the proposed system are collected by analyzing the needs of
the user(s). This phase is concerned about establishing what the ideal system has to perform.
However, it does not determine how the software will be designed or built. Usually, the users are
interviewed and a document called the user requirements document is generated. The user
requirements document will typically describe the system’s functional, physical, interface,
performance, data, security requirements etc as expected by the user. It is one which the business
analysts use to communicate their understanding of the system back to the users. The users
carefully review this document as this document would serve as the guideline for the system
designers in the system design phase. The user acceptance tests are designed in this phase.

[11]
 2. System Design:

System engineers analyze and understand the business of the proposed system by studying the
user requirements document. They figure out possibilities and techniques by which the user
requirements can be implemented. If any of the requirements are not feasible, the user is
informed of the issue. A resolution is found and the user requirement document is edited
accordingly.

The software specification document which serves as a blueprint for the development phase is
generated. This document contains the general system organization, menu structures, data
structures etc. It may also hold example business scenarios, sample windows, reports for the
better understanding. Other technical documentation like entity diagrams, data dictionary will
also be produced in this phase. The documents for system testing is prepared in this phase.

3. Architecture Design:

This phase can also be called as high-level design. The baseline in selecting the architecture is
that it should realize all which typically consists of the list of modules, brief functionality of each
module, their interface relationships, dependencies, database tables, architecture
diagrams, technology details etc. The integration testing design is carried out in this phase.

4. Module Design:

This phase can also be called as low-level design. The designed system is broken up in to smaller
units or modules and each of them is explained so that the programmer can start coding directly.
The low level design document or program specifications will contain a detailed functional logic
of the module, in pseudocode - database tables, with all elements, including their type and size -
all interface details with complete API references- all dependency issues- error message listings-
complete input and outputs for a module. The unit test design is developed in this stage.

Spiral Model

History

The spiral model was defined by Barry Boehm in his 1988 article A Spiral Model of Software
Development and Enhancement. This model was not the first model to discuss iterative
development, but it was the first model to explain why the iteration matters. As originally
envisioned, the iterations were typically 6 months to 2 years long. Each phase starts with a
design goal and ends with the client (who may be internal) reviewing the progress thus far.
Analysis and engineering efforts are applied at each phase of the project, with an eye toward the
end goal of the project.

[12]
The Spiral Model

The spiral model, also known as the spiral lifecycle model, is a systems development method
(SDM) used in information technology (IT). This model of development combines the features
of the prototyping model and the waterfall model. The spiral model is intended for large,
expensive, and complicated projects.

The steps in the spiral model can be generalized as follows:

1. The new system requirements are defined in as much detail as possible. This usually
involves interviewing a number of users representing all the external or internal users and
other aspects of the existing system.

2. A preliminary design is created for the new system.

3. A first prototype of the new system is constructed from the preliminary design. This is
usually a scaled-down system, and represents an approximation of the characteristics of
the final product.

4. A second prototype is evolved by a fourfold procedure: (1) evaluating the first prototype
in terms of its strengths, weaknesses, and risks; (2) defining the requirements of the
second prototype; (3) planning and designing the second prototype; (4) constructing and
testing the second prototype.

5. At the customer's option, the entire project can be aborted if the risk is deemed too great.
Risk factors might involve development cost overruns, operating-cost miscalculation, or
any other factor that could, in the customer's judgment, result in a less-than-satisfactory
final product.

6. The existing prototype is evaluated in the same manner as was the previous prototype,
and, if necessary, another prototype is developed from it according to the fourfold
procedure outlined above.

7. The preceding steps are iterated until the customer is satisfied that the refined prototype
represents the final product desired.

8. The final system is constructed, based on the refined prototype.

9. The final system is thoroughly evaluated and tested. Routine maintenance is carried out
on a continuing basis to prevent large-scale failures and to minimize downtime.

Advantages

1. Estimates (i.e. budget, schedule, etc.) become more realistic as work progresses, because
important issues are discovered earlier.

[13]
 2. It is more able to cope with the (nearly inevitable) changes that software development
generally entails.

3. Software engineers (who can get restless with protracted design processes) can get their
hands in and start working on a project earlier.

Disadvantages

1. Highly customized limiting re-usability

2. Applied differently for each application

3. Risk of not meeting budget or schedule

4. Risk of not meeting budget or schedule

[14]
Reference

Acharya, Kamal. "STUDENT INFORMATION MANAGEMENT

SYSTEM." Authorea Preprints (2023).
Acharya, Kamal. "Library Management System." Available at SSRN4807104 (2019).
ACHARYA, KAMAL, et al. "LIBRARY MANAGEMENT SYSTEM." (2019).
Acharya, Kamal. "Online bus reservation system project report." Authorea
Preprints (2024).
Acharya, Kamal. "Online bus reservation system project report." (2024).
Acharya, Kamal. “Online Bus Reservation System.” SSRN ElectroNIC ASIA
Journal (2024): n. pag.
Acharya, Kamal. “Student Information Management System Project.” SSRN
ElectroNIC ASIA Journal (2024): n. pag.
Acharya, Kamal. “ATTENDANCE MANAGEMENT SYSTEM.” International
Research Journal of Modernization in Engineering Technology and
Science (2023): n. pag.
Acharya, Kamal. “College Information Management System.” SSRN ElectroNIC
ASIA Journal (2024): n. pag.
Acharya, Kamal, Attendance Management System Project (April 28, 2024).
Available at
SSRN: https://ssrn.com/abstract=4810251 or http://dx.doi.org/10.2139/ssrn.4810251
Acharya, Kamal, Online Food Order System (May 2, 2024). Available at
SSRN: https://ssrn.com/abstract=4814732 or http://dx.doi.org/10.2139/ssrn.4814732
Acharya, Kamal, University management system project. (May 1, 2024). Availableat
SSRN: https://ssrn.com/abstract=4814103 or http://dx.doi.org/10.2139/ssrn.4814103
Acharya, Kamal, Online banking management system. (May 1, 2024). Available at
SSRN: https://ssrn.com/abstract=4813597 or http://dx.doi.org/10.2139/ssrn.4813597
Acharya, Kamal, Online Job Portal Management System (May 5, 2024). Available at
SSRN: https://ssrn.com/abstract=4817534 or http://dx.doi.org/10.2139/ssrn.4817534
Acharya, Kamal, Employee leave management system. (May 7, 2024). Available
at SSRN: https://ssrn.com/abstract=4819626 or http://dx.doi.org/10.2139/ssrn.4819626
Acharya, Kamal, Online electricity billing project report. (May 7, 2024). Available at
SSRN: https://ssrn.com/abstract=4819630 or http://dx.doi.org/10.2139/ssrn.4819630

[15]