CCS366 - SOFTWARE TESTING AND AUTOMATION

UNIT IV
ADVANCED TESTING CONCEPTS
Performance Testing: Load Testing, Stress Testing, Volume Testing, Fail-Over Testing,
Recovery Testing, Configuration Testing, Compatibility Testing, Usability Testing, Testing
the Documentation, Security testing, Testing in the Agile Environment, Testing Web and
Mobile Applications.
4.1 PERFORMANCE TESTING
Performance Testing is a type of software testing that ensures software applications
perform properly under their expected workload. It is a testing technique carried out to
determine system performance in terms of sensitivity, reactivity, and stability under a
particular workload.
Performance Testing is the process of analysing the quality and capability of a
product. It is a testing method performed to determine the system’s performance in terms of
speed, reliability, and stability under varying workloads. Performance testing is also known
as Perf Testing.

Performance Testing Attributes:

• Speed: It determines whether the software product responds rapidly.
• Scalability: It determines the amount of load the software product can handle at a
time.
• Stability: It determines whether the software product is stable in case of varying
workloads.
• Reliability: It determines whether the software product is secure or not.

Objective of Performance Testing:

• The objective of performance testing is to eliminate performance congestion.
• It uncovers what needs to be improved before the product is launched in the market.
• The objective of performance testing is to make software rapid.
• The objective of performance testing is to make software stable and reliable.
• The objective of performance testing is to evaluate the performance and scalability of
a system or application under various loads and conditions. It helps identify
bottlenecks, measure system performance, and ensure that the system can handle the
expected number of users or transactions. It also helps to ensure that the system is
reliable, stable, and can handle the expected load in a production environment.

Types of Performance Testing:

Load testing: It checks the product’s ability to perform under anticipated user loads. The
objective is to identify performance congestion before the software product is launched in the
market.
Stress testing: It involves testing a product under extreme workloads to see whether it
handles high traffic or not. The objective is to identify the breaking point of a software
product. Spike testing: It is a type of load testing that tests the system’s ability to handle
sudden spikes in traffic. It helps identify any issues that may occur when the system is
suddenly hit with a high number of requests.

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Soak testing: Soak testing is a type of load testing that tests the system’s ability to handle a
sustained load over a prolonged period. It helps identify any issues that may occur after
prolonged usage of the system.
Endurance testing: This type of testing is similar to soak testing, but it focuses on the long-
term behaviour of the system under a constant load.
Volume testing: In volume testing, large number of data is saved in a database and the
overall software system’s behaviour is observed. The objective is to check the product’s
performance under varying database volumes.
4.2 LOAD TESTING
Load testing determines the behavior of the application when multiple users use it at the
same time. It is the response of the system measured under varying load conditions.
• The load testing is carried out for normal and extreme load conditions.
• Load testing is a type of performance testing that simulates a real-world load on a
system or application to see how it performs under stress.
• The goal of load testing is to identify bottlenecks and determine the maximum
number of users or transactions the system can handle.
• It is an important aspect of software testing as it helps ensure that the system can
handle the expected usage levels and identify any potential issues before the system is
deployed to production.
During load testing, various scenarios are simulated to test the system’s behavior under
different load conditions. This can include simulating a high number of concurrent users,
simulating numerous requests, and simulating heavy network traffic. The system’s
performance is then measured and analyzed to identify any bottlenecks or issues that may
occur.

Load Testing Techniques:

Stress testing: Testing the system’s ability to handle a high load above normal usage levels.
Spike testing: Testing the system’s ability to handle sudden spikes in traffic.
Soak testing: Testing the system’s ability to handle a sustained load over a prolonged period
of time.
Tools for Performance Testing: Make use of specialized load testing tools like Locust,
Gatling, JMeter, LoadRunner, and Apache Benchmark. These tools assist in gathering
performance measurements and simulating a large number of users.
Specify the Test Objectives: Clearly state what your load test’s goals are. Recognize the
required response times, transaction volumes and expected user behavior.
Determine Crucial Situations: Determine the essential user scenarios that correspond to
common usage patterns. A variety of actions, including user logins, searches, form
submissions and other significant interactions, should be covered by these scenarios.

Objectives of Load Testing:

Evaluation of Scalability: Assess the system’s ability to handle growing user and
transaction demands. Find the point at which the system begins to function badly.
Planning for Capacity: Describe the system’s ability to accommodate anticipated future
increases in the number of users, transactions and volume of data. Making well-informed
decisions regarding infrastructure upgrades is made easier by this.

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Determine bottlenecks: Identify and localize bottlenecks in the application or
infrastructure’s performance. Finding the places where the system’s performance can suffer
under load is part of this.
Analysis of Response Time: For crucial transactions and user interactions, track and
evaluate response times. Make that the system responds to changes in load with reasonable
response times.
Finding Memory Leaks: Find and fix memory leaks that may eventually cause a decline in
performance. Make sure the programme doesn’t use up too many resources when it’s
running.

Load Testing Process:

1. Test Environment Setup: Firstly create a dedicated test environment setup for
performing the load testing. It ensures that testing would be done in a proper way.
2. Load Test Scenario: In second step load test scenarios are created. Then load testing
transactions are determined for an application and data is prepared for each
transaction.
3. Test Scenario Execution: Load test scenarios that were created in previous step are
now executed. Different measurements and metrices are gathered to collect the
information.
4. Test Result Analysis: Results of the testing performed is analyzed and various
recommendations are made.
5. Re-test: If the test is failed then the test is performed again in order to get the result in
correct way.

Metrics of Load Testing:

Metrics are used in knowing the performance of load testing under different
circumstances. It tells how accurately the load testing is working under different test cases. It
is usually carried out after the preparation of load test scripts/cases. There are many metrics
to evaluate the load testing. Some of them are listed below.
1. Average Response Time
It tells the average time taken to respond to the request generated by the clients or
customers or users. It also shows the speed of the application depending upon the time taken
to respond to the all requests generated.

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
2. Error Rate
The Error Rate is mentioned in terms of percentage denotes the number of errors
occurred during the requests to the total number of requests. These errors are usually raised
when the application is no longer handling the request at the given time or for some other
technical problems. It makes the application less efficient when the error rate keeps on
increasing.
3. Throughput
This metric is used in knowing the range of bandwidth consumed during the load scripts or
tests and it is also used in knowing the amount of data which is being used for checking the
request that flows between the user server and application main server. It is measured in
kilobytes per second. 4. Requests Per Second
It tells that how many requests are being generated to the application server per second. The
requests could be anything like requesting of images, documents, web pages, articles or any
other resources.
5. Concurrent Users
This metric is used to take the count of the users who are actively present at the particular
time or at any time. It just keeps track of count those who are visiting the application at any
time without raising any request in the application. From this, we can easily know that at
which time the high number of users are visiting the application or website.
6. Peak Response Time
Peak Response Time measures the time taken to handle the request. It also helps in finding
the duration of the peak time(longest time) at which the request and response cycle is handled
and finding that which resource is taking longer time to respond the request.

Load Testing Tools: Apache Jmeter, WebLoad, NeoLoad, LoadNinja, HP Performance, Tester,
LoadUI Pro, LoadView.

4.3 STRESS TESTING

Stress Testing is a software testing technique that determines the robustness of
software by testing beyond the limits of normal operation. Stress testing is particularly
important for critical software but is used for all types of software. Stress testing emphasizes
robustness, availability, and error handling under a heavy load rather than what is correct
behavior under normal situations.
Stress testing is defined as a type of software testing that verifies the stability and
reliability of the system. This test particularly determines the system on its robustness and
error handling under extremely heavy load conditions. It even tests beyond the normal
operating point and analyses how the system works under extreme conditions. Stress testing
is performed to ensure that the system would not crash under crunch situations. Stress testing
is also known as Endurance Testing or Torture Testing.

Characteristics of Stress Testing:

1. Identification of Risk: Stress testing’s main objective is to locate and evaluate a
system’s possible hazards and weaknesses.
2. Quantitative and Qualitative Analysis: While numerical data are crucial, it’s also
critical to comprehend the qualitative characteristics of the system’s response and
potential weak points.

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
3. Variable Parameters: Stress testing include changing variables including interest
rates, market conditions, transaction volumes and outside influences that could have
an impact on the system.
4. Cross-Functional Involvement: Many departments within an organization must
work together and participate in stress testing. This cross-functional strategy makes
sure that the stress testing procedure benefits from a variety of viewpoints and
specialties.
5. Open and Honest Communication: Stress testing necessitates open and honest
communication regarding the goal, approach, and outcomes of the testing procedure.

Need for Stress Testing:

• To accommodate the sudden surges in traffic: It is important to perform stress
testing to accommodate abnormal traffic spikes. For example, when there is a sale
announcement on the e-commerce website there is a sudden increase in traffic. Failure
to accommodate such needs may lead to a loss of revenue and reputation.
• Display error messages in stress conditions: Stress testing is important to check
whether the system is capable to display appropriate error messages when the system
is under stress conditions.
• The system works under abnormal conditions: Stress testing checks whether the
system can continue to function in abnormal conditions.
• Prepared for stress conditions: Stress testing helps to make sure there are sufficient
contingency plans in case of sudden failure due to stress conditions. It is better to be
prepared for extreme conditions by executing stress testing.

Purpose of Stress Testing:

• Analyze the behavior of the application after failure: The purpose of stress testing
is to analyze the behavior of the application after failure and the software should
display the appropriate error messages while it is under extreme conditions.
• System recovers after failure: Stress testing aims to make sure that there are plans
for recovering the system to the working state so that the system recovers after failure.
• Uncover Hardware issues: Stress testing helps to uncover hardware issues and data
corruption issues.
• Uncover Security Weakness: Stress testing helps to uncover the security
vulnerabilities that may enter into the system during the constant peak load and
compromise the system.
• Ensures data integrity: Stress testing helps to determine the application’s data
integrity throughout the extreme load, which means that the data should be in a
dependable state even after a failure.

Stress Testing Process:

The stress testing process is divided into 5 steps:

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

1. Planning the stress test: This step involves gathering the system data, analyzing the
system, and defining the stress test goals.
2. Create Automation Scripts: This step involves creating the stress testing automation
scripts and generating the test data for the stress test scenarios.
3. Script Execution: This step involves running the stress test automation scripts and
storing the stress test results.
4. Result Analysis: This phase involves analyzing stress test results and identifying the
bottlenecks.
5. Tweaking and Optimization: This step involves fine-tuning the system and
optimizing the code with the goal meet the desired benchmarks.

Types of Stress Testing:

1. Server-client Stress Testing: Server-client stress testing also known as distributed
stress testing is carried out across all clients from the server.
2. Product Stress Testing: Product stress testing concentrates on discovering defects
related to data locking and blocking, network issues, and performance congestion in a
software product.
3. Transactional Stress Testing: Transaction stress testing is performed on one or more
transactions between two or more applications. It is carried out for fine-tuning and
optimizing the system.
4. Systematic Stress Testing: Systematic stress testing is integrated testing that is used
to perform tests across multiple systems running on the same server. It is used to
discover defects where one application data blocks another application.
5. Analytical Stress Testing: Analytical or exploratory stress testing is performed to
test the system with abnormal parameters or conditions that are unlikely to happen in
a real scenario. It is carried out to find defects in unusual scenarios like a large
number of users logged at the same time or a database going offline when it is
accessed from a website.
6. Application Stress Testing: Application stress testing also known as product stress
testing is focused on identifying the performance bottleneck, and network issues in a
software product.

Stress Testing Tools:

1. Jmeter: Apache JMeter is a stress testing tool is an open-source, pure Java-based
software that is used to stress test websites. It is an Apache project and can be used
for load testing for analyzing and measuring the performance of a variety of services.

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
2. LoadNinja: LoadNinja is a stress testing tool developed by SmartBear that enables
users to develop codeless load tests, substitutes load emulators with actual browsers,
and helps to achieve high speed and efficiency with browser-based metrics.
3. WebLoad: WebLoad is a stress testing tool that combines performance, stability, and
integrity as a single process for the verification of mobile and web applications.
4. Neoload: Neoload is a powerful performance testing tool that simulates large
numbers of users and analyzes the server’s behavior. It is designed for both mobile
and web applications. Neoload supports API testing and integrates with different CI/
CD applications.
5. SmartMeter: SmartMeter is a user-friendly tool that helps to create simple tests
without coding. It has a graphical user interface and has no necessary plugins. This
tool automatically generates advanced test reports with complete and detailed test
results.

Metrics of Stress Testing:

Metrics are used to evaluate the performance of the stress and it is usually carried out at the
end of the stress scripts or tests. Some of the metrics are given below.
1. Pages Per Second: Number of pages requested per second and number of pages
loaded per second.
2. Pages Retrieved: Average time is taken to retrieve all information from a particular
page.
3. Byte Retrieved: Average time is taken to retrieve the first byte of information from
the page.
4. Transaction Response Time: Average time is taken to load or perform transactions
between the applications.
5. Transactions per Second: It takes count of the number of transactions loaded per
second successfully and it also counts the number of failures that occurred.
6. Failure of Connection: It takes count of the number of times that the client faced
connection failure in their system.
7. Failure of System Attempts: It takes count of the number of failed attempts in the
system.
8. Rounds: It takes count of the number of test or script conditions executed by the
clients successfully and it keeps track of the number of rounds failed.

4.4 VOLUME TESTING

Volume Testing is a type of software testing which is carried out to test a software
application with a certain amount of data. The amount used in volume testing could be a
database size or it could also be the size of an interface file that is the subject of volume
testing.
While testing the application with a specific database size, database is extended to
that size and after that the performance of the application is tested. When an application
needs interaction with an interface file this could be either reading or writing the file or same
from the file. A sample file of the size required is created and then functionality of the
application is tested with that file in order to test the performance.
In volume testing a huge volume of data is acted upon the software. It is basically
performed to analyze the performance of the system by increasing the volume of data in the
database. Volume testing is performed to study the impact on response time and behavior of
the system when the volume of data is increased in the database.
Volume Testing is also known as Flood Testing.

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Characteristics of Volume Testing
Following are the characteristics of the Volume Testing:
• Performance of the software decline as passing of the time as there is huge amount of
data overtime.
• Basically the test data is created by test data generator.
• Only small amount of data is tested during development phase.
• The test data need to be logically correct.
• The test data is used to assess the performance of the system.

Objectives of Volume Testing

The objectives of volume testing is:
• To recognize the problems that may be created with large amount of data.
• To check The system’s performance by increasing the volume of data in the database.
• To find the point at which the stability of the system reduces.
• To identify the capacity of the system or application.
Volume Testing Attributes
Following are the important attributes that are checked during the volume testing:
• System’s Response Time: During the volume testing, the response time of the
system or the application is tested. It is also tested whether the system responses
within the finite time or not. If the response time is large then the system is
redesigned.
• Data Loss: During the volume testing, it is also tested that there is no data loss. If
there is data loss some key information might be missing.
• Data Storage: During the volume testing, it is also tested that the data is stored
correctly or not. If the data is not stored correctly then it is restored accordingly in
proper place.
• Data Overwriting: In volume testing, it is tested that whether the data is overwritten
without giving prior information to the developer. If it so then developer is notified.

Advantages of Volume Testing

• Volume testing is helpful in saving maintenance cost that will be spent on application
maintenance.
• Volume testing is also helpful in a rapid start for scalability plans.
• Volume testing also helps in early identification of bottlenecks.
• Volume testing ensures that the system is capable of real world usage.

Disadvantages of Volume Testing

• More number of skilled resources are needed to carry out this testing.
• It is sometimes difficult to prepare test cases with respect to the number of volume of
data to be tested.
• It is a time consuming technique since it requires lot of time to decide the number of
volume of data and test scenarios.
• It is a bit costly as compared to another testing technique.
• It is not possible to have the exact break down of memory used in the real world
application.

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

10

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
4.5 FAIL-OVER TESTING
Software products/services are tested multiple times before delivery to ensure that it
is providing the required service. Testing before delivery doesn’t guarantee that no problem
will occur in the future. Even sometimes the software application fails due to some unwanted
event due to network issues or due to server-related problems. Failover testing aims to
respond to these types of failures.
Suppose that the PC gets off due to some technical issue, and on restarting we open the
browser, then a pop-up is shown saying Do you want to restore all pages? On clicking
restore, all tabs are restored. The process of ensuring such restorations is known as
FAILOVER TESTING.

What is Failover Testing :

Failover testing is a technique that validates if a system can allocate extra resources
and back up all the information and operations when a system fails abruptly due to some
reason. This test determines the ability of a system to handle critical failures and handle extra
servers. So, the testing is independent of the physical hardware component of a server.
It is preferred that testing should be performed by servers. Active-active and active-
passive standby are the two most common configurations. Both the techniques achieve
failover in a very different manner but both of them are performed to improve the server’s
reliability.
For example, if we have three servers, one of them fails due to heavy load, and then
two situations occur. Either that failed server will restart on its own or in another situation
when the failed server cannot be restarted, the remaining servers will handle the load. Such
situations are tested during this test.

Considerable Factors Before Performing Failover Testing:

1. The budget has to be the first thing to be taken into consideration before thinking
about performing the Failover test.
2. The budget is connected to the frameworks that might crash or break down under
pressure/load.
3. Always keep in mind how much time it will take to fix all the issues caused by the
failure of the system.
4. Note down the most likely failures and organize the outcomes according to how much
harm is caused by the failure.

Considerable Factors While Performing Failover Testing:

1. Keep a plan of measures to be taken after performing a test.
2. Focus on the execution of the test plan.
3. Set up a benchmark so that performance requirements can be achieved.
4. Prepare a report concerning issue requirements and/or requirements of the asset.

Working of Failover testing:

11

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

1. Consider the factors: Before performing failover testing like budget, time, team,
technology, etc.
2. Analysis on failover reasons and design solutions: Determine probable failure
situations that the system might experience. Examine the causes of failover, including
software bugs, hardware malfunctions, network problems, etc. It provides fixes for
any flaws or vulnerabilities found in the failover procedure.
3. Testing failover scenarios: It develops extensive test cases to replicate various failover
scenarios. This covers both unplanned failovers (system or component failures) and
scheduled failovers (maintenance). Test cases ought to address many facets of
failover, such as load balancing, user impact, network rerouting, and data
synchronization.
4. Executing the test plan: To reduce the impact on production systems, carry out the
failover test plan in a controlled setting. Keep an eye on how the system behaves
during failover to make sure it satisfies the recovery point and recovery time
objectives (RPO and RTO, respectively).
5. Detailed report on failover: Keep a record of the failover testing findings, including
any problems you ran across, how long it took to failover and how it affected
customers or services. Assess problems according to their severity and offer
suggestions for improvements.
6. Necessary actions based on the report: Distribute the report on the failover test to all
pertinent parties, such as project managers, developers, and system administrators.
Determine what needs to be done and prioritize it based on the report’s conclusions.
This might involve fixing found flaws in the system, updating failover setups or
improving the documentation.

Benefits of Failover Testing:

1. Determines Vulnerabilities and Weaknesses: Failover testing helps in locating holes
and vulnerabilities in the system by simulating different failure scenarios.
Organizations can address any problems before they have an impact on production
settings by taking a proactive approach.
2. Verifies Redundancy Procedures: Redundancy mechanisms are frequently incorporated
into systems to improve reliability. In order to make sure that these systems such as
redundant servers, databases, or network paths, work flawlessly in the event of a
breakdown, failover testing verifies their efficacy.
3. Improving the User Experience: Consumers anticipate consistent and dependable
service availability. By reducing interruptions and ensuring service availability even
in

12

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

13

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
the face of unanticipated circumstances, failover testing contributes to the provision
of a positive user experience.
4. Encourages Compliance: Failover testing supports organizations in demonstrating
compliance with these rules by ensuring that systems can recover within prescribed
times.
5. Encourages Continuous Improvement: Organizations that regularly test their systems
can learn from each test cycle, fix problems found and upgrade their failover
procedures to meet changing business needs and technological advancements.

Examples of Failover Testing:

1. Banking and Financial applications
2. Telecom applications
3. Visa applications
4. Trading applications
5. Emergency service business applications
6. Government applications
7. Defense service-related applications
4.6 RECOVERY TESTING
Recovery testing is a type of system testing which aims at testing whether a system
can recover from failures or not. The technique involves failing the system and then verifying
that the system recovery is performed properly.
To ensure that a system is fault-tolerant and can recover well from failures, recovery
testing is important to perform. A system is expected to recover from faults and resume its
work within a pre-specified time period. Recovery testing is essential for any mission-critical
system, for example, the defense systems, medical devices, etc. In such systems, there is a
strict protocol that is imposed on how and within what time period the system should recover
from failure and how the system should behave during the failure.
A system or software should be recovery tested for failures like:
• Power supply failure
• The external server is unreachable
• Wireless network signal loss
• Physical conditions
• The external device not responding
• The external device is not responding as expected, etc.

Steps to be performed before executing a Recovery Test

A tester must ensure that the following steps are performed before carrying out the
Recovery testing procedure :
1. Recovery Analysis – It is important to analyze the system’s ability to allocate extra
resources like servers or additional CPUs. This would help to better understand the
recovery-related changes that can impact the working of the system. Also, each of the
possible failures, their possible impact, their severity, and how to perform them
should be studied.
2. Test Plan preparation – Designing the test cases keeping in mind the environment
and results obtained in recovery analysis.

14

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
3. Test environment preparation – Designing the test environment according to the
recovery analysis results.
4. Maintaining Back-up – Information related to the software, like various states of the
software and database should be backed up. Also, if the data is important, then the
backing up of the data at multiple locations is important.
5. Recovery personnel Allocation – For the recovery testing process, it is important to
allocate recovery personnel who are aware and educated enough for the recovery
testing being conducted.
6. Documentation – This step emphasizes on documenting all the steps performed
before and during the recovery testing so that the system can be analyzed for its
performance in case of a failure.

Example of Recovery Testing

• When a system is receiving some data over a network for processing purposes, we
can stimulate software failure by unplugging the system power. After a while, we can
plug in the system again and test its ability to recover and continue receiving the data
from where it stopped.
• Another example could be when a browser is working on multiple sessions, we can
stimulate software failure by restarting the system. After restarting the system, we can
check if it recovers from the failure and reloads all the sessions it was previously
working on.
• While downloading a movie over a Wifi network, if we move to a place where there
is no network, then the downloading process will be interrupted. Now to check if the
process recovers from the interruption and continues working as before, we move
back to a place where there is a Wifi network. If the downloading resumes, then the
software has a good recovery rate.

Types of Recovery Testing

• Database Recovery Testing: Evaluate the system’s capacity to recover from
corrupted or malfunctioning databases. In order to test how well the system can
restore the database to a consistent and useful condition, it involves intentionally
destroying or damaging it.
• Load and Stress Recovery Testing: Determine how effectively the system bounces
back from variables that affect performance, including heavy loads or stressful
situations. It helps in determining if the system is capable of handling higher loads
and in the event that it cannot, how soon it will resume normal operation after the
load is dropped.
• Crash Recovery Testing: Determine how well the system bounces back from a
hardware or software failure. To make sure the system can resume regular operations
without losing data, it can involve unexpected shutdowns, abrupt power failures or a
sudden halt of services.
• Security Recovery Testing: Examine the system’s resilience to security lapses,
illegal access, and other security-related events by conducting security recovery
testing. It guarantees that the system can recover from security breaches and
helps discover loopholes in the security procedures, reducing the impact of any
unauthorized access. Data Recovery Testing: Evaluate the system’s capacity to
restore data following an unplanned disruption or failure. To make sure that data
backups, restoration procedures

15

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

16

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
and recovery mechanisms are efficient and dependable, this might involve planned
data loss scenarios.
• Environment Recovery Testing: Examine the software’s ability to adjust to changes
in dependencies or configurations in the environment. It guarantees that in the event
of modifications to the underlying structure or environmental circumstances, the
system can recover and go on operating as anticipated.

Advantages of Recovery Testing

• Improves the quality of the system by eliminating the potential flaws in the system
so that the system works as expected.
• Recovery testing is also referred to as Disaster Recovery Testing. A lot of
companies have disaster recovery centers to make sure that if any of the systems is
damaged or fails due to some reason, then there is back up to recover from the failure.
• Risk elimination is possible as the potential flaws are detected and removed from the
system.
• Improved performance as faults are removed, and the system becomes more reliable
and performs better in case a failure occurs.

Disadvantages of Recovery testing

• Recovery testing is a time-consuming process as it involves multiple steps and
preparations before and during the process.
• The recovery personnel must be trained as the process of recovery testing takes
place under his supervision. So, the tester needs to be trained to ensure that recovery
testing is performed in the proper way. For performing recovery testing, he should
have enough data and back up files to perform recovery testing.
• The potential flaws or issues are unpredictable in a few cases. It is difficult to
point out the exact reason for the same, however, since the quality of the software
must be maintained, so random test cases are created and executed to ensure such
potential flaws are removed.
4.7 CONFIGURATION TESTING
Configuration Testing is the type of Software Testing that verifies the performance
of the system under development against various combinations of software and hardware to
find out the best configuration under which the system can work without any flaws or issues
while matching its functional requirements.

What is Configuration Testing?

Configuration Testing is the process of testing the system under each configuration of the
supported software and hardware. Here, the different configurations of hardware and
software mean the multiple operating system versions, various browsers, various supported
drivers, distinct memory sizes, different hard drive types, various types of CPU, etc.

The various configurations are Win XP, Win 7 32/64 bit, Win 8 32/64 bit, Win 10, etc.
1. Database Configuration: Oracle, DB2, MySQL, MSSQL Server, Sybase etc.
2. Browser Configuration: IE 8, IE 9, FF 16.0, Chrome, Microsoft Edge etc.

Objectives of Configuration Testing:

17

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
1. Adaptability to Different Configurations: Check that the program’s basic features
work consistently and dependably in all configurations. Testing the behavior of the
program with different setups and settings is part of this process.
2. Evaluation of Stability: Examine the software’s stability under various configurations.
Find and fix any configuration-specific problems that might be causing crashes,
unstable systems or strange behavior.
3. Testing the User Experience: Assess the value and consistency of the user
experience across various setups. Make that the graphical user interface (GUI) of the
software adjusts to various screen sizes, resolutions and display settings.
4. Security Throughout Configurations: To make sure that sensitive data is kept safe,
test the software’s security features in various setups. Determine and fix any
vulnerabilities that might be configuration-specific.
5. Compatibility of Networks: Examine the software’s behavior with various network
setups. Evaluate its compatibility with various network types, speeds and latency.
6. Data Compatibility: Check if the programme can manage a range of data
configurations, such as those from diverse sources, databases and file formats. Verify
the consistency and integrity of the data across various setups.

Configuration Testing Process:

Types of Configuration Testing:

Configuration testing is of 2 types:
1. Software Configuration Testing: Software configuration testing is done over the
Application Under Test with various operating system versions and various browser
versions etc. It is a time-consuming testing as it takes long time to install and uninstall
the various software which are to be used for testing. When the build is released,
software configuration begins after passing through the unit test and integration test.

2. Hardware Configuration Testing: Hardware configuration testing is typically

performed in labs where physical machines are used with various hardware connected
to them. When a build is released, the software is installed in all the physical
machines to which the hardware is attached and the test is carried out on each and
every machine to confirm that the application is working fine. While doing hardware
configuration test, the kind of hardware to be tested is spelled out and there are
several computer hardware and peripherals which make it next to impossible to
execute all the tests.

Design the Test Cases to Run on Each Configuration:

Let consider the steps required to test each configuration. This can be as simple as
1. Select and set up the next test configuration from the list.

18

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

19

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
2. Start the software.
3. Load in the file configtest.doc.
4. Confirm that the file is displayed correctly.
5. Print the document.
6. Confirm that there are no error messages and that the printed document matches
the standard.
7. Log any discrepancies as a bug.

Execute the Tests on Each Configuration:

You need to run the test cases and carefully log and report your results to your team
and to the hardware manufacturers if necessary. It’s often difficult and time-consuming to
identify the specific source of configuration problems. You’ll need to work closely with the
programmers and white-box testers to isolate the cause and decide if the bugs you find are
due to your software or to the hardware.
If the bug is specific to the hardware, consult the manufacturer’s website for
information on reporting problems to them. Be sure to identify yourself as a software tester
and what company you work for. Many companies have separate staff set up to assist
software companies writing software to work with their hardware. They may ask you to send
copies of your test software, your test cases, and supporting details to help them isolate the
problem.

Rerun the Tests Until the Results Satisfy Your Team: It’s not uncommon for
configuration testing to run the entire course of a project. Initially a few configurations might
be tried, then a full test pass, then smaller and smaller sets to confirm bug fixes. Eventually
you will get to a point where there are no known bugs or to where the bugs that still exist are
in uncommon or unlikely test configurations. At that point, you can call your configuration
testing complete.

4.8 COMPATIBILITY TESTING

Compatibility testing is software testing which comes under the non-functional testing
category, and it is performed on an application to check its compatibility (running capability)
on different platform/environments. This testing is done only when the application becomes
stable. Means simply this compatibility test aims to check the developed software application
functionality on various software, hardware platforms, network and browser etc. This
compatibility testing is very important in product production and implementation point of
view as it is performed to avoid future issues regarding compatibility.

Types of Compatibility Testing:

Several examples of compatibility testing are given below.

1. Software:
• Testing the compatibility of an application with an Operating System like Linux,
Mac, Windows
• Testing compatibility on Database like Oracle SQL server, MongoDB server. Testing
compatibility on different devices like in mobile phones, computers.

Types based on Version Testing:

There are two types of compatibility testing based on version testing

20

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
1. Forward compatibility testing: When the behavior and compatibility of a software
or hardware is checked with its newer version then it is called as forward
compatibility testing.
2. Backward compatibility testing: When the behavior and compatibility of a software
or hardware is checked with its older version then it is called as backward
compatibility testing.

2. Hardware:
Checking compatibility with a particular size of
• RAM
• ROM
• Hard Disk
• Memory Cards
• Processor Graphics Card
3. Smartphones:
Checking compatibility with different mobile platforms like android, iOS etc.
4. Network:
Checking compatibility with different:
• Bandwidth
• Operating speed
• Capacity

Along with this there are other types of compatibility testing are also performed such as
browser compatibility to check software compatibility with different browsers like Google
Chrome, Internet Explorer etc. device compatibility, version of the software and others.

How to perform Compatibility testing?

Testing the application in a same environment but having different versions. For
example, to test compatibility of Facebook application in your android mobile. First check
for the compatibility with Android 9.0 and then with Android 10.0 for the same version of
Facebook App.
Testing the application in a same version but having different environment. For
example, to test compatibility of Facebook application in your android mobile. First check
for the compatibility with a Facebook application of lower version with a Android 10.0(or
your choice) and then with a Facebook application of higher version with a same version of
Android.

Why compatibility testing is important?

1. It ensures complete customer satisfaction.
2. It provides service across multiple platforms.
3. Identifying bugs during development process.

Compatibility testing defects:

1. Variety of user interface.
2. Changes with respect to font size.
3. Alignment issues.
4. Issues related to existence of broken frames.
5. Issues related to overlapping of content.

21

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

22

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

4.9 USABILITY TESTING

Usability Testing in software testing is a type of testing, that is done from an end
user’s perspective to determine if the system is easily usable. Usability testing is generally
the practice of testing how easy a design is to use on a group of representative users.
A very common mistake in usability testing is conducting a study too late in the
design process. If you wait until right before your product is released, you won’t have the
time or money to fix any issues – and you’ll have wasted a lot of effort developing your
product the wrong way.
This testing has a cycle wherein when:
1. the product is ready,
2. customers are asked to test it,
3. If any further changes,
4. product (software) is returned to the development team with feedback to update the
changes,
5. again the software had to run usability testing,
6. if there are no more changes required, 7. the software is launched in the market.
This whole process from 1 to 5 is repeated unless the software is completely ready and there
are no further changes required. This process helps you to meet customers’ needs and
identify the problems faced by customers during the usage of the software. Usability testing
is also referred to as User Experience.

Phases of Usability Testing:

There are five phases in usability testing which are followed by the system when usability
testing is performed. These are given below:

1. Prepare your product or design to test: The first phase of usability testing is
choosing a product and then making it ready for usability testing. For usability testing,
more functions and operations are required than this phase provided that type of
requirement. Hence, this is one of the most significant phases in usability testing.
2. Find your participants: The second phase of usability testing is finding an employee
who is helping you with performing usability testing. Generally, the number of
participants that you need is based on several case studies. Mostly, five participants
can find almost as many usability problems as you’d find using many more test
participants.
3. Write a test plan: This is the third phase of usability testing. The plan is one of the
first steps in each round of usability testing is to develop a plan for the test. The main
purpose of the plan is to document what you are going to do, how you are going to
conduct the test, what metrics you are going to find, the number of participants you
are going to test, and what scenarios you will use.
4. Take on the role of the moderator: This is the fourth phase of usability testing and

23

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
here the moderator plays a vital role that involves building a partnership with the
participant. Most of the research findings are derived by observing the participant’s
actions and gathering verbal feedback to be an effective moderator, you need to be
able to make instant decisions while simultaneously overseeing various aspects of the
research session.
5. Present your findings/ final report: This phase generally involves combining your
results into an overall score and presenting it meaningfully to your audience. An easy
method to do this is to compare each data point to a target goal and represent this as
one single metric based on the percentage of users who achieved this goal.

Why we Used Usability Testing?

When software is ready, it is important to make sure that the user experience with the product
should be seamless. It should be easy to navigate and all the functions should be working
properly, the competitor’s website will win the race. Therefore, usability testing is
performed. The objective of usability testing is to understand customers’ needs and
requirements and also how users interact with the product (software). With the test, all the
features, functions, and purposes of the software are checked.
The primary goals of usability testing are – discovering problems (hidden issues) and
opportunities, comparing benchmarks, and comparison against other websites. The
parameters tested during usability testing are efficiency, effectiveness, and satisfaction. It
should be performed before any new design is made. This test should be iterated unless all
the necessary changes have been made. Improving the site consistently by performing
usability testing enhances its performance which in return makes it the best website.

Factors Affecting Cost of Usability Testing:

The testing cost will depend on the following factors:
1. No. of participants for testing.
2. Number of Days which you need for testing.
3. which type of testing.
4. the size of the team used for testing.
Remember to budget for the usability testing, making usability testing into a product or any
website is an iterative process and the elements that are needed are as follows:
• Time: Time is an important factor with considering usability testing, it will use the
specialist of usability and the team to evolve the site and also need to test the test
scenarios. Be sure to budget in time for this test preparation as well as for running test
cases, report writing, analysing the data, and presenting the findings.
• Rental cost: If you are not considering the equipment, you will need to ensure the
budget cost for all other equipment, and also need to allot the location for the testing
purpose. For example the rental room like a conference room which is used to
perform all operations.
• Recruiting Costs: Consider how and where you have recruited your participants.
You will need to allow the staff to engage a recruiting team to schedule participants
based on requirements.
• Participants Compensation based on: You will need to compensate the participants
for their time and travel purposes that also important to finding the testing budget.

Techniques and Methods of Usability Testing:

24

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
There are various types of usability testing that when performed lead to efficient software.
But few of them which are the most widely used have been discussed here.
1. Guerilla Testing
It is a type of testing where testers wander to public places and ask random users about the
prototype. Also, a thank gift is offered to the users as a gesture of token. It is the best way to
perform usability testing during the early phases of the product development process. Users
mostly spare 5–10 minutes and give instant feedback on the product. Also, the cost is
comparatively low as you don’t need to hire participants. It is also known as corridor or
hallway testing. 2. Usability Lab
Usability lab testing is conducted in a lab environment where moderators (who ask for
feedback on the product) hire participants and ask them to take a survey on the product. This
test is performed on a tablet/desktop. The participant count can be 8-10 which is a bit costlier
than guerrilla testing as you need to hire participants, arrange a place, and conduct testing.
3. Screen or Video Recording
Screen or video recording kind of testing is in which a screen is recorded as per the user’s
action (navigation and usage of the product). This testing describes how the user’s mind runs
while using a product. This kind of testing involves the participation of almost 10 users for
15 minutes. It helps in describing the issues users may face while interacting with the
product.

Generally, there are two studies in usability testing –

1. Moderated – the Moderator guides the participant for the changes required in the
product (software)
2. Unmoderated – There’s no moderator (no human guidance), participants gets a set of
questions on which he/she has to work.
While performing usability testing, all kinds of biases (be it friendly bias, social bias,
etc.) by the participants are avoided to have honest feedback on the product so as to improve
its durability.

Advantages and Disadvantages of Usability Testing

Usability testing is preferred to evaluate a product or service by testing it with the proper
users. In Usability testing, the development and design teams will use to identify issues before
coding and the result will be earlier issues will be solved. During a Usability test, you can,
Learn if participants will be able to complete the specific task completely. identify how long
it will take to complete the specific task.
• Gives excellent features and functionalities to the product
• Improves user satisfaction and fulfils requirements based on user’s feedback
• The product becomes more efficient and effective
The biggest cons of usability testing are the cost and time. The more usability testing is
performed, the more cost and time is being used.

4.10 TESTING THE DOCUMENTATION

As a software tester, you typically aren’t constrained to just testing the software. Your
responsibility will likely cover all the parts that make up the entire software product.
Assuring that the documentation is correct is your job, too.

4.10.1 Type of Software Documentation:

25

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Here’s a list of software components that can be classified as documentation. Obviously, not
all software will have all the components, but it’s possible:
• Packaging text and graphics. This includes the box, carton, wrapping, and so on. The
documentation might contain screen shots from the software, lists of features, system
requirements, and copyright information.
• Marketing material, ads, and other inserts. These are all the pieces of paper you usually
throw away, but they are important tools used to promote the sale of related software, add-
on content, service contracts, and so on. The information for them must be correct for a
customer to take them seriously.
• Warranty/registration. This is the card that the customer fills out and sends in to register
the software. It can also be part of the software, being displayed onscreen for the user to
read, acknowledge, and complete online.
• EULA. Pronounced “you-la,” it stands for End User License Agreement. This is the legal
document that the customer agrees to that says, among other things, that he won’t copy the
software nor sue the manufacturer if he’s harmed by a bug. The EULA is sometimes printed
on the envelope containing the media—the floppy or CD. It also may pop up onscreen
during the software’s installation.
• Labels and stickers. These may appear on the media, on the box, or on the printed
material. There may also be serial number stickers and labels that seal the EULA envelope.
Figure 4.10 shows an example of a disk label and all the information that needs to be
checked.

26

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

27

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Figure 4.10 There’s lots of documentation on this disk label for the software tester to check.

• Installation and setup instructions. Sometimes this information is printed directly on the
discs, but it also can be included on the CD sleeve or as a CD jewel box insert. If it’s
complex software, there could be an entire installation manual.
• User’s manual. The usefulness and flexibility of online manuals has made printed manuals
much less common than they once were. Most software now comes with a small, concise
“getting started”–type manual with the detailed information moved to online format. The
online manuals can be distributed on the software’s media, on a website, or a combination
of both.
• Online help. Online help often gets intertwined with the user’s manual, sometimes even
replacing it. Online help is indexed and searchable, making it much easier for users to find
the information they’re looking for. Many online help systems allow natural language
queries so users can type Tell me how to copy text from one program to another and receive
an appropriate response.
• Tutorials, wizards, and CBT (Computer Based Training). These tools blend programming
code and written documentation. They’re often a mixture of both content and high-level,
macro-like programming and are often tied in with the online help system. A user can ask a
question and the software then guides him through the steps to complete the task.
• Samples, examples, and templates. An example of these would be a word processor with
forms or samples that a user can simply fill in to quickly create professional-looking results.
A compiler could have snippets of code that demonstrate how to use certain aspects of the
language.
• Error messages.

4.10.2 What to Look for When Reviewing Documentation:

Table 4.1 is a simple checklist to use as a basis for building your documentation test cases.

28

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

Table 4.1 A Documentation Testing Checklist

4.10.3 The Importance of Documentation Testing:
Good software documentation contributes to the product’s overall quality in three ways:
• It improves usability.
• It improves reliability.
• It lowers support costs. The reason is that users who are confused or run into unexpected
problems will call the company for help, which is expensive. Good documentation can
prevent these calls by adequately explaining and leading users through difficult areas.

29

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
4.11 SECURITY TESTING
Security testing is an integral part of software testing, which is used to discover the
weaknesses, risks, or threats in the software application and also help us to stop the nasty
attack from the outsiders and make sure the security of our software applications.
The primary objective of security testing is to find all the potential ambiguities and
vulnerabilities of the application so that the software does not stop working. If we perform
security testing, then it helps us to identify all the possible security threats and also help the
programmer to fix those errors.
It is a testing procedure, which is used to define that the data will be safe and also
continue the working process of the software.

4.11.1 Principle of Security testing:

Availability: In this, the data must be retained by an official person, and they also guarantee
that the data and statement services will be ready to use whenever we need it.
Integrity: In this, we will secure those data which have been changed by the unofficial
person. The primary objective of integrity is to permit the receiver to control the data that is
given by the system.
The integrity systems regularly use some of the similar fundamental approaches as
confidentiality structures. Still, they generally include the data for the communication to
create the source of an algorithmic check rather than encrypting all of the communication.
And also verify that correct data is conveyed from one application to another.
Authorization: It is the process of defining that a client is permitted to perform an action and
also receive the services. The example of authorization is Access control.

Confidentiality: It is a security process that protracts the leak of the data from the outsider's
because it is the only way where we can make sure the security of our data.
Authentication: The authentication process comprises confirming the individuality of a
person, tracing the source of a product that is necessary to allow access to the private
information or the system.

30

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

Non-repudiation: It is used as a reference to the digital security, and it a way of assurance

that the sender of a message cannot disagree with having sent the message and that the
recipient cannot repudiate having received the message.
The non-repudiation is used to ensure that a conveyed message has been sent and
received by the person who claims to have sent and received the message.

4.11.2 Key Areas in Security Testing:

While performing the security testing on the web application, we need to concentrate on the
following areas to test the application:

System software security: In this, we will evaluate the vulnerabilities of the application
based on different software such as Operating system, Database system, etc.
Network security: In this, we will check the weakness of the network structure, such as
policies and resources.
Server-side application security: We will do the server-side application security to ensure
that the server encryption and its tools are sufficient to protect the software from any
disturbance.
Client-side application security: In this, we will make sure that any intruders cannot
operate on any browser or any tool which is used by customers.

4.11.3 Types of Security testing:

As per Open Source Security Testing techniques, we have different types of security
testing which as follows:

Security Scanning: Security scanning can be done for both automation testing and manual
testing. This scanning will be used to find the vulnerability or unwanted file modification in a
web-based application, websites, network, or the file system. After that, it will deliver the
results which help us to decrease those threats. Security scanning is needed for those
systems, which depends on the structure they use.

31

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Risk Assessment: To moderate the risk of an application, we will go for risk assessment. In
this, we will explore the security risk, which can be detected in the association. The risk can
be further divided into three parts, and those are high, medium, and low. The primary
purpose of the risk assessment process is to assess the vulnerabilities and control the
significant threat.

Vulnerability Scanning: It is an application that is used to determine and generates a list of

all the systems which contain the desktops, servers, laptops, virtual machines, printers,
switches, and firewalls related to a network. The vulnerability scanning can be performed
over the automated application and also identifies those software and systems which have
acknowledged the security vulnerabilities.

Penetration testing: Penetration testing is a security implementation where a cyber-security

professional tries to identify and exploit the weakness in the computer system. The primary
objective of this testing is to simulate outbreaks and also finds the loophole in the system and
similarly save from the intruders who can take the benefits.

Security Auditing: Security auditing is a structured method for evaluating the security
measures of the organization. In this, we will do the inside review of the application and the
control system for the security faults.

Ethical hacking: Ethical hacking is used to discover the weakness in the system and also
helps the organization to fix those security loopholes before the nasty hacker exposes them.
The ethical hacking will help us to increase the security position of the association because
sometimes the ethical hackers use the same tricks, tools, and techniques that nasty hackers
will use, but with the approval of the official person.
The objective of ethical hacking is to enhance security and to protect the systems
from malicious users' attacks.

Posture Assessment: It is a combination of ethical hacking, risk assessments, and security

scanning, which helps us to display the complete security posture of an organization.

4.11.4 Security testing tools:

We have various security testing tools available in the market, which are as
follows: SonarQube, ZAP, Netsparker, Arachni, IronWASP

32

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
4.12 TESTING IN THE AGILE ENVIRONMENT
• Agile Testing is a type of software testing that follows the principles of agile software
development to test the software application.
• All members of the project team along with the special experts and testers are
involved in agile testing.
• Agile testing is not a separate phase and it is carried out with all the development
phases
i.e. requirements, design and coding, and test case generation. Agile testing takes place
simultaneously throughout the Development Life Cycle.
• Agile testers participate in the entire development life cycle along with development
team members and the testers help in building the software according to the customer
requirements and with better design and thus code becomes possible.
• Agile Testing has shorter time frames called iterations or loops. This methodology is
also called the delivery-driven approach because it provides a better prediction on the
workable products in less duration time.

4.12.1 Features of Agile Testing

Some of the key features of agile software testing are:
Simplistic approach: In agile testing, testers perform only the necessary tests but at the same
time do not leave behind any essential tests. This approach delivers a product that is simple
and provides value.
Continuous improvement: In agile testing, agile testers depend mainly on feedback and
self- learning for improvement and they perform their activities efficiently continuously.
Self-organized: Agile testers are highly efficient and tend to solve problems by bringing
teams together to resolve them.
Testers enjoy work: In agile testing, testers enjoy their work and thus will be able to deliver
a product with the greatest value to the consumer.
Encourage Constant communication: In agile testing, efficient communication channels
are set up with all the stakeholders of the project to reduce errors and miscommunications.
Constant feedback: Agile testers need to constantly provide feedback to the developers if
necessary.

4.12.2 Agile Testing Principles

Shortening feedback iteration: In Agile Testing, the testing team gets to know the product
development and its quality for each and every iteration. Thus continuous feedback
minimizes the feedback response time and the fixing cost is also reduced.
Testing is performed alongside: Agile testing is not a different phase. It is performed
alongside the development phase. It ensures that the features implemented during that
iteration are actually done. Testing is not kept pending for a later phase.
Involvement of all members: Agile testing involves each and every member of the
development team and the testing team. It includes various developers and experts.
Documentation is weightless: In place of global test documentation, agile testers use
reusable checklists to suggest tests and focus on the essence of the test rather than the
incidental details. Lightweight documentation tools are used.
Clean code: The defects that are detected are fixed within the same iteration. This ensures
clean code at any stage of development.
Constant response: Agile testing helps to deliver responses or feedback on an ongoing
basis. Thus, the product can meet the business needs.

33

Downloaded by Selva Jothi V

CCS366 - SOFTWARE TESTING AND AUTOMATION

34

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Customer satisfaction: In agile testing, customers are exposed to the product throughout the
development process. Throughout the development process, the customer can modify the
requirements, and update the requirements and the tests can also be changed as per the
changed requirements.
Test-driven: In agile testing, the testing needs to be conducted alongside the development
process to shorten the development time. But testing is implemented after the implementation
or when the software is developed in the traditional process.

4.12.3 Agile Testing Life Cycle

The agile testing life cycle has 5 different phases:
1. Impact Assessment: This is the first phase of the agile testing life cycle also known as
the feedback phase where the inputs and responses are collected from the users and
stakeholders. This phase supports the test engineers to set the objective for the next phase in
the cycle.
2. Agile Testing Planning: In this phase, the developers, customers, test engineers, and
stakeholders team up to plan the testing process schedules, regular meetings, and
deliverables.
3. Release Readiness: This is the third phase in the agile testing lifecycle where the test
engineers review the features which have been created entirely and test if the features are
ready to go live or not and the features that need to be sent again to the previous development
phase.
4. Daily Scrums: This phase involves the daily morning meetings to check on testing and
determine the objectives for the day. The goals are set daily to enable test engineers to
understand the status of testing.
5. Test Agility Review: This is the last phase of the agile testing lifecycle that includes
weekly meetings with the stakeholders to evaluate and assess the progress against the goals.

4.12.3 Agile Testing Strategies:

35

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

1. Iteration 0
It is the first stage of the testing process and the initial setup is performed in this stage. The
testing environment is set in this iteration.
• This stage involves executing the preliminary setup tasks such as finding people for
testing, preparing the usability testing lab, preparing resources, etc.
• The business case for the project, boundary situations, and project scope are verified.
• Important requirements and use cases are summarized.
• Initial project and cost valuation are planned.
• Risks are identified.
• Outline one or more candidate designs for the project.

2. Construction Iteration
It is the second phase of the testing process. It is the major phase of the testing and most of
the work is performed in this phase. It is a set of iterations to build an increment of the
solution. This process is divided into two types of testing:
Confirmatory testing: This type of testing concentrates on verifying that the system meets
the stakeholder’s requirements as described to the team to date and is performed by the team.
It is further divided into 2 types of testing:
Agile acceptance testing: It is the combination of acceptance testing and functional
testing. It can be executed by the development team and the stakeholders.
Developer testing: It is the combination of unit testing and integration testing and
verifies both the application code and database schema.
Investigative testing: Investigative testing detects the problems that are skipped or ignored
during confirmatory testing. In this type of testing, the tester determines the potential
problems in the form of defect stories. It focuses on issues like integration testing, load
testing, security testing, and stress testing.

3. Release End Game

This phase is also known as the transition phase. This phase includes the full system testing
and the acceptance testing. To finish the testing stage, the product is tested more relentlessly
while it is in construction iterations. In this phase, testers work on the defect stories. This
phase involves activities like:
• Training end-users.
• Support people and operational people.
• Marketing of the product release.
• Back-up and restoration.

36

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
• Finalization of the system and user documentation.

4. Production
It is the last phase of agile testing. The product is finalized in this stage after the removal of
all defects and issues raised.

4.12.4 Agile Testing Quadrants

The whole agile testing process is divided into four quadrants:

1. Quadrant 1 (Automated)
The first agile quadrat focuses on the internal quality of code which contains the test cases
and test components that are executed by the test engineers. All test cases are technology-
driven and used for automation testing. All through the agile first quadrant of testing, the
following testing can be executed:
• Unit testing.
• Component testing.

2. Quadrant 2 (Manual and Automated)

The second agile quadrant focuses on the customer requirements that are provided to the
testing team before and throughout the testing process. The test cases in this quadrant are
business- driven and are used for manual and automated functional testing. The following
testing will be executed in this quadrant:
• Pair testing.
• Testing scenarios and workflow.
• Testing user stories and experiences like prototypes.

37

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
3. Quadrant 3 (Manual)
The third agile quadrant provides feedback to the first and the second quadrant. This quadrant
involves executing many iterations of testing, these reviews and responses are then used to
strengthen the code. The test cases in this quadrant are developed to implement automation
testing. The testing that can be carried out in this quadrant are:
• Usability testing.
• Collaborative testing.
• User acceptance testing.
• Pair testing with customers.

4. Quadrant 4 (Tools)
The fourth agile quadrant focuses on the non-functional requirements of the product like
performance, security, stability, etc. Various types of testing are performed in this quadrant
to deliver non-functional qualities and the expected value. The testing activities that can be
performed in this quadrant are:
• Non-functional testing such as stress testing, load testing, performance testing, etc.
• Security testing.
• Scalability testing.
• Infrastructure testing.
• Data migration testing.

4.12.5 Challenges During Agile Testing:

Below are some of the challenges that are faced during agile testing:
Changing requirements: Sometimes during product development changes in the
requirements or the specifications occur but when they occur near the end of the sprint, the
changes are moved to the next sprint and thus become the overhead for developers and
testers. Inadequate test coverage: In agile testing, testers sometimes miss critical test cases
because of the continuously changing requirements and continuous integration. This problem
can be solved by keeping track of test coverage by analyzing the agile test metrics.
Tester’s availability: Sometimes the testers don’t have adequate skills to perform API and
Integration testing, which results in missing important test cases. One solution to this
problem is to provide training for the testers so that they can carry out essential tests
effectively. Less Documentation: In agile testing, there is less or no documentation which
makes the task of the QA team more tedious.
Performance Bottlenecks: Sometimes developer builds products without understanding the
end-user requirements and following only the specification requirements, resulting in
performance issues in the product. Using load testing tools performance bottlenecks can be
identified and fixed.
Early detection of defects: In agile testing, defects are detected at the production stage or at
the testing stage, which makes it very difficult to fix them.
Skipping essential tests: In agile testing, sometimes agile testers due to time constraints and
the complexity of the test cases put some of the non-functional tests on hold. This may cause
some bugs later that may be difficult to fix.

38

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
4.13 TESTING WEB AND MOBILE APPLICATIONS
4.13.1 Introduction:
Web Applications
Web applications refer to computer programs that run in a web browser. Commonly built
with the help of HTML5, CSS and JavaScript, web applications offer more interactivity than
websites and can be accessed via a desktop or laptop. The classic examples of web
applications include webmail, online stores and web banking.

Mobile Applications
A mobile application is a program that was built to be used on mobile devices (smartphones,
tablets and various wearables). Mobile apps are not as straightforward as desktop web apps
and fall into three varieties: mobile web, native and hybrid apps.
Mobile web applications
A mobile web application is a program that can be accessed via a mobile browser, meaning
that you don’t have to download them to your device to start using them. Like web apps,
mobile web applications are usually built using JavaScript, CSS and HTML5; however, there
is no standard software kit. Contrary to other mobile applications, web apps for mobile use
are easier to build and test, but they’re usually much more primitive in terms of functionality.
Native applications
Fast and more advanced when it comes to features, native mobile applications run on the
device itself, so you have to download them before using them. Since they are platform-
specific, native mobile apps are built using specific languages and integrated development
environments (IDEs). For example, Android native applications are developed using Java or
Kotlin and Android Studio or Eclipse IDE. At the same time, to build an app for an Apple
device, you’ll need to use Objective-C or Swift and the XCode IDE. Native apps are secure,
integrate with the hardware perfectly and have the best UI/UX experience.
Hybrid applications
Hybrid apps combine the characteristics of native and mobile web apps. Built with the help
of the “standard web” stack (JavaScript, CSS and HTML5), they are then wrapped in a native
environment, so you can use the same code for different platforms. While running on your
mobile browser, hybrid applications are downloadable and have access to your camera, GPS,
contact list, etc. Though such applications are easier to build and maintain, they are slower
and offer less advanced functionality than their native counterparts.

4.13.2 Types of Mobile App Testing vs. Web App Testing

Whether it comes to testing web or mobile applications, the aim is to ensure that an app is
user- friendly and functions properly under different circumstances, even if it’s already in
production. Furthermore, both application testing varieties include common types:
• Functional testing
• User-interface testing
• Usability testing
• Configuration and Compatibility testing
• Security testing
• Performance testing
• Database testing

39

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
1. Functional Testing
Functional testing involves checking of the specified functionality of a web
application. Functional test cases for web applications may be generated using boundary
value analysis, equivalence class testing, decision table testing and many other techniques.
Example:
Let us consider the eCommerce application sells products such as computers, mobile
phones, cameras, electronics, etc. The home page of this web application is given in Figure
4.13.

Figure 4.13 Homepage of online shopping web application

Table 4.13 presents some sample functional test cases of the order process form of an
online shopping website.

40

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

Table 4.13. Sample functional test cases of order process of an online shopping web
application
2. User-interface Testing
User interface testing tests that the user interaction features work correctly. These
features include hyperlinks, tables, forms, frames and user interface items such as text fields,
radio buttons, check boxes, list boxes, combo boxes, command buttons and dialog boxes.
User interface testing ensures that the application handles mouse and keyboard events
correctly and displays hyperlinks, tables, frames, buttons, menus, dialog boxes, error
message boxes, and toolbars properly.
2.1. Navigation Testing
Navigation testing investigates the proper functioning of all the internal and external links.
Navigation testing must ensure that websites provide consistent, well-organized links and
should also provide alternative navigation schemes such as search options and site maps. The
placement of navigation links on each page must be checked. Search based navigation
facility must also be thoroughly tested and search items should be consistent across one page
to another. All the combinations of keywords and search criteria must be verified in
navigation testing. Table 4.14 presents test cases for navigation testing for an online
shopping website as given in Figure 4.13.

41

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

Table 4.14 Navigation testing test cases for online shopping website
Manual checking of hyperlinks can be very time consuming. There are various online
tools available for checking broken links, accuracy and availability of links and obtaining
advice on search engines. Some tools for navigation testing include Performance
Technologies’ TestLink, W3C’s Link checker, Xenu’s LinkSleuth, Dead Links’ Dead Links,
LinkTiger’s LinkTiger, Viable Software Alternative’s LinkRunner, Elsop’s LinkScan, REl
Software’s Link Validator, UCI’s MQMspider and Illumit’s WebLight.

2.2 Form Based Testing

Websites that include forms need to ensure that all the fields in the form are working
properly. Form-based testing involves the following issues:
1. Proper navigation from one field of the form to another using the tab key.
2. Ensures that the data entered in the form is in a valid format.
3. Checks that all the mandatory fields are entered in the form.
Consider the registration form of an online shopping website (web page is given in Figure
4.13). Its inputs include login id, password, repeat password, name, last name, email address,
phone number, shipping details (address, city, state, zip code) and billing details (address,
city, state, zip code). Its form based test cases are shown in Table 4.15.

42

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

Table 4.15 Test cases of registration form of an online shopping web application

3. Usability Testing
Usability testing refers to the procedure employed to evaluate the degree to which the software
satisfies the specified usability criteria.
4. Configuration and Compatibility Testing
One of the significant challenges of web testing is that it must ensure the proper
functioning of a web application on all the supported platforms and suitable environments.
Configuration testing determines the behaviour of the software with respect to various
configurations whereas compatibility testing determines whether the web application behaves
as expected with respect to various supported configurations.
5. Security Testing
Security is the procedure used to protect information from various threats. It is very
important to protect sensitive and critical information and data while communicating over the
network. The user wants implementation of a safeguard to protect personal, sensitive and
financial information. We want data to be accurate, reliable and protected against
unauthorized access.
Security involves various threats such as unauthorized users, malicious users, message
sent to an unintended user, etc. The primary requirement of security includes:
i. Authentication: Is the information sent from an authenticated user?
ii. Access Control: Is data protected from unauthorized users?
iii. Integrity: Does the user receive exactly what is sent?
iv. Delivery: Is the information delivered to the intended user?
v. Reliability: What is the frequency of a failure? How much time does the
network take to recover from a failure? What measures are taken to counter
catastrophic failure? vi. Non-repudiation: Is the receiver able to prove that the data
received came from a specific sender?

43

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
6. Performance Testing The goal of performance testing is to evaluate the application’s
performance with respect to real world scenarios. The following issues must be addressed
during performance testing:
i. Performance of the system during peak hours (response time, reliability and
availability).
ii. Points at which the system performance degrades or system fails. iii. Impact
of the degraded performance on the customer loyalty, sales and profits.
6.1 Load Testing
Load testing involves testing the web application under real world scenarios by
simulating numerous users accessing the web application simultaneously. It tests the web
application by providing it maximum load.
6.2 Stress Testing
Stress testing involves execution of a web application with more than maximum and
varying loads for long periods. Unlike performance and load testing, stress testing evaluates
the response of the system when the system is given a load beyond its specified limits. The
behaviour of the system is monitored to determine when the system under stress test fails and
how does it recover from the failure.

7. Database Testing
In web applications, many applications are database driven, for example, e-commerce
related websites or business-to-business applications. It is important for these applications to
work properly and provide security to the user’s sensitive data such as personal details and
credit card information. Testing data-centric web applications is important to ensure their
error- free operation and increased customer satisfaction.
For example, consider the example for purchasing items from an online store. If the
user performs a search based on some keywords and price preferences, a database query is
created by the database server. Suppose due to some programming fault in the query, the
query does not consider the price preferences given by the customer, this will produce
erroneous results. These kinds of faults must be tested and removed during database testing.

44

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION

Table 4.16 shows sample test cases based on a user operation in an online shopping
website.

4.13.3 Difference between Mobile App and Web App Testing:

The divergence between mobile app testing and web app testing is more discernible
in terms of user interaction, Internet connection and compatibility.
User Interaction
From the viewpoint of interaction with users, most desktop browser-based applications are
limited to a mouse and a keyboard, and all the operations can be performed by clicking or
hitting a particular key. Mobile apps, on the other hand, offer a wider scope of options in this
respect. While all the tapping, swiping, pulling and pinching is not as challenging to test,
there are voice assistants and move commanders to be included in your testing checklist.

Internet Connection
The majority of web applications don’t function offline, and the same rule applies to mobile
web applications. To this end, when dealing with this type of apps, you should test if they
perform well under different speeds of the Internet connection. When it comes to native and
some hybrid mobile apps, it is critical to verify if an app works properly in the offline mode,
how it reacts to interrupted connection and how it functions with 3/4/5 G or WI-FI.

Compatibility
Web applications should be tested against different browsers, which also applies to their
mobile web and hybrid counterparts. From the device compatibility viewpoint, web-based
applications are usually more simple to test due to the functionality of a desktop. With
mobile apps, the testing procedure is much more complicated because of the wide variety of
mobile devices, which expand their functionality at an exponential rate. That is why, when
checking

45

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
mobile applications for compatibility, it is critical to pay attention to the technical
characteristics of each mobile device and how they affect the behavior of your app. The most
important of these characteristics are:
Capacity limitations
Note that RAM and SSD are absolutely different for web and mobile. While some mobile
devices can ship 3-4GB of RAM, the capacity of even the best smartphone falls behind that
of an average desktop. To this effect, if your app isn’t properly optimized for mobile use, it
will function much slower than its desktop fellow. Furthermore, since mobile apps are
updated more frequently than their web counterparts, it is critical to test how your app affects
memory and whether it resumes proper work after a number of updates and installations.
Battery life
Desktop computers don’t have problems like this, but a smartphone battery that’s running
low has become an actual nightmare for thousands of people. Even when you’re using a
laptop, having a dead computer is not as bad as having a dead mobile phone. An application
that drains battery life will be deleted soon. So, it is paramount for your QA team to check
the performance of a mobile application on a fully-charged and a low-charged device.
Screen size
While desktops are limited to a few screen dimensions, mobile devices offer much greater
variety in this respect. It is important to examine specific mobile screen dimensions and
make sure that all the app fields are visible and easy to access.

4.13.4 Different Tools for Mobile vs. Web App Testing

The reasons for choosing the right tool for test automation are rather obvious: a higher level
of test coverage and parallel testing, better reliability and faster test execution.

Tools for Web App Testing

To be able to work without a glitch under continuous, high load and growing market
expectations, web apps need to undergo a plethora of tests to ensure compliance with the UI
standards, compatibility and usability.

Certainly, there’re a lot more commercial and open-source tools, but Selenium is still the most
popular open-source tool for automated testing against web apps and websites.

Selenium
Selenium is a powerful open-source automated testing framework consists of Selenium IDE,
Selenium WebDriver and Selenium Grid. Selenium supports multiple programming
languages for script creation, allows users to record and re-run saved scripts and works well
on different browsers and operating systems. While it’s not particularly self-sufficient,
Selenium can be easily integrated with other tools for test management, image-based testing,
performing load tests and efficient functioning in an agile working environment.

To execute Selenium WebDriver scripts, you need to ensure that you have the initial
configuration done. The WebDriver consists of three separate parts, take Chrome as an
example, the browser itself (“i.e. Chrome“), the language bindings provided (“the Driver”)
and an executable ChromeDriver downloaded from the Chromium project which acts as a
bridge between “chrome” and the “driver”. All of them is discussed here.

46

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Tools for Mobile App Testing
While the tooling world for web app testing looks quite simple, as Selenium is widely used by
web developers and testers, the tools used for mobile app testing is not as straightforward.

First of all, not all web application QA tools are applicable to mobile app testing.
Secondly, when choosing a tool or framework for your mobile app, consider if it is cross-
platform or designed for a particular OS.
Finally, not every mobile app testing tool is best for all application types: native, web mobile
and hybrid.
But don’t let the number of tools and frameworks scare you away from mobile app test
automation. We’ve gathered the best of them right here.

Appium
If you need a black-box mobile app testing tool, look no further than Appium. Based on
Selenium, it is an open-source tool for testing hybrid, web and native Android and iOS
mobile applications. Offering the Selenium set of features, Appium runs tests on real devices,
emulators and simulators.

It allows you to create test cases on a variety of programming languages, including Java, C#,
Python, PHP, etc., using the same API. Still, you’ll need to create separate iOS and Android
scripts to run UI tests. Like Selenium, Appium is a flexible tool and can be easily integrated
with other frameworks. Learn all tips and tricks about Appium here.

Espresso
Espresso is a UI quality assurance framework designed by Google for white box testing.
Since it was created to test Android native applications, Espresso tests can be written in Java
and Kotlin, the programming languages used to develop Android applications.

Due to the simple API, it easily adapts to your needs without the complexity of multi-
threaded testing. But the most outstanding feature of Espresso is that it provides automatic
synchronization between tests and applications (the test doesn’t continue until the application
is ready).
XCUITest / XCTest
For iOS functional and UI testing, XCUITest and XCTest are getting their popularity among
iOS developers. The reason being, they run tests inside the XCode IDE and testers can write
tests in either Objective-C and Swift.

XCUITest and XCTest can execute iOS tests much faster than other cross-platform
frameworks. Their tests are usually more reliable and trustable. However, it is because they
are native to iOS development and testing that keeps away many mobile developers that are
more looking into the flexibility of the used tools.

Calabash
Although the team behind it has stopped developing it, Calabash was once an open-source
automated QA tool for running functional and acceptance tests of native and hybrid
applications.

47

Downloaded by Selva Jothi V

 CCS366 - SOFTWARE TESTING AND AUTOMATION
Users can run simultaneous tests against multiple devices in parallel with Calabash, and it
provides an API for testing native applications on touch screen devices. The only minor
setback is that the test scripts have to be written in Ruby only.

Selenium Appium Espresso XCUITest Calabash

XCTest
Platform Desktop Both Android & Android iOS Both Android
type browsers iOS & iOS
App type Web Native, web and Native, web Native, web Native and
hybrid and hybrid and hybrid hybrid
Areas Functional, Functional, UI UI Functional, UI
to test regression regression, UI acceptance
Scripting Java, C#, Java, C#, Java, Kotlin Swift, Ruby
language Perl, Python, Python, PHP, Objective-C
JavaScript, Ruby, JavaScript
Ruby, PHP
CI Popular Popular CI tools Popular Popular Popular
CI tools (e.g., Jenkins, CI tools CI tools CI tools
(e.g., Teamcity) (e.g., (e.g., (e.g.,
Jenkins, Jenkins, Jenkins, Jenkins,
Teamcity) Teamcity) Teamcity) Teamcity)
License Open-source Open-source Open-source Open-source Open-source
type

48

Downloaded by Selva Jothi V