UNIT-V

TOPIC-1

Test Management and Automation Test Planning

Test management, process of managing the tests. A test management is also performed using
tools to manage both types of tests, automated and manual, that have been previously specified
by a test procedure.

Test management tools allow automatic generation of the requirement test matrix (RTM), which
is an indication of functional coverage of the application under test (SUT).

Test Management tool often has multifunctional capabilities such as testware management, test
scheduling, the logging of results, test tracking, incident management and test reporting.

Test Management Responsibilities:

 Test Management has a clear set of roles and responsibilities for improving the quality of the product.

 Test management helps the development and maintenance of product metrics during the course of
project.

 Test management enables developers to make sure that there are fewer design or coding faults.
Automation Testing Test Plan Sections:

Automated testing or test automation is a method in software testing that makes use of special software
tools to control the execution of tests and then compares actual test results with predicted or expected
results.

1. Choose the test cases/scenarios that are to be regressed over and over across multiple
cycles.
2. Sometimes the simplest of test cases need lots of complicated solutions to be automated.
...
3. Automation Testing does not/cannot perform exploratory testing.
Use these top tips to ensure that your software testing is successful and you get the maximum
return on investment (ROI):
 1.Decide what Test Cases to Automate
2.Test Early and Test Often
3.Select the Right Automated Testing Tool
4. Divide your Automated Testing Efforts
5. Create Good, Quality Test Data
6. Create Automated Tests that are Resistant to Changes in the UI

Test tool selection

Test Tool selection largely depends on the technology the Application Under Test is built on.
For instance, QTP does not support Informatica. So QTP cannot be used for
testing Informatica applications. It's a good idea to conduct Proof of Concept of Tool on
AUT.

Define the scope of Automation

The scope of automation is the area of your Application Under Test which will be automated.
Following points help determine scope:

 The features that are important for the business

 Scenarios which have a large amount of data
  Common functionalities across applications
 Technical feasibility
 The extent to which business components are reused
 The complexity of test cases
 Ability to use the same test cases for cross-browser testing

Planning, Design, and Development

During this phase, you create Automation strategy & plan, which contains following details-

 Automation tools selected

 Framework design and its features
 In-Scope and Out-of-scope items of automation
 Automation testbed preparation
 Schedule and Timeline of scripting and execution
 Deliverables of Automation Testing

Test Execution
Automation Scripts are executed during this phase. The scripts need input test data before
there are set to run. Once executed they provide detailed test reports.

Execution can be performed using the automation tool directly or through the Test
Management tool which will invoke the automation tool.

Example: Quality center is the Test Management tool which in turn it will invoke QTP for
execution of automation scripts. Scripts can be executed in a single machine or a group of
machines. The execution can be done during the night, to save time.

Maintenance
As new functionalities are added to the System Under Test with successive cycles,
Automation Scripts need to be added, reviewed and maintained for each release
cycle. Maintenance becomes necessary to improve the effectiveness of Automation
Scripts.
 Automation Framework:
Automation Framework is not a single tool or process, but it is a collection of tools and processes
working together to support automated testing of any application. It integrates various functions
like libraries, test data, and various reusable modules.

Importance of Automation Framework in Software Testing

Framework supports automation testing as a technical implementation guideline. For example,
consider a scenario where a testing team includes members who are based on different
automation testing code. And, they are not able to grasp the common pieces of code and scripts
updated by a team member in a project.
The automation framework not only offers the benefit of reusing the code in various scenarios,
but it also helps the team to write down the test script in a standard format. Hence, the test
automation framework handles all the issues. Besides, there are many other benefits of using
automation framework testing as listed below:
 Maintain a well-defined strategy across the test suites
 Enhanced speed at which testing progresses
 Maintaining the test code will be easy
 The URL or Application can be tested accurately
 Continuous testing of coding and delivery will be achieved
Test automation framework is helpful when you need to execute the same test scripts multiple
times with different builds to examine the application and validate output. It is better to avoid
automated testing for functionality, which you used only once since building automation script
itself is time-consuming.
Test Automation Frameworks Type:
Each test automation framework has its own architecture, advantages, and disadvantages. Some
of these test automation frameworks are:
 Linear Automation Framework
 Modular Driven Framework
 Behavior Driven framework
 Data-Driven Framework
 Keyword-Driven Framework
 Hybrid Testing Framework
1) Linear Automation Framework
The linear Automation framework is commonly used in the testing of small applications. This
framework is also called as a Record and playback framework.
Pros: There is no need to write custom code, so expertise in test automation is not necessary.
Cons: The data is hardcoded in the test script; hence, the test cases cannot be re-run with multiple
sets. You need to make some changes if the data is altered.

2) Modular Driven Framework

In this Framework, the tester can create test scripts module wise by breaking down the whole
application into smaller modules as per the client requirements and create test scripts
individually.
Pros: Modular driven framework ensures the division of scripts that leads to easier maintenance
and scalability. You can write independent test scripts.
Cons: The modular driven framework requires additional time in analyzing the test cases and
identifying reusable flows.

3) Behavior Driven Development Framework

Behavior Driven Development framework is to create a platform, which allows every person,
like Developers, Testers, business analyst, etc., to participate actively. It also increases
collaboration between the tester and the developers on your project.
Pros: You can use non-technical, natural language to create test specifications on this behavior-
driven testing.
Cons: To work with this framework, sufficient technical skills as well as prior experience in Test
driven development is required.
4) Data-driven Testing Framework
Generally, Test Data is read from the external files like Excel Files, Text Files, CSV Files,
ODBC Sources, DAO Objects and they are loaded into the variables inside the Test Script. The
data-driven framework allows us to create test automation scripts by passing different sets of test
data.
Pros: It reduces the number of scripts required. Hence, multiple scenarios can be tested in less
code.
Cons: You will need a highly experienced tester who should be proficient in various
programming languages to completely utilize the design of this framework.
5) The Keyword-Driven Testing Framework
The keyword-Driven Testing framework is also known as table-driven testing. This framework is
suitable only for small projects or applications. The automation test scripts performed are based
on the keywords specified in the excel sheet of the project.
Pros: A single keyword can be used across multiple test scripts, so the code is reusable.
Cons: The initial cost of setting up the framework is high, and it is time-consuming & complex.
6) The Hybrid test Automation Framework
Hybrid Framework is used to combine the benefits of Keyword Driven and Data-Driven
frameworks.
Pros: This type leverages the advantages of all kinds of related frameworks.
Cons: Tests are fully scripted in a Hybrid Testing Framework thus increases the automation
effort.
All of these test automation frameworks can be effectively used to handle the code in a
systematic way, which can be reviewed by a third person easily. You can choose the framework
as per your project requirements, team expertise, time as well as budget. Test automation
frameworks boost productivity with standardization. By adopting the framework, you can ensure
maximum test coverage in your development process.

Automation Tool Best Practices

To get maximum ROI of automation, observe the following

 The scope of Automation needs to be determined in detail before the start of the
project. This sets expectations from Automation right.
 Select the right automation tool: A tool must not be selected based on its popularity,
but it's fit to the automation requirements.
 Choose appropriate framework
 Scripting Standards- Standards have to be followed while writing the scripts for
Automation. Some of them are-
o Create uniform scripts, comments, and indentation of the code
o Adequate Exception handling - How error is handled on system failure or
unexpected behavior of the application.
o User-defined messages should be coded or standardized for Error Logging for
testers to understand.
 Measure metrics- Success of automation cannot be determined by comparing the
manual effort with the automation effort but by also capturing the following metrics.
o Percent of defects found
o The time required for automation testing for each and every release cycle
o Minimal Time is taken for release
o Customer Satisfaction Index
 o Productivity improvement

The above guidelines if observed can greatly help in making your automation successful.

Benefits of Automation Testing

Following are benefits of automated testing:

 70% faster than the manual testing

 Wider test coverage of application features
 Reliable in results
 Ensure Consistency
 Saves Time and Cost
 Improves accuracy
  Human Intervention is not required while execution
 Increases Efficiency
 Better speed in executing tests
 Re-usable test scripts
 Test Frequently and thoroughly
 More cycle of execution can be achieved through automation
 Early time to market

Different types of software testing that can be

automated
 Smoke Testing
 Unit Testing
 Integration Testing
 Functional Testing
 Keyword Testing
 Regression Testing
 Data Driven Testing
 Black Box Testing

How to Choose an Automation Tool?

Selecting the right tool can be a tricky task. Following criterion will help you select the best
tool for your requirement-

 Environment Support
 Ease of use
 Testing of Database
 Object identification
 Image Testing
 Error Recovery Testing
  Object Mapping
 Scripting Language Used
 Support for various types of test - including functional, test management, mobile,
etc...
 Support for multiple testing frameworks
 Easy to debug the automation software scripts
 Ability to recognize objects in any environment
 Extensive test reports and results
 Minimize training cost of selected tools

Tool selection is one of biggest challenges to be tackled before going for automation. First,
Identify the requirements, explore various tools and its capabilities, set the expectation from
the tool and go for a Proof Of Concept.

Automation Testing Tools

There are tons of Functional and Regression Testing Tools available in market. Here are 5
best tools certified by our experts

1. Tricentis
Tricentis is the industry’s #1 Automation Testing platform and is recognized for reinventing
software testing for DevOps. Tricentis is the only vendor to achieve “leader” status in all
three top analyst reports.

Features:

 Risk-based testing
 Scriptless end-to-end test automation
 Supports API, BI, Mobile, Cross Browser, RPA and other automation testing types
 The industry’s most extensive technical support

Download Tricentis
 2. mabl
mabl delivers scriptless end-to-end test automation, integrated with your delivery pipeline, so
you can focus on improving your app.

Features:

 Proprietary machine learning models automatically identify and surface application

issues
 Tests are automatically repaired when UI changes
 Automated regression insights on every build

Learn more about mabl

3. TestIM
Testim.io leverages artificial intelligence for the authoring, execution and maintenance of
automated test cases.

Features:

 Proprietary technology learns and adapts automated tests to code changes, reducing
90% of maintenance time and 70% of authoring time.
 Helps companies accelerate time-to-market and achieve unparalleled quality at a
fraction of time of conventional solutions.

Learn more about TestIM

4. Selenium
It is a software testing tool used for Regression Testing. It is an open source testing tool that
provides playback and recording facility for Regression Testing. The Selenium IDE only
supports Mozilla Firefox web browser.

 It provides the provision to export recorded script in other languages like Java, Ruby,
RSpec, Python, C#, etc
 It can be used with frameworks like JUnit and TestNG
 It can execute multiple tests at a time
 Auto complete for Selenium commands that are common
 Walkthrough tests
  Identifies the element using id, name, X-path, etc.
 Store tests as Ruby Script, HTML, and any other format
 It provides an option to assert the title for every page
 It supports selenium user-extensions.js file
 It allows to insert comments in the middle of the script for better understanding and
debugging

Download Selenium

5. QTP (Micro Focus UFT)

QTP is widely used for functional and regression testing, it addresses every major software
application and environment. To simplify test creation and maintenance, it uses the concept
of keyword driven testing. It allows the tester to build test cases directly from the application.

 It is easier to use for a non-technical person to adapt to and create working test cases
 It fix defects faster by thoroughly documenting and replicating defects for developer
 Collapse test creation and test documentation at a single site
 Parameterization is easy than Win Runner
 QTP supports .NET development environment
 It has better object identification mechanism
 It can enhance existing QTP scripts without "Application Under Test" being
available, by using the Active Screen

6. Rational Functional Tester

It is an Object-Oriented automated Functional Testing tool that is capable of performing
automated functional, regression, data-driven testing and GUI testing. The main features of
this tool are

 It supports a wide range of protocols and applications like Java, HTML, NET,
Windows, SAP, Visual Basic, etc.
 It can record and replay the actions on demand
 It integrates well with source control management tools such as Rational Clear Case
and Rational Team Concert integration
  It allows developers to create keyword associated script so that it can be re-used
 Eclipse Java Developer Toolkit editor facilitates the team to code test scripts in Java
with Eclipse
 It supports custom controls through proxy SDK (Java/.Net)
 It supports version control to enable parallel development of test scripts and
concurrent usage by geographically distributed team

TEST MANAGEMENT:

Test management most commonly refers to the activity of managing the computer software
testing process. A test management tool is software used to manage tests(automated or manual) that
have been previously specified by a test procedure.
 The activities of testing can be divided into the following basic steps:
 Planning and Control.
 Analysis and Design.
 Implementation and Execution.
 Evaluating exit criteria and Reporting.
 Test Closure activities.

1) Planning and Control

Test Planning : Test planning involves producing a document that describes an overall approach
and test objectives. It involves reviewing the test basis, identifying the test conditions based on
analysis of test items, writing test cases and Designing the test environment. Completion or exit
criteria must be specified so that we know when testing (at any stage) is complete.

2) Analysis and Design

Test analysis and Test Design has the following major tasks:
 To review the test basis. The test basis is the information on which test cases are based,
such as requirements, design specifications, product risk analysis, architecture and interfaces
 To identify test conditions
 To design the tests
 To design the test environment set-up and identify the required infrastructure and tools

3) Implementation and Execution

Test execution involves actually running the specified test on a computer system either manually or by using
an automated test tool.It is a Fundamental Test Process in which actual work is done.
Test implementation has the following major task:
 To develop and prioritize test cases by using techniques and create test data for those tests.
 To create test suites from the test cases for efficient test execution.Test suite is a collection of test
cases that are used to test a software program
 To re-execute the tests that previously failed in order to confirm a fix.
 To log the outcome of the test execution. A test log is the status of the test case (pass/fail).
 To Compare actual results with expected result.

4) Evaluating Exit criteria and Reporting

Evaluating exit criteria is a process defining when to stop testing. It depends on coverage of code, functionality
or risk. Basically it also depends on business risk, cost and time and vary from project to project. Exit criteria
come into picture, when:
 Maximum test cases are executed with certain pass percentage
 Bug rate falls below certain level
 When we achieve the deadlines
Evaluating exit criteria has the following major tasks:
 To assess if more test are needed or if the exit criteria specified should be changed
 To write a test summary report for stakeholders

5) Test Closure activities:

Test closure activities are done when software is ready to be delivered. The testing can be closed for the other
reasons also like:
 When a project is cancelled
 When some target is achieved
 When a maintenance release or update is done
Test closure activities have the following major tasks:
 To check which planned deliverables are actually delivered and to ensure that all incident reports have
been resolved
 To finalize and archive testware such as scripts, test environments, etc. for later reuse
 To handover the testware to the maintenance organization. They will give support to the software
 To evaluate how the testing went and learn lessons for future releases and projects

Execution and Reporting:

Test Execution:
Test execution is the process of executing the code and comparing the expected and actual
results. Following factors are to be considered for a test execution process:

 Based on a risk, select a subset of test suite to be executed for this cycle.

 Assign the test cases in each test suite to testers for execution.

 Execute tests, report bugs, and capture test status continuously.

 Resolve blocking issues as they arise.

  Report status, adjust assignments, and reconsider plans and priorities daily.

Manner of executing and test the actual system result against the expected result is test
execution. Test execution can be done manually and by using automation suit. During the
execution tester needs to make sure, that the user’s need of the software is occupied in the
software. Test execution is conducted by referring the document created during test design as
step by step process. Tester needs to keep the track while executing the test cases.

Example for static testing:

 Test the requirement specification document.

 Test the design document
 Test the user guide

Example for dynamic testing:

 Unit testing
 Functional testing
 Integration testing

Test Reporting:
Test reporting gives the picture of test process and result for the particular testing cycle. To
define the element in the test reporting the first thing that needs to be considered is whom the
audiences of the test report are. For an example a project manager will like to see the high level
picture of the testing, intermediate people will wish to view more detail and the client will expect
the test reporting in the criteria such as requirement basis, feature basis. Test report is prepared
and communicated periodically like daily, weekly, month etc. This needs to be sent in different
stages and time.In the future project result of test reports needs to be analysed and apply the
lesson learns. Test report contain elements such as test execution status, completed percentage,
plan vs. executed test cases, test environment, test execution by resources, risk and mitigation if
any, defect summary, test scenario and conditions, any assumption, any note etc.

Test coverage report: (Elements of test coverage report)

 Percentage completed
  Test scenario

 Software area

 Tested resource

 Tested date

 Test result

Defect summary report: (Elements of defect summary report)

 Defect by severity
 Defects by priority

 Defects by assigned developer

 Defects by function

 Defects by software area

 Open and closed defects

Risk and mitigation report: (Elements of risk and mitigation report)

 Identified risk
 Likelihood

 Risk level

 Risk type

 Mitigation plan

Scope of automation:
1. Automated tools are used to facilitate generally all testing designs and processes. ...

2. A QA team keen on test design and execution develops the automated

testing framework.

3. The extent to which automation is used in the testing process defines the scope of
automated testing

4. It is required for the successful execution of testing process for a project. It contains
comprehensive information to carry out the testing activities.

5. A software test plan document is divided into different sections such as introduction,
objectives, scope, test items, features to be tested, and environmental needs.
 6. Automation is the process of evaluating the AUT(Application under Test) against the
specification with the help of a tool. In this article we are gong to discuss theScope of
automation in testing.

7. Depending on the nature of testing there are two main branches under automation.
Functional testing with automation.

8. A TEST PLAN is a document describing software testing scope and activities.

9. It is the basis for formally testing any software/product in a project. ISTQB

Definition.test plan:

10. A document describing the scope, approach, resources and schedule of

intended test activities.

A Software Test Plan is a document describing the testing scope,approach,

resources, schedule, deliverable, communication, entry and exit criteria. It is the basis of
formally testing any software/product in a project. Test plans can be of different levels
and types, depending upon the scope of testing.

Test tool selection.

Testing Tools:
Tools from a software testing context can be defined as a product that supports one or more test
activities right from planning, requirements, creating a build, test execution, defect logging and
test analysis.
Classification of Tools
Tools can be classified based on several parameters. They include:

 The purpose of the tool

 The Activities that are supported within the tool

 The Type/level of testing it supports

 The Kind of licensing (open source, freeware, commercial)

 The technology used

Types of Tools:
S.No. Tool Type Used for Used by

1. Test Management Tool Test Managing, scheduling, defect logging, testers

tracking and analysis.

2. Configuration management For Implementation, execution, tracking All Team

tool changes members

3. Static Analysis Tools Static Testing Developers

4. Test data Preparation Tools Analysis and Design, Test data generation Testers

5. Test Execution Tools Implementation, Execution Testers

6. Test Comparators Comparing expected and actual results All Team

members

7. Coverage measurement Provides structural coverage Developers

tools

8. Performance Testing tools Monitoring the performance, response time Testers

9. Project planning and For Planning Project

 Tracking Tools Managers

10. Incident Management For managing the tests Testers

Tools

Tools Implementation - process

 Analyse the problem carefully to identify strengths, weaknesses and opportunities

 The Constraints such as budgets, time and other requirements are noted.

 Evaluating the options and Shortlisting the ones that are meets the requirement

 Developing the Proof of Concept which captures the pros and cons

 Create a Pilot Project using the selected tool within a specified team

 Rolling out the tool phase wise across the organization

Simple Steps to Select the Right Test Automation tool

Step 1: Understand your project requirements thoroughly. ...
Step 2: Consider your existing test automation tool as a benchmark. ...
Step 3: Identify the key criteria suitable for a project. ...
Step 4: Leverage Pugh Matrix Technique for Analysis.

How to Select Best Automation Testing

Tool?
In the article Test Process Improvement, you learned the improvement points
from the project HDFC Bank. It is using automated Testing instead
of Manual Testing. But team members may ask you –

Answer to this question is challenging because there are many factors to

consider
  You want to support your test activities by means of a software tool,
but you don’t know tools currently available in market
 Which type of tool will best fit the requirement and the project budget?
 Who on the team has the skills to use the tool once you have purchased
it

The Importance of the Software Testing Tool

Selection
Success in any test automation depends on identifying the right tool for
automation. Selecting the “correct” Testing Tool for your project is one of the
best ways to achieve the project target. The following example will show you
the benefit of the testing tool selection

In the project HDFC Bank, to save the effort of testing, the project team
decided to use an automated testing tool for the test execution. After many
meetings, your team selected a suitable tool for the project.

One month later, you got the report from the project team about this tool
The results are great. The new automated tool doubled the testing
productivity. It means we saved 50% cost of test execution. If you’re
interested in exploring more options for your projects, check out this list
of automated testing tools.

This is an example of the benefit of using the testing tool in the project.
Selecting the right testing tool help you to improve the project productivity
and save project cost.

Type of Test Tools

There’re many types of test tool, which Test Manager can consider when
selecting the test tools.

Open-Source Tools
Open source tools are the program wherein the source code
is openly published for use and/or modification from its original design, free
of charge.

Open-source tools are available for almost any phase of the testing process,
from Test Case management to Defect tracking. Compared to commercial
tools Open source tools may have fewer features.
Commercial Tools
Commercial tools are the software which are produced for sale or to serve
commercial purposes.

Commercial tools have more support and more features from a vendor than
open-source tools.

Custom Tools
In some Testing project, the testing environment, and the testing process has
special characteristics. No open-source or commercial tool can meet the
requirement. Therefore, the Test Manager has to consider the development of
the custom tool.

Example: You want to find a Testing tool for the project HDFC Bank. You
want this tool to meet some specific requirement of the project.

Automation Feasibility Analysis

Back to the above example, the project team decided to develop a custom
tool which can meet project requirements. Suppose they have been given 100
test cases to automate and they estimated 5 days to develop a tool which can
automate all of those test cases.

Here is the result of their work

As in above scenario, the issue is that the test tool cannot automate all the test
cases of test specification. It means that not all application features can
be thoroughly tested using the test tool.

If the functionality of application under test changes frequently or is

too complicated, it is difficult to create test automation for all the application
features, because every tool has its own limitations.

If you don’t want to be in such situation, before selecting the test tool, you
must analyze the test cases and decide which test cases should be automated
and which test cases should not. This is the Automation Feasibility
Analysis activity.

Automation Feasibility Analysis is the very significant contributor in testing.

In this analysis, you need to check if the application under test
is qualified for automated test.

Some factors you need to consider:

To know more about these factors, you can use a checklist which is included
in this article

How to Select Best Automation Testing Tool

To select the most suitable testing tool for the project, the Test Manager
should follow the below tools selection process
Step 1) Identify the Requirement for Tools
How can you select a testing tool if you do not know what you are looking
for?

You to precisely identify your test tool requirements. All the requirement
must be documented and reviewed by project teams and the management
board.

Step 2) Evaluate the Tools and Vendors

After baselining the requirement of the tool, the Test Manager should

 Analyze the commercial and open source tools that are available in the
market, based on the project requirement.
 Create a tool shortlist which best meets your criteria
 One factor you should consider is vendors. You should consider the
vendor’s reputation, after sale support, tool update frequency, etc. while
taking your decision.
 Evaluate the quality of the tool by taking the trial usage & launching
a pilot. Many vendors often make trial versions of their software
available for download

Step 3) Estimate Cost and Benefit

To ensure the test tool is beneficial for business, the Test Manager have
to balance the following factors:
A cost-benefit analysis should be performed before acquiring or building a
tool

Example: After spending considerable time to investigate testing tools, the

project team found the perfect testing tool for the project Guru99 Bank
website. The evaluation results concluded that this tool could

 Double the current productivity of test execution

 Reduce the management effort by 30%

However, after discussing with the software vendor, you found that
the cost of this tool is too high compare to the value and benefit that it can
bring to the teamwork.

In such a case, the balance between cost & benefit of the tool may affect the
final decision.
Step 4) Make the Final Decision

To make the final decision, the Test Manager must have:

 Have a strong awareness of the tool. It means you must understand

which is the strong points and the weak points of the tool
 Balance cost and benefit.

Even with hours spent reading software manual and vendor information, you
may still need to try the tool in your actual working environment before
buying the license.

You should have the meeting with the project team, consultants to get the
deeper knowledge of the tool.

Your decision may adversely impact the project, the testing process, and the
business goals; you should spend a good time to think hard about it.