All contents hereby presented are copyrighted material from Jose Fajardo.

 Copyrighted
2002. All rights reserved. Must obtain permission from Jose Fajardo to reproduce,
disseminate or publish this article. Email: [email protected]

Author’s Biography:
Jose Fajardo has worked as either a test lead or a test manager for various companies utilizing
automated testing tools from vendors such as Mercury Interactive, Compuware, Segue, and
Rational to test ERP applications such as SAP R/3, Peoplesoft, SAP R/3 Bolt-ons, custom
applications and non-SAP applications interfacing with SAP R/3. Jose Fajardo has helped
various companies: create testing standards, implement testing best practices, create Quality
Assurance Teams from scratch, mentor junior programmers, staff-up testing efforts with
resources, perform V&V activities, audit testing results, execute test scripts, and implement
automated testing strategies. Jose Fajardo is Ivy League educated with graduate engineering
education from the University of Pennsylvania and undergraduate engineering education from
the University of Virginia. Jose Fajardo has given SAP R/3 presentations at the Wharton School
of Business, conferences, and user groups and he is also SAP R/3 certified. He is currently a
senior SAP project manager at a large defense contractor and has over 7 years of IT
experience at various SAP R/3 implementations in these sectors: public, automotive,
pharmaceutical, chemical, consumer products, and high-tech. To contact Jose Fajardo directly
please send an email to: [email protected]

Summary:

Unfortunately many corporations make some blunders with their automation strategy even
before a single test script has been automated with a testing tool. In this article, Jose Fajardo
will discuss some considerations that should be examined, and assayed before purchasing
expensive automated testing tools and working with them. The author also suggests some
robust automated testing approaches and dispels some common testing misconceptions based
on his actual hands-on automated testing experience. The article should help test managers
streamline and ameliorate their techniques for working and implementing automated testing
tools and should also serve as a baseline for companies that are interested in purchasing and
selecting automated testing tools as part of their automated testing strategy.

Title: Working with automated testing tools from a pragmatic point of view.

Automated testing tool selection and acquisition

Many companies first learn about automated testing tools through conferences, trade
shows, internal contractors, consulting companies, or from employees that have previous
experience with one or more automated testing tool. Usually after companies learn about
automated testing tools and decide to acquire an automated software solution they get
product demos from the vendors, then the vendors provide the companies a copy of their
software copy with an evaluation license. After a company’s evaluation period of the
software expires the company may wind up spending hundreds of thousands of dollars on
automated testing software solutions that may not even meet a partial listing of the
company’s testing needs.

The need to select and purchase automated software solutions should be compelling and
critical while meeting a vast majority of the company’s testing efforts. The reader should
note that it might not be possible to find an automated testing software solution from a

Jose Fajardo Page 1 of 14 Email: [email protected]

single vendor to meet all of the company’s automated testing needs. Companies should
have a test manager or a testing champion on-site who is thoroughly familiar with
automated testing tools software in general and can provide substantial feedback to the
company researching test tools for possible acquisition during the evaluation period. The
test manager should articulate clearly and have a vision as to how the automated software
solution will be used, how compatible the automated software is with the existing IT
environment, who will use the software, and develop a realistic schedule of how many
test scripts can actually be developed with the automated testing tool given the
company’s already established deadlines for a given software release or deployment.

Test managers should also recognize that the mere fact that a software vendor provides a
satisfactory product demonstration for its automated testing tool is not sufficient to
guarantee that the automated testing tool will be compatible with the project’s application
or with the testing team’s on-going testing objectives. The test manager should perform
due diligence in asking the automated software vendor for a demonstration of its
automated software against the company’s IT environment. Furthermore, the test
manager or the project’s testing champion with previous knowledge of an automated
testing tool should compose a “wish-list of automated test scripts” from the project’s
current test scenarios ranging from low complexity to medium complexity that they
would like to have the vendor fully automate in their environment before acquiring a
specific automated tool from a vendor. The onus and burden should fall squarely on the
vendors to demonstrate how their automated testing tools will fit the needs and be
compatible with the company’s software application.

Test managers should be cognizant of the fact that many automated testing vendors will
make many promises about their tools during their product demonstration or “dog and
pony show” to make a sale for their tools when in fact these promises may not be
consistent with the company’s testing goals or may go unfulfilled. The test manager
should seek software demonstrations from at least 3 different vendors during the tool
selection phase and should rely on various readily available automated tool selection
matrices before recommending making a purchase for an automated testing tool. Armed
with the knowledge of another automated testing tool the appointed test manager or test
champion should inquire and verify that the product that the vendor is demonstrating has
features and capabilities that exceed or are similar to automated testing products that they
have worked with at previous projects.

Minimum Criteria Factors to consider before tools acquisition

At the very least the acquired automated testing software should meet the criteria
specified below. The reader should note that it might be necessary to purchase more than
one software solution from a vendor or vendors to meet the various testing needs of the
project. The criteria is enumerated below.

1. Automated testing tools should have version control capability

2. Automated testing tools should have workflow capability for the reporting and
closing of defects.

Jose Fajardo Page 2 of 14 Email: [email protected]

 3. The recording testing tool should recognize the custom controls, objects, GUI,
and generic controls (i.e Active-x controls) for the application under test to allow
playback of the recorded scripts.
4. Automated testing tools should allow for the sequencing of test sets with
dependencies for the execution of the automated test scripts. For instance execute
test script B only after test script C has successfully completed.
5. The recording testing tool should have a scripting language that is widely
accepted, robust and recognized (i.e. Visual Basic)
6. The recording testing tool should produce reports to verify the execution of the
scripts and provide a means to store the execution reports.
7. The automated testing tool should be compatible and integrated with standard
word processors and spreadsheets
8. The vendor of the automated testing tool should offer online support to allow the
customer to report identified problems or bugs with the automated testing
software.
9. The recording testing tool should work with external/internal data sheets, to allow
for the creation of parameterized data driven scripts
10. The automated testing tool should have the capability for report generation to
track and collect metrics for the number of scripts that passed, failed, number of
defects open/closes, number of test cases that have been developed, etc.
11. The automated testing tool should have email notification for the reporting and
closing of defects.
12. The automated testing tool should have an open architecture that makes it
flexible enough to modify it to extend the tool’s functionality.
13. The automated testing tool should have the capability to allow storage of
automated and manual scripts and to serve as a repository for test artifacts.

Again the reader should be aware that the delineated criteria 1-14 above is by no means
an all exhaustive list of attributes that the automated testing tools should possess before
acquisition but rather the aforementioned criteria should serve as a baseline for the
minimum features that the automated testing tools should possess.

Below are other considerations for selecting automated testing tools:

1. Straightjacket effect:

Companies that fall under this category purchase all their automated testing tools from a
single vendor and make a significant commitment of time, money and resources to the
automated testing solutions from a single vendor. The company is highly dependent on a
single vendor for all their automated testing needs. Although, this approach may be cost
effective if a single vendor is capable of meeting all of a company’s testing needs, in
actuality this is highly unlikely due to a company’s bevy of heterogeneous IT
applications. The test manager and the company may instead opt for a hybrid solution of
automated testing tools where automated solutions are procured from 2 vendors or more
to avoid the straightjacket effect of working with a single vendor and its potential

Jose Fajardo Page 3 of 14 Email: [email protected]

limitations. Below is an example from an actual project where the straightjacket effect
hampered one of my client’s ability to automate test cases.

Example: My client, a consumer products company for which I did consulting for had
spent over 500,000 dollars in automated testing software from a single vendor. The
company bought a bolt on to their existing ERP system running in production and wanted
to test the ERP bolt-on with their automated testing tools. The vendor offered an add-in
program at a cost of an extra $10,000 dollars that supposedly could recognize the objects
on the ERP bolt on for record and playback. As it turns out my client company could not
record or script at all the newly acquired ERP bolt-on despite the vendor’s promise.
Having stretched the budget for automated testing software on a single vendor my client
had to abandon its plans to automate the testing of the ERP bolt-on even though there
were other vendors in the market that had demonstrated the compatibility between their
automated testing software and the ERP bolt-on in question. My client’s only vendor for
automated testing software made promises that within 6 months they would deliver a
solution to test the ERP bolt-on but in the meantime my client had to test by hand and
manually the newly acquired ERP bolt-on and thus rendering the automated tools useless
for this effort.

2. Future of the application

Test managers in selecting automated testing software should not only consider the
current state of the architecture of their software application but future releases of their
software. For instance an automated testing tool may be compatible with a currently
installed client server software application that is invoked via a GUI on a desktop, but
what if within a future release of the software the new architecture for the software is to
become a completely web-enabled solution that is not compatible with the automated
testing tools that were recently purchased? If the purchased automated testing software
will not be compatible with a future release of the software that will be tested then the
test manager may want to postpone obtaining the automated testing software.

3. Vendor Support
Another guideline for a test manager to follow in selecting automated testing software is
the quality of the vendor’s software support. I was once in a project where the existing
automated software did not recognize newly introduced active-x controls within the
application under test. I worked closely with the vendor to get a beta version of their
software to support my testing needs at the project for the application’s active-x controls.
The vendor sent me within a workweek 3 different beta versions of their software until I
was able to record and playback successfully against the active-x controls within my
application. During the evaluation period for the automated testing software the company
purchasing the software should have its test engineers call the vendor for support and
evaluate the quality and responsiveness of the vendor’s support when problems arise with
the recording of test scripts during the evaluation period.

Automated Tool Administration

Jose Fajardo Page 4 of 14 Email: [email protected]

After a company has selected and purchased automated testing software from a vendor
that meets a vast majority of the company’s automated testing needs the next step is to
assign roles and responsibilities for administering the software.
The test manager should appoint primary and back-up administrators for managing the
administration of the automated testing software. The roles of the tool administrator
would be to install the software, install software patches, report bugs and defects with the
software to the vendor, configure and customize the software as needed, provide log-on
user access to the software as needed, maintain the software’s documentation and user’s
guides.
The test manager is hereby reminded that in many corporations the automated testing
tools are not part of the help-desk standard software image and therefore the company’s
help desk does not support the automated test tools at all when end users report problems
with the automated test tools.
Should end users of the automated test tools encounter problems with the automated test
tools within the project that are not quickly resolved then the end users may lose faith in
the testing tool and resist using it. I have seen this happen in particularly with test
management tools that serve as the repository for test cases, and test sets, and defect
reporting test tools and have large population of end users. The tool administrator should
be able to answer and respond to the questions that the end users present when they have
problems using the test tools.

Automated tool’s process owner

In addition to appointing the automated testing tool administrator the test manager should
also appoint the testers or group that own the processes for using and working with the
automated testing tools. The automated software tool’s process owners develop the
specifications as to how they want to see the testing software customized. A paradigm of
a process that could be customized is the workflow process for reporting defects and the
fields that need to be populated for creating a defect based on a company’s defect
reporting procedures. Another example of a process that could be customized are the
fields that need to be populated or created from scratch, within a test management tool in
order to create a test case.

Another task assigned to the automated tool’s process owners is the creation of
customized internal training materials for the end users of the automated testing tools
within the project.

The roles of the tool administrator and the tool process owner should be complementary.
For example the process owner may specify how the particular test management tool will
be customized and in turn the tool administrator would perform the associated tasks to
customize the test management tool.

Who will perform the automation

Jose Fajardo Page 5 of 14 Email: [email protected]

Developing and creating automated test scripts requires programming skills. Many
organizations erroneously assume that any tester within the testing team, or QA team has
the ability to write code for test scripts because they believe that creating script is a
simple matter of just “record/and play back”. This is a fallacy and it behooves test
managers to avoid this fallacy.

Writing and developing scripts with an automated testing tool requires knowledge of the
test script language, the ability to embed exception handling logic within the script,
insertion of logic operators, while loops, for loops, parameterization of test scripts,
correlation of test scripts, adding verification points to the script, adding logic to
recognize objects within an application, etc. Merely recording a script does not assure
that a script will playback successfully specially when the script is data-driven. After the
test script is recorded the automation test engineer needs to “massage” the script and
tailor the script until it successfully playbacks back in a manner consistent in which an
end user would execute the transactional steps of a test script.

Just in the same way that writing code and developing code to produce an application
requires technical skills so too is developing automated test scripts. The test manager
should recognize which members from his/her team will be the core automators with the
automated testing tools for the test scripts that need automation and which testers will be
responsible for the creation and documentation of the test design and test scripts that will
be automated. In some instances a test manager will need to create a test team structure in
which some of the testers are divided into subject matter experts with functional and in-
depth knowledge of the application under test and core test automators that have
knowledge of the automated testing tool but limited functional knowledge of the
application under test.

The reader should note that for some applications specially ERP applications the
functional testers have much expertise as to how the ERP application was configured and
what data sets are appropriate to define test cases and test sets to test the ERP application,
however these functional testers have little knowledge if any as to how to properly
develop an automated test script that successfully plays back with an automated testing
tool.

The test manager should identify which testers within his/her team have the ability to
develop automated test scripts and identify a tool champion for script automation within
his/her team. In the event that the test manager does not have the in-house expertise to
assign an automated test tool champion he/she should seek to hire an automation expert
or contractor to impart tool knowledge to the other core test automators. The reader
should note that the technical tester might rely heavily for support on the functional tester
for functional expertise and subject matter expertise associated with the application under
test.

Jose Fajardo Page 6 of 14 Email: [email protected]

Automated tool training

The test manager will in all likelihood get some days of training for the testing tools from
the vendor after the purchase of the automated tools. The test manager should decide
which testers within his/her team would be the core test automators and should therefore
provide training for the testing tools to the core automators and users in general of the
automated testing tool such as the test management tool, and the defect reporting test
tool.

Some test managers after choosing test automators to receive training for a test tool, sent
the testers training with the concept of “train-the-trainers”. The purpose of this concept is
to have the “trained-trainers” share their knowledge with other testers within the project
that may work with the automated test tools in the future. I have seen this concept worked
with mixed results since at the beginning the newly trained core automators and testers do
not have sufficient knowledge or know-how of the automated tools to teach other testers
right away, and these core automators and testers only gain enough knowledge of the
automated test tool after having worked with the automated tool extensively for a period
of at least 1 year.

The test manager should also be cautious that often times the vendor of the automated
test tool only has generic training with generic training materials on a generic
environment which might be considerably different from the test environment in which
the automated tools will actually be used against for the project needs. In such a scenario
the test manager may request customized training from the vendor for the project specific
applications and testing needs.

In any event the test manager should ascertain that the recently trained core automators
have a tool champion with them for at least one release of the software in case they
encounter difficulties automating test cases and test scenarios.

What to automate

In this document under the section of misconceptions I provide some scenarios under
which test cases should not be automated. In general one should automate processes: 1.
That will be repeated, 2. That require testing with multiple values of data, 3. That are
stable and constant, 4. That require verification of multiple business rules, 5. That are
resource intensive, 5. Critical to the application, 6. That will be part of regression testing,
and 7. That require integration testing. This is not an all-inclusive list of all the general
rules for automating a test script but a set of criteria guidelines for reviewing which test
cases are good candidates for automation. The reader should note that in order to
automate a test script, a stable or non-changing environment is needed to gain the most
benefit from an automated test script. For example if a script is automated and signed-off
for a given process on a Monday but the script will be executed on Thursday of the same

Jose Fajardo Page 7 of 14 Email: [email protected]

week but between Monday and Thursday of the same week the underlying recorded
business process has changed then the automated test script quite possible may not
playback successfully on Thursday. A better approach would be to wait until the process
that requires automation is stable and will not be changing until a future release and thus
the process can be automated and successfully played back multiple times within the
same release or future releases with minor test script modifications if the process has
changed between releases.

The reader should note that an automated test script should trace back to either: a
requirement, business process procedure, or test scenario.

Hybrid Solution (mixture of tools)

As previously mentioned it might not be possible to acquire a one size fits all automated
testing software solution from a single vendor. A company might have to buy automated
testing solutions from more than one vendor to meet all the project’s testing needs since
an IT project may be composed of several heterogeneous IT solutions.

The concept of hybrid testing solutions is not unique to automated testing some
companies for instance might purchase a human resource solution from one ERP vendor,
yet the company’s supply chain solution might come from a second ERP vendor, and the
company’s help desk solution might come from a third CRM vendor. The company goes
through the challenge of integration solutions from competing vendors because they want
the best possible solutions for its IT infrastructure since it is possible that a single vendor
will not have a world class solution for every critical process such as Human Resources,
Supply Chain, and Helpdesk. The challenge now is to obtain the necessary automated
testing tools that will enable the company to test how all the pieces of its integrated IT
solutions work as a single component and as individual stand along components.

Avoid Redundancies

The test manager should strive to have the project wisely and consistently utilize the
purchased testing tools. I was in a project once where various teams were implementing
several modules from an ERP vendor. The project had purchased a test management
solution that contained defect-reporting capabilities that everyone in the project could
have utilized effectively. However one of the teams implementing the ERP solution was
not aware of the test management tool’s capability to report defect and this one team had
plans to create from scratch a database for reporting defects. This was an unnecessary
waste of resources and man-hours. I brought this issue to the attention of the project
director and the plans to create a customized a database to report defects from scratch
were abandoned in favor of all the teams implementing the ERP solution using the
defect-reporting capabilities from the purchased test management tool.

Jose Fajardo Page 8 of 14 Email: [email protected]

Production rates for testers

A skilled technical tester with a few years of hands on experience should produce 3-4
maintainable, and robust automated test scripts per working day. A fully automated test
script means that the script has been recorded and modified with all the necessary
programming logic (i.e while loops, data driven, contains error handling logic, is
correlated, contains functional calls, contains verification points, etc) and the automated
test script has been tested and has functional tester sign-off.

These rates presume that the automated test scripts are recorded for test scenarios that
range from medium to complex difficulty and require more than just record and playback.

ROI
The return on investment on automated testing tools is best qualified as a series of
intangible benefits since demonstrating how a particular automated testing tool saves
money over manual testing might be an intractable challenge. Many vendors drum up
their automated testing tools with a plethora of benefits that few projects ever actually
manage to gain.

The main benefits of purchasing automated testing tools and using them to automate test
scripts are: 1. Consistency, 2.Under certain conditions (i.e. regression testing, and
benchmark testing) a previously recorded test script is repeatable, 3. Automated test
scripts may help save time during the test execution phase, 4. Electronic storage of test
results after test execution, 5. Some automated test tools produce graphs, results that
would be inconvenient to produce by hand, 6. Execution of test scenarios with multiple
data sets for parameterized data-driven test scripts, 7. Integration testing of various
business processes for correlated test scripts, 8. Automatic sequencing of test sets with
dependencies, 9. Automatic validation and verification of business objects for the
application under test and 10. Batch scheduling of test scripts for execution. There are
various other benefits gained from using automated testing tools such as: the workflow
activities during the reporting and closing of defects, the automatic collection of test
metrics from test management tools, the mapping of requirements to test cases, a central
repository to store all test scripts and automated test scripts, etc. Again this is not an all-
inclusive list of benefits of working with automated testing tools but just some of the
salient benefits of working with automated testing tools.

A company may save money by not hiring as many testers to perform test execution
during the test execution phase after the company purchased an automated testing
solution and thereby lowering the overheard costs associated with test execution and this
may be perceived as a return on investment from the purchased test tools. But what about
all the costs associated with maintaining the automated test tools, and the costs of hiring
technical testers to produce and maintain automated test scripts? These costs may easily
outweigh any savings incurred by not hiring as many testers to execute test scripts by

Jose Fajardo Page 9 of 14 Email: [email protected]

hands and thus the return on investment from the automated test tool may prove difficult
to quantify.

However I do think that over time although the automated test tools might not generate
any tangible ROI the test tools do bring the project a level of convenience and assurance
that may not otherwise be available in the absence of an automated testing tool.

As an example of the convenience that the automated tools may bring to a project I will
relate my experience from a previous project. I was at a project where a tester executed
test scripts by hand during the middle of the night and the tester had to verify the test
results before the morning shift started. I created an automated test script for the same
process that this tester executed by hand and embedded programming error handling
logic within the script to send an alarm to a pager if the application under test
encountered a given set of errors during testing. This tester no longer had to work the
night shift to manually test the application since I had automated the process for him and
I embedded test script logic to alert him if anything fails with the application during the
batch execution of the test. Needless to say the automated testing tool brought much
convenience to this tester.

When built correctly automated test scripts that are data driven can cover more test
scenarios and permutations of business processes than a tester can cover through manual
testing. For some companies covering more test scenarios with different data values may
mean the identification of more defects and bugs before an application under test is
released commercially. Some companies may prevent lawsuits, loss of contracts,
financial losses and damage to their reputation by building more robust software
applications that meet all the intended requirements. These are areas where automated
test tools may maximize their ROI. For example the automated testing tools may prevent
a company from losing millions of dollars though a lawsuit due to their faulty software
product or may help a company generate more revenue by keeping satisfied customers.

Tool location

The test manager should decide where the test tools would be installed. For instance
some projects that I have worked with have dedicated test labs with hardware equipment
for the testing team, where the tool administrator can install the automated testing tools.
Other projects that I have been involved with do not have a dedicated test lab and the
automated testing tools have to be installed on individual testers’ desktops. I have also
seen examples where the installation of the testing tools whether at a test lab or individual
desktop is irrelevant since the automated testing software is web based and can be
invoked from a web browser without requiring any installation at the client machines.

Jose Fajardo Page 10 of 14 Email: [email protected]

Misconceptions of automated testing

1. Repeatability
Often vendors trump their automated testing products as the panacea for having
consistent and repeatable test scripts. In fact a selling point for a software vendor is the
perceived notion that test scripts once automated can be re-used over and over and thus
saving testing time. Although I would agree that this argument from the vendor has
merits, this argument is shortsighted and suffers from the tunnel vision effect. In order to
have a repeatable test script that is valid for re-use for various software releases the
underlying test conditions under which the test script was recorded, and the underlying
application under test’s data and environment would have to be static and constant every
time the automated test script is executed. This is seldom if ever the case. Most projects
have their software undergo some sort of change from release to release, after bugs are
discovered, after customer complaints, or after undergoing a gap analysis. Once a test
script is recorded and it is underlying conditions on which it was recorded have changed
the script needs to be modified and maintained for correct playback on an application that
undergoes changes.

We can safely say that test scripts are repeatable when conditions for the application
under test stay constant after the script was recorded. And this concept works during a
single release when the testers conduct regression testing and in essence re-execute the
previously executed test script without making any modifications at all to the script or
possibly to the data that the test script is accessing if the test script has been
parameterized.

However, for the most part given the nature of ever changing IT applications one may
assume that test scripts will require modifications after the underlying conditions in
which the test script have changed and are repeatable only to the extent in which the
underlying test conditions under which the test script were recorded remain constant or
static.

2. Automating a script only involves Record/Playback

As previously mentioned automating a test script is more than merely recording
and playing back the test script.

The challenge of automation is to identify skilled personnel with a sufficiently proficient

technical background to create scripts, develop or revise the code to automate the script,
inserting loop statements, insert verification points, add error-handling logic, etc. Merely
recording a user action or keystroke against a business process does not by itself create a
fully robust automated script or even a valid reusable script for that matter. A functional
tester may have some mild success in creating scripts for simple tasks or business
processes such as viewing a report or viewing a business process in read-only mode, but
this does not necessarily mean that the functional tester can create data driven, correlated,
maintainable and fully robust scripts.

Jose Fajardo Page 11 of 14 Email: [email protected]

I hereby place a special emphasis in separating the roles of the functional expert and the
technical expert while these two types of testing roles need not be completely
independent they can certainly specialize in different areas. For the functional tester an
are of specialization might be drafting test cycles, drafting test scenarios, sequencing of
test cases, drafting test sets, providing navigation support and transactional steps within
the test script, identifying test data for the test script, ensuring that requirements are tested
via RTVMs, etc. For the technical tester (core automator) on the other hand an area of
specialization might be to create fully robust, data driven, and maintainable automated
test scripts and analyzing the results from the test script execution.

3. Everything under the sun can be automated

A common misconception is to expect that every single requirement, test scenario and
test case can be automated. According to industry best practices one does not automate
testing cases for all business processes and test scenarios. The following criteria for
instance identify business processes and test scenarios that would not be good candidates
for automation: business processes/test scenarios that will only be executed once;
business processes/test scenarios that require intuitive testing, test scenarios that require
negative testing, test scenarios that rely extensively or solely on analog and bitmap
testing which could be impractical since recoding a test script based on pixels and screen
coordinates may be unpredictable, and recording of test scripts on an unstable or
frequently changing environment. Under these aforementioned conditions the test
manager should employ manual testing techniques.

4. The automated tool is the means to document the test design, and test script

This is a misconception that I have seen at some projects in which test leads incorrectly
believe that recorded automated scripts serve as the basis for a documented test script.
There is no substitute for a well-documented test script that is written in coherent
English. The test script should include at the very minimum a description of the steps to
be captured with the automation tool, the expected results of the step, the data values that
need to be populated within the application under test to record the test script, the output
that needs to be captured, any pre-work needed to record the test script, and a description
of the warning and information messages that the system may display during the
recording of the testing script.

A technical tester (core test script automator) should have the ability to work with a self-
explanatory, thoroughly documented test script before proceeding to automate it. An
approach that requires a technical tester to divine what the testing steps ought to be from
a poorly recorded script is not a suitable approach for script automation and is
counterproductive as well as inefficient. To augment this argument in the absence of a
well written test script the technical tester would not know if the steps that he/she is
recording are malfunctioning because the application under test has a problem or is
malfunctioning. In order to have fully maintainable and properly recorded automated test
scripts the technical tester or core test automator should have access to a well-constructed

Jose Fajardo Page 12 of 14 Email: [email protected]

and fully documented test script that serves as the basis for recording automated test
scripts.

5. Automated testing speeds up the testing effort

This misconception is pointed out and that is that many test managers and project
directors mistakenly believe that creating automated scripts speeds up corresponding
testing activities. Creating automated scripts is a daunting, time-consuming and
monumental task in particular when one is doing so for the very first time on an entirely
new release of a software system. Creating automated scripts requires having valid
documented test cycles, test scenarios and test scripts that can be translated into an
automated script. Planning for the creation of a test case and test script with valid data is
in itself an initial input for the automated script (the output), since the automated script is
a by-product of the testing cycle. Identifying the requirements and scenarios that need to
be tested, and creating the testing cycles and automated scripts are resource intensive and
time-consuming activities that pay-off for subsequent releases of the software where the
previously created automated scripts can be re-executed with minor modifications or “as
is”.

The initial hurdle and obstacle is to create the first batch of automated scripts and then
leverage off these automated scripts for later software versions or releases with minor
modifications or no modifications at all. Realistically, a testing team that can successfully
automate 40-50% of the test cases that are applicable for automated testing when test case
automation has been attempted for the first time has indeed met a major testing challenge.

6. Maintaining scripts a previously recorded test script is always easy

This assumption is wrong! Maintaining some automated test scripts may prove to be a
logistical nightmare. Sometimes a core automator may have to modify an automated test
script that another core automator created but has left the company already. If
information is missing as to what the automated test script is supposed to do, or if the
previous core automator that left the company did not follow best practices in developing
the test script the new core automator might struggle to modify the previously recorded
the script or worst yet might have to create the new test script from scratch. Another
obstacle that makes test script maintenance difficult is that some parameterized scripts
require multiple data sets, which may create multiple permutations for executing the
automated test script since each data set in the parameterized test script might require
error handling or if/else programming logic.

The test manager should ascertain that best practices are followed in creating the
automated test script along with appropriate documentation for executing the script and
comments embedded within the test script. Furthermore, to the extent possible automated
test scripts should be modularized.

7. Functional testers record scripts and the technical testers revise the poorly
recorded scripts

Jose Fajardo Page 13 of 14 Email: [email protected]

A final misconception that I would like to bring to everyone’s attention is that some test
managers actually have functional testers record test scripts even though they know that
the functional testers have little or no knowledge of the automated testing tool. The
rationale for this misconception is that test managers want to have skilled technical
testers (core automators) worked off a previously recorded script and then revise the test
scripts that were poorly recorded by the functional tester as a means to speed up the
automation testing activities. This practice is highly inefficient and I would advise against
doing this.

Functional testers often introduce recorded steps into the test script that were recorded by
accident or erroneously then the test managers expect that the core automators can “fix”
any poorly recorded test script. The test manager should separate the roles of the
functional tester and the technical tester (‘core automator”) and only have the technical
tester recording automated test scripts.

An analogous situation would be with software development where you have a team of
functional analysts gathering requirements and creating the documentation for the
project’s specifications from which the development team would work off the functional
specs to write the programs for the application. The functional analysts would NOT
initially start coding in an object oriented language or some other programming language
and then the development team would go and “fix” the programs that the functional
analysts coded. The same principle applies to the automation of test scripts. The
functional testers perform the initial test planning activities and the creation of the test
scripts, test scenarios, test cycles, test sets, etc then the technical tester would work off
the fully documented test scripts and test scenarios and proceed to automate the test script
with an automated testing tool.

The functional tester and the technical tester can complement each other’s work for
instance after the technical tester has created a robust test script the functional tester can
verify that the script correctly plays back against the application and provide sign-off on
the automated test script.

Conclusions

This article illustrates some commonly held misconceptions about automated testing and
articulates some insights that are optimal for working with automated testing tools such
as guidelines for automating test scripts. The reader also has is informed on techniques
for selecting ad acquiring an automated testing tool, how to implement the test tool
within a QA team, and what are the expected benefits of working with automated testing
tools.

Jose Fajardo Page 14 of 14 Email: [email protected]