http://www.ijccr.

com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

Comparative Study of Testing Tools:

Apache JMeter and Load Runner

Vandana Chandel Shilpa Patial Sonal Guleria

Department of CSE Department of CSE, Department of CSE
Bahra University Bahra University Bahra University
Solan, India Solan, India Solan ,India
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

II. INTRODUCTION
I.ABSTRACT Software testing is a process used to help
Testing automation tools enables developers and identify the correctness, completeness and
testers to easily automate the entire process of quality of developed computer software. With
testing in software development. The objective that in mind, testing can never completely
of the paper is to conduct a comparative study of establish the correctness of computer software.
automated tools such as the Apache JMeter There are many approaches to software testing,
,Load Runner , QTP, based on criteria such as but effective testing of complex products is
the performance , speed, throughput and essentially a process of investigation, not merely
efficiency. The fundamental goal is to analyze a matter of creating and following rote
the features supported by these performance procedure. One definition of testing is "the
testing tools that aid in minimizing the resources process of questioning a product in order to
in script maintenance and increasing efficiency evaluate it", where the "questions" are things the
for script reuse. tester tries to do with the product, and the
product answers with its behavior in reaction to
Keywords: Apache Jmeter , Load Runner, the probing of the tester. Although most of the
Testing Tools, Software Testing. intellectual processes of testing are nearly
identical to that of review or inspection, the
word testing is connoted to mean the dynamic
analysis of the product—putting the product
through its paces.
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

The quality of the application can and normally III.SIGNIFICANCE

does vary widely from system to system but In Internet, We can see lot of articles
some of the common quality attributes include explaining/listing loss made by poor low-quality
reliability, stability, portability, maintainability software products.
and usability. Refer to the ISO standard ISO For example, how will you feel if a bug in a
9126 for more complete list of attributes and bank software shows your bank balance as 0
criteria. Testing helps is verifying and validating instead of some thousands?
if the Software is working as it is intended to be And if you are a student, what will you be your
working.This involves using Static and Dynamic state if your mark sheet shows your score as 0
methodologies to Test the application. Because instead of some good score?
of the fallibility of its human designers and its Here, we will be feeling good if we see some
own abstract, complex nature, software notification (e.g. Not able to show your balance
development must be accompanied by quality due to some unexpected error/couldn’t print
assurance activities. It is not unusual for your mark sheet because of unexpected issue)
developers to spend 40% of the total project instead of seeing wrong data.
time on testing. For life-critical software (e.g. Testing plays an important role to avoid these
flight control, reactor monitoring), testing can situations.
cost 3 to 5 times as much as all other activities So we can say that testing is necessary/important
combined. The destructive nature of testing even when it couldn't guarantee 100% error free
requires that the developer discard preconceived software application.
notions of the correctness of his/her developed
software[1]. Testing enables us:
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

• Cost of fixing the bug will be more if it webpage. As we know that Users satisfaction is
is found in later stage than it is found more important for growth of any business,
earlier. testing becomes more important.
• Quality can be ensured by testing only. So we can assume/treat the Testers as the
In the competitive market, only Quality representatives of the Users [2].
product can exist for long time.
IV. Scope of Software Testing
When we’re doing excellent testing, we’re
Testing will be necessary even if it is not
learning things about the product on behalf of
possible to do 100% testing for an application.
our clients and the project community.
One more important reason for doing testing is
Confirmation, verification, and validation are
user/production environment will be completely
important, but they’re mostly about checking.
different from development environment.
Checking is especially important, especially
useful, and especially inexpensive when
For example, a webpage developer may be using
programmers are doing it as part of the process
Firefox as browser for doing his webpage
of writing a program. Testing in the Bright
development. But the user may be using
Future is something more than just checking.
different browser such as Internet Explorer,
Among other things, testing means actively
Safari, Chrome and Opera.
engaging with the product, interacting with it,
The web page appearing good in Firefox may
providing it with challenging inputs, seeking
not appear good in other browsers (particularly
extents and limitations, exercising it with
IE). So ultimately, user will not be happy even if
complex scenarios, giving it a lengthy set of
the developer puts more efforts to develop the
tasks—one thing after another after another.
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

Testing is also about challenging the constraints V.SDLC

of the situation to obtain maximum relevant A Systems Development Life Cycle (SDLC)
information for our clients at maximum speed. adheres to important phases that are essential for
Often it’s about taking notice of things that are developers, such as planning, analysis, design,
hidden in plain sight; listening to the way people and implementation, and are explained in the
answer questions in addition to the content of section below. A number of system development
the answer; observing the way people interact life cycle (SDLC) models have been created:
with one another; and watching for the ways in waterfall, fountain, and spiral build and fix,
which they might be gaming the management rapid prototyping, incremental, and synchronize
systems. Confirmation is about answering the and stabilize. The oldest of these, and the best
questions “Does it do what we expect?” or “Is it known, is the waterfall model: a sequence of
still working?” In general, such tests tend stages in which the output of each stage
towards the scripted end of the becomes the input for the next. These stages can
exploratory/scripted continuum. In general, we be characterized and divided up in different
ask and answer the question “Can it work?” with ways, including the following:
tests that tend to be more exploratory and more
challenging. Still more challenging and more • Project planning, feasibility study:-
exploratory tests probe the product, asking “Will Establishes a high-level view of the
it work?” and “Is there a problem here?” Part of intended project and determines its
our job is to help defend against anticipated goals.
problems—but another part is to discover • Systems analysis, requirements
problems that no one else anticipated [2]. definition: - Defines project goals into
defined functions and operation of the
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

intended application. Analyzes end-user most important step of all, goes on

information needs. seemingly forever.
• Systems design: - Describes desired
features and operations in detail,
including screen layouts, business rules,
process diagrams, pseudo code and
other documentation.
• Implementation: - The real code is
written here.
• Integration and testing: - Brings all the
pieces together into a special testing
environment, then checks for errors,
bugs and interoperability.
• Acceptance, installation, deployment: -
The final stage of initial development,
FIG-1
where the software is put into
production and runs actual business.
Another way to say this is: The desired result of
• Maintenance: - What happens during the
testing is a level of confidence in the software so
rest of the software's life: changes,
that the organization is confident that the
correction, additions, and moves to a
software has an acceptable defect rate. What
different computing platform and more.
constitutes an acceptable defect rate depends on
This, the least glamorous and perhaps
the nature of the software. Another way of
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

saying this in an even more businesslike manner measure performance. It was originally designed
is: Testing is a process of technical for testing Web Applications but has since
investigation, performed on behalf of expanded to other test functions. Apache JMeter
stakeholders, that is intended to reveal quality- may be used to test performance both on static
related information about the product with and dynamic resources (files, Servlets, Perl
respect to the context in which it is intended to scripts, Java Objects, Data Bases and Queries,
operate. This includes, but is not limited to, the FTP Servers and more). It can be used to
process of executing a program or application simulate a heavy load on a server, network or
with the intent of finding errors. And what are object to test its strength or to analyze overall
some the common things that necessitate the performance under different load types. You can
need for a testing plan: use it to make a graphical analysis of
performance or to test your server/script/object
• Communication gaps between the behavior under heavy concurrent load.
requirement specified and the developer.
• Over commitment by the developer. A) Advantages
• Not proper understanding of all the
• Can load and performance test many
complexities of the system.
different server types:
• Inadequate requirements gathering [3].
o Web - HTTP, HTTPS

VI.Apache Jmeter: o SOAP

The Apache Jmeter desktop application is open o Database via JDBC

source software, a 100% pure Java application o LDAP

designed to load test functional behavior and o JMS

 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

o Mail - POP3(S) and IMAP(S) • Scriptable Samplers (Bean Shell is fully

• Complete portability and 100% Java supported; and there is a sampler which
purity. supports BSF-compatible languages)
• Full multithreading framework allows
concurrent sampling by many threads B) Disadvantages

and simultaneous sampling of different

• It’s open source! There’s no guarantee
functions by separate thread groups.
of support or further development.
• Careful GUI design allows faster
However, I don’t consider this a real
operation and more precise timings.
problem with a tool like JMeter, which
• Caching and offline analysis/replaying
easily can be replaced by another tool.
of test results.
• Getting "correct" response times from
• Highly Extensible:
JMeter forces you to understand how to
o Pluggable Samplers allow
make a realistic client/server setup. If
unlimited testing capabilities.
you’re not careful you might end up
o Several load statistics may be
measuring some incidental bottleneck,
chosen with pluggable timers.
like your own workstation’s capacity[4]
o Data analysis and visualization
plug-in allow great extendibility
VII. Load Runner
as well as personalization.
Load Runner is an industry-leading performance
• Functions can be used to provide
and load testing product by Hewlett-Packard
dynamic input to a test or provide
(since it acquired Mercury Interactive in
data manipulation.
November 2006) for examining system
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

behaviour and performance, while generating when to activate them, and how to group
actual load. Load Runner can emulate hundreds the Users and keep track of them.
or thousands of concurrent user’s to put the • The Results and Analysis program gives
application through the rigors of real-life user us all the results of the load test in
loads, while collecting information from key various forms. It allows us to see
Infrastructure components (Web servers, summaries of data, as well as the details
database servers etc). The results can then be of the load test for pinpointing problems
analysed in detail, to explore the reasons for or bottlenecks.
particular behaviour.LoadRunner is divided up
into three smaller applications: A) Advantages:
• No need to install it on the server under
• The Virtual User Generator allows us to
test. It uses native monitors. For Ex:
determine what actions we would like
perform for windows or for Unix
our Vusers, or virtual users, to perform
• Uses ANSI C as the default
within the application. We create scripts
programming language1 and other
that generate a series of actions, such as
languages like Java and VB.
logging on, navigating through the
• Excellent monitoring and analysis
application, and exiting the program.
interface where you can see reports in
• The Controller takes the scripts that we
easy to understand colored charts and
have made and runs them through a
graphics.
schedule that we set up.We tell the
• Supports most of the protocols.
Controller how many Users to activate,
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

• Makes correlation3 much easier. We will Correlation is a way to substitute values

dig into correlation through a series of in dynamic data to enable successful
posts later. playback [5].
• Nice GUI generated script through a one
click recording; of course you would
need to modify the script according to
VIII.PROBLEM STATEMENT
your needs.
Testing is a critical part of the software
• Excellent tutorials, exhaustive
development process.There are a lot of different
documentation and active tool support
software testing tools currently on the
from HP.
market.Some of these are only able to perform

B) Disadvantages: specific kinds of testing and only work.When we

research for the right software testing tool it is
The only disadvantage is the prohibitive cost
important to create list of requirements that help
associated with the tool but that can also be
us in choosing tool for performance.
compensated in the long run when you start
This research evaluate two major testing tools
getting a good ROI from the tool.
that are apache Jmeter and Load Runner. These
• Programming/Scripting language is used tools are evaluated on the basis of performance ,
to represent the captured protocol data speed, throughput an Evaluation Study.
and manipulate the data for play-back.
• Protocol is simply a language that your
client uses to communicate with the
system.
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

IX.ANALAYSIS
A)UNLIMITEDGENERATE
LOAD
This is a make-or-break item. There are many
scenarios I just can't cover if I can only open a
Item Load Jmeter few thousand socket connections to the server.
Runner a) Load Runner

Unlimited Load _ + Load runner restricts the number of users you

can run. Even large amounts of money only
Generator
allow a license for a modest number of users.
Ease Installation _ +
Historically, the rate for 10,000 HTTP users has
Large Download + _ been $250,000. However, on a per agent basis,
Performance load is generated very efficiently so it may take
Result Reporting + _ less hardware to generate the same amount of
load. (But for the money you spend on the load

Cost _ + runner license, you could buy a LOT of load

generation hardware! )
Technical Level + +
b) JMeter
Stability/Bugginess Neutral _
Since this is Free/Open Source, you may run as
many agents as you have hardware to put them
on. You can add more and more load virtually
forever, as long as you have more hardware to
run additional agents on. However, in specific
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

uncast scenarios, such as repeatedly B ) EASE OF INSTALLATION

downloading very large files (like a) Load Runner
PIPEDSCHEDULE), the ability of agents to Installation: - Installation takes a ton of time, a
generate load falls off abruptly due to memory lot of disk space, and a very specific version of
issues. Windows. But it's as simple as running a
windows installer, followed by 3 or 4 product
updaters.

• Setting up Simple tests:-For HTTP tests,

Load Runner is strong in this category,
with its browser recorder and icon-based
scripts.

• Running Tests:- The UI of the controller

is complex and a bit daunting. There is
great power in the UI if you can find it.

b) JMeter

• Installation:-Be sure Sun's JRE is

installed. Unpack the tar file. Simple.

• Setting up Simple tests Very quick. Start

up the console, a few clicks of the
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

mouse, and you are ready to generate capacity is strong. Licensing constraints may
load. Add thread group, add a sampler, limit actual load generated.
and you have the basics. Throw in an b) Jmeter:-With the exception of the high-
assertion or two on your sampler to bandwidth case, per agent capacity is good
validate server responses.
D)Results Reporting
• Running Tests both distributed and local Having key graphs generated at the conclusion
tests can be started form the UI. A menu of a scale run, such as load over time, server
shows the available agents, and grays out CPU, transactions per second, etc, can save a lot
the ones that are already busy. Standalone of tedium, since manually generating these
tests can be started from the command line.
graphs from log files is quite time consuming.
JMeter wins this category hands down.
a) Load Runner

C) Download Performance Load runner has an excellent integrated analysis

JMeter and Load Runner are two different tool that can dynamically generate graphs on any
performance testing tools. Performance testing of the myriad performance counters available to
tools are tools in the software field by which it. The downfall of this approach is that there are
different types of applications used in the only a small number of performance metrics it
software are tested. The performance of these can gather on Solaris. And while it can gather
applications is tested by increasing the load on additional server metrics using sar, vmstat, trace,
them and checking the maximum limit up to instat, mpstat, etc., integrating this information
which they can work in an efficient and effective in to the load runner framework will be difficult
manner. at best.
a) Load Runner:-Per-agent load generation
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

b) Jmeter b) Jmeter
JMeter does not gather any server-side Free. (Apache License)
performance metrics. But it can generate a
limited number of client-side graphs while the
F) TECHNICAL LEVEL
test is in progress. These graphs can be saved
after the test is over. Fortunately, all the test data a) Load Runner
is written in a standard format. So it probably Load Runner has the widest audience of all

makes more sense to generate all the desired these tools; perhaps not surprising given its
graphs via shell scripts during post-processing. maturity as a commercial product. It's browser-
This is the same approach I used with our recording and icon-based script development
previous test harness. give it the lowest technical barriers to entry of
any of the products. A QA engineer with modest
E) COST technical background and little to no coding
a) Load Runner skills can still be productive with tool. And it's
Expect to pay in the low to mid six-figures for a ability to load Windows and other libraries give
license allowing any kind of robust load- it a power and flexibility useful to developers
generation capacity. But that's not all, there are and other more advanced users.
high ongoing support costs as well. For the b)Jmeter
same kind of money I could get over 100
JMeter does not require developer skills to
powerful machines to use as scale agents, as
perform basic tests in any of the protocols it
well as associated network switches, cabling,
support out of the box. A form-driven UI allows
etc.
the user to design their own scenario. This
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

scenario is then auto-deployed to all agents

during test initialization.

G)STABILITY/BUGGINESS
a)LoadRunner
The controller crashes occasionally under heavy
load, but this is infrequent and largely
manageable. Other than this, the product
seems robust enough.

b) Jmeter
JMeter fares poorly in this area [4],[5].

Fig 2
IX. GRAPHICAL REPRESENTATION X. LINE REPRESENTATION
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

Prof. Miss. Pratibha Sharma (in Bahra

University,Shimla Hills) for their guidance and
support throughout the project.We
acknowledge the help and cooperation received
from all the staff members of the Computer
Science and Engineering department of Bahra
University, Shimla Hills.Finally,we wish to
express our sincere thanks to our family
members for their moral support.

XII). CONCLUSION

Software testing has become necessity of

companies because it saves both time and
money.Apache Jmeter and Load Runner are very
good tool for stress and performance testing.
FIG-3 [4], [5] Based on this research, we recommended The
Apache Jmeter as the tool to go forward with. It
has a simple, clean UI that clearly shows what is
going on without trying to do too much, and
XI.ACKNOWLEDGMENT offers great power and simplicity with its
Many persons have contributed to make this unique.Moreover,it is free of cost and having
project work a reality.We would like to express Complete portability and 100% Java purity .
our gratituide to Dr. Rajan Vohra and Assistant Both of the open source projects have merits, but
 http://www.ijccr.com

International Manuscript ID : ISSN2249054X-V3I3M3-052013

VOLUME 3 ISSUE 3 May 2013

neither one is ideal. Our approach will be to http://www.compuware.com/products/qacenter/4

work with the Apache Jmeter development team 02_ENG_HTML.htm
to resolve the most serious offenders. http://www.compuware.com/products/qacenter/q

XIII. REFERENCES aload.html http://jakarta.apache.org/jmeter/

http://jakarta.apache.org/jmeter/usermanual/inde
[1] Introduction to Software Testing – by Paul x.html
Ammann,JeffOffutt EmilyH.Halili,”ApacheJmeter”.
[2]http;//qualitypoint.blogspot.com/2009/04/imp [5]http://www.mercury.com/us/products/perfor
ortance-of-software-testing.html
mance-center/loadrunner/requirements.html
Automated Software Testing Elfriede Dustin,
Linux/Solaris server monitoring (weak)
Jeff Rashka, John Paul, 2005.
http://www.mercury.com/us/products/performan
[3] Software Testing: Testing Across the ce-center/loadrunner/monitors/unix.html
Entire Software Development Life Cycle – by http://grinder.sourceforge.net/g3/getting-
Gerald D. Everett, Raymond, Jr. McLeod started.html
Software Testing Process: Test Management – http://www.nrgglobal.com/application-testing-
by Andreas Spillner, Tilo Linz, Thomas tools/end-user-metrics-performance-monitoring-
Rossner, Mario Winter appswatch.html
Automated Software Testing – by Elfriede http://www.merc-int.com/
Dustin and Jeff Rashka
[4]http://jakarta.apache.org/jmeter/usermanual/i
ndex.html
http://www.w3.org/Protocols/rfc2616/rfc2616-
sec14.html