Load Factor - Performance Testing For Web Applications Business White Paper
Load Factor - Performance Testing For Web Applications Business White Paper
Load factor:
performance
testing for
Web
applications
Table of contents
Internet
Clients Routers Switches Web servers Internet Load Application Database servers
firewall balancers servers and other
and Applications database servers
connecting
with external
applications
3
Figure 2
The VuGen allows you to emulate production workloads on an entire system.
This complexity triggers many uncertainties about the integrity Manual testing is also not the most practical option because in order
and performance capabilities of a Web application. Will it provide to accurately diagnose problems, you must repeat tests multiple
consistently fast response times for all users—regardless of where times: first to identify performance problems, tune the system, and
they are coming from or what type of connection they are using? then retest to verify that tuning has had a positive impact. Manual
Can an application sustain a large user volume? Will it remain fast testing is a option for business failure.
and reliable after a sudden increase in the number of users? Is there
sufficient hardware on each tier to handle high traffic volumes? With automated performance testing tools, you can automatically
Can all the interoperable pieces of the system work reliably run tests and measure results. You can also easily and cost-
together? What is the maximum capacity of the application? To effectively create and maintain scripts and scenarios to maximize
answer these questions, you must be able to predict how your Web reuse between test runs and accurately compare results. Automated
applications will behave in a production environment before going testing tools help reduce the risk of human error during testing. As
live, and continue to measure and monitor your system throughout a result, automated performance testing has become the preferred
its lifecycle. choice for companies of all types and sizes for performance testing
of their business critical applications. Automated testing tools
typically use three major components to execute a test:
Put your apps through the paces. • A control console that organizes, drives, and manages the load
and use behavior
A typical response to growing traffic on your website is to add
more hardware to each component of the system. However, • Virtual users (Vusers)—processes used to imitate a real user
such random attempts to improve performance often lead to performing a business process on a client application
overbuying hardware—and not necessarily to better performance • Load servers (used to run the Vusers)
and scalability since many times the application was not designed
to be elastic. The only way to accurately predict system behavior Using these components, automated load testing tools can:
is to execute performance testing in your applications end to end, • Replace manual testers with automated Vusers
measure response times from the end-user perspective, and • Simultaneously run many Vusers on a single load generating
diagnose and remedy any performance issues prior to going live. machine from a unique location or from multiple locations around
the globe
Application performance testing is the way to measure your
Web application’s ability to conduct multiple transactions • Automatically measure transaction response times and monitor
simultaneously while maintaining adequate response times. the infrastructure components
It also helps you isolate bottlenecks in any component of the • Easily repeat load scenarios to validate any changes you make to
network infrastructure to accurately pinpoint potential problems. optimize performance
Two common approaches to performance testing methods are
manual and automated testing. Unfortunately, manual testing This advanced functionality definitely allows you to save time
has several inherent challenges, such as: and resources.
• Difficulty emulating the behavior of thousands of concurrent users
• Coordinating user operations
• Accurately measuring response times
• Comparing results between test runs, scenarios, and iterations.
4
Table 1
Accuracy and scalability are key attributes of an automated performance testing solution.
Record load testing scripts against a real client application. Generate more Vusers that can be run on a single machine before exceeding its capacity.
Capture protocol-level communication between the client application and the Generate more hits per second against a Web server.
rest of the system.
Provide flexibility and the ability to define configurations for user behavior (for Manage thousands of Vusers.
example, think times, connection speeds, cache settings, iterations).
Verify that the requested content returns to the browser to facilitate a Increase the number of Vusers in a controlled manner.
successful transaction.
Show detailed performance results that can be easily understood and analyzed to Simulate the effect of scaling out to remote locations over wireless area networks (WANs).
identify the source of the problem.
5
• Defining testing approach
You must determine the appropriate strategy for testing
applications. You can choose one or more types of performance
testing: load testing, stress testing, and capacity testing. Load
testing is typically used to gauge how an application will perform
under a specific user load. Your goal is to determine whether your
Web application can sustain this requested number of users with
acceptable response times. Stress testing is the way to validate
an application’s stability and reliability by applying a load for an
extended period of time. Capacity testing helps you determine
the maximum number of users who can access your application
simultaneously before experiencing system failure.
6
Figure 3 Figure 4
This generic graph shows application performance under load. In this This generic graph shows transactions that passed or failed. In this example,
example, the application response times increase sharply once the number transaction number two failed approximately one-third of the time.
of concurrent users reaches 7,500. Depending on how critical this transaction is, such high failure rate is likely
not acceptable for the business.
16
120
Number of transactions
14
100
Response time (sec)
12
80
10
60
8
40
6
20
4
0
2
1 2 3 4 5 6
0
1 2 3 4 5 6 7 8 9 10 Business transaction
Number of users (thousands)
Pass Fail
Step 3: Define user behavior. −− Emulate cache: Emulates a user browsing with a specific cache
Run-time settings define the way the script runs in order to size. Caching can be turned off based on server requirements.
accurately emulate real users. Settings can configure think times, −− Browser emulation: Enables you to specify which browser the
connection speeds, and error handling rules. Think times can Vuser simulates.
vary depending on your customer’s actions and complexity of a −− Number of connections: Allows the Vuser to control the number
transaction. For example, a customer can move quickly through the of connections to a server.
account information update page, while taking longer comparing
−− IP spoofing: Tests the performance impact of IP-dependent
product options or entering payment information.
components by assigning Vusers their own IP addresses from
Your system’s response times will also vary depending on the users’ the same physical machine.
connection speed. WAN emulation accurately imitates a variety of −− Iterations: Commands repetition of Vuser scripts and paces
connection options at varying network bandwidth and latencies. Vusers, instructing how long to wait between intervals. Iterative
This can help you determine how the underlying network affects testing defines the amount of work a user does based on the
application response times. Because errors can arise throughout number of times a process is performed using varying data.
a scenario and impede testing, you also need to configure error −− Error handling: Regulates how a Vuser handles errors during
handling. You can configure Vusers to handle these errors so you can script execution. HP LoadRunner can enable the continue on
run the tests uninterrupted. Network communications errors can error feature when the Vuser encounters an error during replay.
also profoundly influence application response times. You can also
−− Log files: Stores information about a Vuser’s server
configure WAN emulation to introduce underlying network errors to
communication. Standard logging maps all transactions,
understand their impact and measure the application’s tolerance
rendezvous, and output messages. Extended logging also
for them.
tracks warnings and other messages.
• Creating user behavior with HP LoadRunner
HP LoadRunner provides comprehensive run-time settings to
configure scripts that emulate the behavior of real users. Below
are some examples of run-time settings:
−− Think time: Controls the speed at which the Vuser interacts
with the system by including pauses of think times during test
execution.
−− Connection speed: Simulates a user connected to the system
from different network speeds.
7
Figure 5
How HP LoadRunner works
Step 4: Defining the performance tests scenarios multiuser load test scenarios. Using five easy-to-follow screens,
The performance test scenario contains information about the the scenario wizard leads you through a process of selecting the
groups of Vusers that can run the scripts and the load machines workstations that can host the Vusers, as well as the test scripts
on which the groups are running. To run a successful scenario, you to run.
must first categorize your customers into groups based on their
During this step-by-step process, you also create simulation
transactions. Then, you need to define and distribute the total
groups of Vusers. The HP LoadRunner scheduler is used to ramp
number of Vusers. You can assign a varying number of Vusers to
Vuser numbers up or down in order to position Vusers in both the
individual business processes to emulate user groups performing
ready state and the running state. For example, you may want to
multiple transactions. One group may be searching for a book, while
gradually increase the number of customers logging onto your site
the second and third groups are entering payment information or
with a fixed batch size. This is referred to as the ready state. This
shipping addresses. After Vuser groups have been assigned, you
method helps avoid unnecessary strain on the system.
must determine which load-generating machines the Vusers can
run on. You can add load generator machines to the client side of The scheduler also manages scheduling and features an
the system architecture to run additional Vusers. Finally, you need automated process that allows the user to run the script without
to specify how the scenario will run. You can run the Vuser groups being present. In real time, this would be similar to running a script
either in staggered or parallel formation. Staggering the Vusers during off-peak hours—such as 6 p.m. to 6 a.m. To schedule a test,
allows you to examine a gradual increase of the user load to a peak. you simply click the Run Scenario button and enter the desired
• Creating load testing scenarios with HP LoadRunner starting time.
HP LoadRunner Controller is used to create scenarios, providing • Create network impact tests.
you with complete visibility of the tests and the Vusers from a The network test uses information about where the groups of
single point of control. The Controller facilitates the process of Vusers can be located relative to the server. During the test, only
creating a load test scenario by allowing you to: the network characteristics—such as bandwidth availability,
−− Assign scripts to individual groups contention, latency, errors, and jitter—can change; the
−− Define the total number of Vusers needed to run the tests number of Vusers in this test remains constant. By staggering
the decreases in network bandwidth or increases in latency,
−− Define the host machines on which Vusers are running
errors, and jitter, you can understand their relative influence
In addition, HP LoadRunner offers a scenario wizard, a scheduler on application behavior. This data can be used to set network
and TurboLoad to enhance your experience during testing. The requirements for the application when it is deployed. You can
HP LoadRunner scenario wizard allows you to quickly compose conduct this kind of testing directly over the network to remote
8
Figure 6
The VuGen allows you to capture business processes to create Vusers.
Business process
Browser
Application
locations, but for testing purposes it is generally more practical to • Run the load test scenario and monitor the performance
emulate the network where you can easily establish a variety of with HP LoadRunner
conditions using WAN emulation. This can allow you to predict the Once the scenario is built, you are ready to run the test. The HP
performance for remote users. LoadRunner controller provides a suite of performance monitors
that can monitor each component of a multi-tier system during the
• Defining Load Testing per locations with HP LoadRunner load test. By capturing performance data over the entire system,
HP LoadRunner has the ability to handle WAN emulation, so you can correlate this information with the end-user loads and
you can use the same virtual-user scripts used in the previous response times to identify bottlenecks. HP LoadRunner provides
steps for network impact tests. Network characteristics such performance monitors for all parts of the system under test—
as connection speed, latency, and error rates are modified for including Web servers, application servers, databases, ERP and
groups of Vusers that are simultaneously emulated during a CRM systems, firewalls, and load balancers. HP LoadRunner can
single test run. You can then accurately determine the impact of identify hardware limitations and software configuration issues
the network on response time on the different groups, as well as that might otherwise go undetected. All performance monitors
the sensitivities of the application to the network. You can record are non-intrusive and hardware and OS independent, so they do
expected response time data and network requirement set for use not require that agents be installed on the remotely monitored
later when the application goes live. servers. To look deeper into each transaction—from end-user
response time all the way down to the application method level—
Step 5: Running the load test scenario and monitor the you can use HP Diagnostics software. During a performance test,
performance. HP Diagnostics traces business processes from the client side
You can view the application’s performance at any time during the across all tiers of the infrastructure, and then breaks down each
test using real-time monitoring. Every component of the system transaction response time into time spent in the various tiers and
requires monitoring: the clients, the network, the Web server, the within individual components. HP Diagnostics provides:
application server, the database, and all server hardware. You can −− An intuitive, easy-to-use view of how individual tiers,
detect performance bottlenecks early during testing using real- components, memory, and SQL statements impact overall
time monitoring. You then have the ability to view the performance performance of a business process under load conditions.
of every single tier, server, and component of the system during
−− The ability to more easily find components relevant to a specific
testing. As a result, you can instantly identify performance
business process under test.
bottlenecks during load testing. This can allow you to accelerate the
test process and achieve a more stable application. −− The ability to triage and find problems effectively with business
context, enabling teams to focus on problems impacting
business processes.
9
Figure 7
HP LoadRunner and HP Diagnostics allow you to monitor system performance during tests, diagnose performance bottlenecks as they occur, and identify root
cause of the issues.
First, a network specialist (such as a database administrator or −− Performance under load: indicates transaction times relative
consultant) can make the necessary adjustments to the system. to the number of Vusers running at any given point during the
Then you need to rerun the scripts to verify that the changes have scenario
taken place. −− Transaction performance: displays the average time taken to
perform transactions during each second of the scenario run
Finally, you can measure the amount of improvement the −− Transaction performance summary: displays the minimum,
system has undergone by comparing results from before and maximum and average performance times for all the
after testing. transactions in the scenario
The HP LoadRunner analysis component provides a single −− Transaction performance by Vuser: displays the time taken by
integration environment that collectively gathers all the data an individual Vuser to perform transactions during the scenario
generated throughout the testing cycle. HP LoadRunner analysis
−− Transaction distribution: displays the distribution of the time
provides advanced capabilities that let you compare results
taken to perform a transaction
across scenarios and drill down into network layers to identify
10
Figure 8 Figure 9
This sample activity graph displays the number of completed transactions This graph displays the number of transactions that passed, failed, aborted,
(successful and unsuccessful) performed during each second of a load test. or ended with errors. For example, these results show the “Submit Search”
This graph helps you determine the actual transaction load on your system business process passed all its transactions at a rate of approximately 96
at any given moment. The results show that after six minutes an application percent.
is under a load of 200 transactions per second.
HP LoadRunner also offers two types of Web graphs: Using minimal hardware resources, HP LoadRunner emulates
−− Connections per second: shows the number of connections hundreds or thousands of concurrent users to apply production
made to the Web server by Vusers during each second of the workloads to almost any application platform or environment. HP
scenario run LoadRunner stresses an application from end-to-end—applying
consistent, measurable, and repeatable loads—then uses the data
−− Throughput: shows the amount of throughput on the server to identify scalability issues that can affect real users in production.
during each second of the scenario run HP LoadRunner analysis
includes a correlation of results feature to enhance the user As it drives load against the system, HP LoadRunner captures
analysis process for the data. This allows you to custom design a end-user response times for business processes and transactions.
graph beyond the basics using any two metrics. As a result, you Non-intrusive, real-time performance monitors obtain and display
can identify and troubleshoot performance problems performance data from every tier, server, and system component.
more quickly. Strong analysis capabilities help isolate performance bottlenecks
quickly and efficiently.
HP LoadRunner software: getting the
picture of application performance
HP LoadRunner software is the industry-standard solution for
application performance testing. Because HP LoadRunner is
extremely flexible customers can design the performance testing
based as their own specific project requirements. HP LoadRunner
helps you obtain an accurate picture of end-to-end system
performance before going live. It helps:
• Verify that new or upgraded applications meet specified
performance requirements
• Identify and reduce performance bottlenecks during the
development lifecycle
• Pinpoint end-user, system-level, and code-level bottlenecks
rapidly and with ease
11
Test against a broad range of HP LoadRunner Key features and benefits
applications and protocols • Supports performance testing of new technologies together
with your existing, legacy applications
HP LoadRunner supports performance testing for a wide range
• Accurately tests a mix of mobile and Internet users
of application environments and protocols, including Ajax, Flex,
Microsoft ® Silverlight, Web, SOA, Web services, RDP, Database, • Decreases the risk of deploying systems that do not meet
Terminal, Citrix, Java, .NET, Oracle, and SAP. An updated and performance requirements
easy-to-use scripting and debugging engine leverages data • Reduces hardware and software costs by accurately predicting
format extensions and correlation studio to make scripting application scalability and capacity
faster and easier. • Helps you establish intelligent service-level agreements before
applications go live
To enable load testing earlier in the application lifecycle, HP
LoadRunner integrates with the leading J2EE, Microsoft ® Visual • Shortens test cycles to accelerate delivery of high‑quality
Studio, and Microsoft .NET environments. This integration lets you applications
create HP LoadRunner scripts directly within the IDE, so developers • Pinpoints end-user, system-level, and code-level bottlenecks
can participate in the performance testing efforts earlier. rapidly and with ease
• Reduces the cost of defects by testing earlier in the
As part of the lifecycle approach to application testing,
application lifecycle
HP LoadRunner is also integrated with HP QuickTest Professional,
HP Quality Center, and HP Application Performance Center
(APM) software. To discover how you can measure application
Leveraging these complementary solutions together with HP
performance using load testing software from HP,
LoadRunner provides a comprehensive approach for managing the please visit hp.com/go/loadrunner.
overall testing process, measuring release risk, and increasing the
reuse of testing assets from pre-release to production.
The intricacy of today’s Web technologies, high traffic volumes, For an overview of HP software services, visit
complexity of transactions, and frequent changes demand that managementsoftware.hp.com/service.
you take a lifecycle approach to performance testing. Make it a
To access technical interactive support, visit Software Support
part of the delivery process for all applications. By following the
Online at hp.com/managementsoftware/services.
best practices and using the right tools for planning, building,
maintaining, and reusing your performance testing assets, you will To learn more about HP Software Customer Connection, a one-stop
greatly reduce the risk of failure and enable that your applications information and learning portal for software products and services,
remain scalable and reliable throughout the lifecycle. visit hp.com/go/swcustomerconnection.
Get connected
hp.com/go/getconnected
Get the insider view on tech trends,
support alerts, and HP solutions
© Copyright 2007, 2010, 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and
services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors
or omissions contained herein.
Microsoft is a U.S. registered trademark of Microsoft Corporation. Oracle and Java are trademarks of Oracle and/or its affiliates.