The Effects of Active Queue Management on Web Performance

Long Le Jay Aikat Kevin Jeffay F. Donelson Smith

Department of Computer Science
University of North Carolina at Chapel Hill
http://www.cs.unc.edu/Research/dirt

ABSTRACT viding queue space to absorb bursts of packet arrivals, (2) avoiding
We present an empirical study of the effects of active queue man- lock-out and bias effects from a few flows dominating queue
agement (AQM) on the distribution of response times experienced space, and (3) providing lower delays for interactive applications
by a population of web users. Three prominent AQM schemes are such as web browsing [3].
considered: the Proportional Integrator (PI) controller, the Random
Exponential Marking (REM) controller, and Adaptive Random All AQM designs function by detecting impending queue buildup
Early Detection (ARED). The effects of these AQM schemes were and notifying sources before the queue in a router overflows. The
studied alone and in combination with Explicit Congestion Notifi- various designs proposed for AQM differ in the mechanisms used
cation (ECN). Our major results are: to detect congestion and in the type of control mechanisms used to
1. For offered loads up to 80% of bottleneck link capacity, no achieve a stable operating point for the queue size. Another dimen-
AQM scheme provides better response times than simple drop- sion that has a significant impact on performance is how the con-
tail FIFO queue management.
gestion signal is delivered to the sender. In today’s Internet where
2. For loads of 90% of link capacity or greater when ECN is not the dominant transport protocol is TCP (which reacts to segment
used, PI results in a modest improvement over drop-tail and the
other AQM schemes. loss as an indicator of congestion), the signal is usually delivered
3. With ECN, both PI and REM provide significant response time implicitly by dropping packets at the router when the AQM algo-
improvement at offered loads above 90% of link capacity. More- rithm detects queue buildup. An IETF proposed standard adds an
over, at a load of 90% PI and REM with ECN provide response explicit signalling mechanism, called explicit congestion notifica-
times competitive to that achieved on an unloaded network. tion (ECN) [12], by allocating bits in the IP and TCP headers for
4. ARED with recommended parameter settings consistently re- this purpose. With ECN a router can signal congestion to an end-
sulted in the poorest response times which was unimproved by system by “marking” a packet (setting a bit in the header).
the addition of ECN.
We conclude that without ECN there is little end-user performance In this work we report the results of an empirical evaluation of
gain to be realized by employing the AQM designs studied here. three prominent examples of AQM designs. These are the Propor-
However, with ECN, response times can be significantly im- tional Integrator (PI) controller [8], the Random Exponential
proved. In addition it appears likely that provider links may be Marking (REM) controller [2] and a contemporary redesign of the
operated at near saturation levels without significant degradation in classic RED controller, Adaptive RED [6] (here called ARED).
user-perceived performance. While these designs differ in many respects, each is an attempt to
realize a control mechanism that achieves a stable operating point
Categories and Subject Descriptors for the size of the router queue. Thus a user of each of these
C.2.2 [Computer Systems Organization]: Computer Communi- mechanisms can determine a desired operating point for the control
cation Networks — Network Protocols. mechanism by simply specifying a desired target mean queue size.
Choosing the desired queue size may represent a tradeoff between
General Terms link utilization and queuing delay — a short queue reduces latency
Algorithms, Measurement, Performance, Experimentation. at the router but setting the target queue size too small may reduce
link utilization by causing the queue to drain needlessly.
Keywords
Congestion control, Active queue management, Web performance. Our goal in this study was first and foremost to compare the per-
formance of control theoretic AQM algorithms (PI and REM) with
the more traditional randomized dropping found in RED. For per-
1 INTRODUCTION AND MOTIVATION formance metrics we chose both user-centric measures of perform-
The random early detection (RED) algorithm, first described ten ance such as response times for the request-response transactions
years ago [7], inspired a new focus for congestion control research that comprise Web browsing, as well as more traditional metrics
on the area of active queue management (AQM). The common such as achievable link utilization and loss rates. The distribution
goal of all AQM designs is to keep the average queue size small in of response times that would be experienced by a population of
routers. This has a number of desirable effects including (1) pro- web users is used to assess the user-perceived performance of the
AQM schemes. Link utilization is used to assess the impact on
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
network resources. Of particular interest was the implication of
not made or distributed for profit or commercial advantage and that copies ECN support on performance. ECN requires the participation of
bear this notice and the full citation on the first page. To copy otherwise, or end-systems in the AQM scheme and hence it is important to quan-
republish, to post on servers or to redistribute to lists, requires prior specific tify the performance gain to be had at the expense of a more com-
permission and/or a fee. plex protocol stack and migration issues for the end-system.
SIGCOMM’03, August 25–29, 2003, Karlsruhe, Germany.
Copyright 2003 ACM 1-58113-735-4/03/0008…$5.00.

265
Our experimental platform was a laboratory testbed consisting of a presents our results for AQM with ECN. The results are discussed
large collection of computers arranged to emulate a peering point in Section 6. We conclude in Section 7 with a summary of our
between two ISPs operated at 100 Mbps (see Figure 1). We emu- major results.
lated the Web browsing behavior of tens of thousands of users
whose traffic transits the link connecting the ISPs and investigated 2 BACKGROUND AND RELATED WORK
the performance of each AQM scheme in the border-routers con- The original RED design uses a weighted-average queue size as a
necting the ISPs. Each scheme was investigated both with and measure of congestion. When this weighted average is smaller than
without ECN support across a variety of AQM parameter settings a minimum threshold (minth), no packets are marked or dropped.
that represented a range of target router-queue lengths. For each When the average queue length is between the minimum threshold
target queue length we varied the offered load on the physical link and the maximum threshold (maxth), the probability of marking or
connecting the ISPs to determine how (or if) AQM performance dropping packets varies linearly between 0 and a maximum drop
was affected by load. probability (maxp, typically 0.10). If the average queue length ex-
Our results were that for offered loads up to 80% of the bottleneck ceeds maxth, all packets are marked or dropped. (The actual size of
link capacity, no AQM scheme provided better response time per- the queue must be greater than maxth to absorb transient bursts of
formance than simple drop-tail FIFO queue management. In addi- packet arrivals.) A modification to the original design introduced a
tion, all schemes resulted in similar loss rates and link utilization. “gentle mode” in which the mark or drop probability increases
For offered loads above 80% of link capacity there was an advan- linearly between maxp and 1 as the average queue length varies
tage to employing control theoretic AQM. When ECN is not used, between maxth and 2 x maxth. This fixes a problem in the original
at offered loads of 90% of link capacity, PI resulted in a modest RED design caused by the non-linearity in drop probability (in-
improvement over drop-tail and the other AQM schemes. Web creasing from maxp to 1.0 immediately when maxth is reached).
browsing response time was improved for responses requiring A weakness of RED is that it does not take into consideration the
more than approximately 400 milliseconds to complete but at the number of flows sharing a bottleneck link [5]. Given the TCP con-
cost of slightly lower achievable link utilization (compared to gestion control mechanism, a packet mark or drop reduces the
drop-tail). Of particular note was the fact that without ECN, PI offered load by a factor of (1 – 0.5n-1) where n is the number of
gave performance superior to REM. flows sharing the bottleneck link. Thus, RED is not effective in
Our most striking result is that with ECN, both REM and PI sig- controlling the queue length when n is large. On the other hand,
nificantly outperform drop-tail at 90% load and provide response RED can be too aggressive and can cause under-utilization of the
time performance that is competitive to that achieved on an link when n is small. Feng et al. concluded that RED needs to be
unloaded network. The improved response time performance, tuned for the dynamic characteristics of the aggregate traffic on a
however, comes at some loss of achievable link utilization. In light given link [5]. They proposed a self-configuring algorithm for
of these results, an additional striking result was the fact that the RED by adjusting maxp every time the average queue length falls
addition of ECN did not improve ARED performance. ARED con- out of the target range between minth and maxth. When the average
sistently resulted in the poorest response time performance across queue length is smaller than minth, maxp is decreased multiplica-
all offered loads and resulted in the lowest link utilizations. tively to reduce RED’s aggressiveness in marking or dropping
packets; when the queue length is larger than maxth, maxp is in-
We conclude that without ECN there is little end-user or provider creased multiplicatively. Floyd et al. improved upon this original
performance gain to be realized by employing the AQM algo- adaptive RED proposal by replacing the MIMD (multiplicative
rithms studied here. However, with ECN performance can be sig- increase multiplicative decrease) approach with an AIMD (additive
nificantly improved. In addition, our experiments provide evidence increase multiplicative decrease) approach [6]. They also provided
that provider links may be operated at near saturation levels (90% guidelines for choosing minth, maxth, and the weight for computing
average utilization with bursty traffic sources) without significant a target average queue length. The RED version that we imple-
degradation in user-perceived performance and with only very mented and studied in our work (referred to herein as “ARED”)
modest decreases in link utilization (when compared to drop-tail). includes both the adaptive and gentle refinements to the original
Thus unlike a similar earlier study [4] which was negative on the design. It is based on the description given in [6].
use of AQM, we view the ECN results as a significant indicator
that the stated goals of AQM can be realized in practice. In [11], Misra et al. applied control theory to develop a model for
TCP and AQM dynamics and used this model to analyze RED.
While the results of this study are intriguing, the study was none- They pointed out two limitations in the original RED design: (1)
theless limited. The design space of AQM schemes is large with RED is either unstable or has slow responses to changes in net-
each algorithm typically characterized by a number of independent work traffic, and (2) RED’s use of a weighted-average queue
parameters. We limited our consideration of AQM algorithms to a length to detect congestion and its use of loss probability as a feed-
comparison between two classes of algorithms: those based on back signal to the senders were flawed. Because of this, in over-
control theoretic principles and those based on the original ran- load situations, flows can suffer both high delay and a high packet
domized dropping paradigm of RED. Moreover, we studied a link loss rate. Hollot et al. simplified the TCP/AQM model to a linear
carrying only Web-like traffic. More realistic mixes of HTTP and system and designed a Proportional Integrator (PI) controller that
other TCP traffic as well as traffic from UDP-based applications regulates the queue length to a target value called the “queue refer-
need to be examined. ence,” qref [8]. The PI controller uses instantaneous samples of the
The following section reviews the salient design principles of cur- queue length taken at a constant sampling frequency as its input.
rent AQM schemes and reviews the major algorithms that have The drop probability is computed as
been proposed. Section 3 presents our experimental methodology p(kT) = a x (q(kT) – qref) – b x (q((k–1)T) – qref) + p((k–1)T)
and discusses the generation of synthetic Web traffic. Section 4
presents our results for AQM with packet drops and Section 5

266
where p(kT) is the drop probability at the kth sampling interval, Network Monitor
q(kT) is the instantaneous sample of the queue length and T is
1/sampling-frequency. A close examination of this equation shows ISP 1 ISP 2
Ethernet Router Router Ethernet
that the drop probability increases in sampling intervals when the Switches Switches
queue length is higher than its target value. Furthermore, the drop 100/1,000
probability also increases if the queue has grown since the last Mbps

...

...
1 1
sample (reflecting an increase in network traffic). Conversely, the 100 Gbps Gbps 100
drop probability in a PI controller is reduced when the queue Mbps Mbps
length is lower than its target value or the queue length has de- ISP 1 Network ISP 2
creased since its last sample. The sampling interval and the coeffi- Browsers/Servers Monitor Browsers/Servers
cients in the equation depend on the link capacity, the maximum
RTT and the expected number of active flows using the link. Figure 1: Experimental network setup.

In [2], Athuraliya et al. proposed the Random Exponential Mark- Web response generator as the “server machines” (or “servers”).
ing (REM) AQM scheme. REM periodically updates a congestion The browser and server machines have 100 Mbps Ethernet inter-
measure called “price” that reflects any mismatch between packet faces and are attached to switched VLANs with both 100 Mbps
arrival and departure rates at the link (i.e., the difference between and 1 Gbps ports on 3Com 10/100/1000 Ethernet switches.
the demand and the service rate) and any queue size mismatch (i.e., At the core of this network are two router machines running the
the difference between the actual queue length and its target ALTQ extensions to FreeBSD. ALTQ extends IP-output queuing
value). The measure (p) is computed by: at the network interfaces to include alternative queue-management
p(t) = max(0, p(t–1) + γ x ( α x (q(t) – qref) ) + x(t) – c) ) disciplines [10]. We used the ALTQ infrastructure to implement
PI, REM, and ARED. The routers are 1 GHz Pentium IIIs with
where c is the link capacity (in packet departures per unit time), over 1 GB of memory. Each router has one 1000-SX fiber Gigabit
p(t) is the congestion measure, q(t) is the queue length, and x(t) is Ethernet NIC attached to one of the 3Com switches. Each router
the packet arrival rate, all determined at time t. As with ARED and also has three additional Ethernet interfaces (a second 1000-SX
PI, the control target is only expressed by the queue size. fiber Gigabit Ethernet NIC and two 100 Mpbs Fast Ethernet NICs)
The mark/drop probability in REM is defined as prob(t) = 1 – φ– configured to create point-to-point Ethernet segments that connect
p(t)
, where φ > 1 is a constant. In overload situations, the congestion the routers as shown in Figure 1. When conducting measurements
price increases due to the rate mismatch and the queue mismatch. to calibrate the traffic generators on an un-congested network,
Thus, more packets are dropped or marked to signal TCP senders static routes are configured on the routers so that all traffic uses the
to reduce their transmission rate. When congestion abates, the full-duplex Gigabit Ethernet segment. When we need to create a
congestion price is reduced because the mismatches are now nega- bottleneck between the two routers, the static routes are reconfig-
tive. This causes REM to drop or mark fewer packets and allows ured so that all traffic flowing in one direction uses one 100 Mbps
the senders to potentially increase their transmission rate. It is easy Ethernet segment and all traffic flowing in the opposite direction
to see that a positive rate mismatch over a time interval will cause uses the other 100 Mbps Ethernet segment.1 These configurations
the queue size to increase. Conversely, a negative rate mismatch allow us to emulate the full-duplex behavior of the typical wide-
over a time interval will drain the queue. Thus, REM is similar to area network link.
PI because the rate mismatch can be detected by comparing the Another important factor in emulating this network is the effect of
instantaneous queue length with its previous sampled value. Fur- end-to-end latency. We use a locally-modified version of the
thermore, when drop or mark probability is small, the exponential dummynet [9] component of FreeBSD to configure out-bound
function can be approximated by a linear function [1]. packet delays on browser machines to emulate different round-trip
times on each TCP connection (giving per-flow delays). This is
3 EXPERIMENTAL METHODOLOGY accomplished by extending the dummynet mechanisms for regulat-
For our experiments we constructed a laboratory network that ing per-flow bandwidth to include a mode for adding a randomly-
emulates the interconnection between two Internet service provider chosen minimum delay to all packets from each flow. The same
(ISP) networks. Specifically, we emulate one peering link that minimum delay is applied to all packets in a given flow (identified
carries web traffic between sources and destinations on both sides by IP addressing 5-tuple). The minimum delay in milliseconds
of the peering link and where the traffic carried between the two assigned to each flow is sampled from a discrete uniform distribu-
ISP networks is evenly balanced in both directions. tion on the range [10, 150] (a mean of 80 milliseconds). The mini-
mum and maximum values for this distribution were chosen to
The laboratory network used to emulate this configuration is
approximate a typical range of Internet round-trip times within the
shown in Figure 1. All systems shown in this figure are Intel-based
continental U.S. and the uniform distribution ensures a large vari-
machines running FreeBSD 4.5. At each edge of this network are a
ance in the values selected over this range. We configured the
set of 14 machines that run instances of a Web request generator
dummynet delays only on the browser’s outbound packets to sim-
(described below) each of which emulates the browsing behavior
plify the experimental setup. Most of the data transmitted in these
of thousands of human users. Also at each edge of the network is
experiments flow from the server to the browser and the TCP con-
another set of 8 machines that run instances of a Web response
generator (also described below) that creates the traffic flowing in
response to the browsing requests. A total of 44 traffic generating 1
machines are in the testbed. In the remainder of this paper we refer We use two 100 Mbps Ethernet segments and static routes to separate the
forward and reverse path flows in this configuration so we can use Ethernet
to the machines running the Web request generator simply as the hubs to monitor the traffic in each direction independently. Traffic on the
“browser machines” (or “browsers”) and the machines running the Gigabit link is monitored using passive fiber splitters to monitor each direc-
tion independently.

267
gestion control loop at the server (the one AQM causes to react) is Table 1: Elements of the HTTP traffic model.
influenced by the total RTT, not by asymmetry in the delays rela-
tive to the receiver’s side. Because these delays at the browsers Element Description
effectively delay the ACKs received by the servers, the round-trip Request size HTTP request length in bytes
times experienced by the TCP senders (servers) will be the combi- Response size HTTP reply length in bytes (top-level & embedded)
nation of the flow’s minimum delay and any additional delay in- Page size Number of embedded (file) references per page
troduced by queues at the routers or on the end systems. (End sys-
Think time Time between retrieval of two successive pages
tems are configured to ensure no resource constraints were present
hence delays there are minimal, ~1 millisecond.) A TCP window Persistent con- Number of requests per persistent connection
nection use
size of 16K bytes was used on all the end systems because widely
used OS platforms, e.g., most versions of Windows, typically have Servers per page Number of unique servers used for all objects in a
page
default windows this small or smaller.
Consecutive page Number of consecutive top-level pages requested
The instrumentation used to collect network data during experi- retrievals from a given server
ments consists of two monitoring programs. One program monitors
the router interface where we are examining the effects of the determine how many requests will use each persistent connection.
AQM algorithms. It creates a log of the queue size (number of One other parameter of the program is the number of parallel TCP
packets in the queue) sampled every 10 milliseconds along with connections allowed on behalf of each browsing user to make em-
complete counts of the number of packets entering the queue and bedded requests within a page. This parameter is used to mimic the
the number dropped. Also, a link-monitoring machine is connected parallel connections used in Netscape (typically 4) and Internet
to the links between the routers (through hubs on the 100 Mbps Explorer (typically 2).
segments or fiber splitters on the Gigabit link). It collects (using a For each request, a message of random size (sampled from the
locally-modified version of the tcpdump utility) the TCP/IP head- request size distribution) is sent over the network to an instance of
ers in each frame traversing the links and processes these in real- the server program. This message specifies the number of bytes the
time to produce a log of link utilization over selected time intervals server is to return as a response (a random sample from the distri-
(typically 100 milliseconds). bution of response sizes depending on whether it is a top-level or
embedded request). The server sends this number of bytes back
3.1 Web-Like Traffic Generation through the network to the browser. The browser is responsible for
The traffic that drives our experiments is based on a recent large- closing the connection after the selected number of requests have
scale analysis of web traffic [13]. The resulting model is an appli- completed (1 request for non-persistent connections and a random
cation-level description of the critical elements that characterize variable greater than 1 for persistent connections). For the experi-
how HTTP/1.0 and HTTP/1.1 protocols are used. It is based on ments reported here, the server’s “service time” is set to zero so the
empirical data and is intended for use in generating synthetic Web response begins as soon as the request message has been received
workloads. An important property of the model is that it reflects and parsed. This very roughly models the behavior of a Web server
the use of persistent HTTP connections as implemented in many or proxy having a large main-memory cache with a hit-ratio near 1.
contemporary browsers and servers. Further, the analysis presented For each request/response pair, the browser program logs its re-
in [13] distinguishes between Web objects that are “top-level” sponse time. Response time is defined as the elapsed time between
(typically an HTML file) and those that are embedded objects either the time of the socket connect() operation (for a non-
(e.g., an image file). At the time these data were gathered, ap- persistent connection) or the initial request (on a persistent connec-
proximately 15% of all TCP connections carrying HTTP protocols tion) or the socket write() operation (for subsequent requests on a
were effectively persistent (were used to request two or more ob- persistent connection) and the time the last byte of the response is
jects) but more than 50% of all objects (40% of bytes) were trans- returned. Note that this response time is for each element of a page,
ferred over these persistent connections. not the total time to load all elements of a page.
The model is expressed as empirical distributions describing the When all the request/response pairs for a page have been com-
elements necessary to generate synthetic HTTP workloads. The pleted, the emulated browsing user enters the thinking state and
elements of the model that have the most pronounced effects on makes no more requests for a random period of time sampled from
generated traffic are summarized in Table 1. Most of the behav- the think-time distribution. The number of page requests the user
ioral elements of Web browsing are emulated in the client-side makes in succession to a given server machine is sampled from the
request-generating program (the “browser”). Its primary parameter distribution of consecutive page requests. When that number of
is the number of emulated browsing users (typically several hun- page requests has been completed, the next server to handle the
dred to a few thousand). For each user to be emulated, the program next top-level request is selected randomly and uniformly from the
implements a simple state machine that represents the user’s state set of active servers. The number of emulated users is constant
as either “thinking” or requesting a web page. If requesting a web throughout the execution of each experiment.
page, a request is made to the server-side portion of the program
(executing on a remote machine) for the primary page. Then re- 3.2 Experiment Calibrations
quests for each embedded reference are sent to some number of Offered load for our experiments is defined as the network traffic
servers (the number of servers and number of embedded references resulting from emulating the browsing behavior of a fixed-size
are drawn as random samples from the appropriate distributions). population of web users. It is expressed as the long-term average
The browser also determines the appropriate usage of persistent throughput (bits/second) on an un-congested link that would be
and non-persistent connections; 15% of all new connections are generated by that user population. There are three critical elements
randomly selected to be persistent. Another random selection from of our experimental procedures that had to be calibrated before
the distribution of requests per persistent connection is used to performing experiments:

268
1. Ensuring that no element on the end-to-end path represented a 2e+08
primary bottleneck other than the links connecting the two 1.8e+08
routers when they are limited to 100 Mbps,
1.6e+08
2. The offered load on the network can be predictably controlled

Link throughput (bps)

using the number of emulated users as a parameter to the traffic 1.4e+08
generators, and
1.2e+08
3. Ensuring that the resulting packet arrival time-series (e.g.,
1e+08
packet counts per millisecond) is long-range dependent as ex-
pected because the distribution of response sizes is a heavy- 8e+07
tailed distribution [13].
6e+07
To perform these calibrations, we first configured the network 4e+07
connecting the routers to eliminate congestion by running at 1 Measured
10457.7012 * x + 423996
Gbps. All calibration experiments were run with drop-tail queues 2e+07
2000 4000 6000 8000 10000 12000 14000 16000 18000
having 2,400 queue elements (the reasons for this choice are dis- Browsers
cussed in Section 4). We ran one instance of the browser program
on each of the browser machines and one instance of the server Figure 2: Link throughput v. number of emulated browsing users
program on all the server machines. Each browser was configured compared to a straight line.
to emulate the same number of active users and the total active 1
users varied from 7,000 to 35,000 over several experiments. Figure 0.9
2 shows the aggregate traffic on one direction of the 1 Gbps link as 0.8
a function of the number of emulated users. The load in the oppo-

Cumulative probability
0.7
site direction was measured to be essentially the same and is not
plotted in this figure. The offered load expressed as link through- 0.6

put is a linear function of the number of emulated users indicating 0.5

there are no fundamental resource limitations in the system and 0.4

generated loads can easily exceed the capacity of a 100 Mbps link. 0.3
With these data we can determine the number of emulated users 0.2
that would generate a specific offered load if there were no bottle-
0.1
neck link present. This capability is used in subsequent experi- Empirical distribution
Generated response sizes
ments to control the offered loads on the network. For example, if 0
1 10 100 1000 10000 100000 1e+06 1e+07 1e+08 1e+09
we want to generate an offered load equal to the capacity of a 100 Response size (bytes)
Mbps link, we use Figure 2 to determine that we need to emulate
approximately 19,040 users (9,520 on each side of the link). Note Figure 3: CDF of empirical v. generated response sizes.
that for offered loads approaching saturation of the 100 Mbps link, 1
the actual link utilization will, in general, be less than the intended
Complementary cumulative probability

0.1
offered load. This is because as response times become longer,
users have to wait longer before they can generate new requests 0.01
and hence generate fewer requests per unit time.
0.001
A motivation for using Web-like traffic in our experiments was the
0.0001
assumption that properly generated traffic would exhibit demands
on the laboratory network consistent with those found in empirical 1e-05
studies of real networks, specifically, a long-range dependent
(LRD) packet arrival process. The empirical data used to generate 1e-06

our web traffic showed heavy-tailed distributions for both user 1e-07
“think” times and response sizes [13]. (Figures 3-4 compare the Empirical distribution
Generated response sizes
cumulative distribution function (CDF), F(x) = Pr[X ≤ x], and 1e-08
1 10 100 1000 100001000001e+06 1e+07 1e+08 1e+09
complementary cumulative distribution function (CCDF), 1 – F(x), Response size (bytes)
of the generated responses in the calibration experiments with the
empirical distribution from [13]. Note from Figure 3 that while the Figure 4: CCDF of empirical v. generated response sizes.
median response size in our simulations will be approximately
1,000 bytes, responses as large as 109 bytes will also be generated.) In all cases the 95% confidence intervals for the estimates fell in
the range 0.8 to 0.9 which indicates a significant LRD component
That our web traffic showed heavy-tailed distributions for both in the time series.
think times (OFF times) and response size (ON times), implies that
the aggregate traffic generated by our large collection of sources
should be LRD [14]. To verify that such LRD behavior is indeed 3.3 Experimental Procedures
realized with our experimental setup, we recorded tcpdumps of all Each experiment was run using the following automated proce-
TCP/IP headers during the calibration experiments and derived a dures. After initializing and configuring all router and end-system
time series of the number of packets and bytes arriving on the 1 parameters, the server programs were started followed by the
Gbps link between the routers in 1 millisecond time intervals. We browser programs. Each browser program emulated an equal num-
used this time series with a number of analysis methods (aggre- ber of users chosen, as described above, to place a nominal offered
gated variance, Whittle, Wavelets) to estimate the Hurst parameter. load on an unconstrained network. The offered loads used in the

269
experiments were chosen to represent user populations that could ering equivalent to 100 milliseconds at the link’s transmission
consume 80%, 90%, 98%, or 105% of the capacity of the 100 speed. FreeBSD queues are allocated in terms of a number of
Mbps link connecting the two router machines (i.e., consume 80, buffer elements (mbufs) each with capacity to hold an IP datagram
90, 98 or 105 Mbps, respectively). It is important to emphasize of Ethernet MTU size. For our experimental environment where
again that terms like “105% load” are used as a shorthand notation the link bandwidth is 100 Mbps and the mean frame size is a little
for “a population of web users that would generate a long-term over 500 bytes, this implies that a FIFO queue should have avail-
average load of 105 Mbps on a 1 Gbps link.” Each experiment was able about 2,400 mbufs for 100 milliseconds of buffering.
run for 120 minutes to ensure very large samples (over 10,000,000
Figures 5-7 give the response-time performance of a drop-tail
request/response exchanges in each experiment) but data were
queue with 24, 240 and 2,400 queue elements for offered loads of
collected only during a 90-minute interval to eliminate startup
80%, 90%, and 98% compared to the performance on the un-
effects at the beginning and termination synchronization anomalies
congested 1 Gbps link. Loss rates and link utilizations are given in
at the end. Each experiment for a given AQM schemes was re-
Table 2. At 80% load (80 Mbps on a 100 Mbps link) the results for
peated three times with a different set of random number seeds for
any queue size are indistinguishable from the results on the un-
each repetition. To facilitate comparisons among different AQM
congested link. At 90% load we see some significant degradation
schemes, experiments for different schemes were run with the
in response times for all queue sizes but note that, as expected, a
same sets of initial seeds for each random number generator (both
queue size of 24 or 240 elements is superior for responses that are
those in the traffic generators for sampling various random vari-
small enough to complete in under 500 milliseconds. For the
ables and in dummynet for sampling minimum per-flow delays).
longer queue of 2,400 elements performance is somewhat better
The key indicator of performance we use in reporting our results for the longer responses. At a load of 98% there is a severe per-
are the end-to-end response times for each request/response pair. formance penalty to response times but, clearly, a shorter queue of
We report these as plots of the cumulative distributions of response 240 elements is more desirable than one of 2,400 elements. In
times up to 2 seconds.2 In these plots we show only the results Figure 7 we also see a feature that is found in all our results at high
from one of the three repetitions for each experiment (usually there loads where there are significant numbers of dropped packets (see
were not noticeable differences between repetitions; where there Table 2). The flat area in the curves for 24 and 240 queue sizes
were, we always selected the one experiment most favorable to the shows the impact of RTO granularity in TCP — most responses
AQM scheme under consideration for these plots). We also report with a timeout take at least 1 second. In this study we made no
the fraction of IP datagrams dropped at the link queues, the link attempt to empirically determine the “optimal” queue size for the
utilization on the bottleneck link, and the number of re- drop-tail queue in our routers (which is likely to be somewhere
quest/response exchanges completed in the experiment. These between 300 and 2,000 elements). Finding such a drop-tail queue
results are reported in Table 2 where the values shown are means size involves a tradeoff between improving response times for the
over the three repetitions of an experiment. very large number of small objects versus the small number of
very large objects that consume most of the network’s capacity.
4 AQM EXPERIMENTS WITH PACKET DROPS Instead, we use a drop-tail queue of 240 elements as a baseline for
For both PI and REM we chose two target queue lengths to evalu- comparing with AQM mechanisms because it corresponds to one
ate: 24 and 240 packets. These were chosen to provide two operat- of the targets selected for AQM and provides reasonable perform-
ing points: one that potentially yields minimum latency (24) and ance for drop-tail.
one that potentially provides high link utilization (240). The values
used for the coefficients in the control equations above are those 4.1 Results for PI with Packet Drops
recommended in [1, 8] and confirmed by the algorithm designers. Figure 8 gives the results for PI at target queue lengths of 24 and
For ARED we chose the same two target queue lengths to evaluate. 240, and offered loads of 80%, 90%, and 98%. At 80% load, there
The calculations for all the ARED parameter settings follow the is essentially no difference in response times between the two tar-
guidelines given in [6] for achieving the desired target delay get queue lengths, and their performance is very close to that ob-
(queue size). In all three cases we set the maximum queue size to a tained on the un-congested network. At 90% load, the two target
number of packets sufficient to ensure tail drops do not occur. queue sizes provide nearly identical results except for the 10% of
responses requiring more than 500 milliseconds to complete. For
To establish a baseline for evaluating the effects of using various
these latter requests, the longer target size of 240 is somewhat
AQM designs, we use the results from a conventional drop-tail
better. At 98% load, the shorter queue target of 24 is a better
FIFO queue. In addition to baseline results for drop-tail at the
choice as it improves response times for shorter responses but does
queue sizes 24 and 240 chosen for AQM, we also attempted to find
not degrade response times for the longer responses.
a queue size for drop-tail that would represent a “best practice”
choice. Guidelines (or “rules of thumb”) for determining the “best”
allocations of queue size have been widely debated in various ven- 4.2 Results for REM with Packet Drops
ues including the IRTF end2end-interest mailing list. One guide- Figure 9 gives the results for REM at target queue lengths of 24
line that appears to have attracted a rough consensus is to provide and 240, and offered loads of 80%, 90%, and 98%. At 80% load,
buffering approximately equal to 2-4 times the bandwidth-delay there is essentially no difference in response times between the two
product of the link. Bandwidth in this expression is that of the link target queue lengths, and their performance is very close to that
and the delay is the mean round-trip time for all connections shar- obtained on the un-congested network. At 90% load, a queue refer-
ing the link — a value that is, in general, difficult to determine. ence of 24 performs much better than a target queue of 240. At
Other mailing list contributors have recently tended to favor buff- 98% load, a queue reference of 24 continues to perform slightly
better than 240. Overall, REM performs best when used with a
target queue reference of 24.
2
Because of space restrictions, only plots of summary results are shown
for105% load.

270
 100 100

80 80
Cumulative probability (%)

Cumulative probability (%)

60 60

40 40
Uncongested network
PI 80load - qref=24
20 PI 80load - qref=240
Uncongested network 20 PI 90load - qref=24
drop-tail - qlen=24 PI 90load - qref=240
drop-tail - qlen=240 PI 98load - qref=24
drop-tail - qlen=2400 PI 98load - qref=240
0 0
0 500 1000 1500 2000 0 500 1000 1500 2000
Response time (ms) Response time (ms)

Figure 5: Drop-tail performance, 80% offered load. Figure 8: PI performance with packet drops.
100 100

80 80
Cumulative probability (%)

Cumulative probability (%)

60 60

40 40
Uncongested network
REM 80load - qref=24
20 REM 80load - qref=240
Uncongested network 20 REM 90load - qref=24
drop-tail - qlen=24 REM 90load - qref=240
drop-tail - qlen=240 REM 98load - qref=24
drop-tail - qlen=2400 REM 98load - qref=240
0 0
0 500 1000 1500 2000 0 500 1000 1500 2000
Response time (ms) Response time (ms)

Figure 6: Drop-tail performance, 90% offered load. Figure 9: REM performance with packet drops.
100 100

80 80
Cumulative probability (%)

Cumulative probability (%)

60 60

40 40
Uncongested network
ARED 80load - thmin=12 thmax=36 w=1/8192
ARED 80load - thmin=120 thmax=360 w=1/8192
20 Uncongested network 20 ARED 90load - thmin=12 thmax=36 w=1/8192
drop-tail - qlen=24 ARED 90load - thmin=120 thmax=360 w=1/8192
drop-tail - qlen=240 ARED 98load - thmin=12 thmax=36 w=1/8192
drop-tail - qlen=2400 ARED 98load - thmin=120 thmax=360 w=1/8192
0 0
0 500 1000 1500 2000 0 500 1000 1500 2000
Response time (ms) Response time (ms)

Figure 7: Drop-tail performance, 98% offered load. Figure 10: ARED performance with packet drops.

tings for ARED are again almost indistinguishable from each other
4.3 Results for ARED with Packet Drops and response times, overall, are very poor.
Figure 10 gives the results for ARED at the two target queue
lengths, and offered loads of 80%, 90%, and 98%. These results Because of the consistently poor performance of ARED, we tried
are all quite surprising. In contrast to drop-tail, PI, and REM, the several different sets of parameters. We tried the recommended
results at 80% load show some degradation relative to the results settings for our network capacity of minth at 60, maxth at 180 and
on the un-congested link. At this load there is essentially no differ- wq at 1/16384. We also tried varying the parameter wq from 1/1024
ence in response times between the two target queue lengths. At up to 1/16384, but none of the settings yielded results that were
90% load, there is still little difference in performance among the better than the ones presented here. We cannot recommend a set-
two target queues for ARED but the overall degradation from an ting for ARED based on our experiments since the performance for
un-congested link is more substantial. At 98% load, the two set- all of them are very close to each other, and yet, unsatisfactory.

271
 100 1

Complementary cumulative probability (%)

0.1
80
Cumulative probability (%)

0.01

0.001
60

0.0001

40
1e-05

Uncongested network 1e-06 Uncongested network

20 drop-tail 90load - qlen=240 drop-tail 98load - qlen=240
PI 90load - qref=240 1e-07 PI 98load - qref=24
REM 90load - qref=24 REM 98load - qref=24
ARED 90load - thmin=120 thmax=360 w=1/8192 ARED 98load - thmin=12 thmax=36 w=1/8192
0 1e-08
0 500 1000 1500 2000 10 100 1000 10000 100000 1e+06 1e+07
Response time (ms) Reponse time (ms)

Figure 11: Comparison of all schemes at 90% load. Figure 14: CCDF of all schemes without ECN, 98% load.
100
stantial range of response times and comparable in the remaining
range. At 90% load, PI, REM, and drop-tail all provide reasonable
80 performance for the 80% of responses that can be completed in
Cumulative probability (%)

400 milliseconds or less. For the remaining 20% of responses, PI

60
with a target queue length of 240 is better than the other schemes.
Overall, PI with a target queue of 240 provides very good perform-
ance at this load. At 98% load, PI is again somewhat superior to
40 the other schemes but note that the best performance is obtained
with a target queue length of 24 and that overall, no AQM scheme
Uncongested network can offset the performance degradation at this extreme load. At
20 drop-tail 98load - qlen=240
PI 98load - qref=24
105% load performance for all schemes degrades uniformly from
REM 98load - qref=24 the 98% case. Table 2 also presents the link utilization, loss ratios,
ARED 98load - thmin=12 thmax=36 w=1/8192
0 and the number of completed requests for each experiment for each
0 500 1000 1500 2000 AQM scheme. At 90% and 98% offered loads, drop-tail with a
Response time (ms) queue of 240 gives slightly better link utilization than any of the
Figure 12: Comparison of all schemes at 98% load. AQM schemes. It also completes slightly more request-response
exchanges than the other schemes at the same load. Drop-tail does,
100
however, have higher loss ratios than the other schemes. PI has
better loss ratios than REM, completes more requests, and has
80
better link utilization at all loads.
Cumulative probability (%)

Figures 11-12 show that at least 90% of all responses complete in

under 2 seconds for the best AQM schemes. Figure 14 shows the
60
remainder of the distribution at 98% load. The conclusions drawn
from Figures 11-13 also hold for responses that experience re-
40 sponse times up to approximately 50 seconds (~99.95% of all re-
sponses). The remaining responses perform best under drop-tail.
Eventually ARED performance approaches that of drop-tail and is
Uncongested network
20 drop-tail 105load - qlen=240 superior to PI and REM but only for a handful of responses.
PI 105load - qref=24
REM 105load - qref=24
0
ARED 105load - thmin=12 thmax=36 w=1/8192 5 AQM EXPERIMENTS WITH ECN
0 500 1000 1500 2000 AQM schemes drop packets as an indirect means of signaling con-
Reponse time (ms) gestion to end-systems. The explicit congestion notification (ECN)
packet marking scheme was developed as a means of explicitly
Figure 13: Comparison of all schemes at 105% load.
signaling congestion to end-systems [12]. To signal congestion a
router can “mark” a packet by setting a specified bit in the TCP/IP
4.4 Comparing all Schemes with Packet Drops header of the packet. This marking is not modified by subsequent
At 80% load, all schemes but ARED perform comparably to an un- routers. Upon receipt of a marked data segment, a TCP receiver
congested network, and are barely distinguishable from each other. will mark the TCP header of its next outbound segment (typically
Figures 11-14 compare the best settings, based on the overall dis- an ACK) destined for the sender of the original marked segment.
tribution of response times, for each AQM scheme for offered Upon receipt of this marked segment, the original sender will react
loads of 90%, 98%, and 105%. In comparing results for two AQM as if a single segment had been lost within a send window. In addi-
schemes, we claim that the response time performance is better for tion, the sender will mark its next outbound segment (with a differ-
one of them if its CDF is clearly above the other’s in some sub- ent marking) to confirm that it has reacted to the congestion.

272
 Table 2: Loss, completed requests, and link utilizations. nificantly improves performance for PI at both target queue
lengths. REM shows the most significant improvement in perform-
Link
Offered Loss ratio
Completed
utilization/
ance with ECN. Although PI performed better than REM when
requests used without ECN at almost all loads, at 90% and 98% loads PI
Load (%) throughput
(millions) and REM with ECN give very similar performance, and ECN has a
(Mbps)
No No No significant effect on PI and REM performance in almost all cases.
ECN ECN ECN
ECN ECN ECN Overall, and contrary to the PI and REM results, ECN has very
Uncongested 80% 0 13.2 80.6 little effect on the performance of ARED at all tested target queue
1 Gbps lengths at all loads.
90% 0 15.0 91.3
network Table 2 again presents the link utilization, loss ratios, and the
(drop-tail) 98% 0 16.2 98.2
number of completed requests for each ECN experiment. PI with
105% 0 17.3 105.9
ECN clearly seems to have better loss ratios, although there is little
drop-tail 80% 0.2 13.2 80.3 difference in link utilization and number of requests completed.
queue size = 90% 2.7 14.4 88.4 REM’s improvement when ECN is used derives from lowered loss
24
98% 6.5 14.9 91.1 ratios, increases in link utilization, and increases in number of
105% 9.1 15.0 91.8 completed requests. With ARED, there is very little improvement
drop-tail 80% 0.04 13.2 80.6 in link utilization or number of completed requests. Its loss ratios
queue size = are only marginally better with ECN.
90% 1.8 14.6 89.9
240
98% 6.0 15.1 92.0
5.1 Comparisons of PI, REM, & ARED with ECN
105% 8.8 15.0 92.4 Recall that at 80% load, no AQM scheme provides better response
drop-tail 80% 0 13.1 80.4 time performance than a simple drop-tail queue. This result is not
queue size = 90% 0.1 14.7 88.6 changed by the addition of ECN. Here we compare the best set-
2,400 tings for PI, REM, and ARED when combined with ECN for loads
98% 3.6 15.1 91.3
105% 7.9 15.0 91.1 of 90%, 98%, and 105%. The results for drop-tail (queue length
240) are also included as a baseline for comparison. Figures 21-23
PI 80% 0 0 13.3 13.2 80.2 79.3
qref = 24
show these results. At 90% load, both PI and REM provide re-
90% 1.3 0.3 14.4 14.6 87.9 88.6
sponse time performance that is both surprisingly close to that on
98% 3.9 1.8 15.1 14.9 89.3 89.4 an un-congested link and is better than drop-tail. At 98% load there
105% 6.5 2.5 15.1 15.0 89.9 89.5 is noticeable response time degradation with either PI or REM,
PI 80% 0 0 13.1 13.1 80.1 80.1 however, the results are far superior to those obtained with drop-
qref = 240 90% 0.1 0.1 14.7 14.7 87.2 88.2 tail. Further, both PI and REM combined with ECN have substan-
98% 3.7 1.7 14.9 15.1 90.0 89.6 tially lower packet loss rates than drop-tail and link utilizations that
are only modestly lower. At 105% load the performance of PI and
105% 6.9 2.3 15.0 15.2 90.5 90.8
REM is virtually identical and only marginally worse than was
REM 80% 0.01 0 13.2 13.1 79.8 80.1 observed at 98% load. This is an artifact of our traffic generation
qref = 24 90% 1.8 0.1 14.4 14.6 86.4 88.2 model wherein browsers generate requests less frequently as re-
98% 5.0 1.7 14.5 14.9 87.6 89.6 sponse times increase. Table 2 shows that few additional request-
105% 7.7 2.4 14.6 14.9 87.5 89.3 response exchanges are completed at 105% load than at 98% load.
REM 80% 0 0 13.2 13.2 79.3 80.3 For ARED, even when used with ECN, response time performance
qref = 240 at all load levels is significantly worse than PI and REM except for
90% 3.3 0.2 14.0 14.7 83.3 88.6
the shortest 40% of responses where performance is comparable.
98% 5.4 1.6 14.4 15.1 86.2 90.4
105% 7.3 2.3 14.6 15.1 87.7 90.4 Figure 24 shows the tails of the response time distribution at 98%
load. For AQM with ECN, drop-tail again eventually provides
ARED 80% 0.02 0.03 13.0 12.9 79.4 78.8
better response time performance, however, the crossover point
thmin = 12 90% 1.5 1.3 13.8 13.8 85.5 85.5
thmax= 36 occurs earlier, at approximately 5 seconds. The 1% of responses
98% 4.1 4.1 14.0 13.9 87.4 88.0 experiencing response times longer than 5 seconds receive better
wq = 1/8192
105% 5.1 5.1 14.1 14.1 87.3 87.7 performance under drop-tail. ARED performance again eventually
ARED 80% 0.02 0.02 13.0 13.1 80.2 80.5 approaches that of drop-tail for a handful of responses.
thmin = 120 90% 1.4 1.2 14.0 14.1 85.5 86.2
thmax = 360 6 DISCUSSION
98% 4.8 4.7 14.2 14.1 87.9 88.2
wq = 1/8192
105% 6.8 6.3 13.9 13.9 85.2 85.8 Our experiments have demonstrated several interesting differences
in the performance of Web browsing traffic under control theoretic
We repeated each of the above experiments with PI, REM, and and pure random-dropping AQM. Most striking is the response
ARED using packet marking and ECN instead of packet drops. Up time performance achieved under PI and REM with ECN at loads
to 80% offered load, ECN has no effect on response times of any of 90% and 98%. In particular, at 90% load response time per-
of the AQM schemes. Figures 15-20 show the results for loads of formance surprisingly approximates that achieved on an un-
90% and 98%. At 90% load, with target queue length of 24, PI congested network. Approximately 90% of all responses complete
performs better with ECN, however, with a target queue length of in 500 milliseconds or less whereas only approximately 95% of
240, there is little change in performance. At 98% load, ECN sig- responses complete within the same threshold on the un-congested
network.

273
 100 100

80 80

Cumulative probability (%)

Cumulative probability (%)

60 60

40 40

Uncongested network Uncongested network

drop-tail 90load - qlen=240 drop-tail 98load - qlen=240
20 20 PI 98load - qref=24
PI 90load - qref=24
PI 90load - qref=240 PI 98load - qref=240
PI/ECN 90load - qref=24 PI/ECN 98load - qref=24
PI/ECN 90load - qref=240 PI/ECN 98load - qref=240
0 0
0 500 1000 1500 2000 0 500 1000 1500 2000
Response time (ms) Response time (ms)

Figure 15: PI performance with/without ECN, 90% load. Figure 16: PI performance with/without ECN, 98% load.
100 100

80 80
Cumulative probability (%)

Cumulative probability (%)

60 60

40 40

Uncongested network Uncongested network

drop-tail 90load - qlen=240 drop-tail 98load - qlen=240
20 REM 90load - qref=24 20 REM 98load - qref=24
REM 90load - qref=240 REM 98load - qref=240
REM/ECN 90load - qref=24 REM/ECN 98load - qref=24
REM/ECN 90load - qref=240 REM/ECN 98load - qref=240
0 0
0 500 1000 1500 2000 0 500 1000 1500 2000
Response time (ms) Response time (ms)

Figure 17: REM performance with/without ECN, 90% load. Figure 18: REM performance with/without ECN, 98% load.
100 100

80 80
Cumulative probability (%)
Cumulative probability (%)

60 60

40 40

Uncongested network Uncongested network

drop-tail 90load - qlen=240 drop-tail 98load - qlen=240
20 ARED 90load - thmin=12 thmax=36 w=1/8192 20 ARED 98load - thmin=12 thmax=36 w=1/8192
ARED 90load - thmin=120 thmax=360 w=1/8192 ARED 98load - thmin=120 thmax=360 w=1/8192
ARED/ECN 90load - thmin=12 thmax=36 w=1/8192 ARED/ECN 98load - thmin=12 thmax=36 w=1/8192
ARED/ECN 90load - thmin=120 thmax=360 w=1/8192 ARED/ECN 98load - thmin=120 thmax=360 w=1/8192
0 0
0 500 1000 1500 2000 0 500 1000 1500 2000
Response time (ms) Response time (ms)

Figure 19: ARED performance with/without ECN, 90% load. Figure 20: ARED performance with/without ECN, 98% load.

To better understand PI’s distributions of response times and the enables the vast majority of all responses to experience response
positive impact of ECN, Figures 25-26 show scatter plots of re- times proportional to their RTT divided by their window size. This
sponse size versus response time for PI at 98% load. (In interpret- is seen in Figure 26 by observing the dense triangle-shaped mass
ing these plots it is important to remember that the median re- of points starting at the origin and extending outward to the points
sponse size is under 1,000 bytes and the 90th percentile response is (100,000, 6,000) and (100,000, 500). (Note as well the existence of
slightly over 10,000 bytes (see Figure 3).) For small responses, similar triangles offset vertically by multiples of 1 second — the
strong banding effects are seen at multiples of 1 second represent- canonical packet loss timeout.)
ing the effects of timeouts. Of special interest is the density of the
The second most striking result is that performance varied substan-
band at 6 seconds representing the effects of a dropped SYN seg-
tially between PI and REM with packet dropping and this perform-
ment. While it appears that PI forces a large number of small re-
ance gap was closed through the addition of ECN. A preliminary
sponses to experience multi-second response times, PI in fact does
analysis of REM’s behavior suggests that ECN is not so much
better in this regard than all the other AQM schemes. With the
improving REM’s behavior as it is ameliorating a fundamental
addition of ECN, the number of timeouts is greatly reduced and PI
design problem. Without ECN, REM consistently causes flows to

274
 100 1

Complementary cumulative probability (%)

0.1
80
Cumulative probability (%)

0.01

60 0.001

0.0001
40
1e-05

Uncongested network
20 drop-tail 90load - qlen=240 1e-06 Uncongested network
PI/ECN 90load - qref=24 drop-tail 98load - qlen=240
REM/ECN 90load - qref=24 1e-07 PI/ECN 98load - qref=24
ARED/ECN 90load - thmin=120 thmax=360 w=1/8192 REM/ECN 98load - qref=24
0 ARED/ECN 98load - thmin=12 thmax=36 w=1/8192
0 500 1000 1500 2000 1e-08
Response time (ms) 10 100 1000 10000 100000 1e+06 1e+07
Reponse time (ms)
Figure 21: Comparison of all schemes with ECN, 90% load.
100
Figure 24: CCDF of all schemes with ECN, 98% load.

The final point of note is the difference in performance between

80 ARED and the other AQM schemes, in particular, the fact that
Cumulative probability (%)

response time performance is consistently worse with ARED than

60
with drop-tail. The exact reasons for the observed differences re-
mains the subject of continued study, however, the experiments
reported here and others lead us to speculate that there are three
40 primary factors influencing these results. First, PI and REM oper-
ate in “byte mode” by default — they monitor the queue length in
Uncongested network bytes rather than packets. While ARED also has a byte mode,
20 drop-tail 98load - qlen=240
PI/ECN 98load - qref=24
“packet mode” is the recommended setting. Byte mode allows for
REM/ECN 98load - qref=24 finer grain queue measurements but more importantly, in PI and
ARED/ECN 98load - thmin=12 thmax=36 w=1/8192
0 REM the marking/dropping probability for an individual packet is
0 500 1000 1500 2000 biased by a factor equal to the ratio of the current packet size to the
Response time (ms)
average (or maximum) packet size. This means that in PI and
Figure 22: Comparison of all schemes with ECN, 98% load. REM, SYNs and pure ACKs experience a lower drop probability
than data segments arriving at the router at the same time and
100
hence fewer SYNs and ACKs are dropped than under ARED.
Second, in ARED’s “gentle mode,” when the average queue size is
80
between maxth and 2xmaxth, ARED drops ECN-marked packets,
Cumulative probability (%)

following the ECN guidelines that state packets should be dropped

60 when the AQM scheme’s queue length threshold is exceeded in
this manner. The motivation for this rule is to more effectively
deal with potential non-responsive flows that are ignoring conges-
40
tion indications [12]. Our analysis indicates that this rule is in fact
counter-productive and explains much of ARED’s inability to
20
Uncongested network benefit from ECN.
drop-tail 105load - qlen=240
PI/ECN 105load - qref=24
REM/ECN 105load - qref=24 Finally, PI and REM periodically sample the (instantaneous) queue
0
ARED/ECN 105load - thmin=12 thmax=36 w=1/8192 length when deciding to mark packets. ARED uses a weighted
0 500 1000 1500 2000 average. We believe that the reliance on the average queue length
Reponse time (ms) significantly limits ARED’s ability to react effectively in the face
Figure 23: Comparison of all schemes with ECN, 105% load. of highly bursty traffic such as the Web traffic generated herein.
Of note was the fact that changing ARED’s weighting factor for
computing average queue length by an order of magnitude had no
experience multiple drops within a source’s congestion window,
effect on performance.
forcing flows more frequently to recover the loss through TCP’s
timeout mechanism rather than its fast recovery mechanism. When
ECN is used, REM simply marks packets and hence even if multi- 7 CONCLUSIONS
ple packets from a flow are marked within a window the timeout From the results reported above we draw the following conclu-
will be avoided. Thus ECN appears to improve REM’s perform- sions. These conclusions are based on a premise that user-
ance by mitigating the effects of its otherwise poor (compared to perceived response times are the primary yardstick of performance
PI) marking/dropping decisions. and that link utilization and packet loss rates are important but
secondary measures.

275
 10000 10000

8000 8000

Response time (ms)

Response time (ms)

6000 6000

4000 4000

2000 2000

0 0
0 20000 40000 60000 80000 100000 0 20000 40000 60000 80000 100000
Response size (bytes) Response size (bytes)

Figure 25: Scatter plot of PI performance without ECN, 98% load. Figure 26: Scatter plot of PI performance with ECN, 98% load.

• For offered loads up to 80% of bottleneck link capacity, no 9 REFERENCES

AQM scheme provides better response time performance than
simple drop-tail FIFO queue management. Further, the response [1] S. Athuraliya, A Note on Parameter Values of REM with Reno-like
times achieved on a 100Mbps link are not substantially different Algorithms, http://netlab.caltech.edu, March 2002.
from the response times on a 1 Gbps link with the same number [2] S. Athuraliya, V. H. Li, S.H. Low, Qinghe Yin, REM: Active Queue
of active users that generate this load. This result is not changed Management, IEEE Network, Vol. 15, No. 3, May 2001, pp. 48-53.
by combining any of the AQM schemes with ECN.
[3] B. Braden, et al, Recommendations on Queue Management and Con-
• For loads of 90% of link capacity or greater, PI results in a mod- gestion Avoidance in the Internet, RFC 2309, April, 1998.
est improvement over drop-tail and the other AQM schemes
when ECN is not used. [4] M. Christiansen, K. Jeffay, D. Ott, and F.D. Smith, Tuning RED for
Web Traffic, Proc., ACM SIGCOMM 2000, Sept. 2000, pp. 139-150.
• With ECN, both PI and REM provide significant response time
improvement at offered loads at or above 90% of link capacity. [5] W. Feng, D. Kandlur, D. Saha, K. Shin, A Self-Configuring RED
Moreover, at a load of 90%, PI and REM with ECN provide per- Gateway, Proc., INFOCOM ‘99, March 1999, pp. 1320-1328.
formance on a 100 Mbps link competitive with that achieved [6] S. Floyd, R. Gummadi, S. Shenker, Adaptive RED: An Algorithm for
with a 1 Gbps link with the same number of active users. Increasing the Robustness of RED’s Active Queue Management,
• ARED with recommended parameter settings consistently re- http://www.icir.org/floyd/papers/ adaptiveRed.pdf, August 1, 2001.
sulted in the poorest response time performance. This result was [7] S. Floyd, and V. Jacobson, Random Early Detection Gateways for
not changed with the addition of ECN. Congestion Avoidance, IEEE/ACM Transactions on Networking,
We conclude that without ECN there is little end-user performance Vol. 1 No. 4, August 1993, p. 397-413.
gain to be realized by employing any of the AQM schemes studied [8] C.V. Hollot, V. Misra, W.-B. Gong, D. Towsley, On Designing Im-
here. However, with ECN performance can be significantly im- proved Controllers for AQM Routers Supporting TCP Flows, Proc.,
proved at near-saturation loads with either PI or REM. Thus, it IEEE INFOCOM 2001, April 2001, pp. 1726-1734
appears likely that provider links may be operated at 80% of ca- [9] L. Rizzo, Dummynet: A simple approach to the evaluation of network
pacity even when not deploying any AQM (with or without ECN). protocols, ACM CCR, Vol. 27, No. 1, January 1997, pp. 31-41.
Further, providers may be able to operate their links at even higher
[10] C. Kenjiro, A Framework for Alternate Queueing: Towards Traffic
load levels without significant degradation in user-perceived per- Management by PC-UNIX Based Routers, Proc., USENIX 1998 An-
formance provided PI or REM combined with ECN is deployed in nual Technical Conf., New Orleans LA, June 1998, pp. 247-258.
their routers and ECN is implemented in TCP/IP stacks on the end-
[11] V. Misra, W.-B. Gong,, D. Towsley, Fluid-based Analysis of a Net-
systems. work of AQM Routers Supporting TCP Flows with an Application to
RED, Proc., ACM SIGCOMM 2000, pp. 151-160.
8 ACKNOWLEDGEMENTS [12] K. Ramakrishnan, S. Floyd, D. Black, The Addition of Explicit Con-
We are indebted to Sanjeewa Athuraliya, Sally Floyd, Steven Low, gestion Notification (ECN) to IP, RFC 3168, September 2001.
Vishal Misra, and Don Towsley, for their assistance in performing
[13] F.D. Smith, F. Hernandez Campos, K. Jeffay, D. Ott, What TCP/IP
the experiments described herein. In addition, we are grateful for Protocol Headers Can Tell Us About the Web, Proc. ACM SIGMET-
the constructive comments of the anonymous referees and for the RICS 2001, June 2001, pp. 245-256.
help of our shepherd, Dina Katabi.
[14] W. Willinger, M.S. Taqqu, R. Sherman, D. Wilson, Self-similarity
This work was supported in parts by the National Science Founda- through high variability: statistical analysis of ethernet LAN traffic at
tion (grants ITR-0082870, CCR-0208924, and EIA-0303590), the source level, IEEE/ACM Transactions on Networking, Vol. 5, No.
Cisco Systems Inc., and the IBM Corporation. 1, February 1997, pp. 71-86.

276