RIGHT-SIZING CABLE MODEM BUFFERS FOR MAXIMUM

APPLICATION PERFORMANCE
Greg White (CableLabs®)
Richard Woundy, Yiu Lee, Carl Williams (Comcast)

Abstract significant impact on performance – either

imposing limits on upstream data rates or
The sizing of the cable modem (CM) upstream increasing delay for latency sensitive
buffer can significantly impact application applications.
performance and end-user quality of experience. Too
little buffering will result in poor TCP throughput in the Historically, upstream buffer
upstream direction. Too much buffering will have a
implementations in CMs have been sized
negative impact on latency sensitive applications when
the upstream link is heavily utilized. Studies have (statically) by the modem vendor, and the
indicated that cable modems in the field today have network operator has had no control or
perhaps an order of magnitude greater buffering than visibility as to the size of the buffer
is needed to ensure TCP throughput, with the result implementation. In order to ensure optimal
being a sluggish user experience for web browsing, and performance across the wide range of
outright failure of VoIP and real-time games at times potential upstream configured data rates,
when the upstream is being used simultaneously by modem suppliers have sized their upstream
TCP.
buffers such that there is sufficient buffering
This paper provides a background on the network
for TCP to work well at the highest possible
dynamics involved, describes a new feature in the upstream data rate, and at the greatest
DOCSIS® 3.0 specification that allows the operator to expected round trip time (i.e. the greatest
tune the upstream buffer, and provides some initial possible bandwidth delay product). In the
experimental work that seeks to provide guidance to majority of deployment scenarios, the
operators in the use of this feature. configured upstream data rate is significantly
less than the highest possible rate, and the
average TCP round trip time is significantly
INTRODUCTION less than the greatest expected. The result is
that in the majority of cases the cable modem
Buffering of data traffic is a critical has an upstream buffer that greatly exceeds
function implemented by network elements what is necessary to ensure optimal TCP
(e.g. hosts, switches, routers) that serves to performance.
minimize packet loss and maximize efficient
use of the next-hop link. It is generally In today's mix of upstream TCP and UDP
thought that for the Transmission Control traffic, some of which is latency sensitive,
Protocol (TCP) to work most efficiently, each some of which is not, the impact of a vastly
element in a network needs to support oversized upstream buffer is that the upstream
sufficient buffering to allow the TCP window TCP sessions attempt to keep the buffer as
to open up to a value greater than or equal to full as possible, and the latency sensitive
the product of the next hop link bandwidth traffic can suffer enormously. The result is
and the total round trip time for the TCP poorer than expected application performance
session (the bandwidth delay product). In for VoIP, gaming, web surfing, and other
many residential broadband scenarios, the applications. Studies have shown upstream
cable modem is the head of the bottleneck buffering in deployed cable modems on the
link in the upstream direction, and as a result order of multiple seconds.
its buffering implementation can have a
 Recent work has established a standardized Two key congestion control algorithms are
means by which a cable operator can set the “slow start” and “congestion avoidance”. In
upstream buffer size at the cable modem via slow start, the sender increases the congestion
the modem's configuration file (alongside window by one segment for each TCP
other service defining parameters, such as the acknowledgement (ACK) received; this action
maximum traffic rate). This paper provides doubles the transmission rate for each
an introduction to this new capability, roundtrip time, leading to exponential growth.
investigates the performance impact of buffer Once in congestion avoidance, the sender
size on a range of traffic scenarios, and seeks increases the congestion window by one
to provide some initial guidance for operators segment per roundtrip time; this action
to properly configure buffer size for their increases the transmission rate linearly. If a
service offerings. segment is dropped due to a timeout, the
threshold for entering congestion avoidance is
set to one-half the current congestion window,
RESEARCH AND BEST PRACTICES and the congestion window itself is reset to
RELATING TO BUFFER SIZING one segment. This “sawtooth” behavior is
illustrated in Figure 1 [16].
The TCP [2] is a connection-oriented, end-
to-end protocol that enables the reliable
transmission of data across the Internet. The
TCP is designed to recover data that is
damaged, lost, duplicated, or delivered out of
order by the network.

The original TCP specification provides

reliability and end-to-end flow control
through the use of sequence numbers and the
use of a “receive window”. All transmitted Figure 1
data is tracked by sequence number, which
enables the TCP receiver to detect missing or Modern TCP implementations also
duplicate data. The receive window enables implement “fast retransmit” and “fast
flow control by identifying the range of recovery” algorithms. According to fast
sequence numbers that the receiver is retransmit, when the sender receives three
prepared to receive, to prevent overflow of the duplicate ACKs, the sender assumes a
receiver’s data buffer space. segment has been lost and re-transmits that
segment. If that segment is subsequently
However, these controls were insufficient transmitted successfully, then under fast
to avoid “congestion collapse” in the Internet recovery the sender cuts its congestion
in the mid-1980s [4]. Van Jacobson [1] window in half (rather than resetting to one
developed the initial mechanisms for TCP segment).
congestion control, which many researchers
have continued to extend and to refine [6]. In Using modern congestion control, the TCP
particular, TCP congestion control utilizes a sender is constantly attempting to increase its
“congestion window” to limit the transmission transmission rate to the maximum possible,
rate of the sender in response to network and it decreases the transmission rate in the
congestion indications. presence of segment loss. As a result of this
behavior, the TCP sender will send packets at
a continuously increasing rate, filling the
buffer of the network device at the head of the minimum delay was typically under 15
bottleneck link, until one or more packets are milliseconds, the upstream buffer sizes were
dropped. (This discussion ignores the found to be quite large – resulting in 600
presence of Active Queue Management [3] or milliseconds of queuing delay for most DSL
Explicit Congestion Notification [5], neither links, and several seconds for many cable
of which are commonly implemented in modems. These results suggest that broadband
broadband modems.) buffer sizes are much larger than the BDP
rule-of-thumb.
Since the TCP sender’s behavior is to keep
the buffer full at the bottleneck link, it is Gettys [17] shows that such large buffer
helpful to review the research sizes, or “bufferbloat”, is an issue today with
recommendations for the size of network broadband modems both for wireline and for
buffers. wireless networks. As one example, some
have observed six seconds of latency in 3G
Back in 1994, Villamizar and Song [7] wireless networks related to queuing. The
provided performance test results using up to issue of bufferbloat has also been observed in
eight TCP flows over a national backbone other consumer equipment, such as home
infrastructure. Their results led to the rule-of- routers and laptops. Gettys points to excessive
thumb that bottleneck links need a buffer size buffering at the head of the bottleneck link as
equal to the bandwidth delay product (BDP) – being disruptive to many user applications.
that is, equal to the available bandwidth at the
bottleneck multiplied by the roundtrip time of
the TCP sessions – for the TCP flows to CABLE MODEM UPSTREAM
saturate the bottleneck link. A decade later, BUFFERING AND IMPACTS ON
Appenzeller, Keslassy and McKeown [8] APPLICATION PERFORMANCE/USER
confirmed the BDP rule-of-thumb for a single EXPERIENCE
long-lived TCP flow, which may apply best to
a bottleneck link at the edge of the network, In the upstream traffic direction, the cable
such as a broadband modem’s upstream link. modem is generally the entry point of the
bottleneck link, both topologically and in
Since that time, the topic of buffering at terms of link capacity. As a result of this, its
network hops has become an active one, with buffer size can have a significant impact on
some beginning to look at the impact that the application performance. As noted previously,
rule-of-thumb has on non-TCP traffic, as well TCP upload performance can be sensitive to
as whether the rule-of-thumb holds up under buffer size on the bottleneck link, and
more real-world conditions. Vishwanath, et al. furthermore will seek to keep the bottleneck
[10] provide a good survey of recent work in link buffer full. When latency-sensitive or
this area. loss-sensitive traffic is mixed with TCP traffic
in the same cable modem service flow, issues
If the optimal buffer size for TCP can arise as a result of the modem buffer
performance is presumed to be equal to the being kept full by the TCP sessions.
bandwidth delay product, then one might
inquire how that compares to the buffer size One impact will be that applications will
implemented in current broadband modems. see an upstream latency that can be predicted
In 2007, Dischinger et al. [9] measured the by the buffer size divided by the upstream
performance of a number of DSL and cable configured rate. For example, in the case of a
operators in Europe and in North America. 256KiB buffer and a configured upstream rate
While the measured broadband modem of 2Mb/s, the upstream latency would be
expected to be on the order of 1 second. More most sensitive to latency, where some games
buffering will result in a proportional increase require that players have less than 130ms
in latency. round-trip time between their host and the
game server in order to be allowed to play,
Another impact will be that upstream and lower is generally better.
applications will experience packet loss due to
buffer overflow. Bulk TCP sessions are Even more latency sensitive are games that
immune to packet loss resulting from buffer are run on a cloud server, with a video stream
overflow, and in fact their congestion sent to the player, and a control stream sent
avoidance algorithm relies on it. However, from the player back to the server. Services
other applications may be more sensitive. In such as these may require a maximum of 20-
contrast to the latency impact, the packet loss 50ms round-trip time across the cable
rate will increase if the buffer is undersized. operator's network.

While not typically considered to be a

How Much Buffering Should There Be?
latency sensitive application, web-browsing
activities can also be impacted by upstream
Commonly used VoIP and video latency. Consider that a webpage load
conferencing applications such as Vonage, consists of a cascade of perhaps a dozen or
Skype, iChat, FaceTime, and umi, provide a more round trips to do DNS lookups and
better user experience when end-to-end HTTP object fetches. An upstream latency on
latency is kept low. The ITU has developed the order of 500ms or more is likely to cause
models and recommendations for end-to-end the web browsing experience to be much
delay for voice and other interactive services. slower than a user might find acceptable,
The ITU-T Recommendation G.114 [11] particularly if they are paying for a
suggests one-way latency be kept below downstream connection speed measured in
150ms in order to achieve "essentially tens of megabits per second.
transparent interactivity", and that delays
greater than 400ms are unacceptable for In addition, any downstream-centric TCP
interactive applications. Furthermore it has application (such as web browsing) has the
defined the "E-model" (ITU-T G.107 [12], potential to be throttled due to the latency
G.108 [13] & G.109 [14]) used for predicting experienced by the upstream TCP-ACK
digital voice quality based on system stream from the receiver. In practice this is
parameters. The E-model has been reduced to not an issue, because many cable modems
a simpler form for VoIP applications by Cole support proprietary means to accelerate TCP-
& Rosenbluth, which points to significant ACKs by queuing them separately from bulk
degradation in voice quality beginning when upstream data.
the one-way latency exceeds ~177ms. Packet
loss also degrades quality, with losses of less How Much Queuing Latency Is There?
than 2.5% being necessary to achieve "Best"
quality. Included in the packet loss statistic
are any packets that exceed the ability of the Historically, cable modems have
receiver de-jitter buffer to deliver them implemented static buffer sizes regardless of
isochronously. upstream data rate. Evidence suggests that
cable modems in the field today may have
Multiplayer online games can also be buffers that are sized (using the BDP rule-of-
sensitive to latency and packet loss. Fast- thumb) for the maximum possible upstream
paced, interactive games are typically the data rate (~25Mb/s for DOCSIS 2.0 CMs and
~100Mb/s for current DOCSIS 3.0 CMs) and offered some implementation advantages.
the maximum coast-to-coast Round Trip Time When presented with a Buffer Control TLV,
(RTT) that might be experienced (100ms or such a CM would then choose a number of
more). blocks that results in a buffer size that is
within the range of acceptable sizes, and is as
Our observations indicate that most close to the target buffer size as possible.
conventional cable modems are built with
buffer size between 60KiB and 300KiB. The Buffer Control TLV is comprised of
three parameters:
Since the majority of modems (particularly
those in residential service) are currently Minimum Buffer - defines the lower limit
operated with upstream rates in the range of 1 of acceptable buffer sizes. If the device
to 2Mb/s, the result (as corroborated by cannot provide at least this amount of
Dischinger [9]) is that the modems have buffering, it will reject the service flow. If
buffering latencies on the order of several this parameter is omitted, there is no
seconds, which may be 1 or 2 orders of minimum buffer size.
magnitude greater than would be ideal.
Target Buffer - defines the desired size of
the buffer. The device will select a buffer size
DOCSIS 3.0 BUFFER CONTROL that is as close to this value as possible, given
FUNCTIONALITY its implementation. If this parameter is
omitted, the device selects any buffer size
A recent addition to the DOCSIS 3.0 within the allowed range, via a supplier-
specifications provides, for the first time, the specific algorithm.
ability for cable operators to tune thetransmit
buffers for cable modems and CMTSs in their Maximum Buffer - defines the upper limit
networks. This new feature, support for of acceptable buffer sizes. If the device
which became mandatory for cable modems cannot provide a buffer size that is less than or
submitted for CableLabs certification in early equal to this size, it will reject the service
April 2011 (and is optional for CMTSs), gives flow. If this parameter is omitted, there is no
the operator control of the configured buffer maximum buffer size.
size for each DOCSIS Service Flow. The
feature controls upstream buffering in the Each of these parameters is defined in
cable modem, and downstream buffering in bytes, with an allowed range of 0 -
the CMTS. 4,294,967,295 (4 GiB).

The new feature is referred to as Buffer As noted, each of the three parameters is
Control, and is defined as a Quality of Service optional. If all three parameters are omitted,
(QoS) Parameter of a DOCSIS Service Flow. the interpretation by the device is that there is
Specifically, the Buffer Control parameter is a no limitation (minimum or maximum) on
set of Type-Length-Value (TLV) tuples that allowed buffer size for this service flow, and
define a range of acceptable buffer sizes for that the device should select a buffer size via a
the service flow, as well as a target size. This vendor specific algorithm. This is exactly the
allows the CM/CMTS implementer some situation that existed prior to the introduction
flexibility on the resolution of its buffer of this feature. As a result, if the operator
configuration. For example, a CM does not change its modem configurations to
implementer might choose to manage include Buffer Control, the operator should
buffering in blocks of 1024 bytes, if that see no difference in behavior between a
modem that supports this new feature and one necessary in order to ensure that the CMTS
that does not. properly sends the TLVs to the CM,
regardless of which of the above methods is
In many configuration cases it will be most utilized.
appropriate to omit the Minimum Buffer and
Maximum Buffer limits, and to simply set the
Target Buffer. The result is that the modem EXPERIMENTAL WORK
will not reject the service flow due to
buffering configuration, and will provide a Methodology
buffer as close as the implementation allows
to the Target Buffer value.
The new Buffer Control parameters
In certain cases however, the operator may described in a previous section enable us to
wish to be assured that the buffer is within provision the buffer size in the cable modem,
certain bounds, and so would prefer an and thus provide the opportunity to
explicit signal (i.e., a rejection of the investigate the application performance
configuration) if the modem cannot provide a impact in various scenarios. For the purposes
buffer within those bounds. Hard limits are of this paper, we ran three sets of tests:
provided for these cases.
In the first set of tests, we correlate the
The cable modem is required to support buffer size and upstream speed. The purpose
buffer configurations of up to 24KiB per of the test is to understand the impact buffer
service flow, but is expected to support size would have on the ability of the customer
significantly more, particularly when a small to utilize his/her provisioned upstream
number of service flows is in use. Put another bandwidth. As an illustration, if the round-
way, if the Minimum Buffer parameter is set trip time for a TCP session is 40ms and the
to a value less than or equal to 24576, the upstream service is provisioned for 5Mb/s, the
cable modem is guaranteed not to reject the rule-of-thumb for buffer size would indicate
configuration due to that parameter value. If 5000Kb/s * 0.04s = 200Kb or 24.4KiB. If the
the Minimum Buffer parameter is set to a buffer size was then set to 16KiB, the
value that is greater than 24576, then there is expectation is that a single TCP session would
some risk that a modem implementation will not be able to utilize the 5Mb/s service. In this
reject the configuration. set of tests, we validate that expectation, and
we additionally look at the effect when
Since they are part of the QoS Parameter multiple simultaneous TCP sessions are
Set, the Buffer Control TLVs can be set sharing the upstream link.
directly in the cable modem's configuration
boot file; they can be set indirectly via a In the second set of tests we correlate the
named Service Class defined at the CMTS; buffer size and QoS of real-time applications
and they can be set and/or modified via the during upstream self-congestion (i.e.
PacketCableTM Multimedia (PCMM) interface saturation of the user's allotted bandwidth by a
to the CMTS. mix of applications). During congestion,
packets from real-time applications may
In order to use this new feature to control queue up in the buffer. If the buffer size is too
upstream buffering in DOCSIS 3.0 cable large, this will increase the latency and may
modems, it is necessary that the CM software severely impact real-time applications such as
be updated to a version that supports it. online games and Voice over IP (VoIP). This
Furthermore, CMTS software updates are set of tests examines whether changing the
buffer size can improve real-time gateway between the CM and the rest of the
applications’ QoS or not. home network equipment. To eliminate any
bias that might be introduced by the packet
The third set of tests examines the forwarding performance of a home gateway,
implications of buffer size when the token we instead used a Gigabit switch to connect
bucket feature defined in the DOCSIS 3.0 the home network equipment to the CM. We
specification is utilized to provide the user a then configured the CM to allow multiple
high burst transmit rate (as is currently the devices connecting to it.
practice by several operators). In this test, the
token bucket is set to 5MB and, after a PSTN

suitable idle period, a 5MB file is transferred ICMP Test

using a single TCP session (FTP). The TCP

Latency
Test
Server VoIP

session runs through the process discussed

Latency Gateway
Test
Client

earlier, beginning with slow start and RTP Packets

transitioning to congestion avoidance. Since MOS Tester ATA

Cable
Modem
CMTS Router Internet

the token bucket size is equal to the size of the Cable

Modem
CMTS

file, the rate shaping function in the CM

Network Network
Test System Test System
Sender Receiver

doesn't come into play, and thus the limits to Web

Server Dum m yNet
Web
Client

the transfer speed in this test are: a) the HTTP Upload Test

maximum speed of the upstream channel,

which in this case is approximately 25Mb/s;
and b) ability of a single TCP session to Figure 2
utilize the link in the presence of limited
buffering. The CM was installed with the new
DOCSIS 3.0 firmware that provides the
We used upstream buffer sizes of 8KiB, capability of using the Buffer Control feature
16KiB, 32KiB, 64KiB, 128KiB, and 256KiB, so that specific buffer queue sizes might be
although in some test scenarios we limited the set. A CMTS with knowledge of such settings
choice of buffer sizes due to time constraints. was connected to the CM. Inside the home
In the experiments the upstream bandwidth network, there are four applications: (1) a
was provisioned to 1Mb/s, 2Mb/s, 5Mb/s, or PING test client, (2) a VoIP Analog Terminal
10Mb/s. These upstream speeds are indicative Adaptor (ATA), (3) a Network Test System,
of services currently available in the market. and (4) a Web Server. The PING test client
The tests simulated RTTs of 40ms and 100ms. measured the RTT to the PING server in the
40ms represents a local file transfer within a test network. This was set up to validate the
metropolitan network region (perhaps from a PING latency while the network was
cache). 100ms represents a coast-to-coast congested. The VoIP ATA was set up for the
session. QoS test for real-time applications. A VoIP
tester was connected to the ATA. This tester
Network Setup would measure the Mean Opinion Score
(MOS) when the ATA made a phone call to
Figure 2 depicts the experimental setup, the VoIP gateway in the Internet. The
which emulates a typical home network Network Test System allowed an easy way to
behind a CM. The home network consisted of simulate multiple TCP devices
PCs running test applications and other simultaneously. The Web server was setup to
hardware for measuring network generate packets over TCP by uploading a
characteristics and performance. Typical large file via HTTP to the client behind the
home networks would include a home CMTS. In order to simulate the network
latency, Dummynet was used. Dummynet
[18] is an internal packet filter for FreeBSD Multi-TCP Data Rate (non-rate-limited)
which can inject artificial delay in the
network. It was connected between the CM 30
and Web server. Delay was injected in both 25

Data Rate (Mb/s)

upstream and downstream. For example: we 20
configured Dummynet to inject 20ms in both 15
directions for the HTTP upload test to 10
simulate 40ms RTT delay. 5
0

Baseline Results 8 16 32 64 128 256

Buffer Size (KiB)

As an initial test, we configured the CM 40ms 100ms

with an upstream rate limit of 35Mb/s 40ms - BDP 100ms - BDP

(effectively no rate-limiting), and investigated
the achievable TCP data rate for a range of Figure 3
buffer sizes, both for the 40ms RTT case and
for the 100ms RTT case. Figure 3 shows the
resulting achieved data rate for 10
simultaneous upstream TCP sessions, along Test 1 Results
with the data rate that would be predicted
based on the BDP. The maximum data rate
that a single DOCSIS upstream channel could Figure 4 and Figure 5 show the results of
support is approximately 25Mb/s, so the data rate in various buffer sizes. Figure 4
achieved rates for 256KiB, 128KiB and shows the data rate of 1Mb/s and 2Mb/s
perhaps 64KiB should be considered to be upstream services. Figure 5 shows the data
artificially limited by this constraint. It is rate of 5Mb/s and 10Mb/s upstream services.
interesting to note that in nearly all cases, The tests were executed with either 40ms or
when multiple TCP sessions were utilized, we 100ms delay. All the tests were run with a
were able to achieve a higher data rate (and in single TCP stream.
some cases a much higher data rate) than
would be predicted based on the rule-of- Data Rate (1 Mbps/2 Mbps)
thumb. Our conjecture is that the multiple 2
TCP sessions were desynchronized (that is,
1.5
the TCP sawtooths were not synchronized
Data Rate (Mb/s)

across sessions), and Appenzeller, Keslassy 1

and McKeown [8] demonstrate that 0.5

significantly less buffering than the bandwidth
0
delay product is needed in this scenario.
16 64 256

Buffer Size (KiB)

2Mb/s (40msRTT)
2Mb/s (100msRTT)
1Mb/s (40msRTT)
1Mb/s (100msRTT)

Figure 4
 Data Rate (5 Mbps/10 Mbps)
single TCP session and 10 TCP sessions.
Both were able to achieve the full 10Mb/s rate
10 limit.
Data Rate (Mb/s)

8
6 1 TCP vs 10 TCPs (40ms RTT)
12
4

Data Rate (Mb/s)

10
2 8
0 6
16 64 256 4
2
Buffer Size (KiB)
0
10Mb/s (40msRTT) 10Mb/s (100msRTT)
16 64
5Mb/s (40msRTT) 5Mb/s (100msRTT)
Buffer Size (KiB)
1 TCP (40ms RTT) 10 TCP (40ms RTT)

Figure 5
Figure 6a
For 1Mb/s and 2Mb/s services, 16KiB buffer
size was sufficient to utilize the upstream Figure 6b shows further results with a 100ms
bandwidth with either 40ms and 100ms delay. delay. Bandwidth utilization could be
For 5Mb/s, results showed that at least 64KiB improved for both 16KiB and 64KiB buffers
buffer was needed when delay was 100ms. when aggregating 10 TCP sessions. The test
For 10Mb/s, results showed that 64KiB buffer using 16KiB buffer size with a 100ms delay
was needed for 40ms delay and 256KiB resulted in 4.95Mb/s upstream utilization, a
buffer was needed for 100ms delay. three times improvement over the test that
was conducted using a single TCP sessions.
From the previous section, the tests However, 5Mb/s of capacity remains unused.
performed with 10Mb/s upstream service for When using a 64KiB buffer size with 10 TCP
16KiB and 64KiB showed significant sessions, there was improvement with a data
bandwidth under-utilization. The question rate of 8.77Mb/s compared to 6.05Mb/s for
becomes: is it possible to achieve utilization the single TCP flow test. Here again 1Mb/s
of the whole bandwidth with multiple parallel of capacity was unused.
TCP sessions? In additional experiments we
used 10 TCP sessions established in parallel
1 TCP vs 10 TCPs (100ms RTT)
recording individual data rates and provided 10
the aggregation plots in the graph provided in
Data Rate (Mb/s)

8
Figure 6a and Figure 6b for 10Mb/s upstream 6
service with buffer sizes of 16KiB and 64KiB. 4
2
Figure 6a shows that with a 16KiB buffer,
0
a single TCP session with 40ms delay is
16 64
unable to achieve the full upstream rate, with
only 1.97Mb/s of the 10Mb/s upstream Buffer Size (KiB)
1 TCP (100ms RTT) 10 TCP (100ms RTT)
service being used. By aggregating 10 TCP
sessions and keeping other factors fixed, the
channel utilization improved by a factor of Figure 6b
four to 7.11Mb/s. Still, full utilization wasn’t
realized. When 64KiB buffer size was used, Figure 7 and Figure 8 show the results of
we saw no difference in throughput between a packet latency when an upstream link was
congested with a single TCP session. Figure 7 upstream service, the PING latency was
shows the latency of 1Mb/s and 2Mb/s 52.04ms for a 16KiB buffer. The PING
upstream services. Figure 8 shows the latency latency increased to 200ms for a 64KiB buffer
of 5Mb/s and 10Mb/s upstream services when and 1019.17ms for a 256KiB buffer.
the upstream link was congested with a single However, this pattern was not observed when
TCP session. using a 10Mb/s service for 16KiB and 64KiB
buffer sizes. This is because a single TCP
PING Latency (1 Mbps/2 Mbps) session was unable to utilize the full 10Mb/s
10000
service with 40ms and 100ms delay.
Therefore, the PING packets proceeded
without congestion impact.
Latency (ms)

1000

100 Test 2 Results

10 In this test, we wanted to understand how

16 64 256 buffer size would impact the VoIP MOS
Buffer Size (KiB) under congestion. The VoIP ATA was
connected into the simulated home network in
2Mb/s (40msRTT) 2Mb/s (100msRTT) our test environment. We started a large
1Mb/s (40msRTT) 1Mb/s (100msRTT) upstream file transfer (using a single TCP
session) to simulate a self-congested
Figure 7 upstream. The Dummynet test set injected
either 40ms or 100ms round-trip latency to the
PING Latency (5 Mbps/10 Mbps)
TCP stream. The two latency values impact
the dynamics of the TCP congestion
1000
avoidance algorithm differently, and thus
Latency (ms)

result in a different pattern of CM buffer

100 usage. For each bandwidth/buffer/delay
permutation, we ran 10 tests and computed
10
the average MOS.
16 64 256

Buffer Size (KiB)

MOS vs Buffer Size (1Mbps)
5
10Mb/s (40msRTT) 10Mb/s (100msRTT)
4
5Mb/s (40msRTT) 5Mb/s (100msRTT)
MOS

2
Figure 8
1
For all upstream services, the pattern is that 8 16 32 64 128 256
the bigger the buffer, the higher the PING Buffer Size (KiB)
latency. The observed pattern of the test
results show (as expected) that the latency is 1Mbps (40ms) 1Mbps (100ms)
directly proportional to the buffer size. If the
buffer size is increased to four times the size,
Figure 9a
the latency would increase roughly four times.
For example, when testing with a 2Mb/s
 Figure 9a shows the results when the both of these cases, the best MOS results were
upstream was configured to 1Mb/s. In this recorded when the buffer size was set between
case, the 16KiB buffer gave the best MOS 32KiB and 64KiB. What was interesting is
result regardless of the TCP RTT. When the when the buffer was set below 64KiB, there
buffer size increased to 64KiB, the MOS should not be any congestion because the
started to drop. In fact, when the buffer size buffer size was too small for the competing
was set to 256KiB with 100 ms delay, many TCP session to utilize the full upstream
tests couldn’t be completed. The test set bandwidth. Also when the buffer size was set
reported a Timeout. The timeout could be to 8KiB or 16KiB, the VoIP traffic should see
caused by long delay introduced by the large very low buffering latency (between 6ms and
buffer. 26ms). Both of these factors should result in
high MOS scores. However, the MOS score
was degraded when the buffer size was set to
MOS vs Buffer Size (2Mbps) 8KiB or 16KiB, and this low MOS is likely to
be caused by packet loss due to the periodic
5
saturation of the small CM buffer by the
4 competing TCP congestion window.
MOS

2
MOS vs Buffer Size (10Mbps)
1
8 16 32 64 128 256 5

Buffer Size (KiB) 4

MOS

3
2Mbps (40ms) 2Mbps (100ms)
2

1
Figure 9b 8 16 32 64 128 256

Buffer Size (KiB)

Similar results were obtained for 2MB/s, as
shown in Figure 9b. 10Mbps (40ms) 10Mbps (100ms)

Figure 9d
MOS vs Buffer Size (5Mbps)
5 When we increased the buffer size above
4 128KiB, the MOS started to deteriorate. In
MOS

3 fact, the test set failed to complete some tests

2
and required us to re-run the test when we set
1
the upstream bandwidth to 10Mb/s with
8 16 32 64 128 256
256KiB buffer and 100ms delay. This could
Buffer Size (KiB) be caused by the long delay of the VoIP
signaling packets for the call setup.
5Mbps (40ms) 5Mbps (100ms)

Another interesting observation is that,

Figure 9c while adding latency to the TCP session did in
some cases affect the VoIP MOS, it didn't
Figures 9c and 9d show the results of point to a different optimal buffer size.
5Mb/s and 10Mb/s upstream bandwidths. For
 Average Throughput
100.0
Test 3 Results

Throughput (Mbps)
10.0
In this test, the CM's rate-shaping token
bucket depth was set to 5MB to mimic the
configuration that some operators use to boost 1.0
performance for bursty interactive traffic. The
test set was then used to transit a 5MB file in
the upstream direction using FTP. The test 0.1

measured the transfer time and, from this, an 8 16 32 64 128 256

average data rate was calculated. Due to the Buffer Size (KiB)
40ms 100ms
fact that the file size was equal to the token
bucket depth, there was effectively no
upstream rate-shaping in play. Figure 10 Figure 11
shows the result of Average Transfer Time for
the scenario with 40ms RTT and the scenario
with 100ms RTT. FUTURE WORK

Additional work is necessary to more fully

Average Transfer Time
120 understand the impact of CM buffering on
100
application performance and user experience.
The overarching goal of this work is to enable
Transfer Time (s)

80
cable operators to optimize network
60 performance and user experience across a
40 wide range of application and usage scenarios.
20
The experimental work described in this
0 paper focused both on upstream bulk TCP
8 16 32 64 128 256 performance, and on the impact that bulk TCP
Buffer Size (KiB) traffic would have on a VoIP session across a
40ms 100ms range of buffer sizes in fairly isolated and
controlled scenarios. Further work will seek
Figure 10
to examine the effect of buffering on some
more real-world application scenarios, such as
Figure 11 shows the calculated average a scenario in which TCP sessions are
throughput based on the average transfer time numerous and short-lived, such that many of
and the file size. It is clear that the choice of them stay in the slow-start phase for their
buffer size can have a dramatic impact on entire duration, and never (or rarely) enter the
achievable throughput during this congestion avoidance phase. Additionally,
performance boost period. It is important to future work should assess the impact of buffer
note that the average throughput shown in sizing on other latency or loss sensitive
Figure 11 is for the entire file transfer, and so applications beyond VoIP.
includes the slow-start effect as well as the
initial FTP hand-shaking.
 CONCLUSIONS Further work is needed to evaluate the
performance impact that buffer size has on a
We have shown that the size of the wider range of application scenarios.
upstream service flow buffer in the cable
modem can have significant effects on
application performance. For applications REFERENCES
that use TCP to perform large upstream
transmissions (i.e. upstream file transfers such [1] V. Jacobson, Congestion Avoidance and
as uploading a large video clip), insufficient Control, ACM SIGCOMM '88, August
buffering capacity can limit throughput. 1988.
Applications utilizing a single TCP session
see the most pronounced effect. Applications [2] J. Postel, Transmission Control Protocol,
that are latency sensitive, on the other hand, STD 7, RFC793, September 1981.
see degraded performance when too much
[3] Braden, B., et al., Recommendations on
buffer capacity is provided.
Queue Management and Congestion
Cable modems deployed today appear to Avoidance in the Internet, RFC 2309,
have significantly greater buffering than is April 1998.
needed to sustain TCP throughput, potentially
[4] S. Floyd, Congestion Control Principles,
causing high latency when the upstream is
BCP 41, RFC 2914, September 2000.
congested. The result is poorer than expected
application performance for VoIP, gaming, [5] K. Ramakrishnan, S. Floyd and D. Black,
Web surfing, and other applications that are The Addition of Explicit Congestion
latency-sensitive. Notification (ECN) to IP, RFC 3168,
September 2001.
A new feature for DOCSIS 3.0 CMs allows
operators to configure the upstream buffer [6] M. Allman, V. Paxson and E. Blanton,
size for each upstream service flow in order to TCP Congestion Control, RFC 5681,
optimize application performance and to September 2009.
improve user experience. In choosing the
buffer size, the operator will need to consider [7] C. Villamizar and C. Song, High
the upstream QoS parameters for the service Performance TCP in ANSNet. ACM
flow, the expected application usage for the CCR, 24(5):45-60, 1994.
flow, as well as the service goals for the flow.
[8] G. Appenzeller, I. Keslassy, and N.
Service features that utilize a large token McKeown, Sizing Router Buffers, ACM
bucket size (in order to provide high SIGCOMM, USA, 2004.
throughput for short bursts) complicate
matters since the buffer size cannot [9] M. Dischinger, et al., Characterizing
realistically be resized in real time. Thus a Residential Broadband Networks,
buffer configuration that may be optimized to Proceedings of the 7th ACM SIGCOMM
provide a good balance in performance Conference on Internet Measurement,
between TCP uploads and real-time services Oct. 24-26, 2007, San Diego, CA, USA.
for the configured sustained traffic rate, may
result in poorer than expected burst speeds. [10] A. Vishwanath, V. Sivaraman, M.
Thottan, Perspectives on Router Buffer
Sizing: Recent Results and Open
Problems, ACM CCR, 39(2):34-39, 2009.
[11] ITU-T Recommendation G.114, One-way
Transmission Time, http://www.itu.int

[12] ITU-T Recommendation G.107, The E-

model, a computational model for use in
transmission planning, http://www.itu.int

[13] ITU-T Recommendation G.108,

Application of the E-model: A planning
guide, http://www.itu.int

[14] ITU-T Recommendation G.109,

Definition of categories of speech
transmission quality, http://www.itu.int

[15] R. G. Cole and J. H. Rosenbluth, Voice

over IP performance monitoring,
ACMCCR, 31(2), 2001.

[16] Internetworking lectures from La Trobe

University,
http://ironbark.bendigo.latrobe.edu.au/sub
jects/INW/lectures/19/, 2011.

[17] J. Gettys, Bufferbloat: Dark Buffers in

the Internet,
http://mirrors.bufferbloat.net/Talks/Pragu
eIETF/IETFBloat7.pdf, April 2011.

[18] L. Rizzo, Dummynet: a simple approach

to the evaluation of network protocols,
ACM Computer Communication Review,
Vol. 27, No. 1, pp. 31-41, January 1997.