CUBIC: A New TCP-Friendly High-Speed TCP Variant: Sangtae Ha, Injong Rhee Lisong Xu
CUBIC: A New TCP-Friendly High-Speed TCP Variant: Sangtae Ha, Injong Rhee Lisong Xu
12000
(packets)
10000 Concave Growth
10000
8000
6000 Convex Growth
1000
4000
2000 Multiplicative DecreaseCUBIC flow1
100 CUBIC flow2
0
0 100 200 300 400 500 600
Time (second)
10
1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 (a) CUBIC window curves.
Random Loss Rate (packets)
400
(a) Networks with 10ms RTT. 350
1e+06
Standard TCP 300
HSTCP
CUBIC 250
(Mbps)
100000
Avg. Window Size (packets)
200
150
10000
100
50 CUBIC flow1
1000 CUBIC flow2
0
0 100 200 300 400 500 600
100 Time (second)
3000
tocols. Figure 8 (a) and 8 (b) show the intra-protocol fair-
2000 ness and link utilization for the tested protocols. CUBIC
1000
and BIC-TCP show higher fairness index than TCP-SACK
and HSTCP, representing better fair sharing between the
0 flows. With 16ms RTT, TCP-SACK shows the best fairness
0 100 200 300 400 500 600
Time (second) index indicating that Standard TCP works fairly well un-
(b) RTT 82ms. der small RTT networks. CUBIC, BIC-TCP, and HSTCP
utilize the link regardless of RTTs while TCP-SACK suffers
Figure 5: One CUBIC flow and one TCP-SACK under-utilization with larger RTTs.
flow. Bandwidth is set to 400Mbps.
6.3 Inter-RTT Fairness
2500
TCP-SACK flow1
We measure the fairness in sharing the bottleneck band-
TCP-SACK flow2 width between two competing flows that have different RTTs.
TCP-SACK flow3
2000 TCP-SACK flow3 For this experiment, we fix RTT of one flow to 162ms and
CUBIC flow1
CUBIC flow2 vary RTT of the other flow between 16ms and 164ms. This
1500 CUBIC flow3
(packets)
CUBIC flow4 setting gives us the RTT ratio up to 10. We test CU-
1000
BIC, BIC-TCP, HSTCP, and TCP-SACK protocols. Fig-
ure 9 (a) shows that TCP-SACK achieves RTT fairness lin-
500 early proportional to the inverse of the RTT ratio, which
means that the short RTT flow has proportionally more
0 bandwidth shares than the longer RTT flow. Even though
0 100 200 300 400 500 600
Time (second)
there is no commonly accepted notion of RTT-fairness, we
think the proportional fairness of TCP-SACK is desirable
because long RTT flows tend to use more resources along
Figure 6: Four TCP-SACK flows and four CUBIC
the longer path than short RTT flows. But some of high-
flows over 40ms RTT
speed protocols are desiged to provide an equal bandwidth
sharing among the flows with different RTTs (e.g., H-TCP
FreeBSD systems are shown in [5]. Note that Netem [22] in and FAST). Based on this notion of RTT fairness, if the
Linux provides the same functionality with the dummynet RTT fairness of a protocol has a similar slope with TCP-
software in FreeBSD. For this experiment, the maximum SACK, we can say the protocol is “acceptable”. Figure 9
bandwidth of the bottleneck router is set to 400Mbps. The (a) confirms that CUBIC has a similar slope with TCP-
bottleneck buffer size is set to 100% BDP if it is not explic- SACK but with a higher fairness ratio indicating better
itly specified. The amount of background traffic comparable share of resources (bandwidth) while HSTCP fails in achiev-
to around 15% of the bottleneck bandwidth is pushed into ing a similar slope. Even though BIC-TCP shows the simi-
forward and backward directions of the dumbbell. We use lar slope with TCP-SACK, it shows the lowest fairness ra-
the drop-tail router at the bottleneck. tios among tested protocols. This is what CUBIC improves
over BIC-TCP for RTT-fairness. We also observe that even
6.2 Intra-Protocol Fairness though two HSTCP flows fully utilize the link regardless of
We measure the intra-protocol fairness between two flows their RTT ratio (See Figure 9 (b)), the slow convergence of
of a protocol with the same RTT. We use a throughput HSTCP flows hinders even two flows of the same RTT from
ratio between these two flows for representing the intra- reaching to a fair share within a reasonable amount of time.
protocol fairness. This metric represents a degree of band-
width shares between two flows of the same protocol. For 6.4 Impact on standard TCP traffic
this experiment, we vary RTTs between 16ms and 324ms As many new high-speed TCP protocols modify the win-
and test CUBIC, BIC-TCP, HSTCP, and TCP-SACK pro- dow growth function of TCP in a more scalable fashion,
1 1
0.9
0.8
Throughput Ratio
Throughput Ratio
0.8
0.6
0.7
0.4
0.6
TCP-SACK 0.2 TCP-SACK
0.5 HSTCP HSTCP
BIC-TCP BIC-TCP
CUBIC CUBIC
0.4 0
0 50 100 150 200 250 300 350 0 20 40 60 80 100 120 140 160 180
RTT (ms) RTT (ms)
90 90
80 80
Link Utilization
Link Utilization
70 70
60 60
TCP-SACK TCP-SACK
50 HSTCP 50 HSTCP
BIC-TCP BIC-TCP
CUBIC CUBIC
40 40
0 50 100 150 200 250 300 350 0 20 40 60 80 100 120 140 160 180
RTT (ms) RTT (ms)
Figure 8: Intra-protocol fairness (a) and link uti- Figure 9: Inter-RTT fairness. The bottleneck band-
lization (b). The bottleneck bandwidth is set to width is set to 400Mbps and 2Mbyte buffer is used.
400Mbps and 2Mbyte bottleneck buffer is used. One flow has a fixed RTT of 162ms and the other
RTT is varied between 16ms and 324ms and two flow varies its RTT from 16ms to 162ms.
flows have the same RTT.
Throughput (Mbps)
liness by having some form of “TCP modes” during which 250
they behave in the same way as TCP. BIC-TCP and HSTCP 200
enter their TCP mode when the window size is less than 150
algorithm and implementation. Through extensive testing, (c) RTT 100ms with varying bandwidth
we confirm that CUBIC tackles the shortcomings of BIC-
TCP and achieves fairly good Intra-protocol fairness, RTT- Figure 10: Impact on standard TCP traffic.
fairness and TCP-friendliness.
[2] Iperf. http://sourceforge.net/projects/iperf . [7] Aikat, J., Kaur, J., Smith, F., and Jeffay, K.
Variability in TCP round-trip times. In Proceedings of the
[3] Linux CUBIC source navigation. ACM SIGCOMM Internet Measurement Conference
http://lxr.linux.no/linux/net/ipv4/tcp cubic.c. (Miami, FL, October 2003).
[4] TCP Testing Wiki. [8] Andrew, L., Marcondes, C., Floyd, S., Dunn, L.,
http://netsrv.csc.ncsu.edu/wiki/index.php/TCP Testing. Guillier, R., Gang, W., Eggert, L., Ha, S., and Rhee,
†
I. Towards a common TCP evaluation suite. In Proceedings
We used the latest update of CUBIC (v2.2) which improved of the fourth PFLDNet Workshop (UK, March 2008).
the scalability and convergence speed of the protocol, which
doesn’t clamp the increment in both convex and cocave re- [9] Barford, P., and Crovella, M. Generating
gions. A slight increase of aggressivenss is the trade-off be- representative web workloads for network and server
tween scalability and TCP-friendlines. Our extensive testing performance evaluation. In Measurement and Modeling of
confirms that CUBIC has better scalability and convergence Computer Systems (1998), pp. 151–160.
speed with this small change (trade-off) while obtaining rea- [10] Brakmo, L., and Peterson, L. TCP vegas: End to end
sonable TCP-friendliness. congestion avoidance on a global internet. IEEE Journal of
Selected Areas in Communications (October 1995).
[11] Bullot, H., Cottrell, R. L., and Hughes-Jones, R.
Evaluation of advanced TCP stacks on fast long-distance
production networks. In Proceedings of the third PFLDNet
Workshop (Illinois, February 2004). Version Kernel Updates
[12] Cai, H., Eun, D., Ha, S., Rhee, I., and Xu, L. Stochastic 2.0-pre 2.6.13 The authors releases the first CUBIC
ordering for internet congestion control and its
implementation in Linux to the Linux
applications. In Proceedings of IEEE INFOCOM
(Anchorage, Alaska, May 2007). community [18].
[13] Caini, C., and Firrincieli, R. TCP hybla: a TCP 2.0 2.6.15 CUBIC is officially included in the
enhancement for heterogeneous networks. International Linux kernel.
Journal of Satellite Communication and Networking 22, 5
(September 2004), 547–566. 2.6.18 CUBIC replaces BIC-TCP as the de-
[14] Casetti, C., Gerla, M., Mascolo, S., Sanadidi, M. Y., fault TCP protocol in Linux kernel.
and Wang, R. TCP Westwood: Bandwidth estimation for 2.6.19 The original implementation of CUBIC
enhanced transport over wireless links. In Proceedings of
has a scaling bug. It has taken about
ACM Mobicom (Rome, Italy, July 2001).
a month to fix this bug since CUBIC
[15] Floyd, S. HighSpeed TCP for Large Congestion Windows.
RFC 3649 (Experimental), Dec. 2003.
replaced BIC-TCP.
[16] Floyd, S., Handley, M., and Padhye, J. A Comparison 2.6.21 Its original implementation by the au-
of Equation-Based and AIMD Congestion Control, May thors are optimized by the Linux de-
2000. veloper for better performance [20, 29].
[17] Fu, C. P., and Liew, S. C. TCP Veno: TCP Enhancement In particular, the cubic root calculation
for Transmission Over Wireless Access Networks. IEEE in CUBIC, originally implemented in
Journal of Selected Areas in Communications (Feb 2003). the bisection method, is now replaced
[18] Ha, S. Cubic v2.0-pre patch. by a Newton-Raphson method with ta-
http://netsrv.csc.ncsu.edu/twiki/pub/Main/BIC/cubic-
ble loopups for small values. This re-
kernel-2.6.13.patch.
sults in more than 10 times perfor-
[19] Hatano, T., Fukuhara, M., Shigeno, H., and Okada, K.
TCP-friendly SQRT TCP for High Speed Networks. In
mance improvement in the cubic root
Proceedings of APSITT (November 2003), pp. 455–460. calculation. On average, the bisection
[20] Hemminger, S. Cubic root benchmark code. method costs 1032 clocks while the im-
http://lkml.org/lkml/2007/3/13/331 . proved version costs only 79 clocks.
[21] Hemminger, S. Linux TCP Performance Improvements. 2.1 2.6.22 The original implementation of CU-
Linux World 2004 (2004). BIC clamped the maximum window in-
[22] Hemminger, S. Network Emulation with NetEm. Linux crement to 32 packets per RTT. This
Conf Au (2005). feature is inherited from BIC-TCP
[23] Hemminger, S. TCP infrastructure split out. (Smax ). An extensive lab testing con-
http://lwn.net/Articles/128626/ (2005). firmed that CUBIC can safely remove
[24] Jin, C., Wei, D. X., and Low, S. H. FAST TCP: this window clamping in the concave
motivation, architecture, algorithms, performance. In
Proceedings of IEEE INFOCOM (Hong Kong, March
region. This enhances the scalability of
2004). CUBIC over very large BDP network
[25] Kelly, T. Scalable TCP: Improving performance in paths. This is incorporated in CUBIC
highspeed wide area networks. ACM SIGCOMM Computer 2.1 (Linux 2.6.22).
Communication Review 33, 2 (April 2003), 83–91. 2.6.22-rc4 CUBIC improves slow start for fast
[26] Liu, S., Basar, T., and Srikant, R. TCP-Illinois: A loss start-up by removing initial ssthresh.
and delay-based congestion control algorithm for
high-speed networks. In Proceedings of VALUETOOLS 2.6.23 The use of received timestamp op-
(Pisa, Italy, October 2006). tion value from RTT calculation is
[27] Rhee, I., and Xu, L. CUBIC: A new TCP-friendly removed for preventing possible ma-
high-speed TCP variant. In Proceedings of the third licious receiver attacks that reports
PFLDNet Workshop (France, February 2005). wrong timestamps to reduce RTTs for
[28] Shorten, R. N., and Leith, D. J. H-TCP: TCP for more throughput.
high-speed and long-distance networks. In Proceedings of
the Second PFLDNet Workshop (Argonne, Illinois, 2.2 2.6.25-rc3 The window clamping during the con-
February 2004). vex growth phase is also removed. This
[29] Tarreau, W. Cubic optimization. feature allows CUBIC to improve its
http://git.kernel.org/?p=linux/kernel/git/davem/net- convergence speed while maintaining
2.6.git;a=commit;h=7e58886b45bc4a309aeaa8178ef89ff767daaf7f . its fairness and TCP friendliness.
[30] Xu, L., Harfoush, K., and Rhee, I. Binary increase
congestion control for fast long-distance networks. In
Proceedings of IEEE INFOCOM (Hong Kong, March Table 1: CUBIC version history
2004).
[31] Zhang, Y., Duffield, N., Paxson, V., and Shenker, S.
On the constancy of Internet path properties. In
Proceedings of ACM SIGCOMM Internet Measurement
Workshop (November 2001).