Core-Stateless Fair Queueing:

A Scalable Architecture to
Approximate Fair Bandwidth Allocations
in High Speed Networks
Ion Stoica,Scott Shenker, and Hui Zhang
SIGCOMM’98, Vancouver, August 1998
subsequently
IEEE/ACM Transactions on Networking
11(1), 2003, pp. 33-46.

Presented by Bob Kinicki

1
 Outline
 Introduction
 Core-Stateless Fair Queueing (CSFQ)
– Fluid Model Algorithm
– Packet Algorithm
– Flow Arrival Rate
– Link Fair Share Rate Estimation
 NS Simulations
 Conclusions

Advanced Computer Networks: CSFQ Paper 2

 Introduction
 This paper brings forward the concept of “fair”
allocation.
 The claim is that fair allocation inherently
requires routers to maintain state and perform
operations on a per flow basis.
 The authors present an architecture and a set
of algorithms that is “approximately” fair while
using FIFO queueing at internal routers.

Advanced Computer Networks: CSFQ Paper 3

 An “Island” of Routers

Destination
Source

Edge
Router

Core
Router

Destination

Advanced Computer Networks: CSFQ Paper 4

 Outline
 Introduction
 Core-Stateless Fair Queueing (CSFQ)
– Fluid Model Algorithm
– Packet Algorithm
– Flow Arrival Rate
– Link Fair Share Rate Estimation
 NS Simulations
 Conclusions

Advanced Computer Networks: CSFQ Paper 5

Core-Stateless Fair Queueing
 Ingress edge routers compute per-flow
rate estimates and insert these estimates
as labels into each packet header.
 Only edge routers maintain per flow state.
 Labels are updated at each router based
only on aggregate information.
 FIFO queuing with probabilistic dropping
of packets on input is employed at the
core routers.

Advanced Computer Networks: CSFQ Paper 6

Edge – Core Router Architecture

Advanced Computer Networks: CSFQ Paper 7

 Fluid Model Algorithm
 Assume the bottleneck router has an output
link with capacity C.
 Assume each flow’s arrival rate, ri (t) , is
known precisely.
The main idea is that max-min fair bandwidth
allocations are characterized such that all
flows that are bottlenecked by a router
have the same output rate.
 This rate is called the fair share rate of the link.
 Let α(t) be the fair share rate at time t.

Advanced Computer Networks: CSFQ Paper 8

 Fluid Model Algorithm
If max-min bandwidth allocations are achieved, each
flow receives service at a rate given by

min (ri (t), α(t))

Let A(t) denote the total arrival rate:

If A(t) > C then the fair share is the unique solution to

Advanced Computer Networks: CSFQ Paper 9

 Fluid Model Algorithm
Thus, the probabilistic fluid forwarding
algorithm that achieves fair bandwidth
allocation is:
Each incoming bit of flow i is dropped
with probability
max (0,1-α(t)/ri(t)) (2)
These dropping probabilities yield fair
share arrival rates at the next hop.

Advanced Computer Networks: CSFQ Paper 10

 Packet Algorithm
 Moving from a bit-level, buffer-less fluid
model to a packet-based, buffer model
leaves two challenges:
– Estimate the flow arrival rates ri(t)
– Estimate the fair share α(t)
 This is possible because the rate
estimator incorporates the packet size.

Advanced Computer Networks: CSFQ Paper 11

 Flow Arrival Rate
At each edge router, use exponential averaging to estimate the
rate of a flow. For flow i, let
lik be the length of the kth packet.
t ik be the arrival time of the kth packet.

Then the estimated rate of flow i, ri is updated every time a new

packet is received:

rinew = (1-e-T/K)*L / T + e-T/K* riold (3)

where
T = Tik = tik – tik-1
L = l ik and K is a constant

Advanced Computer Networks: CSFQ Paper 12

 Link Fair Rate Estimation
If we denote the estimate of the fair share by
and the acceptance rate by ,
we have

Note – if we know ri(t) then can be

determined by finding the unique solution to
F(x) = C.
However, this requires per-flow state !
Advanced Computer Networks: CSFQ Paper 13
 Heuristic Algorithm
 The heuristic algorithm needs three aggregate state
variables:
, , where is the estimated aggregate
arrival rate.
 When a packet arrives, the router computes:

(5)

 and similarly computes .

Advanced Computer Networks: CSFQ Paper 14

 CSFQ Algorithm
When a packet arrives, is updated using
exponential averaging (equation 5).
If the packet is dropped, remains the same.
If the packet is not dropped, is updated
using exponential averaging.
At the end of an epoch (defined by Kc ), if the
link is congested during the whole epoch,
update :

Advanced Computer Networks: CSFQ Paper 15

 CSFQ Algorithm (cont.)
 If the link is not congested, is set to
the largest rate of any active flow.

 feeds into the calculation of drop

probability, p, for the next arriving packet
as α in
p = max (0 , 1 – α / label)

Advanced Computer Networks: CSFQ Paper 16

 CSFQ Algorithm (cont.)
 Estimation inaccuracies may cause
to exceed link capacity.
 Thus, to limit the effect of Drop Tail
buffer overflows, every time buffer
overflows is decreased by 1% in
simulations.

Advanced Computer Networks: CSFQ Paper 17

CSFQ Pseudo Code

Figure 3
Advanced Computer Networks: CSFQ Paper 18
CSFQ
Pseudo
Code

Advanced Computer Networks: CSFQ Paper 19

 Label Rewriting
 At core routers, outgoing rate is merely
the minimum between the incoming rate
and the fair rate, α .
 Hence, the packet label L can be
rewritten by
L new = min (L old , α )

Advanced Computer Networks: CSFQ Paper 20

 Outline
 Introduction
 Core-Stateless Fair Queueing (CSFQ)
– Fluid Model Algorithm
– Packet Algorithm
– Flow Arrival Rate
– Link Fair Share Rate Estimation
 NS Simulations
 Conclusions

Advanced Computer Networks: CSFQ Paper 21

 Simulations
 A major effort of the paper is to
compare CSFQ to four algorithms via
ns-2 simulations.
 FIFO
 RED
 FRED (Flow Random Early Drop)
 DRR (Deficit Round Robin)

Advanced Computer Networks: CSFQ Paper 22

FRED (Flow Random Early Drop)
 Maintains per flow state in router.
 FRED preferentially drops a packet of a flow
that has either:
– Had many packets dropped in the past
– A queue larger than the average queue size
 Main goal : Fairness
 FRED-2 guarantees to each flow a minimum
number of buffers.

Advanced Computer Networks: CSFQ Paper 23

 DRR (Deficit Round Robin)
 Represents an efficient implementation of
WFQ.
 A sophisticated per-flow queueing
algorithm.
 Scheme assumes that when router buffer is
full the packet from the longest queue is
dropped.
 Can be viewed as “best case” algorithm
with respect to fairness.

Advanced Computer Networks: CSFQ Paper 24

 ns-2 Simulation Details
 Use TCP, UDP, RLM and On-Off traffic
sources in separate simulations.
 Bottleneck link: 10 Mbps, 1ms latency,
64KB buffer
 RED, FRED (min, max) thresholds:
(16KB, 32KB)
 K and Kc = 100 ms. Kα = 200ms.

Advanced Computer Networks: CSFQ Paper 25

 A Single Congested Link
 First Experiment : 32 UDP CBR flows
– Each UDP flow is indexed from 0 to 31 with
flow 0 sending at 0.3125 Mbps and each of
the i subsequent flows sending (i+ 1) times
its fair share of 0.3125 Mbps.
 Second Experiment : 1 UDP CBR flow,
31 TCP flows
– UDP flow sends at 10 Mbps
– 31 TCP flows share a single 10 Mbps link.

Advanced Computer Networks: CSFQ Paper 26

Figure 5b: 32 UDP Flows

Only CSFQ, DRR

and FRED-2 can
contain UDP flows!!

Advanced Computer Networks: CSFQ Paper 27

Figure 6b : One UDP Flow, 31 TCP Flows

Only CSFQ and

DRR can contain
Flow 0 – the only
UDP flow!

Advanced Computer Networks: CSFQ Paper 28

 A Single Congested Link
 Third Experiment Set : 31 simulations
– Each simulation has a different N,
N = 2 … 32.
– One TCP and N-1 UDP flows with each
UDP flow sending at twice fair share rate of
10/N Mbps.

Advanced Computer Networks: CSFQ Paper 29

 Figure 6b : One TCP Flow, N-1 UDP Flows

Normalized fair share

throughput for one TCP source
DRR good for less
than 22 flows.

CSFQ better than

DRR when a large
number of flows.

CSFQ beats FRED.

Advanced Computer Networks: CSFQ Paper 30

 Multiple Congested Links
1-10 K1-K10
UDP
Sinks
TCP/UDP-0
Source

Router Router Router K Router K+1

TCP/UDP-0
UDP Sink
Sources
1 10 11 20 K1 K10

Advanced Computer Networks: CSFQ Paper 31

 Multiple Congested Links
 First experiment : UDP flow 0 sends at
its fair share rate, 0.909 Mbps while the
other ten “crossing” UDP flows send at
2 Mbps.
 Second experiment: Replace the UDP
flow with one TCP flow and leave the
ten crossing UDP flows.

Advanced Computer Networks: CSFQ Paper 32

Figure 8a : UDP source

Fraction of UDP-0 traffic forwarded

versus the number of congested links

Advanced Computer Networks: CSFQ Paper 33

Figure 8b : TCP Source
Fraction of TCP-0 traffic forwarded
versus the number of congested links

Advanced Computer Networks: CSFQ Paper 34

 Receiver-driven Layered
Multicast
 RLM is an adaptive scheme in which
the source sends the information
encoded in a number of layers.
 Each layer represents a different
multicast group.
 Receivers join and leave multicast
groups based on packet drop rates
experienced.

Advanced Computer Networks: CSFQ Paper 35

 Receiver-driven Layered
Multicast
 Simulation of three RLM flows and one
TCP flow with a 4 Mbps link.
 Fair share for each is 1 Mbps.
 Since router buffer set to 64 KB, K, Kc,
and Kα are set to 250 ms.
 Each RLM layer I sends 2i+4 Kbps with
each receiver subscribing to the first five
layers.

Advanced Computer Networks: CSFQ Paper 36

Figure 9b : FRED

Advanced Computer Networks: CSFQ Paper 37

Figure 9e : RED

Advanced Computer Networks: CSFQ Paper 38

Figure 9f : FIFO

Advanced Computer Networks: CSFQ Paper 39

Figure 9a : DRR

Advanced Computer Networks: CSFQ Paper 40

Conference Figure : CSFQ

K,= Kc = Kα = 250 ms.

Advanced Computer Networks: CSFQ Paper 41

Figure 9c: CSFQ

Advanced Computer Networks: CSFQ Paper 42

Figure 9d: CSFQ

Advanced Computer Networks: CSFQ Paper 43

 On-Off Flow Model
 One approach to modeling interactive,
Web traffic :: OFF represents “think
time”
 ON and OFF drawn from exponential
distribution with means of 100 ms and
1900 ms respectively.
 During ON period source sends at 10
Mbps.

Advanced Computer Networks: CSFQ Paper 44

Table 1 : One On-Off Flow, 19 TCP
Flows
Algorithm Delivered Dropped
DRR 1080 3819
CSFQ 1000 3889
FRED 1064 3825
RED 2819 2080
FIFO 3771 1128

4899 packets sent!

Advanced Computer Networks: CSFQ Paper 45
 Web Traffic
 A second approach to modeling Web
traffic that uses Pareto Distribution to
model the length of a TCP connection.
 In this simulation 60 TCP flows whose
interarrivals are exponentially
distributed with mean 0.05 ms and
Pareto distribution that yields a mean
connection length of 20,1 KB packets.

Advanced Computer Networks: CSFQ Paper 46

Table 2 : 60 Short TCP Flows, One UDP
Flow
Algorithm Mean Transfer Standard
Time for TCP Deviation
DRR 25 99
CSFQ 62 142
FRED 40 174
RED 592 1274
FIFO 840 1695

Advanced Computer Networks: CSFQ Paper 47

Table 3 : 19 TCP Flows, One UDP Flow
with propagation delay of 100 ms.
Algorithm Mean Standard
Throughput Deviation
DRR 6080 64
CSFQ 5761 220
FRED 4974 190
RED 628 80
FIFO 378 69

Advanced Computer Networks: CSFQ Paper 48

 Packet Relabeling
Sources

10 Mbps
Flow 1
Link 1
Router 10 Mbps
10 Mbps Link 2
10 Mbps
Flow 2 Router Sink
10 Mbps

Flow 3

Advanced Computer Networks: CSFQ Paper 49

Table 4 : UDP and TCP with
Packet Relabeling
Traffic Flow 1 Flow 2 Flow 3

UDP 3.267 3.262 3.458

TCP 3.232 3.336 3.358

Advanced Computer Networks: CSFQ Paper 50

 Unfriendly Flows
 Using TCP congestion control requires
cooperation from other flows.
 Three types cooperation violators:
– Unresponsive flows (e.g., Real Audio)
– Not TCP-friendly flows (e.g., RLM)
– Flows that lie to cheat.
This paper deals with unfriendly
flows!!

Advanced Computer Networks: CSFQ Paper 51

 Outline
 Introduction
 Core-Stateless Fair Queueing (CSFQ)
– Fluid Model Algorithm
– Packet Algorithm
– Flow Arrival Rate
– Link Fair Share Rate Estimation
 NS Simulations
 Conclusions

Advanced Computer Networks: CSFQ Paper 52

 Conclusions
 This paper presents Core Stateless Fair
Queueing and offers many simulations
to show how CSFQ provides better
fairness than RED or FIFO.
 They mention issue of “large latencies”.
This is the robust versus fragile flow
issue from FRED paper.
 CSFQ ‘clobbers’ UDP flows!

Advanced Computer Networks: CSFQ Paper 53

 Significance
 First paper to use hints from the edge of
the subnet.
 Deals with UDP. Many AQM algorithms
ignore UDP.
 Makes a reasonable attempt to look at a
variety of traffic types.

Advanced Computer Networks: CSFQ Paper 54

 Problems/ Weaknesses
 “Epoch” is related to three constants in
a way that can produce different results.
 How does one set K constants for a
variety of situations.
 No discussion of algorithm “stability”

Advanced Computer Networks: CSFQ Paper 55

 Acknowledgments
 Figures extracted from presentation by
Nagaraj Shirali and Choong-Soo Lee in
Spring 2002 and modified for
annotations.

Advanced Computer Networks: CSFQ Paper 56