Pathlet Routing

∗
P. Brighten Godfrey† , Igor Ganichev‡ , Scott Shenker‡§ , and Ion Stoica‡
† University of Illinois at Urbana-Champaign ‡ UC Berkeley § ICSI
[email protected], {igor,shenker,istoica}@cs.berkeley.edu

ABSTRACT that scales linearly in the number of IP prefixes advertised

We present a new routing protocol, pathlet routing, in which in the Internet. This is particularly a concern in the data
networks advertise fragments of paths, called pathlets, that plane where the router stores the routing table, or forward-
sources concatenate into end-to-end source routes. Intu- ing information base (FIB). Because it has to operate at
itively, the pathlet is a highly flexible building block, captur- high speeds and often uses SRAM rather than commodity
ing policy constraints as well as enabling an exponentially DRAM, FIB memory is arguably more constrained and ex-
large number of path choices. In particular, we show that pensive than other resources in a router [22]. Moreover, the
pathlet routing can emulate the policies of BGP, source rout- number of IP prefixes is increasing at an increasing rate [15],
ing, and several recent multipath proposals. leading to the need for expensive hardware and upgrades.
This flexibility lets us address two major challenges for The Internet Architecture Board Workshop on Routing and
Internet routing: scalability and source-controlled routing. Addressing recently identified FIB growth as one of the key
When a router’s routing policy has only “local” constraints, concerns for future scalability of the routing system [22].
it can be represented using a small number of pathlets, lead- A second challenge for interdomain routing is to provide
ing to very small forwarding tables and many choices of multipath routing, in which a packet’s source (an end host or
routes for senders. Crucially, pathlet routing does not im- edge router) selects its path from among multiple options.
pose a global requirement on what style of policy is used, but For network users, multipath routing is a solution to two
rather allows multiple styles to coexist. The protocol thus important deficiencies of BGP: poor reliability [1, 14, 17]
supports complex routing policies while enabling and incen- and suboptimal path quality, in terms of metrics such as la-
tivizing the adoption of policies that yield small forwarding tency, throughput, or loss rate [1, 27]. Sources can observe
plane state and a high degree of path choice. end-to-end failures and path quality and their effect on the
particular application in use. If multiple paths are exposed,
the end-hosts could react to these observations by switching
Categories and Subject Descriptors paths much more quickly and in a more informed way than
C.2.1 [Network Architecture and Design]: Packet-switching BGP’s control plane, which takes minutes or tens of minutes
networks; C.2.2 [Network Protocols]: Routing Protocols; to converge [19, 21]. For network providers, multipath rout-
C.2.6 [Internetworking]: Routers ing represents a new service that can be sold. In fact, route
control products exist today which dynamically select paths
General Terms based on availability, performance, and cost for multi-homed
edge networks [3]; exposing more flexibility in route selection
Design, Experimentation, Performance, Reliability could improve their effectiveness. Greater choice in routes
may bring other benefits as well, such as enabling competi-
1. INTRODUCTION tion and encouraging “tussles” between different parties to
be resolved within the protocol [6].
Challenges for interdomain routing. Interdomain rout- But providing multiple paths while respecting network
ing faces several fundamental challenges. One is scalabil- owners’ policies is nontrivial. BGP provides no multipath
ity: routers running the Internet’s interdomain routing pro- service; it selects a single path for each destination, which
tocol, Border Gateway Protocol (BGP) [25], require state it installs in its FIB and advertises to its neighbors. Several
∗The first and fourth authors were supported in part by a multipath routing protocols have been proposed, but these
Cisco Collaborative Research Initiative grant. have tradeoffs such as not supporting all of BGP’s routing
policies [32, 30], exposing only a limited number of addi-
tional paths [28], making it difficult to know which paths
will be used [31, 23], or increase the size of the FIB [28, 31,
Permission to make digital or hard copies of all or part of this work for 23], which would exacerbate the scalability challenge.
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies Our contributions. This paper addresses the challenges
bear this notice and the full citation on the first page. To copy otherwise, to of scalability and multipath routing with a novel protocol
republish, to post on servers or to redistribute to lists, requires prior specific called pathlet routing. In pathlet routing, each network
permission and/or a fee.
SIGCOMM’09, August 17–21, 2009, Barcelona, Spain. advertises pathlets—fragments of paths represented as se-
Copyright 2009 ACM 978-1-60558-594-9/09/08 ...$10.00. quences of virtual nodes (vnodes) along which the network
is willing to route. A sender concatenates its selection of A router
pathlets into a full end-to-end source route.
3 pathlet
From this architecture come three key ideas. First, path- (labeled with FID)
lets and vnodes are highly flexible building blocks, able to
a vnode
express many kinds of policies as well as enabling a dra- 12.34.56.0 / 24
matic amount of path choice in a clean protocol. Intuitively,
like path vector routing (i.e. BGP), pathlets can be used to a b 2
3 c 7 1
constrain how a packet transits an autonomous system (AS) d e
A B C D E
and where it goes after it leaves. But like source routing,
pathlets may be concatenated in exponentially many ways. Route in packet header arriving at each hop:
In fact, we show that pathlet routing’s data plane can em- 3,2 2 7,1 1
ulate the routing policies of BGP, loose and strict source
routing, and three recent multipath proposals: NIRA [30],
MIRO [28], and LISP [9]. We are not aware of any protocol Figure 1: A pathlet routing example.
which can emulate pathlet routing’s policies, although there
are several that pathlet routing cannot emulate [32, 31, 23].
The second key idea is that an AS whose policies have only Besides this core protocol, we must specify how path-
“local” constraints can represent its policies using a small lets are disseminated throughout the network. This is a
number of pathlets, leading to small FIBs and many allowed largely separable component of the design, because (unlike
paths. We suggest a new class of policies of this type, lo- BGP) route policy is enforced by what pathlets are con-
cal transit (LT) policies, that allow networks to control the structed, rather than by how they are announced. For now,
portions of routes which transit across their own networks, the reader may assume “telepathic” routers which know ev-
but otherwise expose the full flexibility of the Internet’s AS- ery constructed pathlet. We will present out dissemination
level routes to sources. A special case of LT policies are protocol in Section 3.
valley-free routes, the common export policy used in BGP.
We show that this kind of LT policy has a large amount of 2.1 Example
path choice which significantly improves reliability, and has Before defining the protocol in detail, we give a simple
FIBs whose size scales with the number of neighbors of a example to illustrate the pathlet mechanism. Consider the
router, rather than with the number of destinations—thus topology in Fig. 1 with routers A, B, C, D, E, each of which
reducing the average number of FIB entries for the Internet has one vnode (a, b, c, d, e, respectively). Initially, the routers
AS topology by more than 10, 000× compared with BGP. learn the vnodes of their neighbors. They can then construct
The third key idea is that pathlet routing does not impose one-hop pathlets to their neighbors, as A, C, and D have
a global requirement on what style of policy is used. It cleanly done. A pathlet is given a forwarding identifier (FID) which
allows multiple styles to coexist, for example with BGP-style identifies it in the routing table of its first vnode. For ex-
policies at some ASes, and LT-style policies at others. A ample, entry 7 in the routing table of vnode c instructs the
convenient consequence of our architecture is that regardless router to forward the packet to D.
of what the other ASes choose, the LT routers obtain the A sender determines its destination by finding a vnode
entire benefit of small FIBs, and part of the improved path tagged with the appropriate IP prefix, such as e in the ex-
choice. Intuitively, a router needs space in its forwarding ample. It specifies a route as a list of FIDs in the packet
table only for the pathlets that it constructs. header. A packet starting at c with route (7, 1) will have its
We confirm these results in experiments with an imple- first FID popped off and will be forwarded to d with route
mentation of pathlet routing. Our implementation also shows (1) in the header, whereupon the next FID is popped off
that while our protocol can have greater messaging and con- and the packet is forwarded to e with the empty route (),
trol plane memory overhead than path vector protocols like indicating that it has reached its destination.
BGP, the overhead is small in Internet-like topologies. After one-hop pathlets are constructed, multihop pathlets
Thus, pathlet routing supports complex BGP-style poli- can be built using them. Here, B builds a pathlet b →
cies while enabling the adoption of policies that yield small c → d → e. It picks FID 2, and sets the forwarding table
forwarding plane state and a high degree of path choice. instructions to push (7, 1) onto the front of the packet’s route
and then forward it to C. A packet beginning at a can now
Paper outline. We introduce the core of the protocol in
reach e using route (3, 2), as shown at the bottom of Fig. 1.
Sec. 2, and the scheme for disseminating pathlets in Sec. 3.
Note that the routers in this example have used different
Sec. 4 discusses new uses of pathlet routing, including LT
styles of routing policy. A, C, and D have policies that are
and mixed policies. In Sec. 5 we show pathlet routing can
“local”, i.e., they depend only on their neighbors. B has a
emulate the policies of several other protocols. Sec. 6 de-
BGP-like policy which depends on the destination: it allows
scribes and evaluates our implementation of pathlet routing.
transit from B to C only when the ultimate destination is
We discuss related work in Sec. 7 and conclude in Sec. 8.
E. We will see these two styles again in Sec. 4.
In the rest of Sec. 2, we give a more detailed description
2. THE PATHLET ROUTING PROTOCOL of the pathlet routing protocol.
This section begins with a simple example (§2.1). We
then describe the core pathlet routing protocol: its building 2.2 Building blocks
blocks of vnodes and pathlets (§2.2), how pathlets are built Pathlet routing can be seen as source routing over a vir-
(§2.3), how packets are forwarded (§2.4), and how the sender tual topology whose nodes are vnodes and whose edges are
picks its route from the pathlets it knows (§2.5). pathlets. We describe these two building blocks next.
Vnodes. A vnode is a virtual node created by an AS to Examples include transferring the packet to another
represent the structure of routes within its own network. vnode at the same router; sending it on a certain out-
Vnodes can be used in many ways (some of which we will going interface; or tunneling it across an intradomain
see later), but the simplest case is that each router has one routing protocol like OSPF to an egress point.
vnode.
An AS creates a vnode by setting up a routing table for • the remaining FIDs: the list of FIDs necessary to
the vnode at one or more routers. Initially, each router is route the packet from the second vnode to the final
configured with at least one vnode. When a router X opens vnode in the pathlet. If the pathlet has just one hop,
a control plane connection to a neighbor Y , it designates an this list is empty. Otherwise it is a list r1 , . . . , rk , where
ingress vnode for Y : it sends Y a globally-unique vnode each ri is the FID for pathlet Pi .
identifier v, indicating that every packet X receives from Y Readers nostalgic for LISP might think of these two fields
will be directed to v. The ingress vnode can be different as the CAR and CDR of the pathlet’s forwarding information.
for each of X’s neighbors, thus allowing X to control routes For example, in Fig. 1, most of the pathlets are the “base
based on a packet’s previous hop. For example, in Sec. 4.1 case” one-hop pathlet, which can be constructed immedi-
we will construct vnodes representing ingress from neighbors ately after each router X opens the control plane connec-
grouped into classes according to customer, provider, and tions to its neighbors and learns their ingress vnodes for
peer business relationships. X. After the one-hop pathlets are disseminated through the
Routers learn most vnodes implicitly, when they are men- network (according to the protocol of Sec. 3), router B can
tioned as part of a pathlet’s path. Additionally, a vnode can create the “general case” pathlet b → c → d → e by concate-
be tagged with a destination IP prefix, as in e in Fig. 1. Tag nating the two pathlets c → d and d → e. In the forwarding
announcements are handled with the same dissemination al- table for vnode b, it maps the FID 2 to the remaining FIDs
gorithm as pathlets (see Sec. 3). (“push (7, 1) onto the front of the route”) and the next hop
Pathlets. A pathlet represents a sequence of vnodes v1 → rule (“send the packet to router C”).
· · · → vn along which the AS X that originated the an-
nouncement is willing to route. The first vnode v1 is in X, 2.4 Packet forwarding
but the others may be in X or other ASes. Each router has multiple forwarding tables: one for each
A pathlet is identified by a forwarding identifier, or of its vnodes. The table for each vnode v is an exact-match
FID, f . To routers other than the pathlet’s creator, f is lookup table with one entry for each pathlet beginning at v.
simply an opaque variable-length string of bits with the se- The entry maps the pathlet’s FID to the next hop rule and
mantics that if a packet arrives at v1 and the packet’s source the remaining FIDs, as described above.
route begins with f , then it will be forwarded along the path When a packet is sent from router Y to X, it is interpreted
v1 → · · · → vn , arriving at vn with f popped off the front of as arriving at the specific vnode v which is X’s ingress vnode
the source route. The sender of a packet will place a series for Y . It is safe to assume that the previous hop cannot be
of FIDs in the packet to indicate the full route. spoofed: X knows on which of its interfaces it received the
To the router or routers on which v1 is instantiated, f is packet, and the interface would be associated with a specific
used as an index into the vnode’s forwarding table. Thus, f neighboring AS.
must uniquely identify the pathlet among all pathlets begin- The router X then inspects the route in the packet header.
ning at v1 —but importantly, f need not be globally unique If it is empty, the packet is delivered locally. Otherwise the
like the identifiers in IP source routing, or even unique within route is a sequence of FIDs (f1 , f2 , . . . , fn ) of the pathlets
an AS. The result is very compact FIDs. For example, in forming the route to its destination. Initially, this is set by
Fig. 1, it would actually have been possible to give all path- the sender to be the selected route. The router checks for f1
lets the same FID since they originate at different vnodes. in the forwarding table for the ingress vnode v. If no entry is
Our implementation uses the following variable-length FID found, the packet is malformed and is dropped. Otherwise,
encoding scheme. The first bits of the FID indicate its the table maps f1 to a next hop rule and the remaining FIDs
length: FIDs that begin with 0, 10, 110, 1110, and 1111 r1 , . . . , rk for this pathlet. It pops off the packet’s first FID
have total lengths of 4, 8, 16, 24, and 32 bits, respectively. f1 and pushes the remaining FIDs, resulting in a route of
The remaining bits contain the unique identifier. In our LT r1 , . . . , rk , f2 , . . . , fn . Finally, it forwards the packet to the
policies, most vnodes originate ≤ 8 pathlets, so these use next vnode, according to the next hop rule.
the short 4-bit FIDs. Other encoding schemes are possible, Fig. 1 gives a packet forwarding example. At a, the first
and an AS can unilaterally pick its own scheme, since other FID (3) is popped off and used to look up the next-hop
routers simply view the FID as an opaque string of bits. rule and remaining FIDs. There are no remaining FIDs be-
cause it is a one-hop pathlet, so the packet is forwarded to
2.3 Pathlet construction b according to the next-hop rule. At b, the next FID (2)
In the “base case” of pathlet construction, a pathlet con- is popped. Here the remaining FIDs (7, 1) are pushed onto
nects two vnodes, v1 → v2 . In the general case, after X the route since this is a multihop pathlet, and then it is
has learned or constructed some pathlets, it can concate- forwarded to c according to the next-hop rule. At c and d
nate these to build longer pathlets which take the form FIDs are popped off the route, none are pushed on, and the
v1 → P1 → · · · → Pk , where each Pi is an existing pathlet. packet is forwarded. Finally the packet arrives at e with the
The router gives the pathlet a FID f , and in the forwarding empty route, where it is delivered.
table for v1 , it associates f with: Note that it is impossible for a sender to “cheat” and use
a source route that violates the routers’ policies, even if it
• a next hop rule to reach v2 : instructions for forward- somehow gains knowledge of every pathlet. This is simply
ing packets to the next vnode in the pathlet’s path. because there are no forwarding entries for invalid routes.
2.5 Route selection intended, then policy is enforced in the data plane, rather
Each router learns a set of pathlets, via control plane than through obscurity in the control plane. If a certain
mechanisms to be discussed later. It can select from among route is not allowed, then the appropriate forwarding table
these a route for each packet that it sends. A simple way to entries do not exist, so no sequence of bits placed in the
do this is to build a graph in which each vnode is a node, packet header can cause a packet to traverse the route.
and each pathlet v1 → . . . → vn is a single edge v1 → vn Instead, the problem with this approach is simply that
(perhaps given a cost equal to the number of ASes through there may be too many pathlets. Certain cases would be
which the pathlet travels). Then, similar to link state rout- manageable, such as if all ASes use the Local Transit policies
ing, run a shortest path algorithm on this graph to produce that we propose; but there is no fundamental constraint on
a sequence of edges (i.e., pathlets) to each destination. After how many pathlets an AS constructs.
the router has made its path selection, it places the sequence Straw man #2. Suppose now that we use a standard
of FIDs associated with the chosen pathlets into the packet broadcasting algorithm, except any router may decide to
header, and sends it. propagate only a subset of its known pathlets. This solves
Note that this algorithm requires a map of the entire In- the scalability problem. However, it leads to a subtle issue
ternet, as learned by the control plane. However, as we when there are multiple simultaneous failures: it is possible
will see (§6), it is not dramatically more state than BGP for a router to learn of a pathlet which later fails, but not
disseminates. And memory use in this algorithm is not as be notified of the failure because the channel over which it
critical as in the FIB, for several reasons. First, it is per- would have learned about the pathlet’s state has also failed.
formed by edge routers which do not need as high packet We omit an example due to space constraints. The key point
processing speeds as core routers, so the map can be stored is that it would be possible for a router to continue to use a
in slower (larger, cheaper) memory. Second, an edge router pathlet for an arbitrarily long time after it has failed.
can cache routes for the (presumably small) set of destina-
tions to which it is sending at any one point in time, so that 3.2 Pathlet dissemination algorithm
the route computation is needed only for the first packet in We choose a path vector algorithm to disseminate path-
each connection, or the caching could be offloaded to the lets, much as BGP notifies the Internet of the existence of
senders. In fact, it would be feasible for many end-hosts IP prefixes. We use a standard, bare-bones path vector pro-
(rather that edge routers) to choose routes themselves, but tocol, with a pathlet announcement containing the pathlet’s
the protocol between end-hosts and their gateway routers is FID and its sequence of vnode identifiers.
beyond the scope of this work.1 Path vector has two important properties. First, it solves
Routers have the freedom to make more intelligent choices the scalability problem from the first straw man, by allow-
than the simple shortest path algorithm presented above, ing routers to propagate an arbitrary subset of their known
through information learned outside of the pathlet routing pathlets (much as BGP allows routers to choose which routes
protocol. Options include learning path properties based to export). Second, it guarantees that if a pathlet fails, it
on observations of performance or availability [32, 4]; com- will eventually be withdrawn from all routers that learned
mercial route selection products [3]; each network operat- of it (intuitively because path vector remembers the dissem-
ing a route monitoring service for its users [30]; third-party ination channel).
route selection services [20], or even a global “Internet weath- This choice of path vector might seem ironic, given that
ermap” service that would assist end-hosts in their route our goal is to improve on BGP’s path vector routing; but
selection. we use path vector only to make pathlets known, not to
pick packets’ routes. In particular, the dissemination path
3. PATHLET DISSEMINATION attribute (corresponding to BGP’s ASPATH) is used only
We motivate our dissemination algorithm with two straw for detection of loops in the dissemination channel, not as a
man proposals (§3.1). We then describe our chosen dissemi- path along which to route data packets.
nation algorithm (§3.2), the choice of which pathlets to dis- The main disadvantage of path vector is that a router
seminate (§3.3), and an extension where nonadjacent routers must remember up to O(δ`) state for each pathlet rather
exchange pathlets (§3.4). than O(1) state as in a broadcasting algorithm, where δ is
Note that the algorithm we present here and evaluate in the number of neighbors and ` is the average path length.
Sec. 6 should be seen only as one feasible design. The choice However, we have a great deal of flexibility to optimize the
of dissemination algorithm largely does not affect the rest overhead of the algorithm, compared with BGP’s use of path
of our design, and could easily be replaced with another vector. The simplest optimization is that we never need
approach. to switch to a “more preferred” dissemination path, since
they are all equally acceptable; see our experimental results
3.1 Design rationale concerning message rates in Sec. 6.3. Other optimizations
to reduce both messaging and memory overhead would be
Straw man #1. Suppose we simply broadcast all pathlets feasible, as we discuss in Sec. 8.
to the entire network using a link state algorithm. This is
in fact not a policy concern: if pathlet routing is used as 3.3 Which pathlets are disseminated?
1
Since we use a path vector dissemination protocol, each
For many end-hosts, it would be feasible to learn pathlets router can choose exactly which subset of its known pathlets
from the gateway router and run the same algorithm pre- to announce to its neighbors, just as in BGP’s export filter.
sented here. Alternately, the gateway router could give the
end-host several choices without detailing what those routes The rule we choose is as follows.
are, as in [31, 23]; or path selection could be performed en- Suppose v is router X’s ingress vnode for neighbor Y .
tirely by routers. From among all the pathlets that X knows (both those
it constructed and those it learned from neighbors), it an- customer or X is a customer. This is a function only of the
nounces a subset to Y , as follows: ingress and egress points, and hence is an LT policy.
To the best of our knowledge, other multipath routing pro-
1. Announce pathlets which form a shortest path tree
tocols [8, 28, 30, 18, 23, 32] cannot implement local transit
from v to all destination vnodes reachable from v.
policies, though some [30] can implement special cases.
2. Announce any additional pathlets that are reachable
from v, up to limit(δ) pathlets originating at each AS Advantages, disadvantages, incentives. LT policies are
with δ AS-level neighbors. an example of policies in which constraints are local. In
general, this locality leads to two advantages over BGP-style
Step 1 ensures that sufficient pathlets are supplied in order policies. First, the pathlets can be combined in potentially
that Y can reach every destination reachable from v. Step 2 an exponentially large number of ways, leading to improved
adds desirable redundancy. In our implementation, we use performance and reliability for senders. Second, they can be
limit(δ) = 10 + δ, so that higher-degree ASes can announce represented with a small number of pathlets, leading to small
more pathlets. But this particular choice is not critical since forwarding tables. (We will demonstrate both advantages in
Step 1 handles reachability. Sec. 6.)
The primary disadvantage is that policies cannot depend
3.4 Extensions on a route’s destination. Such policies are used in BGP,
Information dissemination in distributed systems can gen- where ASes select and advertise their most preferred route
erally be described as either “push” or “pull”. The above dis- on a per-destination basis. Thus, ASes would end up per-
semination algorithm, like BGP, pulls routing state. But an mitting routes other than the one for each destination that
easy extension of our protocol allows any router to initiate is cheapest. A more minor issue is that LT policies use
a control plane connection with any other (perhaps non- slightly more control plane state than BGP (as we will see
adjacent) router, and pull certain pathlets, such as those in Sec. 6), but this is not as constrained a resource as data
relevant to a specified destination. This extension can be plane memory.
used to emulate control-plane features of three recent mul- We argue that for a large fraction of ASes, the advantages
tipath routing proposals: MIRO, NIRA, and LISP (§5). It of LT policies may outweigh the disadvantages. LT-capable
is also similar to BGP’s multihop option, where an eBGP routers require less data plane memory and hence less cost,
session connects to non-adjacent routers. and are much less susceptible to scalability problems as the
Internet grows; and providing more path choice is a ser-
4. NEW USES OF PATHLET ROUTING vice that customers may be willing to pay for. Providing
In this section, we discuss three practical examples of what more paths may also attract more traffic and hence more
one can do conveniently with pathlet routing, but not with revenue. Moreover, the disadvantage of policy locality is
BGP or most other protocols: Local Transit policies (Sec- limited since the common BGP export policy of enforcing
tion 4.1) allow networks to control one of the most impor- valley-free routes is an LT policy. At a higher level, we note
tant aspects of routing policy—the portions of routes which that offering multipath network services at a constant price
transit across their own networks—while permitting very regardless of the path may be a feasible business model for
small forwarding tables and a large degree of route choice the same reasons that today’s transit services have a fixed
for senders. Pathlet routing cleanly supports mixed poli- rate regardless of the destination.
cies (Section 4.2), for example with some ASes using BGP- Ultimately, whether ASes choose to use LT policies de-
style policies and some using LT policies. The LT adopters pends on business decisions and payment mechanisms, but
still have small forwarding tables, and some of the benefit pathlet routing makes it feasible at a technical level. Fur-
of additional route choice. Finally, pathlet routing may be thermore, as we will see in Section 4.2, this is a decision that
a more flexible way of offering multiple types of service can be made by each AS individually.
along the same physical path (Section 4.3).
Implementation. Figure 2 depicts two ways of implement-
4.1 Local Transit Policies ing LT policies in pathlet routing. The simplest way uses
one ingress vnode for each neighbor. Then, the AS con-
Definition. A network X has a local transit (LT) pol- structs a pathlet between each ingress-egress pair for which
icy when X’s willingness to carry traffic along some route it wants to allow transit service. However, this results in
depends only on the portion of the route that crosses X’s O(δ 2 ) pathlets for a network with δ neighbors.
network. In other words, under an LT policy the permissi- Fortunately, we can use our vnode abstraction for a much
bility and cost of some path, according to X, is a function more compact representation. We assign each neighbor to a
only of the ingress and egress points of the path in X. An “class”, an abstract group of neighbors which are equivalent
alternate definition is that the policy can be constructed from a policy standpoint. Examples are business relation-
using pathlets which never extend beyond X’s neighboring ships (customer, provider, peer) or potentially geographic
vnodes—hence, the policy is local. (Note that LT policies regions. We have vnodes representing ingress and egress for
concern only how traffic may transit X’s network; X may each class (rather than each neighbor), and then construct
still have arbitrary preferences on the paths taken by traffic the appropriate pathlets between these vnodes.
that X sends.) This reduces the number of pathlets from O(δ 2 ) to O(c2 +
An important example of LT policies is valley-freeness [11], δ), where c is the number of classes. We argue that the num-
a common export policy in BGP today. Valley-free routes ber of classes would typically be small. For example, in the
can be defined as follows: each neighbor is labeled as a cus- case of valley-free routes, we need only 2 classes, customer
tomer, provider, or peer; a BGP route is exported to a neigh- and provider/peer, for a total of 4 + δ pathlets per AS. This
bor X if and only if either the next hop of the route is a is depicted in Fig. 2.
 providers and peers providers and peers pathlet which adds one hop to one of the pathlets it learned
from a neighbor. This is depicted below in a topology with
a single IP prefix destination:

in out v
w

in out
To mimic BGP’s export rules, in which routes are ex-
ported to only certain neighbors, there are two options.
First, the router may simply not announce certain path-
lets to its neighbors. This mimics BGP “too closely”, since
a neighbor could still use a prohibited path if, for example,
customers customers
it managed to guess the pathlet’s FID. (Similarly, in BGP,
AS vnode
a neighbor could send a packet even when there is no an-
destination vnode
nounced route for the destination IP.) A solution which is
pathlet better than BGP is to enforce policy in the forwarding tables
(tagged with IP prefix)
themselves. For the common valley-free export policy, this
Figure 2: Two ways to implement a local transit can be done with a small constant number of additional vn-
policy are to connect the appropriate ingress-egress odes and pathlets, similar to the LT construction of Fig. 2;
pairs (left), or to group neighbors into classes and we omit the details.
connect the appropriate classes (right). Here we
A BGP-policy router in a mixed setting. In the previ-
show the vnodes and pathlets in one AS to permit
ous example all routers used BGP-style policies. Now con-
valley-free routes.
sider a “mixed” network, where the leftmost router in the
previous example continues using BGP-style policies, but
One difficulty arises with class-based LT policies. If an the others use LT policies. It runs the same algorithm as
AS is internally partitioned, it may not be able to represent before, building a pathlet to each destination. However, it
reachability as a class-based LT policy, which implicitly as- may now have an exponentially large set of possible paths to
sumes that if the provider class can reach the customer class, each destination, rather than one path through each neigh-
then all providers can reach all customers. A solution is for bor. This necessitates a generalization of the BGP decision
the AS to advertise two sets of LT policies, one on each process to select the best path without constructing each op-
side of the partition, in the rare case that it becomes inter- tion explicitly. For example, it could run a least-cost path
nally partitioned. Alternately, the AS can simply continue algorithm, with large costs assigned to its provider links, less
announcing the pathlets; sources will realize that some path- cost on its peering links, and lowest cost on its customer
lets have failed and will switch to a different route, assuming links.
another is available. The resulting pathlet is also slightly different: instead of
adding one hop to a multihop pathlet, it is built by con-
4.2 Mixed Policies catenating multiple short pathlets from the LT nodes, high-
lighted below:
In this section we describe how pathlet routing supports
mixed policies, in particular with some ASes using tradi-
tional BGP-style policies, and some using LT-style. Mixed
policies are important because it allows ASes to make in-
dependent choices about their policies. Some may require The result is a single long pathlet:
restrictive BGP style policies; others may use LT-style poli-
cies, giving up some control but getting the benefit of small
forwarding tables and providing many possible paths. It is
unlikely that either choice would be preferred by all ASes.
We require no new protocol mechanisms to support mixed
policies; routers run the algorithms we have already de- An LT-policy router in a mixed setting. Due to lo-
scribed in Sections 2 and 3, constructing the pathlets ap- cality, an LT router’s pathlets do not depend on pathlets
propriately to match their routing policy. However, since constructed elsewhere. Therefore, it has the same number
we believe this is an important way of using pathlet routing, of pathlets originating from its vnodes, and hence the same
we illustrate the process here. small routing tables regardless of what other routers do.
However, it does have to deal with disseminating other
Emulating BGP. As a prelude, we illustrate the non-mixed routers’ pathlets in the control plane. If an LT node has δ
case when all ASes use BGP-style policies. To emulate BGP, neighbors using BGP-style policies, then it receives O(δn)
each AS constructs one vnode v from which all its pathlets pathlets for a network of size n, and all of these are reach-
originate, and which is its ingress vnode for all neighbors. If able from its ingress vnodes. The router would therefore
it owns an IP prefix, then it has a second vnode w tagged be happy to disseminate them all. In fact, a mixed setting
with the prefix, from which no pathlets depart. It then of BGP-style and LT policy routers could result in O(n2 )
constructs a pathlet from v to every destination it can reach, pathlets usable by LT nodes. However, our dissemination-
along its most-preferred available path. If the destination is limiting mechanisms from Sec. 3.3 will take effect, and only
at w, this is a one-hop pathlet; otherwise it is a multihop a small number of these will be propagated.
4.3 Quality of Service Pathlet routing
We discuss one final novel use of pathlet routing. Our
Feedback-based
protocol’s abstraction of the topology results in a convenient routing
MIRO NIRA
mechanism to set up multiple paths along the same physical
Routing
path. In particular, an AS can construct multiple “parallel” Loose
LISP Deflections,
source routing
pathlets, over the same sequence of vnodes. Packets can Path splicing
then be treated differently depending on which pathlet is
used. For example, an AS could provide better or worse Strict
BGP
source routing
throughput or delay, or more non-traditional services like
reliable packet delivery across that pathlet rather than a full Figure 3: Relative policy expressiveness of the data
end-to-end route, similar to new services proposed by Ford planes of routing protocols. P → Q indicates P can
and Iyengar [10]. To inform sources of these different types emulate the routing policies of Q.
of service, pathlet announcements can be tagged with labels,
or sources could discover performance by measurement or
by a third-party routing appraisal service, as discussed in the graph. Pathlet routing allows routing policies to be spec-
Section 2.5. ified as an arbitrary virtual graph, whose nodes are vnodes
This approach is more flexible and extensible than other and whose directed edges are pathlets.
approaches for several reasons. First, the service is selected
for a specific segment of a path, rather than for the entire
5.2 Definition of policy emulation
path. Thus, different combinations of services are possible Our analysis focuses on the data plane: the information
and an AS which is not involved in providing a particular carried in a packet and in the forwarding table, and forward-
type of service does not need to support that service or even ing operations. We found that considering only the data
be aware of what services a packet is using in other ASes. plane gives a much cleaner and more well-defined way of rea-
Second, the bits in the packet header are not tied to a small soning about protocols, compared with modeling both the
number of predetermined meanings (as in, for example, IP’s data and control planes. We will also discuss control plane
Type of Service bits). Instead, the header bits (FIDs) are differences between the protocols, leaving rigorous analysis
tied to a much more extensible control plane announcement of their control planes to future work.
by the AS providing the service. A configuration of a protocol is defined by an arrange-
One drawback is that the amount of routing state in- ment of forwarding table state at each router. Given a con-
creases as more types of service are offered. However, in figuration c1 from protocol P and a configuration c2 from
terms of forwarding plane state, this only affects the ASes protocol Q, we say that c1 covers c2 when
that are offering the service; and if an AS is using policies
• every end-to-end route which is allowed in c1 is also
like Local Transit policies, the amount of forwarding plane
allowed in c2 , and every end-to-end route which is pro-
state is quite small to begin with.
hibited in c1 is also prohibited in c2 ; and
5. POLICY EXPRESSIVENESS ANALYSIS • for each router i, |c1 (i)| = O(|c2 (i)|), where |cj (i)| de-
In this section, we develop a framework for comparing notes the amount of forwarding state for router i in
routing protocols. We show that pathlet routing’s data configuration j.
plane can emulate any policy expressible in the data plane
of several existing protocols using a similar amount of state, Finally, we say that protocol P can emulate Q if for every
even though these protocols have substantially different for- configuration c2 of Q, there is a configuration c1 of P such
warding architectures. Our goal is to support the claim that that c1 covers c2 . In other words, if P can emulate Q, then
pathlet routing is highly expressive, but we also believe this P can match every possible outcome of Q in terms of allowed
framework is of independent interest since it takes a step paths, prohibited paths, and forwarding table size.
towards making sense of the large number of proposed pro- This definition is limited. We have chosen not to incor-
tocols in this area. porate aspects of policy like the price of a route, visibility
We begin by discussing intuition for pathlet routing’s flex- of routes, misbehavior outside the rules of the protocol, or
ibility (§5.1). We then we define policy emulation (§5.2) and game-theoretic aspects like which outcomes might actually
discuss the protocols pathlet routing can emulate (§5.3) and arise under selfish behavior, or the control plane as discussed
those it cannot (§5.4), highlighting some limitations of our above. However, we believe that policy emulation provides
protocol. These results are summarized in Figure 3. a way to begin reasoning about the relative strengths of dif-
ferent protocols.
5.1 Intuition
Why does pathlet routing appear to be so flexible, gener-
5.3 Protocols pathlet routing can emulate
alizing multiple other routing protocols? One piece of the Due to space constraints, we only briefly describe some of
intuition is that many protocols incorporate some variant of the protocols and the emulation relationships between them.
tunneling. Our pathlet construct elevates the tunnel to be BGP [25]. We outlined how pathlet routing can emulate
a “first-class object” that can be advertised, concatenated, BGP (in both the data an control planes) in Section 4.2.
and source-routed. We will repeatedly use pathlets to emu-
late tunnel-like features of other protocols. However, this is NIRA [30] offers more choice to sources than BGP, while
only part of the story since we also utilize vnodes. simultaneously reducing control plane state. IP addresses
At a higher level, the following intuition is simple yet pow- are assigned so they encode a specific path between a router
erful. One of the most flexible and general abstractions is and the “core” of the Internet or a peering point. A source
and destination IP address together specify a full end-to- “tag” [31] or “splicing bits” [23] to specify which alternative
end route. The most challenging part of emulating NIRA is used at each step. The selected path is essentially a pseu-
is that an AS can allow or prohibit routes based on the up- dorandom function of the tag.
stream hops as far back as the core providers. In contrast, Pathlet routing cannot emulate these protocols due to the
in the usages of pathlet routing we have seen so far, policies handling of tags. In [31] the tag has too few bits to repre-
depend only on the immediately prior hop and any down- sent the set of all possible routes that the routers intended
stream hops. However, it is possible to construct vnodes to to permit, so the de facto set of allowed routes is a pseudo-
encode the packet’s upstream hops, using the same amount random subset of those—an effect which to the best of our
of state as NIRA. This requires some coordination of vnode knowledge pathlet routing cannot reproduce. One mode of
names in the control plane between neighbors, but no data operation in [23] behaves the same way. A second mode of
plane changes. operation in [23] uses a list of source-route bits to explicitly
The basic NIRA protocol is limited to valley-free routes, select the next hop at each router; pathlet routing would be
so it cannot emulate the other protocols we consider. An ex- able to emulate this version.
tension including a source-routing option is described in [29] But there is also a problem in the control plane. Even
but is not analyzed here. if pathlet routing can match the allowed paths in the data
plane, actually using the paths would require the senders to
Locator/ID Separation Protocol (LISP) [9] maps a know what FIDs to put in the packet header. Because [31,
topology-independent endpoint ID into an egress tunnel router. 23] allow per-destination policies as in BGP, this can result
BGP is used to tunnel a packet to its destination’s egress in a large amount of control plane state in pathlet rout-
tunnel router, which delivers it to the final destination us- ing. Routing deflections and path splicing avoid this control
ing the endpoint ID. This arrangement can reduce forward- plane state by not propagating alternate path information;
ing state since most routers need only know about egresses, the tradeoff is that the senders and ASes in those protocols
rather than all endpoints. Pathlet routing can emulate LISP cannot identify which end-to-end paths are being used.
by concatenating a pathlet representing the tunnel and a
pathlet representing the remainder of the route.
6. EXPERIMENTAL EVALUATION
IP strict source routing [8], IP loose source rout- We implemented pathlet routing as a custom software
ing [8] and MIRO [28]. Pathlet routing can emulate these router, and evaluated it in a cluster environment. We de-
protocols in a straightforward manner. The common thread scribe the structure of our implementation in Sec. 6.1, the
in these protocols and LISP is routing via waypoints or tun- evaluation scenarios in Sec. 6.2, and the results in Sec. 6.3.
nels, which we find can be emulated with MIRO’s data plane.
(Note, however, that MIRO’s data and control planes were 6.1 Implementation
intended for a significantly different usage scenario in which We implemented pathlet routing as a user-space software
most paths are set up via standard BGP and a relatively router, depicted in Fig. 4. Each router runs as a separate
small number of additional tunnels are constructed.) We process and connects to its neighbors using TCP connec-
omit a full description due to space constraints. tions, on which it sends both data and control traffic.
5.4 Protocols pathlet routing cannot emulate Our router contains three main modules: a vnode man-
ager, a disseminator, and a controller. Through the imple-
Feedback Based Routing (FBR) [32] is source routing mentation we found that it is possible to shield the core
at the AS level, with each link tagged with an access control policy module (the controller) from the details of pathlet
rule. A rule either whitelists or blacklists a packet based dissemination and vnode management, making it compact
on prefixes of its source or destination IP address. Pathlet and easy to tune to the specific needs an AS might have.
routing cannot emulate FBR for two reasons. First, it is We describe these three modules briefly.
difficult for pathlet routing to represent policies based on
upstream hops. Essentially, the only way to carry utilizable
information about a packet’s origin is to encode the origin Install/remove
into the vnodes the packet follows, which would increase the forwarding entries

number of necessary vnodes and hence the amount of state Controller

Create/delete
by a factor of n relative to FBR, where n is the number of Vnodes
nodes in the network. (Note that we solved a similar prob- Store these
lem in emulating NIRA, but NIRA’s solution used the same local pathlets Vnode
amount of state as ours.) The second problem in emulat-
ing FBR is that FBR has both blacklisting and whitelisting, Advertise
these Pathlet Vnode
while pathlet routing effectively has only whitelisting. FBR pathlets to (un)available Manager
can therefore represent some policies efficiently for which these peers

pathlet routing would require significantly more state.

Data Ingress
But FBR cannot emulate pathlet routing either. For ex- Pathlet Store Packets Vnode
ample, controlling access based only on source and destina- Announce-
Disseminator ment
tion address ignores intermediate hops which can be taken Messages
into account by pathlet routing (and BGP). Pathlet advertisements
and withdrawals
Routing deflections [31] and interdomain path splic- wire
ing [23]. In these protocols, each hop can permit multiple
alternate paths to the destination. The packet includes a Figure 4: Structure of the software pathlet router.
 The vnode manager is responsible for directing incoming ber of entries varies with the node degree for LT routers, and
data packets to vnodes, which store their forwarding tables with the size of the network for PV nodes. As a result, the
and perform the lookup, as well as for sending the data LT nodes have a mean of 5.19 entries in the all-LT case and
packets out to the next hop. The vnode manager also sends 5.23 in the mixed case. PV averages 400.5 entries in the
and receives control messages that inform peers about the mixed and all-PV cases.
ingress vnodes. We also analyzed an AS-level topology of the Internet gen-
The disseminator stores the pathlets and sends and re- erated by CAIDA [5] from Jan 5 2009. Using LT policies in
ceives pathlet announcements and withdrawals. When a this topology results in a maximum of 2, 264 and a mean of
new pathlet becomes available or a pathlet is withdrawn, only 8.48 pathlets to represent an AS. In comparison, BGP
it notifies the controller. When the controller decides to ad- FIBs would need to store entries for the between 132, 158
vertise or withdraw a pathlet from a particular neighbor, it and 275, 779 currently announced IP prefixes, depending on
calls the disseminator to execute the decision. aggregation [2]. Thus, in this case LT policies offer more
While the vnode manager and the disseminator are gen- than a 15, 000× reduction in forwarding state relative to
eral and oblivious to the AS’s policy, the controller encap- BGP.
sulates the policy logic. It implements the policy by con-
structing and deleting pathlets and vnodes and by deciding Route availability. One of the principal advantages of
which pathlets to announce to which peers. In our imple- multipath routing is higher availability in the case of failure,
mentation, one makes a router an LT router or BGP-style since the source can select a different path without waiting
router by picking the corresponding controller. for the control plane to reconverge.
We measure how much LT policies improve availability as
6.2 Evaluation scenarios follows. We allow the network to converge, and then take a
snapshot of the routers’ forwarding tables and the pathlets
Policies. We tested three types of policies: LT policies, they know. We then select a random set of links to be failed,
Path Vector-like policies, and mixed policies, with 50% of and determine for each pair of routers (X, Y ) whether there
nodes randomly chosen to use LT policies and the rest using is a working X ; Y route in the data plane, using only the
PV. pathlets that X knows. If so, then X has a continuously
The special case of LT policies that we test are valley-free working path to Y , without waiting for the control plane to
routes, as in Fig. 2. The PV policies emulate BGP. Specif- detect the failures and re-converge. Thus, we are measuring
ically, we mimic the common BGP decision process of pre- how often we can avoid PV’s transient disconnectivity. (This
ferring routes through customers as a first choice, through measurement ignores the algorithm X uses to find a working
peers as a second choice, and providers last. We then break path among its options, which is outside the scope of this
ties based on path length and router ID, similar to BGP. We work. However, a simple strategy, like trying a maximally
use the common BGP export policies of valley-free routes. disjoint path after a failure, is likely to perform well.)
Topologies. We tested two types of topology. Internet- The results of this experiment are shown in Fig. 6. Rel-
like topologies annotated with customer-provider-peer re- ative to PV, which emulates the availability of BGP, LT
lationships were generated using the algorithm of [7]. Each has significantly improved availability. In fact, it has the
AS is represented as a single router. Random graphs were best possible availability assuming ASes allow only valley-
generated using the G(n, m) model, i.e., n nodes and m ran- free routes, because every path allowed by policy is usable
dom edges, excluding disconnected graphs. In the random in the data plane. In the random graph topology, we see a
graph there are no business relationship annotations; ASes bigger connectivity improvement in part because the possi-
prefer shortest paths and all paths are exported. Unless oth- ble paths are not constrained by valley-freeness.
erwise stated, these graphs have 400 nodes and an average In the mixed environment, the LT nodes obtain the ma-
of 3.8 neighbors per node. jority of the improvement of a full deployment of LT nodes.
To further illustrate this relationship, Fig. 6 shows how con-
Event patterns. We show results for several cases: the nectivity improves as a function of the fraction of nodes that
initial convergence process; the state of the network after use LT policies, in an 80-node random graph. Here the num-
convergence; and a sequence of events in which each link ber of failed links is fixed at 5, and we report the mean of 5
fails and recovers, one at a time in uniform-random order, trials.
with 8 seconds between events for a total experiment length In the random graph topology in Fig. 6, we also show
of 3.6 hours. We implemented link failures by dropping the availability when fewer pathlets are advertised: in particu-
TCP connection and link recovery by establishing a new lar, each router advertises only the shortest path tree (see
TCP connection. Sec. 3.3). This worsens availability, but improves the num-
Metrics. We record connectivity, packet header size, for- ber of control messages, as we describe next.
warding table size, number of control plane messages, and
Control plane messages. Fig. 7 shows the CDF of the
control plane memory use. The CDFs of these metrics that
number of messages received by a router following a link
we present are the result of three trials for each evaluation
failure or recovery event.
scenario, each with a fresh topology. We show all data points
Consider the Internet-like topology. LT has more objects
(routers or source-destination pairs) from all trials in a single
being announced than PV: an average of 5.19 pathlets per
CDF.
node, vs. 1 per destination in the PV case. One might
6.3 Results therefore expect 5.19× as many messages. However, LT has
only 1.69× as many as PV. There are two factors that cause
Forwarding plane memory. Fig. 5 shows a CDF of the this. First, when a link recovers, PV needs to send many
number of forwarding table entries at each router. The num- messages because its path preferences (selected by the BGP
 Internet-Like Random
1.0 1.0
LT
LT in Mixed

Fraction of Samples
0.8 LT 0.8

Fraction of Samples
Figure 5: Forwarding table LT in Mixed PV in Mixed
0.6 PV in Mixed 0.6 PV
(FIB) size for the Internet-
0.4 PV 0.4
like topology (left) and the
random graph (right). 0.2 0.2
0.0 0.0
0 100 200 300 400 0 100 200 300 400
Number of Forwarding Table Entries Number of Forwarding Table Entries

Effect of LT Nodes on Connectivity

Internet-Like 0.40 Random
0.10

Fraction of (src, dst) Pairs Disconnected

PV PV
Fraction of (src, dst) Pairs Disconnected

Fraction of (src, dst) Pairs Disconnected

PV in Mixed 0.35 PV in Mixed
0.25 LT in Mixed LT Tree
LT 0.30 LT in Mixed
0.08
0.20 LT
0.25
0.06 From PV Nodes
0.15 0.20 From LT Nodes
0.15 0.04
0.10
0.10
0.05 0.02
0.05
0.00
0.00 0.02 0.04 0.06 0.08 0.10 0.00
0.00 0.02 0.04 0.06 0.08 0.10 0.000.0 0.1 0.2 0.3 0.4 0.5
Fraction of Links Failed Fraction of Links Failed Fraction of LT Nodes in the Graph

Figure 6: Probability of disconnection for a varying number of link failures in the Internet-like topology (left)
and the random graph (center), and as a function of the number of LT nodes (right).

Internet-Like Random
1.0 1.0

0.8 0.8
Figure 7: CDF of the
Fraction of Samples

Fraction of Samples

number of messages re- PV PV

0.6 0.6 PV in Mixed
ceived by a router follow- PV in Mixed LT in Mixed
ing a link state change, for LT in Mixed LT Tree
0.4 LT 0.4 LT
the Internet-like topology
(left) and the random graph
(right). 0.2 0.2

0.0 0.0
100 101 102 103 100 101 102 103 104
Number of Messages Number of Messages

Average Number of Messages Received by a Router Average Maximum Control Memory at a Router During
During Initial Convergence. Convergence and Failure/Recovery of Each Link
3000 180
LT PV in MX
PV in MX 160 LT
2500 LT in MX LT in MX
140
Control Memory Size in KB

PV PV
Number of Messages

2000
Figure 8: Scaling of messag- 120
ing and control memory in 1500 100
the Internet-like graph, for
100, 200, 300, 400, and 500 80
1000
nodes. 60
500
40
0
100 150 200 250 300 350 400 450 500 20
100 150 200 250 300 350 400 450 500
Topology Size Topology Size
 Internet-Like Random
1.0 1.0

0.8 0.8

Fraction of Samples

Fraction of Samples
Figure 9: CDF of the size of 0.6 PV 0.6 PV
the route field in the packet PV in Mixed PV in Mixed
header, for the Internet-like LT in Mixed LT in Mixed
0.4 LT 0.4 LT
topology (left) and the ran-
dom graph (right).
0.2 0.2

0.0 0.0
0 2 4 6 8 10 12 0 2 4 6 8 10 12
Maximum Header Size (bytes) Maximum Header Size (bytes)

decision process) cause it to switch back to paths using the of bits may vary as the packet travels along the route; we
recovered link. In contrast, in the LT case we need only report the maximum (which for LT policies occurs at the
advertise a single pathlet for the recovered link (see discus- source).
sion in Sec. 3.2). Second, according to our dissemination Header length scales with the path length. In our Internet-
algorithm (Sec. 3.3), many of those 5.19 pathlets are never like graph, the mean path length is 2.96 and headers average
disseminated to some parts of the network, because they are 4.21 bytes. An AS-level topology of the Internet mapped by
not reachable. CAIDA [5] from Jan 22 2009 has mean path length 3.77,
In the random graph topology, the second factor is not so we could therefore extrapolate that mean header length
true. Not only does every router learn all pathlets, but it would be about 27% greater in a full Internet topology, i.e.,
learns them from all of its neighbors. The result is 10.3× less than 6 bytes, assuming other characteristics of the topol-
as many messages as PV. However, this can be reduced if ogy remain constant. Even the maximum header length of
nodes simply advertise fewer pathlets, which is safe as long 12.5 bytes in our implementation would not add prohibitive
as the advertised pathlets reach every destination. The line overhead to packet headers.
“LT Tree” in Fig. 7 shows the case when we advertise only
a tree reaching all destinations (described in Sec. 3.3). This
reduces the messaging overhead to just 2.23×. The tradeoff 7. RELATED WORK
is that there are fewer choices of paths, and connectivity in We have compared the policy expressiveness of a num-
the face of failures is worsened, as shown in Fig. 6. ber of multipath routing protocols in Section 5. Here we
Fig. 8 plots the messaging cost for initial convergence as a compare other aspects of the protocols.
function of n, the number of nodes in the topology. LT and MIRO [28] uses BGP routes by default. Only those ASes
PV have similar scaling properties for this metric because which need an alternate path (say, to avoid routing through
both, like BGP, use a path vector dissemination algorithm one particularly undesirable AS) need to set up a path.
to announce objects; PV announces one object per node and But this increases the latency of constructing a path, and
LT announces O(δ) (where δ is the number of neighbors), increases forwarding plane state if there are many paths.
which is independent of n. MIRO is likely easier to deploy than pathlet routing in to-
day’s network of BGP routers.
Control plane memory. Since control plane memory use
NIRA [30] provides multiple paths and very small for-
is dependent on the implementation, we evaluate its scal-
warding and control state as long as all routers have short
ing properties. Asymptotically, if the n nodes each have δ
paths to the “core”; but for arbitrary networks, forwarding
neighbors and the mean path length is `, BGP and PV poli-
state may be exponentially large. The scheme requires a po-
cies need to store Θ(n) announcements of mean size Θ(`)
tentially complicated assignment of IP addresses, and works
from up to δ neighbors, for a total of Θ(nδ`) state in the
primarily for Internet-like topologies with valley-free routing
worst case. In a full LT deployment there are Θ(δn) path-
policies. Exceptions to these policies may be difficult.
lets; each announcement has size Θ(`) and is learned from
Routing deflections [31] and path splicing [23] permit poli-
up to d neighbors, for a total of Θ(nδ 2 `) in the worst case.
cies specified on a per-destination basis, while still providing
Fig. 8 shows the mean (over routers and over three trials)
many path choices with limited deviations from the primary
of the maximum (over time for each of trial) of the control
paths. However, sources are not aware of what paths they
plane state at each router. A trial consists of allowing the
are using, [31] has relatively limited choice of paths, and [23]
network to converge and then failing and recovering each
can encounter forwarding loops.
link once. The results confirm that PV and LT policies scale
IP’s strict source routing and loose source routing [8] pro-
similarly as a function of n. It is also apparent that with
vide source-controlled routing but have limited control for
the Internet-like topologies, we don’t reach the worst case
policies. For this reason and due to security concerns un-
in which LT is a factor δ worse than P V .
related to our setting, they are not widely used. They can
It may be possible to optimize our dissemination algo-
also result in long headers because each hop is specified as
rithm to reduce control state and messaging; see Sec. 8.
an IP address, unlike our compact forwarding identifiers.
Header size. Fig. 9 shows the CDF over source-destination Feedback based routing [32] suggested source-based route
pairs (X, Y ) of the number of bits in the route field of the quality monitoring that is likely to be a useful approach for
packet header for the shortest X ; Y route. The number pathlet routing.
 Platypus [24] is similar to loose source routing except each 9. REFERENCES
waypoint can be used only by authorized sources to reach ei- [1] D. G. Andersen, H. Balakrishnan, M. F. Kaashoek, and
R. Morris. Resilient overlay networks. In Proc. 18th ACM
ther any destination, or a specified IP prefix. Pathlet routing SOSP, October 2001.
supports a different set of policies and enforces these using [2] Routing table report.
the presence or absence of forwarding tables, rather than http://thyme.apnic.net/ap-data/2009/01/05/0400/mail-global.
cryptography. [3] Avaya. Converged network analyzer.
http://www.avaya.com/master-usa/en-
R-BGP [18] adds a small number of backup paths that us/resource/assets/whitepapers/ef-lb2687.pdf.
ensure continuous connectivity under a single failure, with [4] B. Awerbuch, D. Holmer, H. Rubens, and R. Kleinberg.
relatively minimal changes to BGP. However, it somewhat Provably competitive adaptive routing. In INFOCOM, 2005.
increases forwarding plane state and is not a full multipath [5] CAIDA AS ranking. http://as-rank.caida.org/.
[6] D. Clark, J. Wroclawski, K. Sollins, and R. Braden. Tussle in
solution. For example, sources could not use alternate paths cyberspace: defining tomorrow’s Internet. In SIGCOMM, 2002.
to improve path quality. [7] X. Dimitropoulos, D. Krioukov, A. Vahdat, and G. Riley.
LISP [9] reduces forwarding state and provides multiple Graph annotations in modeling complex network topologies.
paths while remaining compatible with today’s Internet. Al- ACM Transactions on Modeling and Computer Simulation
(to appear), 2009.
though it can limit expansion of forwarding table size, LISP’s [8] J. P. (ed.). DARPA internet program protocol specification. In
forwarding tables would still scale with the size of the non- RFC791, September 1981.
stub Internet, as opposed to scaling with the number of [9] D. Farinacci, V. Fuller, D. Meyer, and D. Lewis. Locator/ID
neighbors as in our LT policies. separation protocol (LISP). In Internet-Draft, March 2009.
[10] B. Ford and J. Iyengar. Breaking up the transport logjam. In
MPLS [26] has tunnels and labels similar to our pathlets HOTNETS, 2008.
and FIDs. It also shares the high level design of having the [11] L. Gao and J. Rexford. Stable Internet routing without global
source or ingress router map an IP address to a sequence coordination. IEEE/ACM Transactions on Networking,
of labels forming a source route. However the common use 9(6):681–692, December 2001.
[12] P. B. Godfrey, M. Caesar, I. Haken, S. Shenker, and I. Stoica.
of these mechanisms is substantially different from pathlet Stable Internet route selection. In NANOG 40, June 2007.
routing: tunnels are not typically concatenated into new, [13] T. Griffin and J. Sobrinho. Metarouting. In ACM SIGCOMM,
longer tunnels, or inductively built by adding one hop at 2005.
a time. To the best of our knowledge MPLS has not been [14] K. P. Gummadi, H. V. Madhyastha, S. D. Gribble, H. M. Levy,
and D. Wetherall. Improving the reliability of internet paths
adapted to an interdomain policy-aware routing. with one-hop source routing. In Proc. OSDI, 2004.
Metarouting [13], like pathlet routing, generalizes routing [15] G. Huston. BGP routing table analysis reports, 2009.
protocols. It would be interesting to explore whether pathlet http://bgp.potaroo.net/.
routing can be represented in the language of [13]. [16] E. Karpilovsky and J. Rexford. Using forgetful routing to
control BGP table size. In CoNEXT, 2006.
[17] N. Kushman, S. Kandula, and D. Katabi. Can you hear me
8. CONCLUSION now?! it must be BGP. In Computer Communication Review,
2007.
Pathlet routing offers a novel routing architecture. Through [18] N. Kushman, S. Kandula, D. Katabi, and B. Maggs. R-BGP:
its building blocks of vnodes and pathlets, it supports com- Staying connected in a connected world. In NSDI, 2007.
plex BGP-style policies while enabling and incentivizing the [19] C. Labovitz, A. Ahuja, A. Bose, and F. Jahanian. Delayed
adoption of policies that yield small forwarding plane state Internet routing convergence. In ACM SIGCOMM, 2000.
[20] K. Lakshminarayanan, I. Stoica, S. Shenker, and J. Rexford.
and a high degree of path choice. We next briefly discuss Routing as a service. Technical Report UCB/EECS-2006-19,
some limitations and future directions. UC Berkeley, February 2006.
We suspect it is possible to optimize our path vector-based [21] Z. M. Mao, R. Bush, T. Griffin, and M. Roughan. BGP
pathlet dissemination algorithm. The techniques of [16] may beacons. In IMC, 2003.
[22] D. Meyer, L. Zhang, and K. Fall. Report from the iab workshop
be very easy to apply in our setting to reduce control plane on routing and addressing. In RFC2439, September 2007.
memory use from O(δ`) to O(`) per pathlet, where δ is the [23] M. Motiwala, M. Elmore, N. Feamster, and S. Vempala. Path
number of neighbors and ` is the mean path length. Routers splicing. In ACM SIGCOMM, 2008.
could also pick dissemination paths based on heuristics to [24] B. Raghavan and A. C. Snoeren. A system for authenticated
policy-compliant routing. In ACM SIGCOMM, 2004.
predict stability, which for common failure patterns can sig-
[25] Y. Rekhter, T. Li, and S. Hares. A border gateway protocol 4
nificantly reduce the number of update messages [12]. The (BGP-4). In RFC4271, January 2006.
more radical approach of [30] could also be used to dramat- [26] E. Rosen, A. Viswanathan, and R. Callon. Multiprotocol label
ically reduce state in Internet-like environments. switching architecture. In RFC3031, January 2001.
Traffic engineering is an important aspect of routing that [27] S. Savage, T. Anderson, A. Aggarwal, D. Becker, N. Cardwell,
A. Collins, E. Hoffman, J. Snell, A. Vahdat, G. Voelker, and
we have not evaluated. One common technique—advertising J. Zahorjan. Detour: Informed Internet routing and transport.
different IP to different neighbors to control inbound traffic— In IEEE Micro, January 1999.
is straightforward to do in our LT policies. But source- [28] W. Xu and J. Rexford. MIRO: Multi-path Interdomain
ROuting. In SIGCOMM, 2006.
controlled routing would dramatically change the nature of
[29] X. Yang. NIRA: a new Internet routing architecture. Technical
traffic engineering, potentially making it more difficult for Report Ph.D. Thesis, MIT-LCS-TR-967, Massachusetts
ISPs (since they have less control) and potentially making Institute of Technology, September 2004.
it easier (since sources can dynamically balance load). [30] X. Yang, D. Clark, and A. Berger. NIRA: a new inter-domain
routing architecture. IEEE/ACM Transactions on
Networking, 15(4):775–788, 2007.
Acknowledgements [31] X. Yang and D. Wetherall. Source selectable path diversity via
routing deflections. In ACM SIGCOMM, 2006.
We thank the authors of [7] for supplying the Internet-like [32] D. Zhu, M. Gritter, and D. Cheriton. Feedback based routing.
topologies. Computer Communication Review (CCR), 33(1):71–76, 2003.