PASTRY

Please do not cite this document.

1 Introduction after a node failure or the arrival of a new node, the in-
variants in all affected routing tables can be restored by
In this note we describe the current design of Pastry, a exchanging O(log 2b N ) messages.
peer-to-peer location and routing substrate upon which For the purposes of routing, nodeIds and keys are
Scribe was built. Pastry forms a robust, self-organizing thought of as a sequence of digits with base 2 b . A node’s
overlay network in the Internet. Any Internet-connected routing table is organized into dlog 2b N e rows with 2b 1
host that runs the Pastry software and has proper creden- entries each. The 2 b 1 entries in row n of the routing ta-
tials can participate in the overlay network. ble each refer to a node whose nodeId matches the present
There have been three primary papers written talking node’s nodeId in the first n digits, but whose n + 1th digit
about different aspects of Pastry: has one of the 2 b 1 possible values other than the n + 1th
In [5] the original design of Pastry was described. Since digit in the present node’s id. The uniform distribution of
this paper was published we have removed the neighbor- nodeIds ensures an even population of the nodeId space;
hood set, and simplified the joining algorithm [2]. We thus, only dlog2b N e levels are populated in the routing
have also examined how to make Pastry resilient to fail- table. Each entry in the routing table refers to one of po-
ure and attacks [1]. tentially many nodes whose nodeId have the appropriate
prefix. Among such nodes, the one closest to the present
node (according to a scalar proximity metric, such as the
2 Pastry round trip time) is chosen.
Each Pastry node has a unique, 128-bit nodeId. The set In addition to the routing table, each node maintains
of existing nodeIds is uniformly distributed; this can be IP addresses for the nodes in its leaf set, i.e., the set of
achieved, for instance, by basing the nodeId on a secure nodes with the l=2 numerically closest larger nodeIds, and
hash of the node’s public key or IP address. Given a mes- the l=2 nodes with numerically closest smaller nodeIds,
sage and a key, Pastry reliably routes the message to the relative to the present node’s nodeId.
Pastry node with the nodeId that is numerically closest to Figure 2 shows the path of an example message. In
the key, among all live Pastry nodes. Assuming a Pastry each routing step, the current node normally forwards the
network consisting of N nodes, Pastry can route to any message to a node whose nodeId shares with the key a
node in less than dlog 2b N e steps on average (b is a config- prefix that is at least one digit (or b bits) longer than the
uration parameter with typical value 4). With concurrent prefix that the key shares with the current nodeId. If no
node failures, eventual delivery is guaranteed unless l=2 such node is found in the routing table, the message is
or more nodes with adjacent nodeIds fail simultaneously forwarded to a node whose nodeId shares a prefix with the
(l is an even integer parameter with typical value 16). key as long as the current node, but is numerically closer
The tables required in each Pastry node have only to the key than the current nodeId. Such a node must exist
(2b 1)  dlog2b N e + l entries, where each entry maps in the leaf set unless the nodeId of the current node or its
a nodeId to the associated node’s IP address. Moreover, immediate neighbour is numerically closest to the key, or

1
 0 1 2 3 4 5 7 8 9 a b c d e f O 2128 - 1
x x x x x x x x x x x x x x x

6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 d471f1
0 1 2 3 4 6 7 8 9 a b c d e f d467c4
x x x x x x x x x x x x x x x d462ba
d46a1c
6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 d4213f
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
0 1 2 3 4 5 6 7 8 9 b c d e f
x x x x x x x x x x x x x x x

6 6 6 6 6 6 6 6 6 6 6 6 6 6 6
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 Route(d46a1c) d13da3
a a a a a a a a a a a a a a a
0 2 3 4 5 6 7 8 9 a b c d e f
x x x x x x x x x x x x x x x
65a1fc

Figure 1: Routing table of a Pastry node with Figure 2: Routing a message from node
nodeId 65a1x, b = 4. Digits are in base 16, x 65a1f c with key d46a1c. The dots depict live
represents an arbitrary suffix. The IP address nodes in Pastry’s circular namespace.
associated with each entry is not shown.

l=2 adjacent nodes in the leaf set have failed concurrently. distance traveled by two messages sent to the same key
before their routes converge. Simulations show that, given
our network topology model, the average distance trav-
2.1 Locality eled by each of the two messages before their routes con-
verge is approximately equal to the distance between their
Next, we discuss Pastry’s locality properties, i.e., the
respective source nodes.
properties of Pastry’s routes with respect to the proxim-
ity metric. The proximity metric is a scalar value that
reflects the “distance” between any pair of nodes, such as 2.2 Node addition and failure
the round trip time. It is assumed that a function exists
that allows each Pastry node to determine the “distance” A key design issue in Pastry is how to efficiently and dy-
between itself and a node with a given IP address. namically maintain the node state, i.e., the routing table,
We limit our discussion to two of Pastry’s locality prop- leaf set and neighborhood sets, in the presence of node
erties that are relevant to Scribe. The short routes prop- failures, node recoveries, and new node arrivals.
erty concerns the total distance, in terms of the proximity Briefly, an arriving node with the newly chosen nodeId
metric, that messages travel along Pastry routes. Recall X can initialize its state by contacting a nearby node A
that each entry in the node routing tables is chosen to re- (according to the proximity metric) and asking A to route
fer to the nearest node, according to the proximity metric, a special message using X as the key. This message is
with the appropriate nodeId prefix. As a result, in each routed to the existing node Z with nodeId numerically
step a message is routed to the nearest node with a longer closest to X 1 . X then obtains the leaf set from Z , and
prefix match. Simulations performed on several network the ith row of the routing table from the ith node encoun-
topology models show that the average distance traveled tered along the route from A to Z . One can show that
by a message is between 1.59 and 2.2 times the distance using this information, X can correctly initialize its state
between the source and destination in the underlying In- and notify nodes that need to know of its arrival.
ternet [2]. 1 In the exceedingly unlikely event that X and Z are equal, the new
The route convergence property is concerned with the node must obtain a new nodeId.

2
 To handle node failures, neighboring nodes in the forward(msg,key,nextId) called by Pastry just before a
nodeId space (which are aware of each other by virtue of message is forwarded to the node with nodeId = nex-
being in each other’s leaf set) periodically exchange keep- tId. The application may change the contents of the
alive messages. If a node is unresponsive for a period T , it message or the value of nextId. Setting the nextId to
is presumed failed. All members of the failed node’s leaf NULL will terminate the message at the local node.
set are then notified and they update their leaf sets. Since
the leaf sets of nodes with adjacent nodeIds overlap, this newLeafs(leafSet) called by Pastry whenever there is a
update is trivial. A recovering node contacts the nodes change in the leaf set. This provides the application
in its last known leaf set, obtains their current leaf sets, with an opportunity to adjust application-specific in-
updates its own leaf set and then notifies the members of variants based on the leaf set.
its new leaf set of its presence. Routing table entries that
refer to failed nodes are repaired lazily; the details are de-
scribed in [5, 2]. 3 Applications
A number of applications have been built using Pastry,
2.3 Pastry API including PAST, a persistent, global storage utility [3, 6],
In this section, we briefly describe the application pro- Scribe, an application level-multicast system [7, 8], and a
gramming interface (API) exported by Pastry to applica- cooperative web cache [4].
tions such as Scribe. The presented API is slightly simpli- For more information please visit the Pastry web
fied for clarity. Pastry exports the following operations: site: www.research.microsoft.com\˜antr\
Pastry
nodeId = pastryInit(Credentials) causes the local node
to join an existing Pastry network (or start a new
one) and initialize all relevant state; returns the lo- References
cal node’s nodeId. The credentials are provided by
[1] M. Castro, P. Druschel, A. Ganesh, A. Rowstron, and D. S.
the application and contain information needed to
Wallach. Security for peer-to-peer routing overlays, 2002.
authenticate the local node and to securely join the
Submitted for publication.
Pastry network. A full discussion of Pastry’s secu-
rity model is beyond the scope of this paper. [2] M. Castro, P. Druschel, Y. C. Hu, and A. Rowstron.
Topology-aware routing in structured peer-to-peer overlay
route(msg,key) causes Pastry to route the given message networks, 2002. Submitted for publication.
to the node with nodeId numerically closest to key, [3] P. Druschel and A. Rowstron. PAST: A large-scale, per-
among all live Pastry nodes. sistent peer-to-peer storage utility. In Proc. HotOS VIII,
Schloss Elmau, Germany, May 2001.
send(msg,IP-addr) causes Pastry to send the given mes-
sage to the node with the specified IP address, if that [4] S. Iyer, A. Rowstron, and P. Druschel. Squirrel: A decen-
tralized peer-to-peer web cache. In 12th ACM Symposium
node is live. The message is received by that node
on Principles of Distributed Computing (PODC 2002), July
through the deliver method.
2002.
Applications layered on top of Pastry must export the fol- [5] A. Rowstron and P. Druschel. Pastry: Scalable, distributed
lowing operations: object location and routing for large-scale peer-to-peer sys-
tems. In Proc. IFIP/ACM Middleware 2001, Heidelberg,
deliver(msg,key) called by Pastry when a message is re- Germany, Nov. 2001.
ceived and the local node’s nodeId is numerically [6] A. Rowstron and P. Druschel. Storage management and
closest to key among all live nodes, or when a mes- caching in PAST, a large-scale, persistent peer-to-peer stor-
sage is received that was transmitted via send, using age utility. In Proc. ACM SOSP’01, Banff, Canada, Oct.
the IP address of the local node. 2001.

3
[7] A. Rowstron, A.-M. Kermarrec, P. Druschel, and M. Castro.
Scribe: The design of a large-scale event notification infras-
tructure. In Proc. NGC’2001, London, UK, Nov. 2001.
[8] A. Rowstron, A.-M. Kermarrec, P. Druschel, and M. Castro.
SCRIBE: A large-scale and decentralized publish-subscribe
infrastructure, 2002. Accepted for Journal Selected
Areas in Communications. http://www.research.
microsoft.com/˜antr/SCRIBE/.