Peer-to-Peer Middleware

Routing Overlays
Peer to Peer Distributed Systems
Peer-to-Peer Middleware Routing Overlays
Peer-to-Peer Middleware Routing Overlays

Overview:

● Routing overlays in P2P systems are virtual networks built on top of the internet’s infrastructure to facilitate efficient data routing between
peers. They allow peers to discover each other and locate data efficiently in a decentralized manner.

● There are two types of P2P overlay networks: structured and unstructured. Structured P2P overlays distribute data in a deterministic manner,
which allows for high performance in terms of resource discovery and access.

● Common P2P routing protocols include Distributed Hash Tables (DHTs) - Chord, Pastry, and Kademlia.

Problems Solved:

● Efficient Data Location: Routing overlays ensure that data can be quickly located and retrieved from decentralized nodes without a central
directory (unlike Napster).
Efficient routing: P2P networks can offer an efficient routing architecture that is self-organizing and robust.
● Scalability: Routing overlays distribute the responsibility of routing and data storage across all peers, making the system more scalable.
● Fault Tolerance: Since data and routing paths are distributed across multiple nodes, the failure of one peer doesn't bring down the network.
Distributed Hash Table
A Distributed Hash Table (DHT) is a decentralized system for mapping keys to values across a network of nodes, commonly used in
peer-to-peer networks.

Key Points:

1. Hashing & Key-Value Storage:

○ Keys (unique identifiers) are generated by hashing data (e.g., file names).
○ Values are stored at nodes with IDs close to the key in the network’s address space.
2. Efficient Lookup:
○ Uses structured overlays (like Chord or Kademlia) to enable efficient O(log N) lookups, where N is the number of
nodes.
○ Nodes route data to other nodes based on proximity in the DHT’s address space.
3. Fault Tolerance & Scalability:
○ Replicates data across multiple nodes for redundancy.
○ Handles node joins and departures dynamically, making it scalable and resilient.

DHTs are widely used in decentralized storage, file-sharing networks, and applications requiring efficient distributed data retrieval.
Distributed Hash Table
DHT in BitTorrent
● Step 1: A user wants to find peers sharing a specific file. Their BitTorrent client calculates the hash (key) of the torrent file.
● Step 2: The client sends a DHT lookup request to its known neighbors. These neighbors forward the request to peers whose
IDs are closer to the hash.
● Step 3: Eventually, the request reaches a peer responsible for the hash. This peer responds with a list of IP addresses of other
peers sharing the file.
● Step 4: The client then connects directly to those peers to download the file.

DHT enables peer discovery without centralized trackers and makes systems like BitTorrent scalable and robust.
Distributed Hash Table Example
1. Chord

● Structure: Ring-based Distributed Hash Table (DHT).

● Routing Method: Nodes maintain a "finger table" with pointers to other nodes at increasing distances, making lookups efficient.
● Lookup Complexity: O(log N).
● Use Cases: Frequently used in distributed file systems and resource discovery systems.

2. Kademlia

● Structure: XOR-based DHT.

● Routing Method: Nodes use XOR distance to determine closeness and maintain k-buckets to store information about nearby nodes.
● Lookup Complexity: O(log N).
● Use Cases: Popular in file-sharing networks like BitTorrent and decentralized applications like IPFS.

3. Pastry

● Structure: Prefix-based DHT.

● Routing Method: Nodes route requests based on shared prefixes, matching closer prefixes at each hop.
● Lookup Complexity: O(log N).
● Use Cases: Used in resilient distributed applications, such as large-scale object storage and content location systems.
Kademlia
Kademlia is a peer-to-peer (P2P) distributed hash table (DHT) that was designed in 2002 by Petar Maymounkov and David Mazières.
It's used to structure networks and exchange information through node lookups.

Here are some key features of Kademlia:

- Node IDs: Each Kademlia node and key is identified by a 160-bit identifier.
- Distance metric: The distance between two IDs is defined as the XOR of the two IDs. For example, the distance between 1 and
4 is 5.
- Binary tree: Kademlia treats each node on a network as a leaf on a binary tree.
- Routing table: The routing table organizes peers based on how similar their keys are.
- Key-value storage: Each value is stored at the k nodes whose node IDs are closest to the key ID.
- Protocol: The Kademlia protocol ensures that each node knows of at least one node on each of its sub-trees.

- Kademlia was the first P2P system to combine provable consistency and performance, latency-minimizing routing, and a symmetric,
unidirectional topology.
Why XOR?
Routing Tables
Kademlia - O(log n) hop routing
BitTorrent DHT
Most implementations of BitTorrent DHT use a modified version of Kademlia, specifically adapted to suit the needs
of BitTorrent peer discovery and file sharing.

Key points about BitTorrent DHT and Kademlia:

1. Kademlia Structure: BitTorrent DHT uses Kademlia's XOR-based distance metric and routing structure to
locate peers storing information about a particular torrent.

2. Modifications: BitTorrent DHT has simplified some aspects of Kademlia, such as reduced bucket sizes and
simplified message types, to make it more efficient in high-churn environments (frequent node joins/leaves).

3. Routing Efficiency: The modifications maintain the essential efficiency of Kademlia’s O(log N) lookups,
making it highly effective for decentralized peer discovery.

While it’s based on Kademlia, BitTorrent DHT is not a pure Kademlia implementation but rather an adapted
version tailored to BitTorrent’s unique requirements.
Kademlia’s applications
Kademlia is a versatile Distributed Hash Table (DHT) protocol that has a variety of applications in decentralized networking and peer-to-peer systems. Here are some
key applications of Kademlia:

1. File Sharing Networks

● Example: BitTorrent DHT

● Use Case: Enables efficient peer discovery and file retrieval by allowing users to find and download files from multiple peers without a central server.

2. Decentralized Storage Systems

● Example: IPFS (InterPlanetary File System)

● Use Case: Kademlia facilitates distributed data storage and retrieval, allowing files to be stored across multiple nodes, improving redundancy and accessibility.

3. Content Delivery Networks (CDNs)

● Use Case: Enhances the delivery of content by locating and serving files from the nearest peers, reducing latency and bandwidth usage.

4. Blockchain and Cryptocurrencies

● Example: Ethereum’s Swarm

● Use Case: Kademlia is used for decentralized storage of smart contracts and transaction data, ensuring data availability and retrieval efficiency in blockchain
environments.

Many more…
Bittorrent Tracker vs Kademila
Aspect Centralized Decentralized
BitTorrent DHT
Tracker
Architecture Centralized Decentralized
Peer Discovery Tracker provides Nodes find peers
a list of peers by querying other
nodes in the DHT
Single Point of Yes, if the tracker No, fault-tolerant
Failure goes down, peer due to distributed
discovery stops structure
Scalability Limited by tracker Highly scalable,
capacity nodes distribute
load
Privacy Tracker knows all More private, as
peer IPs no single node
has full network
view
Network Low for peers, as Higher, as peers
Overhead tracker provides need to perform
full peer list recursive lookups
directly
Usage Traditional Modern BitTorrent
BitTorrent clients with DHT
support
Home work…
How does a new node join the Network? How is its routing table populated? What
happens to Routing table of other nodes?
What happens when a node leave the Network?
References
- Recommended -> https://www.youtube.com/watch?v=1QdKhNpsj8M
- https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf