Virtual LANs (VLANs)

 Allow us to split switches into separate

(virtual) switches
 Only members of a VLAN can see that
VLAN & VLAN’s traffic
Data Center Networking  Inter-vlan traffic must go through a router
Switch
VLAN X VLAN Y
Edge ports

VLAN X nodes VLAN Y nodes

Local VLANs VLAN operation

 2 VLANs or more within a single switch
 Edge ports, where end nodes are connected, are  As a device enters the network, it
configured as members of a VLAN
automatically assumes the VLAN
 The switch behaves as several virtual switches, sending
traffic only within VLAN members. membership of the port to which it is
 Switches may not bridge any traffic between VLANs, attached.
as this would violate the integrity of the VLAN  The default VLAN for every port in the
broadcast domain.
switch is VLAN 1 and cannot be deleted.
 Traffic should only be routed between VLANs.
 Ports on the switch may be reassigned to
alternate VLANs.
 ARP
Request
VLANs Switch 1 VLANs Switch 1
172.30.1.21 172.30.1.21
172.30.2.12 172.30.2.12
255.255.255.0 255.255.255.0
255.255.255.0 255.255.255.0
VLAN 1 VLAN 1
VLAN 2 VLAN 2

172.30.2.10 172.30.1.23 172.30.2.10 172.30.1.23

1 2 3 4 5 6 . Port 255.255.255.0 255.255.255.0 1 2 3 4 5 6 . Port 255.255.255.0 255.255.255.0
1 2 1 2 2 1 . VLAN VLAN 2 VLAN 1 1 2 1 2 2 1 . VLAN VLAN 2 VLAN 1

TwoTwo
VLANs
VLANs= Two subnets TwoTwo
VLANs
VLANs= Two subnets
 Two Subnets  Two Subnets
• Important notes on VLANs:
 VLANs are assigned to switch ports. There is no “VLAN”  VLANs separate broadcast domains!
assignment done on the host (usually). e.g. without VLAN the ARP would be seen on all subnets.
 In order for a host to be a part of that VLAN, it must be
assigned an IP address that belongs to the proper subnet.
Remember: VLAN = Subnet
 Assigning a host to the correct VLAN is a 2-step process:
 Connect the host to the correct port on the switch.
 Assign to the host the correct IP address depending on the
VLAN membership

VLANs VLANs across switches

 Two switches can exchange traffic from

one or more VLANs
 Inter-switch links are configured as
trunks, carrying frames from all or a
subset of a switch’s VLANs
 VLANs logically segment switched networks based on the functions,
 Each frame carries a tag that identifies
project teams, or applications of the organization regardless of the which VLAN it belongs to
physical location or connections to the network.
 All workstations and servers used by a particular workgroup share the
same VLAN, regardless of the physical connection or location.
 VLANs across switches VLANs across switches
No VLAN Tagged Frames
Tagging

802.1Q Trunk
Trunk Port
VLAN X VLAN Y VLAN X VLAN Y
VLAN Tagging Edge Ports

This is called “VLAN Trunking”

 VLAN tagging is used when a single link needs
to carry traffic for more than one VLAN.

Tagged vs. Untagged 802.1Q

 Edge ports are not tagged, they are just  The IEEE standard that defines how
“members” of a VLAN ethernet frames should be tagged when
 You only need to tag frames in switch-to- moving across switch trunks
switch links (trunks), when transporting  This means that switches from different
multiple VLANs vendors are able to exchange VLAN
 A trunk can transport both tagged and traffic.
untagged VLANs
 As long as the two switches agree on how to handle those
 802.1Q tagged frame 802.1Q Header
• A 4-byte tag header containing a tag protocol identifier
(TPID) and tag control information (TCI) with the following
elements:
TPID
• A 2-byte TPID with a fixed value of 0x8100.
• This value indicates that the frame carries the
802.1Q/802.1p tag information.
TCI
• A TCI containing the following elements:
- Three-bit user priority (8 priority levels, 0 thru 7)
- One-bit canonical format (CFI indicator), 0 = canonical, 1 = noncanonical,
to signal bit order in the encapsulated frame
(www.faqs.org/rfcs/rfc2469.html - “A Caution On the Canonical
Ordering of Link-Layer Addresses”)
- Twelve-bit VLAN identifier (VID)-Uniquely identifies the VLAN to which
the frame belongs, defining 4,096 VLANs, with 0 and 4095 reserved.

VLANS increase complexity Data Center Networking

Major Theme:
What are new networking issues posed by
 You can no longer “just replace” a switch large-scale data centers?
 Now you have VLAN configuration to maintain
 Field technicians need more skills • Network Architecture?
 You have to make sure that all the switch- • Topology design?
to-switch trunks are carrying all the • Addressing?
necessary VLANs • Routing?
 Need to keep in mind when adding/removing VLANs
• Forwarding?
 Data Center Interconnection Cisco Recommended DC Structure:
Structure Illustration
• Nodes in the system: racks of servers Internet Internet
• How are the nodes (racks) inter- Data Center
CR CR

connected? Layer 3
AR AR … AR AR
– Typically a hierarchical inter-connection structure

• Today’s typical data center structure Layer 2 LB S S LB

Cisco recommended data center structure: Key:

starting from the bottom level S S S S … • CR = L3 Core Router
• AR = L3 Access Router
– rack switches • S = L2 Switch
• LB = Load Balancer
– 1-2 layers of (layer-2) aggregation switches • A = Rack of 20 servers
… …
– access routers with Top of Rack switch

– core routers
• Is such an architecture good enough? 18

Data Center Costs Data Center Design Requirements

Amortized Cost* Component Sub-Components
~45% Servers CPU, memory, disk
• Data centers typically run two types of
~25% Power infrastructure UPS, cooling, power distribution applications
~15% Power draw Electrical utility costs – outward facing (e.g., serving web pages to users)
~15% Network Switches, links, transit – internal computations (e.g., MapReduce for web indexing)

*3 yr amortization for servers, 15 yr for infrastructure; 5% cost of money • Workloads often unpredictable:
– Multiple services run concurrently within a DC
• Total cost varies
– upwards of $1/4 B for mega data center – Demand for new services may spike unexpected
– server costs dominate • Spike of demands for new services mean success!
– network costs significant • But this is when success spells trouble (if not prepared)!
• Long provisioning timescales: • Failures of servers are the norm
– new servers purchased quarterly at best – GFS, MapReduce, etc., resort to dynamic re-assignment of
Source: the Cost of a Cloud: Research Problems in Data Center Networks. Sigcomm CCR 2009. chunkservers, jobs/tasks (worker servers) to deal with
Greenberg, Hamilton, Maltz, Patel.
failures; data is often replicated across racks, …
– “Traffic matrix” between servers are constantly changing
 Overall Data Center Design Goal Achieving Agility …
Agility – Any service, Any Server • Workload Management
• Turn the servers into a single large fungible – means for rapidly installing a service’s code on a server
– dynamical cluster scheduling and server assignment 
pool • E.g., MapReduce, …
– Let services “breathe” : dynamically expand and contract – virtual machines, disk images 
their footprint as needed
• Storage Management
• Benefits
– means for a server to access persistent data
– Increase service developer productivity
– distributed file systems (e.g., GFS) 
– Lower cost
– Achieve high performance and reliability • Network Management
– Means for communicating with other servers, regardless of
These are the three motivators for most data center where they are in the data center
infrastructure projects! – Achieve high performance and reliability

Networking Objectives Is Today’s DC Architecture Adequate?

• Hierarchical network; 1+1 redundancy
1. Uniform high capacity • Equipment higher in the hierarchy handles more traffic
– Capacity between servers limited only by their NICs • more expensive, more efforts made at availability
• Servers connect via 1 Gbps UTP to Top-of-Rack switches
– No need to consider topology when adding servers
• Other links are mix of 1G, 10G; fiber, copper
=> In other words, high capacity between two any servers no
matter which racks they are located ! • Uniform high capacity? Internet
Internet
• Performance isolation? CR CR
2. Performance isolation Data Center
typically via VLANs Layer 3
– Traffic of one service should be unaffected by others AR AR … AR AR
• Agility in terms of
3. Ease of management: “Plug-&-Play” (layer-2 dynamically adding or Layer 2 LB S S LB Key:
semantics) shrinking servers? • CR = L3 Core
Router
– Flat addressing, so any server can have any IP address • Agility in terms of S S S S … • AR = L3 Access
Router
– Server configuration is the same as in a LAN adapting to failures, and • S = L2 Switch
• LB = Load
to traffic dynamics?
– Legacy applications depending on broadcast must work … … Balancer
• A = Top of Rack
• Ease of management? switch

24
A Scalable, Commodity Data Center Fat-Tree Based DC Architecture
Network Architecture • Inter-connect racks (of servers) using a fat-tree topology
• Main Goal: addressing the limitations of • Fat-Tree: a special type of Clos Networks (after C. Clos)
K-ary fat tree: three-layer topology (edge, aggregation and core)
today’s data center network architecture – each pod consists of (k/2)2 servers & 2 layers of k/2 k-port switches
– sing point of failure – each edge switch connects to k/2 servers & k/2 aggr. switches
– each aggr. switch connects to k/2 edge & k/2 core switches
– over subscript of links higher up in the topology – (k/2)2 core switches: each connects to k pods
• trade-offs between cost and providing

• Key Design Considerations/Goals Fat-tree

– Allows host communication at line speed with K=4
• no matter where they are located!
– Backwards compatible with existing infrastructure
• no changes in application & support of layer 2 (Ethernet)
– Cost effective
• cheap infrastructure
• and low power consumption & heat emission

Fat-Tree Based Topology …

Cost of Maintaining Switches
• Why Fat-Tree?
– Fat tree has identical bandwidth at any bisections
– Each layer has the same aggregated bandwidth
• Can be built using cheap devices with uniform capacity
– Each port supports same speed as end host
– All devices can transmit at line speed if packets are distributed
uniform along available paths
• Great scalability: k-port switch supports k3/4 servers

Fat tree network with K = 6 supporting 54 hosts

 Fat-tree Topology is Great, But … Addressing Scheme
• Enforce a special (IP) addressing scheme in DC
Does using fat-tree topology to inter-connect racks
– Allocate IP addresses within the private block 10.0.0.0/8 block
of servers in itself sufficient?
– Switch address: 10.pod.switch.1
• What routing protocols should we run on these • pod [0,…,k-1] left to right, switch [0,…,k-1] left to right, bottom
switches? to top
• Layer 2 switch algorithm: data plane flooding! – Core switch address: 10.k.j.i, j,i [1,(k/2)]
– Host address: 10.pod.switch.ID, ID [2,(k/2)+1]
• Layer 3 IP routing:
– shortest path IP routing will typically use only one path despite
– Allows host attached to same switch to route only through
the path diversity in the topology switch
– if using equal-cost multi-path routing at each switch – Allows inter-pod traffic to stay within pod
independently and blindly, packet re-ordering may occur;
further load may not necessarily be well-balanced
– Aside: control plane flooding!

FAT-Tree Modified FAT-Tree Modified

• Use two level look-ups to distribute traffic • Use two level look-ups to distribute traffic
and maintain packet ordering and maintain packet ordering
• First level is prefix lookup • Core Switches: terminating 1st level prefixes for all network ID,
– used to route down the topology 10.2.2.1 pointing to the appropriate ID
to servers table • 1 link from each core to each pod
• Second level is a suffix lookup • /16 prefix (10.pod.0.0/16, port)
– used to route up towards core
– maintain packet ordering by using
same ports for same server