Use cases and pitfalls

in MPLS/VPLS networks

MUM EU 2018 Berlin | Sebastian Inacker | © FMS Internetservice GmbH

 Contact

 Phone: +49 761 2926500

 Email: [email protected]

 Shop: https://www.mikrotik-shop.de

 MikroTik Mirror: http://www.mikrotik-software.de

 Twitter: https://twitter.com/fmsweb_de

 Website: http://www.fmsweb.de

 Wiki: http://wiki.fmsweb.de

 Presentations: http://wiki.fmsweb.de/wiki/MUM-Presentations

 Facebook: https://www.facebook.com/fmsinternetservice
 About me

 Sebastian Inacker
<[email protected]>
 FMS Internetservice GmbH,
Germany
 MikroTik Trainer TR0011
(May 2007)
 MTCNA, MTCRE, MTCTCE,
MTCUME, MTCWE,
MTCIPv6E, MTCINE
 MikroTik trainings and workshops

 Own training center and on site

(Austria, Germany, Greenland, Hungary,
Luxembourg, Malta, Netherlands,
Switzerland, Uganda)
 Inquiries: [email protected]
 Overview / big picture

“Implementing and running a MPLS/VPLS network is easy.

As long as it is running well.”

Topics:
 Typical use cases of our (ISP) customers
 Typical pitfalls
 Surprising pitfalls
 Real world examples
 Overview / big picture

“Implementing and running a MPLS/VPLS network is easy.

As long as it is running well.”

Not main topics

 Step-by-step guide for each setup (focus on pitfalls)
 Reason for MPLS/VPLS (You should know, why)
 Reasons for MPLS/VPLS

Ok, very short and incomplete…

Benefits of MPLS
 Routing more complex than MPLS
 Some future setups (L3 VPN, TE) require MPLS

Benefits of VPLS vs. EoIP

 VPLS: No fragmentation (if done right)
 EoIP: Big overhead (42 bytes) & might cause fragmentation
 Overview / big picture

Pitfalls:
 Incomplete (of course)
 Not limited to MPLS/VPLS

Needs for VPLS

 MPLS
 Routing (OSPF here)
 Physical infrastructure
 Warning / heads-up / caution

This presentation will include errors, mistakes

and wrong configuration attempts to show
resulting errors!

Examples are simplified.

Keep that in mind.

The beginning
 Existing setup

 Existing OSPF network

 One PPPoE server
 EoIP (L2 tunnel) for client connections
 Existing setup

OSPF
R4

R2 R3 R6

R1 R5

R7

PPPoE

Internet

Uplink router
 Existing setup

OSPF
R4

R2 R3 R6

EoIP tunnel

EoIP tunnel

R1 R5
EoIP tunnel

R7

PPPoE

Internet

Uplink router
 Requirements for MPLS

MPLS can be integrated without service disruption

Running MPLS on top of OSPF:

 Enable LDP (Label Distribution Protocol)
 Set
 LSR ID (Label Switching Router's ID)
 Transport Address
If left unset: Lowest IP of router will be used
 Create LDP interfaces
 1st possible issue: LSR ID not unique

R1

R2 NAT R3
Uplink: 192.168.2.2/30 Uplink: 192.168.3.2/30
LAN: 10.0.0.1/24 LAN: 10.0.0.1/24
LSR ID = 10.0.0.1 R3 LSR ID = 10.0.0.1
R2

10.0.0.0/24 10.0.0.0/24
 Unique IP for LDP

Unique IP for
 LDP (LSR ID and Transport Address)

Let‘s try 10.255.255.<Router>/32 on physical interface

 Unique IP for OSPF

Unique IP for
 OSPF (Router ID) – same issue as with LSR ID

Take care: Setting of Router ID

 Restart of OSPF
 Loss of routing table
Service affecting action!
 LDP interfaces

Set LDP interfaces

 Don‘t forget your backup path!
 Compare OSPF interfaces and LDP interfaces
Create VPLS tunnels
Check VPLS interface

VPLS interface not running!

 Check MPLS

Empty:
 MPLS Local Bindings
 MPLS Remote Bindings
 MPLS Forwarding Table
 Check routing

IP routes to 10.255.255.x are missing

Routing ok, VPLS ok
 Lesson learned

MPLS is based on routing

 Broken/incomplete routing, broken/incomplete MPLS
 Broken MPLS, broken VPLS

Debugging:
Consider dependencies!
Working traceroute
Let‘s break things
 Maintenance at backup link

OSPF
R4

R2 R3 R6

VPLS tunnel

VPLS tunnel

R1 R5
VPLS tunnel

R7

PPPoE
Maintenance at R4 (backup link). OSPF is going through R5.
Customers at R3 complain. Customers at R6, R7 are fine.
 Maintenance at backup link

OSPF
R4

R2 R3 R6

VPLS tunnel

VPLS tunnel

R1 R5
VPLS tunnel

R7

PPPoE
Maintenance at backup link

R3: No link on ether4

10.255.255.3/32 on ether4
 Maintenance at backup link

No MPLS Forwarding / IP Route

for 10.255.255.3/32
Loopback bridge

loopback bridge is a good idea

Loopback bridge:
Empty bridge with IP 10.255.255.x/32
 Failure at main link

OSPF
R4

failure
R2 R3 R6

VPLS tunnel

VPLS tunnel

R1 R5
VPLS tunnel

R7

PPPoE
 Failure at main link

Expected behaviour
 Routing through R4
 PPPoE customers at R3, R6, R7 online

Observed behaviour OSPF

 Routing through R4
R4

failure
R2 R3 R6

 PPPoE customers at R6, R7 offline VPLS tunnel

VPLS tunnel

R1 R5
VPLS tunnel

R7

PPPoE
 Failure at main link

Ping from R1 to R7 ok
 Wrong LDP interfaces at R3

 LDP:
 ether2
 ether3
 ether4
 OSPF
 ether3
 ether4
 ether5
Examine setup
 Monitor a PPPoE session

Bandwidth-test: PPPoE client to PPPoE server (download)

R4

R2 R3 R6

VPLS tunnel

VPLS tunnel

R1 VPLS tunnel R5
PPPoE-Tunnel

R7

PPPoE 203.0.113.1

PPPoE-client
 Monitor a PPPoE session

Bandwidth-test: PPPoE client to PPPoE server (download)

MTU PPPoE Client: 1492  Bandwidth-test with 1492
 Monitor a PPPoE session

On R1
 Interface to R2: 1697 p/s
 Interface to PPPoE: 846 p/s
R2

ether2

R1
ether3

PPPoE 203.0.113.1
 Fragmentation

Packet fragmentation?

Benefits of VPLS vs. EoIP

 VPLS: No fragmentation (if done right)
 Packet sizes

Original frame
 L3 Size = 1500
MTU = 1500
 Full Frame Size = 1514
ETH: 14 IP (20) + DATA (1480)

ETH: 14 PPPoE (8) + DATA (1492)

 Packet sizes

Insertion of 1500 bytes (MTU) packet into VPLS tunnel:

No fragmentation
MTU = 1500
ETH: 14 IP (20) + DATA (1480)

ETH: 14 PPPoE (8) + DATA (1492)

VPLS tunnel Original frame

ETH: 14 MPLS (4) VPLS (4) CW (4) ETH (14) PPPoE (8) + DATA (1492)

Full Frame MTU

MPLS-MTU = L2 MTU = 1526 = 4 + 4 + 4 + 14 + 8 + 1492
 Packet sizes

VPLS packet is fragmented because:

Resulting MPLS-MTU: 1526
Interface MPLS MTU: 1508 (default)

ETH: 14 MPLS (4) VPLS (4) CW (4) ETH (14) PPPoE (8) + DATA (1492)

MPLS-MTU = L2 MTU = 1526

 Increase interface MPLS MTU

If hardware capable: Increase interface MPLS MTU

 L2 MTU (see Maximum Transmission Unit on RouterBoards)
 RB433, RB450, RB493: ether1: 1526, ether2-last: 1522
 RB433GL, RB450G, RB493G: all interfaces: 1520
 …
 Switches, media converters, …
ether1 ether2
L2 MTU 1526 L2 MTU 1522
RB450
 MPLS MTU set to 1526

MPLS Interface MTU: 1526 

Corresponding packet counters
 PPPoE client
 Interface to backbone
 Interface to PPPoE server
 Why 1508?

1508 is enough for

 MPLS for packet forwarding (1 MPLS label)
MTU = 1500
ETH: 14 IP (20) + DATA (1480)

MTU = 1500
ETH (14) MPLS (4) IP (20) + DATA (1480)
MPLS-MTU = L2 MTU = 1504 = 4 + 1500
 Why 1508?

1508 is enough for

 MPLS for packet forwarding (1 MPLS label)
 Targeted LDP (2 MPLS labels) MTU = 1500
ETH: 14 IP (20) + DATA (1480)

Default 1526 MTU = 1500

 Too large (?) ETH (14) MPLS (4) IP (20) + DATA (1480)
MPLS-MTU = L2 MTU = 1504 = 4 + 1500

MTU = 1500
ETH (14) MPLS (4) MPLS (4) IP (20) + DATA (1480)
MPLS-MTU = L2 MTU = 1508 = 4 + 4 + 1500
Network improvements
 Current network

OSPF
R4

R2 R3 R6

VPLS tunnel

VPLS tunnel

R1 R5
VPLS tunnel

R7

PPPoE
 Redundancy

Redundancy:
 Type / coverage depends on
 setup
 needs
 customer / network
 No claim for completeness
 Examples

Redundancy can become complex. Complexity can result in

issues.
 Redundancy at main site

optical fiber
R1 R2

Switch

PPPoE #1 PPPoE #2

Switch ISP #1 Internet

ISP #2

Green frame: See presentation of Patrik Schaub

(Access all FMS Internetservice presentations: click)
 Redundancy at backbone

 Additional link / ip subnet between R1/R2 and R2/R3

 2nd link is backup – same as on R3 R4

 OSPF interfaces: R2 R3 R6

High(er) cost for backup link

VPLS Tunnel

 Don‘t forget to add LDP interface

R5
VPLS Tunnel
R1
VPLS Tunnel
R7

main site
 Redundancy at backbone

R4

R2 R3 R6

VPLS Tunnel
R5
VPLS Tunnel
R1
VPLS Tunnel
R7

main site
 Redundancy for R1

 Clone R1:
 R1-Main (10.255.255.11)
 R1-Backup (10.255.255.12)
 Main link connected to R1-Main
 Backup link connected to R1-Backup
 VPLS go to R1-Main (10.255.255.1) R4

R2 R3 R6

VPLS Tunnel R5
R1-Main
VPLS Tunnel
R1-Backup
VPLS Tunnel
R7

main site
 Redundancy for R1

R4

R2 R3 R6

VPLS Tunnel R5
R1-Main
R4
VPLS Tunnel
R1-Backup R2 R3 R6
VPLS Tunnel
R7

VPLS Tunnel
R5
VPLS Tunnel
R1
VPLS Tunnel
R7

main site
main site
 Redundancy for R1

Who is R1-Main / R1-Backup?

R2
Who is 10.255.255.1?

R1-Main
VPLS Tunnel
 No VRRP between Main / Backup
VPLS Tunnel
R1-Backup
VPLS Tunnel
on Interface to R2 (different L3
Switch
networks)

main site
 Redundancy for R1

Who is R1-Main / R1-Backup?

R2
Who is 10.255.255.1?
Switch

R1-Main VPLS Tunnel

 Same L2 for R1-Main, R1-Backup
VPLS Tunnel
R1-Backup
VPLS Tunnel
and R2
Switch
 VRRP on R2 side
 Backup path: Decission by RSTP
main site
 Redundancy for R1

Who is R1-Main / R1-Backup?

R2
Who is 10.255.255.1?
Switch

R1-Main VPLS Tunnel

 Same L2 for R1-Main, R1-Backup
VPLS Tunnel
R1-Backup
VPLS Tunnel
and R2
Switch
 VRRP on R2 side
 Backup path: Decission by RSTP
main site
 Failure on link to main site
 VRRP is fine
 Clients offline
 Redundancy for R1

Who is R1-Main / R1-Backup?

R2
Who is 10.255.255.1?

R1-Main
VPLS Tunnel
 R1-Main and R1-Backup:
VPLS Tunnel
R1-Backup
VPLS Tunnel
Connected to main site switch
Switch
 VRRP on this side
 Management VLAN?
main site
 Redundancy for R1

VRRP and MPLS on R1-Main

 IP 10.255.255.1/32 on VRRP interface
 LSR ID = Transport address 10.255.255.1
R2

 10.255.255.11/32 on loopback, for OSPF

VPLS Tunnel
R1-Main

VRRP and MPLS on R1-Backup R1-Backup VPLS Tunnel

 IP 10.255.255.1/32 on VRRP interface

VPLS Tunnel

Switch

 LSR ID = Transport address 10.255.255.1

main site

 10.255.255.12/32 on loopback, for OSPF

 Let’s break test things

Failure of R1-Main or failure of link to main site

Expected behaviour
R2

 10.255.255.1 on R1-Backup
 VPLS tunnels to R1-Backup  up VPLS Tunnel
R1-Main

 PPPoE clients reconnecting R1-Backup VPLS Tunnel

VPLS Tunnel

Switch

Observed behavour
main site

 Everything fine (stop testing!)

 Let’s break test things

Failure of link R1-Main to R2

Expected behaviour
R2

 10.255.255.1 on R1-Main
 R1-Main VPLS master VPLS Tunnel
R1-Main

 R2: No route to 10.255.255.1 (OSPF) R1-Backup VPLS Tunnel

 Clients offline
VPLS Tunnel

Switch

main site
 Let’s break test things

OSPF and LDP on crosslink

R2

Switch
Expected behaviour
R1-Main VPLS Tunnel
 10.255.255.1 on R1-Main
R1-Backup VPLS Tunnel

VPLS Tunnel
 R2: route to 10.255.255.1
Switch
 VPLS ok & clients online

main site

Observed behaviour
 Clients offline
 Let’s break test things

Tests from R2:

 Route via R1-Backup
 Ping to 10.255.255.1 ok
 Traceroute ok
 Let’s break test things

Tests from R7:

 Ping to 10.255.255.1 ok
 Traceroute… ?
 Let’s break test things

Routing between R1-Backup and R1-Main ok

MPLS/LDP broken on R1-Backup
 No forwarding Table

Routing is not enough for VPLS!

 Let’s break test things

Simple reason:
 LSR ID and Transport Address 10.255.255.1 is used on
R1-Backup and R1-Main(!)
 IP 10.255.255.1 is active only on R1-Main (VRRP master)
 Duplicate ID (and transport address): Good idea? (No.)
 Let’s fix things

(One possible) Solution:

 On VRRP Master:
Set LSR ID and Transport Address to 10.255.255.1
 On VRRP Backup:
Set LSR ID and Transport Address to router unique address
(available on loopback)

Result: Working MPLS between routers

(OSPF was useing unique address as Router ID.)
 Let’s fix things

/interface vrrp
add interface=ether3 name=vrrp-directed-to-pppoe \
on-backup="/mpls ldp set transport-address=10.255.255.11 lsr-id=10.255.255.11" \
on-master="/mpls ldp set transport-address=10.255.255.1 lsr-id=10.255.255.1" \
preemption-mode=no vrid=5

R1-Main: 10.255.255.11
R1-Backup: 10.255.255.12

Note: Change of LSR ID

Service affecting
Traffic improvement
 Use backup link

 Traffic from R7 to R1 through R4

R4

But:
 OSPF goes through R5 R2 R3 R6

 MPLS goes through R5

VPLS Tunnel R5

 VPLS goes through R5

R1-Main
VPLS Tunnel
R1-Backup
VPLS Tunnel
R7

main site
 Traffic engineering (TE) tunnel

 Enable TE support on all involved interfaces

For example on R3:

/mpls traffic-eng interface
add interface=ether3
add interface=ether4
add interface=ether5

(Compare with MPLS interfaces)

 Traffic engineering (TE) tunnel

Use TE tunnel.

Here:
 No need for OSPF adjustments / single OSPF area
 No need for bandwith reservation / definition
 No need for Constrained Shortest Path First (CSPF)
 Traffic engineering (TE) tunnel

 Configure primary and secondary tunnel path (R1, R7)

/mpls traffic-eng tunnel-path
add name=tunnel-path-via-r4 use-cspf=no hops=10.255.255.4:loose
add name=dynamic-path use-cspf=no

R4

R2 R3 R6

VPLS Tunnel R5
R1-Main
VPLS Tunnel
R1-Backup
VPLS Tunnel
R7

main site
 Traffic engineering (TE) tunnel

 Create TE Tunnel (R1, R7)

/interface traffic-eng add \

name=traffic-eng-to-r7 \
from-address=10.255.255.1 \
to-address=10.255.255.7 \
primary-path=tunnel-path-via-r4 \
secondary-paths=dynamic-path
 Result

10 Mbit/s 10 Mbit/s
R4

20 Mbit/s
R2 R3 10 Mbit/s R6

20 Mbit/s 10 Mbit/s

VPLS Tunnel 10 Mbit/s R5

R1-Main
VPLS Tunnel
R1-Backup
VPLS Tunnel
R7

10 Mbit/s to PPPoE client at R6 and R7

 Result

R4

20 Mbit/s
R2 R3 20 Mbit/s R6

20 Mbit/s 10 Mbit/s

VPLS Tunnel 20 Mbit/s R5

R1-Main
VPLS Tunnel
R1-Backup
VPLS Tunnel
R7

10 Mbit/s to PPPoE client at R6 and R7

Failure of R4: Traffic through R5 (same for R5)

OSPF issue
 OSPF setup (simplified)

R12
R01, R11 and R21 on same subnet
 Bridge on R01 10.30.2.0/27
R21

 Same horizon value R11

10.30.1.0/27
 R01 OSPF neigbors: R11, R21 10.30.1.0/27

R01
 OSPF setup (simplified)

R12
Expected behaviour on R21
 OSPF neighbour (only) R01 10.30.2.0/27
R21

 Route to 10.30.2.0/27 R11

10.30.1.0/27
10.30.1.0/27

Observed behaviour
R01

 As expected
 OSPF setup (simplified)

R12
Reboot R01. No config change.
10.30.2.0/27
R21

R11
Expected behaviour on R21
10.30.1.0/27
 OSPF neighbour (only) R01 10.30.1.0/27

 Route to 10.30.2.0/27 R01

Observed behaviour
 10.30.2.0/27 missing
 Debug R21

R12
Debug R21
 OSPF state to R01 full 10.30.2.0/27
R21

 10.30.2.0/27 missing R11

10.30.1.0/27
10.30.1.0/27

R01
 Debug R01

R12
Debug R01
 OSPF state to R11 & R21 full 10.30.2.0/27
R21

 10.30.2.0/27 missing R11

10.30.1.0/27
10.30.1.0/27

R01
 OSPF Designated Router

R12
OSPF with network type Broadcast
will elect Designated Router (DR). 10.30.2.0/27
R21

R11

10.30.1.0/27
Who is DR? R21 is DR! 10.30.1.0/27

R01

 R11 tries to update R21 - not allowed

 by bridge horizon
 or wireless default forward
 or bridge filter
 …
 Possibilities

R12
Possible solutions
10.30.2.0/27
R21

 Force R01 to be DR R11

10.30.1.0/27
 Use network type ptmp 10.30.1.0/27

R01
Thank you
 FMS Internetservice GmbH

Phone: +49 761 2926500

Web: www.fmsweb.de
Shop: www.mikrotik-shop.de
Email: [email protected]
Twitter: https://twitter.com/fmsweb_de

MUM 2018 Berlin | Sebastian Inacker | © FMS Internetservice GmbH