I.

Configuring SD WAN Load balancing for

Multiple Internet Links

A. Topology:

LAN1 10.10.1.0/24

LAN2 10.10.2.0/24
ISP1
LAN3 10.10.3.0/24

LAN4 10.10.4.0/24 WAN port1 192.168.0.108

WAN port2 14.140.40.108

LAN port5 10.10.10.108 ISP2

WAN port3 85.85.85.108

ISP3
WAN port4 95.95.95.108

ISP4

1. LAN Port5 10.10.10.108

2. LAN1 10.10.1.0/24, LAN2 10.10.2.0/24, LAN3 10.10.3.0/24, LAN4 10.10.4.0/24
3. WAN Port1 -> ISP1
4. WAN Port 2-> ISP 2
5. WAN Port1 -> ISP3
6. WAN Port 2-> ISP 4
7. WAN Port 1 Segment -> 192.168.0.108
8. WAN Port 2 Segment -> 14.140.40.108
9. WAN Port 3 Segment -> 85.85.85.108
10. WAN Port 4 Segment -> 95.95.95.108
 B. Scenarios:

We have 4 ISPs ISP1, ISP2, ISP3 and ISP4

1- All ISPs are active for all users

2- ISP1 main and isp2 first backup and isp3 second backup and isp4 third backup for LAN1
And ISP3 main and isp4 first backup and isp1 second backup and isp2 third backup for LAN2
3- ISP1 and isp2 are main active and isp3 first standby and isp4 second standby for LAN 3
And ISP3 and isp4 are main active and isp1 first standby and isp2 second standby for LAN 4

Below is the network setup on which we will configure FortiGate SD-WAN for the above scenarios.

1. Scenario 1: (All ISPs are active for all users)

All active load balanced round robin ISP1

LAN1 10.10.1.0/24
ISP2
LAN2 10.10.2.0/24

LAN3 10.10.3.0/24

LAN4 10.10.4.0/24

ISP3

ISP4
 a) 1. Enable SD-WAN feature in FortiGate

Go to Feature Visibility option and select SD-WAN Interface. You must enable this feature to
configure SD-WAN interfaces in the firewall.

• System ->Feature Visibility

• Select -> SD-WAN Interface

• Configure Interfaces as per above network diagram.

• Here, we have configured ISP1 (Port1)-> 192.168.0.108/24
• ISP2 (Port2) ->14.140.40.108/24
• ISP3 (Port3)-> 85.85.85.108/24
• ISP4 (Port4) ->95.95.95.108/24
• Configure LAN port on port 5 (for downstream Switch)

•
 b) 2. Create SD-WAN Zone

• Create SD-WAN Zone

• Named as SD-WAN-Zone

• SD-WAN->Select SD-WAN-ZONE
• Create New ->SD-WAN-Member
• Add ISP-1 Values
• Interface-> ISP1 (port1)
• SD-WAN-Zone-> SD-WAN-ZONE
• Gateway-> 192.168.0.1
• Cost-> 0
• Status-> Enable
• OK
In a similar way add ISP2 in SD-WAN-Zone member

• Interface->ISP2(port2)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 14.140.40.109
• Cost-> 0
• Status -> Enable
• OK

In a similar way add ISP3 in SD-WAN-Zone member

• Interface->ISP3(port3)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 85.85.85.109
• Cost-> 0
• Status -> Enable
• OK

In a similar way add ISP4 in SD-WAN-Zone member

• Interface->ISP4(port4)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 95.95.95.109
• Cost-> 0
• Status -> Enable
• OK
 c) 3. Configure Performance SLA

Next move to configure Performance SLAs Policy.

• Select -> SD-WAN

• Go to -> Performance SLAs

• Select-> Create New and add values in the tab

• Name-> SDWAN_SLA
• Detection Mode-> ACTIVE
• Protocol -> PING
• Server -> DNS Server/ Global DNS IP -> 8.8.8.8
• Enable SLA Target and put values in it
• Add values to Link Status
• Click OK

d) SLA Targets

• Latency Threshold -> maximum latency a link can manage to make decision
• Jitter Threshold ->Jitter for SLA to make the decisions
• Packet Loss Threshold->how much packet can loss when SD-WAN select SLA
Performance SLA shown in below diagram which contains values of both ISP1 and ISP2

1. Packet loss percentage of ISP1 and ISP2

2. Latency data of ISP1 and ISP2
3. Jitter values of ISP1 and ISP2
 e) 4. Configure SD-WAN Rules

• Go to SD-WAN ->SD-WAN Rules

We will not create any new rules and we will use the implicit rule

• Source-Address -> all

• Destination -> Allow for ALL
• Criteria: Source IP
• Protocol -> ANY

Implicit rule
SD-WAN rules define specific policy routing options to route traffic to an SD-WAN member.
When no explicit SD-WAN rules are defined, or if none of the rules are matched, then the
default implicit rule is used.

In an SD-WAN configuration, the default route usually points to the SD-WAN interface, so each
active member's gateway is added to the routing table's default route. FortiOS uses equal-cost
multipath (ECMP) to balance traffic between the interfaces. One of five load balancing
algorithms can be selected:

f) 5. Configure Static Routes

Now, it’s turn to configure static routes for the destination subnet. Here we have configured
static routes from all internal subnets by SD-WAN interface.

• Create New Static Route Rule

• Destination ->0.0.0.0/0 or All
• Interface -> SD-WAN
• Status -> Enable
 g) 6. Firewall Policy

• Create Firewall policy to the Internet to allow LAN-to-WAN traffic.

• Name-> Add Policy Name
• Incoming Interface -> LAN (Port-3)
• Outgoing Interface -> SD-WAN
• Source IP Address -> LAN Subnet (10.10.0.0/16)
• Destination -> ALL
• Service-> ALL
• Action-> Accept
• IP Pool Configuration -> Use Outgoing Interface Address
• OK
• Check Traffic stream from Firewall CLI.
• As per below logs traffic is going via ISP-1
2. Scenario 2:
(LAN1 → ISP1 main ISP2 first backup ISP3 second backup ISP4 third backup )
(LAN2→ ISP3 main ISP4 first backup ISP1 second backup ISP2 third backup )

Main

Backup 1 ISP1

Backup 2

Backup 3

LAN1 10.10.1.0/24
ISP2

ISP3

ISP4

Main

Backup 1 ISP1

Backup 2
Backup 3

LAN2 10.10.2.0/24
ISP2

ISP3

ISP4
 a) 1. Enable SD-WAN feature in FortiGate

Go to Feature Visibility option and select SD-WAN Interface. You must enable this feature to
configure SD-WAN interfaces in the firewall.

• System ->Feature Visibility

• Select -> SD-WAN Interface

• Configure Interfaces as per above network diagram.

• Here, we have configured ISP1 (Port1)-> 192.168.0.108/24
• ISP2 (Port2) ->14.140.40.108/24
• ISP3 (Port3)-> 85.85.85.108/24
• ISP4 (Port4) ->95.95.95.108/24
• Configure LAN port on port 5 (for downstream Switch)

•
 b) 2. Create SD-WAN Zone

• Create SD-WAN Zone

• Named as SD-WAN-Zone

• SD-WAN->Select SD-WAN-ZONE
• Create New ->SD-WAN-Member
• Add ISP-1 Values
• Interface-> ISP1 (port1)
• SD-WAN-Zone-> SD-WAN-ZONE
• Gateway-> 192.168.0.1
• Cost-> 0
• Status-> Enable
• OK
In a similar way add ISP2 in SD-WAN-Zone member

• Interface->ISP2(port2)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 14.140.40.109
• Cost-> 0
• Status -> Enable
• OK

In a similar way add ISP3 in SD-WAN-Zone member

• Interface->ISP3(port3)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 85.85.85.109
• Cost-> 0
• Status -> Enable
• OK

In a similar way add ISP4 in SD-WAN-Zone member

• Interface->ISP4(port4)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 95.95.95.109
• Cost-> 0
• Status -> Enable
• OK
 c) 3. Configure Performance SLA

Next move to configure Performance SLAs Policy.

• Select -> SD-WAN

• Go to -> Performance SLAs

• Select-> Create New and add values in the tab

• Name-> SDWAN_SLA
• Detection Mode-> ACTIVE
• Protocol -> PING
• Server -> DNS Server/ Global DNS IP -> 8.8.8.8
• Enable SLA Target and put values in it
• Add values to Link Status
• Click OK

d) SLA Targets

• Latency Threshold -> maximum latency a link can manage to make decision
• Jitter Threshold ->Jitter for SLA to make the decisions
• Packet Loss Threshold->how much packet can loss when SD-WAN select SLA
Performance SLA shown in below diagram which contains values of both ISP1 and ISP2

4. Packet loss percentage of ISP1 and ISP2

5. Latency data of ISP1 and ISP2
6. Jitter values of ISP1 and ISP2
 e) 4. Configure SD-WAN Rules

• Go to SD-WAN ->SD-WAN Rules

We will create two rules,

1- Rule 1 for LAN1 with order (ISP1, ISP2, ISP3, ISP3)

2- Rule 2 for LAN2 with order (ISP3, ISP4, ISP1, ISP2)

Rule for LAN1:

• Source-Address -> LAN1

• Destination -> Allow for ALL
• Protocol -> TCP/UDP or ANY
• Select strategy for how outgoing interfaces will be chosen

Manual: We can manually send traffic to any specific interface and provide preference to that
particular WAN interface. However only one WAN interface can take part in Performance SLA
and another WAN interface (example -WAN2) act as a backup link.
ISP1 main ISP2 first backup ISP3 second backup ISP4 third backup

• Interface Preferences -> Select this order ISP1, ISP2, ISP3 and ISP4 as the order here is
very important
• Status -> Enable
• OK

Rule for LAN2:

• Source-Address -> LAN2

• Destination -> Allow for ALL
• Protocol -> TCP/UDP or ANY
• Select strategy for how outgoing interfaces will be chosen

Manual: We can manually send traffic to any specific interface and provide preference to that
particular WAN interface. However only one WAN interface can take part in Performance SLA
and another WAN interface (example -WAN2) act as a backup link.

ISP3 main ISP4 first backup ISP1 second backup ISP2 third backup

• Interface Preferences -> Select this order ISP3, ISP4, ISP1 and ISP2 as the order here is
very important
• Status -> Enable
• OK
 f) 5. Configure Static Routes

Now, it’s turn to configure static routes for the destination subnet. Here we have configured
static routes from all internal subnets by SD-WAN interface.

• Create New Static Route Rule

• Destination ->0.0.0.0/0 or All
• Interface -> SD-WAN
• Status -> Enable
 g) 6. Firewall Policy

• Create Firewall policy to the Internet to allow LAN-to-WAN traffic.

• Name-> Add Policy Name
• Incoming Interface -> LAN (Port-3)
• Outgoing Interface -> SD-WAN
• Source IP Address -> LAN Subnet (10.10.0.0/16)
• Destination -> ALL
• Service-> ALL
• Action-> Accept
• IP Pool Configuration -> Use Outgoing Interface Address
• OK
• Check Traffic stream from Firewall CLI.
• As per below logs traffic is going via ISP-1
3. Scenario 3:
(LAN3 → ISP1 main ISP2 main ISP3 first backup ISP4 second backup )
(LAN4→ ISP3 main ISP4 main ISP1 first backup ISP2 second backup )

Main

Main
ISP1

Backup 1
Backup 2

LAN3 10.10.3.0/24 ISP2

ISP3

ISP4

Main

Main
ISP1

Backup 1
Backup 2

LAN4 10.10.4.0/24 ISP2

ISP3

ISP4
 a) 1. Enable SD-WAN feature in FortiGate

Go to Feature Visibility option and select SD-WAN Interface. You must enable this feature to
configure SD-WAN interfaces in the firewall.

• System ->Feature Visibility

• Select -> SD-WAN Interface

• Configure Interfaces as per above network diagram.

• Here, we have configured ISP1 (Port1)-> 192.168.0.108/24
• ISP2 (Port2) ->14.140.40.108/24
• ISP3 (Port3)-> 85.85.85.108/24
• ISP4 (Port4) ->95.95.95.108/24
• Configure LAN port on port 5 (for downstream Switch)

•
 b) 2. Create SD-WAN Zone

• Create SD-WAN Zone

• Named as SD-WAN-Zone

• SD-WAN->Select SD-WAN-ZONE
• Create New ->SD-WAN-Member
• Add ISP-1 Values
• Interface-> ISP1 (port1)
• SD-WAN-Zone-> SD-WAN-ZONE
• Gateway-> 192.168.0.1
• Cost-> 0
• Status-> Enable
• OK
In a similar way add ISP2 in SD-WAN-Zone member

• Interface->ISP2(port2)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 14.140.40.109
• Cost-> 0
• Status -> Enable
• OK

In a similar way add ISP3 in SD-WAN-Zone member

• Interface->ISP3(port3)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 85.85.85.109
• Cost-> 0
• Status -> Enable
• OK

In a similar way add ISP4 in SD-WAN-Zone member

• Interface->ISP4(port4)
• SD-WAN-ZONE (Zone must be same in both member 1 and member 2)
• Gateway-> 95.95.95.109
• Cost-> 0
• Status -> Enable
• OK
 c) 3. Configure Performance SLA

Next move to configure Performance SLAs Policy.

• Select -> SD-WAN

• Go to -> Performance SLAs

• Select-> Create New and add values in the tab

• Name-> SDWAN_SLA
• Detection Mode-> ACTIVE
• Protocol -> PING
• Server -> DNS Server/ Global DNS IP -> 8.8.8.8
• Enable SLA Target and put values in it
• Add values to Link Status
• Click OK

d) SLA Targets

• Latency Threshold -> maximum latency a link can manage to make decision
• Jitter Threshold ->Jitter for SLA to make the decisions
• Packet Loss Threshold->how much packet can loss when SD-WAN select SLA
We will create another Performance SLA to be used in the SD-WAN rules called LoadBalance

• Name: LoadBalance
• Participants: All SD-WAN Memebers
• Server: 8.8.8.8
• Latency Threshold -> 100 MS
• Jitter Threshold ->100 MS
 e) 4. Configure SD-WAN Rules

• Go to SD-WAN ->SD-WAN Rules

We will create four rules, two for LAN1 and two for LAN2 and also we will create Performance
SLA LoadBalance

(LAN3 → ISP1 main ISP2 main ISP3 first backup ISP4 second backup )
(LAN4→ ISP3 main ISP4 main ISP1 first backup ISP2 second backup )

3- Rule 1 LAN3 with strategy: Maximise Bandwidth (SLA) and with members
(ISP1, ISP2)
4- Rule 2 LAN3 with strategy: Manual and with members order (ISP3, ISP4)
5- Rule 3 LAN4 with strategy: Maximise Bandwidth (SLA) and with members
(ISP3, ISP4)
6- Rule 4 LAN4 with strategy: Manual and with members order (ISP1, ISP2)
Rule 1 for LAN3:

• Source-Address -> LAN3

• Destination -> Allow for ALL
• Protocol -> TCP/UDP or ANY
• Select strategy for how outgoing interfaces will be chosen

Maximise Bandwidth (SLA): Traffic distributed among the available links however, load-
balancing and transfer of traffic takes place after matching Latency parameter of link. By default,
it uses the Round-Robin method.

(LAN3 → ISP1 main ISP2 main ISP3 first backup ISP4 second backup )
(LAN4→ ISP3 main ISP4 main ISP1 first backup ISP2 second backup )

• Interface Preferences -> Select this order ISP1, ISP2

• Required SLA Target: LoadBalance
• Status -> Enable
• OK
Rule 2 for LAN3:

• Source-Address -> LAN3

• Destination -> Allow for ALL
• Protocol -> TCP/UDP or ANY
• Select strategy for how outgoing interfaces will be chosen

Manual: We can manually send traffic to any specific interface and provide preference to that
particular WAN interface. However only one WAN interface can take part in Performance SLA
and another WAN interface (example -WAN2) act as a backup link.

(LAN3 → ISP1 main ISP2 main ISP3 first backup ISP4 second backup )
(LAN4→ ISP3 main ISP4 main ISP1 first backup ISP2 second backup )

• Interface Preferences -> Select this order ISP3, ISP4 as the order here is very important
• Status -> Enable
• OK

Rule 3 for LAN4:

 • Source-Address -> LAN4
• Destination -> Allow for ALL
• Protocol -> TCP/UDP or ANY
• Select strategy for how outgoing interfaces will be chosen

Maximise Bandwidth (SLA): Traffic distributed among the available links however, load-
balancing and transfer of traffic takes place after matching Latency parameter of link. By default,
it uses the Round-Robin method.

(LAN3 → ISP1 main ISP2 main ISP3 first backup ISP4 second backup )
(LAN4→ ISP3 main ISP4 main ISP1 first backup ISP2 second backup )

• Interface Preferences -> Select this order ISP3, ISP4

• Required SLA Target: LoadBalance
• Status -> Enable
• OK

Rule 4 for LAN4:

• Source-Address -> LAN4

• Destination -> Allow for ALL
 • Protocol -> TCP/UDP or ANY
• Select strategy for how outgoing interfaces will be chosen

Manual: We can manually send traffic to any specific interface and provide preference to that
particular WAN interface. However only one WAN interface can take part in Performance SLA
and another WAN interface (example -WAN2) act as a backup link.

(LAN3 → ISP1 main ISP2 main ISP3 first backup ISP4 second backup )
(LAN4→ ISP3 main ISP4 main ISP1 first backup ISP2 second backup )

• Interface Preferences -> Select this order ISP1, ISP2 as the order here is very important
• Status -> Enable
• OK
 f) 5. Configure Static Routes

Now, it’s turn to configure static routes for the destination subnet. Here we have configured
static routes from all internal subnets by SD-WAN interface.

• Create New Static Route Rule

• Destination ->0.0.0.0/0 or All
• Interface -> SD-WAN
• Status -> Enable

g) 6. Firewall Policy

• Create Firewall policy to the Internet to allow LAN-to-WAN traffic.

• Name-> Add Policy Name
• Incoming Interface -> LAN (Port-3)
• Outgoing Interface -> SD-WAN
• Source IP Address -> LAN Subnet (10.10.0.0/16)
• Destination -> ALL
• Service-> ALL
• Action-> Accept
• IP Pool Configuration -> Use Outgoing Interface Address
• OK

• Check Traffic stream from Firewall CLI.

• As per below logs traffic is going via ISP-1