Fortinet SD-WAN Lab Setup (2023 Update)

by andrewtravisFebruary 15, 20236

Last year, I blogged about how I had built a SD-WAN lab for FortiGate and FortiManager demos. I wanted
to better illustrate Branch deployments and in 2023 I rebuilt the lab with more Branch FortiGates. I’ve
also upgraded my home lab from FortiOS 7.0 to 7.2, which introduced a plethora of new features for
FortiManager that I was excited to learn.

This blog post turned into a long one, so I broke it into sections to make it easier to jump around:

 Lab Overview

 Metadata Variables

 FortiManager

 System Templates

 IPsec VPN Tunnel Templates

 BGP Templates

 SD-WAN Templates

 CLI Templates

 Branch FortiGate Deployment

Before I dive into the details of my configuration, let’s start with an overview of how the lab looks today.
Lab Overview

SD-WAN Lab Logical Diagram

My lab uses VLANs for each of the four WAN networks: Hub WAN1 (red), Hub WAN2 (yellow), Branch
WAN1 (blue) and Branch WAN2 (green). Each Branch has two WAN uplinks (through different ISPs) and
three LAN downlinks. If it’s difficult to see adequate detail in the above diagram, right-click it and open in
new tab so that you can see it at full resolution. These WAN and LAN networks are configured as Port
Groups on VMware vSwitches:

Hub FortiGate VLANs

Branch FortiGate VLANs

WANem in the SD-WAN Lab Logical Diagram above is still my “Internet backbone”. If you’ve not played
with WANem before, it’s a WAN Emulation tool. It’s not been updated in quite some time, so it’s an oldie
but still a goodie. Rather than re-type how to deploy WANem, you can hop over to my old blog post for
that section and then come back to this post. The big change from my old blog post is that I use larger
subnets for Branch WAN1 and WAN2 now, since my smaller subnet masks before wouldn’t allow for as
many Branches as I wanted to deploy; the new IP addressing is as follows:

 eth0 = Network Adapter 1 = Infrastructure = 192.168.5.102/24

 eth1 = Network Adapter 2 = Hub-WAN1 = 100.100.100.1 /24

 eth2 = Network Adapter 3 = Hub-WAN2 = 200.200.200.1 /24

 eth3 = Network Adapter 4 = Branch-WAN1 = 10.231.224.1 /24

 eth4 = Network Adapter 4 = Branch-WAN2 = 10.231.225.1 /24

 eth5 = Network Adapter 5 = WANem-Uplink = 192.168.99.2 /24

My Branch WAN DHCP server was also modified to use these new subnets of 10.231.224.x and
10.231.225.x; you can reference the old blog post for the ISC DHCP configuration and modify the subnets
to match the new ones above (if you’re following this setup identically).

And finally in the initial setup, you’ll deploy your Hub FortiGates the same way we did in the old blog
post. No big changes to the Hubs as far as networking in the lab redesign this year.

Now that our underlying framework is up and running, we can pivot to FortiManager to configure the
Branches!

Metadata Variables

Branch FortiGate VM Interfaces

For our templates in FortiManager to make sense, I wanted to illustrate how I configured the networking
for my Branch FortiGate VMs. The above diagram shows the port layout and which port maps to which
network.

To make our design and templates scalable, we are utilizing Metadata Variables in FortiManager. A
Metadata Variable is a variable that can have a different value for different devices or platforms and then
be referenced in templates. It’s similar to Meta Fields (if you’re familiar with them, they’re under FMG >
System Settings > Advanced > Meta Fields), but NOW in FOS 7.2 you need to use Metadata Variables
under FMG > Policy & Objects > Advanced) in all your templates. For those of you migrating from FMG
7.0 to 7.2, you’ll need to manually recreate these Metadata Variables. If you’re on a fresh install of FMG
7.2, just start here under Metadata Variables and forget Meta Fields even exist. In our lab, we have the
following Meta Variables:

Metadata Variables Overview

And when you edit a Metadata Variable, you can assign a per-device value to be used in your
Provisioning Templates:
Metadata Variables Per-Device Mapping

For my lab, here’s an example of the values given to a Branch device’s Metadata Variables:

 branch_guest = 192.168.254.0/24

 branch_guest_gw = 192.168.254.1/24

 branch_id = 1

 branch_lan = 172.16.200.0/24

 branch_lan_gw = 172.16.200.1/24

 branch_onboard = 192.168.254.0/24

 branch_onboard_gw = 192.168.254.1/24

 branch_ot = 172.16.100.0/24

 branch_ot_gw = 172.16.100.1/24

 hostname = Branch-1

 mgmt_ip = 192.168.5.114
  loopback = 192.168.250.201

Now let’s go to FortiManager.

FortiManager

FortiManager 7.2 added a lot more configurability to templates than existed in FOS 7.0. When I built the
lab on FOS 7.0, I had to use CLI Templates to configure IPsec VPN and BGP because these sections in the
Provisioning Templates just didn’t have the level of detail I needed. Thankfully the Import feature in
FortiManager 7.2 allowed me to import my BGP and IPsec VPN config from my existing Branch FortiGate
and save that config as a Provisioning Template. Here is how my Provisioning Templates are broken
down:

1. System Templates (Branch-Template-HomeLab) – this has my SNMP and logging config

2. IPsec Tunnel Templates (Branch) – this is my Hub and Spoke (with ADVPN) IPsec VPN config

3. BGP Templates (Branch_BGP) – this is my BGP config to peer with the Hubs and support ADVPN

4. SD-WAN Templates (VM-Branch-SD_WAN) – this is my template for SD-WAN members, rules

and SLAs

5. CLI Templates (VM-Branch-CLI-Templates) – this is how I configure hostname, network

interfaces, etc.

System Templates

I use System Templates in my lab to configure SNMP monitoring and logging to FortiAnalyzer. You can
also configure other system settings such as DNS, NTP, FortiGuard overrides, etc.
System Templates

IPsec Tunnel Templates

Since my design uses redundant VPN tunnels (one over Branch WAN1 to Hub WAN1 and likewise for
WAN2), I have configured two VPN interfaces for my Branch IPsec VPN template:

IPsec Tunnel Template

The key on this is to create as many VPN tunnels as you plan to need in your architecture and ensure the
Outgoing Interface matches your WAN interfaces on your Branch FortiGates. I have two VPN tunnels in
my lab: one tunnel over WAN1 and another tunnel over WAN2. Since I had different model FortiGates at
branches with different WAN interfaces (i.e. some were “wan1” whereas some were “port1”), I had to
use a different IPsec Tunnel Template for different FortiGate device types.

When you Create or Edit the VPN1 tunnel, you’ll use settings similar to these:
VPN1 Tunnel Configuration

Our design uses 100.100.100.2 for Hub WAN1, 200.200.200.2 for Hub WAN2, 169.254.100.0/24 for VPN1
tunnel addressing and 169.254.200.0/24 for VPN2 tunnel addressing. Further down in the config (not
shown in the screenshot), you’ll configure your Phase 2 encryption settings. Repeating the above for
VPN2, you can see the pattern:
VPN2 Tunnel Configuration

And for a sanity check, this is what the config looks like on the Branch FortiGate once applied later on:

config vpn ipsec phase1-interface

edit “VPN1”
set interface “port1”
set ike-version 2
set peertype any
set net-device enable
set mode-cfg enable
set proposal aes256-sha256
set add-route disable
set localid “Branch-1”
set comments “VPN: VPN1 [Created by IPSEC Template]”
set idle-timeout enable
set auto-discovery-receiver enable
set auto-discovery-shortcuts dependent
set network-overlay enable
set network-id 1
set remote-gw 100.100.100.2
set psksecret <preshared key>
set dpd-retrycount 2
set dpd-retryinterval 2
next
edit “VPN2”
set interface “port2”
set ike-version 2
set peertype any
set net-device enable
set mode-cfg enable
set proposal aes256-sha256
set add-route disable
set localid “Branch-1”
set comments “VPN: VPN2 [Created by IPSEC Template]”
set idle-timeout enable
set auto-discovery-receiver enable
set auto-discovery-shortcuts dependent
set network-overlay enable
set network-id 2
set remote-gw 200.200.200.2
set psksecret <preshared key>
set dpd-retrycount 2
set dpd-retryinterval 2
next
end
config vpn ipsec phase2-interface
edit “VPN1”
set phase1name “VPN1”
set proposal aes256-sha256
set auto-negotiate enable
set comments “VPN: VPN1 [Created by IPSEC Template]”
next
edit “VPN2”
set phase1name “VPN2”
set proposal aes256-sha256
set auto-negotiate enable
set comments “VPN: VPN2 [Created by IPSEC Template]”
next
end

BGP Templates

We use iBGP across the VPN tunnels for route sharing and the ability to build spoke to spoke VPN tunnel
shortcuts (using Auto-Discovery VPN or ADVPN). Here is how we configure our BGP template for our
Branches:
Our Hub cluster is 169.254.100.253 (via VPN1) and 169.254.200.253 (via VPN2) and we’re using the
loopback IP address Metadata Variable for the Router ID. In our next screenshot we can see how we’re
using the various network Metadata Variables to define our advertised networks:

Metadata Variables are that secret sauce that allow us to templatize our BGP config, yet still keep it
unique for each target device we’ll install it on. There are more BGP config settings that are important for
graceful restart, multipath, etc. and I’ve captured them in the below config (also as a sanity check if your
config looks different once installed on a device):
config router bgp
set as 65000
set router-id 192.168.250.201
set ibgp-multipath enable
set additional-path enable
set graceful-restart enable
set additional-path-select 4
config neighbor
edit “169.254.100.253”
set advertisement-interval 1
set capability-graceful-restart enable
set link-down-failover enable
set soft-reconfiguration enable
set description “VPN1”
set interface “VPN1”
set remote-as 65000
set route-map-out “Out-of-SLA”
set route-map-out-preferable “VPN1”
set connect-timer 1
set additional-path receive
next
edit “169.254.200.253”
set advertisement-interval 1
set capability-graceful-restart enable
set link-down-failover enable
set soft-reconfiguration enable
set description “VPN2”
set interface “VPN2”
set remote-as 65000
set route-map-out “Out-of-SLA”
set route-map-out-preferable “VPN2”
set connect-timer 1
set additional-path receive
next
end
config network
edit 1
set prefix 192.168.250.201 255.255.255.255
next
edit 2
set prefix 172.16.200.0 255.255.255.0
next
edit 3
set prefix 172.16.100.0 255.255.255.0
next
end

You may be looking at that config and wondering “what’s up with the route tags?”. Great question! But
it’s too deep a rabbit hole to go down here and I promise a future blog post to explain the structure. But
in a simple explanation, we map Route Tag 1 to VPN1 and we use them in our SD-WAN Rules if a
member is out of SLA or not. More to come on this one though… (and as promised, here you
go: https://www.andrewtravis.com/2023/02/23/fortigate-bgp-and-sd-wan/)

SD-WAN Templates

Next up is an SD-WAN Template to apply to our Branches. The beauty of this method is that if something
changes (i.e. you change the IP address that one of your SD-WAN SLAs is pinging), you just change it here
once and push it to all your devices.

Step 1 – Zones and Member Interfaces

For starters, you need to define your SD-WAN Zones. Think of them as a logical grouping of interfaces
that can then be referenced in Firewall Security Policies. In my environment, I place both VPN interfaces
in a SD-WAN Zone, but leave my WAN interfaces in separate Zones since I want them to operate
independently.

And don’t let the naming trip you up. We say Underlay for an Internet-connected interface and Overlay
for VPN tunnel interfaces (that ride over those Underlay interfaces).

Step 2 – SD-WAN Performance SLAs

I have the following Performance SLAs that monitor some resource and define how many times the
criteria fails before that Performance SLA is out of the desired service level agreement.
My Performance SLAs are pretty simple — they just ping different resources, though we have the option
for other protocols. Here’s more detail on the Direct Internet Access (DIA) Performance SLA we use to
ensure our redundant Internet uplinks are healthy and choose our preferred one:

It pings the resource (i.e. 8.8.8.8) over the participating members (i.e. port1, port2) and if it has latency >
150ms or jitter > 50ms or packet loss > 5%, then it’s out of SLA. We’ll use these Performance SLAs in our
SD-WAN Rules where we make traffic steering decisions based upon the Performance SLA being in SLA or
out of SLA.

Step 3 – SD-WAN Rules

Now’s the time to use the previous two steps and steer some traffic! Usually you’ll define your SD-WAN
rules around business applications to ensure they go the route you desire and use a backup path if the
primary path falls out of SLA. In my example, my lab’s business critical applications are SAP and Microsoft
365/Teams/OneDrive so I created rules for each of them:

Think of a SD-WAN rule like you do a Firewall Rule: they are matched in order and the first match applies
to the traffic. We’re able to tap into the FortiGate’s vast database of applications to select our desired
application to steer. Let’s drill into our “MS_Steering_Rule” for Microsoft 365 traffic:

First section is defining our source to match upon — it could be an IP address, subnet, user, group, etc.
We use the source subnet for our LAN subnet at the Branch:
The next section is defining our destination to match upon — it could be an Internet service, application,
IP address, QoS value, etc. We’re matching upon the Microsoft 365 Internet services:
The last section is defining what interfaces we want to participate in this rule, which Performance SLA to
measure health against and what strategy to use to route the traffic. In ours, we’re using both the
Underlay interfaces connected to the Internet and it will prefer port2 as long as it meets our SLA
(because we’ve selected port2 higher in the order than port1):
Now let’s check the same thing for our Direct Internet Access (DIA) Rule… This one matches
on any destination, so it’s our last rule of the list, otherwise it would match our Microsoft traffic. This
rule uses port1 for any traffic matching it.
In our design, we want Microsoft 365 traffic to use port2 (WAN2, ISP2) and all other Internet-bound
traffic to use port1 (WAN1, ISP1). We’re ensuring our business critical traffic gets the “better” ISP and all
our other traffic uses the other ISP. Now we don’t have to worry about generic Internet-bound traffic
squashing our business critical traffic. And if the SLAs fail on either SD-WAN Rule, it’ll choose the
surviving Internet uplink seamlessly. It’s a thing of beauty!

CLI Templates

CLI Templates are amazing to do all the things you want to that aren’t in the FortiManager GUI (yet). I
use these templates to configure interfaces, set the FortiGate’s hostname and set many other settings.

“General Settings” sets the hostname and defines our local admin password (otherwise it’s blank until
you first login manually):

config system global

set hostname $(hostname)
end
config system admin
edit “admin”
set password <super secret local admin password>
next
end

Interface Setup is the quick and dirty way to create our interfaces so much faster than manually in the
FortiManager GUI, per device. So much faster… Here’s the CLI config I use for my lab:

config system interface

edit “port1”
set vdom “root”
set mode dhcp
set allowaccess ping
set type physical
set alias “WAN1”
set role wan
next
edit “port2”
set vdom “root”
set mode dhcp
set allowaccess ping
set type physical
set alias “WAN2”
set role wan
next
edit “port3”
set vdom “root”
set ip $(mgmt_ip) 255.255.255.0
set allowaccess ping https ssh snmp http fgfm
set type physical
set alias “MGMT”
next
edit “port4”
set vdom “root”
set alias “Branch-LAN”
set dhcp-relay-service enable
set ip $(branch_lan_gw)
set allowaccess ping
set device-identification enable
set dhcp-relay-ip “10.0.0.20”
next
edit “port5”
set vdom “root”
set alias “Branch-OT”
set dhcp-relay-service enable
set ip $(branch_ot_gw)
set allowaccess ping
set device-identification enable
set dhcp-relay-ip “10.0.0.20”
next
edit “port6”
set vdom “root”
set alias “Branch-Guest”
set dhcp-relay-service enable
set ip $(branch_guest_gw)
set allowaccess ping
set device-identification enable
set dhcp-relay-ip “10.0.0.20”
next
edit VPN1
set allowaccess ping
next
edit VPN2
set allowaccess ping
next
edit “lo1”
set vdom “root”
set ip $(loopback) 255.255.255.255
set allowaccess ping
set type loopback
next
end

For this last one I probably could’ve created a Static Route Template in the FortiManager GUI, but I
already had this config handy:

config router static

edit 10
set device “port1”
set dynamic-gateway enable
next
edit 11
set device “port2”
set dynamic-gateway enable
next
end

That wraps up all of our templates. Whew, what a long read/write up! But now we’re ready to deploy
our Branches!

Branch FortiGate Deployment

In my lab, I mostly use FortiGate VMs as my Branches based upon availability, rack space, etc. so this
walkthrough will begin with how to deploy the FortiGate VM in ESXi. Shortly after, we’ll dive right into
configuring it in FortiManager.

Step 1 – Deploy FortiGate VM in ESXi

Deploy your FortiGate VM OVF without powering it on and map the interfaces similar to those I have
above. I deleted NICs 7 and up since I didn’t use them, plus it makes the FortiGate interfaces look
cleaner.

Step 2 – Power Up the VM, Address Management Interface

Now power your FortiGate VM up and give the management interface (port3 for my lab) an IP address
you can reach across your network.

Step 3 – Access GUI and Upload License

Once you login as admin with no password, you’ll give the FortiGate any password (since we’ll reset it in
a few steps). The only option you have now is an eval license or to upload your license downloaded from
FortiCare.
The FortiGate will reboot and once it does, go ahead and factory reset it so that we can configure it from
FortiManager. The trick is running the “exec factoryreset keepvmlicense” command. Without completing
this licensing step, my installation from FortiManager failed because I didn’t have entitlement to push all
my config (and create all the additional interfaces) without a valid license.

Step 4 – Authorize in FortiManager

When we deployed our WAN DHCP server, we specified DHCP option 240 with the IP address of
FortiManager. And just like magic, our FortiGate VM registered itself to FortiManager so that we can
configure it! Option 240 is the workaround to using Forti Deploy and works great in a lab environment. I
have a separate Administrative Domain (ADOM) in FortiManager for my SD-WAN Lab and my FortiGate
VM first shows up under my root ADOM. From here, I select the device, click Authorize and specify the
following:

1. ADOM = <destination ADOM> (if not root)

2. New Device Name = Branch-X

3. Policy Package = Branch (this package’s creation was outside the scope of this post, it’s already
long enough!)

4. Provisioning Template = Branch_VM_Templates

Step 5 – Set Metadata Variables

We defined what Metadata Variables are earlier and now we’re ready to assign them values. Go into the
ADOM with the device > Device Manager > Right-click the device > Edit Variable Mapping.
And give them values similar to these:
Step 6 – Install Device Settings
You’re now ready to install the device’s settings from Forti Manager. This will basically set everything on
the device besides the Policy Package. It will create interfaces, build VPN tunnels, setup BGP adjacencies,
define SD-WAN rules, etc. You’ll click Install Wizard on the top blue bar in Forti Manager and step
through the wizard to install the updates to the Branch FortiGate. I will warn you that unless you’re
perfect and don’t make mistakes (I’m not that person, trust me), you definitely want to preview the
changes before installing them. When you preview, it will look like this plus another 100+ lines of config:
After finishing the install wizard, your Branch FortiGate is almost finished. We just need to map our
interfaces to Normalized Interfaces so that our Policy Packages will install correctly.

Step 7 – Map Interfaces and Global Address Objects

Drill into your device in Forti Manager > Device Manager > System > Interfaces and edit each interface to
map the device’s interfaces to the Normalized Interfaces used in your Policy Packages:

And likewise, if you’re using the Global Database in FortiManager and giving objects unique values for
each device, then you will want to edit each object and give it a per-device value:
Step 8 – Install Branch Policy Package

This step assumes you already have a Policy Package with firewall rules, security profiles, etc. And it
assumes that your Policy Package uses the Normalized Interfaces you mapped to in the previous step. I
guess I should also say it assumes you’ve assigned your Policy Package to your target device(s) too:
In FortiManager, go to Policy & Objects and click Install on the top blue bar. As before, definitely preview
the install before going through with it in case you need to tweak anything.
Viola! Your Branch FortiGate is deployed and you’re ready to rinse and repeat on the remaining
Branches.

Conclusion

This was a long post! But it wrapped up a solid week of reconfiguring my lab, learning the templates in
FortiManager 7.2 and realizing where I needed to adapt my FortiManager config to make it all reusable
across many Branches. I hope you enjoyed reading and please comment with anything you’d like me to
deeper dive into. I’ll write future posts on how we use Route Tags and SD-WAN as well as testing all this
with WANem. Thanks! Andrew