FortiGate Infrastructure

Diagnostics

FortiOS 7.0
© Copyright Fortinet Inc. All rights reserved. LastLast
Modified:
Modified:
Saturday,
Saturday,
August
August
19, 19,
20232023
Lesson Overview

General Diagnosis

Debug Flow

CPU and Memory

Firmware and Hardware

© Fortinet Inc. All Rights Reserved. 2
General Diagnosis

Objectives
• Identify your network’s normal behavior
• Monitor for abnormal behavior, such as traffic spikes
• Diagnose problems at the physical and network layers

3
Before a Problem Occurs
• Know what normal is (baseline): Abnormal
• CPU usage
• Memory usage
• Traffic volume
• Traffic directions
• Protocols and port numbers
• Traffic pattern and distribution
• Why?
• Abnormal behavior is difficult to identify,
unless you know, relatively, what normal is

Now
Baseline (Average)
Normal Range

© Fortinet Inc. All Rights Reserved. 4

Network Diagrams
• Why?
• Explaining or analyzing complex networks is port2
192.168.5.1/30
difficult and time-consuming without them
port1
• Physical diagrams: 10.0.30.1/24

port2
• Include cables, ports, and physical network 192.168.3.1/30
devices
• Show relationships at Layer 1 and Layer 2 port1
192.168.1.2/30

• Logical diagrams:
wan1
• Include subnets, routers, logical devices 192.168.1.1/30
port1 port2
• Show relationships at Layer 3 10.0.10.1/24 192.168.2.1/24
Web server
192.168.2.2/24

port4

vlan 100 vlan 110

10.0.20.1/24 10.0.50.1/24

© Fortinet Inc. All Rights Reserved. 5

Monitoring Traffic Flows and Resource Usage
• Get normal data before problems or complaints

• Tools:
• Security Fabric
• Dashboard
• SNMP
• Alert email
• Logging/Syslog/FortiAnalyzer
• CLI debug commands

Traffic spikes

© Fortinet Inc. All Rights Reserved. 6

System Information FortiGate # get system status
Version: FortiGate-VM64 v7.0.0,build0066,210330 (GA)
Virus-DB: 84.00735(2021-03-15 18:07)
Extended DB: 84.00735(2021-03-15 18:07)
Extreme DB: 1.00000(2018-04-09 18:07)
IPS-DB: 6.00741(2015-12-01 02:30)
IPS-ETDB: 15.00796(2020-03-14 03:19)
APP-DB: 15.00796(2020-03-14 03:19)
INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)
Serial-Number: FGVM010000064692
IPS Malicious URL Database: 2.00584(2020-03-16 04:32)
License Status: Valid
VM Resources: 1 CPU/1 allowed, 2010 MB RAM
Log hard disk: Available
Hostname: Local-FortiGate
Private Encryption: Disable
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: standalone
Branch point: 0066
Release Version Information: GA
FortiOS x86-64: Yes
System time: Tue Apr 6 02:34:53 2021
Last reboot reason: warm reboot
© Fortinet Inc. All Rights Reserved. 7
System Information (Contd)
FortiGate # get hardware nic <interface_name>
...
Name: port1 Rx errors: 0
Driver: virtio_net
Rx Length err: 0
Version: 1.0.0
Bus: 0000:00:03.0 Rx Buf overflow: 0
Hwaddr: 02:09:0f:00:02:01 Rx Crc err: 0
Permanent Hwaddr:02:09:0f:00:02:01 Rx Frame err: 0
State: up Rx Fifo overrun: 0
Link: up Rx Missed packets: 0
Mtu: 1500 Tx packets: 57752
Supported: 1000full 10000full Tx bytes: 4993066
Advertised: Tx compressed: 0
Speed: 10000full
Auto: disabled Tx dropped: 0
RX Ring: 256 Tx errors: 0
TX Ring: 256 Tx Aborted err: 0
Rx packets: 670785 Tx Carrier err: 0
Rx bytes: 949908714 Tx Fifo overrun: 0
Rx compressed: 0 Tx Heartbeat err: 0
Rx dropped: 0 Tx Window err: 0
... Multicasts: 0
Collisions: 0

© Fortinet Inc. All Rights Reserved. 8

ARP Table
# get system arp
Address Age(min) Hardware Addr Interface
10.0.1.10 0 00:0c:29:e0:c1:87 port3
10.200.1.254 0 00:0c:29:1c:28:d7 port1

FortiGate Interface
Connecting device IP address
and MAC address

© Fortinet Inc. All Rights Reserved. 9

Network Layer Troubleshooting
# execute ping-options
adaptive-ping Adaptive ping <enable|disable>.
data-size Integer value to specify datagram size in bytes.
df-bit Set DF bit in IP header <yes | no>.
interface Auto | <outgoing interface>.
interval Integer value to specify seconds between two pings.
pattern Hex format of pattern, e.g. 00ffaabb.
repeat-count Integer value to specify how many times to repeat PING.
...

# execute ping <ip> IP address or domain name

# execute traceroute <dest> IP address or hostname

© Fortinet Inc. All Rights Reserved. 10

Knowledge Check
1. Which CLI command can be used to determine the MAC address of a FortiGate
default gateway?
A. get system arp
B. get hardware nic

2. Which CLI command can be used to diagnose a physical layer problem?

A. execute traceroute
B. get hardware nic

© Fortinet Inc. All Rights Reserved. 11

Lesson Progress

General Diagnosis

Debug Flow

CPU and Memory

Firmware and Hardware

© Fortinet Inc. All Rights Reserved. 12

Debug Flow

Objectives
• Diagnose connectivity problems using the debug flow

13
Debug Flow
• Shows what the CPU is doing, step-by-step, with the packets
• If a packet is dropped, it shows the reason

• Multi-step command
1. Define a filter: diagnose debug flow filter <filter>
2. Enable debug output: diagnose debug enable
3. Start the trace: diagnose debug flow trace start <xxx> Repeat number
4. Stop the trace: diagnose debug flow trace stop

© Fortinet Inc. All Rights Reserved. 14

Debug Flow Example—SYN IP addresses, port numbers,
id=2 line=4677 msg="vd-root received a packet(proto=6, and incoming interface
10.0.1.10:49886->66.171.121.44:80) from port3. flag [S], seq 2176715501, ack 0,
win 8192"

id=2 line=4831 msg="allocate a new session-00007fc0" Create a new session

id=2 line=2582 msg="find a route: flag=04000000 Found a matching route.

Shows next-hop IP address
gw-10.200.1.254 via port1" and outgoing interface.

id=2 line=699 msg="Allowed by Policy-1: SNAT" Matching firewall policy

id=2 line=2719 msg="SNAT 10.0.1.10->10.200.1.1:49886"

Source NAT

© Fortinet Inc. All Rights Reserved. 15

Debug Flow Example—SYN/ACK
id=2 line=4677 msg="vd-root received a packet(proto=6,
66.171.121.44:80->10.200.1.1:49886) from port1. flag [S.], IP addresses, port numbers,
and incoming interface
seq 3567496940, ack 2176715502, win 5840"

id=2 line=4739 msg="Find an existing session, Using an existing session

id-00007fc0,reply direction"

Destination NAT
id=2 line=2733 msg="DNAT 10.200.1.1:49886->10.0.1.10:49886"

id=2 line=2582 msg="find a route: flag=00000000 gw-10.0.1.10 via port3"

Found a matching route.

Shows next-hop IP address
and outgoing interface.

© Fortinet Inc. All Rights Reserved. 16

Knowledge Check
1. Which information is displayed in the output of a debug flow?
A. Incoming interface and matching firewall policy
B. Matching security profile and traffic log

2. When is a new TCP session allocated?

A. When a SYN packet is allowed
B. When a SYN/ACK packet is allowed

© Fortinet Inc. All Rights Reserved. 17

Lesson Progress

General Diagnosis

Debug Flow

CPU and Memory

Firmware and Hardware

© Fortinet Inc. All Rights Reserved. 18

CPU and Memory

Objectives
• Diagnose resource problems, such as high CPU or memory
usage
• Diagnose memory conserve mode
• Diagnose fail-open session mode

19
Slowness
• High CPU usage
• High memory usage

• What was the last feature you enabled?

• Enable one at a time

• How high is the CPU usage? Why?

• # get system performance status
• # diagnose sys top 1

© Fortinet Inc. All Rights Reserved. 20

CPU and Memory Usage
CPU states: 11% user 8% system 0% nice 67% idle 0% iowait 0% irq 14% softirq CPU usage
CPU0 states: 2% user 2% system 0% nice 96% idle 0% iowait 0% irq 0% softirq
CPU1 states: 4% user 22% system 0% nice 74% idle 0% iowait 0% irq 0% softirq
CPU2 states: 38% user 2% system 0% nice 2% idle 0% iowait 0% irq 58% softirq RAM usage
CPU3 states: 1% user 7% system 0% nice 92% idle 0% iowait 0% irq 0% softirq
Memory: 1911056k total, 858976k used (44.9%), 1019792k free (53.4%), 32288k freeable (1.7%)
Average network usage: 12813 / 3784 kbps in 1 minute, 6551 / 1385 kbps in 10 minutes, 1908 / 463 kbps in
30 minutes
Average sessions: 8 sessions in 1 minute, 7 sessions in 10 minutes, 4 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10
minutes, 0 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30
minutes
Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last
30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Network usage
Uptime: 0 days, 5 hours, 18 minutes

© Fortinet Inc. All Rights Reserved. 21

High CPU and Memory Troubleshooting
# diagnose sys top
Run Time: 0 days, 0 hours and 18 minutes
1U, 4N, 0S, 95I, 0WA, 0HI, 0SI, 0ST; 994T, 421F
pyfcgid 248 S 2.9 3.8
newcli 251 R 0.1 1.0
merged_daemons 185 S 0.1 0.7
miglogd 177 S 0.0 6.8
pyfcgid 249 S 0.0 3.0
pyfcgid 246 S 0.0 2.8
reportd 197 S 0.0 2.7
Process Memory
cmdbsvr 113 S 0.0 2.4
name usage (%)

Sort by CPU: Shift + P Process CPU usage

Sort by RAM: Shift + M Process ID
state (%)

© Fortinet Inc. All Rights Reserved. 22

Memory Conserve Mode
• FortiOS protects itself when memory usage is high
• It prevents using so much memory that FortiGate becomes unresponsive
• Three configurable thresholds:

Threshold Definition Default (% of total RAM)

Green Threshold at which FortiGate exits conserve mode 82%

Red Threshold at which FortiGate enters conserve mode 88%

Extreme Threshold at which new sessions are dropped 95%

config system global

set memory-use-threshold-red <percentage>
set memory-use-threshold-extreme <percentage>
set memory-use-threshold-green <percentage>
end

© Fortinet Inc. All Rights Reserved. 23

What Happens During Conserve Mode?
• System configuration cannot be changed

• FortiGate skips quarantine actions (including FortiSandbox analysis)

• For packets that require any inspection by the IPS engine:

config ips global
set fail-open [enable|disable]
end
• enable: Packets can still be transmitted without IPS scanning while in conserve mode
• disable: Packets are dropped for new incoming sessions, but try to make the existing sessions work the same
as non-conserve mode

© Fortinet Inc. All Rights Reserved. 24

What Happens During Conserve Mode? (Contd)
• For traffic that requires any proxy-based inspection (and if memory usage has not
exceeded the extreme threshold yet):
config system global
set av-failopen [off | pass | one-shot]

end
• off :All new sessions with content scanning enabled are not passed
• pass (default): All new sessions pass without inspection
• one-shot: Similar to pass in that traffic is not inspected. However, it will keep bypassing the AV proxy even
after leaving conserve mode. Administrators must either change this setting, or restart the unit, to restart the AV
scanning

• The av-failopen setting also applies to flow-based antivirus inspection

• If memory usage exceeds the extreme threshold, all new sessions that require
inspection (flow-based or proxy-based) are blocked

© Fortinet Inc. All Rights Reserved. 25

System Memory Conserve Mode Diagnostics
# diagnose hardware sysinfo conserve Off = no conserve mode
memory conserve mode: on on = conserve mode
total RAM: 3040 MB
memory used: 2706 MB 89% of total RAM
memory freeable: 334 MB 11% of total RAM
memory used + freeable threshold extreme: 2887 MB 95% of total RAM
memory used threshold red: 2675 MB 88% of total RAM
memory used threshold green: 2492 MB 82% of total RAM

© Fortinet Inc. All Rights Reserved. 26

Fail-Open Session Setting
• The following setting controls how FortiOS handles a session that is impacted by a UTM
scan error when doing http/mapi proxy or explicit webproxy

config system global

set av-failopen-session [enable | disable]

• enable = Sessions are allowed

• disable(default) = Block all new sessions that require proxy-based inspection

© Fortinet Inc. All Rights Reserved. 27

Knowledge Check
1. Which action does FortiGate take during memory conserve mode?
A. Configuration changes are not allowed.
B. Administrative access is denied.

2. Which threshold is used to determine when FortiGate enters conserve mode?

A. Green
B. Red

© Fortinet Inc. All Rights Reserved. 28

Lesson Progress

General Diagnosis

Debug Flow

CPU and Memory

Firmware and Hardware

© Fortinet Inc. All Rights Reserved. 29

Firmware and Hardware

Objectives
• Format the flash memory
• Load a firmware image from the BIOS menu
• Run hardware tests
• Display crash log information

30
Access to BIOS Menu
FGT60D (18:34-05.05.2021)
Ver:04000005 BIOS version. Options in the BIOS
Serial number:FG60DXXXXXXXX menu depend on the version.
RAM activation
Total RAM: 512MB
Enabling cache...Done.
Scanning PCI bus...Done.
Allocating PCI resources...Done.
Enabling PCI resources...Done. Press any key at this
Zeroing IRQ settings...Done. prompt to enter the BIOS
Verifying PIRQ tables...Done. menu.
Enabling Interrupts...Done.
Boot up, boot device capacity: 122MB.
Press any key to display configuration menu...
......

Reading boot image 1375833 bytes.

Initializing firewall...
System is started.

© Fortinet Inc. All Rights Reserved. 31

Format Flash Memory
[G]: Get firmware image from TFTP server.
[F]: Format boot device.
[B]: Boot with backup firmware and set as default.
[I]: Configuration and information. Recommended for a clean
[Q]: Quit menu and continue to boot with default firmware. installation and problems possibly
[H]: Display this list of options. related to corrupted firmware

Enter Selection [G]:

Enter G,F,B,I,Q,or H:

All data will be erased,continue:[Y/N]?

Formatting boot device...
..............................................................
Format boot device completed.

CAUTION: Formatting the flash memory deletes the firmware,

configuration, and digital certificates

© Fortinet Inc. All Rights Reserved. 32

Firmware Installation From Console

Make sure that a TFTP server application is installed on

your PC

Configure the TFTP server directory and copy the

FortiGate firmware [image.out]

Connect your PC NIC to the FortiGate TFTP install

interface

Select get firmware image from the BIOS menu

© Fortinet Inc. All Rights Reserved. 33

BIOS Firmware Transfer
Please connect TFTP server to Ethernet port "3".

Enter TFTP server address [192.168.1.168]: 192.168.1.110

CAUTION: Transferring a firmware
Enter local address [192.168.1.188]:
image deletes the configuration and
Enter firmware image file name [image.out]:
installs the factory default configuration
MAC:00090FC371BE
######################
Total 23299683 bytes data downloaded.
Verifying the integrity of the firmware image.

Total 40000kB unzipped.

Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]?d
Programming the boot device now.
.......................................
Reading boot image 1375833 bytes.
Initializing firewall...
System is started.
Formatting shared data partition ... done!

© Fortinet Inc. All Rights Reserved. 34

Hardware Tests
• Designed for both manufacturing testing and for end users to verify major hardware
components:
• CPU
• RAM memory
• Network interfaces
• Hard disk
• Flash memory
• USB interface
• Front panel LEDs
• Wi-Fi
• And so on

© Fortinet Inc. All Rights Reserved. 35

How to Run the Hardware Tests
• In some E,F, and D-series models, the hardware tests can be run directly from FortiOS
• Can run a single test, or multiple tests
• For other models, a special HQIP image must be loaded using TFTP and run from the
BIOS menu
• Instructions: https://support.fortinet.com/Download/HQIPImages.aspx

© Fortinet Inc. All Rights Reserved. 36

FortiOS Hardware Tests Command
# diagnose hardware test suite all

- Please connect ethernet cables:

[WAN - Any of PORT1...PORT4]
To skip this test, please press 'N'.
Do you want to continue? (y/n) (default is n) N
Following tests will request you to check the colours of the system LEDs.
To skip this test, please press 'N'.
Do you want to continue? (y/n) (default is n) N
Following tests will request you to check the colours of the NIC LEDs.
- Please connect ethernet cables:
[WAN - Any of PORT1...PORT4]
To skip this test, please press 'N'.
Do you want to continue? (y/n) (default is n) N
Test Begin at UTC Time Wed May 05 21:08:53 2021

.....
© Fortinet Inc. All Rights Reserved. 37
Crash Logs
• Inspect crash logs for debugging purposes
• Any time a process closes, it is recorded as killed
• Some are normal (for example, closing scanunit to update definitions)

# diagnose debug crashlog history

Crash log interval is 3600 seconds
lldptx crashed 1 times. The lastest crash was at 2021-04-02 06:40:15
fgfmsd crashed 1 times. The lastest crash was at 2021-04-02 06:50:31

# diagnose debug crashlog read

14379: 2021-04-02 06:40:15 <14640> firmware FortiGate-61F v7.0.0,build0066,210330 (GA) (Release)
14380: 2021-04-02 06:40:15 <14640> application lldptx
14381: 2021-04-02 06:40:15 <14640> *** signal 11 (Segmentation fault) received ***
14382: 2021-04-02 06:40:15 <14640> Register dump:
14383: 2021-04-02 06:40:15 <14640> R0: 0000000003b58e10 R1: 0000007fd4dd70dc R2: 0000007fd4dd7120
...

© Fortinet Inc. All Rights Reserved. 38

Conserve Mode Events in Crash Logs
• The crash log also records conserve mode events

• Entering:
12: 2021-04-06 14:10:16 logdesc="Kernel enters conserve mode" service=kernel
conserve=on free="127962
13: 2021-04-06 14:10:16 pages" red="128000 pages" msg="Kernel enters conserve
mode"

• Exiting:
14: 2021-04-06 14:19:55 logdesc="Kernel leaves conserve mode" service=kernel
conserve=exit
15: 2021-04-06 14:19:55 free="192987 pages" green="192000 pages" msg="Kernel
leaves conserve mode"

© Fortinet Inc. All Rights Reserved. 39

Knowledge Check
1. Which types of information are stored in the crash log?
A. Process crashes and conserve mode events
B. Traffic logs and security logs

2. Which protocol is used to upload new firmware from the console?

A. HTTP/HTTPS
B. TFTP

© Fortinet Inc. All Rights Reserved. 40

Lesson Progress

General Diagnosis

Debug Flow

CPU and Memory

Firmware and Hardware

© Fortinet Inc. All Rights Reserved. 41

Review
 Identify the normal behavior of your network
 Monitor for abnormal behavior, such as traffic spikes
 Diagnose problems at the physical and network layers
 Diagnose connectivity problems using the debug flow
 Diagnose resource problems, such as high CPU or memory usage
 Diagnose memory conserve mode
 Diagnose fail-open session mode
 Format the flash memory
 Load a firmware image from the BIOS menu
 Run hardware tests
 Display crash log information

© Fortinet Inc. All Rights Reserved. 42