Layer 3 Troubleshooting

EDU-311

PAN-OS® 8.0

Courseware Version A
Agenda
§ Virtual routers

§ Dynamic routing

§ Troubleshooting potential network issues

§ Troubleshooting session issues

2 | ©2017, Palo Alto Networks, Inc.

Virtual Routers

3 | ©2017, Palo Alto Networks, Inc.

Virtual Router Overview

§ A virtual router (VR) contains routing information.

§ Each Layer 3 interface must be added to a VR.
§ A firewall supports multiple VRs.
§ VRs support:
• Standard routing protocols, such as OSPF, RIP, BGP
• Dual-stack IPv4 and IPv6
• VR-to-VR routing

4 | ©2017, Palo Alto Networks, Inc.

Virtual Router Configuration
Network > Virtual Routers

5 | ©2017, Palo Alto Networks, Inc.

Configuring Static Routes
Network > Virtual Routers > [VR name] > Static Routes > [name or Add]

6 | ©2017, Palo Alto Networks, Inc.

Dynamic Routing

7 | ©2017, Palo Alto Networks, Inc.

Troubleshooting Dynamic Routing with the CLI
§ show commands:
> show system resources
> show log system ...
> show routing protocol [bgp | ospf | rip] ...

§ routed.log can display routing errors:

> tail follow yes mp-log routed.log

§ debug commands:
> debug routing pcap [bgp | ospf | rip] on
> debug routing pcap show
> debug routing pcap [bgp | ospf | rip] view
> debug routing restart (restarts the routing process)

8 | ©2017, Palo Alto Networks, Inc.

Virtual Router Runtime Statistics
A virtual router’s More Runtime Stats link opens a window to display information
about that virtual router.
Network > Virtual Router

9 | ©2017, Palo Alto Networks, Inc.

Troubleshooting OSPF Using the WebUI
Verify the routing table:
§ On the Routing > Route Table tab, look for routes learned via OSPF (“O” flag).
§ Routes that are inactive will not have the letter “A” in the Flags column:

§ The routing table shows internal network routes and default routes propagated
from the upstream routers.

10 | ©2017, Palo Alto Networks, Inc.

Troubleshooting OSPF with the WebUI (Cont.)
§ Verify that OSPF neighbors are in the full state to confirm that OSPF
adjacencies are established:
OSPF > Neighbor

§ Examine the System log to confirm that OSPF connections are established:
Monitor > Logs > System

11 | ©2017, Palo Alto Networks, Inc.

Troubleshooting with the test Command
> test routing fib-lookup ip 196.168.120.153 virtual-router default
--------------------------------------------------------------------------------
runtime route lookup
--------------------------------------------------------------------------------
virtual-router: default
destination: 196.168.120.153
result: via 192.168.2.1 interface ethernet1/1, source 192.168.2.223,
metric 10
--------------------------------------------------------------------------------

> test routing fib-lookup ip 192.168.220.101 virtual-router default

--------------------------------------------------------------------------------
runtime route lookup
--------------------------------------------------------------------------------
virtual-router: default
destination: 192.168.220.101
result: to host, interface ethernet1/3
--------------------------------------------------------------------------------

12 | ©2017, Palo Alto Networks, Inc.

Troubleshooting Potential Network Issues

13 | ©2017, Palo Alto Networks, Inc.

Problem: No Network Connectivity
§ Ping the remote host or firewall interface to test network connectivity.

§ Ping from the client to device, making sure that ping is enabled in the Interface
Management Profile.

§ Ping from the firewall to a host, using the ping source option to specify the
interface IP address to originate the ICMP request.

14 | ©2017, Palo Alto Networks, Inc.

Check Firewall Statistics
§ Verify that the packet is received/transmitted:
> show counter global
> show counter interface <name>
§ Verify policy configuration:
> test nat-policy-match ...
> test security-policy-match ...
§ Verify routing:
> test routing fib-lookup ...
> show routing fib ...
§ Verify that a session was created:
> show session ...
> show session info
§ If session application is “incomplete” or “unknown,” evaluate for asymmetric routing.

15 | ©2017, Palo Alto Networks, Inc.

Verify Connectivity
§ Verify that ARP entries are learned:
> show arp {all | [interface name]}

§ Verify that ARP entries are being added:

> show arp {all | [interface name]}

§ If appropriate, trigger tunnel rekey to verify IKE/IPsec tunnel setting:

> test vpn {ike-sa | ipsec-sa}

§ If security associations (SAs) are not active, view the system logs for details.

§ Run packet captures to confirm that packets are reaching the firewall:
> debug dataplane packet-diag show

16 | ©2017, Palo Alto Networks, Inc.

Check for DHCP Issues (if Configured)
To display the options that a DHCP server has assigned to clients, use the
following command:
> show dhcp server settings { all | <value>}

Interface GW DNS1 DNS2 DNS-Suffix Inherit source

-------------------------------------------------------------------------------------------------------------------------
------
ethernet1/2 192.168.0.1 10.30.11.60 4.2.2.2 ethernet1/3

17 | ©2017, Palo Alto Networks, Inc.

DHCP Server Commands
Use the lease option to display DHCP leases:
> show dhcp server lease {all | <interface name>}
interface: ethernet1/1
ip mac state duration lease_time
192.168.0.52 00:18:8b:b2:1b:b6 committed 0 Mon Oct 6 08:43:10 2016
192.168.0.60 00:14:22:d8:c0:c0 reserved

The clear command will release DHCP leases for an interface:

> clear dhcp lease interface <interface name>
ip 192.168.3.1
mac f0:2c:ae:29:71:34
expired-only

18 | ©2017, Palo Alto Networks, Inc.

DHCP Client Commands
Display DHCP client information:
> show dhcp client state {all | vlan | <interface-name>}
Interface State IP Gateway Leased-until
-------------------------------------------------------------------------------
ethernet1/1 Bound 10.43.14.80 10.43.14.1 110315

19 | ©2017, Palo Alto Networks, Inc.

Packet Captures of DHCP Session
To create a packet capture of DHCP packets:
> debug dhcpd pcap on
> debug dhcpd pcap show
feature dhcp, vr_id 0, output_file /opt/pan/.debug/pcap/dhcp-vr-0.pcap, max_size 1000000,
cur_size 24 bytes, rollover 0 times

--- pcap file ---

-rw-rw-rw- 1 root root 24 Oct 26 10:18 dhcp-vr-0.pcap

> debug dhcpd pcap off

> debug dhcpd pcap view

20 | ©2017, Palo Alto Networks, Inc.

DHCP Troubleshooting Options
PAN-OS® DHCP implementation supports most DHCP options.

Unsupported options are as follows:

§ No support for fully qualified domain names (FQDNs)

§ No support for relay agents

Review the DHCP options for more information:

https://www.paloaltonetworks.com/documentation/71/pan-os/pan-os/networking/dhcp-options

21 | ©2017, Palo Alto Networks, Inc.

Problem: Performance-Related Problems
§ Look for rapidly incrementing counters:
> show counter global filter delta yes

§ Rapidly incrementing counters indicate a latency issue:

Use an application-override policy to discover whether the issue is related to application-
layer processing.

22 | ©2017, Palo Alto Networks, Inc.

Problem: Performance-Related Problems (Cont.)
§ Toggle the URL Filtering Profile setting.

§ Run a URL test in the CLI; the response should return in less than a second:
> test url-info-cloud <URL>

§ View QoS counters:

> show counter global filter delta yes aspect qos

§ View security zone DoS counters. The device might be under attack:
> show counter global filter delta yes aspect dos

§ Determine if there are excessive IP fragments:

> show counter global filter delta yes aspect ipfrag
> show counter global filter | match fragment
> show counter global filter delta yes | match fragment

23 | ©2017, Palo Alto Networks, Inc.

Troubleshooting Session Issues

24 | ©2017, Palo Alto Networks, Inc.

Session Browser
Monitor > Session Browser

Show session information from the CLI:

> show session all

25 | ©2017, Palo Alto Networks, Inc.

show session Command Options
• all
Potentially large amount of output

• id <value>
Detailed view of one session

• info
• Firewall-specific summary
• Displays only traffic statistics to the CPU (software switched)
• No information for fastpath (hardware) switched traffic

• meter
Session count

• rematch
Displays statistics of last session rematch

26 | ©2017, Palo Alto Networks, Inc.

show session info Command
admin@PA-820(active)> show session info
--------------------------------------------------------------------------------
Number of sessions supported: 65530
Number of active sessions: 99 Maximum number of sessions for the platform
Number of active TCP sessions: 44
Number of active UDP sessions:
Number of active ICMP sessions:
53
0
Number of sessions allocated from free pool
Number of active BCAST sessions:
Number of active MCAST sessions:
0
0 pps and Mbps rate from DP. Does not include
Number of active predict sessions:
Session table utilization:
0
0%
offloaded traffic.
Number of sessions created since bootup: 1803
Packet rate: 210/s New connections rate
Throughput: 860160 kbps
New connection establish rate: 5 cps
--------------------------------------------------------------------------------
Session timeout
TCP default timeout: 3600 secs Default and/or configured timeout
TCP session timeout before SYN-ACK received: 5 secs
TCP session timeout before 3-way handshaking: 10 secs settings including in discard state
TCP session timeout after FIN/RST: 30 secs
UDP default timeout: 30 secs
ICMP default timeout: 6 secs
other IP default timeout: 30 secs Accelerated aging settings,
Session timeout in discard state:
TCP: 90 secs, UDP: 60 secs, other IP protocols: 60 secs (True=on)
--------------------------------------------------------------------------------
Session accelerated aging: True
Accelerated aging threshold: 80% of utilization
Scaling factor: 2 X
--------------------------------------------------------------------------------
Session setup
TCP - reject non-SYN first packet: True TCP SYN check on? (True=on)
Hardware session offloading: True
IPv6 firewalling: False Session offload on? (True=on)
--------------------------------------------------------------------------------
Application trickling scan parameters:
Timeout to determine application trickling: 10 secs
Resource utilization threshold to start scan: 80%
Scan scaling factor over regular aging: 8
If DP resources exceed threshold and
-------------------------------------------------------------------------------- if session has been inactive longer
Session behavior when resource limit is reached: drop
-------------------------------------------------------------------------------- than the trickling threshold, then
resources will be freed.

27 | ©2017, Palo Alto Networks, Inc.

show session all Command

28 | ©2017, Palo Alto Networks, Inc.

show session id Command
c2s direction source/destination IP and
admin@PA-820(active)> show session id 3561 port. This was the original packet that
Session 3561 started the session.
c2s flow:
source: 192.168.210.100 [Trust-L3]
dst: 74.125.71.113
proto: 6 Session state
sport: 2723 dport: 80
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown s2c direction source/destination IP and
s2c flow: port. This is for the reply direction
source:
dst:
74.125.71.113 [UnTrust-L3]
10.30.6.210
session. Note that the destination IP
proto:
sport:
6
80 dport: 9069
shows NAT IP and translated port.
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
Session start time
start time : Tue Jan 29 16:25:30 2013
timeout
time to live
:
:
1800 sec
1738 sec
Time to live resets to the original timeout
total byte count : 1604 value, meaning that traffic is reaching the
layer7 packet count : 7
vsys : vsys1 session time.
application : google-analytics
rule : Trust to Untrust
session to be logged at end : True
session in session ager
session synced from HA peer
:
:
True
False
Application and rule that was applied
address/port translation : source + destination
nat-rule : Src_nat(vsys1)
layer7 processing
URL filtering enabled
:
:
completed
False
For HA, session is owned locally
session via syn-cookies : False
session terminated on host : False
session traverses tunnel
captive portal session
:
:
False
False NAT rule that was applied
ingress interface : ethernet1/6
egress interface : ethernet1/3

completed = session offloaded

29 | ©2017, Palo Alto Networks, Inc.

show session all Command Options
show session all admin> show session all filter

show session all start-at

<value> <1-2097152> Show next 1K sessions

30 | ©2017, Palo Alto Networks, Inc.

Flow Session Example
admin> show session all filter type flow
-------------------------------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------------------------
3609 bittorrent ACTIVE FLOW 10.154.4.74[14696]/L3-untrust-3/17 (70.59.59.61[12548])
vsys1 70.59.59.61[12548]/L3-untrust-3 (10.154.4.74[14696])

admin> show session id 3609

Session 62870

c2s flow:
source: 10.154.4.74 [L3-untrust-3]
dst: 70.59.59.61
proto: 17
sport: 14696 dport: 12548
state: ACTIVE type: FLOW
src user: domain\user1
dst user: unknown
s2c flow:
source: 70.59.59.61 [L3-untrust-3]
dst: 10.154.4.74
proto: 17
sport: 12548 dport: 14696
state: OPENING type: FLOW
src user: unknown
dst user: unknown
qos node: ethernet1/8, qos member N/A Qid 0

start time : Sat Oct 26 17:01:42 2015

timeout : 1200 sec
time to live : 843 sec
total byte count(c2s) : 145
total byte count(s2c) : 319
layer7 packet count(c2s) : 1
layer7 packet count(s2c) : 1

31 | ©2017, Palo Alto Networks, Inc.

Predict Session Example
admin@lab1-820> show session all filter type predict
--------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
62870 bittorrent ACTIVE PRED 0.0.0.0[0]/L3-untrust-3/17 (0.0.0.0[0])
vsys1 68.60.187.177[56895]/L3-untrust-3 (68.60.187.177[56895])

admin@wtam-lab-500> show session id 62870

Session 62870

c2s flow:
source: 0.0.0.0 [L3-untrust-3]
dst: 68.60.187.177
proto: 17
sport: 0 dport: 56895
state: ACTIVE type: PRED
src user: unknown
dst user: unknown
qos node: ethernet1/8, qos member N/A Qid 0

s2c flow:
source: 68.60.187.177 [L3-untrust-3]
dst: 0.0.0.0
proto: 17
sport: 56895 dport: 0
state: OPENING type: PRED
src user: unknown
dst user: unknown
qos node: ethernet1/8, qos member N/A Qid 0

start time : Sat Jan 26 17:01:42 2013

timeout : 900 sec
time to live : 843 sec
total byte count(c2s) : 0
total byte count(s2c) : 0
layer7 packet count(c2s) : 0
layer7 packet count(s2c) : 0

32 | ©2017, Palo Alto Networks, Inc.

Disable the SIP Application-Level Gateway
Search for the sip application and select the option to Disable ALG.
Objects > Applications > [search]

33 | ©2017, Palo Alto Networks, Inc.

Questions?

34 | ©2017, Palo Alto Networks, Inc.

Secures the Network