Firepower Data Path Troubleshooting Phase

1: Packet Ingress
Contents
Introduction
Platform Guide
Troubleshooting the Packet Ingress Phase
Identify the Traffic in Question
Check for Connection Events
Capturing Packets on the Ingress and Egress Interfaces
SFR - Capture on the ASA Interfaces
FTD (non-SSP and FPR-2100) - Capture on the Ingress and Egress Interfaces
FTD (SSP) - Capture on the Logical FTD Interfaces
Check for Interface Errors
SFR - Check ASA Interfaces
FTD (non-SSP and FPR-2100) - Check for Interface Errors
FTD (SSP) - Navigating the Data Path to Look for Interface Errors
Data to Provide to Cisco Technical Assistance Center (TAC)
Next Step: Troubleshoot the Firepower DAQ Layer

Introduction
This article is part of a series of articles which explain how to systematically troubleshoot the data
path on Firepower systems to determine whether components of Firepower may be affecting
traffic. Please refer to the Overview article for information about the architecture of Firepower
platforms and links to the other Data Path Troubleshooting articles.

In this article, we will look at the first stage of the Firepower data path troubleshooting, the Packet
Ingress stage.

Platform Guide
The following table describes the platforms covered by this article.

Platform Applicable
Code Description Hardware Notes
Name Platforms
ASA with FirePOWER Services (SFR) ASA-5500-X
SFR N/A
module installed. series
FTD (non- Firepower Threat Defense (FTD) image ASA-5500-X
N/A
SSP and installed on an Adaptive Security Appliance series, virtual
FPR-2100) (ASA) or a Virtual Platform NGFW platforms
FTD installed as a logical device on a The 2100 series does
FPR-9300, FPR-
FTD (SSP) Firepower eXtensible Operative System not use the FXOS
4100, FPR-2100
(FXOS) based chassis Chassis Manager

Troubleshooting the Packet Ingress Phase

The first data path troubleshooting step is to make sure that there are no drops occurring at the
ingress or egress stage of packet processing. If a packet is ingressing but not egressing, then you
can be sure that the packet is being dropped by the device at some place within the data-path or
that the device is unable to create the egress packet (for example, a missing ARP entry).

Identify the Traffic in Question

The first step in troubleshooting the packet ingress stage is to isolate the flow and the interfaces
involved in the problem traffic. This includes:

Flow Information Interface Information

Protocol
Source IP Address
Ingress Interface
Source Port
Egress Interface
Destination IP
Destination Port

For example:

TCP inside 172.16.100.101:38974 outside 192.168.1.10:80

Tip: You may not be able to identify the exact source port since it is often different in each
flow, but the destination (server) port should suffice.

Check for Connection Events

After getting an idea of the ingress and egress interface the traffic should be matching as well as
the flow information, the first step to identify whether Firepower is blocking the flow is to check the
Connection Events for the traffic in question. These can be viewed in the Firepower Management
Center under Analysis > Connections > Events

Note: Prior to checking Connection Events, ensure that logging is enabled in your Access
Control Policy rules. Logging is configured in the "Logging" tab within each Access Control
Policy rule as well as the Security Intelligence tab. Make sure the suspect rules are
configured to send the logs to the "Event Viewer".
In the example above, "Edit Search" is clicked and a unique source (Initiator) IP is added as a filter
to see the flows which were being detected by Firepower. The Action column shows "Allow" for
this host traffic.

If Firepower is intentionally blocking traffic, the Action contains the word "Block". Clicking on
"Table View of Connection Events" provides more data. The following fields in the Connection
Events can be noted if the action is "Block":

- Reason

- Access Control Rule

This, combined with the other fields in the event in question, can help to narrow down which
component is blocking the traffic.

For more information about troubleshooting Access Control Rules, you can click here.

Capturing Packets on the Ingress and Egress Interfaces

If there are no events or the Firepower is still suspected of blocking despite the Connection Events
displaying a rule action of "Allow" or "Trust", the data path troubleshooting continues.

Here are instructions on how to run an ingress and egress packet capture on the various platforms
mentioned above:

SFR - Capture on the ASA Interfaces

Since the SFR module is simply a module running on the ASA Firewall, it is best to first capture on
the ingress and egress interfaces of the ASA to make sure that the same packets which ingress
are also egressing.

This article contains instructions on how to perform the captures on the ASA.

If it has been determined that the packets which are ingressing the ASA are not egressing,
continue to the next phase in troubleshooting (the DAQ phase).
 Note: If packets are seen on the ASA ingress interface, it may be worth checking the
connected devices.

FTD (non-SSP and FPR-2100) - Capture on the Ingress and Egress Interfaces

Capturing on a non-SSP FTD device is similar to capturing on the ASA. However, you can run the
capture commands directly from the CLI initial prompt. When troubleshooting dropped packets it is
advised to add the "trace" option to the capture.

Here is an example of configuring an ingress capture for TCP traffic on port 22:

If you add the "trace" option, you can then select an individual packet to trace through the system
to see how it came to the final verdict. It also helps to make sure that the proper modifications are
done to the packet such as Network Address Translation (NAT) IP modification and that the proper
egress interface has been chosen.
In the example above, we see that the traffic make it to Snort inspection and that it finally reached
an allow verdict and overall was passed through the device. Since the traffic can be seen in both
directions you can be sure traffic is flowing through the device for this session, so an egress
capture may not be needed, but you can take one there as well to make sure the traffic is
egressing properly as shown in the trace output.

Note: If the device is unable to create the egress packet, the trace action is still "allow" but
the packet is not created or seen on the egress interface capture. This is a very common
scenario where the FTD doesn't have an ARP entry for the next hop or destination IP (if this
last one is directly connected).

FTD (SSP) - Capture on the Logical FTD Interfaces

The same steps to generate a packet capture on FTD as mentioned above can be followed on an
SSP platform. You can connect using SSH into the IP address of the FTD logical interface and
enter the following command:

Firepower-module1> connect ftd

>
You can also navigate to the FTD logical device shell from the FXOS command prompt with the
following commands:

# connect module 1 console

Firepower-module1> connect ftd
>
If a Firepower 9300 is used, the module number can vary depending on which Security Module is
being used. These modules can support up to 3 logical devices.

If multi-instances are being used, the instance ID must be included on the "connect" command.
Telnet command can be used to connect to different instances at the same time.

# connect module 1 telnet

Firepower-module1>connect ftd ftd1
Connecting to container ftd(ftd1) console... enter "exit" to return to Boot CLI
>

Check for Interface Errors

Interface level issues can also be checked during this phase. This is especially helpful if packets
are missing in the ingress interface capture. If interface errors are seen, checking the connected
devices can be helpful.

SFR - Check ASA Interfaces

Since the FirePOWER (SFR) module is basically a virtual machine running on an ASA, the actual
ASA interfaces are checked for errors. For detailed information on checking the interface statistics
on the ASA, see this ASA Series Command Reference guide section.

FTD (non-SSP and FPR-2100) - Check for Interface Errors

On non-SSP FTD devices, the > show interface command can be run from the initial command
prompt. The interesting output is highlighted in red.

FTD (SSP) - Navigating the Data Path to Look for Interface Errors

The 9300 and 4100 SSP platforms have an internal fabric interconnect which first handles the
packets.
It is worth to check if there are any interface issues at the initial packet ingress. These are the
commands to run on the FXOS system CLI in order to get this information.

# connect module 1 telnet

Firepower-module1>connect ftd ftd1
Connecting to container ftd(ftd1) console... enter "exit" to return to Boot CLI
>
This is a sample output.
After the fabric interconnect handles the packet upon ingress, it is then sent to the interfaces which
are assigned to the logical device hosting the FTD device.

Here is a diagram for reference:

In order to check for any interface level issues, enter the following commands:
# connect module 1 telnet
Firepower-module1>connect ftd ftd1
Connecting to container ftd(ftd1) console... enter "exit" to return to Boot CLI
>
This is an output example (possible issues highlighted in red):

If any errors are seen, the actual FTD software can be checked for interface errors as well.
In order to get to the FTD prompt, it is first necessary to navigate to the FTD CLI prompt.

# connect module 1 telnet

Firepower-module1>connect ftd ftd1
Connecting to container ftd(ftd1) console... enter "exit" to return to Boot CLI
>
For multi-instances:

# connect module 1 telnet

Firepower-module1>connect ftd ftd1
Connecting to container ftd(ftd1) console... enter "exit" to return to Boot CLI
>
This is an output example.
Data to Provide to Cisco Technical Assistance Center (TAC)
Data Instructions
Connection
Event See this article for instructions
screenshots
'show
interface' See this article for instructions
output
For ASA/LINA: https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next
Packet firewalls/1180...
captures For Firepower: http://www.cisco.com/c/en/us/support/docs/security/sourcefire-firepower-800
appliances/11777...
Log into ASA CLI and have the terminal session saved to a log. Enter the show tech comm
ASA 'show the terminal session output file to TAC.
tech' output This file can be saved to disk or an external storage system with this command.
show tech | redirect disk0:/show_tech.log
Troubleshoot
file from the
Firepower
http://www.cisco.com/c/en/us/support/docs/security/sourcefire-defense-center/117663-techn
device
inspecting
the traffic
Next Step: Troubleshoot the Firepower DAQ Layer
If it is unclear as to whether the Firepower device is dropping packets, the Firepower device itself
can be bypassed to rule out all of the Firepower components at once. This is especially helpful in
mitigating an issue if the traffic in question is ingressing the Firepower device but not egressing.

To proceed, please review the next phase of Firepower data path troubleshooting; The Firepower
DAQ. Click here to continue.