IPsec_GW Reference

Pipeline

User Guide
AMD Pensando Software-in-Silicon Development Kit

UG1671 (v1.0) November 30, 2023

AMD Adaptive Computing is creating an environment where

employees, customers, and partners feel welcome and included. To
that end, we’re removing non-inclusive language from our products
and related collateral. We’ve launched an internal initiative to remove
language that could exclude people or reinforce historical biases,
including terms embedded in our software and IPs. You may still find
examples of non-inclusive language in our older products as we work
to make these changes and align with evolving industry standards.
Follow this link for more information.
Table of Contents
Chapter 1: Introduction.............................................................................................. 3

Chapter 2: Architecture...............................................................................................6
Transport and Tunnel Modes.....................................................................................................6
Encryption Path........................................................................................................................... 7
Decryption Path........................................................................................................................... 8
Stages......................................................................................................................................... 10
Features......................................................................................................................................10
End-to-End Testing....................................................................................................................11

Chapter 3: Building and Running the Reference Pipeline..................... 13

About the SSDK..........................................................................................................................13
Building a Simulator Image..................................................................................................... 14
Managing the Simulation Environment................................................................................. 15
Building a DSC Firmware Image..............................................................................................16
Copying an Image to the DSC..................................................................................................16
Loading an Image..................................................................................................................... 21
Starting the DP-App.................................................................................................................. 22
Cleaning the Build..................................................................................................................... 23
Performing a Packet Test......................................................................................................... 23
Troubleshooting........................................................................................................................ 23

Appendix A: Additional Resources and Legal Notices............................. 24

Finding Additional Documentation......................................................................................... 24
Support Resources.................................................................................................................... 24
References..................................................................................................................................24
Revision History......................................................................................................................... 24
Please Read: Important Legal Notices................................................................................... 25

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 2
 Chapter 1: Introduction

Chapter 1

Introduction
This guide presents an overview of the IPsec_GW reference pipeline (Transport and Tunnel
mode). It provides details of the P4 program that you can load onto an AMD Pensando™ second
generation ("Elba") data processing unit (DPU), or run within the provided x86 simulator. P4
pipeline programmability gives flexible software-defined constructs that enable you to develop
your networking software quickly, load it onto an AMD Pensando DPU, and test it. The reference
pipelines supply P416 source code for P4I and P4E (protocol processing) modules and P416-based
libraries in binary format for P4 RxDMA and P4 TxDMA that you can access via APIs for handling
message transfer with host and local CPU. This combination of source code and libraries
facilitates the implementation of the provided reference pipeline. The provided P4 binary can be
deployed alongside your P4 code on the same DPU to integrate the functionality into the
customer's system. This approach streamlines the implementation process and ensures seamless
compatibility.

In cloud service provider, enterprise, and public sector environments, encryption services are
commonly delivered via Edge/VPN gateway physical or virtual (VNF) appliances. However, this
approach compromises performance and drives up costs if the encryption offering needs to scale
the number of tunnels or if high IPsec throughput is required. Physical appliances tend to
increase in cost and size as performance requirements increase for both throughput and tunnel
scale. VNF or other software-only solutions typically pin a tunnel to a single CPU and can only
achieve a peak throughput of 1.25-2.5 Gb/s per tunnel. Performance challenges of software-only
IPsec solutions include:

• High CPU utilization: Encryption and decryption of IPsec packets can be CPU-intensive,
leading to high CPU utilization on the server or appliance. This can impact the performance of
other applications running on the same server or appliance.

• Limited throughput: CPU speed limits the throughput of an IPsec tunnel on a software-only
solution. This can be a problem for networks that need to support high-bandwidth traffic.

• Latency: Encryption and decryption of IPsec packets can add latency to network traffic. This
can be a problem for applications that are sensitive to latency, such as VoIP and video
conferencing.

When there is a need to encrypt high-speed links or provide a scalable encryption service that
does not consume racks of CPUs, P4-programmable DPUs provide a more scalable and
performant solution.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 3
 Chapter 1: Introduction

Customers of cloud providers often want to encrypt on-ramp circuits between colocation
facilities, enterprise data centers, and their cloud resources. High-speed cloud on-ramp circuits
offer sub 1 Gb/s to 100 Gb/s links with one or more IPsec tunnels, which is challenging to
encrypt using current IPsec implementations that rely on CPUs.

The AMD Pensando DPU can offload encryption services from the x86 server, significantly
increasing throughput per tunnel and the number of tunnels supported, without requiring
additional compute resources from a software-only solution (VNF) or relying on large appliances.
Third-party vendors can use the AMD Pensando DPU to improve the IPsec throughput and
tunnel scale of their appliance offerings, while also reducing their footprint.

The benefits of using an AMD Pensando DPU for encryption services include:

• Scalability: DPUs and current SSDK software can scale to support up to 64,000 encrypted
tunnels.

• Performance: DPUs can encrypt and decrypt IPsec packets at line rate without impacting the
performance of other applications.

• Cost-effectiveness: DPUs are a more cost-effective solution using domain-specific encryption

and network services accelerators versus dedicated crypto engines and CPUs.

• Flexibility: The AMD Pensando DPU can support multi-service offerings, leveraging a flow-
based approach, additional networking and security functions with encryption (Packet
Rewrite, SDN Policy Offload, Flow Offloads, NAT, Stateful Firewall, Observability, and Massive
Control and Data plane scale), or a policy-based VPN for stateless environments.

• Flexible encryption: P4 programmability enables you to select what type of traffic should be
encrypted and how the traffic is mapped to service associations (SAs) and IPsec tunnels.

The IPsec_GW P4 reference pipeline is a robust method to enhance security without sacrificing
network performance. The AMD Pensando DPU and P4 pipeline have achieved 100 Gb/s for a
single IPsec tunnel and up to 260 Gb/s of bidirectional throughput for a single DPU. The
IPsec_GW reference pipeline is a bump-in-the-wire (BITW) implementation ideal for a
SmartSwitch or an appliance form factor, as shown in the following figure. When deployed inside
a SmartSwitch as a top-of-rack device or in an appliance, the DPU accelerates IPsec services for
any traffic that enters or leaves the device. A DPU with the IPsec_GW reference pipeline
typically connects to a switching ASIC as a bump-in-the-wire. Alternatively, the BITW approach
includes an appliance mode using an x86 system with DPUs deployed in PCIe® slots. It takes
unencrypted packets from the wire, encrypts them at line rate, and sends them back to the
switching ASIC to be forwarded to their next hop or destination. To scale out performance,
multiple DPUs can be connected to an ASIC.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 4
 Chapter 1: Introduction

Figure 1: BITW Implementation for SmartSwitch Appliances

DPU ×N x86 System

Optional
200G TX 200G TX
Control Plane
200G RX 200G RX

DPU ×N
ASIC

Traffic Traffic
To/From To/From
Network Network
X28791-110623

The IPsec pipeline can also be enhanced to support a host-to-network deployment, as shown in
the following figure. This allows offloads to be performed on a per-compute node basis.

Figure 2: Host to Network Implementation

x86 System pf0

Traffic
To/From Host

DPU

Traffic
To/From
Network
X28792-110623

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 5
 Chapter 2: Architecture

Chapter 2

Architecture
The P4 pipeline is a software-defined data plane programmed to perform various tasks, including
encryption and decryption. In the case of IPsec, the P4 pipeline is programmed to perform inline
encryption and decryption without the need to offload to a crypto engine. This allows the P4
pipeline to encrypt and decrypt packets at line speed without impacting network performance or
latency-sensitive applications.

Transport and Tunnel Modes

The IPsec_GW reference pipeline implementation supports transport and tunnel modes, as
shown in the following figure:

Figure 3: UDP Transport and Tunnel Modes

Original Packet

IP L4 Header Data

UDP Transport Mode

IP ESP Header L4 Header Data ESP Trailer ESP Auth

Encrypted

Authenticated

Tunnel Mode

New IP ESP IP L4 Header Data ESP Trailer ESP Auth

Encrypted

Authenticated
X28810-110823

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 6
 Chapter 2: Architecture

Encryption Path
Packet paths for encryption and decryption follow P4 Ingress to P4 Egress pipelines.

For the encryption path, where a packet comes in from an uplink un-encrypted, the P4 pipeline
uses the crypto block between the P4I deparser and the packet buffer. The P4 pipeline can also
be used for post-encryption/decryption operations (multi-service), such as packet rewrites, SDN
policy offload, flow offloads, NAT, stateful firewall, observability, or additional flow lookup. See
the following figure.

Figure 4: Packet Encryption

PCIe DPU DDR

Storage and NOC

Crypto Arm
Offloads

P4 TxDMA P4 RxDMA

Inline
Crypto
P4 P4
Egress Packet Buffer Ingress
Traffic Manager
Inline
Crypto

Encrypted Eth 1/1 Eth 1/2 Unencrypted

X28794-110623

It is worth noting that Arm® cores are not in the path of the IPsec data path, unless desired. All
IPsec functionalities can be implemented in P4 with an inline crypto engine, which provides near-
line rate performance for the IPsec data plane. The current reference pipeline supports
configuring static security associations (SAs) and using a custom control plane (strongSwan or
other third party). The SSDK provides corresponding SA APIs, which customers can use to
program the P4 SA table. The P4 pipeline can also be extended to support dynamic SA
configuration (strongSwan or other third party vendors). This allows the SAs to be configured
automatically based on the traffic flows in the network.

Below is the path taken in the P4 pipeline to perform IPsec encryption:

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 7
 Chapter 2: Architecture

• Ingress: lif → p4i_init → nacl → tunnel_mapping → tunnel → ipsec_encrypt → inter_pipe →

Egress

• Egress: p4e_init → ipsec_action → nexthop → checksum → inter_pipe → Out

The following applies:

• Because IPsec_GW is a stateless app, SA index required to encrypt the packet is derived from
packet vlan-id.
• The tunnel-mapping table entry contains the information about vlan-id -> tunnel-id mapping.
• The tunnel table entry provides ipsec tunnel mode, encrypt sa-index, nexthop info, and so on.
• Based on the sa index, the ipsec_info required for encryption is derived in the ipsec_encrypt
table and it is passed to the deparser.
• The P4I deparser sets the IPSec_info_valid bit, populates the IPSec_info, inserts a valid ESP
header and trailer, and adds an empty ESP authentication trailer. The modified packet is sent
to the inline IPsec engine.
• The Inline IPsec engine encrypts the plain text payload and fills the ESP authentication trailer.
The resultant packet is sent to the Packet Buffer.
• The P4E Parser identifies the packet and parses the ESP header. The ipsec_post_encrypt table
sets up esp iv header and UDP NAT (if enabled).
• The Out uplink port and packet MAC address are derived in the nexthop table, and the packet
is sent to the designated uplink port.

Decryption Path
Packet paths for encryption and decryption follow P4 Ingress to P4 Egress pipelines.

For the Decryption path, where a packet comes in from the uplink encrypted, the packet gets
decrypted in P4I. The decrypted packet goes to P4E where it is validated, and can be sent
directly to the uplink interface, or can be sent back to the P4 pipeline for a 'flow lookup' on inner
packet-header fields for additional services such as packet rewrites, SDN policy offload, flow
offloads, NAT, stateful firewall, or observability. See the following figure.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 8
 Chapter 2: Architecture

Figure 5: Additional Services

SDN Policy Flow Offload SDN FW LB NAT Observability

CPU

+
DPU
Stateful with
High Performance
X28812-110823

Below is the path taken in the P4 pipeline to perform IPsec decryption:

• Ingress: lif → p4i_init → nacl → ipsec_decrypt_sa_lookup → tunnel → ipsec_decrypt →

inter_pipe → Egress

• Egress: p4e_init → ipsec_decrypt → nexthop → checksum → inter_pipe → Out

The following applies:

• On tunnel creation, an entry needs to be created in ipsec_decrypt_sa_lookup with a 3-tuple

containing the tunnel source/destination IP and security parameter index (SPI).
• The IPsec decrypt SA index and tunnel info are derived from this ipsec_decrypt_sa_lookup
hash table.
• Based on the SA index, the IPsec decrypt info is derived in the ipsec_decrypt table and passed
to the deparser.
• The deparser gets instruction from the P4 pipeline, sets the IPsec_info_valid bit, and populates
the IPsec_info. The modified packet is sent to the Inline IPsec engine.
• The inline IPsec engine decrypts the encrypted payload and validates the ESP authentication
trailer. A new sideband signal is added to carry the Integrity Check Value (ICV) Check Status
back to the P4 Pipeline via the Packet Buffer.
• The decrypted packet goes to P4E for further processing. The parser extracts and passes the
icv check status to the P4 Pipeline. The IPsec_info, ESP header and trailer, and ESP
authentication trailer are removed by the Deparser. The IPsec_info_valid bit is cleared by the
Deparser.
• The IP protocol is extracted from the ESP trailer next header field, and the reconstructed
packet is sent to the designated uplink port.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 9
 Chapter 2: Architecture

Stages
P4I and P4E consist of eight match-action stages, and have a parser and a deparser unit at the
beginning and end (respectively) of the pipeline. The RxDMA and TxDMA have eight match-
action stages each and do not have a parser or deparser unit but instead feed into dedicated
DMA blocks at the end of the pipeline. The SxDMA pipeline in Elba has four match-action stages
and its own DMA block, and is intended for offload unit control and other memory-to-memory
transfers, not to manipulate packets. The match-action stages in all pipelines are identical and
consist of a lookup unit called Table Engine (TE) and four Match Processing Units (MPUs) at each
stage that execute the P4 action code.

Features
The Elba DPU has the following inline IPsec encryption/decryption features:

• Programmable AES-GCM inline encryption/decryption offload support in P4-Ingress and P4-

Egress pipelines.
• Supports ESP (Encapsulating Security Payload) and AH (Authentication Header) protocols.
• The Inline-crypto blocks (IS block) are positioned next to the P4I and P4E engines. There is
one IS block after P4I (between the P4I deparser and PB), and another identical IS block after
P4E (between the P4E deparser and PB).
• P4I and P4E can access these engines for inline, line-rate performance encryption. The IS
block supports AES-GCM 128/256 bit Key symmetric algorithms for IPsec/DTLS-like packet-
based encryption use cases. In this case, the packet stays in the packet buffer while the IS
block encrypts/decrypts the packet in place; therefore, no redirection/copy to memory is
required.

The IPsec_GW reference pipeline demonstrates the following features:

• Supports UDP encapsulation (can be enabled/disabled).

• Implements stateless IPsec, and supports both IPsec-tunnel and IPsec-transport modes. IPsec
mode is configurable per tunnel.
• Both IPsec tunnel and transport modes are using ESP protocol.
• For encryption, the tunnel index is derived from the packet vlan-id. The tunnel table contains
the IPsec mode and Encrypt-SA index to encrypt the packet.
• For decryption, a static entry needs to be programmed for each tunnel in the
IPsec_decrypt_sa_lookup table by dp-app to handle the decryption.
• VXLAN and IPv6 packets are not supported in the IPsec_GW reference pipeline and are
dropped in the P4 pipeline.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 10
 Chapter 2: Architecture

• Encryption of encapsulated packet, anti-reply and 64-bit ESN support is not added/verified.
• Flow-based forwarding is not supported.

The following crypto offloads are supported:

• AES-GSM/XTS Crypto/Decrypt
• AES CCM/AES CBC/DES/3DES/SHA1/SHA2/SHA3/HMAC
• SHA3-256
• SHA3-384/512
• PKE, DRBG, TLS 1.1 & 1.3

End-to-End Testing
You can perform end-to-end testing for the IPsec_GW reference pipeline.

As shown in the following figure, two DPUs are connected back-to-back using port eth1/2. The
other uplink ports of DSC1 & DSC2 (eth1/1) are connected to Ixia traffic generator port-1 and
port-2 respectively. The ipsec_gw P4 program runs on both DSCs. DSC-1 encrypts Ixia IP
traffic from port-1, fed to DSC-2 via port eth1/2. DSC-2 decrypts the IP packet and sends the
decrypted packet to Ixia port-2. For performance analysis, traffic should be started from Ixia
port-1 and port-2 and run symmetrically for testing bi-directional throughput.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 11
 Chapter 2: Architecture

Figure 6: End-to-End Testing

Traffic Generator (Ixia)

unencrypted

E1/1
Elba DSC-1

Static SAs or strongSwan P4I Inline AES-GCM Packet Buffer

(3rd party Control Plane) P4E (Packet Rewrite)

Arm P4 Pipeline

E1/2

encrypted

E1/2
Elba DSC-2

Static SAs or strongSwan P4I Inline AES-GCM Packet Buffer

(3rd party Control Plane) P4E (Packet Rewrite)

Arm P4 Pipeline

E1/1

unencrypted

Traffic Generator (Ixia)

X28813-112023

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 12
 Chapter 3: Building and Running the Reference Pipeline

Chapter 3

Building and Running the Reference

Pipeline
This chapter describes how to build and run the reference pipeline using the Software-in-Silicon
Development Kit (SSDK).

Note: This chapter assumes you already have installed the developer environment. If not refer to the
following:

• The SSDK Quick Start Guide or Getting Started Guide for installation of the SSDK.
• The SSDK Getting Started Guide to setup your environment.

About the SSDK

The SSDK is installed as a container. It is delivered in three separate software packages:

• A container image that constitutes the build environment and includes generic tools, such as
the Linaro Arm® cross-compile toolchain.
• A tarball with the source code of the reference pipelines and AMD-specific tools, such as the
P4 compiler and the Distributed Services Card (DSC) simulator.
• A tarball with documentation.

The following table lists installed directories and their contents.

Table 1: SSDK Directories

Directory Contents
/sw/nic SSDK main folder for build scripts
/sw/nic/rudra Relevant code and files for development
/sw/nic/rudra/src/pds-core Core Application, Environment Initialization, Starts services,
PDS Agent, gRPC Listener, PDS Layer API, and Common
Libraries
/sw/nic/rudra/src/lib Common code and files for all pipelines, LIFS, Memory Utils,
Debug, Tracing, and Interface Management
/sw/nic/rudra/src/<reference-pipeline> The reference pipelines

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 13
 Chapter 3: Building and Running the Reference Pipeline

Table 1: SSDK Directories (cont'd)

Directory Contents
/sw/nic/rudra/src/<reference-pipeline>/p4/ P4 Source Code for the specific pipeline
p4-16/
/sw/nic/rudra/src/conf Platform, Pipeline, and Infrastructure configuration files
/sw/nic/rudra/src/<reference-pipeline>/dp Dataplane and Pipeline specific DP application
/sw/nic/tools The developer tools

Note: This container can be used to build both x86 and Arm images.

Building a Simulator Image

To build a simulator image, do the following within the container:

1. Change directory to the main folder for build scripts:

cd /sw/nic

2. Clean the build directory:

ARCH=x86_64 ./tools/build.sh --p4_program ipsec_gw --clean

3. Build the simulator image:

ARCH=x86_64 ./tools/build.sh --p4_program ipsec_gw

The full syntax for the tools/build.sh command is:

# tools/build.sh --help
ARCH=<architecture> --p4_program <p4_program> [--clean] [--create-sim-
target]
Architectures: aarch64
x86_64 (default)
--p4_program: hello_world
flow_offload
classic_host_offload
classic_rtr flow_offload
flow_ha
ipsec_gw
sdn_policy_offload
--clean: cleaning target and build directories (optional)
--create-sim-target: build the rudra target sim

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 14
 Chapter 3: Building and Running the Reference Pipeline

Managing the Simulation Environment

Once the build process is complete, start the simulator on the built reference pipeline using the
following command:

./tools/dsc-sim-start.sh --p4_program ipsec_gw

At this point both the ASIC simulator and core application are running, and the two uplink
interfaces (uplink0 and uplink1) are exposed to Linux® as tap interfaces (Eth1-1 and Eth1-2).
Validate this by running show commands such as the following:

pdsctl show lif

pdsctl show port status

As a part of the simulator startup, the dp-app process is started, which reads the file /sw/nic/
rudra/src/conf/ipsec_gw/config.json. This file contains instructions to create the flow
with policy and route results, and to populate specific entries into the tables. To view the file, run
the command:

cat /sw/nic/rudra/src/conf/ipsec_gw/config.json

If desired you can expose the uplink interfaces outside the container, so you can run traffic tools
from outside to inject packets to the uplinks, To do so, run the following commands on the host
of the container:

cd $TTOP/src/github.com/pensando/sw/nic/
DSC_INSTANCE=<ssdk-dev-container-name> ./rudra/test/tools/setup-uplink-
interfaces.sh

Note: <ssdk-dev-container-name> can be obtained via docker ps.

To stop the simulator environment, execute the dsc-sim-stop.sh script. This kills the
processes including the ASIC simulator and pds core application:

./tools/dsc-sim-stop.sh

To restart the simulation environment, execute the dsc-sim-restart.sh script. This restarts
all the processes including the ASIC simulator and pds core application, which is useful in
recovering from scenarios such as a process crashing:

./tools/dsc-sim-restart.sh --p4_program ipsec_gw

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 15
 Chapter 3: Building and Running the Reference Pipeline

Building a DSC Firmware Image

To build a DSC firmware image, do the following within the P4 development container:

1. Change directory to the main folder for build scripts:

cd /sw/nic

2. Clean the build directory:

ARCH=aarch64 ./tools/build.sh --p4_program ipsec_gw --clean

3. Build the DSC firmware image:

ARCH=aarch64 ./tools/build.sh --p4_program ipsec_gw

This creates an image for the DSC/DPU as a tarball in the /sw/nic directory, for example
dsc_fw_elba_.tar.
4. List the built DSC firmware image:
ls /sw/nic/dsc_fw_elba_.tar

Note: This is also aliased to naples_fw_elba.tar.

Copying an Image to the DSC

After building the DSC firmware image you must copy it to the DSC. You can use any of the
following in-band or out-of-band management interfaces:

• The internal management NIC (int_mnic) interface

• The memtun interface
• A serial cable connected to the DCS console
• The external RJ45 out-of-band (OOB) interface

Using the Internal Management NIC Interface

To copy the DSC firmware image using the internal management NIC (int_mnic) interface:

1. Boot up the host with the DSC plugged in.

2. Login to the host.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 16
 Chapter 3: Building and Running the Reference Pipeline

3. Confirm the DSC enumerates as a PCIe® device, and note the PCIe address of its
management controller:
# lspci -d 1dd8:
12:00.0 PCI bridge: Pensando Systems Inc Device 0002
13:00.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
13:01.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
13:02.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
14:00.0 Ethernet controller: Pensando Systems Inc DSC Ethernet Controller
15:00.0 Ethernet controller: Pensando Systems Inc DSC Ethernet Controller
16:00.0 Ethernet controller: Pensando Systems Inc DSC Management
Controller

In this example the PCIe address is 16:00.0.

4. Load the IONIC Ethernet driver on the host.
5. Identify the interface for the DSC management controller.
Look in the dmesg logs from when you were loading the driver, find the PCIe address from
step 3 (16:00.0 in this example), and note the associated interface:
# dmesg | tail
[ 341.163138] ionic Pensando Ethernet NIC Driver, ver 23.02.1-001
[ 341.163334] ionic 0000:14:00.0: 126.016 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:11:00.0 (capable of
252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 341.165911] ionic 0000:14:00.0: FW: 1.51.0-G-37
[ 341.210114] ionic 0000:15:00.0: 126.016 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:11:00.0 (capable of
252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 341.210301] ionic 0000:14:00.0 ens1: renamed from eth0
[ 341.212756] ionic 0000:15:00.0: FW: 1.51.0-G-37
[ 341.261967] ionic 0000:16:00.0: 126.016 Gb/s available PCIe
bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:11:00.0 (capable of
252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 341.263733] ionic 0000:16:00.0: FW: 1.51.0-G-37
[ 341.345005] ionic 0000:16:00.0 eth1: Link up - 1 Gbps

The final line shows that the driver is using is using the eth1 interface for the DSC
management controller.
6. Configure an IP address for this interface:
# ifconfig eth1 169.254.<mgmt_ctlr_pcie_bus_number>.2/24 up

where:
• <mgmt_ctlr_pcie_bus_number> is the first component of the PCIe address for the
management controller from step 3, converted from hexadecimal to decimal.
In this example, the PCIe address is 16:00.0, so the bus address is 0x16 which converts
to 22 decimal.
The command corresponding to the example output shown in step 3 is therefore:
# ifconfig eth1 169.254.22.2/24 up

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 17
 Chapter 3: Building and Running the Reference Pipeline

7. Confirm the IP address has been set:

# ifconfig eth1
ens1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 169.254.22.2 netmask 255.255.255.0 broadcast
169.254.22.255
ether 00:ae:cd:2e:70:d0 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

8. Verify that you can ping the DSC from the host:
Note: The internal management NIC for the DSC has a default IP address of 169.254.22.1.

# ping 169.254.22.1 -c 3
PING 169.254.22.1 (169.254.22.1) 56(84) bytes of data.
64 bytes from 169.254.22.1: icmp_seq=1 ttl=64 time=0.205 ms
64 bytes from 169.254.22.1: icmp_seq=2 ttl=64 time=0.091 ms
64 bytes from 169.254.22.1: icmp_seq=3 ttl=64 time=0.085 ms

--- 169.254.22.1 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2037ms

9. SSH to the DSC from the host as root with a password of pen123 :
# ssh -lroot 169.254.22.1
The authenticity of host '169.254.22.1 (169.254.22.1)' can't be
established.
ECDSA key fingerprint is SHA256:AoU0vi8BifouUOfqSg78t08JgaH7vHHBZfK58CnS
+EI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '169.254.22.1' (ECDSA) to the list of known
hosts.

[email protected]'s password:

10. Confirm the IP address for the internal Management NIC (MNIC) interface on the DSC is
169.254.<pcie_bus_number>.1:
Note: The int_mnic0 interface is the internal MNIC net device on the DSC that is created by default.

# ifconfig int_mnic0
int_mnic0 Link encap:Ethernet HWaddr 00:AE:CD:01:C6:C4
inet addr:169.254.22.1 Bcast:169.254.22.255 Mask:255.255.255.0
inet6 addr: fe80::2ae:cdff:fe01:c6c4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:41 errors:0 dropped:0 overruns:0 frame:0
TX packets:65 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5351 (5.2 KiB) TX bytes:10066 (9.8 KiB)

11. Use scp to copy the image over the internal management NIC interface from the host
(specified by the IP address that you set in step 6) to the /data/ directory on the card:
$ scp <user>@<host_ip_address>:<SSDK_dir>/src/github.com/pensando/sw/nic/
dsc_fw_elba_.tar /data/

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 18
 Chapter 3: Building and Running the Reference Pipeline

For example:
$ scp [email protected]:/home/jdoe/ssdk/src/github.com/pensando/sw/nic/
dsc_fw_elba_.tar /data/

Using the Memtun Interface

Memtun is another PCIe memory-based interface. It is available for Linux hosts, and does not
require the IONIC driver on the host, but is less secure.

To copy the DSC firmware image using the memtun interface:

1. Boot up the host with the DSC plugged in.

2. Login to the host.
3. Confirm the DSC enumerates as a PCIe device, and note the PCIe addresses of its first virtual
downstream port and of its management controller:
# lspci -d 1dd8:
12:00.0 PCI bridge: Pensando Systems Inc Device 0002
13:00.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
13:01.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
13:02.0 PCI bridge: Pensando Systems Inc DSC Virtual Downstream Port
14:00.0 Ethernet controller: Pensando Systems Inc DSC Ethernet Controller
15:00.0 Ethernet controller: Pensando Systems Inc DSC Ethernet Controller
16:00.0 Ethernet controller: Pensando Systems Inc DSC Management
Controller

In this example the PCIe addresses are 13:00.0 and 16:00.0, respectively.
4. Compile the memtun binary for the host using the code from $sw/platform/src/app/
memtun.
5. Start the memtun binary on the host:
# ./memtun -s <downstream_port_pcie_address>
169.1.<mgmt_ctlr_pcie_bus_number>.2 &

where:
• <downstream_port_pcie_address> is the PCIe address of the virtual downstream
port from step 3.
In this example it is 13:00.0.
• <mgmt_ctlr_pcie_bus_number> is the first component of the PCIe address for the
management controller from step 3, converted from hexadecimal to decimal.
In this example, the PCIe address is 16:00.0, so the bus address is 0x16 which converts
to 22 decimal.
The command corresponding to the example output shown in step 3 is therefore:
# ./memtun -s 13:00.0 169.1.22.2 &

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 19
 Chapter 3: Building and Running the Reference Pipeline

The memtun utility is started by default on DSCs running goldfw or ssdk firmware. When
memtun starts on the host it establishes a connection with the DSC and creates a tun device
visible in ifconfig on both the host and the DSC.
6. Confirm the IP address has been set:
# ifconfig tun0
tun0 Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:169.1.22.2 P-t-P:169.1.22.3
Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:96 (96.0 B) TX bytes:96 (96.0 B)

7. Verify that you can ping the DSC from the host:
Note: The tun device for the DSC has a default IP address of 169.254.22.3.

# ping 169.254.22.3 -c 3
PING 169.254.22.3 (169.254.22.3) 56(84) bytes of data.
64 bytes from 169.254.22.3: icmp_seq=1 ttl=64 time=0.205 ms
64 bytes from 169.254.22.3: icmp_seq=2 ttl=64 time=0.091 ms
64 bytes from 169.254.22.3: icmp_seq=3 ttl=64 time=0.085 ms

--- 169.254.22.3 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2037ms

8. SSH to the DSC from the host as root with a password of pen123 :
# ssh -lroot 169.254.22.1
The authenticity of host '169.254.22.1 (169.254.22.1)' can't be
established.
ECDSA key fingerprint is SHA256:AoU0vi8BifouUOfqSg78t08JgaH7vHHBZfK58CnS
+EI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '169.254.22.1' (ECDSA) to the list of known
hosts.

[email protected]'s password:

9. Confirm the IP address for the tun0 interface on the DSC is 169.254.<pcie_bus_number>.3:
Note: The tun0 interface is the memtun device on the DSC that is created by default.

# ifconfig tun0
tun0 Link encap:UNSPEC HWaddr
00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:169.1.22.3 P-t-P:169.1.22.2
Mask:255.255.255.255
inet6 addr: fe80::b1a3:b97a:c6a3:3b12/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:96 (96.0 B) TX bytes:96 (96.0 B)

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 20
 Chapter 3: Building and Running the Reference Pipeline

10. Use scp to copy the image over the memtun interface from the host (specified by the IP
address that you set in step 5) to the /data/ directory on the card:
$ scp <user>@<host_ip_address>:<SSDK_dir>/src/github.com/pensando/sw/nic/
dsc_fw_elba_.tar /data/

For example:
$ scp [email protected]:/home/jdoe/ssdk/src/github.com/pensando/sw/nic/
dsc_fw_elba_.tar /data/

Using the DSC Console

To copy the DSC firmware image using the DSC console:

1. Connect the DSC console to the host with a serial cable.

2. Use scp to copy the image over the serial cable from the host to the /data/ directory on
the card.
$ scp <user>@<host>:<SSDK_dir>/src/github.com/pensando/sw/nic/
dsc_fw_elba_.tar /data/

Using the External RJ45 OOB Interface

If the DSC has an external RJ45 OOB interface you can connect it to the management network,
configure it, and use it to copy images:

1. Ensure the oob_mnic0 interface is up:

ifconfig oob_mnic0

If you get no output:

• Run the following command:
dhclient oob_mnic0

• Boot `gold-fw` if needed.

2. Use scp to copy the image over the out of band interface to the /data directory on the
card:
scp user@server://~dsc_fw_elba.tar

Loading an Image
To load an image you have copied to the DSC, do the following on the DSC:

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 21
 Chapter 3: Building and Running the Reference Pipeline

1. Synchronize data on disk with memory:

# sync

2. Set the copied image as the boot image:

# sysupdate.sh -p /data/dsc_fw_elba_.tar

3. Restart the DSC:

# sysreset.sh

The DSC comes up with the new firmware loaded.

4. Get the details of the loaded firmware and confirm that the version number and build time
are as expected:
# pdsctl show version
Firmware Version : <version_number>
Pipeline : rudra
P4 Program : ipsec_gw
Build Time : <build_time>

Note: You can also use the table access tools (p4ctl or, if one of the reference pipelines had been
used to create the image, pdsctl) to verify that the correct software is executed.

Starting the DP-App

The dp-app is not started by default in hardware. You must start it yourself.

To do so:

1. Start the dp-app as follows:

• To use the default init config from /nic/conf/config.json:
start-dp-app.sh

• To use a different config file specify it with the -c option. For example:
start-dp-app.sh -c /nic/config/config_scale.json

2. Use grep to confirm that the dp-app is running:

# ps -ef | grep pds_dp
root 1659 1 99 00:00 ttyS0 00:07:38 pds_dp_app -c /nic/conf/
config.json
root 1733 578 0 00:00 ttyS0 00:00:00 grep pds_dp

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 22
 Chapter 3: Building and Running the Reference Pipeline

Cleaning the Build

You must clean the current build before doing any of the following:

• Building a P4 program for the first time.

• Building a different P4 program.
• Switching between x86 and Arm targets.

To do so:

1. Change directory to the main folder for build scripts:

cd /sw/nic

2. Clean the build directory:

ARCH=x86_64 ./tools/build.sh --p4_program ipsec_gw --clean

Performing a Packet Test

After you have initialized the dp-app with the default config.json, you can perform packet
tests by sending packets to the exposed uplink interfaces (Eth1-1, Eth1-2).

A sample packet test is described at the following location:

/sw/nic/rudra/docs/ipsec_gw/quickstart.md#sim-packet-test

Troubleshooting
The following files in the container are useful for troubleshooting:

• Persistent logs files are located at /obfl/.

• Non-persistent log files are located at /var/log/pensando/.
• If you start the ASIC simulator in debug mode, a model.log file is generated in the non-
persistent log location (/var/log/pensando/).
• In the event of a primary application core dump, the core file is located at /data/core/.
• Technical Support files are generated at /data/techsupport/.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 23
 Appendix A: Additional Resources and Legal Notices

Appendix A

Additional Resources and Legal

Notices

Finding Additional Documentation

Documentation Portal

The AMD Documentation Hub is an online tool that provides robust search and navigation for
documentation using your web browser. To access the Documentation Hub, go to https://
www.amd.com/en/search/documentation/hub.html.

Support Resources
For support resources such as Answers, Documentation, Downloads, and Forums, see Support.

References
Pensando Documents

1. SSDK Product Brief (PPB22002)

2. Pensando DSC2-200 Data Sheet (PPB23003)

Revision History
The following table shows the revision history for this document.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 24
 Appendix A: Additional Resources and Legal Notices

Section Revision Summary

11/30/2023 Version 1.0
Initial release. N/A

Please Read: Important Legal Notices

The information presented in this document is for informational purposes only and may contain
technical inaccuracies, omissions, and typographical errors. The information contained herein is
subject to change and may be rendered inaccurate for many reasons, including but not limited to
product and roadmap changes, component and motherboard version changes, new model and/or
product releases, product differences between differing manufacturers, software changes, BIOS
flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities
that cannot be completely prevented or mitigated. AMD assumes no obligation to update or
otherwise correct or revise this information. However, AMD reserves the right to revise this
information and to make changes from time to time to the content hereof without obligation of
AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED "AS
IS." AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE
CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES,
ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY
DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR
FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY
PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL
DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF
AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Copyright

© Copyright 2023 Advanced Micro Devices, Inc. AMD, the AMD Arrow logo, Pensando, and
combinations thereof are trademarks of Advanced Micro Devices, Inc. AMBA, AMBA Designer,
Arm, ARM1176JZ-S, CoreSight, Cortex, PrimeCell, Mali, and MPCore are trademarks of Arm
Limited in the US and/or elsewhere. PCI, PCIe, and PCI Express are trademarks of PCI-SIG and
used under license. Other product names used in this publication are for identification purposes
only and may be trademarks of their respective companies.

UG1671 (v1.0) November 30, 2023

Send Feedback
AMD Pensando SSDK IPsec_GW Reference Pipeline User Guide 25