FDIO Quick Start Guide

Installing and Using FDIO

Overview 2

Configuration Utility 2

Installing FDIO 2
Installing with APT (Latest Released Version) 3
Installing with YUM (Latest Released Version) 3
Start and quickly verify VPP installation 4
Setting the Interface IP Addresses 4

CLI 5
vppctl 5
Some useful commands 6
Show interface, clear interfaces 6
Show error, Clear error 7
Show hardware, clear hardware 7
Trace packets 8
Show Fib 9

Configuration 9
Startup (startup.conf) 9
Devices 10
Miscellaneous 10
My Changes 11
Hugepages (80-vpp.conf) 11
Exec command, scripts and startup.conf 12
Using the exec command. 12
 Exec command and startup.conf 13

Downloading and Building FDIO 14

Building binaries and packages 14

Directories 15
src/vpp/conf 15
build-root/build-vpp_debug-native/dpdk/dpdk-17.02/usertools/ 16
build-root/install-vpp_debug-native/vpp/bin/ 16
dpdk-devbind.py 16
src/scripts/vnet/ 16
src/vnet/ 16
src/vnet/devices/ 17

Installing packages with dpkg 17

Install Summary 19

Uninstalling FDIO, giving the ports back 20

1 Overview
This document will describe downloading, installing and running FDIO. We will talk about the
installing FDIO, the build tree, configuration files and scripts. This isn’t meant to be a detailed
description, but more a way to quickly ramp up and understand the basics of FDIO. VPP is a
core technology in FDIO and thus referenced as VPP in many places in this document.. In this
doc commands that start with “# “ should be root and “$ “ are not.

2 Configuration Utility
The VPP configuration can be used to safely and easily configure a system for use with VPP.
Information on how to use this utility can be found at the following:

https://wiki.fd.io/view/VPP/Configuration_Tool

3 Installing FDIO
Later in this document we describe how to build FDIO and its packages. This section will
describe how VPP can be installed and run without building the packages
The following link describes how to install FDIO from the packages. I will summarize it here.

https://wiki.fd.io/view/VPP/Installing_VPP_binaries_from_packages

3.1 Installing with APT (Latest Released Version)

Create the file ​/etc/apt/sources.list.d/99fd.io.list​ with the contents shown below:

# cat /etc/apt/sources.list.d/99fd.io.list
deb [trusted=yes] https://nexus.fd.io/content/repositories/fd.io.ubuntu.xenial.main/ ./

Install the packages with the following:

# apt-get update
…..
Reading package lists...

# apt-get install vpp-lib

Reading package lists... Done
Building dependency tree
…..
# apt-get install vpp
Reading package lists... Done
Building dependency tree
….
# apt-get install vpp-plugins
Reading package lists... Done
Building dependency tree
….

3.2 Installing with YUM (Latest Released Version)

Create a file ​/etc/yum.repos.d/fdio-release.repo​ with contents shown below:

[fdio-stable-1801]
name=fd.io stable/1801 branch latest merge
baseurl=https://nexus.fd.io/content/repositories/fd.io.centos7/
enabled=1
gpgcheck=0

Install VPP:

# yum install vpp

3.3 Start and quickly verify VPP installation
When VPP is started it will take all the available interfaces. VPP should be started after the
install. VPP can also be started or restarted after installation with the service command.

Start VPP:

# service vpp start

Verify VPP is started using the ps command.

# ps -eaf | grep vpp

root 23781 1 99 14:00 ? 00:00:02 /usr/bin/vpp -c /etc/vpp/startup.conf
root 23904 11029 0 14:00 pts/1 00:00:00 grep --color=auto vpp

Verify that vpp is using some interfaces using the vppctl command. The vppctl command is used
to run the vpp cli.

# vppctl show int

Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 down
TenGigabitEthernet2/0/1 2 down
TenGigabitEthernet86/0/0 3 down
TenGigabitEthernet86/0/1 4 down
local0 0 down

3.4 Setting the Interface IP Addresses

We can then add IP addresses to the interfaces. Once this is done we can then ping these
addresses from a live network.

# vppctl set int ip address TenGigabitEthernet86/0/0 30.0.0.2/8

# vppctl set int state TenGigabitEthernet86/0/0 up
# vppctl set int ip address TenGigabitEthernet86/0/1 40.0.0.2/8
# vppctl set int state TenGigabitEthernet86/0/1 up

Verify the addresses are set “show interface address”

# vppctl show interface address

TenGigabitEthernet86/0/0 (up):
30.0.0.2/8
TenGigabitEthernet86/0/1 (up):
40.0.0.2/8
local0 (dn):

The interfaces TenGigabitEthernet86/0/0 and TenGigabitEthernet86/0/1 now have IP addresses.

You can now ping those IP addresses from another system.

$ ping 30.0.0.2
PING 30.0.0.2 (30.0.0.2) 56(84) bytes of data.
64 bytes from 30.0.0.2: icmp_seq=1 ttl=255 time=0.729 ms
64 bytes from 30.0.0.2: icmp_seq=2 ttl=255 time=0.540 ms

VPP is now installed and operational!!

4 CLI
FDIO comes with a rudimentary command line interface. The FDIO CLI is a work in progress. It
is expected to be an ongoing work and to be updated regularly. It does have command line recall
and a rudimentary help.

Although not a complete description of the CLI, a decent description can be found at the
following link.

https://wiki.fd.io/view/VPP/Command-line_Interface_(CLI)_Guide

4.1 vppctl
The usual way to access the cli is using the “vppctl”. It can be used from the linux command line
or used to start a shell where VPP commands can executed from. Use exit or ctrl-d to exit from
the shell.

Executing vppctl from the linux command line:

# vppctl show int

Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 up
TenGigabitEthernet2/0/1 2 up
TenGigabitEthernet86/0/0 3 up
TenGigabitEthernet86/0/1 4 up
local0 0 down

Excuting vppctl from the linux command line and creating a shell :
root@tf-ucs-3:/scratch/jdenisco/fdio.1704/build-root# vppctl
_______ _ _ _____ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /____(_)_/\___/ |___/_/ /_/

vpp# show int

Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 up
TenGigabitEthernet2/0/1 2 up
TenGigabitEthernet86/0/0 3 up
TenGigabitEthernet86/0/1 4 up
local0 0 down

4.2 Some useful commands

Here are a few useful VPP commands.

4.2.1 Show interface, clear interfaces

These commands show and clear the interface counters. The short form of the “show interface”
command is “show int” or “sh int”.

vpp# show interface

Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 up
TenGigabitEthernet2/0/1 2 up
TenGigabitEthernet86/0/0 3 up rx packets 4
rx bytes 354
tx packets 3
tx bytes 238
drops 1
ip4 3
TenGigabitEthernet86/0/1 4 up rx packets 8
rx bytes 746
tx packets 7
tx bytes 630
drops 1
ip4 7
local0 0 down drops 2

vpp# clear interfaces

vpp# sh int
Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 up
TenGigabitEthernet2/0/1 2 up
TenGigabitEthernet86/0/0 3 up
TenGigabitEthernet86/0/1 4 up
local0 0 down

vpp#

4.2.2 Show error, Clear error

These commands show and clear the error counters. The short form of the “show error”
command is “sh err”.

vpp# sh err
Count Node Reason
2 ip4-glean ARP requests sent
10 ip4-icmp-input echo replies sent
2 arp-input ARP replies received

vpp# show errors

Count Node Reason
2 ip4-glean ARP requests sent
10 ip4-icmp-input echo replies sent
2 arp-input ARP replies received

vpp# clear errors

vpp# sh err
Count Node Reason

4.2.3 Show hardware, clear hardware

These commands show and clear the hardware counters. The link state is shown with this
command.

vpp# show hardware

Name Idx Link Hardware
TenGigabitEthernet2/0/0 1 up TenGigabitEthernet2/0/0
Ethernet address 90:e2:ba:96:cb:98
Intel 82599
carrier up full duplex speed 10000 mtu 9216
rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
cpu socket 0
….
TenGigabitEthernet86/0/0 3 up TenGigabitEthernet86/0/0
Ethernet address 90:e2:ba:96:cd:4c
 Intel 82599
carrier up full duplex speed 10000 mtu 9216
rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
cpu socket 1

tx frames ok 3
tx bytes ok 256
….
vpp# clear hardware

vpp# show hardware

Name Idx Link Hardware
TenGigabitEthernet2/0/0 1 up TenGigabitEthernet2/0/0
Ethernet address 90:e2:ba:96:cb:98
Intel 82599
carrier up full duplex speed 10000 mtu 9216
rx queues 1, rx desc 1024, tx queues 1, tx desc 1024
cpu socket 0
….

4.2.4 Trace packets

Packets can be traced using the “trace add” and “show trace” commands.
# vppctl clear trace
# vppctl trace add dpdk-input 20
# vppctl show trace
------------------- Start of thread 0 vpp_main -------------------
No packets in trace buffer
------------------- Start of thread 1 vpp_wk_0 -------------------
No packets in trace buffer
------------------- Start of thread 2 vpp_wk_1 -------------------
No packets in trace buffer
------------------- Start of thread 3 vpp_wk_2 -------------------
No packets in trace buffer
------------------- Start of thread 4 vpp_wk_3 -------------------
No packets in trace buffer
root@tf-ucs-3:/scratch/jdenisco/quickstart# vppctl show trace
------------------- Start of thread 0 vpp_main -------------------
No packets in trace buffer
------------------- Start of thread 1 vpp_wk_0 -------------------
Packet 1

00:07:04:001708: dpdk-input
TenGigabitEthernet86/0/0 rx queue 0
buffer 0xed8bb8: current data 14, length 84, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0
PKT MBUF: port 2, nb_segs 1, pkt_len 98
buf_len 2176, data_len 98, ol_flags 0x182, data_off 128, phys_addr 0x7672ee40
packet_type 0x0
 Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
IP4: 90:e2:ba:4b:fa:70 -> 90:e2:ba:96:cd:4c
ICMP: 30.0.0.1 -> 30.0.0.2
tos 0x00, ttl 64, length 84, checksum 0x73a0
…

4.2.5 Show Fib

L2 and IP FIB can be show with the show fib commands.

# vppctl show ip fib

ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] locks:3
0.0.0.0/0
unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to:[0:0]]
[0] [@0]: dpo-drop ip4
0.0.0.0/32

unicast-ip4-chain
…

# vppctl show l2fib verbose

no l2fib entries

I am not doing any switching yet so there are no l2fib entries.

5 Configuration
There are 2 configuration files you need to know in order to configure FDIO. After FDIO is
installed these files will be in place. The files are /etc/vpp/startup.conf and
/etc/sysctl.d/80-vpp.conf.

5.1 Startup (startup.conf)

The startup.conf file is used when FDIO is started. The following sections show how to use this
file to install my system. It is always helpful to make a copy of your startup.conf before making
changes.
5.1.1 Devices
To choose which devices to be used for the FDIO data path we can use the script
“dpdk-devbind.py”. In this example, I use it to pick the interfaces enp2s0f0 and enp2s0f1. These
ports are physically looped back. I will also use enp134s0f0 and enp134s0f1. These ports are
directly connected to another system. I will make a note of the PCI IDs for these IDs then add
them to startup.conf.

root@tf-ucs-3:fdio.1# dpdk-devbind.py --status

……
Network devices using kernel driver
===================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp2s0f0 drv=ixgbe unused=
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp2s0f1 drv=ixgbe unused=
0000:08:00.0 'I350 Gigabit Network Connection' if=enp8s0f0 drv=igb unused= *Active*
0000:08:00.1 'I350 Gigabit Network Connection' if=enp8s0f1 drv=igb unused=
0000:84:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f0d1,enp132s0f0 drv=i40e unused=
0000:84:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f1d1,enp132s0f1 drv=i40e unused=
0000:86:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp134s0f0 drv=ixgbe unused=
0000:86:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp134s0f1 drv=ixgbe unused=
0000:8f:00.0 'VIC Ethernet NIC' if=enp143s0 drv=enic unused=
0000:90:00.0 'VIC Ethernet NIC' if=enp144s0 drv=enic unused=
……

I will specify these devices in the startup.conf in the section that starts “## Whitelist specific
interface by specifying PCI address”. I will specify my devices, using the PCI IDs like so.

## Whitelist specific interface by specifying PCI address

dev 0000:02:00.0
dev 0000:02:00.1
dev 0000:86:00.0
dev 0000:86:00.1

If no devices are specified VPP will take all the ports that are not link up.

5.1.2 Miscellaneous
There are a couple of other values I will change in this file. In the section “unix {“, I will add the
keyword “interactive”. This will allow us to use the FDIO cli to be enabled when starting FDIO
manually. I will also specify the driver near the section of the file “## Change UIO driver used”
to specify uio_pci_generic.
5.1.3 My Changes
My changes to the default FDIO startup.conf are as follows:

diff --git a/../fdio.1704/src/vpp/conf/startup.conf b/startup.conf

index 19e322a..3f4a723 100644
--- a/../fdio.1704/src/vpp/conf/startup.conf
+++ b/startup.conf
@@ -1,6 +1,7 @@

unix {
nodaemon
+ interactive
log /tmp/vpp.log
full-coredump
}
@@ -72,7 +73,10 @@ dpdk {
# }

## Whitelist specific interface by specifying PCI address

- # dev 0000:02:00.0
+ dev 0000:02:00.0
+ dev 0000:02:00.1
+ dev 0000:86:00.0
+ dev 0000:86:00.1

## Whitelist specific interface by specifying PCI address and in

## addition specify custom parameters for this interface
@@ -83,6 +87,7 @@ dpdk {
## Change UIO driver used by VPP, Options are: igb_uio, vfio-pci
## and uio_pci_generic (default)
# uio-driver vfio-pci
+ uio-driver uio_pci_generic

## Disable mutli-segment buffers, improves performance but

## disables Jumbo MTU support

5.2 Hugepages (80-vpp.conf)

FDIO needs to setup up Linux huge pages. The file that used to configure huge pages is
80-vpp.conf. The default number of huge pages FDIO will setup is 1024. This is a small amount
and should always work, but a more reasonable amount could be 4096. Follow the instructions in
the default example when setting these pages up. However, the instruction (2 *
vm.nr_hugepages) should probably be (2 * vm.nr_hugepages + 1024). Every example I have
seen sets vm.nr_hugepages this way and works, 2 * vm.nr_hugepages does not. The example I
have provided uses 4096 huge pages.
After changing this file execute “sysctl -p /etc/sysctl.d/80-vpp.conf”.

root@tf-ucs-3:quickstart# sysctl -p /etc/sysctl.d/80-vpp.conf

vm.nr_hugepages = 4096
vm.max_map_count = 9216
vm.hugetlb_shm_group = 0
kernel.shmmax = 8589934592

Verify the huge pages are setup using meminfo.

root@tf-ucs-3:quickstart# cat /proc/meminfo

MemTotal: 65863648 kB
MemFree: 50173284 kB
…..
HugePages_Total: 4096
HugePages_Free: 4096
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

In some cases, for example if vpp or dpdk crashes, the huge pages may not get cleaned up. You
will know this because HugePages Free will be 0. If this happens they need to be deleted. You
can do this by finding where they are and then deleting them.

root@tf-ucs-3:/# rm /dev/hugepages/rtmem*

5.3 Exec command, scripts and startup.conf

The “exec” command will execute VPP commands that have been written to file. In this
example, I will set up my interfaces.

5.3.1 Using the exec command.

My file (script) looks like this:

# cat set-up-interfaces
comment { connected to TenGigabitEthernet2/0/1 }
set int ip address TenGigabitEthernet2/0/0 10.0.0.2/8
set int state TenGigabitEthernet2/0/0 up

comment { connected to TenGigabitEthernet2/0/0 }

set int ip address TenGigabitEthernet2/0/1 20.0.0.2/8
set int state TenGigabitEthernet2/0/1 up

comment { connected to a system outside the box }

set int ip address TenGigabitEthernet86/0/0 30.0.0.2/8
set int state TenGigabitEthernet86/0/0 up

comment { connected to a system outside the box }

set int ip address TenGigabitEthernet86/0/1 40.0.0.2/8
set int state TenGigabitEthernet86/0/1 up

From the VPP shell I will execute the command.

# vppctl
_______ _ _ _____ ___
__/ __/ _ \ (_)__ | | / / _ \/ _ \
_/ _// // / / / _ \ | |/ / ___/ ___/
/_/ /____(_)_/\___/ |___/_/ /_/

vpp# exec /scratch/jdenisco/quickstart/set-up-interfaces

vpp# sh int addr
TenGigabitEthernet2/0/0 (up):
10.0.0.2/8
TenGigabitEthernet2/0/1 (up):
20.0.0.2/8
TenGigabitEthernet86/0/0 (up):
30.0.0.2/8
TenGigabitEthernet86/0/1 (up):
40.0.0.2/8
local0 (dn):

vpp# exit

Notice the “show interface address” command show IP addresses associated with the interfaces.

5.4 Exec command and startup.conf

In startup.conf you can use the exec command, to set up your initial configuration. In this case in
startup.conf I have this.

unix {
nodaemon
interactive
log /tmp/vpp.log
full-coredump
exec /scratch/jdenisco/quickstart/set-up-interfaces
}
 6 Downloading and Building FDIO
It is only necessary to download and build FDIO if you are interested in Developing or
debugging FDIO.

First get FDIO:

# git clone -b master ​https://gerrit.fd.io/r/vpp fdio.1704

7 Building binaries and packages

All of the FDIO binaries and package can be built from the root directory, in this case fdio.1704.
However, I will build the actual binaries and packages from the build-root directory. You have a
little bit more flexibility from here.

After downloading FDIO the first time and before building the binaries and packages you need
to install the dependencies and bootstrap the build environment. These 2 steps only need to be
done once per tree.

# make install-dep
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Ign:3 https://nexus.fd.io/content/repositories/fd.io.stable.1701.ubuntu.xenial.main ./ InRelease
…….
# make bootstrap
make[1]: Entering directory 'fdio.1704/build-root'
@@@@ Arch for platform 'native' is native @@@@
@@@@ Finding source for tools @@@@
@@@@ Makefile fragment found in fdio.1704/build-root/packages/tools.mk @@@@
@@@@ Source found in fdio.1704/src @@@@
…….

After setting up and bootstrapping the build environment, using the steps above, the binaries and
packages can be built with the following:

Change to the build-root directory, then execute path_setup.

# cd build-root
# source ./path_setup

The vpp binaries can built using the make command in the build-root directory. To build images
with debug symbols use the following:
# make PLATFORM=vpp TAG=vpp_debug vpp-install
@@@@ Arch for platform 'vpp' is native @@@@
@@@@ Finding source for dpdk @@@@
@@@@ Makefile fragment found in fdio.1704/build-data/packages/dpdk.mk @@@@
@@@@ Source found in fdio.1704/dpdk @@@@
@@@@ Arch for platform 'vpp' is native @@@@
@@@@ Finding source for vpp @@@@
…..

This does the same as executing “make debug” from the root directory (fdio.1704). To build an
image without symbols use “TAG=vpp” instead of “TAG=vpp_debug”. “make release” in the
root directory builds the binaries without debug symbols.

To build the debian packages execute the following:

# make PLATFORM=vpp TAG=vpp_debug install-deb

make[1]: Entering directory 'fdio.1704/build-root'
@@@@ Arch for platform 'vpp' is native @@@@
@@@@ Finding source for dpdk @@@@
@@@@ Makefile fragment found in fdio.1704/build-data/packages/dpdk.mk @@@@
@@@@ Source found in fdio.1704/dpdk @@@@
@@@@ Arch for platform 'vpp' is native @@@@
@@@@ Finding source for vpp @@@@
…..

The packages will be found in the build-root directory.

# ls *.deb
vpp_17.04-rc2~1-g5e189f1_amd64.deb vpp-api-python_17.04-rc2~1-g5e189f1_amd64.deb
vpp-lib_17.04-rc2~1-g5e189f1_amd64.deb vpp-api-java_17.04-rc2~1-g5e189f1_amd64.deb
vpp-dbg_17.04-rc2~1-g5e189f1_amd64.deb vpp-plugins_17.04-rc2~1-g5e189f1_amd64.deb
vpp-api-lua_17.04-rc2~1-g5e189f1_amd64.deb vpp-dev_17.04-rc2~1-g5e189f1_amd64.deb

8 Directories
After pulling and building FDIO there are a few directories worth looking at.

8.1 src/vpp/conf
This directory contains default configuration files.

# ls
80-vpp.conf startup.conf
8.2 build-root/build-vpp_debug-native/dpdk/dpdk-17.02/usertools/
This directory is provided with the dpdk. The 2 import scripts are cpu_layout.py and
dpdk-devbind.py

# ls ./build-root/build-vpp_debug-native/dpdk/dpdk-17.02/usertools/
cpu_layout.py dpdk-devbind.py dpdk-pmdinfo.py dpdk-setup.sh

8.3 build-root/install-vpp_debug-native/vpp/bin/
This directory contains the vpp executables. The most useful files are vpp and vppctl. These files
are copied to/usr/bin after FDIO is installed. You can use the binary file “vpp” located in this
directory with gdb to help debug FDIO.

root@tf-ucs-3# ls ./build-root/install-vpp_debug-native/vpp/bin
elftool svmdbtool svmtool vpp vppapigen vpp_api_test vppctl vpp_get_metrics vpp_json_test
vpp_restart

8.3.1 dpdk-devbind.py
The dpdk-devbind.py script is provided with the Intel DPDK. It is included with FDIO. After the
FDIO is built, this script and other DPDK tools can be found in
build-root/build-vpp_debug-native/dpdk/dpdk-17.02/usertools/.

8.4 src/scripts/vnet/
This directory has some very useful examples using the FDIO traffic generator and general
configuration.

# ls src/scripts/vnet/
arp4 dhcp ip6 l2efpfilter_perf l2flood mcast pcap rightpeer snat_det
…..

8.5 src/vnet/
This directory contains most of the important source code.

# ls src/vnet
adj config.h fib interface.api interface_output.c lawful-intercept misc.c
ppp
….
8.6 src/vnet/devices/
This directory contains the device drivers. For example, the vhost driver is in
src/vnet/devices/virtio.

# ls src/vnet/devices/virtio/
dir.dox vhost_user.api vhost_user_api.c vhost-user.c vhost-user.h

9 Installing packages with dpkg

We are now ready to install the packages. In the previous steps, we built the debian packages.
They can be found in the build-root directory.

# ls *.deb
vpp_17.04-rc2~1-g5e189f1_amd64.deb vpp-api-python_17.04-rc2~1-g5e189f1_amd64.deb
vpp-lib_17.04-rc2~1-g5e189f1_amd64.deb vpp-api-java_17.04-rc2~1-g5e189f1_amd64.deb
vpp-dbg_17.04-rc2~1-g5e189f1_amd64.deb vpp-plugins_17.04-rc2~1-g5e189f1_amd64.deb
vpp-api-lua_17.04-rc2~1-g5e189f1_amd64.deb vpp-dev_17.04-rc2~1-g5e189f1_amd64.deb

We will install the packages vpp-lib, vpp-plugins, vpp, and vpp-api-python. We will install them
in this order so the dependencies work. I have also inserted some faults so we can walk through a
typical debug sequence.

Install the VPP libraries

# dpkg -i *.deb
Selecting previously unselected package vpp-lib.
(Reading database ... 140058 files and directories currently installed.)
Preparing to unpack .../vpp-lib_17.04-rc2~1-g5e189f1_amd64.deb ...
….

After installing VPP it should be started. I will check this using the ps command.

# ps -eaf | grep vpp

root 29365 11029 0 09:56 pts/1 00:00:00 grep --color=auto vpp

In this case VPP is not running. To debug I will run VPP manually. You can also check error
messages using “service vpp status”

# /usr/bin/vpp -c /etc/vpp/startup.conf
vlib_plugin_early_init:360: plugin path /usr/lib/vpp_plugins
load_one_plugin:188: Loaded plugin: acl_plugin.so (Access Control Lists)
….
0: load_one_plugin:63: Loaded plugin: /usr/lib/vpp_api_test_plugins/ioam_export_test_plugin.so
0: load_one_plugin:63: Loaded plugin: /usr/lib/vpp_api_test_plugins/ioam_vxlan_gpe_test_plugin.so
0: dpdk_config: unknown input `uio_pci_generic'
……

VPP once again did not start. In this case, I have an incorrect statement in my startup.conf.
Notice the error “​dpdk_config: unknown input `uio_pci_generic'”.​ I will fix it and try again.

# /usr/bin/vpp -c /etc/vpp/startup.conf
vlib_plugin_early_init:360: plugin path /usr/lib/vpp_plugins
load_one_plugin:188: Loaded plugin: acl_plugin.so (Access Control Lists)
….
0: dpdk_ipsec_process:241: DPDK Cryptodev support is disabled, default to OpenSSL IPsec
0: dpdk_lib_init:536: DPDK drivers found no ports...
0: dpdk_lib_init:540: DPDK drivers found 0 ports...

This looks better and I get to the prompt. The “interactive” keyword we specified in the
startup.conf enables this prompt. However, I do not see any interfaces when I type “sh int” (show
interfaces).

DBGvpp# sh int
Name Idx State Counter Count
local0 0 down

This usually means the driver did not load. Notice “​0: dpdk_lib_init:540: DPDK drivers found 0
ports...”. ​In this case, I did not load the uio_pci_driver. I will do that then try again.

# modprobe uio_pci_generic
# /usr/bin/vpp -c /etc/vpp/startup.conf
vlib_plugin_early_init:360: plugin path /usr/lib/vpp_plugins
load_one_plugin:188: Loaded plugin: acl_plugin.so (Access Control Lists)
load_one_plugin:188: Loaded plugin: dpdk_plugin.so (Data Plane Development Kit (DPDK))

…….
0: dpdk_ipsec_process:241: DPDK Cryptodev support is disabled, default to OpenSSL IPsec
0: dpdk_lib_init:540: DPDK drivers found 4 ports...
……
DBGvpp# sh int
Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 down
TenGigabitEthernet2/0/1 2 down
TenGigabitEthernet86/0/0 3 down
TenGigabitEthernet86/0/1 4 down
local0 0 down
DBGvpp# quit
This looks good. Now I will quit and will start and check VPP with the service command. vppctl
is a command that allows you to run VPP cli from the user prompt.

# service vpp start

# ps -eaf | grep vpp
root 534 1 99 12:19 ? 00:00:01 /usr/bin/vpp -c /etc/vpp/startup.conf
root 542 11029 0 12:19 pts/1 00:00:00 grep --color=auto vpp
# vppctl show int
Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 down
TenGigabitEthernet2/0/1 2 down
TenGigabitEthernet86/0/0 3 down
TenGigabitEthernet86/0/1 4 down
local0 0 down

9.1 Install Summary

To summarize, if my configuration is all correct this is how I would install the FDIO packages.

# modprobe uio_pci_generic
# dpkg -i vpp-lib_17.04-rc2~1-g5e189f1_amd64.deb
Selecting previously unselected package vpp-lib.
(Reading database ... 139943 files and directories currently installed.)
…..
# dpkg -i vpp_17.04-rc2~1-g5e189f1_amd64.deb
Selecting previously unselected package vpp.
(Reading database ... 139983 files and directories currently installed.)
Preparing to unpack vpp_17.04-rc2~1-g5e189f1_amd64.deb ...
Unpacking vpp (17.04-rc2~1-g5e189f1) ...
….
# dpkg -i vpp-plugins_17.04-rc2~1-g5e189f1_amd64.deb
Selecting previously unselected package vpp-plugins.
(Reading database ... 140032 files and directories currently installed.)
Preparing to unpack vpp-plugins_17.04-rc2~1-g5e189f1_amd64.deb ...
Unpacking vpp-plugins (17.04-rc2~1-g5e189f1) ...
Setting up vpp-plugins (17.04-rc2~1-g5e189f1) ...
# service vpp start
# ps -eaf | grep vpp
root 1194 1 96 12:27 ? 00:00:04 /usr/bin/vpp -c /etc/vpp/startup.conf
root 1204 11029 0 12:27 pts/1 00:00:00 grep --color=auto vpp
# vppctl show int
Name Idx State Counter Count
TenGigabitEthernet2/0/0 1 down
TenGigabitEthernet2/0/1 2 down
TenGigabitEthernet86/0/0 3 down
TenGigabitEthernet86/0/1 4 down
local0 0 down

10Uninstalling FDIO, giving the ports back

When VPP is installed VPP will take over the ports that it is using. We can see this using
dpdk-devbind.py. Here we see that VPP is using ports 0000:02:00.0, 0000:02:00.1,
0000:86:00.0, and 0000:86:00.1.

# ./dpdk-devbind.py --status

Network devices using DPDK-compatible driver

============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe
0000:86:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe
0000:86:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=uio_pci_generic unused=ixgbe

Network devices using kernel driver

===================================
0000:08:00.0 'I350 Gigabit Network Connection' if=enp8s0f0 drv=igb unused=uio_pci_generic *Active*
0000:08:00.1 'I350 Gigabit Network Connection' if=enp8s0f1 drv=igb unused=uio_pci_generic
0000:84:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f0d1,enp132s0f0 drv=i40e
unused=uio_pci_generic
0000:84:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f1d1,enp132s0f1 drv=i40e
unused=uio_pci_generic
0000:8f:00.0 'VIC Ethernet NIC' if=enp143s0 drv=enic unused=uio_pci_generic
0000:90:00.0 'VIC Ethernet NIC' if=enp144s0 drv=enic unused=uio_pci_generic

Other network devices

=====================
<none>
…..

We will now uninstall VPP using apt. If we had installed the packages with dpdk we would use
“dpkg –r” to remove the packages.

# dpkg -l | grep vpp

ii vpp 17.04-rc2~3-g2594216~b29 amd64 Vector
Packet Processing--executables
ii vpp-lib 17.04-rc2~3-g2594216~b29 amd64 Vector
Packet Processing--runtime libraries
ii vpp-plugins 17.04-rc2~3-g2594216~b29 amd64 Vector
Packet Processing--runtime plugins
# apt remove vpp-plugins
Reading package lists... Done
Building dependency tree
…..
# apt remove vpp
Reading package lists... Done
Building dependency tree
…..
# apt remove vpp-lib
Reading package lists... Done
…..

With the packages removed we still see that the system does not have the ports back.

# ./dpdk-devbind.py --status

Network devices using DPDK-compatible driver

============================================
<none>

Network devices using kernel driver

===================================
0000:08:00.0 'I350 Gigabit Network Connection' if=enp8s0f0 drv=igb unused=uio_pci_generic *Active*
0000:08:00.1 'I350 Gigabit Network Connection' if=enp8s0f1 drv=igb unused=uio_pci_generic
0000:84:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f0d1,enp132s0f0 drv=i40e
unused=uio_pci_generic
0000:84:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f1d1,enp132s0f1 drv=i40e
unused=uio_pci_generic
0000:8f:00.0 'VIC Ethernet NIC' if=enp143s0 drv=enic unused=uio_pci_generic
0000:90:00.0 'VIC Ethernet NIC' if=enp144s0 drv=enic unused=uio_pci_generic

Other network devices

=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,uio_pci_generic
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,uio_pci_generic
0000:86:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,uio_pci_generic
0000:86:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=ixgbe,uio_pci_generic
…..

We give the ports back to the kernel using the –b option like so:

# dpdk-devbind.py -b ixgbe 0000:02:00.0 0000:02:00.1 0000:86:00.0 0000:86:00.1

# ./dpdk-devbind.py --status

Network devices using DPDK-compatible driver

============================================
<none>

Network devices using kernel driver

===================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp2s0f0 drv=ixgbe unused=uio_pci_generic
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp2s0f1 drv=ixgbe unused=uio_pci_generic
0000:08:00.0 'I350 Gigabit Network Connection' if=enp8s0f0 drv=igb unused=uio_pci_generic *Active*
0000:08:00.1 'I350 Gigabit Network Connection' if=enp8s0f1 drv=igb unused=uio_pci_generic
0000:84:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f0d1,enp132s0f0 drv=i40e
unused=uio_pci_generic
0000:84:00.1 'Ethernet Controller XL710 for 40GbE QSFP+' if=enp132s0f1d1,enp132s0f1 drv=i40e
unused=uio_pci_generic
0000:86:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp134s0f0 drv=ixgbe
unused=uio_pci_generic
0000:86:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' if=enp134s0f1 drv=ixgbe
unused=uio_pci_generic
0000:8f:00.0 'VIC Ethernet NIC' if=enp143s0 drv=enic unused=uio_pci_generic
0000:90:00.0 'VIC Ethernet NIC' if=enp144s0 drv=enic unused=uio_pci_generic

Other network devices

=====================
<none>
…..