sensors

Article
Enhancing Mitigation of Volumetric DDoS Attacks: A Hybrid
FPGA/Software Filtering Datapath
Denis Salopek * and Miljenko Mikuc

Faculty of Electrical Engineering and Computing, University of Zagreb, 10000 Zagreb, Croatia;
[email protected]
* Correspondence: [email protected]

Abstract: The increasing network speeds of today’s Internet require high-performance, high-
throughput network devices. However, the lack of affordable, flexible, and readily available devices
poses a challenge for packet classification and filtering. This problem is exacerbated by the increase
in volumetric Distributed Denial-of-Service (DDoS) attacks, which require efficient packet processing
and filtering. To meet the demands of high-speed networks and configurable network processing
devices, this paper investigates a hybrid hardware/software packet filter prototype that combines
reconfigurable FPGA technology and high-speed software filtering on commodity hardware. It
uses a novel approach that offloads filtering rules to the hardware and employs a Longest Prefix
Matching (LPM) algorithm and allowlists/blocklists based on millions of IP prefixes. The hybrid filter
demonstrates improvements over software-only filtering, achieving performance gains of nearly 30%,
depending on the rulesets, offloading methods, and traffic types. The significance of this research
lies in developing a cost-effective alternative to more-expensive or less-effective filters, providing
high-speed DDoS packet filtering for IPv4 traffic, as it still dominates over IPv6. Deploying these
filters on commodity hardware at the edge of the network can mitigate the impact of DDoS attacks
on protected networks, enhancing the security of all devices on the network, including Internet of
Things (IoT) devices.

Keywords: hybrid filters; DDoS mitigation; low power; FPGA; hardware/software packet processors;
high performance

Citation: Salopek, D.; Mikuc, M.

Enhancing Mitigation of Volumetric
DDoS Attacks: A Hybrid
1. Introduction
FPGA/Software Filtering Datapath.
Sensors 2023, 23, 7636. https:// The rapid growth of the Internet, amplified by the proliferation of Internet of Things
doi.org/10.3390/s23177636 (IoT) devices and coupled with increasing network speeds, requires the deployment of high-
performance, high-throughput network devices that prioritize energy-efficient solutions. In
Academic Editor: Xiaojie Wang
the context of IoT environments, the challenge lies in developing efficient mechanisms that
Received: 29 July 2023 can process packets rapidly while optimizing energy usage to meet their energy-constrained
Revised: 25 August 2023 demands, even at the network edge.
Accepted: 1 September 2023 Packet processing in high-throughput networks is primarily the task of specialized
Published: 3 September 2023 hardware-based network devices that can quickly classify and filter packets, albeit with
limitations. Performing this task programmatically on a 100 Gbps network requires packet
filtering that can handle a throughput of over 148 million packets per second (Mpps). On a
general-purpose computer with a processor speed of 4 GHz, this would mean that each
Copyright: © 2023 by the authors.
packet should be processed in less than 27 clock cycles.
Licensee MDPI, Basel, Switzerland.
Depending on the type of device, the classification of packets is based on certain
This article is an open access article
criteria, such as searching by destination MAC address or the VLAN tag in switches, or
distributed under the terms and
the source/destination IP address in routers and firewalls. Performing such checks within
conditions of the Creative Commons
Attribution (CC BY) license (https://
the required 27 clock cycles is a major challenge, as multiple operations or memory fetches
creativecommons.org/licenses/by/
must be completed within this time frame. The 27 clock cycle example applies to the
4.0/). smallest frame size traffic, when the device is at its busiest due to the shortest packet arrival

Sensors 2023, 23, 7636. https://doi.org/10.3390/s23177636 https://www.mdpi.com/journal/sensors

Sensors 2023, 23, 7636 2 of 19

times. Regardless of this, devices in the network infrastructure must be able to handle
packets of all sizes at all speeds. Otherwise, if one device fails, its availability can no longer
be guaranteed.
Malicious users on the Internet exploit this fact and attempt to disable access to certain
services through Denial of Service (DoS) attacks and Distributed Denial of Service (DDoS)
attacks. According to various sources [1–3], DDoS attacks are becoming increasingly
common. In these attacks, infected computers under the control of attackers (called bots,
often insecure computers or IoT devices) send traffic to the victim, consuming resources
and disrupting normal users. Defending against such attacks is very difficult, as there may
be millions of these infected devices. Distinguishing the ‘good’ traffic from the ‘bad’ and
at the same time filtering it out is particularly problematic when dealing with very high
network speeds.
Despite current efforts to replace IPv4 with IPv6 in response to IPv4 address exhaus-
tion, there is no clear indication that this transition will occur in the near future. Since
the percentage of total IPv6 traffic remains lower [4], the occurrence of IPv6 DDoS attacks
is also relatively limited. Therefore, this paper focuses exclusively on IPv4 traffic and
the mitigation of IPv4 DDoS attacks, as it is expected that IPv4 will still be in use for an
extended period of time.
It is important to differentiate between DDoS protection (mitigation) and DDoS detec-
tion (recognition). DDoS mitigation systems may include DDoS detection, but this is not
universally the case. This paper focuses primarily on packet filtering and the mitigation of
detected DDoS attacks, assuming that other systems are responsible for the automatic or
non-automatic task of DDoS detection.
The work is organized as follows. Section 2 gives an overview of related work in the
protection against DDoS attacks. Section 3 describes a model of a hybrid hardware/software
datapath used for high-speed packet filtering. Section 4 explains the benchmark method-
ology and showcases the results of the hybrid filter compared to the software-only filter.
Finally, Section 5 provides the conclusion.

2. Related Work
The current state of protection against DDoS attacks relies on using one of three types
of protection approaches: third-party delegation, on-site infrastructure protection, or a
combination of both. Third-party delegation routes all traffic to a DDoS protection service,
which then “scrubs” the dangerous and suspicious traffic as needed and redirects legitimate
traffic to its destination. However, this redirection of traffic raises potential issues if the
traffic is sensitive to even minor delays or contains sensitive and private information that
third parties should not have access to (e.g., in the financial sector).
On-site protection is achieved by devices capable of filtering traffic using specialized
hardware, software, or a hybrid of both. Hardware-based filtering is performed by devices
designed specifically for this purpose. These devices offer high throughput, but come
with high annual licensing costs for the associated software. Apart from the cost, negative
aspects of such devices include inflexibility and complexity when it comes to modifications
or updates [5]; so, after a few years, they no longer meet the requirements of current
network speeds. In addition, the use of primarily TCAM (ternary content-addressable
memory) technology in these devices contributes to their high power consumption [5,6],
which exacerbates their disadvantages. Other technologies used for this type of filtering
include ASIC (application-specific integrated circuit) and FPGAs (field-programmable gate
arrays), with ASIC having similar disadvantages to TCAM, including the high price of its
development, but FPGAs standing out from both of them by being reprogrammable.
Software frameworks for fast packet processing on general-purpose computers have
emerged in recent years. These frameworks, such as Netmap [7], DPDK [8], and XDP/
eBPF [9], when combined with sufficiently adequate hardware, can achieve packet process-
ing results comparable to hardware filters. They provide flexibility and control over the
filters created with them because they are simpler and programmable compared to most
Sensors 2023, 23, 7636 3 of 19

hardware-based systems. Software-defined networking (SDN) using OpenFlow [10] and

content delivery networks (CDNs) [11] have also been explored, but encounter architectural
limitations [12] for the mitigation of large volumetric DDoS attacks [13,14].
Hybrid protection combines hardware- and software-based protection, often utilizing
some form of hardware to partially (or fully) handle the filtering process and “offload”
software-based filtering, which is expected to have lower performance. This is why hy-
brid hardware/software solutions combining software with non-expensive, off-the-shelf
hardware (e.g., FPGAs [15–20], GPUs [21–24], or smart NICs [25–27]) offer a flexible and
cost-effective approach.
In [15], packet processing is performed using eBPF, with its implementation running
entirely on FPGAs, while [16] describes a framework for offloading click router functionality
that also runs entirely on FPGAs. In these systems, the CPU is primarily used for system
preparation and transport to the hardware component, and so there is no “hybridity”
in the packet processing itself. However, in [17,20], offloading is achieved by moving a
limited number of filtering rules to the FPGA, while the remaining rules are executed on
the host machine using the Linux firewall. This reduces the load on the CPU due to the
smaller number of rules on the host. The work of [18] and its improved version [19] achieve
offloading in a similar manner, but for higher traffic throughput, along with improvements
related to storing rules on the FPGA. The filtering method in all of these mentioned
systems has limitations on the number of rules that can be offloaded, depending on the
implementation and the resources available on the FPGA. This makes them inconvenient
for defending against large volumetric DDoS attacks. In addition, modifying rules on the
FPGA requires re-synthesizing the bitfile in the worst case, which is time-consuming and
does not provide a fast defense in case of a DDoS attack.
In [21], the authors present ways to improve CPU packet processing using some
principles of GPU processing (e.g., latency hiding). This work’s actual contribution is that
the CPU is capable of processing packets faster than the GPU in certain cases, without
the negative consequences associated with GPUs. The systems in [22–24] perform packet
operations entirely on the GPU, i.e., they utilize the parallelism of the GPU in different ways
to process a large number of packets simultaneously. In this way, the GPU effectively acts
as a large number of CPUs. A similar mode of operation is found in [25], where all “heavy”
packet processing is conducted on the GPU, while simpler tasks are left for processing on
the CPU. All GPU-assisted hybrid solutions face latency issues due to the batch processing
of packets and may introduce packet re-ordering. They are also not compatible with all
GPUs and are less energy-efficient than FPGAs.
The system in [26], like some of those previously mentioned, involves transferring
a certain number of rules to the (memory-limited) SmartNIC, while leaving the rest on
the host computer. An approach to packet processing similar to our paper can be found
in [27,28]. They also utilize some method of preprocessing of packets (generating metadata)
that takes place before the actual packet processing on the CPU. However, these papers
do not address cases involving defense against large volumetric DDoS attacks and the
challenges associated with them. Instead, they use SmartNICs for packet preprocessing in
conjunction with other applications (e.g., key–value stores or GRE termination).
Protection against DDoS attacks is approached in a similar way to a standard firewall,
where rulelists are created with various fields to be checked (e.g., source or destination IP
address, transport layer protocol, or ports) and each packet traverses these lists, comparing
its header against the specified fields. In some cases, attempts are made to minimize these
rulelists, or Packet Classification Engines (PCEs) are used to achieve a filtering method
that requires minimal memory fetches. However, these tools work under the premise that
protection against DDoS attacks is only possible with a large number of separate rules,
requiring tens of thousands or even millions of such records.
To assess the effectiveness of DDoS filtering solutions, it is important to consider the
type and volume of traffic used in testing. Any solution should be exposed to traffic that
mirrors real DDoS attacks. A simulation of the attack can be achieved by flooding the
Sensors 2023, 23, 7636 4 of 19

filter with synthetic traffic containing randomly generated IP addresses or using existing
traces of actual DDoS attacks. Some vendors [29,30] utilize genuine DDoS traffic, as they
have access to extensive real-world data and traffic with various DDoS attack scenarios.
However, since most researchers do not have access to such data, they resort to synthetic
traffic to simulate DDoS attacks in their tests [18,19,26,31,32]. Synthetic traffic with a large
pool of randomly generated IP addresses can approximate DDoS attacks, but in previously
mentioned works, this is limited to tens of thousands of IP addresses and is therefore
not capable of replicating the scale of today’s volumetric DDoS attacks. For example, the
attack on Dyn in 2016 involved tens of millions of different IP addresses, as shown through
various analyses [33–35]. Therefore, any effective DDoS filtering system must be able to
withstand such large-scale attacks.
In addition to the “active” defenses against DDoS attacks mentioned above, it is
worth mentioning so-called blackhole routing. In this method, the victim’s IP address is
reported to the network service provider, which then redirects all traffic destined for that IP
address to a “black hole”, effectively discarding it. This protects the rest of the network by
saving bandwidth by eliminating a significant portion of malicious traffic, while effectively
fulfilling the attackers’ goal by rendering the victim inaccessible to other users.

3. Hybrid System Model

This paper, based on the research from one of the author’s previous work [36], builds
upon our previous studies [37] which aimed to enhance existing software-based filtering to
protect against volumetric DDoS attacks by replacing large rulelists with more compact
ones. At the same time, additional tables are used to store IP addresses or subnets (e.g.,
whitelists or blacklists), and so these can be retrieved much faster using Longest Prefix
Matching (LPM). In this paper, we propose a hybrid filtering system using a combination
of FPGA hardware and software support based on the aforementioned LPM filtering. This
moves away from the paradigm that defense against DDoS attacks requires the maintenance
of monolithic lists with complex rules.
The packets to be filtered first pass through the hardware, which parses them and,
if necessary, generates metadata that are passed to the software filter. The software filter
receives the packets that are then metadata-enriched and, depending on its programming,
parses them and performs the required actions which are determined from the ruleset given
to the system, explained later in this chapter.
When using allowlists/blocklists for filtering, the LPM algorithm plays a crucial
role in extracting data from each list to determine whether the IP address being checked
belongs to a specific list. Various LPM algorithms can be used for this task, but the
current implementation of the filter uses the DXR algorithm from [38,39], specifically the
D16X4R version.
The DXR algorithm compresses and stores a list in compact structures with a small
memory footprint. The algorithm consists of three stages (D16, X4, and R), all originally
executed in software. However, in the hybrid implementation, a modified version of the
algorithm is used. The first two stages (D16 and X4—indexing and memory retrieval using
a total of 20 bits) are separated from the last one and executed in hardware. Then, the
hardware passes the result of these two stages (the 32-bit index and a range included in the
metadata of the packet) to the software. In the last stage of the algorithm, a binary search is
performed over the received range (R—range lookup) until it reaches its end and returns
the final result representing the “next-hop” for a given LPM table or a null value, indicating
no match.
To select the best ruleset/metadata combination depending on pre-specified assump-
tions, parameters, and factors, we heuristically modeled a distributor part of the hybrid
system. It acts as an intermediary and determines how packets are parsed in hardware,
what metadata are created, and which filter rules are appropriate for hardware offload-
ing. It takes into account various parameters such as the ruleset, hardware, and software
capabilities, network status, and traffic volume.
Sensors 2023, 23, 7636 5 of 19

what metadata are created, and which filter rules are appropriate for hardware offloading.
Sensors 2023, 23, 7636 It takes into account various parameters such as the ruleset, hardware, and software ca-
5 of 19
pabilities, network status, and traffic volume.

3.1. Hardware
3.1. Hardware
FPGA technology was chosen for the hardware component due to its flexibility in
FPGA technology was chosen for the hardware component due to its flexibility in terms
terms of reconfiguration and the utilization of parallelism. This also made the FPGA a
of reconfiguration and the utilization of parallelism. This also made the FPGA a promising
promising technology to improve the efficiency of packet processing in IoT environments
technology to improve the efficiency of packet processing in IoT environments [40]. NetF-
[40]. NetFPGA SUME [41] is a development board for prototyping network functions for
PGA SUME [41] is a development board for prototyping network functions for high-speed
high-speed networks that has been used extensively in various research projects since 2015
networks that has been used extensively in various research projects since 2015 [42–47]. It
[42–47]. It provides prototyping capabilities for such a filter at 10 G network speeds and
provides prototyping capabilities for such a filter at 10 G network speeds and features a
features a Xilinx Virtex-7 690T FPGA, four 10 GbE SFP+ interfaces, QDR II SRAM memory
Xilinx Virtex-7 690T FPGA, four 10 GbE SFP+ interfaces, QDR II SRAM memory modules,
modules, DDR3 SODIMM
DDR3 SODIMM memory modules,
memory modules, and otherand other peripherals.
peripherals.
The
The primary
primary idea
idea for
for our
our model
model was
wasto to configure
configureNetFPGA
NetFPGASUMESUMEas asan
anNIC
NIC onon top
top
of which the software filter would be deployed, and so it would serve
of which the software filter would be deployed, and so it would serve as a standaloneas a standalone
network
networkmiddleware
middlewareelement
elementinstalled
installedatatthe
theedge
edgeof
ofthe
thenetwork,
network,asasshown
shownin inFigure
Figure1.1.
The
The DMA engine used in the existing NetFPGA SUME Reference NIC project [48] was
DMA engine used in the existing NetFPGA SUME Reference NIC project [48] was not
not
designed
designed to fully utilize the PCIe bus, and so the bandwidth between the NetFPGASUME
to fully utilize the PCIe bus, and so the bandwidth between the NetFPGA SUME
card and the
card and theoperating
operatingsystem
systemwas was poor.
poor. Attempting
Attempting to create
to create and implement
and implement an im-
an improved
proved version which would work in a high-speed environment on the existing
version which would work in a high-speed environment on the existing hardware proved hardware
proved impossible
impossible withoutwithout significant
significant and complex
and complex modifications.
modifications.

Figure 1.
Figure Architectureof
1. Architecture of the
the proposed
proposed DDoS
DDoS filtering
filtering system
system using
using NetFPGA
NetFPGASUME
SUMENIC.NIC. Regular
Regular
arrows
arrows represent
represent “real”
“real” packet
packet datapaths,
datapaths, dashed
dashed arrows
arrows represent
represent combined
combined “real”
“real” packet
packet and
and
metadata
metadata datapaths, andand dotted
dotted arrows
arrowsrepresent
representinternal
internalcommunication
communication between
between different
different mod-
modules
ules of system.
of the the system.

Therefore,
Therefore, the model was was modified
modifiedto tono
nolonger
longeruseusethe
thePCIe
PCIe bus
bus forfor packet
packet trans-
transmis-
mission
sion andand to exclusively
to exclusively useuse Ethernet
Ethernet communication
communication between
between the the
FPGAFPGAandand
the the soft-
software
ware
filter. filter.
Packets Packets and metadata
and metadata are passed
are passed from thefromFPGAthetoFPGA to the software
the software filter via filter via
Ethernet,
Ethernet,
achievingachieving
sufficientlysufficiently
high speedshighforspeeds
use infor
10 use in 10 G networks.
G networks. The model, Theasmodel,
shown as in
Figure in
shown 2, Figure
uses an2,additional NIC to receive
uses an additional NIC topackets
receive with
packetsmetadata and forwards
with metadata them to
and forwards
the software
them filter. The
to the software NetFPGA
filter. SUME performs
The NetFPGA the necessary
SUME performs offload offload
the necessary and pre-filtering
and pre-
tasks, but
filtering the but
tasks, automatic forwarding
the automatic of packets
forwarding to the egress
of packets interface
to the egress from hardware
interface from hard- is
disabled.
ware The model
is disabled. Thecan be extended
model by enabling
can be extended additional
by enabling interfaceinterface
additional pairs to increase the
pairs to in-
overallthe
crease throughput.
overall throughput.
The hardware implementation demonstrates the hybrid filter prototype datapath on
the NetFPGA SUME development board, using the AXI4-Stream protocol for inter-module
communication in the system pipeline. The pipeline is composed of modules connected
in series or parallel and consists of two parts: one part carries packets arriving from the
incoming network interface, i.e., packets that are checked (filtered) and forwarded to the
 Sensors 2023, 23, 7636 6 of 19

output interface if necessary, and control packets from the “Distributor” that regulate the
internal logic within the hardware (e.g., setting memory values or enabling and disabling
Sensors 2023, 23, 7636 certain parsers). All data required for filtering (e.g., source and destination IP addresses)
6 of 19
are extracted from the “real” packets, and metadata are created based on this information.
The metadata are appended to the end of the packets and forwarded to the software filter.

Figure 2. Architecture of the implemented DDoS filtering system without using NetFPGA SUME NIC.
Figure 2. Architecture of the implemented DDoS filtering system without using NetFPGA SUME
Regular
NIC. arrows
Regular represent
arrows “real” packet
represent “real” datapaths, dashed arrows
packet datapaths, dashedrepresent combined combined
arrows represent “real” packet
“real”
and metadata datapaths, and dotted arrows represent internal communication between
packet and metadata datapaths, and dotted arrows represent internal communication betweendifferent dif-
modules
ferent of the system.
modules of the system.

Two types of memory modules are used in the implementation: Block Random Access
The hardware implementation demonstrates the hybrid filter prototype datapath on
Memory (BRAM) and Quad Data Rate Static Random Access Memory (QDR SRAM).
the NetFPGA SUME development board, using the AXI4-Stream protocol for inter-mod-
BRAM is a memory integrated on the FPGA board with limited capacity and very low
ule communication
latency in the
(readout requires system
up to pipeline.
two clock cycles),The pipeline
while QDR isisancomposed of modules
external memory modulecon-
nected in series or parallel and consists of two parts: one part carries packets
with a larger capacity but slightly higher latency (about 20 clock cycles to readout). Both arriving from
the
types of memory are suitable for high-speed operation, which is why they are used for to
incoming network interface, i.e., packets that are checked (filtered) and forwarded
the output
packet interface
filtering. if necessary,
In particular, theyand control
store packets
the data from
required forthe
the“Distributor” thatofregulate
partial execution the
the
LPM internal logicorwithin
algorithm, the hardware
other partially (e.g., setting
offloadable memory
rule patterns values
which the or enabling
hardware andto
sends disa-
bling certain filter
the software parsers).
in theAll data required for filtering (e.g., source and destination IP ad-
metadata.
dresses) are extracted from the “real” packets, and metadata are created based on this
3.2. Software The metadata are appended to the end of the packets and forwarded to the
information.
softwareThe filter.
proposed system utilizes the Restricted Feature-set Packet Filter (RFPF)—a soft-
wareTwo filtertypes
developed in our previous
of memory modulesresearch
are used[36].
in RFPF is a high-performance
the implementation: BlockIPv4 traffic Ac-
Random
filter proven to be capable of filtering DDoS traffic at 10 G speeds
cess Memory (BRAM) and Quad Data Rate Static Random Access Memory (QDR SRAM). using only one CPU core.
It works by binding to two network interfaces using the Netmap software framework and
BRAM is a memory integrated on the FPGA board with limited capacity and very low
generates C code from a predefined rulelist. The generated code is converted into a dynam-
latency (readout requires up to two clock cycles), while QDR is an external memory mod-
ically executable program that is inserted between the network interfaces to filter traffic in
ule with a larger capacity but slightly higher latency (about 20 clock cycles to readout).
both directions. In this work, it is adapted to the hybrid mode of operation, considering
Both
how types of memory
the filtering are suitable
is performed for high-speed
in hardware and how the operation, which
information is the
from why they areare
metadata used
for
used in the software filtering. The software filter separates the metadata arriving from the of
packet filtering. In particular, they store the data required for the partial execution
the LPM algorithm,
hardware or other
from the packets partially offloadable
themselves and uses them ruleforpatterns which the hardware sends
further processing.
to the software filter in the metadata.

3.2. Software
The proposed system utilizes the Restricted Feature-set Packet Filter (RFPF)—a soft-
ware filter developed in our previous research [36]. RFPF is a high-performance IPv4 traf-
fic filter proven to be capable of filtering DDoS traffic at 10 G speeds using only one CPU
core. It works by binding to two network interfaces using the Netmap software framework
and generates C code from a predefined rulelist. The generated code is converted into a
Sensors 2023, 23, 7636 7 of 19

3.3. Rule Categorization

The rulesets used in the model consist of rules which are grouped and categorized by
whether they are filtered in hardware, software, or both components. The rules consist of
one action and one or more patterns, in the following format:

action pattern {pattern. . .}

The action can be terminating (if further packet inspection is halted after the rule
is matched, e.g., A—Accept or D—Deny) or non-terminating (N—if the rule matching
is continued even though the rule is matched). Additionally, the action may have a
counting (c—when the software component must be notified that the rule was matched) or
non-counting (n—if the rule does not require incrementing a counter) attribute. Patterns
are divided into three types: those that can be fully or partially processed in hardware
(pO —fully offloadable and pP —partially offloadable) and those that cannot be processed
in hardware (pN —non-offloadable). The combination of patterns in a rule determines the
overall rule offloadability attribute: fully offloadable (O), partially offloadable type 0 (P0 ),
partially offloadable type 1 (P1 ), partially offloadable type 2 (P2 ), and non-offloadable (N),
as shown in Table 1.

Table 1. Rule offloadability matched with pattern combination types. Multiple repetitions (once or
more) of types of patterns combination are marked with ()+ . Multiple repetitions (zero times or more)
of types of patterns are marked with ()*.

Rule Offloadability Combination of Patterns

O (pO )+
P0 (pN )+ (pO )+
P1 (pP )+ (pN )*
P2 (pP )+ (pN )*(pO )+
N (pN )+

Combining the pattern offloadability attributes (five options), terminating attributes

(three options), and counting attributes (two options), there are a total of 30 possible
combinations of rule types that can be categorized by how they can be processed in
hardware: fully, partially, or not at all. In the context of DDoS protection used in this
research, non-terminating actions with the non-counting attribute make no sense, regardless
of the pattern offloadability (*Nn); therefore, they are not considered. Table 2 presents the
remaining 25 combinations, categorized based on the offload type that they belong to.

Table 2. Offload types for rule categories.

Offload Type Rule Categories

Hardware ODn
OAc, OAn, ODc, ONc
P0 Ac, P0 An, P0 Dc, P0 Dn, P0 Nc
Hybrid
P1 Ac, P1 An, P1 Dc, P1 Dn, P1 Nc
P2 Ac, P2 An, P2 Dc, P2 Dn, P2 Nc
Software NAc, NAn, NDc, NDn, NNc

An example of a pseudo ruleset with several different rule types categorized accord-
ingly is shown and explained in Figure 3.
Sensors
Sensors2023,
2023,23,
23,7636
7636 8 of 1919
8 of

Figure3. 3.
Figure Three
Three types
types of rules,
of rules, annotated
annotated withwith
theirtheir categories.
categories. Therule
The first firstspecifies
rule specifies that TCP
that every every
TCP packet
packet with destination
with destination port 22port
should 22 should be dropped
be dropped and notand not counted
counted (Deny with (Deny with non-counting
non-counting attrib-
ute). The second
attribute). specifies
The second that every
specifies packet
that every withwith
packet destination portport
destination 80 or80 443 should
or 443 bebe
should forwarded
forwarded
and
andcounted
counted(Accept
(Acceptwith
withcounting
countingattribute).
attribute).TheThefirst
firsttwo
tworules
rulesare
areterminating—the
terminating—thefilterfilterstops
stops
parsing
parsinganyanysubsequent
subsequentrules
rulesif ifthis
thisrule
ruleisismatched.
matched.The Thethird
thirdrule
ruleisisnon-terminating.
non-terminating.It Itspecifies
specifies
that
thatpackets
packetswith
withthe
thesource
sourceIP IPaddress
address fromfrom the
the GOOD
GOOD table table and
and with
withdestination
destinationport
port8080should
shouldbe
be counted without any action—if there are rules after this one, they are checked.
counted without any action—if there are rules after this one, they are checked.

When
Whentaking
takinginto
intoconsideration
considerationthetheinformation
information totobebeexchanged
exchangedbetween
between the hard-
the hard-
ware
wareandandthe
thesoftware
softwareduring
duringthe
thefiltering
filteringprocess,
process,the
thecategories
categoriescan
canbebeclassified
classifiedinto
into
eight groups, with each group utilizing one of the four different metadata types,
eight groups, with each group utilizing one of the four different metadata types, as demon- as
demonstrated in Table
strated in Table 3: 3:
• • metadata1—data
metadata1—dataused usedininpartially
partiallyoffloaded
offloadedprocessing
processing(e.g.,(e.g.,protocol
protocoltype,
type,IPIPaddress
address
source/destination, port number, or partial data used for an
source/destination, port number, or partial data used for an LPM algorithm).LPM algorithm).
• • metadata2—data
metadata2—dataused usedinin fully
fully offloaded processing(a
offloaded processing (abinary
binaryvaluevaluefor
forevery
everyp0ppattern
0 pat-

tern from
from partially
partially offloaded
offloaded rules,rules, whether
whether the hardware
the hardware processingprocessing
matched matched
the rulethe
or it
rule
didor it did not).
not).
• • metadata3—data
metadata3—dataused usedforforallallcounting
countingrules
rules(a(abinary
binaryvalue
valueforforevery
everycounting
countingrule,
rule,
whether
whetherthethehardware
hardwareprocessing
processingmatched
matchedthetherule
ruleororititdid
didnot).
not).
• • metadata4—data
metadata4—dataused usedwhen
whena aterminating
terminatingrule
ruleisismatched
matched(8-bit(8-bitdata
datanoting
notingthetherule
rule
number
numberthat
thatfirst
firstmatched).
matched).

Table
Table3.3.Metadata
Metadatafields
fieldsfor
fordifferent
differentrule
rulecategories.
categories.The
The‘*’‘*’character
characterreplaces
replacesany
anyother
otherattribute.
attribute.
Metadata Type
Rules Metadata Type
Rules metadata1 metadata2 metadata3 metadata4
ODn metadata1
- metadata2
- metadata3
- metadata4
-/x
OAn ODn - - - - - - x-/x
ONcOAn - - - - x - - x
ONc
O[A|D]c - - - - - x x -
O[A|D]c
P0** - - x - - - - x
P ** - x - -
P1** 0 x - - -
P1 ** x - - -
P2**P ** x x x x - - - -
2
N**N** - - - - - - - -

As shown in Table 3, fully offloadable rules with the “Deny” terminating attribute and
As shown
no counting in Table
attribute 3, fully
(ODn) can offloadable
be offloaded rules
to hardware withoutterminating
with the “Deny” attribute and
sending metadata to
no counting attribute (ODn) can be offloaded to hardware without sending metadata to
software exclusively if they appear at the beginning of the ruleset. Otherwise, the result
software exclusively if they appear at the beginning of the ruleset. Otherwise, the result of
of their check must be sent to the software using the same metadata as the fully offloadable
Sensors
Sensors 2023,
2023, 23,
23, 7636
7636 9 9ofof19
19

rules
their with
checkthe “Accept”
must be sentterminating
to the software attribute
usingand theno samecounting
metadataattribute
as the(OAn). However,
fully offloadable
the implementation architecture used for this filter limits the inclusion
rules with the “Accept” terminating attribute and no counting attribute (OAn). However, of these two cate-the
gories (ODn and OAn) in the performance evaluation presented in
implementation architecture used for this filter limits the inclusion of these two categoriesthis paper.
(ODnThe andevaluation
OAn) in the is solely conducted
performance for all ofpresented
evaluation the otherin groups of rules (except for N**
this paper.
groupThe rules, which are
evaluation non-offloadable
is solely conductedby fordefault).
all of theThe matching
other groupsof ofthe fully
rules offloadable
(except for N**
non-terminating
group rules, which rulesarewith the counting attribute
non-offloadable by default).(ONc) Theshould only of
matching increment
the fullythe counter
offloadable
in the software,rules
non-terminating and with
so they therequire
countingone bit for(ONc)
attribute each should
rule that can
only be counted.
increment Fully of-
the counter in
floadable rules with both terminating and counting attributes (O[A|D]c)
the software, and so they require one bit for each rule that can be counted. Fully offloadable need to send only
the ordinal
rules number
with both of the first
terminating andterminating rule that(O[A|D]c)
counting attributes matches inneed hardware.
to sendForonly each
the combi-
ordinal
nation
number ofofpartially
the firstoffloadable
terminatingPrule 0 rules,
thatonly the results
matches of each fully
in hardware. offloadable
For each combination patternof
(p O) needoffloadable
partially to be sent to P0software
rules, only as athetrue/false
results ofbitmap.
each fullyForoffloadable
every combination
pattern (p ofOpartially
) need to
offloadable P1 rules, as
be sent to software thea hardware
true/falsecomputes
bitmap. For theevery
data for partially offloadable
combination of partiallypatterns (pP)
offloadable
and sendsthe
P1 rules, it hardware
in full to the software.
computes 2 rules
thePdata forhave the same
partially metadata
offloadable requirements
patterns as P0
(pP ) and sends
and
it inPfull
1 rules combined.
to the software. P2 rules have the same metadata requirements as P0 and P1 rules
combined.
3.4. Use Case
3.4. Use Case
Combined with an external DDoS detection system, the filtering system described in
Combined
this paper wouldwith an external
effectively DDoS
utilize the detection
LPM search system,
for IPthe filteringand
addresses system described
subnets againstin
this paper would effectively utilize the LPM search for IP addresses
DDoS attacks even with millions of different attackers in a high-speed networking envi- and subnets against
DDoS attacks
ronment. As aneven with millions
example, this kindof of different
mitigation attackers
could beinachieved
a high-speed
usingnetworking
only seven rules envi-
ronment.
with As an example,
six different this of
tables (lists kind
IP of mitigation
addresses andcould be achieved
subnets), as shown using only seven
in Figure 4. Asrules
the
with six different tables (lists of IP addresses and subnets), as
system is reconfigurable, various versions of rulesets could be made ready to be deployedshown in Figure 4. As the
system is reconfigurable, various
depending on the security status of the network.versions of rulesets could be made ready to be deployed
depending on the security status of the network.

Figure 4.
Figure Pseudoruleset
4. Pseudo rulesetexample
examplefor default security
for default security status.
status. The
The‘#’
‘#’characters
charactersdenote
denote comments.
comments.

All such rulesets include constant rules that always perform the same tasks, regardless
of theAll such rulesets
security level ofinclude
alert, asconstant rules that
well as variable onesalways perform
that change the same on
depending tasks,
the regard-
security
less of the security level of alert, as well as variable ones that change
status. The rules from Figure 4 allow certain safe source hosts/networks to all necessary depending on the
security status. The rules from Figure 4 allow certain safe source
parts of the internal network, possibly further specified by destination ports (ADMIN hosts/networks to all
necessary
table—e.g., third-party administrators that should always have access to the devices ports
parts of the internal network, possibly further specified by destination in the
(ADMIN
network).table—e.g., third-party
Access is blocked to alladministrators
other parts of the that shouldthat
network always havepublicly
are not access to the de-
accessible
vices
(PUBLICin thetable—e.g.,
network). IP Access is blocked
addresses to all other
of private email parts
serversof the network
or IoT thatand
sensors) areto
not pub-
known
licly accessible (PUBLIC table—e.g., IP addresses of private email servers
malicious IP addresses (BAD table—e.g., from publicly available collectors). Additionally, or IoT sensors)
and to considered
traffic known malicious IP addresses
to be suspicious (BAD table—e.g.,
is exclusively monitored from publicly
(SUSP available
table—e.g., collectors).
subnet ranges
Additionally, traffic considered to be
from regions known for espionage or DDoS attacks).suspicious is exclusively monitored (SUSP table—
e.g., subnet
The restranges
of the from
ruleset regions
depends known
on the for espionage
situation andormayDDoS attacks).
change depending on whether
The restisofunder
the network the ruleset
a DDoSdepends
attack and on how
the situation
severe it is.and In may change
the state depending
of normal network on
whether the network
activity (i.e., without DDoS is under a DDoS
attacks), anattack
externalandtool
how severe
that it is. regular
monitors In the state of accumu-
traffic normal
network activity
lates secure (i.e., withoutinDDoS
hosts/networks attacks),
a secure an external
table that is always tool that monitors
forwarded (GOOD regular traffic
table—e.g.,
accumulates
regular or unsuspicious users). Using LPM also allows for the fast classification of source ta-
secure hosts/networks in a secure table that is always forwarded (GOOD ad-
ble—e.g., regular or
dresses according to unsuspicious users).
their geo-location, andUsing LPM also
so it always allows specific
forwards for the fast classification
countries (GEOIP
Sensors 2023, 23, 7636 10 of 19

Sensors 2023, 23, 7636 10 of 19

of source addresses according to their geo-location, and so it always forwards specific

countries (GEOIP table—country for which the service is intended, neighboring countries,
or “friendly” countries).
table—country for which the All service
other traffic is forwarded,
is intended, but also
neighboring monitored
countries, by an external
or “friendly” coun-
tries). All other
automatic DDoStraffic
attackisdetection
forwarded, but also monitored by an external automatic DDoS
system.
attackDuring
detection
thesystem.
DDoS attack, instead of allowing all unknown traffic from the variable
part During the DDoS
of the ruleset, attack,
the filter instead
blocks of allowing
everything exceptallpotentially
unknownsecure traffic tables
from the
(GOOD variable
and
part of the ruleset, the filter blocks everything except potentially secure tables
GEOIP). Since all other traffic is monitored, this helps to isolate the bad IP addresses and (GOOD and
GEOIP).
adds them Since all other
to the trafficwhich
BAD table, is monitored,
is updated thisaccordingly.
helps to isolate the bad IPthe
If necessary, addresses and
filter would
adds them toreject
additionally the BAD table, which
the GEOIP table, is
asupdated
shown inaccordingly.
Figure 5, if itIfisnecessary,
proven tothe filter would
be unsafe, until
additionally reject the
the attack subsides or GEOIP table, as shown
all the attackers in Figure 5, if it is proven to be unsafe, until
are blocked.
the attack subsides or all the attackers are blocked.

Figure 5. Pseudo
Figure 5. Pseudo ruleset
ruleset example for high-alert
example for high-alert security
security status.
status. The
The ‘#’
‘#’ characters
characters denote
denote comments.
comments.
4. Benchmarks
4. Benchmarks
Since the packets in the implementation of this filter are prepared independently by
Since with
hardware, the packets in the implementation
their metadata created before of this filter
reaching theare prepared
input independently
interface of the software by
hardware, with their metadata created before reaching
component, the software is unaffected by how they were created. the input interface of the software
component, the validate
To test and softwarethe is unaffected by how
system without thethey
needwere created. hardware implementa-
for multiple
tions, the measurements were designed to ensure the independencehardware
To test and validate the system without the need for multiple of packetimplementa-
preparation
tions, the measurements were designed to ensure the independence
and metadata creation. This was achieved by simulating the hardware of packet
partpreparation
of the sys-
and by
tem metadata
creatingcreation. This was
the metadata achieved by simulating
programmatically. The pkt-genthe tool
hardware partin
(included ofthe
theNetmap
system
by creating was
framework) the metadata
used on aprogrammatically.
separate computerThe pkt-gen tool
to generate (included
packets, create in
thethe Netmap
necessary
framework)
metadata, wasitused
attach to theon a separate
packets, computer
and send them to generate
to the software packets, create
filter, as showntheinnecessary
Figure 6.
metadata, attach it to the packets, and send them to the software filter, as
The filtered traffic is verified in the traffic sink, which also calculates the throughput shown in Figure
of
6. The filtered
incoming traffic is verified in the traffic sink, which also calculates the throughput of
packets.
incoming packets.
The test results are based on the 60 s average, and the CPU cycle counter in the
software filter is implemented using the assembler instruction rdtsc [49], which acquires
the processor’s timestamp counter before and after processing each batch of packets.
To validate and verify the results of the simulated measurements, some of the mea-
surements were performed on a real hybrid system that used FPGA hardware to generate
metadata.

4.1. Results
We present a comparison of filtering results with and without offloading on hardware,
using different hardware offloading configurations. The average total throughput and the
average number of CPU cycles per packet were measured, and two types of measurements
were performed: using random and specific traffic. Both types used randomly generated
traffic with either completely random source IP addresses or specifically shaped traffic to
seem like a DDoS attack. The specific traffic consisted of a combination of “normal” random
traffic and randomly selected IP addresses from a large pool of malicious IP addresses.
In this way, we could demonstrate how the filter performed under pressure, when the
filtering load was high. All of the tests were set so that the bandwidth never reached
the maximum possible value for the system. Additionally, to allow for better control and
consistency of tests, all of the tests were performed on a system with a single CPU core at
reduced frequency. For this reason, the efficiency of the two filtering methods could be
 Sensors 2023, 23, 7636 11 of 19

Sensors 2023, 23, 7636 11 of 1

compared based on the number of packets processed per second and the number of CPU
cycles required to process one packet.

Figure6.6.Simulated
Figure Simulatedhardware testbed
hardware (bypassing
testbed the NetFPGA
(bypassing SUME).SUME).
the NetFPGA

The measurements were categorized by rulelists, which were tested with associated
The test results are based on the 60 s average, and the CPU cycle counter in the soft
metadata specific to the rules within them. Multiple measurements were conducted for
ware filter is
each rulelist toimplemented
obtain averageusingresultsthe
forassembler instruction
all of the hardware rdtsc [49],
offloading which acquires th
configurations:
processor’s
without timestamp
any hardware counter(only
offloading before theand after filter
software processing
withouteach batchas
metadata) ofapackets.
point of
To validate
reference, and thenandwithverify the results
modified of the
parameters forsimulated measurements,
hardware offloading. some of the meas
Each individual
urements
rulelist were of
consisted performed on a created
multiple rules, real hybrid system
in such a waythat used
to test the FPGA hardware
offloading to generat
of a single
metadata
metadata. type and, in some cases, combinations of multiple types of metadata. The detailed
explanations and extended results of these experiments can be found in [36].
4.1. The
Resultsrulesets used in tests were divided into two groups, depending on the type of
rules used in them. One group used only non-terminating rules, forcing the software filter
We present
to process a comparison
each of them of filtering
before forwarding results
the packet with
to the and interface.
egress without This
offloading
ensuredon hard
ware, using different hardware offloading configurations. The
that the same number of processing operations were performed for each packet, making average total throughpu
andfiltering
the the average number
comparable for of
allCPU
of thecycles perthe
tests for packet
same were
group.measured,
The average and two types of meas
software-only
filtering
urements throughput and cycle count
were performed: using for each type
random and of rulesettraffic.
specific in this Both
grouptypes
are shown
used in randomly
Figure 7.
generated traffic with either completely random source IP addresses or specifically
The traffic
shaped secondtogroup
seemused
like both
a DDoSnon-terminating
attack. The and terminating
specific (Accept) rules.
traffic consisted If the
of a combination o
packet matched the terminating rule, it no longer needed to be processed, and so the
“normal” random traffic and randomly selected IP addresses from a large pool of mali
subsequent rules were not checked. For this reason, the maximum throughput for these
cious IP addresses. In this way, we could demonstrate how the filter performed unde
tests was slightly higher than the throughput for the tests from the first group of rulesets.
pressure,
The averagewhen the filtering
software-only loadthroughput
filtering was high. and All cycle
of thecount
testsfor
wereeachset so of
type that the bandwidth
ruleset in
never reached the maximum
this group are shown in Figure 8. possible value for the system. Additionally, to allow for bet
ter control and consistency of tests, all of the tests were performed on a system with a
single CPU core at reduced frequency. For this reason, the efficiency of the two filtering
methods could be compared based on the number of packets processed per second and
the number of CPU cycles required to process one packet.
The measurements were categorized by rulelists, which were tested with associated
metadata specific to the rules within them. Multiple measurements were conducted fo
each rulelist to obtain average results for all of the hardware offloading configurations
without any hardware offloading (only the software filter without metadata) as a point o
 rules used in them. One group used only non-terminating rules, forcing the software filter
to process each of them before forwarding the packet to the egress interface. This ensured
that the same number of processing operations were performed for each packet, making
the filtering comparable for all of the tests for the same group. The average software-only
Sensors 2023, 23, 7636 filtering throughput and cycle count for each type of ruleset in this group are shown in
12 of 19
Figure 7.

(a)

(b)
Figure 7. Software-only average filtering throughput and average cycle count of rulesets without
terminating rules for (a) random traffic and (b) specific traffic.
 The second group used both non-terminating and terminating (Accept) rules. If the
packet matched the terminating rule, it no longer needed to be processed, and so the sub-
sequent rules were not checked. For this reason, the maximum throughput for these tests
was slightly higher than the throughput for the tests from the first group of rulesets. The
Sensors 2023, 23, 7636 average software-only filtering throughput and cycle count for each type of ruleset13 in this
of 19
group are shown in Figure 8.

(a)

(b)
Figure 8. Software-only average filtering throughput and average cycle count of rulesets with
terminating rules for (a) random traffic and (b) specific traffic.

Figure 9 illustrates the average CPU cycle count reduction achieved when using meta-
data for filter offloading, considering both random and specific traffic. The total throughput
increase closely correlates with the CPU cycle decrease, and so it is not shown.
 Figure 9 illustrates the average CPU cycle count reduction achieved when using
Sensors 2023, 23, 7636
metadata for filter offloading, considering both random and specific traffic. The14total of 19

throughput increase closely correlates with the CPU cycle decrease, and so it is not shown.

(a)

(b)
Figure 9.
Figure 9. Improvements in average
Improvements in average CPU
CPU cycle
cycle count
count for
for (a)
(a) random
random traffic
traffic and
and (b)
(b) specific
specific traffic.
traffic. Both
Both
types of ruleset are combined in this figure: with terminating rules (dark) and without terminating
types of ruleset are combined in this figure: with terminating rules (dark) and without terminating
rules (light).
rules (light).

The results
The results in
in Figure
Figure 99 use
use the
the metadata
metadata notation
notation fromfrom Table
Table 33 and
and show
show thatthat tests
tests
using rulesets with terminating rules (OAc) show the greatest improvements
using rulesets with terminating rules (OAc) show the greatest improvements for both types for both types
of traffic.
of traffic. For
For random
random traffic,
traffic, the
the greatest
greatest improvement
improvement is is seen
seen using
using the
the ruleset
ruleset with
with rules
rules
that partially offload the LPM algorithm, combined with rules that
that partially offload the LPM algorithm, combined with rules that fully offload simple fully offload simple
port checks
port checks to
to hardware
hardware withwith 20.2%
20.2% fewer
fewer CPU
CPU cycles.
cycles. For
For specific
specific traffic,
traffic, it it is
is the
the variation
variation
in aa ruleset
in ruleset that
that fully
fully offloads
offloads simple
simple port
port checks
checks to
to hardware
hardware with with28.9%
28.9%fewer
fewercycles.
cycles.
The second highest test in both random and specific traffic cases uses
The second highest test in both random and specific traffic cases uses the “realistic” the “realistic”
version of
version of the
the LPM
LPM ruleset
ruleset with
with rules
rules that
that can
can be
be assumed
assumed to to be
be used
used in in aa realistic
realistic scenario,
scenario,
similar to the rules shown in Figure 4. Using the P
similar to the rules shown in Figure 4. Using the P1 ** partial LPM offloading, it it
1 ** partial LPM offloading, achieved
achieved a
a 17.8%
17.8% reduction
reduction ininCPU
CPUcycles
cyclesforforrandom
randomandand23.6%
23.6%for forspecific
specifictraffic.
traffic. Other
Other cases
cases with
with
the highest improvements for both types of traffic are those combining P1 ** or P2 ** partial
LPM offloading with other metadata for an around 10% reduction in CPU cycles.
As previously mentioned, the results shown are all based on tests using metadata
pre-generated by the packet generator. To test the hardware partwww.mdpi.com/journal/sensors
Sensors 2023, 23, 7636. https://doi.org/10.3390/s23177636 of the hybrid system, i.e.,
Sensors 2023, 23, 7636 2 of 6

the highest improvements for both types of traffic are those combining P1** or P2** partial
Sensors 2023, 23, 7636 LPM offloading with other metadata for an around 10% reduction in CPU cycles. 15 of 19
As previously mentioned, the results shown are all based on tests using metadata
pre-generated by the packet generator. To test the hardware part of the hybrid system,
i.e., how the system works when the NetFPGA generates the metadata and attaches it to
how the system works when the NetFPGA generates the metadata and attaches it to the
the packets, another set of tests was performed.
packets, another set of tests was performed.
The test environment (testbed) for the hybrid tests, as shown in Figure 10, was similar
The test environment (testbed) for the hybrid tests, as shown in Figure 10, was similar
to the one used when the hardware part was simulated by the software metadata genera-
to the one used when the hardware part was simulated by the software metadata generator.
tor. The traffic generator is connected to the NetFPGA SUME ingress interface, and the
The traffic generator is connected to the NetFPGA SUME ingress interface, and the SUME
SUME egress interface is connected to the software filter ingress interface.
egress interface is connected to the software filter ingress interface.

Figure 10. Hybrid hardware/software system testbed.

Figure 10. Hybrid hardware/software system testbed.
The tests for this hybrid system were performed in the same manner as the simulated
The tests for this hybrid system were performed in the same manner as the simulated
tests, except that the packet generator did not have to create metadata and attach it to the
tests, except
packets. For that
this the packet
reason, generator
it was didthat
expected not have to create
the results metadata
of all and performed
of the tests attach it to the
on
packets. For this reason, it was expected that the results of all
the hybrid system would match the results of the simulated hardware. In all of the tests performed on
of the tests
the hybrid system would match the results of the simulated hardware.
that were performed with the hybrid system, the average results of the hybrid system In all of the tests
that werethe
matched performed with the
average results hybrid
of the system,
simulation the average
almost perfectly.results of theresults,
From these hybriditsystem
can be
matched the average results of the simulation almost perfectly. From these
inferred that it is possible to achieve the same level of improvement over non-offloaded results, it can
be inferred
filtering that
with it is possible
other types of to achieve if
metadata the same level
specific of improvement
offload over
capabilities are non-offloaded
implemented in
filtering with
the hardware. other types of metadata if specific offload capabilities are implemented in
the hardware.
4.2. Improving Hybrid Filtering
4.2. Improving
The extentHybrid Filtering
of improvement in each test may vary depending on the capabilities of
The extentwith
the hardware, of improvement
performancein eachfurther
being test may vary depending
enhanced as moreon the capabilities
metadata of the
are offloaded
hardware,
to with performance
the hardware. being further
Among the different enhanced
metadata types,asmetadata1
more metadata
requiresaretheoffloaded
fewest and to
the hardware.
least Among theupdates
complex hardware different tometadata
achieve atypes, metadata1
significant requires
increase the fewest
in DDoS and least
protection. For
example, to increaseupdates
complex hardware the totalto number
achieve of available LPM
a significant tables
increase for hardware
in DDoS offloading,
protection. For ex-
additional memory the
ample, to increase needs to be
total installed
number in the hardware.
of available The use
LPM tables for of this additional
hardware memory
offloading, ad-
would not significantly impact the internal FPGA logic and overall system performance.
ditional memory needs to be installed in the hardware. The use of this additional memory
On the other
would hand, otherimpact
not significantly metadata thetypes require
internal FPGAmore complicated
logic and overall changes,
system as parallelism
performance.
cannot be leveraged as efficiently as in the case of metadata1. This could lead to delays and
performance degradation, especially when offloading a large number of rules to hardware.
Improving the performance of the filter that uses hardware offloading means finding
a balance between the size of the metadata and their usefulness. Responding to changes
Sensors 2023, 23, 7636 16 of 19

in the type and volume of traffic is also one of the most important matters to consider
when offloading and even beforehand when creating the ruleset. Therefore, in cases where
offloading has a negative impact on throughput, it can be bypassed and replaced with a
better configuration.
It is worth repeating that all of the tests (including those performed with real hardware
offloading) were performed on a system with a single CPU core at a reduced frequency.
Moreover, they were performed on a system corresponding to the model shown in Figure 2.
The results of a hybrid system without the limitations of this model would certainly be
even better.

5. Conclusions
In this paper, we presented a datapath model of a high-speed network traffic classi-
fier/filter based on a hybrid hardware/software combination of FPGA and off-the-shelf
computer software. The hardware component model comprises a reconfigurable FPGA
datapath capable of adapting to runtime packet classification changes in near real-time.
On the other hand, the software component is a modified version of the filter used in
our previous research, now equipped with additional functionality to receive metadata
from the hardware. This integration allows for more efficient packet filtering, leading to
improved performance over software-only packet filtering.
We have shown that the hybrid system can achieve filtering in networks with speeds
of 10 Gbps by heuristically distributing the workload between the hardware and software
components. This is achieved by carefully selecting packet filtering methods and metadata
that can be offloaded to the hardware, which optimizes the throughput of the system. To
test the system, we developed a method to empirically evaluate the distribution of the
workload between the hardware and software components. It bypasses the development
of complex hardware implementations by simulating the necessary offloading of hardware
to software.
The implemented model shows performance improvements in tests that include both
random traffic and traffic specifically designed to simulate DDoS attacks. It is shown
that offloading different types of rules to hardware, fully or partially, results in varying
performance improvements, with reductions of up to 30% in CPU cycles for certain offloads
and rule types. In packet filtering, the use of rules based on LPM offers the advantages
of higher throughput and simpler, more manageable rulesets. Therefore, these rulesets
are well suited for DDoS protection, and their effectiveness can be further enhanced by
hardware offloading.
In addition, the scalability of such a system should be emphasized, because efficient
use of the LPM algorithm for IP address lookup means that filtering does not depend on the
number of rules, but on the method of offloading parts of the filtering to the hardware. With
suitable hardware, it is expected that the improvement in such a system can be maintained
at higher speeds.

Author Contributions: Conceptualization, D.S. and M.M.; methodology, D.S. and M.M.; software,
D.S.; validation, D.S.; investigation, D.S.; resources, M.M.; data curation, D.S.; writing—original
draft preparation, D.S.; writing—review and editing, D.S. and M.M.; visualization, D.S. and M.M.;
supervision, M.M. All authors have read and agreed to the published version of the manuscript.
Funding: This work has been supported by the Croatian Science Foundation under the project
IP-2019-04-1986.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: All data analyzed during this study are included in this article.
Conflicts of Interest: The authors declare no conflict of interest.
Sensors 2023, 23, 7636 17 of 19

References
1. Google. Exponential Growth in DDoS Attack Volumes. Available online: https://cloud.google.com/blog/products/identity-
security/identifying-and-protecting-against-the-largest-ddos-attacks (accessed on 24 July 2023).
2. Microsoft. 2022 in Review: DDoS Attack Trends and Insights. Available online: https://www.microsoft.com/en-us/security/
blog/2023/02/21/2022-in-review-ddos-attack-trends-and-insights/ (accessed on 24 July 2023).
3. Cloudflare. DDoS Threat Report for 2023 Q1. Available online: https://blog.cloudflare.com/ddos-threat-report-2023-q1/
(accessed on 24 July 2023).
4. RIPE Labs. IPv6 10 Years Out: An Analysis in Users, Tables, and Traffic. Available online: https://labs.ripe.net/author/wilhelm/
ipv6-10-years-out-an-analysis-in-users-tables-and-traffic/ (accessed on 24 July 2023).
5. Lakshminarayanan, K.; Rangarajan, A.; Venkatachary, S. Algorithms for advanced packet classification with ternary CAMs. ACM
SIGCOMM Comput. Commun. Rev. 2005, 35, 193–204. [CrossRef]
6. Kannan, K.; Banerjee, S. Compact TCAM: Flow Entry Compaction in TCAM for Power Aware SDN. In Distributed Computing and
Networking. ICDCN 2013; Lecture Notes in Computer Science; Frey, D., Raynal, M., Sarkar, S., Shyamasundar, R.K., Sinha, P., Eds.;
Springer: Berlin/Heidelberg, Germany, 2013; Volume 7730. [CrossRef]
7. Rizzo, L. Netmap: A novel framework for fast packet I/O. In Proceedings of the 21st USENIX Security Symposium (USENIX
Security 12), Bellevue, WA, USA, 8–10 August 2012; pp. 101–112.
8. Intel. Data Plane Development Kit (DPDK*). Available online: https://www.intel.com/content/www/us/en/developer/topic-
technology/networking/dpdk.html (accessed on 24 July 2023).
9. Miano, S.; Bertrone, M.; Risso, F.; Tumolo, M.; Bernal, M.V. Creating complex network services with EBPF: Experience and lessons
learned. In Proceedings of the 2018 IEEE 19th International Conference on High Performance Switching and Routing (HPSR),
Bucharest, Romania, 18–20 June 2018; pp. 1–8. [CrossRef]
10. McKeown, N.; Anderson, T.; Balakrishnan, H.; Parulkar, G.; Peterson, L.; Rexford, J.; Shenker, S.; Turner, J. OpenFlow: Enabling
innovation in campus networks. ACM SIGCOMM Comput. Commun. Rev. 2008, 38, 69–74. [CrossRef]
11. Krishnamurthy, B.; Wills, C.; Zhang, Y. On the use and performance of content distribution networks. In Proceedings of the 1st
ACM SIGCOMM Workshop on Internet Measurement, San Francisco, CA, USA, 1–2 November 2001; pp. 169–182. [CrossRef]
12. Molnár, L.; Pongrácz, G.; Enyedi, G.; Kis, Z.L.; Csikor, L.; Juhász, F.; Kőrösi, A.; Rétvári, G. Dataplane specialization for high-
performance OpenFlow software switching. In Proceedings of the 2016 ACM SIGCOMM Conference, Florianopolis, Brazil,
22–26 August 2016; pp. 539–552. [CrossRef]
13. Mauricio, L.A.; Rubinstein, M.G.; Duarte, O.C. Proposing and evaluating the performance of a firewall implemented as a
virtualized network function. In Proceedings of the 2016 7th International Conference on the Network of the Future (NOF),
Búzios, Brazil, 16–18 November 2016; pp. 1–3. [CrossRef]
14. Imthiyas, M.; Wani, S.; Abdulghafor, R.A.A.; Ibrahim, A.A.; Mohammad, A.H. Ddos mitigation: A review of content delivery
network and its ddos defence techniques. Int. J. Perceptive Cogn. Comput. 2020, 6, 67–76.
15. Pacífico, R.D.; Castanho, M.S.; Vieira, L.F.; Vieira, M.A.; Duarte, L.F.; Nacif, J.A. Application layer packet classifier in hardware.
In Proceedings of the 2021 IFIP/IEEE International Symposium on Integrated Network Management (IM), Bordeaux, France,
18–20 May 2021; pp. 515–522.
16. Li, B.; Tan, K.; Luo, L.; Peng, Y.; Luo, R.; Xu, N.; Xiong, Y.; Cheng, P.; Chen, E. Clicknp: Highly flexible and high performance
network processing with reconfigurable hardware. In Proceedings of the 2016 ACM SIGCOMM Conference, Florianopolis, Brazil,
22–26 August 2016; pp. 1–14. [CrossRef]
17. Chen, M.S.; Liao, M.Y.; Tsai, P.W.; Luo, M.Y.; Yang, C.S.; Yeh, C.E. Using netfpga to offload linux netfilter firewall. In Proceedings
of the 2nd North American NetFPGA Developers Workshop, Stanford, CA, USA, 12–13 August 2010.
18. Fiessler, A.; Hager, S.; Scheuermann, B.; Moore, A.W. HyPaFilter: A versatile hybrid FPGA packet filter. In Proceedings of the
2016 Symposium on Architectures for Networking and Communications Systems, Santa Clara, CA, USA, 17–18 March 2016;
pp. 25–36. [CrossRef]
19. Fiessler, A.; Lorenz, C.; Hager, S.; Scheuermann, B.; Moore, A.W. Hypafilter+: Enhanced hybrid packet filtering using hardware
assisted classification and header space analysis. IEEE/ACM Trans. Netw. 2017, 25, 3655–3669. [CrossRef]
20. Weaver, N.; Paxson, V.; Gonzalez, J.M. The shunt: An FPGA-based accelerator for network intrusion prevention. In Proceedings of
the 2007 ACM/SIGDA 15th international symposium on Field Programmable Gate Arrays, Monterey, CA, USA, 18–20 February
2007; pp. 199–206.
21. Kalia, A.; Zhou, D.; Kaminsky, M.; Andersen, D.G. Raising the Bar for Using GPUs in Software Packet Processing. In Proceedings
of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), Oakland, CA, USA, 4–6 May
2015; pp. 409–423.
22. Go, Y.; Jamshed, M.A.; Moon, Y.; Hwang, C.; Park, K. APUNet: Revitalizing GPU as packet processing accelerator. In Proceedings
of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17), Boston, MA, USA, 27–29 March
2017; pp. 83–96.
23. Sun, W.; Ricci, R. Fast and flexible: Parallel packet processing with GPUs and click. In Proceedings of the IEEE Architectures for
Networking and Communications Systems, San Jose, CA, USA, 21–22 October 2013; pp. 25–35. [CrossRef]
Sensors 2023, 23, 7636 18 of 19

24. Vasiliadis, G.; Koromilas, L.; Polychronakis, M.; Ioannidis, S. GASPP: A GPU-Accelerated stateful packet processing framework.
In Proceedings of the 2014 USENIX Annual Technical Conference (USENIX ATC 14), Philadelphia, PA, USA, 19–20 June 2014;
pp. 321–332.
25. Han, S.; Jang, K.; Park, K.; Moon, S. PacketShader: A GPU-accelerated software router. ACM SIGCOMM Comput. Commun. Rev.
2010, 40, 195–206. [CrossRef]
26. Miano, S.; Doriguzzi-Corin, R.; Risso, F.; Siracusa, D.; Sommese, R. Introducing smartnics in server-based data plane processing:
The ddos mitigation use case. IEEE Access 2019, 7, 107161–107170. [CrossRef]
27. Kaufmann, A.; Peter, S.; Sharma, N.K.; Anderson, T.; Krishnamurthy, A. High performance packet processing with flexnic. In
Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating
Systems, Atlanta, GA, USA, 2–6 April 2016; pp. 67–81. [CrossRef]
28. Li, B.; Ruan, Z.; Xiao, W.; Lu, Y.; Xiong, Y.; Putnam, A.; Chen, E.; Zhang, L. Kv-direct: High-performance in-memory key-value
store with programmable NIC. In Proceedings of the 26th Symposium on Operating Systems Principles, Shanghai, China,
28–31 October 2017; pp. 137–152.
29. Bertin, G. XDP in practice: Integrating XDP into our DDoS mitigation pipeline. In Proceedings of the Technical Conference on Linux
Networking, NetDev, Le Westin Montréal, Canada, 6–8 April 2017; The NetDev Society: Nepean, ON, Canada, 2017; Volume 2,
pp. 1–5.
30. Deepak, A.; Huang, R.; Mehra, P. eBPF/XDP based firewall and packet filtering. In Proceedings of the Linux Plumbers Conference,
Vancouver, BC, Canada, 13–15 November 2018.
31. Kirdan, E.; Raumer, D.; Emmerich, P.; Carle, G. Building a traffic policer for ddos mitigation on top of commodity hardware. In
Proceedings of the IEEE 2018 International Symposium on Networks, Computers and Communications (ISNCC), Rome, Italy,
19–21 June 2018; pp. 1–5. [CrossRef]
32. AL-Musawi, B.Q.M. Mitigating DoS/DDoS attacks using iptables. Int. J. Eng. Technol. 2012, 12, 101–111.
33. Kaspersky. How to Not Break the Internet. Available online: https://www.kaspersky.com/blog/attack-on-dyn-explained/13325/
(accessed on 24 July 2023).
34. Red Button. Dyn (DynDNS) DDoS Attack Analysis. Available online: https://www.red-button.net/blog/dyn-dyndns-ddos-
attack/ (accessed on 24 July 2023).
35. CNBC. Massive Cyber Attack ‘Sophisticated, Highly Distributed’, Involving Millions of IP Addresses. Available online: https:
//www.cnbc.com/2016/10/22/ddos-attack-sophisticated-highly-distributed-involved-millions-of-ip-addresses-dyn.html (ac-
cessed on 24 July 2023).
36. Salopek, D. Hybrid Hardware/Software Datapath for Near Real-Time Reconfigurable High-Speed Packet Filtering. Ph.D. Thesis,
Department of Telecommunications, Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb, Croatia, 2022.
37. Salopek, D.; Zec, M.; Mikuc, M.; Vasić, V. Surgical DDoS Filtering with Fast LPM. IEEE Access 2022, 10, 4200–4208. [CrossRef]
38. Zec, M.; Mikuc, M. Pushing the envelope: Beyond two billion IP routing lookups per second on commodity CPUs. In Proceedings
of the IEEE 2017 25th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split,
Croatia, 21–23 September 2017; pp. 1–6. [CrossRef]
39. Zec, M. Improving Performance in Software Internet Routers through Compact Lookup Structures and Efficient Datapaths. Ph.D.
Thesis, Department of Telecommunications, Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb,
Croatia, 2019.
40. Magyari, A.; Chen, Y. Review of state-of-the-art FPGA applications in IoT Networks. Sensors 2022, 22, 7496. [CrossRef] [PubMed]
41. Zilberman, N.; Audzevich, Y.; Covington, G.A.; Moore, A.W. NetFPGA SUME: Toward 100 Gbps as research commodity. IEEE
Micro 2014, 34, 32–41. [CrossRef]
42. Zilberman, N.; Audzevich, Y.; Kalogeridou, G.; Manihatty-Bojan, N.; Zhang, J.; Moore, A. NetFPGA: Rapid prototyping of
networking devices in open source. ACM SIGCOMM Comput. Commun. Rev. 2015, 45, 363–364. [CrossRef]
43. Zilberman, N.; Audzevich, Y.; Kalogeridou, G.; Bojan, N.M.; Zhang, J.; Moore, A.W. NetFPGA-rapid prototyping of high
bandwidth devices in open source. In Proceedings of the IEEE 2015 25th International Conference on Field Programmable Logic
and Applications (FPL), London, UK, 2–4 September 2015; p. 1. [CrossRef]
44. Su, T.; You, L.; Wang, Q.; Hou, C. The high speed switching experiment based on NetFPGA SUME. In Proceedings of the IEEE
2016 11th International Conference on Computer Science & Education (ICCSE), Nagoya, Japan, 23–25 August 2016; pp. 652–657.
[CrossRef]
45. Lai, Y.K.; Huang, P.Y.; Lee, H.P.; Tsai, C.L.; Chang, C.S.; Nguyen, M.H.; Lin, Y.J.; Liu, T.L.; Chen, J.H. Real-time ddos attack
detection using sketch-based entropy estimation on the netfpga sume platform. In Proceedings of the IEEE 2020 Asia-Pacific
Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Auckland, New Zealand,
7–10 December 2020; pp. 1566–1570.
46. Gondaliya, H.; Sankaran, G.C.; Sivalingam, K.M. Comparative evaluation of IP address anti-spoofing mechanisms using a
P4/NetFPGA-based switch. In Proceedings of the 3rd P4 Workshop in Europe, Barcelona, Spain, 1 December 2020; pp. 1–6.
[CrossRef]
47. Rodrigues, P.; Saquetti, M.; Bueno, G.; Cordeiro, W.; Azambuja, J. Virtualization of programmable forwarding planes with p4vbox.
J. Integr. Circuits Syst. 2021, 16, 1–8. [CrossRef]
Sensors 2023, 23, 7636 19 of 19

48. Github. NetFPGA SUME Reference NIC. Available online: https://github.com/NetFPGA/NetFPGA-SUME-public/wiki/

NetFPGA-SUME-Reference-NIC (accessed on 24 July 2023).
49. Intel. Intel 64 and IA-32 Architectures Software Developer’s Manual. Available online: https://www.intel.com/content/dam/www/
public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-2b-manual.pdf (accessed on 24 July 2023).

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.