Application of huge pages

to the netmap platform

School of Electrical Engineering, University of Belgrade

TELFOR 2017
Introduction
• Netmap platform
• Huge pages
• Motivation for implementing huge pages to the
netmap platform
• Testing the implementation and results

TELFOR 2017
Netmap platform
• General purposes hardware is constantly improving
performance
• Network cards with high speed ports (10 Gbit/s, 25
Gbit/s, 40 Gbit/s, 100 Gbit/s…)
• Flexible general purposes hardware with large
number of high speed network ports
• Linux, Unix, Windows… (not optimized for high
speed packet processing)

TELFOR 2017
Netmap platform
• Netmap platform bypass kernel network stack
• High speed I/O interface for applications in user
space
• Netmap optimization methods (simple packet
representation, zero-copy…)
• All packet processing functions must be
implemented by application
• If netmap based applications are not performance
bottleneck, the main time consuming instructions
are accessing memory locations

TELFOR 2017
Huge pages
8 GB RAM - 8 GB RAM - huge
16 GB RAM – normal pages
normal pages pages

4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
2 MB
4K 4K 4K 4K 4K 4K 4K 4K 4K 4K

4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K

4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K

4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
2 MB
4K 4K 4K 4K 4K 4K 4K 4K 4K 4K

4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
2 MB
4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
. .
. .
. .

TELFOR 2017
Translation Lookaside Buffer (TLB)

Virtual address
VPage # offset

TLB
VPage # PPage# Page table
VPage # PPage# Miss
..
.
VPage # PPage#

Hit

PPage # offset
Physical address

TELFOR 2017
Motivation
• Network ports in netmap mode allocate a large
number of memory locations
• High probability of TLB misses
• Translation of virtual memory addresses to physical
memory addresses might be one of bottlenecks in
packet processing path
• Implementing huge pages can lower TLB misses
and improve performance

TELFOR 2017
Netmap structures
netmap_if netmap rings
num_rings ring_size pkt_buf
ring_ofs[] cur
pkt_buf
flags
buf_ofs
flags len index
pkt_buf

pkt_buf

Shared memory region

TELFOR 2017
Huge page implementation
• When network port is initialized in netmap mode,
packet buffers are allocated in huge pages memory
region instead of regular pages memory region
• Much smaller number of TLB entries point to
packet buffer memory locations

TELFOR 2017
Netmap application changes
• Additional fields are defined in the nmreq structure
in file netmap.h (directory sys/net)
• In the user application global variable huge_pages
needs to be set to 1
• Specifying the number of huge pages to be used by
the netmap platform with parameter
nmr.nr_huge_pages

TELFOR 2017
Testing environment A
M1 M2
DPDK netmap with
hugepages

if 1 10 Gbit/s if 1

bridge
if 2 10 Gbit/s if 2
pktgen

if 3 if 3

if 4 if 4

TELFOR 2017
Testing environment B
M1 M2
DPDK netmap with
hugepages

if 1 10 Gbit/s if 1

bridge
if 2 10 Gbit/s if 2
pktgen

if 3 10 Gbit/s if 3

bridge
if 4 10 Gbit/s if 4

TELFOR 2017
Results for testing environment A
PACKET THROUGHPUTS IN CASE WITH 2 INTERFACES FOR REGULAR PAGE SIZES OF 4KB
Packet Size[B] Tx [Mpps] Rx[Mpps]
64 29.382056 15.714904
128 16.890961 14.193723
192 11.791818 11.197840

PACKET THROUGHPUTS IN CASE WITH 2 INTERFACES FOR HUGE PAGE SIZES OF 2MB
Packet Size[B] Tx [Mpps] Rx[Mpps]
64 29.255485 21.557440
128 16.890955 14.931207
192 11.791893 11.791517
THROUGHPUT OF RECEIVED PACKETS SIGNIFICANTLY INCREASES FROM 15.7 MPPS TO
21.5 MPPS, I.E. BY 37%

TELFOR 2017
Results for testing environment B
PACKET THROUGHPUTS IN CASE WITH 4 INTERFACES FOR REGULAR PAGE SIZES OF 4KB
Packet Size[B] Tx [Mpps] Rx[Mpps]
64 38.315223 17.672381
128 33.775383 18.078533
192 23.595031 13.997585

PACKET THROUGHPUTS IN CASE WITH 4 INTERFACES FOR HUGE PAGE SIZES OF 2MB
Packet Size[B] Tx [Mpps] Rx[Mpps]
64 37.789986 20.571017
128 33.689274 16.914281
192 23.583477 14.343598
THROUGHPUT OF RECEIVED PACKETS SIGNIFICANTLY INCREASES FROM 17.7 MPPS TO
20.6 MPPS, I.E. BY 16.3%

TELFOR 2017
Conclusion
• Implementation of huge pages within the netmap
platform provides significant speedups of packet
processing in the case of short packets
• Configuration of a netmap application to use huge
pages is simple, only two parameters need to be
specified, huge_pages and nr_huge_pages, when
the application starts

TELFOR 2017
Q&A

TELFOR 2017